Project: News Application- part 1
Objective:
1. Implement a distributed system using GKE.
2. Connect and debug the components of the distributed system.
3. Apply patterns from previous labs.
Repository:
https://github.com/GeorgeDaoud3/SOFE4790U_Project
System Description:
The Local News application is used as a sample in the book Kubernetes Native Development by Benjamin
Schmeling and Max Dargatz. It serves to demonstrate a wealth of concepts to use Kubernetes along the
software lifecycle.
Figure 1 News Application
As the name implies, it is about displaying news. The location where the respective news is placed on the
map depends on the location mentioned in the news text. An example would be a news article with the
following title: "Next Sunday, there will be the Museum Embankment Festival in Frankfurt". The Local
News application will analyze the text based on Natural Language Processing (NLP), will find out that the
news refers to Frankfurt, and will place it into the city of Frankfurt on the map. Figure 1 shows a screenshot
of the user interface for the local news application. The pins represent the respective news and when the
user clicks on it will display the details about the news.
Figure 2 System Components
Figure 2 provides a brief overview of the components.
•
•
•
A Feed-Scraper is regularly polling a set of given news feeds and will send the news data such as
title and description to the News-Backend.
The News-Backend first stores the news data in a database and then sends a request to the
Location-Extractor to analyze the text, extract the name of the location (only the first location
that is found is considered), and return it to the backend. The response will contain the longitude
and latitude of the location which will then be added to the original news data and updated in the
database.
If a user wants to see the map with the news, she can use the News-Frontend. This web
application renders the map with the news by requesting the news, in the bounds defined by the
map's current clipping, from the News-Backend. The backend queries its database and returns
the news with the stored location. The front end marks the map with the news' location and if the
user hovers over the markers she will see the news text.
Deployment Procedure:
1. Let’s start by cloning the GitHub repository.
cd ~
git clone https://github.com/GeorgeDaoud3/SOFE4790U_Project.git
2. Then, a Docker image will be created for each component
a) The first one is the feed scraper. Run the following commands after changing <project-id> to
your project id.
cd ~/SOFE4790U_Project/feed-scraper
docker build . -t us.gcr.io/<project-id>/feed-scraper
docker push us.gcr.io/<project-id>/feed-scraper
b) Then, the location Extractor will be created. Again, run the following commands after
changing <project-id> to your project id.
cd ~/SOFE4790U_Project/location-extractor
docker build . -t us.gcr.io/<project-id>/location-extractor
docker push us.gcr.io/<project-id>/location-extractor
c) The next component is the news-backend. Run the following commands after changing
<project-id> to your project id.
cd ~/SOFE4790U_Project/news-backend
docker build . -t us.gcr.io/<project-id>/news-backend
docker push us.gcr.io/<project-id>/news-backend
d) Finally, the docker image of the news-frontend will be created. As usual, change <project-id>
to your project id before running the following commands.
cd ~/SOFE4790U_Project/news-frontend
docker build . -t us.gcr.io/<project-id>/news-frontend
docker push us.gcr.io/<project-id>/news-frontend
e) Note that the used database is of type PostGIS which can store geographic objects in the
PostgreSQL object-relational database. The docker image of PostGIS is pre-existed one as can
be accessed as postgis/postgis. Thus, there is no need now to create it.
3. Create GKE deployments and services for the system. This will follow the following order to
maintain the dependencies of the components.
a) Deploy the PostGIS database into GKE. Both deployment and a ClusterIP service will be
created. Note that the important environment variables in the YAML file are the username,
password, and database name.
cd ~/SOFE4790U_Project
kubectl create -f postgis.yaml
b) Then, deploy the location-extractor component into GKE. Both deployment and a ClusterIP
service will be created.
• First, edit the file ~/SOFE4790U_Project/location-extractor.yaml and update the 34th line
by replacing <project-id> with your project id.
• Then, run the following commands
cd ~/SOFE4790U_Project
kubectl create -f location-extractor.yaml
c) The next step is to deploy the back-end component into GKE. Both deployment and a Load
Balancer service will be created.
• First, edit the file ~/SOFE4790U_Project/news-backend.yaml and update the 40th line by
replacing <project-id> with your project id.
• Then, run the following commands
cd ~/SOFE4790U_Project
kubectl create -f news-backend.yaml
• Note that the news-backend.yaml contains a reference to both PostGIS and locationextractor services as environment variables
d) Now, it’s time to deploy the feed-scraper components. It’s will feed news into the backend
for processing and storage. As no other component will access it, only GKE deployment is
needed. Thus, no services will be deployed for that component.
• First, edit the file ~/SOFE4790U_Project/feed-scraper.yaml and update the 23rd line by
replacing <project-id> with your project id.
• Then, run the following commands
cd ~/SOFE4790U_Project
kubectl create -f feed-scraper.yaml
• Note that the feed-scraper.yaml contains a reference to the backend service as an
environment variable.
e) Finally, the front end will be deployed. The front end will access the backend via its external
IP.
• The first step is to find the external IP of the news-backend service by running
kubectl get services
• First, edit the file ~/SOFE4790U_Project/news-frontend.yaml and update the 40th line by
replacing <project-id> with your project id and the 39th line by replacing <backend IP
address> with the external IP of the news-backend service.
• Then, run the following commands to deploy both the deployment and a LoadBalancer
service into GKE
cd ~/SOFE4790U_Project
kubectl create -f news-frontend.yaml
• Note that the news-frontend.yaml contains only a reference to the backend service as an
environment variable.
f) To run the front end,
• get its external IP. Note, you may wait until an IP is assigned to the service.
kubectl get services
• Navigate it in a browser via http://<frontend-IP>/
4. (optional) You may follow the following steps to debug components.
a) Get the name of pods for each component
kubectl get pods
b) Check the logs of the feed scraper. The logs will contain any errors and logs of processing
news
kubectl logs <feed-scraper pod>
c) Check the logs of the location-extractor components. The logs will contain any errors and logs
of processing locations.
kubectl logs < location-extractor pod>
d) Check the logs of the backend-extractor components. The logs will contain any errors and logs
of processing news.
kubectl logs < news-backend pod>
e) Check the content of the PostGIS database.
• Log into the database CLI
f)
kubectl exec -it <postgis pod> -- psql -d news -U postgres
• Display the table
select * from news limit 4;
• Exit from the database CLI by typing
exit
To call the location-extractor service, we can use the PostGIS shell as
• Log into the shell
kubectl exec --stdin --tty <postgis pod> -- /bin/bash
• Install curl (only once)
apt update
apt install curl
• Call the location extractor service by typing this single-line command.
curl -X GET locationextractor:8081/get_loc?text=Luiz+In%C3%A1cio+Lula+da+Silva+returns+to+power+after+na
rrowly+winning+the+country%27s+presidential+election.
• Exit the shell
exit
Design:
the following patterns/changes are to be applied to the distributed system.
1. Deploy the location-extractor function (get_loc) into openFaas. You should change the service
reference at the backend YAML file.
2. Apply autoscaling to the backend deployment.
3. Use a persistent volume for the path /tmp/pgdata within the PostGIS image
4. Use a request splitter between two deployments of the front end in a ratio of 0.7:0.3. Note that
you should access the front end from the request splitter service only. Thus a single IP is needed
to access any of them.
How they can be implemented? How the YAML files should be updated?
Deliverables:
1. A report that includes the design part.
2. An audible video of about 5 minutes maximum showing the deployment and service for each
distributed system component. You should run the front end as proof of correct implementation.
You may use the debug section (step 4) in your video.
3. Another audible video of 5 minutes maximum showing each component of the design section and
how they are implemented and deployed. You should run the front end as proof of correct
implementation. You may use the debug section (step 4) in your video.