Lecture IV: REST Web Service
with Google App Engine
CS 4593
Cloud-Oriented Big Data and Software Engineering
Outline
•
•
•
•
Course Project
Google App Engine
Google Development Console
Google Custom Search API
2
Course Project
• Topic
– Keyword trends analysis services for a certain topic
• Platform
– Google App Engine
– Any programming language supported by GAE
• Collaboration
– Work individually
– Hopefully you are not working on the same idea, but it
is fine if you happen work on one by accident
3
Course Project
• Requirements
– Develop a web service based on Google Search API
to follow the weekly trends of keywords on a certain
topic
– Search one or multiple websites each day with
Google Search API free quota and record the query
results during a period of time (hopefully more than a
week)
– Perform statistics on query results to provide daily
keywords frequency and the trends of the week
4
Course Project
• Requirements
– Provide at least two web services: one for the
keyword frequency of each day, and the other for
trend of the week
– Provide output as either JSON or XML
– The developed web service should be deployed on
Google App Engine
5
Course Project
• Advanced Requirements
– You are going to choose the topic and the website to
search, but the results should be meaningful
(searching a static website for a week will not give
you any different results)
– Provide HTML presentation for the web services
6
Course Project
• Additional Information
– Feel free to use additional free web services to
support your task
– Searching a more volatile website often provides you
more interesting results, e.g., searching "Obama" in
CNN or NY Times.
7
Course Project
• Evaluation
– fetching and recording Google Search API results (2.5
points)
– extracting keywords (2.5 points)
– presentation of daily results (2.5 points)
– presentation of weekly results (2.5 points)
– The advanced features will be evaluated by quality.
8
Course Project
• Deliverables
–
–
–
–
Demo
Code
Commends and User Guide
Blackboard
• Due date
– Demo: Oct. 5th
– Code: Oct. 4th
9
Outline
•
•
•
•
Course Project
Google App Engine
Google Development Console
Google Custom Search API
10
Google App Engine: Overview
• Google App Engine (GAE) is a PaaS cloud
computing platform for developing and hosting
web applications in Google-managed data
centers.
• Google App Engine lets you run web
applications on Google's infrastructure.
– Easy to build.
– Easy to maintain.
– Easy to scale as the traffic and storage needs grow
11
infrastructure vs. platform - What is
“The Platform”?
Application-specific
code
Libraries: shared by
multiple applications
Platform: same for all
applications
infrastructure: hidden by
platform
Scaling
• Low-usage apps: many apps per physical host
• High-usage apps: multiple physical hosts per
app
• Stateless APIs are trivial to replicate
• Datastore built on top of Bigtable; designed to
scale well
– Abstraction on top of Bigtable
– API influenced by scalability
• No joins
• Recommendations: denormalize schema; precompute joins
13
Google App Engine: Overview
• Free
–
–
–
–
1GB storage
1 virtual machine instance
5 million page views / month
10 applications / Google account
14
Preserving Fairness Through Quotas
• Everything an app does is limited by quotas, for
example:
– request count, bandwidth used, CPU usage,
datastore call count, disk space used, emails sent
• If you run out of quota that particular operation is
blocked (raising an exception) until replenished
15
Preserving Fairness Through Quotas
• Free quotas are tuned so that a well-written app
(light CPU/datastore use) can survive
• The point of quotas is to be able to support a
very large number of small apps (analogy:
baggage limit in air travel)
• Large apps can request raised quotas
• Three types of quotas
– Free Quota
– Billed Limits
– Safety Limits
16
Google App Engine: Programming
Languages
• Java
– App Engine runs JAVA apps on a JAVA 7 virtual
machine (currently supports JAVA 6 as well).
– Uses JAVA Servlet standard for web applications:
•
•
•
•
•
•
WAR (Web Applications ARchive) directory structure.
Servlet classes
Java Server Pages (JSP)
Static and data files
Deployment descriptor (web.xml)
Other configuration files
17
Google App Engine: Programming
Languages
• Python
– Uses WSGI (Web Server Gateway Interface)
standard.
– Python applications can be written using:
• Webapp2 framework
• Django framework
• Any python code that uses the CGI (Common Gateway
Interface) standard
18
Google App Engine: Programming
Languages
• PHP
– Local development servers are available to
anyone for developing and testing local
applications.
– Only whitelisted applications can be deployed
on Google App Engine.
(https://gaeforphp.appspot.com/).
19
Google App Engine: Programming
Languages
• Go
– Go is an Google’s open source programming
environment.
– Tightly coupled with Google App Engine.
– Applications can be written using App
Engine’s Go SDK.
20
Google App Engine: Data Storage
• App Engine Data Store
– NoSQL schema-less object based data storage, with
a query engine and atomic transactions.
– Data object is called a “Entity” that has a kind (~ table
name) and a set of
– properties (~ column names).
– JAVA JDO/ JPA interfaces and Python datastore
interfaces.
21
Google App Engine: Data Storage
• Google Cloud Store
– RESTful service for storing and querying data.
– Fast, scalable and highly available solution.
– Provides Multiple layers of redundancy. All data is
replicated to multiple data centers.
– Provides different levels of access control.
– HTTP based APIs.
22
Google App Engine: Services
• Google Services
– URL Fetch
– Mail
– Memcache
23
Google App Engine: Security
• Sandboxes
– All hosted applications run in a secure environment
that provides limited access to the underlying
operating system.
– Sandbox isolates the application in its own secure,
reliable environment that is independent of hardware,
operating system and physical location of a web
server.
24
Google App Engine: Security
• Sandboxes: Limitations
– Application can only interact with other computers
over internet using URL fetch, email, and HTTP/
HTTPS requests on the standard ports
– Applications cannot write to local file system in any of
the runtime environments.
– Application code runs only in response to a web
request, a queued task or a scheduled task and must
return the response data within 60 seconds.
25
Google App Engine: Demo
• Create a project on Google App Engine
– At https://console.developers.google.com/project
• Initialize the environment of Restlet + eclipse +
GAE
– Tutorial at
– http://restlet.com/technical-resources/restletframework/guide/2.3/introduction/first-steps/firstapplication
– Download Restlet-GAE, GAE SDK, and Eclipse-GAE
plugin
26
Google App Engine: Demo
• Deployment of Applications
– Follow
https://console.developers.google.com/start/appengin
e?_ga=1.67402004.429901446.1441394923
• Google Search API
– Generate your search engine
– Manage API usages
– https://cse.google.com/manage/all
27
Google Map API
• Provide Map Services
–
–
–
–
Http: http://maps.googleapis.com/maps/api
Android
IOS
JavaScript
• Parameters
–
–
–
–
Center
Zoom
Size
Language
28
Google Map API
• Pricing
– Free Quota: 2500 / day
– Bill Quota: 1000 / $0.5
– Safety Quota: 100, 000 / day
29
Google Place API
• Searching for places around certain map
location
• Usage Example
– Finding all food places within 500 miles from (long,lat)
– https://maps.googleapis.com/maps/api/place/nearbys
earch/json?location=long,lat&radius=500&types=food
&name=cruise&key=API_KEY
• Place ID
– A ID of a place that can be used later (e.g., Costco at
I 10 @ UTSA Boulevard)
30
Google Place API
• Parameters
– Type: food, airport, hospital, …
– https://developers.google.com/places/supported_type
s
• Pricing
– Free quota: 1000/day
– Free up to 150,000/day with credit card info
31
Google Place API: response
{
"geometry" : {
"location" : {
"lat" : -33.870775,
"lng" : 151.199025
}
},
…
"name" : "Rhythmboat Cruises",
"opening_hours" : {
"open_now" : true
},
"photos" : […],
"place_id" : "ChIJyWEHuEmuEmsRm9hTkapTCrk",
"scope" : "GOOGLE",
"types" : [ "travel_agency", "restaurant", "food", "establishment" ],
"vicinity" : "Pyrmont Bay Wharf Darling Dr, Sydney"
},
Google+ API
•
•
•
•
Intro
Usages
Parameters
Pricing
33
Google+ API
• A series of APIs working with Google+
– People: Get public profiles with query or user ID
– Activities: List activities of a user or searching
activities with query
– Comments: List comments of an activity or searching
comments with query
– Moments: Retrieve, insert or delete moments
• Usage
– https://www.googleapis.com/plus/v1/
– 10,000/day free quota
34
Google Translation API
• Translate strings to other languages
• Parameters
– source
– target
– q(query)
• Pricing
– $20 for 1M characters
– 50M / day limit
35
Youtube Data API
• Provide various supports to retrieve and manage
data from Youtube
• Some important services
– Video: List, search, Insert, Delete, Rate, …
– Playlist: Insert, delete, List, …
– Captions: List, insert, update, download, delete, …
36
Youtube Data API
• Parameters
– Part
– videoCategoryId
– myRating
• Pricing
– Unit based quota calculation
– Services with different parameters may cost different
quota
– https://developers.google.com/youtube/v3/determine_
quota_cost
– 50M unit / day, and 30K unit / second, always free
37
Facebook Graph API
• A web API library for Facebook social network
• Provide support for various platforms
–
–
–
–
–
RESTful
IOS
Android
JavaScript
PHP
38
Facebook Graph API
• Usages
– Provide similar functions to google+ APIs
– List friends, List events, Fetch comments, …
– Uploading videos, photos, …
• Pricing
– Free
– Need to contact Facebook for more than 100M API
calls per day
39
MediaWiki API
• API to get information from Wikipedia
programmatically
– Entry: https://en.wikipedia.org/w/api.php
• Parameters
–
–
–
–
Action: query, modules
Titles: specify the query
Limit: max results
Section: retrieve from which section
• Pricing
– Free: with unofficial limits
40
Bing API
• Search API from Microsoft
• Root:
– https://api.datamarket.azure.com/Bing/Search/v1/
• Parameters
– Query
– NewsCategory
– Sources: web, image, video, news, …
41
Bing API
• Pricing
– 5000 / month free
– After it: 10K/month for $20
42
Summary
• Google App Engine
• Application Deployment on Google App Engine
• Popular free Web Services
43