Google Custom Search, Amazon, eBay, Flickr, YouTube, Second Life
1

Application Programming Interface (API)
 a particular set of rules and specifications that a
software program can follow to access and make use of
the services and resources provided by another
particular software program that implements that API.
 serves as an interface between different software
programs and facilitates their interaction

Web API
 typically a defined set of HTTP request messages
expressed in SOAP or REST along with a definition of
the structure of response messages, typically expressed
in JSON or XML.
2

Simple Object Access Protocol (SOAP)

SOAP Tutorial:
 http://www.w3schools.com/soap/soap_intro.asp

A SOAP message is an ordinary XML document containing
the following elements:
 An Envelope element that identifies the XML document as a SOAP
message
 A Header element that contains header information
▪ contains application-specific information about the SOAP message
▪ optional
▪ must be the first child element of the Envelope element
 A Body element that contains call and response information
 A Fault element containing errors and status information
3

Representational State Transfer (REST)
 Use HTTP method to invoke remote services (not
XML)


The response of remote service can be in XML or
any textual format
Benefits:
 Easy to develop
 Easy to debug (with standard browser)
 Leverage existing web application infrastructure

We will focus on REST services programming for
the rest of the slides.
4

Really Simple Syndication (RSS, Atom)
 XML-based standard
 Designed for news-oriented websites to “Push” content to
readers
 Excellent to monitor new content from websites

JavaScript Object Notation (JSON)
 Lightweight data-interchange format
 Human readable and writable and also machine friendly
 Wide support from most languages (Java, C, C#, PHP,
Ruby, Python…)
5
http://www.json.org/example.html

Google has many APIs available:
 http://code.google.com/more/

We will introduce two of them:
 Google Custom Search API
 YouTube Data APIs
7

What is custom search?
 to create a search engine tailored to your needs and
interests
 Your custom search engine (CSE) can prioritize or
restrict search results based on websites you
specify
 you can present the results in your website
8

Developer's Guide
 http://code.google.com/apis/customsearch/docs/dev_guide.html

You can use a provided custom search wizard to create your
own CSE:
 http://www.google.com/coop/manage/cse/create/1

You can use the “control panel” to manage your CSE. You can
access the “control panel” from the “My search engines” page.
 http://code.google.com/apis/customsearch/docs/start.html#managing
9

A custom search engine has two main
components housed in XML files:
 Context: specifies the global settings of the search
engine. Each CSE has its own context file.
 Annotations: lists the sites (webpages or websites)
you want your search engine to cover.

You can view or download these two XML
files in “control panel”->”Advanced”.
10
Your CSE ID and your
personer ID
11

Users can access your CSE in four ways:
 Your custom search engine homepage
 A search engine gadget in their iGoogle pages
 A search engine gadget in their webpages or blogs
 A search box in your website
▪ To get your very own search box, copy the generated
code from the “Code” tab of the control panel, and paste
it into the HTML of your webpage.
 http://code.google.com/apis/customsearch/docs/start.html#release
12

The Custom Search API lets you skip the
control panel altogether and use a client
application of your choice to :
 retrieve a list of CSE under an account
 Create, update and delete custom search engines
 retrieve code for a search box
 retrieve current settings of a CSE
 associate a CSE with an AdSense Account
 http://code.google.com/apis/customsearch/docs/api.html
13

Request Format:
 standard HTTP requests, such as: GET, POST, and
DELETE.
 Need to get an authentication token from ClientLogin
using HTTP post Request
▪ Generally an authentication token expires in two weeks
▪ http://code.google.com/apis/accounts/docs/AuthForInstalledAp
ps.html

Response Format:
 XML-based

Format Details:
 http://code.google.com/apis/customsearch/docs/api.html
14

A POST request that create a search engine
specification (by submitting a context xml).
Request Endpoint
authentication token
New context file
content
15

XML feeds of search results are only available
to paid Business account …

Our Method 1:
 Parse the content of the search result page:
▪ http://www.google.com/cse?cx=creatorId:SEid&q=query
&parameter=value
▪ creatorID and Seid can be found in context xml file of your CSE
▪ Parameter and query details can be found in
http://code.google.com/apis/customsearch/docs/structured_search.html
16

Method 2:
 Google Custom Search Element for JavaScript users
▪ The generated code from the control panel use this method
▪ http://code.google.com/apis/customsearch/docs/js/cselement-devguide.html

Method 3:
 Use JSON/Atom Custom Search API (RESTful requests)to
get search results in either JSON or Atom format.
▪ http://code.google.com/apis/customsearch/v1/overview.html

Method 2 and 3 need to visit the Google APIs console to
set up a new project and get an API key.
 Currently an API key allows 100 queries per day and 100 results
at most for each query. You can request additional quota from its
console.
17

Step 1: create your own CSE using the wizard
(web interface) or the CSE API
 CSE API needs to apply for authentication token

Step 2: manage your CSE using the wizard or
the CSE API if necessary

Step 3: access CSE search results by method
1,2,3.
 Method 2 & 3 needs to apply for API key
18

Amazon Web Services (AWS)
 http://aws.amazon.com/
 “With AWS you can requisition compute power, storage, and
other services …”

Amazon SDK for Java
 http://aws.amazon.com/sdkforjava/

Amazon Product Advertising API

https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html
 Access Amazon Product Selection
▪ gives you access to Amazon’s selection of millions of products in
categories such as books, music …
 Leverage Amazon Product Discovery capabilities
▪ lets you leverage Amazon’s customer-centric features such as Product
Search, Customer Reviews, Similar Products...
 Monetize your website
19

Programming Guide
 http://docs.amazonwebservices.com/AWSECommerceService/latest/DG
/index.html?ProgrammingGuide.html

Check “Visual Introduction to Product Advertising API” to
see what this API can do.
 E.g.: to find items
20

Tutorial:
 http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/index.
html?CHAP_MakingRequestsandUnderstandingResponses.html


Format: REST & SOAP
REST Format:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&
Operation=ItemSearch& AWSAccessKeyId=[Access Key ID]&AssociateTag=[ID]&
SearchIndex=Apparel& Keywords=Shirt&Timestamp=[YYYY-MM-DDThh:mm:ssZ]
&Signature=[Request Signature]
21

Tutorial:
 http://docs.amazonwebservices.com/AWSECommerceServi
ce/latest/DG/index.html?RequestAuthenticationArticle.html

To access the Product Advertising API, you must first
create an account at http://aws.amazon.com. AWS will
assign you a pair of related identifiers:
 Access Key ID (a 20-character, alphanumeric sequence)
▪ include it in AWS service requests to identify yourself as the sender
of the request
 Secret Access Key (a 40-character sequence)
▪ To calculate the signature used in your request to verify your identity
22

Two authentication parameters:
 Timestamp:
▪ a dateTime object, with the complete date plus hours,
minutes, and seconds
▪ Format : YYYY-MM-DDThh:mm:ssZ (where T and Z
are literals)
 Signature:
▪ created by using the request type, domain, the URI, and a
sorted string of every parameter in the request
▪ Once properly formatted, you create a base64-encoded
HMAC_SHA256 signature using your AWS secret key
23

Tutorial to sign a request, using sample AWS access
identifiers (READ IT!).
 http://docs.amazonwebservices.com/AWSECommerceService/latest/DG
/index.html?rest-signature.html

Basic Authentication Process
24

Response is a XML file which conforms to a WSDL and schema.

WSDL location:
 http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/index.
html?WSDLLocation.html

Paging Through Results
 http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/index.
html?PagingThroughResults.html
 E.g., for itemsearch request, use “&ItemPage=4 ” to access fourth page of the
results.

Sorting Results
 http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/index.
html?SortingResults.html
 E.g., for itemsearch request, use “&Sort=titlerank” to sort results by title or
description
25

Steps to search Amazon:









Apply for a Access Key ID and Secret Access Key
Generate search conditions
Generate a parameter string
Generate a string used for signature
Generate a signature
Generate a query URL
Send a search request to Amazon server
Receive the query result (XML format)
Parse the result and display it on your web site
26

Main page:
 http://developer.ebay.com/common/api/

eBay Web Services Overview
 http://developer.ebay.com/products/overview/
 Available APIs include:
▪ Finding API, Best Match API, Merchandising API
▪ Shopping API, Feedback API, Trading API, Client Alerts API
▪ Platform Notifications API, Research API for eBay, etc.
 Which API to use with your application?
▪ http://developer.ebay.com/products/overview/api-by-feature/
27

eBay Finding API: to search for eBay items:






http://developer.ebay.com/products/finding/
Search for items based on a keyword query
Search for items associated with a specific product
Browse for items in a specific category or in an eBay store
Filter search results by item listing details, such as price, condition, etc.
Refine searches by the characteristics of an item (i.e., aspects), such as brand,
style, etc.
 Retrieve category and aspect metadata
 Get keyword recommendations to improve search results

Formats: XML, SOAP, Name Value, JSON

Protocols: HTTP GET and POST

Need to apply for a API key
 https://developer.ebay.com/join/default.aspx
28

Tutorial for making a API call:


Each Finding API call consists of the following elements:





http://developer.ebay.com/DevZone/finding/Concepts/MakingACall.html
Service Endpoint
HTTP Headers or URL
Standard Input Fields
Call-specific Fields
URL-based call structure

http://svcs.ebay.com/services/search/FindingService/v1?
OPERATION-NAME=findItemsByKeywords
&SERVICE-NAME=FindingService
&SERVICE-VERSION=1.0.0
&GLOBAL-ID=EBAY-US &SECURITY-APPNAME=MyAppID
&RESPONSE-DATA-FORMAT=XML
&REST-PAYLOAD
&Standard input fields
&Call-specific input fields
29

findItemsByKeywords lets you to search for items on eBay
by a keyword query

http://developer.ebay.com/DevZone/finding/CallRef/findItemsByKeywords.html
 Controlling Results with Item Filters
▪ &itemFilter(0).name=FreeShippingOnly&itemFilter(0).value=true
&itemFilter(1).name=ListingType&itemFilter(1).value=FixedPrice
 Refining Results with Aspect and Domain Filters
▪ &aspectFilter.aspectName=Sub
Style&aspectFilter.aspectValueName=Running Shoes...
▪ &domainFilter.domainName=Digital_Cameras
 Finding Local Items
▪ &buyerPostalCode=95125&sortOrder=Distance
 Sorting the Output:
▪ &sortorder= PricePlusShippingAsc
 Paginating the Results
▪ &paginationInput.pageNumber=2&paginationInput.entriesPerPage=50
30

Call Example: The user is looking for a specific Harry Potter title and has a
maximum price ceiling and doesn't want to pay for shipping.
 http://svcs.ebay.com/services/search/FindingService/v1?
OPERATION-NAME=findItemsByKeywords&
SERVICE-VERSION=1.9.0&
SECURITY-APPNAME=YourAppId&
RESPONSE-DATA-FORMAT=XML&
REST-PAYLOAD&
keywords=harry%20potter%20phoenix&
itemFilter(0).name=MaxPrice&
itemFilter(0).value=10.00&
itemFilter(0).paramName=Currency&
itemFilter(0).paramValue=USD&
itemFilter(1).name=FreeShippingOnly&
itemFilter(1).value=true&
paginationInput.entriesPerPage=2

PHP coding Example and Tutorial:

http://developer.ebay.com/DevZone/finding/HowTo/GettingStarted_PHP_NV_XML/GettingStarted_PHP_NV_XML.html
31

Steps to search eBay:
 Apply for a API key
 Construct the request
 Send the request to eBay Server
 Receive the query result in your preferred format
 Parse the result
32

Flickr API Main Page:
 http://www.flickr.com/services/api/

API Key Application
 http://www.flickr.com/services/api/keys/

Request Format
 Rest, XML_RPC, SOAP

Response Format
 Rest, XML_RPC, SOAP, JSON, PHP
33

Flickr Photo Search Method
 Description: http://www.flickr.com/services/api/flickr.photos.search.html
 Explorer: http://www.flickr.com/services/api/explore/?method=flickr.photos.search

Steps:
 Apply for a Flickr API Key
 Send a search request to Flickr Server in your preferred
format
 Receive the response in your preferred format
 Parse the result and get the photo URLs
▪ To construct photo URLs:
http://www.flickr.com/services/api/misc.urls.html
34

Request Format:


http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key
=yourkey&argument=value
Example:
 Request (to search photos that have “tucson” in its description):
http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=e7c88ea43
041aff2df564188d4609849&text=tucson
 Response:
 Extracted Image URL and Web Page URL:
▪ http://farm6.static.flickr.com/5288/5368668124_d216292c15.jpg
▪ http://www.flickr.com/photos/48553200@N04/5368668124
35

YouTube Data API:
 lets you incorporate YouTube functionality into your own
application or website. You can perform searches, upload
videos, create playlists, and more.
 Main Page:
http://code.google.com/apis/youtube/getting_started.html#data_api
 Java Developer Guide:
http://code.google.com/apis/youtube/2.0/developers_guide_java.html

YouTube Player API
 give you control over YouTube video playback on your
website.
36

Advanced functions of the data API requires its java client
library
 http://code.google.com/apis/youtube/2.0/developers_guide_java.html#Getting_
Started

You will need to apply for a developer key and client id to
execute authenticated functions.
 http://code.google.com/apis/youtube/2.0/developers_guide_java.html#Authenti
cation

Two important class:
 VideoFeed: represent lists of videos, such as standard feeds, uploads,
subscriptions, and favorite videos
 VideoEntry: Each video entry corresponds to exactly one YouTube video and
contains information about that video.
37

Description of the YouTube video search function


Query parameter definition


http://code.google.com/intl/en/apis/youtube/2.0/developers_guide_java.html#Searching_for_Videos
http://code.google.com/intl/en/apis/youtube/2.0/reference.html#Query_parameter_definitions
A search query example for videos that match the search term
“puppy”.
 http://gdata.youtube.com/feeds/api/videos?q=puppy&orderby=viewCount

The query example can be generated using the following
code:

YouTubeQuery query =
new YouTubeQuery(new URL("http://gdata.youtube.com/feeds/api/videos"));
query.setOrderBy(YouTubeQuery.OrderBy.VIEW_COUNT);
query.setFullTextQuery("puppy");
//print result
VideoFeed videoFeed = service.query(query, VideoFeed.class); printVideoFeed
(videoFeed, true);
38

Steps to search YouTube Videos:
 Download client library
 Apply for Developer key and client id
 Construct a search request and send to YouTube
 Receive the query result
 Parse the result and display videos on your
websites.
39

Second Life API Main Page:


http://wiki.secondlife.com/wiki/Web_Services_Portal
There are eight official APIs:








Registration API
Map API
Exchange Risk API
Media Plugin API
Live Data Feeds
Inventory API
Snapshot API
Search API
40

The Second Life Search API enables you to
search Second Life for:
 Events, People, Places, Groups, and information on
its Wiki.
 http://wiki.secondlife.com/wiki/SearchAPI

Linden Lab uses a Google Search Appliance
to index web pages. The Search API allows to
search the indexed web pages.
 No API key is needed.
41

Request format:


http://search.secondlife.com/client_search.php?q=yourquery&pa
raemeter=value
Two popular parameters:
 start
▪ Each query will return 20 results at most (wiki says 10)
▪ “start=0” shows the first 20 results, “start=20” shows 21-40,
etc.
 s
▪ To limit the results to specific categories
▪ This value defaults to "All", but can also be set to one of the
following: All, Events, Groups, People, Places, Wiki.
42


The response format is XHTML.
Example:
 http://search.secondlife.com/client_search.php?q=a
rizona&s=All&start=20
43

Steps:
 Send a search request to Second Life Server
 Parse the XHTML response page to get each result
page URL.
 Parse each result page to get information:
▪ Pages in each category share similar structure. You can
develop separate web page parsers for group page, event
page, place page, etc.
44

A variety of APIs for accessing XML have been developed
and used, and some have been standardized. Existing APIs
for XML processing tend to fall into these categories:
 Stream-oriented APIs accessible from a programming language
▪ E.g., Simple API for XML (SAX)
 Tree-traversal APIs accessible from a programming language.
▪ E.g., Document Object Model (DOM)
 XML data binding, which provides an automated translation
between an XML document and programming-language objects
▪ E.g., Java Architecture for XML Binding (JAXB)
 Declarative transformation languages
▪ E.g., XSLT, XPath and XQuery.
▪ XPath Tutorial: http://www.w3schools.com/xpath/default.asp
45

Document Object Model (DOM)
 DOM accesses an XML document through a tree
structure, composed of element nodes and text
nodes.
46

<a href="http://www.google.com/">
Google's<b>fast</b> home page.</a>
47

Pros of DOM:
 Easy to manipulate the document
 Can traverse the document back and forth
 Good for small XML files

Cons of DOM:
 Consumes lots of memory

JDK provides two packages related to DOM:
 org.w3c.dom
 javax.xml.parsers

Some popular JAVA xml parsers:
 Xerces
 JDOM
 DOM4j
48

JSON Tutorial:
 http://www.json.org/

A lot of developed JSON tools and parsers are
listed in the website.
 Listed by different language

For example: http://www.json.org/java/
49