Important
Note
This instruction has some significant differences against the previous version. You will see
instruction blocks formatted in the same way as this one. Please do pay attention to them.
In this project, we use a thorough examination and selection of the performers. To get more tasks
and earn more money, please be attentive and patient.
Stage 1: you have to pass free training tasks to learn different cases on how to label query-product
pairs.
Stage 2: you take an exam, which is paid if you complete it with a good score. Please do your best,
otherwise, you won't get access to a large number of main tasks.
Main tasks: during your work on the main tasks, we constantly measure your performance. We have
a lot of those tasks which is a good opportunity for you. Therefore, be careful and provide a good
quality not to lose access to those tasks.
For exam, we have setup auto-rejection of the assignment if you score lesser than a
threshold. Please beware that if many tasks are wrong, the assignment is rejected within 1
day. Please be very attentive while labeling to get access to bigger pools and higher skill.
Policies - has updates
Judgements
Imagine that you are shopping on Amazon. You will see a search query and (optional) category
selected, and one or more products to be judged for relevance to that query. You will need to use
one of four labels for each product you see:
1. Exact - has updates
Exact: this product fully satisfies the query. A broad or ambiguous query could be satisfied by any of
a diverse set of products, so long as it is reasonable to expect that a customer who entered that
query might buy that product. A very specific query will only be satisfied if the product matches all of
the characteristics included in the query.
Tips:
o
Availability in the widget/drop-down makes the product exact, despite the change in ASIN
o
Multiple interpretation Search Queries (hereinafter SQs): a query can have multiple
meanings and an ASIN that is Exact for any of the meanings should be judged Exact. For
many broad queries, the customer's intent may be unclear. Multiple interpretations of the
query could result in different judgments for an ASIN. For example –
SQ – "battery scooter";
Offered product 1 (hereinafter OP1) – scooter with battery;
Offered product 2 (hereinafter OP2) – battery for scooter;
The given SQ of scooter battery could be interpreted in 2 ways i.e. a batter-operated scooter
or a battery for scooter. As a reviewer, we should be taking both or all possible interpretations
into account before judging a query.
In the above cases both interpretation lead to an exact judgement (scoter and battery). If one
of the interpretations would result in an Exact judgment for the query, then that should be the
judgment, even if other interpretations would result in Complement or Irrelevant judgments. If
the query is issued within a specific alias, then all the interpretations used for judgment should
respect that alias.
o
Exact Bundle: If a product is being sold with normal accessories which are to be expected for
that purchase, we do not consider it a bundle. For example, an electric guitar being sold with a
guitar case, a small amplifier and a cord is normal, and all these accessories are normally
expected with this type of product. Another example is when you buy an iPhone and
earphones and a charging cable are included in the box. This is normal and not considered a
bundle.
o
Universal products are compatible with all model types: For example SQ "iphone 10
case" and ASIN "universal case" will result in an Exact.
o
"Products" not sold on Amazon. Some of the "products" are not sold on Amazon, like guns,
cars, animals. In case of such a query all items, which could be used with this kind of
"product" should be marked as Exact, like:
SQ: "cat";
OP: cat food - will be an Exact
But notice that merchandise, which only depicts the desired "product" and any
other unrelated products should be marked as Irrelevant, like:
SQ: "car";
OP: printed car poster - will be Irrelevant
Search
Query
ASIN/ISBN
Red TShirt
Red T-shirt (B00WQMX5G6)
Doraemon Plush toy (B00S1QTEJS)
Doraemon
Toy
Book on Cricket (0312094116)
Cricket
Gun
Handgun case (B0000C50G9)
2. Substitute - has updates
Substitute: this product differs from the one intended, but it is functionally equivalent. This will be
the same type of product, but may differ by brand, model, color, title, or other details. If the
requested product is included in a bundle with other associated products, it is judged as Substitute,
so long as it is reasonable that the customer may want to buy both together.
Tips:
o
+/- 1 size and age rule: +/-1 difference in size and age from the asked SQ is
still Substitute for the customer, as different brands might have different size scales. Please
do use standard size scales when evaluating, like 7.5, 8, 8.5 for shoes; S, M, L for clothes;
30', 32', 40' for TVs etc. More then +1 or less than -1 size/age difference should be marked
as Irrelevant.
Also note that this rule should not be aplied to products like cushion covers, pillow covers,
fitted sheets etc., as they won't fit or will be too loose for the customer's insert/mattress. In this
case such a product should be marked as Irrelevant.
o
Substitute Bundle: If OP is a bundle, containing the product requested in the SQ as an
acessory/additional thing to another, "lead" product then this OP should be marked
as Substitute. Same for bundle of several independent products, containing the requested
one. But remember if there are too much unrelated products in the bundle - it will be
always Irrelevant.
o
Same purpose: sometimes an OP could not be a directly requested product, but still serve all
customer's needs. It could be then marked as Substitute as soon as customer could be
interested in buying it, for example:
SQ: "dog perfume";
OP: deodorizing spray for dogs - will be a Substitute since the purpose of animals' perfume
and deodorants is the same - odor relief. For people this would not be correct as those
products will be serving different needs
But notice that all customer needs should be specified, otherwise OP should be marked
as Irrelevant, like:
SQ: "tv";
OP: computer monitor - will be Irrelevant since a monitor could be indeed used to show some
videos, but TVs have much wider functionality
o
Difference in target audience (this is mostly about books): from the SQ you might
understand to what audience the author could be referred. E.g. there could be some queries
for scientific literature. If the OP is still covering the requested theme/topic, but targets a bit
different audience then it could be marked as Substitute, for example:
SQ: "the cancer code";
OP: a cancer patient's guide book - will be a Substitute since the book requested is more
likely to be scientific, written for professionals, while the OP is for regular people.
o
Difference in weightage: while quantity of a product should be ignored, the weightage or unit
of the product should be considered:
SQ: "5kg rice";
OP: 2kg rice - will be a Substitute.
o
Retailer definition prioritized over customer comments: This may not always be true as
there are cases when retailer information in incorrect and customers provide clarification in
reviews. For example – in few cases the title might not match with the description or image.
A holistic approach of overall product and detail page is necessary and recommended. While
product title and description provided via retailers usually holds priority/preference over
customer comments [chiefly when description provides specific details to sizes, dimensions,
compatibility, etc.], however it is also imperative to gather a comprehensive knowledge of the
OP [especially where retailer may inaccurately describe a product].
Search
Query
ASIN/ISBN
Braun Shaver (B07Z6HWZX4)
Panasonic
Shaver
Powerpuff Girls Bedsheet (B08XQ7K7YF)
Disney
Princess
Bedsheet
IPhone XR (B07P978C2R)
IPhone X
tablet for
migraine
migraine pain relief head wrap (B071DCB11C)
SQ: Black t shirt.
OP: 1 black tshirt+2 red tshirts
Decision: Substitute
SQ: Iphone X
OP: Iphone X+ cover
Decision: Exact
SQ: Camera
OP: Bundle of camera, card, reader, case, tripod and charger
Decision: Exact
SQ: Table
OP: Table and Chair
Decision: Substitute
Rational: Lead product with its immediate and compatible accessories should
be marked as Exact.
Rational: Two lead products in a bundle, which
independent of each other.
3. Complementary - has updates
Complement: this product is not the one intended, but it could be useful to the customer if they did
buy the intended product. This could be an accessory, a component, replacement parts, or even
another item designed to be used with the product. Also a complementary product could be one,
which a customer might also want to buy while looking for original one. See rules below for
clariifications.
o
Brand affinity: if SQ contains a product of some brand, other products of this brand should
be considered as Complement, but only if they fall under the same major category, like:
SQ: "dove shampoo";
OP: dove deodorant - will be a Complement since both products have the same brand and
fall under "personal care" category
Note that this rule will not work in case SQ contains requirements (gender, religion, age,
animal kind etc.), which are not met in the OP; and in case SQ and OP are medicines of the
same brand, but for different diseases. In this case those should be marked as Irrelevant:
SQ: "nike shoes for men";
OP: nike shoes for women - will be Irrelevant since the gender requirement is not met
o
Indirectly related products: some products are usually bought/weared with some others. It
could be useful to show such products, that's why they should be marked as Complement.
For example, if a customer is looking for a formal blazer, they may be also interested in pair of
compatible pants. So, before judging a product as irrelevant to the query, think if it could be
sold together with the one requested in the SQ and prefer Complement label in this case.
o
Same theme/occasion: customers could be searching goods of some theme or designed for
some occasion. In case OP is not the same product as requested in the SQ, but still falls
under this theme or is designed for this occasion, it should be marked as Complement, like:
SQ: "avengers stickers";
OP: avengers action figures - will be Complement since they are both avengers-themed
o
Same genre: if an OP is not exactly the type fo the product requested in the SQ, but it falls
under the same media genre, it should be marked as Complement since a customer might be
interested in buying it. For example:
SQ: "jazz music";
OP: jazz dance learning video - will be Complement since both products could be useful for a
person interested in jazz
o
Multiple product in the query: Asking for multiple product in the query and the offered
product ASIN is a single product from the query. Example Query: Shampoo and
conditioner and the offered product is “Shampoo” without conditioner, the judgement should
be “Complement”.
Search
Query
ASIN/ISBN
Samsung S20 Case (B082ZYLC63)
Samsung
S20
Eraser (B00AZEK8TY)
Pencil
dior
perfume
dior deodorant (B014RLXFZ4)
bean bag
bean bag refill pack
birthday
banner
happy birthday badge (B08WBW4KT7)
4. Irrelevant - has updates
Irrelevant: this product does not satisfy a reasonable understanding of the customer's intent. It could
be entirely unrelated or it could differ by one crucial feature that would make it unsuitable for the
customer. Examples: SQ is "pajamas", and OP is a cocktail dress; or SQ is "chair for bedroom", and
OP is a gaming chair.
Please, be acurate when making a decision that the OP is Irrelevant to the SQ. Please consider
rules for other labels, such as Same Purpose, Difference in taget audiences for Substitute label;
and Brand afinity, Indirectly related products, Same theme/occasion, Same genre for
Complement label. Sometimes OP, looking as Irrelevant from the first glance, could be marked
either as Substititue or Complement.
Tips:
o
Complement of a substitute product is irrelevant.
o
Substitute of a complementary product is irrelevant.
o
i. Neem Seeds – Exact
ii. Books about Neem – Complementary
iii. Books about banyan tree – (Substitute of a Complimentary) – Irrelevant
o
Use Irrelevant if the ASIN is offering a main item when the search query is shopping for
its accessory. For example, if the query is searching for an accessory "iphone 6 case" and
the item presented is an iPhone 6 mobile phone, you must rate it as Irrelevant – not
Complement – because it is not an accessory, it is a main item that the customer already
owns. Similar cases: SQ is "tv remote" , and OP is a TV. Or SQ is "mouse for laptop" , and OP
is a laptop.
o
Irrelevant bundle: if the SQ is asking for some kind of accessory and the OP is a bundle
containing this accessory, but with a lead product, this should be marked as Irrelevant as
well, e.g. SQ is "iphone"
o
Miss of crucial features or different theme/occasion: sometimes OP cood look familiar
with the one requested in the SQ (even have same purpose, theme, occasion, genre), but
they could miss some crucial features, like brand, gender, size (more than +/-1), device
compativility, majot category etc. Consider examples:
SQ: "nike shoes for men";
OP: nike shoes for women - will be Irrelevant even if the brand and purpose is the same, the
gender requirement is not met
SQ: "t-shirt S";
OP: t-shirt L - will be Irrelevant since the size difference is more than 1
Remember that +/-1 size rule doens't apply to products like cushion covers, pillow covers,
fitted sheets etc., as they won't fit or will be too loose, like:
SQ: "16x16 cushion cover";
OP: 18x18 cushion cover - will be Irrelevant even if 18 is +1 size in the standard scale, it will
be to loose for a customer's cushion
Search
Query
ASIN/ISBN
Dolittle
(Adventure/
The Past (Horror) (B07W1XDVCM)
Comedy)
Men Shoes (B08CZ7FLKT)
Women
Shoes
Apple watch Band (B07TC5T515)
Xiaomi
Watch
samsung tv
Samsung computer monitor (B087N3JPLF)
Michael Kors phone bag (B08LNYMZ18)
Michael
Kors phone
case
FIFA 23 XBox (B0BGZFBLYM)
FIFA 23
ps5
5. Unjudgable
Unjudgable:: this judgment is only to be used if the product cannot be judged, either because the
query cannot be understood, or because the product is lacking enough information to determine if it
matches the query. An Unjudgeable query is one that is complete gibberish, is seeking help, is
attempting to navigate to a website, is in a non-market language, or is a string of numbers that does
not correspond to an ISBN or model number in Google searches.
o
The product cannot be judged because the browser page will not load at all, the page
redirects you to the Amazon home page, or the page is entirely blank (404 error).
o
The product page is missing all substantial information (item details) needed to make a
judgment. (Missing photos do not count. If a product page is missing photos, base your
judgment on the product title and details).
o
There is no clear way to interpret the query because it is so vague, incoherent or
misspelled to the point where no reasonable intent can be speculated or gathered from
Google or Amazon.
o
The query is completely unreadable and is gibberish or a string of numbers that does not
correspond to an ISBN (product code) or any model number in Google searches.
o
The query or the product is written in a foreign language that does not match your
marketplace. (English is not considered a foreign language for this project – English is
acceptable in all marketplaces).
o
Navigational queries: ebay.com, Facebook.com, Rotten Tomatoes, Home Depot , etc.
should be rated Unjudgeable.
o
Help-seeking queries: “How to log into Facebook” or “what is the measurement of a twin
bed” should be rate Unjudgeable.
o
When there is a catalogue issue, example title says t-shirt and image is of gaming console,
then the judgement should be “Unjudgable”
S. No
Search Query
ASIN/ISBN
Socks
(B0895LL44V)
1
823^%0*
Pen
(B082PN4X5J)
2
Flipkart.com
3
博朗
4
How to cancel my
prime membership?
Chocolate
(B08J37H3QK)
Women’s black Top
(B07SN3KMS6)
Category
Judgemen
category is not
specified
Unjudgab
category is not
specified
Unjudgab
category is not
specified
Unjudgab
category is not
specified
Unjudgab
The Ghost and the Hemispheres Kindle edition
(B08KNSWQ5G)
5
Volume of
hemisphere with
radius 15cm?
category is not
specified
Unjudgab
category is not
specified
Unjudgab
Notebook
(B08TYJYD71)
6
Gift for mother
(searched in .com
EN)
General rules to consider
Supplementary information (price, shipping, sales etc.)
Always disregard any mention of price, shipping, sales, promotions, Amazon Prime, quantity,
condition (example: refurbished), or subjective terms mentioned in the query. See these examples:
o
"Best aloe vera cream": The term "best" is subjective – disregard it and judge the query as if it
only said "aloe vera cream".
o
"Cute affordable womens dress": The terms "cute" and "affordable" are subjective – treat this
as if it only says "womens dress" and judge items accordingly.
o
"Cat tree under 60 dollars": Always disregard any mention of price – treat this as if it only
says, "cat tree". Do not check ASINs prices – they are not important or relevant. We always
need to disregard price when we make our judgment.
o
"Wireless headphones prime": Prime refers to price and shipping, always disregard that from
the query and do not take it into account when you make your judgment of items.
o
"5 pack mens underwear": Disregard the quantity and treat the query as if it only said "mens
underwear". In this case even if the result were a 3-pack of mens underwear we could rate it
Exact because we always disregard quantity from both the query and results.
o
"Mens shirts": Disregard the plural written here because it is related to quantity. We can rate
listings that are selling either one or a set (plural) of men's shirts as Exact.
o
'Samsung monitor black Friday 2021": Black Friday relates to seasonal promotions/sales
Amazon has and should be disregarded completely. Treat the query as if it only says
"samsung monitor".
o
"IPhone 8 refurbished": Always disregard any specifications related to condition of the items
(new, used, and refurbished). The condition of items is not to be considered when you make
your judgment.
Note – Insufficient information rule should not be used as an excuse for insufficient research. Rather
a thorough research of the PDP and external sources should be done, as the answers can usually
be found for an ASIN looking at reviews and questions or Google. See examples below for better
understanding –
o
SQ – bulb dimmable and OP – https://www.amazon.com/dp/B079MCPNMF.
[Judgement: Irrelevant; Rationale: PDP mentions „Dimmable: No‟ under Specifications and it
won‟t work in dimming fixtures – information present in Q&A section, see details below]:
Q: Are these lights dimmable?
A: According to the product information, this A15 LED bulb is not dimmable. You can't use
any dimmer with it or install it on any dimmable light fixture.
By TORCHSTAR customer support on April 23, 2018.
o
SQ – ski socks; OP – https://www.amazon.com/dp/B0744KNYMG.
[Judgement: Irrelevant; The PDP does not mention „ski‟, and Ski Socks is a style of tall sock
that covers calf and is of thick warm material – information available on google searches.
These are not ski socks, so irrelevant].
Product not sold on Amazon - has updates
In each marketplace there are classes of products that are not carried by Amazon for legal and
logistical reasons. Customers are not necessarily aware of these constraints, and still issue queries
for many of these products. When evluating such queries, firstly you should think of a target product.
For SQ "gun" it will be a real gun, not toy one; for the SQ "cat" a real animal is considered.
Rules to evaluate are pretty simple:
o
If the OP could be used with or is desgined for the target product requested in the SQ, it
should be marked as Exact, since the SQ is quite ambiguous
o
In case the OP is just merchandise which only depicts items that are not sold (printed posters,
pet printed tshirt/mugs, poster of cars, toy or airsoft gun, Image of gun etc.), it should be
considered as Irrelevant
Consider following examples:
SQ: "gun";
OP: toy gun - will be Irrelevant as it is not used with the target product requested
SQ: "gun";
OP: gun case - will be Exact as it is designed to be used with real guns
SQ: "cat";
OP: cat poster - will be Irrelevant as it is just a picture of the cat
SQ: "cat";
OP: cat food - will be Exact as it is used to feed real cat
Misspelled queries
Queries which are misspelled may have multiple interpretations depending upon which letters are
substituted. Any spell correction offered may or may not be accurate to the customer's intent. If the
query is unclear, consider all common misspellings as different interpretations for the query,
you can use Google (locale google search engine, if Market place is judged in IN, the google
should be google.in) or Amazon search to identify common spell corrections. If a word is
spelled correctly and clearly a product query, then do not consider misspellings.
Misspelled queries
Query
ASIN
Sp
saree digayn
B07RG5VMR5
saree design
body wasglh
B006LXED4O
Body wash
boat blootuth herphonce
B08S16DMFH
Boat Bluetooth hea
ledis naiti suti
B071JHF6ZH
ladies nighty suit
Name related queries - has updates
Public figures:
o
If a person is one of primary contributors towards an offered book, video, music, art or is the
main actor then the OP is Exact
o
If OP is a media about the public person requested then the OP is Complement
o
Merchandise of the public person will be Complement too
o
If the SQ is asking for a person known for certain genre (like Jackie Chan and martial arts
films) then the OP of another genre, even with this person as a main character, should be
marked as Substitute
Public figures
Query
Exact
Co
Will Smith
Movies with Will Smith as a main character
T-shirt with Will Sm
Stephen king
Books written by him
Poster of Stephen k
Beyonce
Megan Thee Stallion - Savage Remix (feat.
Beyoncé)
Figurine of Beyonc
Historic/sport figures and fictional characters in the SQ are treated as a "subject" or a "topic".
Hence, anything which depicts the asked historic figure or the sports figure including books, t-shirt,
merchandise is Exact, as they do not have a primary product, unlike actors and musicians.
Sport/historical figures and fictional characters
Query
Exact
Mahatma Ghandi
Book on or by Mahatma Ghandi
Mahatma Ghandi fi
Ghandi image
Cristiano Ronaldo
Video of Ronaldo's match / his documentary / his
interview
Jerseys, images, fig
Ronaldo
Mickey Mouse
Mickey Mouse cartoons
Any toys, beach tow
Mickey
Deals related queries - has updates
SQ like "deals/offers of the day" should be evluated respective to the category, which could be either
explicitly specified in the task or be presented in the query itself. In case the requested category
meets the OP's category it will be an Exact. In case category is not specified, such query is
ambigous and the correct label will be Unjudgeable. For example:
o
SQ: "deals/offers of the day";
OP: T-shirt - will be Unjudgeable as the request is ambigious and no category provided with
the target product requested
SQ: "deals/offers of the day";
Category: apparel;
OP: Ladies skirt - will be Exact as the OP belongs to the category provided
SQ: "iphone deals of the day";
OP: iPhone 12 - will be Exact as the OP belongs to the category of rhe SQ
Golden Thumb Rule
Reviewers follow the Golden thumbs rule shown below to take accurate judgement.
o
Step 1: Research the search in Amazon or Google (less preferred);
o
Step 2: Find out which store (category) the query falls;
o
Step 3: Understand the ASIN provided with the help or Amazon or Google;
o
Step 4 (Final Step): Check the query and ASIN relevancy and select appropriate decisions.
Submit the offer.
Category - has updates
How to approach a Query based on alias: Know your product
Category: Not specified
When the search query clearly mentions the product type which clarifies the customer’s intent and the availability of the product
Amazon. The category selected is not specified, then we only have to mark the searched product as Exact.
For example: if the search query is basketball (category is not specified) , now we know Amazon sells basketball in this case w
will mark only the basketball should be marked as exact.
Pr
Category: Not specified
In many cases the search query doesn’t have any product type mentioned. And the category selected here is not specified. We ca
consider our search query as a theme. In such cases we can provide all the possible products related to the theme as SPT.
For example: the search query is Marvel (category is not specified) , now we know Amazon sells a lot of products related to marv
, in this case we will mark all the products as Exact such as t-shirts , books , DVD’s, figurines, etc.
Exception: When the product type is clearly understandable but the category selected doesn't
specify a different intend of the customer. In such cases, we will not take the category into
consideration while taking an enforcement action. For example: If the search query is „Jacket‟ and
the search alias is „Shoes & bags‟, in this case we will ignore the category and consider only Jackets
as the Exact product.
Category specific examples - has updates
1. Electronics
Sub Category
Query
Exact
Substitute
Complement
Cell phones
Samsung note 10+
Samsung note 10+
phone
iPhone XR
note 10+ cover +
protective glass
Computers &
peripherals
Asus Zen book
Asus Zen book 14
HP Elite book
Headphones
Gaming headphones
HyperX gaming
headphones
Apple ear pods+
case
laptop cover 13-15 i
headphone chargi
support
TV
TV support 55 inches
TV support Samsung
45-75 inches
console table with
universal TV stand
screws
Amazon devices
Echo dot
Echo dot
Echo dot + smart
bulb
Echo dot cover
Camera
Canon EOS 2000D
Canon EOS 2000D +
lens + tripod
Nikon cool pix
Watches
men's intelligent
watch
Fitbit versa 3 (unisex)
men's analog watch
SDXC memory car
GB
screen protector for
versa 3
2. Media and Games - has updates
Media primarily includes films/music/books/games etc. If the OP is the same product type, but from
different genre or theme it should be marked as Substitute. In case the OP is another media
product type then requested in the SQ, but falls under the same genre - it should be marked
as Complement. Complements also include reviews, analyses, study guides, anthologies, or
commentary, where the book/video is meant as a companion to the central product they are about.
Games here mainly refer to the video games including all PS models, Nintendo, all X boxes and
Windows games. The category also include game accessories like controllers, consoles, game
codes etc. Model compatibility is paramount in the videogames.
Sub Category
Query
Exact
Substitute
Complementary
Movies
Infinity war
(Action)
Infinity war
Spiderman
Infinity war themed products.
Books
The girl on the
train(
Suspense,
drama)
The girl on the
train(
Suspense,
drama)
The guest
list(Thriller,
suspense)
Book mark, book stickers.
COD ps4
COD PS4
game and all
COD themed
product for
PS4
PS4 GTA game
Games CD stand
ps5 charging
station
Charging
station for PS
5 console
Video games
Cables for charging station for
console
ps5 charging
station
Charging
station for PS
5 console
PS4 Console
Cases/Accessories/Games for Xbo
Console
Movies/Prime video
Harry Potter
Movie
Harry Potter
Movie
Fantastic Beasts
Harry Potter Book/ Harry Potter
Merchandise
DVD format
Endgame
DVD
Endgame
DVD
Endgame Bluray/ Endgame
prime video
DVD holder
One Direction
One Direction
music and
brand products
Linkin park
music
One Direction themed products
Justin Bieber
Vinyl
Justin Bieber
Vinyl
Harry Styles
Vinyl
Vinyl cleaning kit
Harry Potter
Book
Hardcover
Harry Potter
Book
Hardcover
Harry Potter
Kindle Version
Sticker for books
Snapchat
Snapchat
Instagram
Guide to use snapchat, Gift card to
buy Snapchat
Windows 10
Windows 10
Home
Windows server
2019
Windows 10 guide
Music/Vinyl/Audio/
Books
Apps/Software
3. Fashion - has updates
Sub
Category
Softlines
Query
Pink
Cushion
cover 50
x 50
Exact
Substitute
Pink Cushion covers, size
50*50
Blue Cushion covers, size
50*50
Complementary
Any decorative to
decorate the
cushion
covers/straps/zips
for cushion covers
Superman t shirt
Batman pants
Batman t shirt
Batman
tshirt
Differently themed t-shirt
(same purpose)
Pants/ jeans
Any Shirts
Tunic/ blouse
Apparels
Shirt
Sweatshirt/Pullover
(relative product)
Top
(anything
from
neck to
waistline)
Tunic
Pants
Jacket
Coat
Jumpsuit, 2–piece dress
A one piece garment
from top till ankles OR
Top till knees
Clothing
Dress
Buttons/Accessories
for a dress
Any pant / set of clothes,
which can be worn while
sleeping or lounging.
Pajama Top
Pajama /
PJs /
Pyjama
2-Piece Bikini
One piece swimsuit
Bikini Cover ups
Bikini
Necklace set with bracelet
and ring
Any Necklace
Necklace
Pendant
Jewelry
Different gem stone
Ruby Stone
Ruby
Wearables
Any other jewerly
like rings / bracelets
/ anklet
Wrist
Watch
Any Wrist watch / Smart
watch
Any jewelro made out of
ruby stone
Watch with bracelet
Watch strap, screen
protector and watch
tool kit
Any sneakers, sandals,
pumps, sliders, training
shoes
Shoe lace, socks
and insoles
Shoes
4. Luggage and bags - has updates
Luggage: By definition, luggage refers to empty bags or suitcases, used for carrying a person‟s
clothes and things on a journey. Anything which is used to carry personal stuff for the purpose of
travelling can be considered as a luggage.
Bags: a strong container made from cloth, plastic, leather, etc., usually with one or two handles,
used to carry things in when travelling, shopping, etc.
Sub
Category
Luggage
Query
Exact
Luggage
Types of Luggage:
Rolling Suitcases,
Backpacks, Duffel
Bags, Softside
Luggage, Hardside
Substitute
Complementary
Irrelev
Luggage, TwoWheeled Luggage,
Spinner Suitcases
Suitcase
Bags
Skybags
Suitcase
Skybags suitcase
Aristocrat
suitcase
Backpack
(same purpose)
Suitcase cover, wheels,
handle
Types of bags:
Handbag,
Backpacks, tote,
slings, Lunch bag,
carry bags, Gift
bags, etc.
A bag used for any
purpose
Bags
Bags for carr
Handbags
Caprese
Handbag
Caprese Tote bag /
sling bag / purse /
wallet
Lavie Tote bag
/ sling bag /
purse / wallet
Purses
MK Purse
MK Purse
Gucci Purse /
Handbags
Wallets
Allensolly
Wallet
Allensolly
Wallet/clutch
Tommy
Hilfiger wallet
/ clutch /
handbags
Bag Strap, Chain,bag
Charms
Suitcase / b
Suitcase / B
Zips/chains, Wallet charms
suitcase/ba
5. DIY & Tools
Sub Category
Query
Exact
Substitute
Art / Craft / DIY
Patchwork
Patches and scale for
patchwork
Beads or other
decorating products for
Complementar
Needle/ thread f
sewing the patchwo
textile
the textile
(if alias "category is not
specified",
chair/bedsheet having
patchwork)
Colors of wood
1.Varnish for wooden
windows/doors
2.Wooden pencil colors
Cloud decoration
cloud shaped mirror
1. Top coat or unde
used after and bef
painting the woo
2. brushes for pain
the wood
Star shaped decoration
Cloud deco DIY
DIY, Tools and
Garden
Nail (Tools)
Store: DIY
Bundle of different sizes
of nails
Nail & hammer
(bundle)
Solar Light DIY
solar light DIY kit
Store: DIY
Normal light DIY kit
(not solar)
Battery or any oth
single part for solar
1.tools/machines for
working with wood
2. book on the theme
Industrial /
Research and
development
products
wood working
accessory of the w
working tool
6. Beauty
Perfume, Cologne and Eau de Toilette are all exact.
Sub Category
Query
Exact
Substitute
Perfumes
Perfume
Spray(eau de toilette) /
Mist perfume Spray
Hair care &
products
Shampoo
Shampoo
Complementary
Empty perfume spray bo
Shampoo +
Conditioner
Conditioner, Any brush
scalp used for shampoo
Clips / caps to attach a
Wigs
Hair wigs
Oil
Store: Hair Product
Hair oil
Oil + Shampoo
Body care &
products
Moisturizer
Moisturizers
body cream
Face care &
products
Foundation
Foundation in liquid
form and Powder form
Hair Extensions
(relative product)
Empty bottles for oi
Brushes/pad to appl
foundation to face
7. Medicines
For any formula foods we need to ensure that the age of the baby/kids matches
Query
Category
Ratio
All medicines for that illness – E
different illness Irrelevant
Name of illness
category is not specified
EX – Headache
Headache syrup – Exact
illness with the form of medication
category is not specified
Only the form and illness mentio
will be Substitute.
EX – Query: Migraine Tablet
OP 1 – Migraine Tablet – Exact
OP 2 – Migraine soothing patch –
SQ – name of some vitamin or m
Supplements
category is not specified
Sub Category
Query
Exact
Substitute
OP – all the supplements for that
be Exact.
OP – supplement with that partic
minerals will be Substitute.
Complementary
W
Fat Burner
Dietary
Supplement
Book On Weight
Loss
Fever Reducer Pill
Cold and Runny
Nose Suspension
Thermometer
Weight
Loss
Supplements
F
Fever
Drug store
8. Grocery
Category
Query
Exact
Food
Tomato
Tomato, Tomato puree
Substitute
Tomato ketchup, all veggi
Tropicana orange Juice
Beverages
Apple juice
Orange
juice
Bourbon Whiskey
Rum
Alcohol
Whiskey
Freeze dried potato, potato c
Potato (raw)
Amazon
Fresh
Potato
Amazon
Pantry
Wheat
flour
5Kg
Wheat flour 5Kg X 3
Wheat flour 2.5Kg X 2
9. Pets - has updates
Pets are not sold on Amazon so the rule for goods not sold on Amazon should be aplied. Anything
used for pet care would be Exact, any merch/illustrations/clothes with pets will be Irrelevant
Sub Category
Query
Store
Exact
Substitute
Pet
Pet
pets/category is not
specified
Pet food, Pet
accessory
Complementa
10. Toys
Sub Category
Toys
Query
Exact
Substitute
Complementary
Board games
Ludo/Catan
Card games like Uno
Dices, pawns, chips
Harry Potter
costume
Harry potter costume
Any other character
costume from Harry
Potter like Dumbledore
or Hagrid
Harry Potter wand
Games for 5
year old
Games for 0-5 year
old
Games for 4 year old
and 6 year old
Guide to the game
11. Home
Sub Category
Query
Exact
Substitute
Complement
Furniture
Bed
Single bed/
Double bed
Sofa cum bed
Mattress/Bed Shee
Wall Art
Wall paper
Wall paper
Poster
Glue Stick
Bathroom
Bath Towel
Towel
Hand Towel
Hanger
Storage and Organization
Wardrobe
Almirah
Clothes Stand
Wardrobe light
12. Kitchen
Sub Category
Query
Exact
Substitute
Complement
Dinning
Plate
Plate
Thali/Bowl
Spoon
Cookware
Induction
Induction
Gas Stove
Induction cleane
Appliance
Chimney
Chimney
Exhaust Fan
Chimney pipe
Appliance
Microwave
Microwave
OTG (Oven Toaster
Grill)
Microwave plate
13. Automobiles - has updates
Sub Category
Query
Query
Exact
Automobiles
Car
category is not
specified
Acessories or
parts for this car
Substitute
Complementa
14. Shoes
Sub Category
Query
Query
Exact
Substitute
Complementary
Irre
Crocs
category is
not specified
Crocs for
any gender
Bathroom
Slippers
Sports
shoes for
women
category is
not specified
any sports /
running
shoes
formal shoes for
women
shoes and
bags
shoes,
wallet,
shoes laces,
etc from
Tommy
Hilfiger
shoes, wallet
and accessories
from other brand
Sole for sandals
Sport
any
laces for sports shoes
high
wome
shoes
or
Shoes
Tommy
Hilfiger
15. Fresh - has updates
Query
Exact
Substitute
Apple juice/puree
Apple
Apple
lugg
from
hilfige
from T
othe
Processed meat / salami / sausagese etc.
Any animal raw meat
Meat
16. Adult - has updates
Query
Exact
Substitute
Complement
electric
massager
electric
body massager
wooden
body massager
user guide
for electric massager
condoms XL
condoms XL
condoms L
lubricant
sex toy
for men
sex toy
for men
lubricant
17. Books - has updates
Please consider following tips, when evaluating books relevance:
o
Price constraints: if the SQ mentions "free" then any non-free bolks should be
marked Substitute;
o
Bestseller adjectives: if SQ mentions "bestseller" keyword or other similar, the OP should be
judged according to criterias:
Books with bestseller badges or books with more then 100 reviews - Exact
Books with less then 1000 reviews and without badges - Substitute
o
Service queries: SQ could contain request of the particluar media format/service, like
"audible plus", "kindle available" etc. OP, which match the format/service should be Exact, OP
of another product - Substitute
o
Age range: overlapping ages, like 3-4 and 4-5 should be marked as Exact. Adjustent ages,
like 3-4 and 5-6 should be marked as Substitute. If the difference is bigger - Irrelevant
o
Non-media merchandise: if an OP is a merch for a media requested in the SQ, it should be
marked as Substitute
Query
Exact
Substitute
Harry Potter
any Harry Potter
book
Harry Potter coffee mug
(non-media merch)
Complement
Defects types - has updates
This is a list of standard defect types, which shows different options of how the OP could be different
from the product requested in the SQ. You can use it like a cheat sheet to quickly find a response.
NOTE: label specific rules, like Brand Afinity, Same Purpose, Same theme occasion have higher
priority. Please, do consider them before making the final judgement.
Defect Type
Details
Decision
Product Type is Wrong
In this scenario the offered product
is an entirely different product, or
the product is not substitutable
Irrelevant
Query
Offer
Accessory
When the product is not the primary
product but is either accessory to the
product asked for but keeping
model, generation and year of
making in mind.
Complement
Query
Offer
Brand is Wrong
When brand of the result is different
but the product type is same.
Substitute
Query
Offer
Model/Sub-Brand is Wrong (1)
When the sub brand or model is
wrong for the lead product, and it is
functionally equivalent
Substitute
Query
Offer
Model/Sub-Brand is Wrong (2)
When the model/ sub brand is wrong
and it cannot be used for the same
purpose
Irrelevant
Query
Offer
Other Title/Person/Group is Wrong
When product title, person, or group
is different to the specified query.
Substitute
Query
Offer
Gender/Age Wrong
When the product’s gender/age is
different than specified in query.
Irrelevant
Query
Offer
Exception: +/-1 age rule.
Color is Wrong
When the color of the result is
different than specified in query.
Unless the requested color can be
selected on detail page.
Substitute
Query
Offer
Size is Wrong
When the size of product is different
from the one specified in query, by 1
size unit of difference.
Substitute
Query
Offer
Style/Feature/Theme/Material is
Wrong
When the product style, feature,
theme, or material is different that
specified in query.
Substitute
Query
Offer
Compatibility
When the product asked for in the
query is not compatible with the
offered product.
Irrelevant
Query
Offer
Bundled (1)
When the query is the lead product
and the offered product is the lead
product with its direct accessories.
Exact
Query
Offer
ampli
Bundled (2)
When the query is the lead product
and the offered product is lead
product with another
complementary lead product.
Substitute
Query
Offer
Cond
Accessory to Accessory
When the query is an accessory to
the lead product and offered product
is another accessory.
Irrelevant
Query
Offer
Purpose 1
When the product type (PT) is same
but the usage of the product is very
different
Irrelevant
Query
Offer
Purpose 2
When the PT is different but fulfills
the customer’s primary
requirement/demand. Deemed as a
style/feature defect
Substitute
Query
Offer
Misc.
When the query is a product in itself
but is also used as an ingredient.
Could be Substitute if product can
be used for the same purpose, such
as essential oils in a carrier oil
Irrelevant
(Depends on purpose of the product)
Query
Offer
essen