Cottonwood Ad & Web Guidelines Date Updated: 4/08/2020 This task has two parts. The first part is to identify which keywords from a list of candidate keywords reflect topics found within the context of a given document. The second part of the task is to match each identified topic to the Wikipedia entry that best describes the topic from multiple Wikipedia entries. Please read all of the instructions carefully before you begin the task. Objectives • • Part 1: Identify the keywords from a list of candidate keywords that best represent topics contained within the context of a document (ie: an ad or webpage) and select each one. Part 2: Match each topic to the best fit Wikipedia entry. Before You Begin As you begin the task, if any of the following issues occur, please skip the rest of the task by selecting the correct option at the bottom. If none of the issues occur, move on to Steps. • • • • Page Won't Load: Image fails to load. SRT (The rating tool) is throwing a widget error. Missing Context: For cases where the document does render a screenshot, but there is not enough content for the rater to rate like login page, a blank screen, etc. But if there is text and you can select topics from that please continue to do the task with the given text. Wrong Language: If the content is not fully comprehensible because it is in a language that was not intended for the given workflow. o Note: If the text on the document was auto-translated (usually a link that will say 'Show original'), then select 'Wrong Language.' This does not apply to text that has been auto-translated to English or to the expected workflow language from English. Jobs that load in English should be rated regardless of the market/queue language for the workflow. Sensitive: In instances where the webpage is sensitive, please mark as 'Sensitive'. 'Sensitive' is defined as containing or potentially containing any of the following: o Child Exploitation and/or Child Nudity; Self Injury and Suicidal Content; Credible Threats, Violence or Calls to Violence; Sexual Content and/or Nudity; Hate Speech; Acts of Terrorism; Human Trafficking; Bullying and Harassment; Profanity CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 o Instructions for Immediate Escalation — If potential Child Exploitation and/or Child Nudity, Self-Injury/Suicidal or TimeSensitive Credible Threat is the reason for marking as “Sensitive”, escalate the Job ID immediately to your manager. Rating Options 1. Submit: Click this button after you’ve made your selection(s) to advance to the next job. 2. Skip: Click this button if none of the candidate keywords are a topic of within the context of the document. Task Diagram & Definitions 1. The document is the ad or webpage being evaluated in this task. 2. The document text is all of the language content included in the document. 3. Candidate keywords are a list of unselected words and phrases pulled from the document text. 4. A topic is identified by selecting a keyword from the list of candidate keywords. 5. Wikipedia Entries are a list of Wikipedia links for each topic. 6. Relevant Wikipedia Entries is the best fit Wikipedia link for the topic in the given document. 7. The rating buttons are for submitting the rating to complete the task. 8. “|||” acts as break point between texts from different fields. Consider that as a break between two paragraphs. CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 PART 1: Topic Selection For this part of the task please read the text of the document carefully and then select the keywords from the list of candidate keywords that are topics. A topic is an idea or subject presented within the context of a document. A topic can be represented as an interest to the reader or a focus point of what a document is about. A document may contain multiple topics or only one. • Tip: To test whether a candidate qualifies as a topic, try side searching it using your favorite search engine. Look at the search results retrieved, could the document reasonably be among the search results? If yes, then it is a topic. Instructions for selecting the topic: Thought: Read the document carefully. Ask yourself, what is this document about? What ideas and topics are presented? Does the document contain one topic or several? Next, consider each of the candidates in the list. Does the candidate reasonably represent topics and ideas within the context of the document? Action: Click the checkbox next to each topic candidate contained in the context of the document. If there are multiple topics, select all of them, including synonyms and variants. There is no minimum view time required to complete the task. If possible, please do not exceed 60 seconds per task. Steps 1. Review the document carefully. Read all of the text and consider the image. Think about the ideas and topics that are presented to the reader. 2. Consider the list of candidates. Do any of the candidates represent topics contained within the context of the document? a. Yes, check the box of each word and phrase that represents a topic contained within the context of the document. b. No, go to step 3 3. Is the topic(s) present in the text of the document but not included in the candidates? a. Yes, select “Topic Not Highlighted” and click submit to move on to the next job. b. No, there is not an identifiable topic presented in the text of the document. Click Skip to move on to the next job. How to select a topic When selecting topics from the list of candidates, please keep in mind the following conditions: CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 SELECT • Synonyms are words and phrases that have exactly, or nearly the same, meaning. In cases where synonyms are used to refer to the same topic, select all of them. o Ex: “Financial revenue” vs “Financial income.” Please select all the synonyms of the topic from the list of keywords. o Tip: There is no coherence between one topic and another. Consider each candidate keyword individually to determine whether it is a topic. • Variants are slight discrepancies of the same word or phrase. Variants include, but are not limited to, case change, tense change, plural vs singular, and word reordering. In cases where the same topic is represented by variant spellings of the same word, select all of them. o Example: ‘Atlanta’ vs ‘atlanta’; “homes for sale” vs “home sale”. Please select all the variants of the topic from the list of keywords • Select only the whole phrase of a topic and not individual words in the phrase of a topic. o Ex: “Service now” is the topic of the document. “Service Now” is a name of the company but if the words “service” and “now” are individually present in the list of keywords, do not select them both as topics because individually they do not mean “Service now.” o Exception: Select part of topic when its synonymous to the whole topic ▪ Ex: In “Tata consultancy services“, just the keyword ”Tata“ could also refer to the company itself. In that, case please select ”Tata“. o Ex: “Captain Jack sparrow”, can also be inferred from “Jack sparrow” but do not select “captain” as it doesn’t refer to the same topic when selected separately. • Places Only select the places when it is the focus of the document. o Ex: Travel deals to Paris, real estate ads, opening of new store in a location are cases where the place itself is the focus of the document • Topic Not Highlighted: Click this button if none of the candidates represent a topic contained in the context of the document. o When none of the candidates represent a topic contained in the context of the document. o When only some of the topics are listed in the candidates • Enumeration: There can be instances where a document is talking about multiple similar topics. Please select all topics. For ex: A concert ad can mention 10 different cities. Please select all cities. DO NOT SELECT • Ambiguous words or phrases CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 o Unclear or inexact words that are either not meaningful on their own or could have multiple meanings without additional context ▪ Ex: “is to,” “is a” are meaningless phrases, please do not select them. TIP 1. If you are ever in doubt whether the candidate keyword is a topic lean towards not selecting that candidate keyword. 2. Ask yourself, if you have to represent the given document with only #hashtags, would you include that particular candidate keyword? --------------------------------------End of Part 1 Instruction ---------------------------------- PART 2: Match topic to best fit Wikipedia entry For the second part of the task please match each selected topic to its best fit Wikipedia entry. The best fit Wikipedia entry is the one that best describes or defines the topic as it is referenced in the context of the document. For example: In the document, “DJT is hosting a republican rally in Florida on June 7th” DJT has several entries in Wikipedia. In this context “Donald John Trump” would the relevant Wikipedia entry. Guidance for selecting relevant Wikipedia entry: Thought: For each candidate, understand the meaning in the given context of the document. In this context does this word refer to a person? or a place? or a title of a book or movie? or a concept described in the document? Action: Click the checkbox of the Wikipedia entry, which correctly identifies the meaning of the candidate in the given context. CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 Example: LOGIC FOR SELECTING THE WIKIPEDIA ENTRY FOR THE CANDIDATE Steps 1. Review the document carefully. Identify the candidate in the text and understand the meaning of the candidate in the document. 2. Click each of the provided links to review the Wikipedia entries. Does any of these Wikipedia entries reflects the meaning of the candidate in the given context? a. Yes, select the box of the Wikipedia entry. Click submit to advance to the next job. Well done! b. No, go to step 3 3. Do none of the Wikipedia entries define the candidate? a. Yes, select “None of the Wiki is accurate” and then click submit to move on to the next job. How to select a relevant Wikipedia entry When selecting a relevant Wikipedia entry, please keep in mind the following conditions: SELECT: • Only one relevant Wikipedia entry for each candidate • The Wikipedia entry should be specific to the context of the document o Ex: Salem can be town in England or a town in India. If the ad is about a real estate in Salem (England), please select the Wikipedia entry that maps to Salem, England. • Candidates can be used in colloquial terms in the document, please select their Wikipedia entry which maps to its formal term. o Ex: “She is an ambassador for all women of color”. “https://en.Wikipedia.org/Wiki/Ambassador“ says Ambassador is a diplomat representing the country. In the context of the document, ambassador means a representative of a group. Please feel free to select the Wikipedia entries that provides same meaning even if it’s in a more formal way. DO NOT SELECT: • If the Wikipedia link leads to a list of Wikipedia pages, do not select them even if it has a relevant Wikipedia entry in the list. Ex: If the document was about Ruth tester but the Wikipedia entry present was for tester(surname), do not select this entry even if “Ruth tester” is on the list. CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 Example Cases 1.The ad here is talking about a Labor Day sale of 30% off specifically for bottoms and dresses at Plato’s Closet Napercille, IL. • Topic: bottoms, dresses • Plato’s Closet Napercille, IL is not in the list of candidate keywords so please select “Topic not listed” • Even though the store is located in Napercille, IL, it is not the focus of the document. Do not select them. • None of the Wikipedia entry for “bottoms” means an apparel worn on lower body. So please select “None of the Wiki is accurate” • Dresses has both “clothing” and “dress” in the list of Wikipedia entries but “dress” is more specific than “clothing.” Please select the most specific Wikipedia entry. CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 2. The focus of the ad here is about watching Spiderman : Far from Home at Regal Cinemas. • Topic(s): spider - man: far from home, spider - man, regal • None of the Wikipedia entries for spider-man actually means spider - man: far from home movie. Select none of the above when those link don’t represent the exact meaning of the topic in the given context. CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 3.The ad here is about physical therapy. • Topic: physical therapy, physical therapists • Do not select the places, as they are not the focus/idea of the document and only mentioned in passing. • The Wikipedia entry for physical therapist is not physical therapy; please select “None of the Wiki is accurate”. CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 4.The focus of the ad is the Emperor and Beauties rpg game. • Topic: rpg game, simulation Rpg, palace and since the name of the game “ Emperor and Beauties” is not preset select “Topic not listed” CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 5.The focus of the ad is about sale in Andar happening during Labor Day weekend. • Topics: sale, labor day weekend, discount, Andar CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 6.The focus of the ad is Joe Holland Hyundai dealership. • Topics: hyundai, dealership, joe holland • Do not select the location of the dealership as it is not the focus of the document. • None of the Wikipedia entry for Joe holland actually represents the dealership. CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 7.The focus of the ad is Elgin realty LLC in Elgin, OK about relocation to Fort Sill, OK. • Topics: Elgin, Elgin ok, realty, relocation, property, elgin oklahoma, home, new home, fort sill, moving o In real estate ads they are not just selling homes, they are selling homes in a particular location. Places are topics in this case. • Since “Elgin realty LLC” is not in the list select “Topic not listed” as well. CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 Issue Buttons: When a job is not rated, it is important to understand the reason why. Inappropriately choosing an issue button counts as a wrong answer and confuses the data. The examples below illustrate the correct cases for when to select Page Won't Load and Missing Context. CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 Page Won't Load Select 'Page Won't Load' when the image fails to load. Please note, if there is a text on the right side, select the topics! It looks like this: PAGE WON’T LOAD Tip: When only partial page loads in the tool, please give it a few seconds before you choose ‘Page won’t Load.’ The job is not ‘Page won’t load’ when it takes few seconds to load. CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 The following example is NOT “PAGE WONT LOAD”. Please proceed with these tasks normally! Missing Context: Select 'Missing Context' if the document is not ratable. It could be a web portal, login/registration screen or a pop-up blocking the contextual information. Please CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020 note, if there is text and you can select topics from that please continue to do the task with the given text. CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 4/08/2020
