Llifelog: Discovering and Evaluating Research Projects through Peer-to-Peer Exchange ARCIVES MFA(SHUSETTS 'NSTITUTE oR mEoo SJl7 by Li Bian LBRARE_ Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, in partial fulfillment of the requirements for the degree of Master of Science in Media Arts and Sciences at the OF TECHNOLOGY INSTITUTE MASSACHUSETTS June 2012 0 2012 Massachusetts Institute of Technology. All rights reserved Author Li Bian Program in Media Arts and Sciences May 5, 2012 Certified by U (j enry Holtzman Research Scientist Media Laboratory Thesis Supervisor Accepted by Mitchel Resnick LEGO Papert Professor of Learning Research Program in Media Arts and Sciences 1 2 Llifelog: Discovering and Evaluating Research Projects through Peer-to-Peer Exchange by Li Bian Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, in partial fulfillment of the requirements for the degree of Master of Science in Media Arts and Sciences at the Massachusetts Institute of Technology June 2012 Abstract Llifelog is a web platform for discovering, evaluating, and exchanging research projects among students and researchers from both academia and industry. The goal of Llifelog is foster a community of critique among students and researchers in a project market. Those who engage in the market answer the call "find out how your work, projects, and ideas are valued in the community." The unique contribution of the Llifelog system lies in designing market-based mechanisms that stimulate the project exchange while maintaining information confidentiality, and devising a peer-to-peer, direct communication channel between project creators and project viewers. In this thesis, I explain how such market mechanism results more objective evaluation compared to other existing rating mechanisms online. A virtual credit system, Truons, is used in this web platform to facilitate the project evaluation and exchange. The website was built using Java, MySQL, Apache Tomcat, HTML, CSS, and Javascript and hosted on an Ubuntu server. It also uses a Natural Language Processing engine, AlchemyAPI, and various image processing techniques. The platform was tested among about 305 users with 74 projects within 3 weeks. User feedback was also gathered after the testing to further understand the context and rationale for user behavior on the web platform. Thesis Supervisor: Henry Holtzman Title: Principal Research Scientist 3 4 Llifelog: Discover and Evaluate Research Projects through One-on-One Peer-to-Peer Exchange by Li Bian The following people served as readers for this thesis: Thesis Reader Pattie Maes rofessor of Media Arts and Sciences Program in Media Arts and Sciences Thesis Reader Sep Kamvar Associate Professor of Media Arts and Sciences Program in Media Arts and Sciences Thesis Reader Joichi Ito Director Media Laboratory 5 6 Acknowledgements This thesis project represents the evolution of my passion and thinking since entering the Media, Arts, and Sciences program for research almost 2 years ago. Without an open environment, interdisciplinary culture, and tremendous creativity such as that of the Media Lab, I would not have been able to reach this stage today and this project would not have come to fruition. Thus, I would like to express my deep gratitude for the following people: Henry Hotlzman, my advisor and thesis supervisor. Thank you for giving me the freedom to stretch my imagination as far as it could reach and providing me immense research resources. I will never forget the long-hour discussions we had about various research topics, gadgets, fashion, and about how I'd fail the Turing Test as an android. Thank you for your tremendous support for my China Initiative activities outside the "normal" research scope. But believe it or not, my initial inspiration for the market-based project evaluation platform has come under the grilling inquiries from the sponsors about our projects. Pattie Maes, thank you for being an inspirational figure in my life since I first took your class in undergrad of MIT. Your open yet pragmatic approach to solving problems has taught me more than just how to do research in school but also how to manage the ups and downs in life. Henry Lieberman, Sandy Pentland, Sep Kamvar, and Joichi Ito, thank you for all your advice and feedback on my thesis project. Thank you for pushing me hard to think more critically about many important aspects of this research. Matt Hirsch, Greg Elliot, David Carr, Dan Schultz, Arlene Ducao, and Rob Hemsley, thank you for being such wonderful group members! You, and bubble tea, were the spice of my research life at the Media Lab. Wei Pan and Dawei Shen, thank you for being the big brothers in my life in the past two years. You were always there when I needed prompt help or advice. I feel fortunate to have known 7 both of you, worked hard on the China Initiative with both of you, and receiving guidance from you. All the Media Lab sponsors who have given me the chance to demo projects to you, thank you for coming! You were the best channel for me to reach out to the real world for the most practical feedback. Special thanks to Hisense, for being one of the test beds for my thesis project. I owe you much for your support and collaboration. All the friends in the Media Lab community, Jinha Lee, Santiago Alfaro, Xiao Xiao, Lining Yao, Roy Shilkrot, Valentin Heun, and many others whose faces appear in my mind vividly now, it is a pleasure to have known all of you. Thank you for all the fun time together and the serious discussions about research together! Lastly, a big thank you to my friends across the globe who helped me test my thesis project website. That includes my high school / undergraduate friends who now go to other universities or work at top technology companies such as Google, my former managers and colleagues at Microsoft and Nokia, and numerous friends that I met by luck in life throughout the years. Thank you all for using the website and providing me the most valuable insights for how to improve the work in the future! 8 9 Contents A b strac t ....................................................................................................................................... 3 A cknow ledgem ents ...................................................................................................................... 7 1. Introduction ............................................................................................................................. 13 1.1 Motivation ................................................................................................................... 14 1.2 The V alue Proposition Dilemm a ........................................................................... 16 1.3 Approach .............................................................................................................. 16 1.4 Contributions ........................................................................................................... 17 1.5 Outlinend .Rela ek........ 20 ............................................................................................. 2. Background and Related W ork................................................................................ 2.1 Relatedw ork.................................................................................................... ..... 21 ...21 ...................................................................................................... 28 3.1 Design M ethod.......................................................................................................... 28 3.2 Process ....................................................................................................................... 29 3. Design.......... 4. Implem entation .... .................................................................................................... 36 4.1 The Prototype .............................................................................................................. 36 4.2 Overview .............................................................................................................. 45 4.3 System Architecture.............................................................................................. 47 5. Evaluation and. u........... ............................................................................................... 49 5.1 U ser Testing.. ................................................................................................................ 49 5.2 Data Analysis............................................................................... 52 5.3 U ser Feedback ............................................................................ 56 6. Conclusions and Future Work...................................................................... 59 Bibliography .............................................................................................. 62 A ppendi.x ...................................................................................................................................... 71 A . Final Round Survey Questionnaire ............................................................................... 71 10 List of figures Figure 2.1 Quora's interface for asking and answering a question with Quora Credits............22 Figure 2.2 Barter's home page listing all information shared on the website....................23 24 Figure 2.3 Barter's "Economy" page ................................................................. Figure 2.4 Types of rating activities for the three datasets used in the study....................26 Figure 2.5 Dataset from CouchSurfing.com.............................................................27 Figure 3.1 User-Centered Design Process...............................................................29 Figure 3.2 Two rounds of iteration for the development of the project..............................31 Figure 4.1 The home page that a user lands on as soon as she signs in..............................37 Figure 4.2 The preview of a project on the market...................................................39 Figure 4.3 The rating of a project.....................................................................40 Figure 4.4 The display of full content of a project......................................................42 Figure 4.5 The project portfolio page of a user......................................................43 Figure 4.6 Simple and easy browsing of projects on the portfolio page...........................44 Figure 4.7 Built-in feedback system on the website....................................................45 Figure 4.8 An overview of Llifelog web platform.....................................................46 Figure 5.1 Website traffic monitored through Google Analytics................................. 50 Figure 5.2 Geographic usage monitored through Google Analytics..............................51 Figure 5.3 Number of exchanges versus number of images in a preview for a project ....... 53 Figure 5.4 Project rating versus the number of images in a preview for a project ................. 55 11 12 W Introduction "Information Wants To Be Free. Information also wants to be expensive.. .That tension will not go away." -- Stewart Brand, The Media Lab: Inventing the Future at MIT. 1988 This thesis is at the heart of that tension. We live in an information society under an economic system of capitalism. For information markets to flow, we need to be able to quantify and preserve the value of information, yet once we hand over information, we risk having lost control of it. The fickle nature of information poses challenges to creating online information markets: reveal too much of the information in advance of a transaction and the buyer could get all she needs for free; present too little and risk nobody showing interest. Furthermore, the value of information is often greatly affected by how it is framed or organized. The author presents a novel approach to organizing information for exchange over an information market, a method for previewing the information and its organization without diminishing the information's value, and a rating system for assigning quantitative value to the information to ensure that the value of the information can be accurately predicted from the preview. This contribution has been implemented and evaluated in the context of an example information market, a market for research projects in computing technology fields. This market is not about the exchange of money, but rather aims to foster a community of critique around student and workplace projects. Those who engage in the market answer the call "find out how your work, projects, and ideas are valued in the community." 13 1.1 Motivation We live in a world of abundance in information. Our generation has witnessed the evolution from the World Wide Web to the "Web of the World" in which mobile communications, social technologies and sensors are connecting people, the internet and the physical world into one interconnected network. Our identities are authenticated and recorded, our location tracked, our speech translated and gestures recognized; we are scrutinized by today's information technology. With Facebook [13], Linkedin [22], FourSquare [14], and Amazon [4], there seems to be no aspect in our life that is not already recorded in real time, stored in the cloud, and analyzed for better, "personalized" services. To make the case more sound, the more active information citizens among us faithfully and tirelessly contribute to the continued digital encyclopedia of human knowledge, by sharing our ideas, questions and answers on Ask.com [6], Yahoo Answers [33], Quora [25], Wikipedia [31] and the likes. Among the technical savvy ones, making what one has built "open source" is what is popular. And so we hear shout-outs from the overwhelmed crowd, "Information is free!" Quietly yet swiftly, another movement emerges. Diaspora [11], a notable attempt launched in 2010 that allowed every individual person to take control of his or her own information, rose among the numerous Kickstarter [20] backed start-ups as the highest funded one-it even got funding from Facebook CEO Mark Zuckerberg. Following Diaspora, a new wave of start-ups is thriving on the premise that users want to take more control over the information they have been freely sharing. Some start-ups take an even more pragmatic approach: letting users trade their data for monetary value. But these start-ups aren't innovating new concepts: academia for several years has been analyzing the dynamics of personal data privacy and information confidentiality and producing prototypes for trading such information with virtual or real currency. For example, Barter [7], a wisdom exchange platform from Viral Communication Group of the MIT Media Lab, has already run beta testing among 5000 users around the globe in the course of 2 years. A series of work on Reality Mining [26], spearheaded by Nathan Eagle and Alex Pentland over the past decade from the Human Dynamics Group at the Media Lab, has seen its derivatives in the privacy control and information trading domain in the past few years since the rise of online social media and networked data. A comprehensive summary of this 14 ongoing phenomenon is provided in the white paper of the World Economic Forum in 2011, which systematically linked the social, economic, and technological impact as a result of the " 2 1st century's new oil", data [32]. "Information wants to be free. Information also wants to be expensive... That tension will not go away." In 1988, Stewart Brand famously quoted this in his book The Media Lab: Inventing the Future at MIT [8]. As of today, in the year of 2012, this still appears to hold true. The MIT Media Lab, founded in 1985, manifests this tension itself. Numerous projects coming out of the Lab have benefitted the world greatly as open source. For example, Processing, a programming language, development environment, and online community that since 2001 has promoted software literacy within the visual arts, was created by Ben Fry and Casey Reas in 2001 while both were John Maeda's students at the MIT Media Lab [27]. DepthJS, a recent open-source project ran by Aaron Zinman, Doug Fritz, Greg Elliott, and Roy Shilkrot successfully connected Microsoft's Kinect motion-sensing technology with Javascript to allow gesture-based web browsing [10]; the source code is hosted in github, an open source code sharing website for software developers [16]. The list of Media Lab open source projects goes on, but the majority of the Lab's projects are not open source. The sponsorship model of the Media Lab requires each member company pay a minimum of $200,000 annual fee to access the rest of the Lab's project information, ideas, source code, and publications, not to mention engagement with the Lab's most valuable asset, the talent pool [21]. At first glance, the Media Lab is like a vertical Google of research projects, except it resembles one with a "freemium and premium" model where people need to pay money to use this information platform, for unlimited research discovery, evaluation, and exchange. My deep involvement in some of the Lab's sponsor engagement activities, especially that from the Media Lab China Initiative [23], has inevitably forced me to question critically the validity and practicality of the campaign of both "free information" and "monetization of information". And it has come to no surprise that talking with different advisors at the Lab about this issue, I have identified a wide range of thoughts and opinions across the whole spectrum from free to not free. 15 1.2 The Value Proposition Dilemma As the connection between information and its economic potential becomes more evident in today's big data world, businesses and research organizations both face an inevitable value proposition dilemma: which direction to focus now to embrace tomorrow's information transformation: direct financial return from information trading markets, or collective evaluation of information to unleash the potential of new business and research opportunities? While the former seems immediately foreseeable, the latter is an unknown land waiting for exploration. Many have failed before in trying to combine the two. For example, Google Answers was an online knowledge market released by Google in 2002 that allowed users to post bounties for well researched answers to their queries [18]. Asker-accepted answers cost $2 to $200 and Google retained 25% of the researcher's reward and a 50 cent fee per question. However, the service was shut down by Google in the end of 2006 possibly due to decreasing number of users. Quora, another Q&A website launched in 2010, allow its users to spend Quora Credits to get answers to questions and earn Quora Credits when others user give credits to request answers or when their content gets up-votes. As of today, Quora has not allowed cash-in or cash-out of the Quora Credits system yet, but instead has been using the system to direct the site's question and answer flow and maintains the quality of content on its website. 1.3 Approach Llifelog borrows ideas from the real world economy and tries to apply market mechanisms in a digital information exchange platform with virtual currency. Contrary to Google Answers, Llifelog does not encourage the culture of expert promotion through real money transaction; instead, it fosters a community of crowd-sourced critique on research projects. At the same time, Llifelog upholds the value of quantitative rating beyond qualitative rating, due to the fact that a quantified system often provides more stringent metrics for evaluation. However, assigning a numerical rating to creative expression, such as a project description, is difficult, if not only novel. Moreover, digital information exchange pertains to the intrinsic atomic nature of information, i.e. giving too much to lose selling power or giving too little to lose user incentive. Mimicking existing real life examples in the music, movie, and 16 publication industry, Llifelog adopts a preview method to allow users to get a glimpse of the information they are about to receive. Same as in the music, movie or publication industry, the quality of a preview can prove to be a good summary and reference to the actual content or leave the audience uninterested. The author believes that the preview method is the right way to go but that there needs to be many iterative testing and tweak before getting the preview right. 1.4 Contributions Motivated by such problem and opportunity, the author designed and implemented a web platform, Llifelog, which allows students and researchers from both academia and industry across the US-and some other parts of the world-to discover, evaluate, and exchange their research projects with one another. The goal of this platform is to foster a community of critique around student and workplace research projects. Those who engage in the market answer the call "find out how your work, projects, and ideas are valued in the community." Before jumping into descriptions of what this platform is or does, I find it necessary to make a list of things that this platform is not and does not do, lest any confusion comes up later: 1. Exchange of money. Llifelog is not a money exchange platform. It does not allow or facilitate buying or selling of anything with real currency on the website. The market mechanism and virtual credit system used on the website is for the purpose of stimulating project exchange and encouraging project sharing with a gamification flavor. In Chapter 2, Background and Related Work, I will give examples of other web platforms that also use market mechanism and virtual credit for similar purpose for different content sharing. 2. Intellectual Property (IP) trading. The author has been contacted by various peoplegovernment IP trading officers, business professionals, the website's own userswhether IP trading is possible on the website. This thesis project done at the Media Lab, Llifelog, is not for IP trading and does not facilitate or take responsibility for that. 3. "Yet another academic research archive"-Yes and no. Yes, because the author did spend quite a lot of effort in figuring out an interface that allows users to more easily 17 document, upload and save their projects online. No, because this web platform is not for archive purpose, but designed for better discovery, peer-evaluation, and exchange. As mentioned above, Llifelog is a web platform for students and researchers from both research academia and industry to discover, evaluate and exchange their research projects with one another. In order to achieve this goal, a complete package of web features need to be provided, so that the users get to experience the entire interaction flow and the author gets to collect real time data for analysis for this thesis project. The major contributions of this whole package include: 1. Providing a vertical search web platform dedicated to research project discovery across different research organizations. Allowing the users to search based not only on the content of projects but also on meta-data such as topic, project creator, organization, and location. Although with only a few hundred users on the website from the 3-week user testing due to the scope of this thesis project, the diversity of users-from MIT, CMU, Stanford, Qinghua, Singapore University of Technology and Design, Hisense, Microsoft, Nokia, Toshiba-has made it interesting for project discovery and comparison. 2. Designing a minimal user interface for content creation and sharing, as well as a sleek interface for displaying a user's project portfolio. This part of the interface design has received positive feedback from many users, which demonstrated that with today's advanced and mature web technology, designing a tool that is useful, usable, and desirable is more important than the technology tool alone. 3. Devising a peer-to-peer, direct communication channel between project creators and project viewers. This private messaging system for project exchange has saved users' time and effort in searching online for the best way to reach out to one another. It also functions as the backbone for project evaluation as it facilitates back-and-forth discussion and assessment about a project. 4. Designing market-based mechanisms that stimulate project exchange while maintaining information confidentiality. A virtual credit system, Truons, is used in this web platform to help users quantitatively measure and assess the value of each 18 and every project. The exchange and evaluation statistics are anonymously shared for every project so that each user gets to know what the rest of the community collectively thinks of a project. Such quantitative evaluation also comes with qualitative feedback provided through the private messaging system described in #3. 5. Automatically generating a preview based on a project's full content. After users create their project content using the interface described in #3, the system automatically generates a preview of the project using natural language processing and image processing techniques. It then renders the preview on HTML5 canvas. Previews are what go on to the project market when a user shares a project. This way, we allow the students and researchers to take more control of and be aware of what they are sharing with the public on the project market. Llifelog was tested in two rounds, first in October 2011 before the system was completed with 120 users over one week, and second in February 2012 when the system was complete with 300 users over three weeks. The user study and testing provided much insight into the research community's problems and concerns with sharing their work using existing tools and platforms, their desire for more efficient sharing with a targeted audience while maintaining information confidentiality, as well as how the market-based platform can help them better evaluate one another's projects which are not always conducted for the purpose of conference paper publication. Another interesting phenomenon observed is that the platform's user base seemed to quickly get adapted to using the virtual credit system, Truon, for measuring their project value quantitatively. The pattern of average scoring of a project given its content quality and quantity will be discussed in details in Chapter 5. 19 1.5 Outline This thesis has six chapters. The first chapter, Introduction, provides a general introduction and motivation behind the project along with the list of major contributions. In Chapter 2, Related Work, we will show examples of related work and web platforms that are similar to Llifelog. Chapter 3, Design, depicts the whole design process and work flow of this project which has taken almost a full academic year. Chapter 4, Implementation, provides the technical details of the software system and how each of the five contributions listed above was implemented. Then in Chapter 5, Evaluation, we take a look at users' onsite behavior patterns and statistics from a system backend perspective as well as using data from Google Analytics. We also share the feedback that users have provided during and after the testing. Lastly, Chapter 6, Conclusion and Future Work, presents conclusive remarks regarding Llifelog and discusses future possibilities for this project. 20 E Related Work "It was six men of Indostan, To learning much inclined, Who went to see the Elephant, (Though all of them were blind), That each by observation, Might satisfy his mind." --John Godfrey Saxe 2.1 Related Work Quora, the question-and-answer website mentioned in the previous section, is a self- contained market platform that scores every question and answer. Each user is given 500 Quora Credits to spend upon signing up. In order to earn more Quora Credits, the user has to post answers good enough that other users would send request to him and possibly give the answers up-votes. Figure 2.3 shows an example of the Quora interface for asking the question "What is Ashton Kutcher's diet?" and the list of answers along with their creators and asking credits. Such mechanism forces the users to think more when they choose the answer(s), which should help with evaluation of the answers and improving the site's content quality in general. However, as we could see in the screenshot below, there is too little information displayed about the answers or the creators of the answers for a user to decide which answer to give credits to. Because of this extreme lack of meta-information, it is difficult to make sense of the difference implied by the credits "415", "50", and "25". A glimpse of the answer as well as the answerer's background is needed in this case to facilitate an asker's choosing process. In addition, it is unclear how many people have asked the same question and how many times each answer has been given. There is also no space for negotiation or bargain, so for each answer, the asker is left in only a choose-or-not choose situation that does not stimulate further conversation exchange between the asker and the answerer. 21 Quora ISearch your boards and all of Quora Ashton Kutcher Diets / Et VWhat is Ashton Kutchers diet? -I Add Question Detais W Comment Post Wiki Options PeJirect Question You have 461 credits Ask to Answer - Find People Jason Putorti, Co-Founder, Votizen 1 Answer in Ashton Kutcher Brian Wang, Experienced in a a 2 Answers in Diets ... Johanna Brewer, co-founder of frest... 415 credits to Ask 25 credits to Ask 50 credits to Ask View More Suggestions Figure 2.1 Quora's interface for asking and answering a question with Quora Credits. A more sophisticated knowledge sharing web platform similar to Quora is Barter by Dawei Shen from the MIT Media Lab, Viral Communication Group. Barter's goal is "to design and create a market platform that incentivizes decentralized information creation within a community." This platform has been deployed at the Media Lab's Media Ventures class to help students share their venture ideas and exchange knowledge. Comparing with Quora, Barter's interface is more explicit in giving the users a glimpse of the full content that they are about to view or interact with through the content's title and a meta-description field called "details" filled out by the content creator. It also shows the statistics of how many times a file or idea has been "purchased" through an on-click popup. However, a close investigation of the listed files and ideas shared reveals that the "details" that the content creators have filled out are mostly subjective and often times self-promotional. In some cases, the users reveal everything in the content in the "details" field, which defeats the purpose of using the site's virtual credits for exchange, as no one would still have the incentive to give out credits when she already gets every piece of the information. Additionally, Barter is a general knowledge exchange platform, 22 as we can see on the top right navigation menu. One can share ideas, Q&A, documents, and The "Economy" page shown in Figure 2.5 visualizes a user's credit history and news. distribution within the Barter system in studious details. While the numerous functionalities contribute to the technical merit of the platform, from a user interaction design perspective, much would need to be truncated or simplified for a narrowed focus of content type. In the next Chapter, Design, the author will go through the present thesis project's design process and how Llifelog started broad as well and iteratively narrowed down its focus through multiple rounds of user testing and numerous brainstorming, design, and redesign. *QS 9 *Search Al DISPLAY: BOTH - Vote for Funf! The Funf guys are trying to qualfy for Knight Foundation funding Ui im Idea SORT: HOT* Sud" Ners shaed byManuLohiya $10 - 4 +1 - Details- 22 days ago Q&A Doc Auction News Fix My Stats Tt" BonUS Eunried F m, My Recent Transaictions Tribs U ianis not n anytribes. Al Tribes Overview Top Tags http:// radar.oreiny.com/ 2012/03/planning-bigdata.html Idea shared byTomSnith 0 - Details- 26da y ago 00 , OnMotion: real-time tremor monitoring and feedback for Parkinson's patients (41) ACOVy Pric +3 13mns ago Login to Barter +3 14 days ago A montlysusidy amount +200 14 daysago 27dav ago Price shared byJennfwe Jordan 3 - 4, +5 - Detaiis - 4 daysago Idea (23) 34 daysago (15) (14) (14) %arowaeTags Anwao Lognto arter NoMyTransct!on -onowa Mobie Gaming for National Security Idea shared bySi Young 1 -i +3 - Detais . 52 days ago Price + Top Users This Wedi No top eswsyeV Figure 2.2 Barter's home page listing all information (ideas, Q&A, documents, and news) shared on the website 23 IbortSr 0 Ea Economy Idea Money Supply Distrbution of Wealth Dstribution of Earning zoo 250 Q&A Doc News Auction Fix WeathEawg ZOO ISO 1%5 100 100 0 20 0 400 600 S00 100 1200 1400 g00 1000 Weath Transactions by Market 120D 1400 160 100 Earig Transactions by Type Q&A Q&A T-Qu- Figure 2.3 Barter's "Economy" page which visualizes for the user her credit distribution over time and various activities on the site. While Quora and Barter relate to Llifelog by both using market-based mechanisms for information sharing, a key difference between these two systems and Llifelog lies in the focus of the content and user base. Llifelog is a platform dedicated to research projects, and such shared projects are limited to first, the computing technology industry and digital design/art field instead of any field such as business or biology; second, work that has already been done or is ongoing instead of ideas, thoughts, questions which often times are too premature to measure, and third, users who are either graduate students and faculties in the research academia or industry employees in the research division of their companies such as Microsoft Research instead of Microsoft's development sector. 24 From the angle of research work sharing, we can also find some existing platforms related to Llifelog. For example, Academia.edu is "a platform for academics to share research papers" [1], and ResearchGate.org is "a free social networking site for scientists and researchers to share their research, experience, and publications [28]." While both platforms serve over 1 million users, the websites are designed to encourage sharing of mostly academic publications in the field of engineering, sciences, and social studies. Geekli.st [15] is an achievement-based social portfolio builder where "all bad-ass code monkeys around the globe can communicate, brag, build their street cred and get found." Software developers post summaries of their work on Geekli.st, such as an Android application they have written, a website they have built, the technologies they have used or acquired through these experiences. Similarly, StackOverflow is a well-known Q&A website within the software development community that uses a set of gamified mechanisms to keep the high quality of the site content, such as different levels of virtual badges, certifications, points, and thumbs up or down [29]. It is interesting to note that StackOverflow also uses virtual credits and market-based mechanisms to allow users to trade software code that takes much more effort to produce than just a few lines of answer. Although "thumbs up or down" type of mechanisms works well on StackOverflow, it is found to be too subjective a rating method to indicate the actual quality of the content objectively on most of the other websites. In the use case of StackOverflow, a question is usually specific enough to demand a particular working solution, and the answers that get thumbs up are usually well-written software code that can be tested to work-make a software program runimmediately. The use cases on the other web platforms, such as giving a "thumbs up", "like", or "+" to a photo album, a research document, or a description of technical expertise, give little indication of such action and effect relationship. In I rate you. You rate me. Should we do so Publicly, Chun-Yuen Teng et. al exam the several possible factors that contribute to how and why users give ratings, such as age, gender, geography, anonymity, level of trust and friendship [30]. To study these factors, they conducted extensive analysis with three different datasets: collections of user-to-user ratings from the CouchSurfing [9] social network, user-to-user and user-to-article ratings from Epinions [12], and user-to-product reviews from Amazon. Their 25 analysis found the near absence of negative rating compared with the distribution of positive rating especially when users of these websites were required to use their real identity for all rating and feedback activities. This insight echoes with that of another study, Rating Friends without Making Enemis, by Lada A Adamic et. al., where the authors found that "to users, providing faceted ratings presents challenges, including differentiating and quantifying inherently subjective feelings such as friendship and trust, concern over a friend's reaction to a rating, and knowledge of how ratings can affect others' reputations [2]. Amazon: user-to-product ratings revtew Anonymous or idenwed --------------------------------------- Epinions: user-to-user ratings user-to-article ratings raung of trust In artce raung of trust M user ------------------- CouchSurfing: user-to-user ratings L.doo. ratng of fAnemshlp and trust rating of fMndship and trust Figure 2.4 Types of rating activities for the three datasets used in the study. From the paper I rate you. You rate me. Should we do so publicly? by Chun-Yuen Teng et. al 26 8 8 0- extremely negative negative III neutral/ no reference positive extremely positive Figure 2.5 Dataset from CouchSurfing.com shows the distribution of positive and negative references. From the paper I rate you. You rate me. Should we do so publicly? by Chun-Yuen Teng et. al 27 Design "Design is not just what it looks like and feels like. Design is how it works." --Steve Jobs 3.1 Design Method The process of this thesis project follows a participatory User-Centered Design approach, where a great number of ideas were flushed out quickly in the beginning with the famous "No But" rule, followed by sketches or paper prototyping, which soon evolves to become "highresolution, high-fidelity" digital prototype. Users are invited to evaluate an evolving prototype for participatory design until it reaches a release version. Further user testing is then conducted and usage data and user feedback analyzed for the next round of iteration, which begins again with a set of "why" and "what if' concept design questions. While User-Centered Design methodology is widely used across the world, Participatory Design, or sometimes called Cooperative Design, is one school of User-Centered Design originated and popular in Scandinavian region. Participatory Design is an approach to design that actively involves all stakeholders-especially users and customers-as early as possible in the design process in order to ensure that the product designed meets their needs. 28 AK&Lvss USER-CENTERED DESIGN PROCESS DESMG & Figure 3.1 User-Centered Design Process 3.2 Process Llifelog followed the User-Centered Design approach and quickly iterated two rounds within 6 months of development. It began with an ideation stage where the author brainstormed with her advisor, lab colleagues, friends, potential users, and undergraduate students working with her. The initial brainstorming conversations were high level, sporadic, and scattered around many observations and topics, including first, big data, second, online social media as platform for collaboration, third, unbalanced distribution of information, and fourth, the trend of monetization of personal data. Following the first round brainstorming, a low-resolution, lowfidelity web prototype was built to test some of the ideation assumptions and speculations, as well as opening up opportunities for unexpected insights and feedback from users. The 120 users in this 1-week testing round were mostly people from the author's close network, and provided informal or written feedback through phone calls, emails, and in-person conversations after the testing. Such feedback was crucial and inspiring for the second round iteration. The second round began with a close examination of the feedback from users from the first round, coupled with a brainstorming that is much more focused comparing with that in the 29 first round. Specifically, particular user need and desire was identified in information sharing. For example, it is during this brainstorming that the author narrowed down the scope of information content on the website from all life experiences to just research projects related to the computing industry. In addition, the targeted user group was narrowed down to graduate students and researchers from academia and industry, where a diverse set of academic laboratories and institutes in both the computing and design disciplines were identified as potential user testing bases. A high-resolution, high-fidelity prototype was built including all major functionalities, such as search, user portfolio, project documentation, one-on-one, peer-to-peer exchange system, messaging and scoring system for each project exchange. Additional features such as a "Send Us Feedback" label and a referral system were built to better facilitate the engagement with users as extra user needs were identified throughout the development process. The second round user testing took 3 weeks with about 300 users and 50 projects uploaded to the website. User behavior was closely tracked through monitoring from the website's backend and with the help of Google Analytics [17]. More details on user testing and data analysis will be discussed in Chapter 5. Although the author actively asked for more feedback after the second round user testing, most of the feedback was received during the testing, as a result of the "Send Us Feedback" function which allowed the author to make changes and improvement of the website in almost real time according to user feedback. 30 1"round brainstorming Topics: big data, social media platform as a collaboration tool, unbalanced distribution of information, and trend of monetization of information 1 st round prototyping Low-resolution, low-fidelity, website features not complete 1 st round user testing 120 users within 1 week 2 "dround brainstorming Topics: how did users evaluate their content with virtual currency in the What 1 st round? What content seems most valuable for exchange? features to include in the website to facilitate that exchange? 2 nd round prototyping High-resolution, high-fidelity website features complete 2 nd round user testing Over 300 users within 3 weeks Figure 3.4 Two rounds of iteration for the development of the project. As mentioned previously, the initial brainstorming during the first iteration was around four different topics, big data, online social media as platform for collaboration, unbalanced distribution of information, and the trend of monetization of personal data. Each of these topics has many researchers working on it already with concentrated effort, so the goal of the brainstorming was not to pick one of them and go deep down for incremental research value added in that field. Instead, the goal was to look at the big picture, trying to understand each topic area and figure out how they are inter-connected and affecting one another on the entire chess board of global information economy. In the middle of a big data era, many people are working on data mining, data visualization, trying to come up with the best algorithm for organizing data and getting the juicy bite. At the same time, others are focused on trading of such data and figuring out what the best monetary exchange scheme is. An interesting phenomenon is observed that while in some geographical regions and industries, people complain about being overwhelmed by information explosion, in others, people's knowledge and understanding of "viral growth", "terabyte storage" seems to still stay in the Stone Age. We have a lot of information; we also do not have much information. Information differs in its distribution, as well as its value with regard to time, geography, and different people. The 31 complexity of such a networked problem domain is not what one can break through with one magical computer algorithm. Instead, it will take a heuristic approach for the entire network of people, both data producers and consumers, to collectively come up with a network of organization framework of information. Such brainstorming process led the author to believe that the toughest bottleneck problem in understanding big data is the unbalanced distribution of information and that the crux of solving this problem is by channeling its flow from higher density to lower density, in order to reduce the "information potential"-in similar way that electrons flow from high voltage potential to low voltage potential, or channeling the flow with an "like attract" approach by helping people discover projects related to what they are working on. Market-based information exchange comes into the picture when we analyze both social and economic incentives of individuals within the network as part of the information channeling process. As Dawei Shen has pointed out in Barter, a complete value system should take into consideration both aspects to effectively incentivize users of an information platform. Such incentives are not only crucial for keeping the information flow but also the driving force for information evaluation, which circles back to affect users' motivation for progressing or blocking the information flow. In the case of Llifelog, market-based incentives are more appropriate than those based on social bonds alone because the type of project information that users share does not come as effortlessly as that they share on Twitter or Facebook, but is rather time and resources consuming to the users. Whereas a simple tweet of "I went to lunch with my best friend from high school" comes from an hour's lunch, a full documentation of full-length research project might come from a few months or years of painstaking research, development, testing, and analysis. Additionally, a favor economy works well when the network community knows one another well, especially at a personal level. In Llifelog, a professional project exchange platform, most of the users come to discover research projects for work related purposes, instead of trying to find friends or reconnect with old friends as people do on a Facebook kind of social network platform. The targeted users of Llifelog, graduate students and researchers, often find themselves put in the position of having to connect with strangers because of the need for research exploration. A purely favor-based platform may not work as efficiently in helping the users achieve their research goal. 32 So how shall we effectively merge both social incentive and economic incentive into one platform? Or if the incentives already co-exist in a particular context, how can we utilize both to achieve the effective information channeling? Although the first round user testing brought back more questions than answers, it helped the project to be narrowed down to a particular type of information content, i.e. research projects in the design and technology industry, and a particular user base, i.e. graduate students and researchers in both the academia and industry. In this context, there are existing social incentives for project discovery, evaluation and exchange: students and researchers want their work to be discovered and it is mandatory for them to be aware of similar work in their field. A community evaluation helps them get feedback on their research, keep track of their progress, and establish a desired reputation. In fact, in both rounds of user testing, the users have indicated that discovery and evaluation are two important factors that they value highly in their research process. When asked to rate from 1 to 10 the importance, 1 being the lowest and 10 being the highest, an average of 8 was given from all the user survey participants. Note that the "discovery" in the user feedback entails "exchange", which the author has separated purposely here in this thesis writing for mapping them to indicate different technical features on the website. There are also existing economic incentives for the users in this context: researchers want to find potential collaborators who can help advancing their project faster and better; students want to be recognized and hired by their potential employers. After the first round user testing, users from both Hisense and MIT Media Lab have demanded that the author add a feature on to the site in the future, such as listing their project collaborators' name in a project description, as well as being able to receive recommendations from people who are working on projects similar to theirs from outside of their own organizations. Based on initial brainstorming ideas, first round user testing and feedback, and a constant feedback mechanism built into the website for the second round iteration, the author filtered through a plethora of functionalities and features for the website, and finalized on the following list: 1. Search, which allows users to discover projects by content relevancy, creator information, geography, and company or school name. 2. Scoring system, which allows the users to rate each project with a quantitative score. To help users get started with thinking about how to evaluate a project, 33 the scoring system lists two aspects to which users can give rating, "presentation" and "content". The total score for a project is the sum of the scores for both presentation and content of the project. This resembles the existing conference paper peer-review system, where a paper is usually scrutinized from both a technical perspective, where theory, data analysis etc. is one aspect and English grammar, layout, etc. is another. Given the fact that the Llifelog platform gathers projects from both technology and design fields, its scoring system also resembles that for sports such as figure skating, where the total score of a performance is the sum of scores for technicality (number of jumps, rotations) and presentation (artistry). 3. Private messaging system, which allows users to chat with each other in real time about a project. This messaging system is placed right below the scoring system so that it is easier for users to relate both the quantitative evaluation with the qualitative evaluation of the project. 4. A "bargaining" system was built on top of the scoring system to facilitate a constant back-and-forth "negotiation" interaction between the project creator and the project seeker. In the process of a project request, when a user gives a project a rating, the other user can then respond by either accepting the rating, rejecting the rating, or suggest a different rating. A project's market rating is calculated by dividing the sum of all the ratings in successful project exchange by the number of times of exchange for the project. The market rating will be visible to any future viewer who discovers the project. In this way, the system discourages the users from "selling" projects for whatever is offered just to earn the virtual credit. 5. An automatically generated preview for every project that users upload. A preview is consisted of three parts: the first 250 characters of a project text description, the first three images extracted from the project documentation with additional image processing technique to blur and resize the images, and a WordCloud that visualizes the key phrases within the actual documentation and their frequency of occurrence. A preview to project documentation serves two purposes. First, it allows users to quickly glance through many projects 34 without spending unnecessary time on the details before focusing on the projects that interest them. Second, it eases the concerns of many users who worry about the confidentiality of their projects when they share on the market and who, at the same time, desire their projects to be discovered. A project preview resembles a movie trailer, which is an abridged version of the whole content, sufficiently representative yet not revealing all the details, in order to stimulate user interest in purchasing the whole movie. It also parallels an abstract to a paper in online publication platforms, where a user gets to browse through many paper abstracts before deciding which ones are relevant or interesting to him. 35 E Implementation "Usually, if you have a new idea, you very rarely break through to anything like recognizable development or implementation of that idea the first time around - it takes two or three goes for the research community to return to the topic." --Martin Fleischmann 4.1 The Prototype Given the nature of this project, it is difficult to understand the concepts without a concrete prototype. Therefore, in this section, without further delay, we go through the interaction flow of the website that has been built for this project in the course of 6 months. This website was a necessary tool for the author to conduct user testing and pragmatically understand the users' behavior in a context of exchanging research projects using a market-based web platform. Screenshots are provided for some of the important features of the website, coupled with explanation of the interaction as well technology used in the particular part when needed. Figure 4.1 shows the home page that a user sees as soon as she signs in, i.e. the home.jsp page. This page is divided roughly into two portions, 40% left and 60% right. The left side displays a stream of briefings of the user's own project, and on hovering shows the entire project documentation. The right side shows the market, where other users on the site have shared their projects. Each project on the market comes with an image extracted from the project documentation-if no image from the project document then the user's own profile image is displayed instead, the project title, creator's name, the creator's organization, the project's upload time and the current market rating for the project. 36 LI Blan Advanced Search 0 Research Assistant M[T Media Lab MatchMaker: Scal TV meets Friend Recommendaion: MatchMaker is an automated collaborative filtering system that recommends friends to people on Facebook by analyzing and matching people's online profile with the profiles of TV characters. The goal of MatchMaker is to produce ustmldaslmin dtI BIDI Screen Matt Hirsch 8:03AM 04/11/2012 200R Open Event Interpreter Dustin Smith MIT 8:07PM 03/07/2012 1001 DepthJS roy shilkrot MIT Media Lab EyeRing: A compact, intelligent vision s... Suranga Nanayakkara SUITD A Mobile Tool for Language Suranga Nanayakkara SUTD Festival of Learning Sayamindu Dasgupta dg truons: 911 8:o7PM 03/07/2012 21 8:O7PM 03/07/2012 13R 8:07PM 03/07/2012 201Z 8:07PM 03/07/2012 101Z 8:07PM 03/07-2012 s Projet ColSensei: Just acollection of projects I worked on in the year of 2011. to vjiew -My 8:o7P 0 7 2012 The Queen's New Clothes: Inspired by the Danish fairy tale The Emperors New Clothes and Lady Gaga's Orbit dress we designed and implemented a costume. The Queen's New Clothes, which appears plain to naked eyes but exhibits changing patterns on i"E 29i - 8:07PM 0307,2012 Figure 4.1 The home page that a user lands on as soon as she signs in. On the left side is a stream of project briefing of the user's own project, and on the right side is the market of projects that other users have uploaded. Please note that the upload time for all the projects on the market in the screenshot was set to "8:07PM 03/07/2012" due to an inevitable database revamp during the recent development process. It has caused the wrong display time for all the projects uploaded before the revamp, but does not affect any projects that are uploaded after the database change. The 40% and 60% division of home page was designed not for the look and feel of the home page, but rather, for allowing the user to be able to engage with some of the core functionalities of the website without having to navigate through multiple pages. For any website design, the navigation architecture is a key that can either help the users focus on their goal at the moment or confuse the users with extraneous, unintuitive, or even illogical redirections. During the initial design process before implementation, the author went through most of the major social networking and knowledge-sharing websites, such as Facebook, Twitter, Linkedin, and Quora, and did repeated comparison and analysis of both these websites' page layout and functionalities. The 37 comparisons and analysis led the author to the design decision that there was a need to reduce the page navigation, simplify navigation menus, and draw out the most frequently used functionalities of a website to one place while balancing the information density through careful aesthetic design. This home page of Llifelog gives a summary of the user's own projects and the market's most recently projects, provides search functionality of projects and project creators, and allows easy access to the user's history of project exchange. Admittedly, this design is not perfect. For example, the author has been trying to come up with a better design for further simplifying the user's history of project exchange but has not been able to fix on one that convinces herself yet. Both the 40% left side and the 60% right side of the home page allows the user to use independent infinite scroll bar to retrieve more information. As soon as the scroll bar reaches the bottom of the page, home.jsp sends an HTTPRequest to the backend asking for ten more projects. AJAX technique is used here to make the user feel that there is almost a continuous retrieval of more projects whereas in fact it is ten projects at a time. The number ten is also a result of technical balance between page loading speed and the hardware configuration, among other issues encountered throughout the development process. Additionally, AJAX is also used for allowing reuse of the area where market projects are displayed on the screenshot. The user's credit is displayed on the top right above the market. With each market project's rating in display, this credit label gives the user a quick assessment of how many projects she could potentially request from the other people. When the user clicks on a project on the market, a collapsible <div> tag expands and shows the preview of the project, as demonstrated in Figure 4.2. Preview is used throughout the website for projects that the user does not have permission to view the full version yet, or for the user to be aware of the amount of information she is sharing with the other people on the market with regard to her own projects. 38 Circular sensor motion photography: removal of motion blur in arbitrary directions Market Rating: 50 truons Created By: Yosuke Bando 8:07PM 03/0712012 Motion blur is one of the sources of image degradation. It can severely attenuate high frequency image content to an unrecoverable extent. if motion direction were known beforehand, one could pan the camera to track that motion (follow shot). If not, or there are multiple objects moving in different directions, arbitrary directionsattenuateblurry camera Imagecircular sensor motion different directionsfrequency image content - image degradationimage middle imagemotion sensor blur motion camera imagemotion direction multiple objectsoptical axisprototype camera standard camera imagetilted acrylic plate unrecoverable extent Presentation: 4 30 0 Content: 4 0 Your Rating: S 40 Yosuke Bando's Rating: 50 Li Bian:0 Figure 4.2 The preview of a project on the market. Each preview is composed of three parts: the first 400 characters of a project's text description, the first three images from the project documentation, and a WordCloud consisted of the key phrases from the project's full text description. The preview of a project is composed of three parts: the first 400 characters of the full project text description, the first three images extracted from the full project documentation, and a "WordCloud" that displays the key phrases from the full project description. The images extracted from the full project documentation are algorithmically pixelated and resized to 100 pixel by 100 pixel for each image. Thus, in Figure 4.2 where three images are displayed for the project, the preview indicates that this project's creator has included at least three images within his project documentation for explanation. In Figure 4.3 we shall see an example where there is only one image in the preview, which leaves empty space for two images. The lack of images gives the users an indication that the actual project documentation will be mostly text. The 39 WordCloud is created by using a Natural Language Processing engine called AlchemyAPI [3] provided by Yahoo. AlchemyAPI is a cloud-based text mining platform that uses deep linguistic parsing, statistical natural language processing, and machine learning to analyze web content and extract semantic met-data in order to provide semantic tagging. To create WordCloud on Llifelog, we use AlchemyAPI on the backend to scan through the entire document of each project that users upload, extract key phrases from the documentation, and on the front end, we create nested <div> for each extracted key phrase in order to resize the phrases based on their frequency of occurrence in the full project documentation. The more key phrases extracted and displayed in a preview, the longer the actual project documentation in text volume; the bigger size of a phrase in the WordCloud display, the higher the frequency of occurrence of this phrase in the documentation. EyeRing: A compact, intelligent vision system on a ring Market Rating: 10 truons iyakkara EyeRing: A compact, intelligent vision system on EyeRing is a wearable intuitive interface that allows a perse Creator's Rating: 18 truons more information about it. We came up with the idea of a n Exchanged: 1 time finger with a button on the side, which can be ... 0O2012 3ar index buttonfinger-wom configuration future versionsindex finger informationmicro camera mobile phonemyriad non-visual data capturepicture possible appicationsSenSOrSthumb uservisual formvisually impaired wearable intuitive interface FPresentation: fl Content: 4 0 Your Rating: 1 Li 19 19 Suranga Nanayakkara's Rating: 18 Bian: 0 Figure 4.3 The rating of a project depends on both the user's subjective rating when he created the project documentation and the market's accumulative rating based the user's exchange history with the other users of this project. 40 Preview is not the only channel through which a user gets a glimpse of the full content of a project documentation. The market rating and number of exchanges for a project also indicates to the user the popularity of the project. The market rating is calculated by dividing the sum of all ratings for a project through successful exchanges by the number of successful exchanges. As the example in Figure 4.4 shows, the project EyeRing was evaluated by the project creator at 18 Truons, where Truon is the unit of virtual credit in the system and 18 is the sum of the project's presentation rating, 9, and content rating, also 9, given by the project creator, as we can see on the two scroll bars. It has been successfully exchanged with someone once, and as a result, its current marketing rating has dropped from 18 Truons to 10 Truons. When another user comes to request for this project, she looks at both the preview-which contains only one image-and the successful exchange and rating history of the project, and decides that she'd probably give the project a rating similar to the previous user who has requested the project. The number of images in a previous gives a hint of the project's presentation, and the WordCloud gives a hint of the project's content. In the example above, the user who is requesting for the project EyeRing agrees to give the project creator 9 Truons for the project content but is only willing to give 3 Truons for the project presentation instead of 9 Truons due to the lack of enough visual elements. This will add up to a total sum of 12 Truons that she offers to the project creator. The user can explain her rationale for her rating to the project creator or ask him questions about the project through the private messaging box right below the rating scroll bars. When she presses "Submit Rating", her rating as well as her message will get sent to the project creator, who can then respond to the user accordingly. The Llifelog system facilitates such communication not only through the Llifelog website messaging system but also by sending email notifications and summaries of project request, inquiries, and comments to the users' email inbox. The triangle buttons on both sides of the scroll bar are for changing the range of the rating. Because Llifelog adopts a market approach for its project ratings by using virtual credits, theoretically there is not upper limit for a project rating as there is for, say, five-star rating. At the same time, the lower boundary of the rating is 0, since virtual credits can only be nonnegative. Practically, on the market when the user can only view the preview of a project, her rating cannot go above the project creator's rating, because otherwise that would create an efficient market, i.e. the buyer is willing to give the seller higher price for an item than the seller is selling for. However, since the purpose of the Llifelog platform is for project evaluation instead of trying to 41 fit projects into a market setting, the system gives the user a second chance for rating when she successfully obtained the full content of a project, as shown below in Figure 4.5. The full content of the project may contain detailed description of the research purpose, design, implementation diagram, more information or outside links to the same project, and images of the finished prototype. When the user obtains the full content of a project, she can then evaluate the project again by giving a "full version" rating. The full version rating is also divided into two aspects, presentation and content. The difference between a preview rating on the market and a full version rating on the full content of a project is that there is no upper bound for the full version rating for the evaluator whereas the upper bound for the preview rating for the evaluator is the project creator's initial self-rating of the project. The evaluator also gets a second chance to comment on the project. Upon the evaluator pressing on "Submit" at this stage, Llifelog system sends the project creator a summary email about the project, which contains, from the same evaluator, her preview rating, preview comments, full view rating, and full view comments for comparison. Figure 4.4 The display of full content of a project that the user has successfully obtained from another user. 42 Ientrepreneur. am an interaction designer, an engineer, and an I like bilding things and make things U Sian Ii_chakuh@yahoo.com happen. Research Assistant Mif Media Lab 617-335-3781 Projects NJ Figure 4.5 The project portfolio page of a user. If the user is interested in viewing more projects by the same project creator, she can either search the project creator's name on the home page or simply clicks on the project creator's name on the project documentation, which leads her to the creator's project portfolio page. A project portfolio showcases all the projects by one user, as that in Figure 4.5. A user can click on each image extracted from the projects to view either the preview or the full content of that project, depending on whether she has access permission to it. For example, a user can view the full content of projects on her own project portfolio, or she can view the full content of projects which she has already successfully exchanged for with her virtual credits. Figure 4.6 gives the screenshot of the former case. Figure 4.7 shows a screenshot of the dropdown feedback box that allows the users to communicate with the author more easily and efficiently than through emails or only through a post-prototyping user survey. Every time when a user sends a feedback, the author gets it 43 instantly in her Gmail inbox. As a result, over the course of user testing, the author received many emails from users in the targeted testing organizations across the globe. The emails include reports of bugs which the users have identified on the website, complaints about the page loading speed, suggestions for new features and how to further improve existing features, and "thank you" emails. In the next chapter, we will discuss in details the user testing and user feedback results. Figure 4.6 Simple and easy browsing of projects on the portfolio page. 44 >ep wmtkm (rn? VI Figure 4.7 Built-in feedback system on the website for more efficient communication with users. 4.2 Overview The Llifelog website was hosted on an Ubuntu server with 1GB RAM and 25GB storage space. Ubuntu is a free computer operating system based on the Debian Linux distribution. The languages used to build the website include Java, MySQL, HTML, CSS, and Javascript, where MySQL-My Structured Query Language-is the world's most used relational database management system that runs as a server providing multi-user access to a number of databases. The programming environment was Eclipse, a multi-language integrated software development environment and an extensible plug-in system. Eclipse is mostly written in Java. As Figure 4.8 shows, when a user interacts with a webpage of Llifelog, either by uploading a project, sending a search query, or requesting for a project exchange, home.jsp, the website's main JavaServer Page, communicates with Apache Tomcat, which sends requests to MySQL database through Java servlets. Apache Tomcat is an open source web server and servlet container developed by the Apache Software Foundation [5]. Apache Tomcat implements the Java Servlet and the JavaServer Pages specifications, and provides a pure Java HTTP web server environment for Java code to run. The Llifelog system uses Apache Tomcat version 6.0. MySQL provides a query processing layer that pulls requested data from the database and sends Java servlets back to Apache Tomcat, which sends the data back to home.jsp. 45 In order to render the webpage as desired with the data, home.jsp also pulls in data from four files: homeini.js, a Javascript file that initiates the page, home.css, a Cascading Style Sheet that determines the look and feel of the page, home.js, a Javascript file that sets the basic frontend dynamic interaction with the user on the page, and item.js, a Javascript file that sets the dynamic interaction between the user and projects on the page. For easy development purpose, a simple interface, phpMyAdmin, was used to look up data in the database. PhpMyAdmin is an open source written in PHP intended to handle the administration of MySQL with the use of a web browser [24]. It can perform various tasks such as creating, modifying or deleting databases, tables, fields or rows; executing SQL statements; or managing users and permissions. ltemjs 111feloa user I 4e0evit -M&ySQLON servlet AM .. 4servIet 6.- wow Linux J Se rver! Figure 4.8 An overview of Ilifelog web platform. 46 4.3 System Architecture The Llifelog system adopts client-server architecture, with two distinct but interconnected tires: the back-end, and the front-end. As mentioned in 4.2 Overview, the backend, which is composed mainly of a series of Java servlets powered by the open-source Apache Tomcat server, interacts with the MySQL database through a JDBC (Java Database Connectivity) driver. JDBC is an API for the Java programming language that defines how a client may access a database. It provides methods for querying and updating data in a database. The MySQL database itself contains the majority of the data stored on the website: the users of the website are stored in a table called users, and the projects that these users create are stored in a table called items. Each entry in the item table has a foreign key to the users table; this foreign key represents the owner of the object. This allows easy lookup of all the items belonging to a specific user through a simple SQL SELECT query. The database also contains a table called categories, which represents the tags that are attached to each project. The mapping of tags to projects is done in a separate table called items tocategories, which is a table with an ID field and two foreign keys-one for the item and the other for the category. This structure allows us to both look up projects associated with a specific category and the categories associated with a specific project without switch tables. A slightly more complicated structure is found in the "trades" table, which represents the exchanges of projects among users on Llifelog. The trades table has two foreign keys for the users' performing the project exchange, as well as another foreign key for the project being exchanged, and some extra fields that provide meta-data about the on-going trade. These extra fields include a field for the status of the trade, the notification preferences of each user, and the different ratings and point values set forth by both the creator and the "buyer" of the project. The trades can be augmented with messages in the trade-messages table, which has a tradeid foreign key representing the trade to which the message pertains. These messages are the communication among users on Llifelog when they discuss about a particular project under potential exchange. The front end of the website is implemented through JavaServer Pages (jsp), which is HTML for front end layout with inline Java code for server-side logic. Most of the dynamic 47 interactions are coded in Javascript using the popular open source library called JQuery [19], and then loaded into the pages dynamically as needed. Each front end JSP page has a corresponding Javascript file and a CSS file which define the dynamic interactions and styling that are possible on that page, respectively. One issue that all websites have to deal with sooner or later is the issue of loading timethe time it takes for the webpages on that website to load. On the web, the faster the pages load, the more satisfied users will be with their experience on the website. Llifelog tackles the loading challenge in several ways. First, all the Javascript logic and CSS stylesheets are kept in separate files, so that they can be cached by the browser and do not have to be loaded on every page reload. Additionally, Llifelog employs a set of widespread techniques called AJAX (Asynchronous Javascript and XML) to decrease loading time for the users of the website. AJAX programming takes into account the fact that when a user is browsing the website, a lot of the things on each page do not need to be reloaded each time the user sends a request-they are just static content. When performing different search queries on the project database, for example, there is no need to reload the whole webpage and send request for all the external style sheets and Javascript files all over again. Instead, we can make a specific Javascript XMLHttpRequest to the server to request just the desired data, and then dynamically update the website with a callback function that gets called whenever the data is fetched from the server. Llifelog employs these AJAX techniques to speed up the "feel" of the website and to reduce discontinuity between users' actions and their effects. For example, on the home page, the search results are loaded completely via AJAX. The same holds true for the all of the different project request views and even the profile edit view; the use of AJAX simply speeds up the interaction by avoiding all extraneous header and Javascript/CSS file requests. 48 M Evaluation "An approximate answer to the right problem is worth a good deal more than an exact answer to an approximate problem." --John Tukey 5.1 User Testing In Chapter 3, Design, we gave a brief overview of the six months development and user testing process of the Llifelog system. In this chapter, we discuss the detailed results from user testing and user feedback, as well as analyzing the usage data we have collected. As mentioned before, the Llifelog system was developed and tested in two quick iterations within 6 months. In the first round iteration, the system was not completed yet. As a result, the website was made public for only one-week testing among 120 users, with 2/3 of the users from MIT and 1/3 of the users from Hisense based in Qingdao, China. The website allowed the users to exchange any type of life experiences with one another using virtual credits. The goal of this round of user testing was to find out whether people would be able to measure and evaluate digital information quantitatively through a market-based exchange platform. Users expressed a lot of confusion, as expected, but also gave many good suggestions that helped the author narrow down the scope of the site content and user base in preparation for the second round user testing. The second round user testing involved 305 registered and verified users as recorded in our system database. Accordingly to Google Analytics, however, there were 690 unique visitors 49 who visited the Llifelog website 1318 times with 4362 pageviews during the period of February 3 to March 3, as shown in Figure 5.1. One possible reason for this discrepancy in the number of "users" is that Google Analytics relies on each unique IP address for counting "unique visitors", whereas in the Llifelog database, we count each unique user through verified school or corporate email address as well as other personal information. A closer investigation in Google Analytics also reveals that the users have been accessing the website using various mobile devices beside their laptop or desktop computers. Figure 5.1 gives a clear visualization of the daily traffic in the 30-day period. One thing to note is that the official user testing for the second iteration lasted from February 3 to February 25, when the author actively advocated this website within her social network. In Figure 5.1, we see a surge of traffic on the website toward the beginning of March when the author has stopped advocating the website for a week; this is possibly due to the network effect that let the word spread out. Feb 3, 2012 -Mar 3, 2012 - Visitors Overview Advanced Segments f EmaUSETA Export- Add to Dashboard 100.00% oftotal visits overview Visft - vs. Select amtric Hourly DWy Week Month 0 Visits 200 100 Feb8 Feb15 Feb22 Feb29 690 people visitd Usi sie d\~~J Visits:1,318 ,4/\.-J Unique Visitors: 690 /4M. Pageview: 4,362 PageeMVit: 3.31 V.%A ' Avg. Visit Duration: 00:05:47 m 50.23% New vso =Vzawt U 49.7. Returning visitor Bounce Rate: 48.25% Figure 5.1 Website traffic monitored through Google Analytics. 50 Figure 5.2 lists the geographic distribution of users during the second round user testing. As we can see, over 60% of the users were from the United States, about 35% of the users were from China, and the rest from various parts of the globe. The wide spread of the geographic user distribution added to the effect of information gap and weak social bond, which affirms the need for a market-based platform for project exchange, instead of a purely favor-based one. countrylTerrory Visits 4, Pagesvsit Avg. Visit Duration % New viait$ ounceRate 1. United States 894 3.71 00:06:21 4508% 49.55% 2. China 383 2.56 0005:01 59.79% 4334% 3. Canada 9 1.33 00:00:30 66.67% 77.78% 4. 8 1.38 00:01:08 50.00% 62 50% 5 2.00 00:0053 60.00% 60.00% United Kingdom 5. Brazil 6. Hong Kong 3 1 33 00.00:07 10000% 66.67% 7 India 2 2.00 00:01:43 100.00% 50.00% 8. Japan 2 200 0000:19 100.00% 50.00% 9. Norway 2 3.50 00:01:32 50.00% 5000% 10. Taiwan 2 1.50 00:01:54 50.00% 50.00% Showrows: 10 Goto: 1 1- 10of 18 Figure 5.2 Geographic usage monitored through Google Analytics. Among all these users, about 2/3 were university graduate students and research staffs and 1/3 were from the industry, such as Hisense, Microsoft, Nokia, and Google. The 2/3 academia users contributed 95% of the research projects uploaded on Llifelog website and only a few projects were shared by the industry users. In the user testing process, many users expressed concerns over the confidentiality of their projects; and this is especially the case for the industry users, who sent the author feedback throughout the testing that they loved using the site to discover what the academia had been working on but were hesitant to share their projects which were likely to become their company's revenue stream. About 50% of the academia users who shared their research projects also sent the author inquiries regarding the confidentiality or IP issue. Besides responding to the users about their concerns, the author quickly added new features on the website during the user testing period to make the users feel more comfortable to share their projects there. For example, after a user finishes documentation of a project, he can 51 click on a "preview" button to view what other users on the market can see about his project, and can modify his documentation accordingly if he wants. When a user receives a project exchange request, he can also hover over the project creator's name to view the project creator's professional background such as school or company, and geographic location. In about 10 cases when the author was observing the users using the website in person, the author noticed that these users almost always hovered over or clicked on the other users' name to find out more information about them before making a project exchange decision. Most of these users also clicked on the "preview" button to see what they had shared on the market, and in a few cases, the users went back and added more content to their documentation to make a richer presentation of their project preview. Such targeted sharing mechanisms give the users more control over who they want to share project information with and what they want to share. Although the project previews appear to be an encryption tool of projects, creating project previews and allowing users to control what others see about their projects through these previews encouraged more project-sharing on the website. 5.2 Data Analysis There were 305 unique users visiting the Llifelog website during the second round user testing period. Among them, 32 users uploaded 74 research projects, with 42 projects being exchanged at least one time. The total number of exchanges is 139. Thus, the content producer ratio on Llifelog was: (32 / 305) * 100% = 10.5% The average project upload rate per content producer was: 74 / 32 = 2.3 project / user The average project exchange rate among all the projects on the website was: (139 / 74) = 1.9 exchanges / project In Chapter 4, Implementation, we discussed in details the three components of each preview, which was automatically generated for every project that a user creates on Llifelog 52 website. One of the most indicative components is the image extracted from each project. If a project has at least three images in its full content, three images will be displayed in the preview; if a project has less than three images in its full content, 0, 1, or 2 images will be displayed in the preview. Because images are often an essential visual element that indicates the presentation quality of a project as well as the stage of progress of the project, we compare the number of exchanges of each project with the number of images contained in each project's preview in Figure 5.3. Due to the fact that there can be multiple factors affecting a user's project exchange decision, such as the number of images in preview, the project's rating, and the private messages exchanged, in Figure 5.3 we only use the data from projects that have been exchanged at least once, i.e. the 56.8% of all the projects uploaded onto the website. The smaller dots represent 1 or 2 projects and the bigger dots represent 3 or 4 projects at the same coordinate, since most of the exchanged projects fall into the number of exchange range from 1 to 4, with two exceptions exchanged 8 and 9 times respectively. As we can see in Figure 5.3, the more images there are in a preview, the higher the average exchange rate of a project-as indicated by the blue curveand the higher total number of exchanges of projects as indicated by the red line. There is a decrease in the average exchange rate from no image to one image due to extreme low initial rating of projects with no image. We will show this through Figure 5.4. # of exchanges 9 9 8 0 7 6 5 4 - 1 0 0 1 2 3 # of images in preview Figure 5.3 Number of exchanges versus number of images in a preview for a project. 53 Figure 5.4 is a visualization of the rating for all successful project exchanges. The horizontal arrows indicate that the project requesters and project creators agreed at the same rating initially set by the project creators. The vertical arrows indicate that the rating of a project dropped from the project creators' initial rating (the arrows' starting point) to mutually agreed ratings (the arrows' ending point). All the arrows are facing downward because the system was designed in such a way that no project requester's rating will exceed the project creator's initial rating, lest there be an inefficient market. As we can see in Figure 5.4, when the rating for a project is lower than 20, the number of images inside a preview has almost no effect on the exchange rate. In fact, many projects without any images in a preview but with ratings under 20 were successfully exchanged. So we can say that the users' exchange behavior was inelastic to the quality of a project, when a project rating is sufficiently low and does not cost the users much to obtain. We also see an exponential decrease of rating drop as the number of images increases in the preview. This indicates that the higher the initial rating and the fewer images in the preview for a project, the more dramatic drop of rating upon a successful exchange. This implies that the users do care about the quality of a project and behave reasonably rationally in terms of using the market rating to measure the value and worth of a project based on their estimation of the project quality. As the number of images increases in a project preview, we see more "stable" project exchange in the region from rating 40 to 100. A stable project exchange is when the project was successfully exchanged at its initial rating set by its project creator. After manually going through almost every project's full content on the Llifelog website, the author observed that most of the projects successfully exchanged at a rating between 40 and 100 include 2 to 5 images and about 2000 to 2500 words. The author also noticed that most of the projects that had below 20 ratings fell into two categories, first, they had very little content-so that the project creators probably did not feel confident enough to give high evaluation themselves, second, they were created very early on during the user testing-so the initial users were probably still trying to figure out how to use the system and waiting to get a few quick back-and-forth interaction with the other users on the market. But as time goes on, by mid-week in the second week of the user testing with about 150 users on the website and 1 or 2 project upload per day, the users started getting familiar with this market environment and getting to learn how to rate each project better with virtual credits. Thus we started seeing a convergence of rating versus project quantity and quality then. 54 Scanning through all the private messages among the users engaged in project exchanges, the author found that for projects with rating lower than 20, there was very little conversation, for projects with high rating but relatively fewer images and phrases in the WordCloud, the messages seemed to be focused more on bargaining for a mutually-agreeable rating, and for projects whose rating fell in the range from 40 to 100, the private messages were mostly focused on discussions and inquiries of the content of the project. For example, "Do you have X-Ray specs?" "Interesting application, may I look at it?" "Hi? How to know the details? Thanks!" A comparison of rating and messages exchanged at preview versus rating and comments left at full content for each project reveals that in only a few cases did the user change rating after receiving the full project content, when the project's preview rating is within 40 to 100 Truons. Most of the users did not change the rating, or seem satisfied with the fact that what they got was expected. Outside the range of 40 to 100 Truons, however, whether it is rating below 20 or above 150 Truons, the user comments suggest that the preview did not do a good job conveying the quality and quantity of the project documentation. For example, for projects with initial rating below 20 Truons, some users requested them immediately only to be disappointed at how little description there is. This indicates that there is still much space for tweaking the preview to ensure accurate depiction of the actual content. Project rating 200 180 160 140 120 10080 40 20 0 0 - . - - -. .-.-. 1 2 3i-------------------- 2 3 # of images in preview Figure 5.4 Project rating versus the number of images in a preview for a project 55 Although it is common for people to try to game markets like this, and the existing prototype does have many loopholes and shortcomings, it is fortunate to note that out of 305 users, only two people seemed to have tried to game the system by exchanging projects among themselves in an attempt to horde virtual credits. Unfortunately for them, the only way to horde virtual credits in the system right now is by creating multiple accounts per person and the email verification process in the system has prevented most people from signing up with multiple school or corporate email addresses. Another phenomenon that could theoretically happen is users reselling other people's projects that have already been sold on the website. However, this has not happened on the website so far, possibly due to the academic integrity and social reputation concerns of the targeted testing user base. 5.3 User Feedback After finishing the two-round iterations, the author conducted a survey (Appendix A) among Llifelog's existing users to further understand the users' background and determine the usefulness of the website to them. 20 users responded to the survey. The questions cover six categories of user bahavior- frequency of web use, response to market rating system, motivation for research, vehicle for project evaluation, time used for similar project search and Llifelog user experience. Through analysis of the user feedback, the author concluded that the website found its target audience and met the needs of the target audience, despite barebones set up. All of the people surveyed reported to be web users who had used Facebook, Google, Linkedin, Youtube, Amazon or their Chinese equivalent. 70% used Facebook, Youtube and Google at least once a day and most of them spent more time browsing than posting. Thus, the people who were using Llifelog for project exchange were already well-integrated into the digital world. 80% did not find that any of those platforms would be good places to post their research projects. 70% found the feedback systems in sites such as Facebook and Youtube to be lacking in objectivity. They found these sites ineffective in giving relevant feedback besides personal support. One user wrote: "I nearly never post researchprojects on Renren [Chinese Facebook] because I have only friends there, not professional connections. We only share fun and 56 emotional things; the occasional times when I've posted work photos I only expected others to say 'cool!' but did not expect any discussion." Llifelog market system allows users to evaluate objectively how much a project is worth and not merely express support for a friend's experience. All users wanted evaluation and feedback on their projects but found that the traditional conference publication peer review method insufficient. Only 10% selected peer review as their method for evaluation and feedback. Because creating, evaluating and improving projects did not allow them to receive timely feedback from the paper peer review process-as papers are published after the project is polished, 90% received feedback and evaluation on their projects through user research, study, and testing and talking to advisers, colleagues, and friends. Projects aimed for demonstration or commercialization purposes do not fit the traditional mold of conference publication peer review because projects need timely review and usually quick and multiple rounds of development iterations. On the goal of research, 60% of the surveyed users reported that it was because they wanted to create project prototypes, and among the rest the author found a variety of motivations including calling for attention for their research field, being able to share their research findings and becoming pioneers in their research field. None wanted to publish papers as his or her main objective, but 20% wanted that as their second or third. Creating demonstrative projects that demand effective feedback and sharing systems require a new platform that is supplied by Llifelog. 100% of the survey respondents reported that they had not used any system similar to Llifelog for exchanging research projects before. 1/3 found the website to be "a little bit easier", 1/3 "no difference", and 1/3 "a little bit more difficult", when asked to compare Llifelog's existing search system with Google search for research projects. This implies the lack of critical mass of content on Llifelog, as well as Google search's low relevancy on finding research projects aside from the ones that already have paper publications. Among the survey respondents who have used Llifelog's private messaging system during project exchange, 50% found it to be "very efficient", 40% found it "somewhat efficient", and 10% found it no difference from existing peer review system for paper publication. A key improvement for the system to be more satisfying for all the users, as the survey suggests, is gathering sufficient 57 number of users and projects. One user recommended, "Considering linking to external sources for projects." Adding a recommendation system was also on top demand from users who wanted even more targeted research results on top of the existing vertical search system. 58 E Conclusion and Future Work "Logic will get you from A to B. Imagination will take you everywhere." - Albert Einstein In this thesis, the author presented Llifelog, a market-based web platform for research project discovery, evaluation, and exchange. The goal of the Llifelog system was to cultivate a community of critique among students and researchers in both the academia and the industry on their work. Those who engage in the market answer the call "find out how your work, projects, and ideas are valued in the community." The motivation and inspiration that lie behind this project came from the author's crossroad engagement with the research academia and industry, as well as curiosity amongst the tension of information value proposition. A web prototype was built, deployed, and iterated twice among 305 students and researchers from all over the world in the course of six months with thousands of site visits. Natural Language Processing, image processing, and web development techniques such as AJAX were used in composing the core functionalities of the website. Through the Llifelog website, the author observed users' behavior and interaction with one another in real time on this information market focused on research projects. The observed user behavior pattern shows that it took some time for the users to get familiar with the mechanisms in the system before they were able to assign quantitative value to the digital documentation of research projects. During the 2 "dround user testing period, the first week and half seemed to be the most confusing time when users gave random ratings to projects and it was difficult to draw any correlation between these ratings and the quality of the projects. Soon 59 afterwards, however, we started seeing a converging trend in this quantitative evaluation process. Specifically, most projects with at least 3 images and a text description at 2200 to 2500-word range were rated with a score between 40 Truons and 100 Truons. A private messaging system allowed users to communicate with one another about the project quality, content, and rating. It was observed that most of the conversations happened for projects with rating between 40 Truons and 100 Truons. Projects with initial rating below 20 Truons by the project creator were usually requested without any bargaining between the creator and the viewer. In addition, the higher a project's initial rating, the sharper the project's market rating drops after exchange. One of the challenges in creating Llifelog was devising a project preview that can accurately convey to the users the content and presentation of a project yet does not reveal too much information, lest the users lose interest to exchange with virtual currency. Based on user testing data, we can see that the current preview in Llifelog seem to provide users sufficient information for the full content, when a project's initial rating is between 40 Truons and 100 Truons. However, when the rating is outside this range, such as below 20 Truons or above 150 Truons, the comments that users left after having received a project's full content suggest that full content did not meet their expectation. This indicates that there is further space to tweak the project preview so that it helps the user reach an adequate level of expectation in all project rating cases. In a few cases, users gave higher rating to a project after having received the full content. Due to the limitation of user testing data, it is difficult to draw any conclusion from this. However, these rare cases might deserve further study with more user testing, since in real world market, people generally do not give more money to a product than what was asked from them. The current Llifelog system only allows exchange with virtual currency; it would be interesting to see how users' rating behavior changes, if it does, when the same system allows users to convert virtual currency with real currency. Additionally, while most of the user behaviors seem to form a correlated pattern for exchanging projects rated within 40 Truons to 100 Truons range, it will be necessary to test if this was affected by the initial 500 Truons given to every user upon signing up and if this range still exists when the system gives every user different amount of free Truons to spend. Nonetheless, within the 40 to 100 Truons range, we see a positive correlation between the amount of information revealed in previews of projects and the projects' market rating. 60 From the user feedback conducted in the end, we see that the majority of students and researchers working on projects constantly value the importance of feedback on their work, and that existing content-sharing social networking platforms such as Facebook and Linkedin do not provide an environment suitable for research project sharing. Although the current Llifelog platform does not fully serve users' need for such sharing, it is evident that a platform with more standardized evaluation mechanism and more efficient communication and exchange features is in demand. Although the current thesis work shed light on a future promise of wider information market, much remains to be done in order to further validate the patterns observed from the existing system. A critical mass is yet to be reached before we can generalize the insights across the information spectrum and among different user groups. To aggregate content onto the website more efficiently, linking to external sources and allowing networked sharing could help with the viral growth of the market. For continued iterations, machine learning techniques could be adopted and applied in project search, recommendation, ranking and preview generation, in order to keep up with the veracity of market dynamics. 61 Bibliography [1] Academia, http://academia.edu/. [2] Adamic, L.A., Lauterbach, D., Teng, C. and Ackerman, M. Rating Friends Without Making Enemies, AAAI Publications, Fifth InternationalAAAJ Conference on Weblogs and Social Media, July 2011. [3] AlchemyAPI, http://www.alchemyapi.com/. [4] Amazon, http://www.amazon.com/. [5] Apache Tomcat, http://tomcat.apache.org/. [6] Ask.com, http://www.ask.com. [7] Barter, http://barter.mit.edu/. [8] Brand, S. The Media Lab: Inventing the Future at MIT, Penguin Books, 1987. [9] CouchSurfing, http://www.couchsurfing.org/. [10] DepthJS, http://depthjs.media.mit.edu/. [11] Diaspora, https://joindiaspora.com/. [12] Epinions, http://www.epinions.com/. [13] Facebook, http://www.facebook.com/. [14] FourSquare, https://foursquare.com/. [15] Geeklist, http://geekli.st/beta. [16] github, https://github.com/. [17] Google Analytics, http://en.wikipedia.org/wiki/GoogleAnalytics . [18] Google Answers, http://en.wikipedia.org/wiki/GoogleAnswers . [19] JQuery, http://jquery.com/ . [20] Kickstarter, http://www.kickstarter.com/. [21] Levels of Sponsorship, http://media.mit.edu/sponsorship/funding-levels . [22] Linkedin, http://www.linkedin.com/. 62 [23] Media Lab China Initiative, http://china.media.mit.edu/. [24] PhpMyAdmin, http://www.phpmyadmin.net. [25] Quora, http://www.quora.com/. [26] Reality Mining, http://reality.media.mit.edu/publications.php . [27] Reas, C. and Fry, B., Getting Started with Processing, MIT Press, August 2007. [28] ResearchGate, http://www.researchgate.net/. [29] StackOverflow, http://stackoverflow.com/. [30] Teng, C., Lauterbach, D. and Adamic, L. A. I rate you. You rate me. Should we do so Publicly, WOSN'10 Proceedings of the 3rd conference on Online social networks, Boston, MA, 2010. [31] Wikipedia, http://www.wikipedia.org/. [32] World Economic Forum, Personal Data: The Emergence of a New Asset Class, Jan. 2011. [33] Yahoo Answers, http://answers.yahoo.com/. 63 Appendix Final Round Survey Questionnaire for Llifelog A Market-Based Web Platform for Discovering, Evaluating, and Exchanging Research Projects The purpose of this survey is to better understand the context of user behavior on information sharing platforms, and the rationale of users sharing and rating information. The goal is to draw insights from such collective feedback in order to improve an existing market-based web platform that we have built for discovering, evaluating, and exchanging research projects within the academia and industry. Allfeedback response provided through this survey will be kept confidential and only used for analytic purpose for this academic study. Any personal data provided will be kept anonymous. Tell me a little bit about you. 1. You are a o graduate student/ academic researcher 2. Oother (specify) How old are you? o Oto 20 3. 0 industry researcher 0 20 to 25 025 to 30 030 to 40 040 to 0 5 0 0above You are 0 male O female 4. You mainly do your research at o office/School Ohome Usage of related web platforms 5. Which one of the following do you use? (Check all that apply) E Facebook E Google+ E Twitter 0 YouTube E Linkedin E Amazon E Renren (AA) E Sina Weibo (,9krd t) 64 E YouKu ('Uii) EDTaobao (N0) El none of above 6. Please check the frequency at which you use/visit the above websites. a few times a a few times a few times a few times (almost) day a week a month a year never Facebook Google+ Twitter YouTube Linkedin Amazon Renren Sina Weibo YouKu Taobao 7. How would you describe your activities on these websites? (Check all that apply) 50% time posting; 20% time posting; 80% time 80% time posting; browsing others' posting 20% time browsing others' posting 50% time browsing others' posting Facebook Google+ Twitter YouTube Linkedin Amazon Renren Sina Weibo Youku Taobao 8. How often do you rate other people's post, such as "like" or "+" someone's status, give 5 stars to an item for sale, write a customer review? o frequently 0 occasionally 0 almost never 9. How often do you get rating from other people on your post? 0 frequently 0 occasionally 0 almost never 65 10. How indicative do you find the ratings regarding the posts' content quality on social networking sites such as Facebook, Google+, or Twitter? o very objective o somewhat subjective 0 somewhat objective 0 very subjective 11. How likely will you post your research projects on these platforms for documentation and sharing purposes? Why? O Unlikely O maybe O Very likely Reason: Tell me a little bit about your research 12. What motivates you the most in your research? (please rank the following, start from 1 being the most important) 0 building a working prototype for demonstration 0 seeing the commercial impact of your research E getting paper publications E call for attention for a research field or being able to share your research findings E become the pioneer in a research field 13. On a scale of 1 to 10, how much do you care about evaluation and feedback to your research? (1 being the least and 10 being the most) 1 2 3 4 5 6 7 8 9 10 14. How do you usually receive evaluation and feedback on your research? (Check all that apply) E peer reviews in the process of submitting papers to conferences E user research, study, and testing 0 talking to advisers, colleagues, and friends 0 I usually don't get / ask for evaluation or feedback on my research 15. How much time do you usually spend finding out relevant, related, or similar work to that one you are working on? 0 a few months 0 a few weeks 0 a few days 0 a few hours 0 no time 66 16. How much do you care whether someone has worked on / is working on something similar to what you are working on? (1 being the least and 10 being the most) 1 2 3 4 5 6 7 8 9 10 Usage of Llifelog system (URL: Llifelog.media.mit.edu) 17. Have you used something similar to the system before for sharing research projects? 0 yes (please specify f) o no 18. How easy was it for you to find projects of interest using this website's search feature comparing with, say Google, or your university/company's project listing page? o much easier o a little bit easier o no difference o a little bit more difficult o much more difficult 19. Have you used the website's one-on-one private messaging system? If so, how efficient do you find this communication method in terms of finding the creator of a project, sending inquiries, exchange project feedback? o very efficient o somewhat efficient o no more efficient than existing paper review systems o somewhat less efficient 0 very inefficient o I haven't used the website's private messaging system 20. Have you used the website to find projects from universities/companies other than your own? If so, how useful or important do you find being aware of other institutes' research through this website? o very useful / important o somewhat useful / important o somewhat not useful / important o not useful / important 0 I haven't found projects from other universities / companies on the website 67 21. The web platform is still at its early stage of development and more user data and projects can definitely provide more value and better user experience for you. Imagine if we had gathered millions of user data and projects already, what improvement would you like to see on the website? What new features / functionalities would you suggest? 68