CPE/CSC 580: Knowledge Management Dr. Franz J. Kurfess Computer Science Department Cal Poly © 2001 Franz J. Kurfess Knowledge Exchange 1 Course Overview Introduction Knowledge Retrieval Information Retrieval Knowledge Navigation Knowledge Presentation Exchange Knowledge Capture, Transfer, and Distribution Usage Organization Classification, Categorization Ontologies, Taxonomies, Thesauri Knowledge Processing Knowledge Acquisition, Representation and Manipulation Knowledge Knowledge of Knowledge Access Patterns, User Feedback Knowledge Management Techniques Topic Maps, Agents Knowledge Management Tools Knowledge Management in Organizations Knowledge Visualization © 2001 Franz J. Kurfess Knowledge Exchange 2 Overview Knowledge Exchange Motivation Transfer Objectives Knowledge Explicit Capture Extraction From Text Case-based Reasoning Enhancement of Existing Documents Communication Capture of Knowledge Basic Concepts Language and Communication Natural Language Formal Languages Communication Models Distribution of Knowledge Knowledge Repositories Distribution Models Important Concepts and Terms Chapter Summary © 2001 Franz J. Kurfess Knowledge Exchange 3 Logistics Term Project Documentation Schedule Lab and Homework Assignments Schedule Exams Grading © 2001 Franz J. Kurfess Knowledge Exchange 4 The Need for Knowledge Exchange WWW Introductory texts, expert hints, explanations, dialogues, comments, examples, exceptions,... Richer representations More ambiguous More versatile Info. extraction templates, dialogue segments and pegs, filled-out forms, high-level connections,... Descriptions augmented with prototypical examples & exceptions, problem-solving steps and substeps, ... ((( )) ()))) © 2001 Franz J. Kurfess (defconcept bridge ())) Alternative formalizations (KIF, MELD, CML,…), alternative views of the same notion (e.g., what is a threat) [Gil 2000] More formal More concrete More introspectible Knowledge Exchange 5 Knowledge Mobility multiple views and versions of the same information need to provide tools that establish connections among alternative versions/views of the same information hyper-connectivity need to provide tools that suggest further connections to related sources when users compose documents need to annotate hyperlinks basis to support information morphing how one or more knowledge sources are used for alternative purposes track alternative knowledge transformations various renderings and implementations of a knowledge source © 2001 Franz J. Kurfess [Gil 2000] Knowledge Exchange 6 Pre-Test © 2001 Franz J. Kurfess Knowledge Exchange 7 Motivation © 2001 Franz J. Kurfess Knowledge Exchange 8 Objectives © 2001 Franz J. Kurfess Knowledge Exchange 9 Knowledge Capture Explicit Capture Extraction From Text Case-based Reasoning Enhancement of Existing Documents © 2001 Franz J. Kurfess Knowledge Exchange 10 Explicit Capture conventional techniques for knowledge acquisition interviews with experts knowledge engineers advantages carefully constructed suitable knowledge representation methods usually common-sense evaluation sometimes formal evaluation consistency checks, other formal aspects © 2001 Franz J. Kurfess Knowledge Exchange 11 Extraction From Text syntactic level keywords essentially construction of an index descriptive semantic level document structure requires information about structure tags, DDT, RDF sentence features structure natural language processing (NLP) pragmatic level context thesaurus, ontology, NLP © 2001 Franz J. Kurfess Knowledge Exchange 12 Case-based Reasoning solutions to a problem in a specific context are collected represented in a structured format problem, context, solution usable by a computer-based system cases are often represented through frames or similar mechanisms new cases are matched against existing ones patterns in the frames provide the basis for matching the suitability of the solution is judged by the user © 2001 Franz J. Kurfess Knowledge Exchange 13 Enhancement of Existing Documents in addition to the methods mentioned above, collections of documents can be enhanced addition of meta-knowledge integration into an existing framework/ontology manually through categorization automatically through keyword extraction indirectly through statistical correlations with other documents © 2001 Franz J. Kurfess Knowledge Exchange 14 Transfer of Knowledge Communication Basic Concepts Language and Communication Natural Language Formal Languages Communication © 2001 Franz J. Kurfess Models Knowledge Exchange 15 Basic Concepts communication exchange of information requires a shared system of signs greatly enhanced by language speaker produces signs as utterances general: not only spoken language listener (hearer) perceives and interprets signs © 2001 Franz J. Kurfess Knowledge Exchange 16 Purpose of Communication sharing of information among agents or systems query other agents for information responses to queries requests or commands actions to be performed for another agent offer proposition for collaboration acknowledgement confirmation of requests, offers sharing of experiences, feelings © 2001 Franz J. Kurfess Knowledge Exchange 17 Communication Problems intention what is the expected outcome (speaker’s perspective) timing when is a communication act appropriate selection which act is the right one language what sign system should be used interpretation will the intended meaning be conveyed to the listener ambiguity can the intention be expressed without the possibility of misunderstandings © 2001 Franz J. Kurfess Knowledge Exchange 18 Language and Communication Natural Language used by humans evolves over time moderately to highly ambiguous Formal Languages invented rigidly defined little ambiguity © 2001 Franz J. Kurfess Knowledge Exchange 19 Natural Language formal description is very difficult sometimes mostly non-systematic, inconsistent, ambiguous used for human communication easy on humans tough on computers context is critical situation, © 2001 Franz J. Kurfess beliefs, goals Knowledge Exchange 20 Formal Languages symbols terminal symbols finite set of basic words not: alphabet, characters non-terminal symbols intermediate structures composed of terminal or non-terminal symbols strings sequences of symbols phrases sub-strings © 2001 Franz J. Kurfess grouping important parts of a string Knowledge Exchange 21 Formal Languages Cont. sentences allowable strings in a language composed from phrases grammar rules describing correct sentences often captured as rewrite rules in BNF notation lexicon list of allowable vocabulary words © 2001 Franz J. Kurfess Knowledge Exchange 22 Communication Models encoded message model a definite proposition of the speaker is encoded into signs which are transmitted to the listener the listener tries to decode the signs to retrieve the original proposition errors are consequences of transmission problems situated language model the intended meaning of a message depends on the signals as well as the situation in which they are exchanged mis-interpretation may lead to additional problems © 2001 Franz J. Kurfess Knowledge Exchange 23 Communication Types telepathic communication speaker and listener have a shared internal representation communication through Tell/Ask directives language-based communication speaker performs actions that produce signs which other agents can perceive and interpret communication language is different from the internal representation more complex involves several mappings language needs to be generated, encoded, transmitted, decoded, and interpreted © 2001 Franz J. Kurfess Knowledge Exchange 24 Telepathic Communication © 2001 Franz J. Kurfess [Russell & Norvig 1995] Knowledge Exchange 25 Language-Based Communication © 2001 Franz J. Kurfess [Russell & Norvig 1995] Knowledge Exchange 26 Communication Steps: Speaker intention decision about producing a speech act generation conversion of the information to be transferred into the chosen language synthesis actions that produce the generated signs © 2001 Franz J. Kurfess Knowledge Exchange 27 Communication Steps: Listener perception reception of the signs produced by the speaker speech recognition, lip reading, character recognition analysis syntactic interpretation (parsing) semantic interpretation disambiguation selection of the most probable intended meaning incorporation the selected interpretation is added to the existing world model as additional piece of evidence © 2001 Franz J. Kurfess Knowledge Exchange 28 Communication Example © 2001 Franz J. Kurfess [Russell & Norvig 1995] Knowledge Exchange 29 Distribution of Knowledge Knowledge Digital Repositories Libraries Distribution © 2001 Franz J. Kurfess Models Knowledge Exchange 30 Knowledge Repositories persistent internal storage of digital documents representation in the original format loss-less transformation may be acceptable transparent internal organization multiple presentation methods for various users and usage methods multiple access methods according © 2001 Franz J. Kurfess to users’ needs and capabilities Knowledge Exchange 31 Digital Libraries collections of documents and artifacts stored and accessed via computers remotely accessible through networks enhanced functionality compared with paper-based libraries access methods organization principles duplication implementation © 2001 Franz J. Kurfess and usage unclear Knowledge Exchange 32 Vision... “… a network of learning environments and resources for Science, Mathematics, Engineering and Technology education, will ultimately meet the needs of students and teachers at all levels—K-12, undergraduate, graduate, and lifelong learning—in both individual and collaborative settings.” NSF © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 33 NEEDS National Berkeley Stanford Northern Arizona UMBC Virginia Tech digital library emerging from the Synthesis Coalition (1990 - 1999) Integrated database of multimedia courseware Multilevel courseware evaluation system Community © 2001 Franz J. Kurfess of Engineering Educators [NEEDS 2000] Knowledge Exchange 34 www.needs.org - Contents © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 35 www.needs.org - Communities Community - a group of individuals who engage in communication through a common bond. Faculty prefer to learn where they are a member of a community that is built upon shared values and interests regarding teaching, learning, and pedagogy GOAL: Provide faculty a means to learn from one another unconstrained by barriers of time, distance, technology, and geography. © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 36 Towards A National SMETE Digital Library... Should NSF Establish and Fund a National Science, Mathematics, Engineering and Technology Education Digital Library? April 1996 - NSF Committee Meeting (LIBUSE) • August 1997 National Research Council • Digital National Library for SME&T Education Workshop July 1998 National Science Foundation “Towards a National Library for Undergraduate Science Education Resources in Science, Mathematics, Engineering and Technology” SMETE-Lib Workshop January 1999 National Science Foundation Digital Libraries and Education Workshop © 2001 Franz J. Kurfess [NEEDS 2000] Funding through Digital Libraries Initiative – Phase 2 And NSDL Programs Knowledge Exchange 37 www.needs.org Delivery Classrooms Instructional Labs Small Study Groups Residences Libraries Anywhere Database Development Courseware Studios Instructional Labs Faculty Offices & Residences Libraries Anywhere NEEDS is the foundation for the National SMETE Digital Library at www.smete.org. © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 38 NEEDS Aspects Digital Library of Multimedia Engineering Courseware Bibliographic records with downloadable courseware Multimedia elements - downloadable movies, images, and text Multilevel Courseware Evaluation System Peer Review of Courseware Premier Award for Excellence in Engineering Education Courseware Expanding Services and Features © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 39 How does NEEDS help users “re-use” learning materials? Provides mechanisms to help user locate materials Uses standardized descriptions (metadata) to describe resources Provides mechanisms to help users evaluate the “quality of materials” Developed upon an extendable platform to: Support multiple uses Integrate new services and features Integrate research © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 40 Quality Review of Courseware on the NEEDS Database Establish credibility of NEEDS as a source of Quality educational material Enhance recognition of scholarly and creative effort of courseware developers Peer/Expert Review of Courseware Premier Award for Excellence in Engineering Education Courseware © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 41 The Premier Award for Excellence in Engineering Education Courseware A national competition to identify and reward the authors of high-quality, non-commercial courseware designed to enhance engineering education. The Premier Award is about the entire experience of using the courseware by learners, not just the courseware itself A dissemination system to distribute the Premier Courseware (via CD’s and presentation at engineering education conferences). Premier Courseware of 1997-1999 Virtual Disk Drive Design Studio Drill Dissection and Bicycle Dissection Mars Navigator Della Steam Plant MDSolids Structural Engineering Visual Encyclopedia UNH Engineering Graphics Cracking Dams 6,700 CD-ROMs Distributed For more info or to receive copies go to http://www.needs.org/engineering/premier/ © 2001 Franz J. Kurfess Knowledge Exchange 43 Needs Assessment with Members of the Math, Science and Engineering Community Purpose: To understand the math, science and engineering communities of educators and examine their needs in order to design services and structures to support users from multiple communities. Research Questions: What services, features & programs are integral to success? What do users expect with regards to quality of the holdings? Who makes upofthe SMETE digital library community? American Association Physics Teachers, American Mathematical Society, American Association for the Advancement of Science, members of the NSF Chemistry Consortia and the NSF Engineering Education Coalitions © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 44 Translating Findings into Services & Features Quality System to rapidly identify the quality of holding Place to comment about a learning object or regarding something of interest to the community Reviewers should include experts in pedagogy and content © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 45 Translating Findings into Services & Features Community Embedded structures for developing and maintaining communication links Developing community should be on par with building content Build on discipline based communities to establish connection to a broader community Content Useful content and community interaction ensures user participation as authors, reviewers, adapters/adopters, and consumers © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 46 Prototype: www.smete.org Berkeley UCOP Stanford Northern Arizona John Wiley Math Forum ENC UMBC Virginia Tech Prototype Goals (1998-2001) Develop a Prototype National SMETE Digital Library test interoperability of federated searches/shared services with partners expand requirements analysis to include K–12 develop criteria and standards to assess the impact of learning objects across disciplines implement community feedback systems, evaluate [NEEDS 2000] Building a National SMETE Digital Library at www.smete.org Searching for learning resources Cataloging (adding) learning resources Standards, Evaluating IEEE and IMS the quality of learning resources “User” reviews “Expert” reviews Forming a community of users in SMETE PKAL workshops and seminars Research on adapters © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 48 Systems Development Expanding www.smete.org/NEEDS platform Continuing to participate in the development of IEEE/IMS Learning Object Metadata Standards Adopting emerging IEEE standards Expanding user comments Implementing discussion systems Implementing customized Total Collection user profiles Engineering 58% Chemistry 21% Expanding Collections Physics 14% Expanding into Chemistry, Math 5% Physics, and Mathematics Other 2% © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 49 Collaborating with Partners Working with University of California Office of the President - (10 campuses) Identifying courseware under development systemwide Developing TLT@UC Website to showcase teaching and learning with technology at the University of California © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 50 Challenges Toward the Future... Continuing to understand and support changing user needs Improving ability to encapsulate the instructional intent and use of materials Metadata Supporting standards and cataloging practice communities of use and practice pedagogy content © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 51 Challenges Toward the Future... Continuing to understand and support changing user needs Improving ability to encapsulate the instructional intent and use of materials Metadata Supporting standards and cataloging practice communities of use and practice pedagogy Content Which allows for a user to find what they’re really looking for as well as personalization of content. © 2001 Franz J. Kurfess [NEEDS 2000] Knowledge Exchange 52 www.smete.or g NACME WebCT BioQUEST John Wiley Math Forum Berkeley ENC UMBC AAAS UCOP ILT Utah State PKAL Stanford DLESE Virginia Tech Cisco SRI Northern GDL Eduprise Arizona MAA Sun TI NSDL Goals (2000- ) Develop the National SMETE Digital Library Provide seamless access to services and resources Create a dynamic learning community that promotes and supports SMET education in the 21st century Expand Partnerships Expand Services and Community [NEEDS 2000] UNC Digital Library Project Digital Services Grants (In-Kind) REQUEST FOR PROPOSALS (RFP) Spring 2001 Agenda Overview Q&A for RFP Digital Library Background © 2001 Franz J. Kurfess [UNC Digital Library 2000] Knowledge Exchange 54 Overview Current and Future Project Services • Current Structure of the DLP • Prototype Interface for DLP © 2001 Franz J. Kurfess [UNC Digital Library 2000] Knowledge Exchange 55 Services for Curators Improved access to collections Support for a variety of media types 24x7 storage services Standards-based metadata schema Capability to support discipline-specific vocabularies Mini-grant program for digitizing high-impact collections © 2001 Franz J. Kurfess [UNC Digital Library 2000] Knowledge Exchange 56 Services for Instructors Flexible rights management (limit access to students in a course, departments, etc.) Improved access to other supporting materials Presentation options Web-based slide show Side-by-side object comparison Integration with desktop applications Support for annotations and other associated data Documentation, training © 2001 Franz J. Kurfess [UNC Digital Library 2000] Knowledge Exchange 57 Services for Scholars Common web interface to many digital collections Customizable “My Digital Library” user interface Multiple search options (keyword, browsing, etc.) Compliance with guidelines for disability access © 2001 Franz J. Kurfess [UNC Digital Library 2000] Knowledge Exchange 58 Structure of the DLP INTERFACE DATABASE MIDDLEWARE OBJECT STORAGE © 2001 Franz J. Kurfess [UNC Digital Library 2000] Knowledge Exchange 59 OTHER PROJECTS Structure of the DLP INTERFACE DATABASE MIDDLEWARE • Metadata Directory DATABASE • User Management OBJECT STORAGE • Schema Repository © 2001 Franz J. Kurfess • Crosswalk Repository • Controlled Vocabularies [UNC Digital Library 2000] Knowledge Exchange 60 OTHER PROJECTS Structure of the DLP • Metadata Transformer • Authentication and Authorization DATABASE MIDDLEWARE • Discovery Tools • Object Manipulation INTERFACE MIDDLEWARE • Presentation Tools • Class Rolls OBJECT STORAGE © 2001 Franz J. Kurfess [UNC Digital Library 2000] Knowledge Exchange 61 OTHER PROJECTS Structure of the DLP INTERFACE DATABASE MIDDLEWARE INTERFACE • My Folders • Administration, OBJECT Display, and Search STORAGE • The Buzz; What’s New? © 2001 Franz J. Kurfess [UNC Digital Library 2000] Knowledge Exchange 62 OTHER PROJECTS Prototype Interface My Folders A personal space where your uploaded objects as well as links to other objects can be organized. This space will also show the classes and projects you are currently involved in, in the capacity of instructor and/or student. The Buzz / What’s New? This space will include recent additions to the digital library, particularly those related to the visitor’s established areas of interest. The Buzz will specifically display the most accessed objects overall, within my field, etc. This space will also ultimately contain other information, such as live news feeds, both local © 2001 Franz J. Kurfess [UNC Digital Library 2000] and world-wide. Admin, Display, Search ADMIN: Allows the digital library visitor to publish his own material for others to use as well as arrange the contents of his My Folders space. DISPLAY: Allows for different ways of showing off an individual object or group of objects within one’s space. Some common “displays” here would include slideshows and thumbnails. SEARCH: Allows for basic keyword searching at any point in time, as well as access to more extensive search capabilities. Knowledge Exchange 63 Digital Library in-a-Box © 2001 Franz J. Kurfess [Sweeney & Kurfess 1998] Knowledge Exchange 64 The EXPECT Project at USC/ISI (I) EXPECT: architecture to develop knowledge based systems that uses highly declarative representations [Swartout and Gil, KAW-95], [Gil and Melz, AAAI-96] http://www.isi.edu/expect Application focus knowledge-rich planning tools, often integrated within a larger architecture Research focus knowledge acquisition tools that exploit declarative representations to help end users augment KBs exploiting declarative representations to mediate agent communication [Blythe&Gil&Chalupsky&MacGregor 00] © 2001 Franz J. Kurfess [Gil 2000] Knowledge Exchange 65 The What-to-How Spectrum [Feigenbaum 96] The history of software development Software translates “WHAT” to “HOW” © 2001 Franz J. Kurfess [Gil 2000] Knowledge Exchange 66 The Good News => The Bad News As we develop more advanced and higher-level languages, we still require formalisms and logic Complete information Consistent statements Executable/compilable Requires putting knowledge “in a straightjacket” Represent captures only the result of the modeling process Represent only a subset of the overall knowledge i.e., whatever fits what the formalism requires Represent knowledge once it is digested only what there is time to represent the rest is assumed unnecessary © 2001 Franz J. Kurfess [Gil 2000] Knowledge Exchange 67 Connections Are Missing! “There is a real conflict between the logician’s goal and the educator’s. The logician wants to minimize the variety of ideas, and doesn’t mind a long thin path. The educator (rightly) wants to make the paths short and doesn’t mind - in fact, prefers - connections to many other ideas.” [Minsky 1969] Knowledge turns up piecemeal Compartmentalized and seldom rejoined Only truly necessary views/opinions are represented Only truly necessary connections are represented As long as the information is there, the system should figure it out No value to redundancy/alternatives © 2001 Franz J. Kurfess [Gil 2000] Knowledge Exchange 68 The Need for Knowledge Mobility: HPKB Challenge Problem - Year 1 LOOM HPKB Upper Ontology BS Ontologies (military units, vehicles, etc) KB of Eng. Actions COA Generation Problem Solver © 2001 Franz J. Kurfess COA Ontology Eng. Techniques Ontology Features and Terrain (Stub) Ontology Eng. Equpt. Ontology Temporal Estimation/ Assessment Problem Solver Methods for Time Estimation EXPECT [Gil 2000] Knowledge Exchange 69 The Need for Knowledge Mobility: HPKB Challenge Problem - Year 2 © 2001 Franz J. Kurfess [Gil 2000] Knowledge Exchange 70 The Need for Knowledge Mobility: JFACC ISTI Architecture Plan JTF Plan Server MASTERMIND Refined plan Web-Based Planner CJTF Objectives CJTF Objectives (Manual Entry) (CORBA) Plan Expanded Plan JMTK Map Visualization Plan Plan SDA Plan Critiques Knowledge Base Server (LOOM) © 2001 Franz J. Kurfess CPET/APAT Targeting (Automated) INSPECT-II [Gil 2000] [Gil 2000] Knowledge Exchange 71 The Need for Knowledge Mobility: ARPI MAPViS MPA Planning Agents Meta Planning-Cell Manager (PRS) Process Monitor APAT GUI Plan Viewer MPA Agent Evaluation Toolkit ACS VISAGE Advisable Planner (SIPE-2) Planner (SIPE-2) Scheduler (OPIS) Planning-Cell Manager (PRS) Beddown Critic Plan Server Cue: (TEST (ready unit1)) ACT2 ACT1 Cue: Answer query Annotations Inspect Triggers © 2001 Franz J. Kurfess [Gil 2000] Knowledge Exchange 72 The Need for Knowledge Mobility Reuse and integration [Valente et al., IEEE IS 99] Ontology/theory manipulation operations [Knight&Luk, AAAI-94] [Hovy et al. 94] [MacGregor& Chalupsky 99] Aligning, Morphing merging, slicing, etc. [Chalupsky, KR-2000] Transformation-based knowledge morphing Repeated mergers and imports are the norm Ontology-based translation Mismatches resolved on a case by case basis, on demand Hub and spokes model © 2001 Franz Kurfess J. n-degrees Exchange 2000] of indirection[Gil if agents have n-degreesKnowledge of “ontological 73 What is Needed: Resilient Hyper Knowledge Bases (RHKB) Extend our view of what a KB contains Support a variety of knowledge media/formats/representations Include alternative renderings of knowledge Everything should remain part of the knowledge base Initial sources: introductory texts, expert hints, explanations, dialogues, comments, examples, exceptions, … Intermediate analysis and models of those sources as they are digested Final representations Emphasis on knowledge connectivity Among progressively more digested renderings of the same knowledge © 2001 Franz J. Kurfess [Gil 2000] Knowledge Exchange 74 Reference [Kearns 00] © 2001 Franz J. Kurfess [Kearns 00] Knowledge Exchange 76 Reference [Sommerville 01] [Sommerville 01] © 2001 Franz J. Kurfess [Sommerville 01] Knowledge Exchange 77 Post-Test © 2001 Franz J. Kurfess Knowledge Exchange 78 References [Gil 2000] Yolanda Gil, Knowledge Mobility. Dagstuhl Workshop “Semantics for the Web”, March 2000. [NEEDS] National Engineering Digital Library, www.needs.org [Russell & Norvig 1995] Stuart Russell and Peter Norvig, Artificial Intelligence - A Modern Approach. Prentice Hall, 1995. © 2001 Franz J. Kurfess [Gil 2000] Knowledge Exchange 80 Important Concepts and Terms agent automated reasoning belief network cognitive science computer science hidden Markov model intelligence knowledge representation linguistics Lisp logic machine learning microworlds © 2001 Franz J. Kurfess natural language processing neural network predicate logic propositional logic rational agent rationality Turing test Knowledge Exchange 81 Summary Chapter-Topic © 2001 Franz J. Kurfess Knowledge Exchange 82 © 2001 Franz J. Kurfess Knowledge Exchange 83