Digital Library as Infrastructure for Knowledge Management in Academic and Research Institutions: Open Archives Initiative and Educational Demonstrations (NDLTD, CITIDEL/NSDL) Amigos 2002 February 21, 2002 Edward A. Fox fox@vt.edu http://fox.cs.vt.edu CS DLRL Internet TIC Virginia Tech, Blacksburg, VA, USA Comunicación del conocimiento basada en la tecnologia de la información Amigos 2002 February 21, 2002 Eduardo Zorro fox@vt.edu Virginia Tech, Blacksburg, VA, USA Acknowledgements (Selected) • Sponsors: Adobe, CNPq, CONACyT, DFG, IBM, Microsoft, NSF, OCLC, SOLINET, SURA, UNESCO, US Dept. of Ed. (FIPSE), … • VT Faculty/Staff: Tony Atkins, Debra Dudley, John Eaton, Gail McMillan, Len Peters, James Powell, … • VT Students: Emilio Arce, Fernando Das Neves, Robert France, Marcos Goncalves, Rohit Kelapure, Neill Kipp, Aaron Krowne, Ming Luo, Paul Mather, Kate McDevitt, Ryan Richardson, Rao Shen, Priya Shivakumar, Arthi Sundararajan, Hussein Suleman, Jun Wang, Wensi Xi, Ye Zhou, Qinwei Zhu, … Overview • • • • • • • Purpose: Scholarship / Knowledge Management Goals: Discovery of Knowledge, Scholarly Communication Objectives: Learning, Advancing Knowledge, Sharing Mechanisms: Electronic Publishing, Digital Libraries Digital Libraries: Content Mgmnt , Workflow Mgmnt Distributed DLs: Open Archives Initiative, Open DLs Examples: NDLTD, NSDL, CITIDEL • Networked Digital Library of Theses and Dissertations • National STEM education Digital Library • Computing and Information Technology Interactive Digital Educational Library Synchronous Scholarly Communication Asynchronous, Digital Library Mediated Scholarly Communication Information Life Cycle Borgman et al.: Workshop Report on Social Aspects of Digital Libraries: http://www-lis.gseis. ucla.edu/DL/ Information Life Cycle Authoring Modifying Using Creating Retention / Mining Organizing Indexing Accessing Filtering Storing Retrieving Distributing Networking Overview - DL • • • • Purpose: Scholarship / Knowledge Management Goals: Discovery of Knowledge, Scholarly Communication Objectives: Learning, Advancing Knowledge, Sharing Mechanisms: Electronic Publishing, Digital Libraries • Digital Libraries: Content Mgmnt , Workflow Mgmnt • Distributed DLs: Open Archives Initiative, Open DLs • Examples: NDLTD, NSDL, CITIDEL Digital Library Courseware • http://ei.cs.vt.edu/~dlib/ • WWW pages or large PDF copy files • Online quizzes based on book by Michael Lesk (Morgan Kaufmann Publishers; new one in June by Ian Witten too!) • Contents based on book, with several other popular topics added (e.g., agents) • Separate pages to supplement: Definitions, Resources (People, Projects), and References Digital Libraries --- Objectives • World Lit.: 24hr / 7day / from desktop • Integrated “super” information systems -> 5S: streams, structures, spaces, scenarios, societies • Ubiquitous, Higher Quality, Lower Cost • Education, Knowledge Sharing, Discovery • Disintermediation -> Collaboration • Universities Reclaim Property • Interactive Courseware, Student Works • Scalable, Sustainable, Usable, Useful Benefits • Ease of use • Effectiveness • “The benefits of digital libraries will not be appreciated unless they are easy to use effectively.” - IITA Workshop report DLs: Why of Global Interest? • National projects can preserve antiquities and heritage: cultural, historical, linguistic, scholarly • Knowledge and information are essential to economic and technological growth, education • DL - a domain for international collaboration • • • • wherein all can contribute and benefit which leverages investment in networking which provides useful content on Internet & WWW which will tie nations and peoples together more strongly and through deeper understanding Libraries of the Future JCR Licklider, 1965, MIT Press World Nation State City Community DL Challenges • Preservation - so people with trust DLs • Supporting infrastructure - networks, ... • Scalability, sustainability, interoperability • DL industry - critical mass by covering libraries, archives, museums, corporate info, govt info, personal info - “quality WWW” integrating IR, HT, MM, ... • Need tools & methods to make them easier to build Communications (bandwidth, connectivity) Locating Digital Libraries in Computing and Communications Technology Space Digital Libraries technology trajectory: intellectual access to globally distributed information Computing (flops) Digital content less more (Slide from S. Griffin, NSF) Service Machine 1 Service PetaPlex Complex Service Machine 2 Nanoserver FRONT END MACHINE RS/6000, 1G RAM, 4 Proc. Machine 3 Service Machine 4 Nanoserver Nanoserver Nanoserver Nanoserver Nanoserver Nanoserver Nanoserver Nanoserver Nanoserver Nanoserver Nanoserver Nanoserver Nanoserver Definitions • Library ++ (library+archive+museum+…) • Distributed information system + organization + effective interface • User community + collection + services • Digital objects, repositories, IPR management, handles, indexes, federated search, hyperbase, annotation Reading Book Abstract Definition: Digital Libraries are complex systems that • • • • • help satisfy info needs of users (societies) provide info services (scenarios) organize info in usable ways (structures) present info in usable ways (spaces) communicate info with users (streams) 5S Layers Societies Scenarios Spaces Structures Streams DL Components Gateways MM/ HT Renderer User Interfaces Workflow Mgr Search Engines, Classifiers, … DBMS Rights Mgr Data, MM Info Repository Open Digital Library (ODL) Hypothesis (Hussein Suleman) • Can we leverage the successful model of the OAI Protocol for Metadata Harvesting to alleviate our architectural problems ? Maybe … if Digital Libraries can be modeled as • networks of extended Open Archives, where • each extended Open Archive is a • source of data and/or a provider of services. Proposed Solution • (Open) DL = Network of Extended OAs Data Input Local Archive Resource Discovery Search Browse Recommend Metadata Repository legend Remote Archive User Interface OAI/ODL archive OAI/ODL protocol Example Architecture (NDLTD) Virginia Tech User Interface PhysNet Humboldt Search Browse Recent Duisburg CalTech Union Catalog MIT Filter MIT legend Dresden User Interface OAI/ODL archive OAI/ODL protocol Overview - CM • • • • Purpose: Scholarship / Knowledge Management Goals: Discovery of Knowledge, Scholarly Communication Objectives: Learning, Advancing Knowledge, Sharing Mechanisms: Electronic Publishing, Digital Libraries • Digital Libraries: Content Mgmnt , Workflow Mgmnt • Distributed DLs: Open Archives Initiative, Open DLs • Examples: NDLTD, NSDL, CITIDEL Digital Library Content Content Types Text Documents Video Audio Geographic Information Software, Programs Bio Information Images and Graphics Articles, Reports, Books Speech, Music (Aerial) Photos Models Simulations Genome Human, animal, plant 2D, 3D, VR, CAT Integrated CCLINC Translingual Information System DARPA CCLINC SERVER Translation It seems that North Korea launch a missile again After North Korea launched a Daipodong missile last month, NK is perceived to proceed to an additional test launch. Korea, US and Japan enter into an alert state, and prepare for a joint response policy. Korea estimates that the additional launch will be on 09/05. Japan estimates that NK’s missile range is short. US information says that there is no sign of launch yet. Structured Video Browser (making video into hypermedia) www.learn.umd.edu • IBrowse • Expository multimedia • Narrative Structures MPEG- MPEG-7 Image Library Systems Tech. 7 Image Library Systems Users Web Search Engines WWW 1 5 Servlet Servlet 3’ Servlet 4 Servlet Engine 4’ MPEG-7 Description Module 2 5’ Web Server Search Server OS 3 DB and Communication ICU Information University MPEG7 MPEG-7 Video Library Systems Tech. Video Library Systems Tech. Architecture Video Data Description Generator Description Scheme Description Schemes Design Tool Player Video Database Retrieval Server Module Presentation Module Meta Database and Communication ICU Information University How is numeric information created, distributed, consumed? Basic Example About • enumerate founders know numbers, computers, and the Web. • They saw the Web’s potential to solve a big problem: Working with numeric information is too hard! How does enumerate help? RDL • Value - What is the number? • Format - $100 [in thousands] • Semantics - How it translates or corresponds to other formats • Provenance – created by whom and when? • Measure - scale about the number: • Units: feet, meters, $, pounds, RBI • Magnitude: thousands, millions, billions • Modifiers: number been manipulated? • Structure - relationship of numbers to each other How does enumerate help? Capture intelligence about data Save data + intelligence in a file Transmit or store the file Build software to decode the file and automate data tasks RXL does the same thing for analytics The enumerate formula enumerate’s interactive data can be analyzed instantly You Do It Numerator! Publish™ Applications RDL Editor – including Applet / HTML Tag Wizard DataPublisher™ Enterprise Views & Applets Standard Chart View Standard Table View Combination View Full Feature View Native XML/RDL Views Overview – Workflow Mgmnt • • • • • Purpose: Scholarship / Knowledge Management Goals: Discovery of Knowledge, Scholarly Communication Objectives: Learning, Advancing Knowledge, Sharing Mechanisms: Electronic Publishing, Digital Libraries Digital Libraries: Content Mgmnt , Workflow Mgmnt • Distributed DLs: Open Archives Initiative, Open DLs • Examples: NDLTD, NSDL, CITIDEL Digital Libraries Shorten the Chain from Editor Reviewer Publisher A&I Consolidator Library DLs Shorten the Chain to Author Teacher Digital Reader Editor Reviewer Learner Librarian Library NDLTD Workflow • • • • • • • Building Infrastructure Authoring Submission Local Processing Providing Access Discovery and Access Scholarship Contact Our Project Team Video Tape E-mail etd@ndltd.org Phone Call Visit Convene Local Planning Group ETD Build Local ETD Site ETD Workshop/Training Digital Library Policies Inspection/Approval Student Prepares Thesis/Dissertation NDLTD Literature Computer Resources Research Student Defends & Finalizes ETD My Thesis ETD Student Gets Committee Signatures and Submits ETD Signed Grad School Graduate School Approves ETD, Student is Graduated Ph.D. Library Catalogs ETD, Access is Opened to the New Research WWW NDLTD Approaches to Aggregation Build By Institution Build By Discipline Approaches to Aggregation Build By Institution Build By Discipline Author Category Interdisciplinary Year Language Query … User Search Support (multilingual, XML) NDLTD World Federated Search User Interface Virginia Tech ... (univ) Dissertations Online (Germany) OhioLink Portugese NL ... (lib / univ group) (national lib) Australia (regional) OAS, ISTEC (Latin America) Note: All groups shown are connected with NDLTD. Access Possibilities Web search engines www. theses. org Virginia MIT National Tech Library of Portugal www. library openarchives. catalog org clients CBUC (Spain) Ohio Link 3rd Party Services (e.g., UMI) National Projects: AU, GE, … Overview – OAI • Purpose: Scholarship / Knowledge Management • Goals: Discovery of Knowledge, Scholarly Communication • Objectives: Learning, Advancing Knowledge, Sharing • Mechanisms: Electronic Publishing, Digital Libraries • Digital Libraries: Content Mgmnt , Workflow Mgmnt • Distributed DLs: Open DLs Open Archives Initiative, • Examples: NDLTD, NSDL, CITIDEL Open Archives Initiative OAI www.openarchives.org openarchives@openarchives.org OAi Philosophy • • • • • Self-archiving = submission mechanism Long-term storage system = archive Open interface = harvesting mechanism Data provider + service provider Start with “gray literature” • e-prints/pre-prints, reports, dissertations, … Budapest Open Access Initiative • • • • Launched Feb. 14 Open Society Institute Free access to refereed research literature Signatories: Fox, NDLTD, … • Guédon – refer to his talk • http://www.soros.org/openaccess/ commitment.shtml Open Archives Initiative (OAI) • • • • xxx@LANL, high-energy physics (Ginsparg, 1991) CSTR + WATERS = NCSTRL (Lagoze,1994) xxx + NCSTRL = CoRR collaboration (1998) Universal Preprint Service protoproto, Oct. 21-22, 1999, Santa Fe – led by LANL, CNI, DLF, Mellon --> OAi • Santa Fe Convention (see Feb. D-Lib Magazine article) • Follow-on mtgs: 6/3@San Antonio, 9/21@Lisbon (ECDL) • Archives -> Open Archives • • • • Support unique archive identifiers Implement Open Archives metadata set (DC, using XML) Implement OA harvesting protocol (derived from Dienst protocol) Register the archive • Build tools, layer other services: linking, searching, … Open Archives (protoproto) • • • • • • • ArXiv & Los Alamos National Lab CogPrints & U. Southampton NACA & NASA (reports) NCSTRL & Cornell U. NDLTD & Virginia Tech RePEc & U. Surrey Total of around 200K records Original Open Archives Members • • • • • • • • • American Physical Society California Digital Library Caltech Coalition for Networked Info. Cornell University Harvard University Library of Congress Los Alamos Nat’l Lab Mellon Foundation • • • • • • • • • NASA Langley Research Cntr Old Dominion University Stanford University U. of Ghent U. of Surrey U. of Southampton Vanderbilt University Virginia Tech Washington University Harvesting vs. Federation • Competing approaches to interoperability • Federation is when services are run remotely on remote data (e.g., federated searching) • Harvesting is when data/metadata is transferred from the remote source to the destination where the services are located (e.g., union catalogues) • Federation requires more effort at each remote source but is easier for the local system and vice versa for harvesting • OAI (currently) focuses on harvesting OAI – Black Box Perspective OA 7 OA 4 OA 2 OA 1 OA 3 OA 6 OA 5 Metadata vs. Data • Data refers to digital objects or digital representations of objects • Metadata is information about the objects (e.g. title, author, etc.) • OAI focuses on metadata, with the implicit understanding that metadata usually contains useful links to the source digital objects Repository of Digital Objects Repository Access Protocol handle terms and conditions Digital object OAI – Repository Perspective Required: Protocol MDO MDO MDO MDO MDO MDO MDO MDO DO DO DO DO Tiered Model of Interoperability Mediator services Metadata harvesting Document models Mechanisms • Sharing • Join federation, run software • Make metadata and archive available • Aggregating • By discipline • By institution • By genre • Automating • • • • Workflow Harvesting and providing services Federated searching Dynamic linking (e.g., with SFX (OpenURLs)) Technical Umbrella for Practical Interoperability… Reference Libraries Museums Publishers E-Print Archives …that can be exploited by different communities The World According to OAI Service Providers Discovery Current Awareness Data Providers Preservation Key Features of the OAI Metadata Harvesting Protocol • definitions & concepts • repository • record • identifier • datestamp • set • protocol features • HTTP encoding • metadata prefix & schema • flow control • protocol requests • supporting requests • harvesting requests identifiers locally unique key for extracting a record from a repository oai-identifier = oai:archive-identifier:record-identifier Registered URI Scheme Archive Identifier: Registered within OAI Unique ID within archive: (syntax is archivespecific) example = oai:ncstrl:ncstrl.cornellcs/TR94-1418 repository support data harvesting data h a r v e s t e r OAI protocol r e p o s i t o r y items selective harvesting - datestamps harvest within date range record record r e p o s i t o r y selective harvesting - sets harvest within set record record record r e p o s i t o r y S1 S2 OAI Tools • • • • Related resources, e.g., XML, Unicode XML Schema Validator Servers and utilities, e.g., ARC, Kepler, EPrints Repository Explorer • • • • • Interactive Browsing Testing of parameters Multiple views of data Multilingual support Automatic test suite ARC (arc.cs.odu.edu) OAI Repository Explorer • Serves as a compliancy test • Allows browsing of open archives using only OAI protocol • Sends requests on behalf of user, parses and checks responses and displays browsable interface • Will detect most discrepancies in protocol • http://purl.org/net/explorer RE 1.3 Request, Response – OAI, VT ETDs Request http://scholar.lib.vt.edu/theses/OAI/cgi-bin/index.pl? verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai:VTETD:etd-520112859651791 Response Overview - NDLTD • Purpose: Scholarship / Knowledge Management • Goals: Discovery of Knowledge, Scholarly Communication • Objectives: Learning, Advancing Knowledge, Sharing • Mechanisms: Electronic Publishing, Digital Libraries • Digital Libraries: Content Mgmnt , Workflow Mgmnt • Distributed DLs: Open Archives Initiative, Open DLs • Examples: NDLTD, NSDL, CITIDEL NDLTD: DL Case Study • Domain: graduate education, research • Genre: ETDs=electronic theses & dissertations • Submission: http://etd.vt.edu • Collection: http://www.theses.org Project: Networked Digital Library of Theses & Dissertations (NDLTD) http://www.ndltd.org The Networked Digital Library of Theses and Dissertations www.NDLTD.org Training Authors Expanding Access Preserving Knowledge Improving Graduate Education Enhancing Scholarly Communication Empowering Students & Universities Leader of the Worldwide ETD (Electronic Thesis and Dissertation) Initiative NDLTD Grad Program IT Library Ed. (Tech) What led to today’s meeting? • 1987 mtg in Ann Arbor: UMI, VT, … • 1992 mtg in Washington: CNI, CGS, UMI, VT and 10 universities with 3 reps each • 1993 mtg in Atlanta to start Monticello Electronic Library (regional, US Southeast): SURA, SOLINET • 1994 mtg at VT: std: PDF + SGML + multimedia objects • 1996 funding by SURA, US Dept. of Education (FIPSE) • 1997 meetings in UK, Germany, ... • 1998 – 1st symposium – Memphis (20) • 1999 – 2nd symposium – Blacksburg (70) • 2000 – 3rd symposium – St. Petersburg (225) • 2001 – 4th symposium – Caltech (200) • 2002 - May 30 – June 1, BYU; 2003 – Spring, in Berlin What are the long term goals? • 400K US students / year getting grad degrees are exposed / involved • 200K/yr rich hypermedia ETDs that may turn into electronic portfolios (images, video, audio, …) • Dramatic increase in knowledge sharing: literature reviews, bibliographies, … • Services providing lifelong access for students: browse, search, prior searches, citation links • Hundreds/thousands of downloads / year / work What are we doing? • Aiding universities to enhance graduate education, publishing and IPR efforts • Helping improve the availability and content of theses and dissertations • Educating ALL future scholars so they can publish electronically and effectively use digital libraries (i.e., are Information Literate and can be more expressive) Key Ideas: Scalability Networked infrastructure University collaboration Workflow, automation Education is the rationale Maximal Access 8th graders vs. grads Authors must submit Standards PDF, SGML, MM, MARC, DC, URNs, Federated search Status of the Local Project • Approved by university governance Spring 1996; required starting 1/1/97 • Submission & access software in place • Submission workshops for students (and faculty) occur often: beginner/adv. • Faculty training as part of Faculty Development Initiative • Over 3800 ETDs in collection – some have audio, video, large images, software, … ETDs: Library Goals • Improve library services • Better turn-around time • Always available • Reduce work • catalog from e-text • eliminate handling: mailing to UMI, bindery prep, check-out, check-in, reshelving, etc. • Save space Archiving ETDs • Every 15 minutes back-ups made of notyet-approved submissions • Hourly back-ups of newly approved ETDs • Weekly back-ups of entire ETD collection • Copies stored on-site and off-site VT ETD Cataloging • same as current cataloging policies, except: • author-assigned keywords (not LCSH) • generic (not LC) call no. • fields/subfields as required for computer files • full abstracts • time savings • cataloger familiar with computer files • equipment, software for word processing • 5 minutes avg. (10-15 minutes for paper TDs) Author‘s tools www.physik.uni-oldenburg.de/EPS/mmm Library Costs • $12/vol. for paper thesis processing • catalog, bind, security strip, label, shelve • @950 vols./yr. = $11,466 • $3.20/vol. ETD processing • cataloging @950 vols./yr. = $3040 • $.07/vol. shelving • $.04/vol. circulation Costs/Savings at VT • Graduate School stopped shipping to the library 3000 copies of paper TDs/year • Library stopped binding, shelving, and circulating 3000 copies of TDs/year • 166 ft of shelf space saved/year by the library • VT used existing equipment in Library (vs. start-up costs for staff, hardware and software from from a zero-base estimate: $65,000 – see http://scholar.lib.vt.edu/theses/) Popular Works 1997 9920 Liu, Xiangdong. Analysis and Reduction of Moire Patterns in Scanned Halftone Pictures (Ph.D. in Computer Science, May 1996; 6.6Mb) 7656 Petrus, Paul. Novel Adaptive Array Algorithms and Their Impact on Cellular System Capacity (Ph.D. in Electrical Engineering, March 1997; 5Mb) 2781 Agnes, Gregory Stephen. Performance of Nonlinear Mechanical, Resonant-Shunted Piezoelectric, and Electronic Vibration Absorbers for Multi-Degree-of-Freedom Structures (Ph.D. in Engineering Mechanics, Sept. 1997; ? + 7926Kb) 2492 Gonzalez, Reinaldo J. Raman, Infrared, X-ray, and EELS Studies of Nanophase Titania (Ph.D. in Physics, July 1996; 4607Kb) 1877 Shih, Po-Jen. On-Line Consolidation of Thermoplastic Composites (Ph.D. in Engineering Mechanics, Feb. 1997; 3.3Mb) 1791 Saldanha, Kevin J. Performance Evaluation of DECT in Different Radio Environments (MS in Electrical Engineering, Aug. 1996; 3.2Mb) 1431 DeVaux, David. A Tutorial on Authorware (MS in CS, April 1996; 2.3Mb) 1394 Kuhn, William B. Design of Integrated, Low Power, Radio Receivers in BiCMOS Technologies (Ph.D. in Electrical Engineering, Dec. 1995; 2518Kb) Institutional Members • • • • • • • • • • • • • • • • Cinemedia Coalition for Networked Information (CNI) Committee on Institutional Cooperation (CIC) Consorci de Biblioteques Universitàries de Catalunya Diplomica.com Dissertation.com Dissertationen Online (Germany) ETDweb, a Division of Answer4.com Ibero-American Science & Technology Education Consortium (ISTEC) National Documentation Centre (NDC), Greece National Library of Portugal (for all universities) OCLC Online Computer Library Center OhioLINK Organization of American States (SEDI/OAS) Southeastern Library Network (SOLINET) UNESCO (www.unesco.org/webworld/etd) National / Regional Projects • Australia • • • • • • • U. New South Wales (lead) U. of Melbourne U. of Queensland U. of Sydney Australian National U. Curtin U. of Technology Griffith U. • Germany • Humboldt University (lead) • 3 other universities • 5 learned societies: Math, Physics, Chemistry, Sociology, Education • 1 computing center • 2 major libraries • OhioLINK: 79 colleges/univs • Consorci de Biblioteques Universitàries de Catalunya, as group, www.cbuc.es: • Universitat de Barcelona • Universitat Autonòma de Barcelona • Universitat Politècnica de Catalunya • Universitat Pompeu Fabra • Universitat de Girona • Universitat de Lleida • Universitat Rovira i Virgili • Universitat Oberta de Catalunya • Biblioteca de Catalunya • India, … OhioLINK • • • • • • Statewide Consortium Represents 79 colleges, universities, libraries Public Universities Private Universities and Colleges 2-Year Colleges Only a few (e.g., Miami U. of Ohio) are also NDLTD members on their own US University Members (54) • • • • • • • • • • • • • • • • • • • • • • • • • • • Air University (Alabama) Baylor University Brigham Young University (part, whole) Caltech Clemson University College of William & Mary Concordia University (Illinois) East Carolina University East Tenn. State U. – require fall 2000 Florida Institute of Technology Florida International University George Washington University Georgetown University Louisiana State University Marshall University (W. Va.) Miami University of Ohio Michigan Tech Mississippi State University MIT Montana State University Naval Postgraduate School (CA) New Jersey Inst. of Technology New Mexico Tech North Carolina State University Northwestern University Penn. State University Regis University • • • • • • • • • • • • • • • • • • • • • • • • • • • Rochester Institute of Tech. Texas A&M U. of Colorado Health Science Center U. of Florida U. of Georgia University of Hawaii, Manoa U. of Iowa U. of Kentucky U. of Maine U. of North Texas – required since 8/99 U. of Oklahoma U. of New Orleans U. of Pittsburgh U. of Rochester U. of South Florida U. of Tennessee, Knoxville U. of Tennessee, Memphis U. of Texas at Austin – required in 2001 U. of Virginia U. of West Florida U. of Wisconsin - Madison Vanderbilt U. Virginia Commonwealth U. Virginia Tech - required since 1/97 West Virginia U. - required fall 1998 Western Michigan U. Worcester Polytechnic Inst. Other Countries - 53 Members • • • • • • • • • • • Australia Belgium Brazil Canada China, Hong Kong Columbia Germany India Italy Korea Mexico • • • • • • • • • • • Netherland Norway Russia Singapore S. Africa S. Korea Spain Sudan Sweden Taiwan UK Type 1 Members University Requires ETDs • Adobe Acrobat and/or XML/SGML tools • Automated submission & processing • Archive/access through UMI, (OCLC,) Virginia Tech, ... • (Local) WWW site, publicity • (Local) Assistance provided as requested: email, phone, listserv(s) Type 2 Members University Agrees to Require ETDs • Like Type 1 but set date not reached • Usually has an option or pilot • May: wait for new AY; start with all who enter after; … • Build grass roots support • • • • Advisory committee: representative? expert? Champions to spread by word of mouth Approval: Senates, Commissions, Deans, Students Publicity to reach community NDLTD Members, Types 3-7 • • • • 3. Part of university requires ETDs 4. University allows ETDs 5. University investigating, has pilot 6. University consortium joins: • CIC (Big 10 coordinating body) • 7. Non-university organization joins • CNI (Coalition for Networked Info.) Counts of ETDs at Selected U’s University/Institution ETD Collection size ADT: Australian Digital Thesis Program (Australia) 238 University of Bergen (Norway) 45 California Institute of Technology 2 Consorci de Biblioteques Universitaries de Catalunya (Spain) 151 East Tennessee State University 106 Humboldt-University (Germany) 430 Louisiana State University 3 Mississippi State University 33 MIT 62 North Carolina State University 301 Pennsylvania State University 83 Pontifical Catholic University (PUC) (Brazil) 90 Gerhard Mercator Universitat Duisburg (Germany) 126 Universitat Politecnica de Valencia (Spain) 189 University of Florida 174 (continued) Counts of ETDs at Selected U’s (cont’d) University/Institution University of Georgia ETD Collection size 121 University of Iowa 6 University of Kentucky 19 University of Maine 27 University of North Texas 337 University of South Florida 25 University of Tennessee 12 University of Tennessee, Knoxville 28 Uppsala University (Sweden) 178 Virginia Tech 3393 West Virginia University 1006 Worcester Polytechnic Institute TOTAL 83 7268 Counts of University Scanned ETD Collections University/Institution ETD Collection Size MIT 5,581 National Documentation Center, Greece 12,000 New Jersey Institute of Technology 26 University of South Florida 150 TOTAL 17,763 VT ETD Access Logs 1997/98 Requests for PDF (mostly full ETDs) 1998/99 Increase 1997/98 -1998/99 1999/00 Increase 1998/99 -1999/00 files 221,679 481,038 117.0% 578,152 20.2% 165,710 215,539 30.1% 260,699 21.0% Requests for multimedia 1,714 4,468 160.7% 12,633 182.7% Distinct files requested 6,419 21,451 234.2% 16,409 -23.5% 29,816 57,901 94.2% 87,804 51.6% 156 MB 219 MB 40.4% 382 MB 74.4% 55 GB 78 GB 40.4% 137 GB 75.6% Requests for HTML files (mostly tables of contents and abstracts) Distinct hosts served Average data transferred daily Data transferred VT ETD Access by Int’l Sites International Domain 1997/98 1997/98 rank 1998/99 1998/99 rank Increase 1997/98 -1998/99 1999/00 1999/00 rank Increase 1998/99 -1999/00 United Kingdom 6,735 1 11,347 1 68.5% 25,583 1 125.5% Malaysia 876 16 4,190 6 378.3% 16,147 2 285.4% France 2,138 7 4,797 5 124.4% 14,960 3 211.9% Germany 6,727 2 3,374 9 -49.8% 14,384 4 326.3% Canada 3,413 4 9,632 3 182.2% 13,543 5 40.6% 590 18 3,647 8 518.1% 9,918 6 171.9% 1,430 12 3,095 10 116.4% 9,300 7 200.5% Spain Italy Multimedia Use in ETD Collection File type Examples Count Still image BMP, DXF, GIF, JPG, TIFF 328 Video AVI, MOV, MPG, QT 58 Audio AIFF, WAV 18 Text PDF, HTML, TXT, DOC, XLS Other Macromedia, SGML, XML 7601 51 Access Choices at VT (7/2000) Mixed 19% Patent 3% Worldwide 54% VT-Only 24% Who are sponsors / cooperators? • Funding, Donations of hardware/software • • • • • • • • SURA US Dept. of Education (FIPSE) Adobe Systems IBM Microsoft OCLC NSF, DFG, CONACyT UNESCO • Others Serving on Steering Committee • National/Regional Projects: Australia, French speaking group, Germany, IberoAmerica (ISTEC), UK (UTOG) • CGS, CRL, National Lib. Canada, NSF, OAS, SOLINET, UMI, UNESCO, ... For professional societies • Like “writing across the curriculum”, e.g., Chemical Markup Language, MathML, … • Besides writing: computing/communications, information literacy, personal digital library management, tool use, research methods, collaboration, archiving/preservation • Data sets, communities of users of them • Classification systems / browsing / searching • NRC’s “Issues for Science and Engineering Researchers in the Digital Age”, 57 pages Relationship with publishers • Concern of faculty and students that still wish to publish books or journal articles, voiced: campus, Chronicle, NPR, Times • Solution: Approval Form gives students, faculty choices on access, when to change access condition; use IPR controls in DL • Solution: by case, work with publishers and publisher associations to increase access • AAP, AAUP • AAAS, ACM, ACS, Elsevier, ... Some responses from publishers • • • • • ACM: need to acknowledge copyright Elsevier: need to acknowledge copyright IEEE-CS: endorse initiative ACS: After first publication, can release Textbook publishers: different market, manuscript significantly reworked • General: restricting access to local campus will not cause any problems How does this relate to UMI? • Generally, they are independent decisions. • • • • 1987 UMI workshop was first to explore ETDs. UMI wrote support letter for US Dept. of Ed. proposal. UMI is on Steering Committee. ProQuest Direct pilot of scanning works started 1/1/97, with free 2 yr access to front part. • We are collaborating on: • accepting electronic author submissions • standards (e.g., representation) ETD Initiative (and UMI) Students Learn about DL, EPub TDs become more expressive Global TDs become more accessible, archived Universities UMI N. Amer. (T)Ds are accessible, archived ETD Encoding Decisions • Text • UNICODE (with language identifiers) • Structure • MARC (MARC-21 or UNIMARC) PLUS • XML / RDF / DC + ETD • Multimedia • Following international standards • Other schemes may not be amenable to preservation www.theses.org • James Powell student project, D-Lib Magazine description in Sept. 1998 • XML description of each site • type of search engine / service • language • coverage (for resource discovery) • Adding Z39.50 gateway capability and integrating with MARIAN, along with Harvest and Open Archives protocols Access Approaches • Goal: Maximize access and services, e.g., by encouraging: • UMI centralized services • VTLS: planned free union collection of metadata • Distributed service: Dienst, Z39.50 • Regional services (e.g., OhioLink) • Local servers with browse, search • From local catalogs to local archives • WWW robot indexing and search services Why might a university want to be involved? • To improve graduate education / better prepare your students / increase their knowledge (epub, DLs, IPR) and visibility • To enhance university infrastructure (DL) • To unlock university information • To save money for students and for the university / improve workflow • To build an important digital library (of ETDs) Multiple objectives • Sharing research results • Decrease costs, increase services • Increase knowledge of users • Adding to author knowledge/skills • Epub, DL, IPR • Enhancing organization’s infrastructure • CS department, library • University, Laboratory How can a university get involved? • Select planning/implementation team • • • • Graduate School Library Computing / Information Technology Institutional Research / Educational Technology • Send us letter, give us contact names • www.ndltd.org/join • Adapt Virginia Tech solution • Build interest and consensus • Start trial / allow optional submission Pilot to Requirement • Shift as soon as feasible, to reduce costs and speed up collection growth. • Build on leadership wherever it exists. • Encourage student enthusiasm/innovation. • Financial incentive? Awards? Advertising? • New Media Center? FDI/GDI equivalent? • Adapt others’ solutions to reduce effort. • Build on campus’ existing infrastructure & related work. • Solve problems / address questions immediately. Resources Required • • • • • • • • Problem solver(s)/leader(s) Advertising Training Nurture pilot. Assistance: “1-stop-shopping” Documentation Liaison with NDLTD and ETD’2002 … OAI, Union Catalog Assistance • • • • • www.etdguide.org Marcel Dekker book soon in press ETD’2002 starting May 30 at BYU www.ndltd.org Support in Latin America by UNESCO and ISTEC (e.g., workshops for trainers) Support Offered • • • • Software, documentation, tech support Email, listservs (etd-l@listserv.vt.edu) UNESCO training, Guide (www.etdguide.org) NDLTD Committees • Steering • Standards • Conference • Strategic Planning NDLTD Members and ETD-MS • NDLTD members will • Share metadata for their ETDs • Providing that in either ETD-MS • Or if they use a version of MARC locally, work to have that eventually shared in either MARC21 or UNIMARC • Run OAI, either locally or in consortia, so their metadata can be harvested, according to necessary terms and conditions Complex to Simple MARC ($50) Dublin Core (DC) + thesis ETD-MS • ETD Metadata Standard • XML-encoded metadata standard (content and encoding) for Electronic Theses and Dissertations (ETDs) • in part conforming to Dublin Core (DC) • using RDF • using UNICODE • Has specified relationship with MARC ETD-MS Schema Includes • Elements not in dces (Dublin Core Element Set) • e.g., thesis.degree • Elements with wildly divergent semantics • e.g., thesis.advisor rather than dc.contributor • Relationships to other elements • Controlled vocabularies • e.g., {Bachelors, Masters, Doctorate, Other} for thesis.degree.level • Labels in multiple languages RDF for ETDs • WWW Consortium (W3C)’s RDF: Resource Description Framework • NUDL ETD metadata realized as an RDF application profile • Specifying elements from DC element set • Plus new elements from a registered ETD schema • Constraints & policies attached to both • (e.g., “Full title,” “Name as it appears on title page,” “Repeatable”) • Links to authority records encoded as URIs • XML syntax as per RDF standard OCLC and ETD-MS • Identify TDs in WorldCat • Through OAI make available metadata for WorldCat TDs in both DC and ETD-MS • Provide an authority service for personal names for NDLTD • Coordinate with other authority services such as LC VTLS and ETD-MS • Support NDLTD through a union catalog service implemented with Virtua • Accept metadata in MARC21 or UNIMARC, and help identify other converters for other types • Accept metadata in one other format, namely ETD-MS, collected using OAI (harvesting) • Accept data in various character sets, with UNICODE preferred, but in some cases the submitter may be required to convert ETD Union Collection (OAI) VIRTUA MARIAN Future: recommender, … Merged Metadata Collection LEGEND OAI Data Provider Virginia Tech ETD Archive Humboldt ETD Archive Duisburg ETD Archive … OAI Service Provider OAI Harvesting Union Catalog Creation Name Authority Service (e.g. OCLC) NDLTD Central VTLS Union Catalog NDLTD Site / Member Librarian Verification / Validation / Enrichment / Maintenance Student Entry OAI Server Local DB MARIAN Union Catalog Virtua MARC DB OAI Harvester Conversion Local Search / Brow se Alternate MARC Transport (ftp?) tapes?) NUDL (www.nudl.org) Int’l Research Support • Networked University Digital Library • Partners: Mexico (Puebla and Monterrey), Germany, Brazil • Problems: Multilingual search, high performance DLs, requirements/usability, … • Start with ETDs, then expand to other student works, portfolios, data sets, (CS) courseware, ... NUDL ETDs Reports Doctoral Masters Bachelors Courseware Demos Computer Science Tech Reports Viz Lab Test Exercises Banks Future Work - 1 of 2 • Working with publishers to increase level of access as much as possible • Interoperability tests among universities and with UMI to provide integrated services • Study with testbed that emerges, to improve information retrieval, browsing, interface, and other types of user support • Evaluation, improving learning experience, spread to worldwide initiative, sustainable support and coordination Future Work - 2 of 2 • Adding services currently prototyped • annotation and SDI (routing) capabilities • Dublin Core metadata, crosswalk to MARC • Adding other services planned • building and using citation database (w. SFX) • implementing plagiarism check (like “SCAM”) • Developing NUDL as a sustainable self governing global institution (w. committees) Overview - NSDL • Purpose: Scholarship / Knowledge Management • Goals: Discovery of Knowledge, Scholarly Communication • Objectives: Learning, Advancing Knowledge, Sharing • Mechanisms: Electronic Publishing, Digital Libraries • Digital Libraries: Content Mgmnt , Workflow Mgmnt • Distributed DLs: Open Archives Initiative, Open DLs • Examples: NDLTD, NSDL, CITIDEL NSDL: DL Case Study • Domain: lifelong (K-gray) education/learning • Genre: learning resources • Submission: http://www.cstc.org, … • Collection: http://www.nsdl.nsf.gov Project: National STEM (science, technology, engineering, and mathematics) education Digital Library NSDL Colls Services Core Integration Research Expectations of NSDL ProgramTracks • Core Integration: coordinate a distributed alliance of resource collection and service providers; and ensure reliable and extensible access to and usability of the resulting network of learning environments and resources • Collections: aggregate and actively manage a subset of the digital library’s content within a coherent theme / specialty • Services: increase the impact, reach, efficiency, and value of the digital library in its fully operational form • Targeted (Applied) Research: have immediate impact on one or more of the other three tracks CI Key Concepts and Principles • NSDL is an “Education Layer” over Web • NSDL will be seen by • Users, as one library with many portals • Builders, as offering a spectrum of interoperability • Major objectives of Core Integration (CI) are to help NSDL successfully achieve • Diverse collections • User interfaces & library services • Educational excellence • Access & content management • Framework for evolution Services • Help services, frequently asked questions, etc. • Synchronous/asynchronous collaborative learning environments using shared resources • Mechanisms for building personal annotated digital information spaces • Reliability testing for applets or other digital learning objects • Audio, image, and video search capability • Metadata system translation • Community feedback mechanisms Collections • • • • Discovery of content Classification and cataloguing Acquisition and/or linking; referencing Disciplinary-based themes define a natural body of content, but other possibilities are also encouraged • Access to massive real-time or archived datasets • Software tool suites for analysis, modeling, simulation, or visualization • Reviewed commentary on learning materials and pedagogy A Learning Environments and Resources Network for STEM Education (LEARNS) “The network is the library.” LEARNS Connects: Users: students, educators, life-long learners Content: structured learning materials; large real-time or archived datasets; audio, images, animations; primary sources; digital learning objects (e.g. applets); interactive (virtual, remote) laboratories; ... Tools: search; refer; validate; integrate; create; customize; publish; share; notify; collaborate; ... LEARNS Supports: Learning communities Users (profiles) Application services Tools Customizable collections Content (metadata) (protocols) LEARNS Enables: Environments for • Discovery • Communication • Stability • Collaboration • Reliability • Creation AND • Reusability • Validation • Interoperability • Evaluation • Customizability • Recognition • ... • ... of Resources Goal Core Integration Track (FY00 pilots, FY01 full) Collections Track Services Track Targeted Research Track LEARNS operational by 2002 NSDL Information Architecture Essentially as developed by the Technical Infrastructure Workgroup Portals & Portals & Clients Portals & Clients Clients User Interfaces Core NSDL “Bus” NSDL NSDL NSDL Collections Collections Collections Collection Building referenced referenced items&& Special items collections Databases collections Core Core Services: Collectionmetadata Building Core gathering CollectionServices protocols Building Services harvesting NSDL NSDL Services Other NSDL Services Services Usage Enhancement Core Services: CI Services information retrieval CI Services browsing CI Services authentication CI Services personalization CI Services discussion annotation Overview - CITIDEL • Purpose: Scholarship / Knowledge Management • Goals: Discovery of Knowledge, Scholarly Communication • Objectives: Learning, Advancing Knowledge, Sharing • Mechanisms: Electronic Publishing, Digital Libraries • Digital Libraries: Content Mgmnt , Workflow Mgmnt • Distributed DLs: Open Archives Initiative, Open DLs • Examples: NDLTD, NSDL, CITIDEL NSDL -> CITIDEL • Computing and Information Technology Interactive Digital Education Library • A collection project in the National STEM (science, technolgy, engineering, and mathematics) education Digital Library – NSDL (www.nsdl.nsf.gov) CITIDEL: DL Case Study • Domain: computing & IT, lifelong (K-gray) education/learning • Genre: learning resources • Submission: http://www.cstc.org, … • Collection: http://www.citidel.org Project: Computing and Information Technology Interactive Digital Educational Library (NSDL collection project) CS Teaching Center (CSTC) • Instead of building large, expensive multimedia packages, that become obsolete and are difficult to re-use, concentrate on small knowledge units. • Learners benefit from having well-crafted modules that have been reviewed and tested. • Use digital libraries to build a powerful base of support for learners, upon which a variety of courses, self-study tutorials & reference resources can be built. • ACM support led to Journal of Educational Resources in Computing (JERIC), accessible from www.cstc.org Browsing (1) Browsing (2) www.CITIDEL.org • Computing and Information Technology Interactive Digital Education Library • Builds on CSTC, JERIC, ACM DL, IEEE DL, ResearchIndex. NCSTRL, … • Led by Virginia Tech, with co-PIs: • Fox (director, DL systems) • Lee (history) • Perez (user interface, Spanish support) • Partners • College of New Jersey (Knox) • Hofstra (Impagliazzo) • Villanova (Cassel) • Penn State (Giles) Overview of CITIDEL architecture USER PORTALS DIGITAL LIBRARY SERVICES REPOSITORIES Multi-dimensional categorization Quality Peer reviewed Editor reviewed Nominated Identified by crawl Algorithms Java English Multimedia Spanish Language Topic Summary of spring 2001 survey of CITIDELrelated collections and their sizes Size of Collection 1-5 items 100-300 Number of Collections Identified 6-100 items 101-999 items +1000 items 50 10-25 20-35 Distributed repository structure Digital Library Services OAI Data Provider Applets Repository OAI Data Harvester Union Metadata Repository Laboratories Repository Syllabi Repository Papers Repository ... Digital library architecture for local and interoperable CITIDEL services EDUCATORS Multilingual Searching LEARNERS Browsing Union Metadata Filtering Filtering Profiles OAI Data Provider Annotating ADMINISTRATORS Revising Administering User Profiles Annotations OAI Data Harvester Remote and Peer Digital Libraries (eg. NSDL -CIS) PORTALS SERVICES REPOSITORIES Summary • Purpose: Scholarship / Knowledge Management • Goals: Discovery of Knowledge, Scholarly Communication • Objectives: Learning, Advancing Knowledge, Sharing • Mechanisms: Electronic Publishing, Digital Libraries • Digital Libraries: Content Mgmnt , Workflow Mgmnt • Distributed DLs: Open Archives Initiative, Open DLs • Examples: NDLTD, NSDL, CITIDEL Highlights • DL-mediated sharing • Students thru universities share theses/dissertations • Faculty thru NSDL share courseware (innovative!), and with learners • Mechanisms • • • • Authors/Creators submitting content, with (initial) metadata Open Archives Initiative for sharing / aggregating metadata Open Digital Libraries (components connected using XOAI) for building DLs Support by NSF, universities, … • Invitation to participate • NDLTD, NSDL, CITIDEL Links • Open Archives Initiative • http://www.openarchives.org • OAI Protocol for Metadata Harvesting • http://www.openarchives.org/OAI/openarchivesprotocol.htm • Virginia Tech DLRL OAI Projects • http://www.dlib.vt.edu/projects/OAI/ • Repository Explorer • http://purl.org/net/oai_explorer • NDLTD • http://www.ndltd.org • NSDL • http://www.nsdl.nsf.gov • CITIDEL • http://www.citidel.org • NCSTRL • http://www.ncstrl.org More Links • ARC Cross-Archive Search Service • http://arc.cs.odu.edu/ • XML Schema Validator • http://www.w3.org/2001/03/webdata/xsv • Dublin Core Metadata Initiative • http://www.dublincore.org • E-Prints DL-in-a-box • http://www.eprints.org • XML Tools at W3C • http://www.w3.org/XML/#software • Fox Homepage • http://fox.cs.vt.edu