From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 1 From Shellac to Audio Layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org Desiree Wallen San Jose State University December 9th, 2013 From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 2 Abstract This work is an examination of the current state of the push to digitize 78 RPM recordings, which are records of some of the earliest recordings in existence, and historically and culturally vital to the development of society. By listing the most disadvantageous issues that arise within various attempted projects, the challenges become goals that need to be overcome within the organizations who want to accomplish this objective. The spotlighted organizations The Library of Congress and archives.org are analyzed for their handling of the stated challenges. From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 3 Introduction The historical and cultural implications of recorded music stem from both the recording’s importance to the moment of its creation (and the people of that moment) and the technology best available at the time of creation that was used to capture the audio. The first explosive rise in recording music came with the advent of disc players that could play a standard disc at 78 revolutions per minute, and appropriately named the 78-RPM record (which is shortened to 78 RPMs throughout the rest of this document). These types of discs contain some of the most important audio recordings of the early twentieth century, and as a result are in high demand to be acquired for digital archival preservation. Since the beginning of the twenty-first century, there have been major technological developments in the field of music-as-archive. Record creation is able to be completed on the personal device (tablet, laptop, even smartphones), so it logically beckons that society requires access to all audio recordings digitally, whether they were created in a digital format or not. This has led to the demand for early twentieth century audio recordings to be digitized, so as to preserve some of these archives on a new medium when the physical record unfortunately yet unavoidably is in danger of deterioration. With any delicate physical medium, there will be challenges during the process of conversion to the digital medium. These can range from issues with the original recording and the original disc to the software being used to remove those issues, from the near-philosophical question of whether altering the recording (even if to improve clarity) detracts from its historical value to ensuring the recording keeps the sense of culture that was present at time of creation (i.e., saving the best recording possible). Finding ways to overcome these challenges could From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 4 require some time, which is another challenge in itself due to the risk that is digitally preserving these types of records in the first place. The resources to accomplish this vast project can come from unlikely sources, as will be focused on in this literature. This intends to analyze two programs that specialize in the digitization of 78 RPMs: the Library of Congress’ National Jukebox and the Audio Archives section of archives.org and interpret how they met the challenges detailed in this document. This is not projected to be a comparison between the two organizations, but rather an individual analysis of each because of their diverse methodologies in conversion. At the basis, these different methodologies stem from the way the resources are provided to these organizations: the Library of Congress is funded through government allocations and donations, as well as given the freedom to use the newest conversion technologies (even if it is off-site), while archives.org relies on the public at large to do the conversions of records found by the public (or the more punctual terms, collectors converting their collectables). It is both practical and opportune to analyze these particular two organizations at this time, because one is indicative of a government-run organization becoming interested in the digital conversion of 78-RPM recordings while the other is an example of using crowdsourcing as a funding and technological resource, with a built-in public interest in the digital conversion project. Therefore, this document aims to compile the many challenges in converting 78-RPMs to the various media formats (the most common being .wav, FLAC, and .mp3) into one place, as well as analyze the potential solutions to these challenges through the lens of two ideal organizations that have become representative of certain methodologies used. No From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 5 experimentation will be conducted, but rather explored through the investigation of previous literature. Literature Review Previous documentation of challenges The compilation of challenges will be selected from previous literature written on it, but they will not be stated in detail here. However, the challenges in the digital conversion of 78 RPMs are numerous. Sergio Canazza, a professor of information engineering at the University of Padova in Italy, specializes in the digital conversion of music and sound from physical mediums, and has written numerous articles on challenges and solutions that arise. In his 2010 article ‘The restoration of low-quality audio recordings based on non-negative matrix factorization and perceptual assessment by means of the ebu mushra test method’, he emphasizes the lack of detection built into conversion software to help the user identify and remove background noise by determining the non-negative matrix value that best shows the least variation in recordings, testing a series of values to define the prominent vocal or instrumental tone, training the software to look for certain qualities of background noise or crackling, tracking the original source of unwanted noise, and isolating and suppressing said unwanted noise. With rigorous testing, the results pointed to the development of a standard algorithm to measure the values of quantifiable background noise. Allegedly, the algorithm method worked better than the other softwares tested in the study, but that emphasizes the difficulty in removing the background noise. Canazza highlights another complication with digitizing shellac-based records such as 78 RPMs in his essay ‘Toward an audio digital library 2.0: Smash, a social music archive of Shellac From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 6 phonographic discs’ by analyzing software that can read the grooves on a record without humans touching the record to gather metadata and further causing risk of deterioration. The ‘limitation’ is that these recordings are not embedded with metadata and such metadata has to be created, and the social media program Smash is spotlighted as something that has the capability to solve this issue as well as minimize physical human interaction with the already at-risk disc. Canazza’s work is essential to the identification of issues that span all mediums of older recordings, but particularly the issues that are pervasive to 78 RPMs and digitizing those records. The case could be argued that the so-called ‘background noise’ in recordings, especially those on 78 RPMs, are part of the historical and cultural context at the time of their recording. Morange, Dubois, and Fontaine conducted a survey to attempt to answer the human ability to differentiate between qualities of sound. The survey measured the trajectory of the vocal quality of Enrico Caruso, an early twentieth-century singer and recording technician, over various remasterings on different formats. Caruso was chosen because his recorded voice was originally captured on his equipment, which was created prior to the development of 78 RPMs, and his recording has been remastered on subsequent phonographic and electronic devices. The creators of the survey selected thirty-two people, with a difference in age and expertise on recordings of the time period in which Caruso recorded. The results of this survey indicated that the older, more contextually aware people were the ones to prefer some background noise. Ridding the recording of the background noise entirely could detract from the factual evidence of the time and place of creation. Regardless that the technology to curtail the potentially harmful handling of 78 RPMs does exist, Imre and Cox’s survey of several research libraries unfortunately shows the inability From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 7 for most organizations to afford such solutions as Canazza had demonstrated. Some of the problems in digitizing were the completeness of the audio catalog, which thirty-eight percent of respondents said was not completed, and the lack of resources for digitization, in which only nineteen percent of respondents considered digitization as a means for preservation. This lack of resources is a problem in smaller facilities because it requires expensive software for which no funding is allocated and requires the time of trained staff, which is unavailable in terms of paid hours or training programs for staff. Bell and Stokes’ chapter in the Jackson book The 21stCentury Black Librarian in America: Issues and Challenges reaffirms these challenges, as well as adding that the servers of most research libraries do not have the capability to host highfunctioning conversion softwares. Without the apt resources, solutions are somewhat futile, which poses another threat. The notion of incomplete audio catalogs is reaffirmed as well by the Lai, Fujinaga, and Leive presentation ‘Metadata for phonograph records: Facilitating new forms of use and access to analog sound recordings’. The cataloguing process is important because without a set, complete standard to aspire to, the digital versions of 78 RPMs would ultimately prove impossible to find and access, rendering the process of digitization nearly unnecessary. Accounts of methodology pertaining to the Library of Congress As one of the primary sources being evaluated, the Library of Congress has a page, including a slideshow, documenting the process by which they have documented the National Jukebox, the digital music collection derived from 78 RPMs that includes 10,000 important recordings from the early days of recorded music selected for preservation via digitization. The From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 8 existence of the documentation of this process is a resource for the general public to understand how the National Jukebox came to be, even if they are not authorized to use the equipment or make the decision of which recordings to select for digitization. Doherty’s review of the National Jukebox, however, contains a brief summarization of how the online collection came to be, from the deal with Sony Records, who had a large portion of the rights. The catalog seems to be the highlight of the entire website, as the reviewer counts several ways in which the recordings can be searched. The two-star rating of the project given by Doherty will not have any bearing the content of the analysis, but is indicative of the impact beyond the organization. Accounts of methodology pertaining to archvies.org Zorker’s blog post from archives.org is another primary source chronicling the methodologies of physical-to-digital LP conversion through the lens of an archival organization. The unique perspective this document provides is that this is not a retrospective document, but an instructional one for the volunteers who are digitizing their own collections. It lists the products necessary for digitization-at-home and warns of taking care of copyright problems before uploading. As with the Library of Congress’ National Jukebox, the journal Choice took on the task of reviewing the digital audio conversion program created by archives.org, naturally called the Audio section. It features commentary on the useful, complex catalog search engine and the interesting mix the collection provides, as well as the offering of a page that shows what recordings were the most popular on the site over the course of the previous week. The audio quality is, according to the review, determined by the quality of the original recording, which presents a problem in archiving such audio collections, and leaving the digitization process of From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 9 such recordings to non-professionals. Again, the rating bestowed by Hogan will not have an effect on the content of the forthcoming analysis of archives.org and the Audio section, but provides an outside perspective. Challenges of conversion Deterioration of physical medium The beginning of the heyday of the 78 RPMs is fast approaching the century mark, and the shellac-based record needs less and less force for breakage as it ages. The cut grooves deliver the sound when a record player needle runs along them. It becomes much easier for the audio to become warped if those grooves deteriorate. According to Canazza and Dattolo in ‘Toward an audio digital library 2.0: Smash, a social music archive of Shellac phonographic discs’, “There are more than 1,000,000 Shellac discs in the worldwide audio archives. They are very important because some of these discs contain music never re-recorded (R&B, Jazz, Ethnic, Western classical, etc.)” (2010, p. 5). Even the article by Imre and Cox mentions that “only 14% of historic sound recordings issued between 1890 and 1964 were legally reissued on CD” (2009, p. 2). With the lack of those recordings being offered on compact disc, which is the most used physical medium for listening to audio, the only current physical access to these historic recordings is by listening to them on the decaying 78 RPMs. Digitization is the seeming solution to ‘save’ these recordings, but the handling and conversion processes used could further damage them. From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 10 Manual metadata creation Unlike compact discs, 78 RPMs do not have the demand for software where metadata is compiled from reading the physical data. When the audio from 78 RPMs is converted into digital files, the information is not read automatically and applied to the data description. This presents a problem due to the potential lack of information about the audio record. The metadata is subjected to whatever information is already known about the recording and what metadata and descriptors were recorded at the time of the record’s creation. The near-century of time passed might certainly have separated some details on the way to current preservation. Just as well, the metadata standards for other records might not adequately capture the extent of the descriptions necessary. As Lai, Fujinaga, and Leive surmise, the “standards are generally limited to bibliographic description of relatively few elements; they have weak relationships between fields (e.g., performers or conductors) describing separate works on the same album; and they make no distinction between publications of the complete work versus parts of the work” (2005, p. 1). The challenge here is to find a balance between developing a maplike standard of metadata while acknowledging potentially incomplete information. The standard should be like a formula that has variables able to be edited and filled in should new information arise. The solution in Lai, Fujinaga, and Leive’s work was to create a standard where “metadata at different levels were created to facilitate the management of a wide variety and combinations of objects (e.g., tracks, discs, performers, recording sessions) that comprise sound recordings” and “to automate the metadata extraction process” (2005, p. 1). The easiest suggested path to fix the issue of nonautomatic metadata generation, then, is to create a standard that creates metadata both moveable with time yet easy to compose. From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 11 Background noise: suffering quality or historical detail? A major issue with the digital conversion of 78 RPMs is the debate over whether remastering should be kept to a minimum or extended, as there are arguments for both sides that claim to be for the ultimate integrity in preserving a recording. The environment (being the synchronization of time period and location) in which these recordings were created occurred prior to such elements as soundproof studios and master mixing, resulting in a pervasive clouding of moving air on most, if not all 78 RPM recordings. This sound on the recordings is reminiscent of television static and can detract from the intended composition of the record. On the other hand, this background noise is representative of the resources the makers of the record had at the time, and therefore can be seen as the most accessible way to understand the environment of creating audio records on 78 RPMs. A survey, conducted by Morange, Dubois, and Fontaine, explores the generational gap between hearing background noise and accepting it as part of the historical value. By using the original recording of Enrico Caruso singing on his own developed physical format, as well as 78 RPMs and subsequent mediums, the respondents are able to hear various remasterings of the recording and answer the question of whether recordings can sound better without background noise or whether background noise is crucial to historical value. The conclusion was that the younger respondents preferred the recordings with the least background noise, and the older respondents preferred the recordings with some background noise (although, significantly, not the original), acknowledging the contextual importance of the background noise being a component of the original recording devices. Their discussion then poses this notion: “The preservation and restoration of musical recordings involve a necessary collaboration of physical From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 12 sciences, signal processing knowledge, and human sciences. The first ones are obviously concerned with the recordings as physical (acoustic) objects, whereas the latter deals with the evaluation of them as cultural (cognitive) objects that may therefore vary across time and across the diversity of individuals who give meanings and evaluation to them” (2010, p. 455). If the definition of ‘background noise’ varies, then, by the perception of the listener, is the preserver alone to decide how much background noise to keep? Is it necessary to find a balance between how much background noise is to be considered overwhelming the information presented in the original recording and how much tells the story of the environment of creation? Resources for digitization While digitization is an ideal way to create a backup for the physical medium, many research libraries and archival repositories lack the resources to attempt the large undertaking of digitizing their 78 RPM collections. Bell and Stokes’ analysis of the digitization project at Hampton University accepts the reality of such a situation. For the digital conversion of audio records, the main resource “lies in the commitments of a firm or institutional budget to support staffing on all levels: cataloging, musical editing, preservation, and information technology” (2012, p. 176). The staffing is an important resource because they are the component that will ensure the process of the conversion goes as accurately as possible. However, such resources as software, hardware, storage, and funding could be considered luxuries for smaller research libraries and archival repositories. The Imre and Cox survey of academic libraries deduced that nearly half of the respondents had collections that were not fully catalogued, and 19% have any kind of digitization program (2009, p. 11). This is likely due to the fact that “without catalog records these items are hidden from our users and with our users’ preference for digital music From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 13 and the inconvenience of turntables, very little future use of these items can be expected” (p. 12). If the repository does not have the resources to even complete cataloguing, it is difficult to incorporate a project such as digital conversion into the budget. What methods could an organization take to utilize as little resources as possible while maximizing those types of goals? Conversion with government resources The Library of Congress’ National Jukebox was an effort by the US’s official library to take 10,000 rare recordings from the Victor Talking Machine Company, which was one of the major companies from 1901 to 1925 to manufacture and record 78 RPMs, and digitize them for listening on the Library of Congress website. The project was fully completed with the current collection in early 2013 (the site launched in 2011) while they actively are looking to partner with more companies that own recordings from that time period. For their resources, they were allocated funding in their annual budget and were able to place staff in an off-site facility called the Packard Campus devoted solely to audio/visual preservation. Staffing was a non-issue because they were able to hire people solely for this project. The selection process was careful, as it involved a handful of curators designated just for the job, and then technicians compared multiple copies and selected the one in the best condition to digitize. Each file was uniquely named using a system designed by the technicians as well. The discs were taken offsite to be cleaned and digitized on a turntable. Hired engineers listen for any extraneous noise that can be removed and adjust pitch for clarity. The files (including images of the original disc and label to accompany the files) are then transferred from the offsite server to the Library of Congress server where they are made searchable on the website (Library of Congress, 2013). From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 14 The Library of Congress is fortunate in overcoming a potential lack of resources due to government-funded allocation. However, this is not indicative of a typical repository, but rather the ideal. The review in Choice explains that this is focused on making these records available and accessible with these resources (particularly the license from Sony Entertainment) (Doherty, 2011, p. 274). The technicians were very careful with the transfer of information and even consulted the Victor sources for help with the thorough creation of metadata. This metadata assists in the making various categories searchable. By being as specific as ‘target audience’ with the metadata, the Library of Congress has set a standard where the record can be easily identified and deemed worthy of preservation in the future. To keep the physical medium safe during the conversion process, the Library of Congress will clean the record with a custom, non-invasive solution. While on the turntable for conversion, the record is kept in a temperature-controlled room with a non-damaging lamp right over the turntable. During the playback process, the staff selects a stylus needle with the closest size to fit the grooves of the record with the least amount of room for error. If the disc itself is scanned, the scanner lid remains open to protect the delicate record. As for resolving the background noise issue, the staff, as experts of both the technological aspect of converting 78 RPMs as well as the historical meaning behind the discs, are relied upon to play the record, convert it into two channels (one for each side of the groove) and compare the results for excess noise. If one channel has more noise than the other visually, then that noise is removed on the software until the audio waves are even on both channels. They make a fairly conscious effort to preserve the integrity of the original recording; the transfer is captured at the From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 15 standard audio file size, 24 bit/96 kHz Broadcast Wave format and is not ‘remastered’ in high definition (Library of Congress, 2013). While the Library of Congress might have the site and the technology to do this undertaking, they still maintain standards to aspire to in the digital conversion of 78 RPMs. They do not attempt to go beyond what they consider preservation or the providing of access. The Library of Congress developed a methodology that keeps the original record safe and welldescribed while keeping the truth of the original in its conversion. Conversion via crowdsourcing The Audio section of archives.org is run by hired staff, but the conversion process is completely maintained by users of the site. The reliance on these donations of collection and labor make up the thousands of recordings on the site. A how-to conversion document by Jordon Zorker chronicles the methodologies of physical-to-digital LP conversion.The main difference is that this is not a retrospective document, but an instructional blog post for the volunteers who are digitizing their own collections. It lists the products necessary for digitization-at-home and warns of taking care of copyright problems before uploading. The sound card listed allegedly doubles as a microphone and the LP is played to a recording Audacity software. This appears to indicate that the initial digital file is limited to the quality of the original physical medium. However, there is a how-to guide on the editing of such music in Audacity, including track divisions so there can be a reduction in unnecessary noise. The process is supremely detailed, but able to be understood by those who are technologically savvy. From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 16 Clearly, the resources of archives.org are philosophical in nature. The opening line of Zorker’s post beckons “Help us preserve and distribute ephemeral culture!” (archives.org). The hardware and software are ultimately supplied by the at-home user, in which it is nearly impossible to tell a standard condition for converting these records. The editing, as well, is completed by the user. The staff are essentially there as verifiers and performing the task of making the converted records accessible. Not that this is inherently bad, but merely risky. There is no special rule or direction for keeping the original record safe. Granted, the user/owner of the original 78 RPM would likely know how to keep it preserved and convert it without damage. Unfortunately, there is no documented guide from the archives.org organization, which is troublesome if someone is new to this (which the Zorker document claims the how-to is for). As for metadata, that is under the jurisdiction of the staff, and they appear to have searchable metadata and the post does ask for either a scan of the original disc cover or to save each file as specifically as possible. This is one of the challenges they better meet in practice. There is no editing of background noise listed in the responsibilities of the user, which indicates that the record being digitized is as close to the original recording as possible, and as stated by Hogan in his review of the site, “limited by the fidelity of the original” (2012, p. 441). The troubling aspect is the conversion process, which consists of the use of an external sound card and a microphone to basically re-record the playback, which if done in a non-soundproof environment, has potentially dangerous consequences of adding additional background noise, in which the historical context could be muddled. From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 17 This concept is philosophically ideal because it entails the sharing of privately-owned information with the public through digital conversion. The program could use somewhat more monitoring, however, due to the immense amount of trust that every record is in a safe environment and that every digitally converted recording is untainted. The amount of sortable metatdata is astounding and delivers a lot of information and directs users to exactly what they are seeking. Conclusion The digital conversion of 78 RPMs is a project of immense importance that will bring the origins of recorded music into the 21st century concept of technological storage as the physical mediums are in more danger of deterioration as time continues. This type of project brings with it a great amount of challenges to overcome: the potential destruction of the disc during conversion, the correct and thorough execution of metadata input, finding the correct balance between eliminating excess background noise and keeping cultural significance, and maintaining the resources needed for this enormous undertaking. While no program is able to overcome them all, the ones that are implementing such a project have their own solutions. The selection of the Library of Congress and archives.org was a showcasing of two philosophical ideals: the use of government resources to ensure the best methodologies possible, and the use of crowdsourcing to obtain information for the peers of the digital conversion creators. The acknowledgment of such programs can help other programs to develop and improve upon them, where the benefit of such analysis is passed onto the preservation of these vital audio recordings. From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 18 Reference List Bell, G. S., & Stokes, H. J. (2012). Advancing digital resources from the black musical experience: An archival and digital challenge at Hampton University. In A.P. Jackson, J. Jefferson, A. Nosakhere (Eds.), The 21st-Century Black Librarian in America: Issues and Challenges (173-176). New York: Scarecrow Press. Burnett, G. (2009). Colliding norms, community, and the place of online information: The case of archive.org. Library Trends, 57(4), 694-710. doi:10.1353/lib.0.0057 Cabras, G., Canazza, S., Montessoro, P., & Rinaldo, R. (2010). The restoration of low-quality audio recordings based on non-negative matrix factorization and perceptual assessment by means of the ebu mushra test method. Proceedings of the Second Workshop on eHeritage and Digital Art Preservation, Taipei, Taiwan. 19-24. doi:10.1145/1877922.1877930 Canazza, S., & Dattolo, A. (2010). Toward an audio digital library 2.0: Smash, a social music archive of Shellac phonographic discs. Proceedings of the 6th Italian Research Conference of Digital Libraries, Padua, Italy. 205-217. doi:10.1007/978-3-642-158506_20 Doherty, B. (2011). National jukebox: Historical recordings from the Library of Congress. Choice: Current Reviews for Academic Libraries, 49(2), 274. Retrieved from http://go.galegroup.com/ps/i.do?id=GALE%7CA269337735&v=2.1&u=csusj&it=r&p=L itRC&sw=w&asid=2ce2a718c97aa407f46661001b08a133 Hogan, W. P. (2012). Audio archive. Choice: Current Reviews for Academic Libraries, 50(3), 441. Retrieved from From shellac to audio layers: The challenges of converting 78 RPM records and the steps taken by the Library of Congress and archives.org 19 http://go.galegroup.com/ps/i.do?id=GALE|A307525992&v=2.1&u=csusj&it=r&p=LitRC &sw=w&asid=0ea14598d7fee8489713a2544f8b1818 Imre, A. & Cox, E. J. (2009). Are we on the right track?: Issues with LP record collections in U.S. academic libraries. Notes, 65(3), 475-476-486. doi:10.1353/not.0.0116 Lai, C., Fujinaga, I., & Leive, C. (2005). Metadata for phonograph records: Facilitating new forms of use and access to analog sound recordings. Proceedings of the 5th ACM/IEEECS Joint Conference on Digital Libraries, Denver, Colorado. 385. doi:10.1145/1065385.1065488 Library of Congress. (2013). Making the national jukebox. Retrieved November 4, 2013, from http://www.loc.gov/jukebox/about/making-the-jukebox Morange, S., Dubois, D., & Fontaine, J. (2010). Perception of recorded singing voice quality and expertise: Cognitive linguistics and acoustic approaches. Journal of Voice, 24(4), 450457. doi:10.1016/j.jvoice.2008.08.006 Zorker, J. (2008). How to digitize an LP. Retrieved November 4, 2013, from http://blog.archive.org/2008/06/19/how-to-digitize-a-lp/