Hourly Sea Level Data Processing and Quality Control Software: Update for 64-bit Microsoft Operating Systems SLP64 User Manual (Version 4.0) Patrick Caldwell Joint Archive for Sea Level of the National Oceanographic Data Center and University of Hawaii Sea Level Center October 2014 JIMAR CONTRIBUTION NO. 14-389 SLP64 USER MANUAL Abstract This software was originally developed in 1988 for capacity building of hydrographic agencies that collect and process tidal (sea level) data. The package was designed for IBMcompatible personal computers since they were inexpensive and readily available. It was referred to as SLPRC, which was the name of the primary working directory. An upgrade to the package was made in 1998 for the year 2000 issues, referred to as SLPR2 (new directory name). The original Fortran software was compiled under 16 or 32-bit DOS operating systems. When the 64-bit operating systems arrived, the programs from the original package would not execute. This required a re-compilation. The plotting libraries of the original set are not available for 64bit systems. Thus, the new plotting software uses Python/Matplotlib. The updated version is referred to as SLP64 (new directory name) in order to clearly distinguishes the 64-bit compatibility. Additional improvements were made to several routines moving from SLPR2 to SLP64. The working data file format remains the same, which is referred to as the SLP64 processing format, though also known by other names, such as the SLPR2, TOGA, or the University of Hawaii Sea Level Center (UHSLC) processing format. This format is kept because it is easier for the quality control steps that require viewing the observed and residual files in text editors simultaneously, an essential component to processing. The format can readily be changed to a more contemporary format where each row or line has date, time, and a single sea level value or missing data flag. This manual serves as a user's guide for processing and quality control of hourly sea level data. The goal is to prepare a scientifically-valid set for enhancing the application of the data. The package was assembled and documented by the Joint Archive for Sea Level (JASL), a collaboration between the UHSLC and the National Oceanographic Data Center (NODC), a branch of the National Oceanic and Atmospheric Administration (NOAA). The software includes the Canadian Institute for Ocean Sciences Tidal Package for tidal analysis and prediction routines. The tidal predictions should be applied to quality control. The tidal predictions should not be used for nautical or coastal engineering purposes. ii SLP64 USER MANUAL Table of Contents 1 Introduction and Setup ................................................................................................................ 1-1 1.1 Important Points .......................................................................................................... 1-2 1.2 System Requirements .................................................................................................. 1-3 1.3 Installation and Software Organization ....................................................................... 1-3 1.3.1 Python .............................................................................................................. 1-3 1.3.2 SLP64 .............................................................................................................. 1-3 1.4 Station Information Setup ........................................................................................... 1-4 1.5 Filename Conventions and Data Formats ................................................................... 1-4 1.5.1 Files with Hourly Sampling Intervals Grouped by Year ................................. 1-5 1.5.2 Files with Hourly or Daily Sampling Intervals Grouped by Month ................ 1-6 1.5.3 Files of Daily and Monthly Means .................................................................. 1-7 2 3 4 Utilities ............................................................................................................................................. 2-1 2.1 Convert to SLP64 Format for Observed Hourly Data ................................................. 2-1 2.2 Convert Units .............................................................................................................. 2-3 2.3 List Missing Data ........................................................................................................ 2-4 2.4 Make Blocks of Missing Data Flags ........................................................................... 2-4 2.5 Shift Hourly Data in Time by Increments of An Hour................................................ 2-5 2.6 Convert SLP64 Format to CSV ................................................................................... 2-6 2.7 Add Constant to Each Hourly Value ........................................................................... 2-6 Plotting Routines ........................................................................................................................... 3-1 3.1 Hourly Files Grouped by Year .................................................................................... 3-1 3.2 Expanded Residual Plot............................................................................................... 3-2 3.3 Scatter Diagram for Tide Staff-Gauge Pairs ............................................................... 3-2 3.4 Daily and Monthly Data Plots ..................................................................................... 3-2 Tidal Analysis and Prediction .................................................................................................... 4-1 4.1 Tidal Analysis.............................................................................................................. 4-1 4.2 Tidal Prediction ........................................................................................................... 4-4 iii SLP64 USER MANUAL 5 6 7 Quality Control .............................................................................................................................. 5-1 5.1 Hourly Residuals ......................................................................................................... 5-1 5.2 Reference Level Stability ............................................................................................ 5-2 5.3 Timing Errors .............................................................................................................. 5-6 5.4 Short Gaps and Data Spikes ........................................................................................ 5-8 Filtering ........................................................................................................................................... 6-1 6.1 The Filter ..................................................................................................................... 6-1 6.2 Running the Filter Program ......................................................................................... 6-2 6.3 Examine the Output ..................................................................................................... 6-2 Final Comment .............................................................................................................................. 7-1 7.1 Acknowledgements ..................................................................................................... 7-1 7.2 References ................................................................................................................... 7-2 Appendices ................................................................................................................................. A-1 A Summary of Upgrades from SLPR2 to SLP64 .......................................................... A-1 B Access to Package ...................................................................................................... B-1 C Directory Tree ............................................................................................................ C-1 D Formats ....................................................................................................................... D-1 E Station Information File ..............................................................................................E-1 F Notes on Foreman Tidal Analysis Program ................................................................ F-1 G Harmonic Constants: Analysis Output ....................................................................... G-1 H Make Predictions other than Hourly .......................................................................... H-1 I Test for Magnitude of Timing Error............................................................................. I-1 J Figures ......................................................................................................................... J-1 iv SLP64 USER MANUAL 1 Introduction and Setup The Joint Archive for Sea Level (JASL) has prepared a user-friendly, interactive software package for hourly sea level data processing and quality control under Microsoft (MS) operating systems. The JASL software utilizes the Canadian Institute for Ocean Sciences (IOS) Tidal Package. The IOS tidal analysis and prediction routines were written in Fortran by Dr. Mike Foreman. See http://www.pac.dfo-mpo.gc.ca/science/oceans/tidal-marees/index-eng.html for more details. The package includes three principle tasks: 1) tidal analysis and prediction, 2) quality control, and 3) filtering hourly into daily and monthly values. The software is geared to those with prior knowledge of computer basics as well as a general understanding of sea level processing. The goal is to share software on a commonly available platform (the personal computer) that will enhance the quality of sea level data sets. This in turn will promote the growth of the international archives of sea level, thus, providing scientists with greater opportunities to understand the ocean. JASL encourages contributions to its expanding global archive (Caldwell and Merrifield, 2013). This package is derived from existing routines that were commonly used for data processing at the Tropical Ocean Global Atmosphere (TOGA) Sea Level Center (TSLC), under direction of Dr. Klaus Wyrtki (International TOGA Project Office, 1992). Mr. Bernie Kilonsky and Mrs. Shikiko Nakahara wrote the earlier versions of some core routines, many of which date back to the 1970s under the North Pacific Experiment (NORPAX) (Wyrtki, 1979). Under the TOGA program, major efforts were made to centralize hourly sea level for a scientific archive. Mr. Patrick Caldwell was hired by the NODC in 1987 to support this task through the JASL. With the goal of transferring the latest information technology and enhancing the quality of data for various agencies in South and Central America, Mr. Caldwell put together the first public domain package in 1988 (Version 1.0). In 1990, the experts of Intergovernmental Oceanographic Commission (IOC) Global Sea Level Observing System (GLOSS) suggested improvements for the first version, resulting in a more complete second version, which was finalized in 1991 (Version 2.0), with copies available in English and Spanish. Over 150 copies have been distributed and the package has been demonstrated at several GLOSS technical workshops (UNESCO, 1993 and 1995). The TOGA program ended in 1995 but the JASL program continued at the UHSLC. Under UHSLC guidance, modifications to the 1991 package have been made to comply with the Year 2000 (Version 3.0, 1998). Since 1998, the software has been distributed through FTP and the Internet to a large user base. Versions 1-3 of the JASL software package were compiled under 16- or 32-bit MS operating systems. In recent years, 64-bit operating systems are the standard. Since the original plotting routines used PLOT88 libraries, which are unavailable for 64-bit systems, a new method 1-1 SLP64 USER MANUAL of plotting was necessary. The solution was to adapt Python/Matplotlib, which is open source software. The new version of the JASL sea level processing and quality control package is Version 4.0 and designated “SLP64”. While updating the plotting routines, other improvements were made to SLPR2 routines. A summary of changes from Version 3.0 (SLPR2) to Version 4.0 (SLP64) is given in Appendix A. 1.1 Important Points The package has been designed to work with hourly data sets. Data stored at higher sampling intervals must be filtered to the hourly level and placed in the SLP64 data processing format. The choice of filtering method is left up to the individual. At the UHSLC, we have used a three-point Hanning filter centered on the hour with weights of 0.25, 0.50, and 0.25, respectively. Utilities are provided to facilitate the conversion of originator’s hourly data files into SLP64 format. This package contains quality control procedures which require modification of data; therefore, make copies of all original data prior to application of this software. Store the original data in a safe location. Data files provided with this package and examples in this manual are partially fabricated. These files were purposely altered to show examples of data errors. Do not use the sample data for scientific or engineering purposes. An option is available for the creation of tide tables (times and heights of low and high tides). These are to be used as a general reference only. Do not use these for navigational purposes. For official nautical tidal predictions, please refer to the National Ocean Service (NOS) of the National Oceanic and Atmospheric Administration (NOAA). Python, an open source programming language, is required to run some SLP64 routines. The user must ensure Python is properly set up within the MS operating system. Matplotlib can be acquired and set up concurrently with Python, since it is part of the overall Python package. Users should be familiar with basic computer concepts, such as directories (folders) and files. We suggest operating the package via Windows Explorer. A text editor is necessary for some steps. MS Notepad is sufficient though there are other editors. The software in this package contains some internal consistency checks for the interactive input, though it is impossible to predict all possible erroneous input. The user is expected to provide input data files in the correct format, and to answer all interactive prompts correctly. The programs will respond to incorrect answers and formats in an unpredictable 1-2 SLP64 USER MANUAL fashion, including occasionally aborting the program. If you are having trouble with the package, first check the correctness of your formats and interactive responses. Ensure that output files from previous runs of the same program do not exist. If so, delete them. Then execute the program again. One can experiment with the sample data files provided under \SLP64\SAMP. The Fortran programs were compiled using “gfortran –static filename.for –o filename.exe”, where filename is the Fortran source code file. One undesirable characteristic of Fortran is its inability to overwrite existing files. Thus, if a program must be run more than once, be sure to eliminate the output files from the prior attempt. This is the most common cause for programs to crash. Some of the new Python programs take care of this problem, but others do not. 1.2 System Requirements This package has been developed for use on computers with Microsoft 64-bit operating systems. It has been successfully tested under Windows 7 and 8. 1.3 Installation and Software Organization 1.3.1 Python Python can be freely downloaded from https://www.python.org/. Code within SLP64 was prepared in 2014 with Python 2.7.6. It is stated on the Python “About” web tab, “Python is developed under an OSI-approved open source license, making it freely usable and distributable, even for commercial use. Python's license is administered by the Python Software Foundation. If one chooses not to download the entire Python package, ensure Matplotlib is one of the selected module options. 1.3.2 SLP64 1. Download the zipped package “slp64.zip” (Appendix B), which will place a copy of the file in your DOWNLOAD folder. 2. Double click on SLP64.ZIP, which takes you to a WinZip utility. Choose to extract the directories and files into the root directory of the desired hard disk or disk partition. The root drive can have any letter designation (default hard drive for most computers is C:\). 1-3 SLP64 USER MANUAL The software is to be operated within the \SLP64 subdirectory off the root directory. Windows Explorer is the recommended utility for viewing and operating the package. SLP64 subdirectories and programs have been designed to enhance organization and facilitate file management by utilizing a fixed directory structure (Appendix C). Note, one must place your original data files into the data directory, \SLP64\DAT. 1.4 Station Information Setup The first step after loading the software is to set up the STAtion INFOrmation Data INput (STAINFO.DIN) file (Appendix E). One must assign a three-digit number to each unique tide station, along with other information about the station. Ensure columns of each parameter line up exactly. Fortran programs read this file. Fortran requires fixed column positions of fields. The STAINFO.DIN provided with SLP64 has stations and number assignments that exist at the UHSLC. Users of SLP64 have the liberty to change or remove any of the stations in the UHSLC list to cater to your own set of stations. A summary of the fields in STAINFO.DIN is given in Appendix E. Mandatory fields are the Station Number and Name, Latitude, and Longitude. The following field is the time-of-day schema, with 0 denoting hours 1-24 (British system) and 1 denoting hours 0-23. This should be fixed as 1. The remaining fields can be left blank. 1.5 Filename Conventions and Data Formats The file naming convention for SLP64 data files conforms to old standards; that is, 8 characters for the root of the name and 3 characters for the file extension. All formats are ASCII and can be viewed with a text editor (Appendix D). Within this package, the format is referred to as the SLP64 processing format. All hourly sea level observations and residuals files are kept in subdirectory \SLP64\DAT. It also holds the daily and monthly data files. The predicted tides are placed in subdirectory \SLP64\PRD. Calibrations files are maintained in \SLP64\CAL. 1-4 SLP64 USER MANUAL 1.5.1 Files with Hourly Sampling Intervals Grouped by Year This format was chosen because it makes examination of the data by eye within an editor easier. There is a utility under \SLP64\UTIL to convert from SLP64 hourly format to a simple format (each row with one hour of tide data) in comma-separated version (CSV) format, which is compatible with spreadsheets and can be viewed with a text editor. The observed hourly data, the predicted tides, and the residuals (defined as observed data minus predicted tides) have a similar SLP64 format and file naming convention. Each file consists of a year of values at an hourly sampling interval. The SLP64 processing format of the OBSERVED HOURLY DATA is described in Appendix D and the file names have the following form (can be capital or small letters): CVSSSYY.DAT C: century (U:1800-1899, V:1900-1999, W:2000-2099) V: file version (letters A-Z) SSS: station number YY: the last two digits of the year For example, VA00385.DAT is the observed hourly data file (version A) for station 003 and year 1985. WA00385.DAT would be the same for year 2085. The file version is useful for keeping track of modifications to a data file. For instance, the original data could be named version A. Then, after editing and interpolation, the modified version could be called version B, and so on. A useful convention at the UHSLC is to rename the "final" data as file version "Z" to denote that the data have received as much processing as possible. A routine is available (Section 2.1) to facilitate conversion of observed data into the SLP64 observed data format. The PREDICTED TIDES files have the same format as the observed hourly data files, except columns 7-9 of the monthly headers are filled with the letters "PRD". As mentioned in the previous footnote, each predicted tides file has suffixed one day from the previous year and has appended one day from the following year. This is to accommodate some routines which require time shifts. The file naming convention for the predicted tides is as follows: CVSSSYY.DAT C: century (O:1800-1899, P:1900-1999, Q:2000-2099) V: file version (letters A-Z) SSS: station number YY: the last two digits of the year 1-5 SLP64 USER MANUAL For example, PA00385.DAT is the predicted tides file (version A) for station 003 and year 1985. QA00385.DAT would be the same for year 2085. The file version for the predicted files can be used to keep track of which version of harmonic constants was used, as discussed in Section 4.2. Versions of harmonic constants depend on the time span which was input into the tidal analysis routine. The RESIDUALS FILES also have the same format as the observed hourly data files, except columns 7-9 of the monthly headers are filled with the letters "RES". The first letter of the file name is always "R" and the remaining letters are the same as the observed data file (file version, station number, and two-byte year). For example, RA00385.DAT is the residuals file (version A) for station 003 and year 1985. RA00385.DAT would also be the file name for year 2085. A convention to denote the century was not incorporated because the residuals files are temporary and one normally only works with a few years at a time during quality control, so the need for multiple centuries would not arise. The file version for the residuals file normally mirrors the file version of the observed hourly data file from which it was computed. For example, the observed hourly data file, VA00385.DAT, would have a corresponding residuals file, RA00385.DAT. 1.5.2 Files with Hourly or Daily Sampling Intervals Grouped by Month As will be described in Section 5.2 on reference level quality control, monthly files can be maintained for preliminary incoming data. The file naming convention is as follows: VSSSYYMM.DAT V: fixed (always "V") SSS: station number YY: the last two digits of the year MM: month For example, V0038807.DAT denotes the file for station 003 during July of 1988. Because these files are temporary, a convention to account for the next century was not required. Related with the monthly file is the PAIR file (Section 5.2) as described in Appendix D.4. It has a file name the same as the monthly file above, except the first letter is an "X". For example, X0038807.DAT denotes the file for station 003 during July of 1988. The PAIR file is maintained in subdirectory \SLP64\CAL. 1-6 SLP64 USER MANUAL 1.5.3 Files of Daily and Monthly Means Daily and monthly means are obtained through use of a sophisticated filter as described in Section 6. All years for a given site are kept in a single file per type (daily or monthly). The daily data file has an ASCII format (Appendix D.2) and a file naming convention as follows: DVSSS.DAT D: fixed (always "D") V: file version (letters A-Z) SSS: station number For example, DA003.DAT is the daily data file for station 003, version A. The monthly data file has an ASCII format (Appendix D.3) and a file naming convention as follows: MVSSS.DAT M: fixed (always "M") V: file version (letters A-Z) SSS: station number For example, MA003.DAT is the monthly data file for station 003, version A. The daily and monthly SLP64 formats are awkward. To make a more convenient format that is compatible with spreadsheets, Matlab, etc., see programs dsl2csv and msl2csv in \SLP64\UTIL. 1-7 SLP64 USER MANUAL 2 Utilities Several utilities are available for a variety of applications. The programs can be found under subdirectory \SLP64\UTIL. One can either use Windows Explorer and click on the desired program, or open a Command Prompt window, cd \SLP64\UTIL, and type the name of the desired program for execution. 2.1 Convert to SLP64 Format for Observed Hourly Data The package has been designed to work with hourly data sets in the SLP64 processing format. If originator data are stored at higher sampling intervals, they must be filtered to the hourly intervals. The choice of filtering method is left up to the individual. At the UHSLC, we have used a three- point Hanning filter centered on the hour with weights of 0.25, 0.50, and 0.25, respectively. SLP64 provides a Python program, \SLP64\UTIL\convert.py to generate the SLP64 hourly data file format. This may involve a few steps. The goal is to change the original format to a simple format with each row in the file containing a single hourly sea level value along with the Date and Time. The software to generate this simple format is not provided and must be written by the user. Examples of the simple format are given below. Once a simple Date-Time Hourly-value format is created, there is an additional requirement. The positions of each field must line up vertically. If the format is an output from a spreadsheet (comma-separated version, CSV), and the fields do not line up, then there is a utility to make the fields align. This is program \SLP64\UTIL\csv_to_fixed_columns.py. As an example of CSV formats, the UHSLC provides this option. Go to web site http://uhslc.soest.hawaii.edu/ and click on the Data tab. Under Download, click on either Fast Delivery or Research Quality. Choose a station and click on hourly under the 2nd to last column labelled CSV. Save the data file to your computer and place the file within \SLP64\DAT. Open the file with Wordpad (in Windows Explorer, right click the filename, then choose open with, and then select Wordpad). Wordpad is chosen since UHSLC provides data with Unix end-of-line delimiters. If Notepad is chosen, since it assumes MS end-of-line delimiters, all rows wrap onto a single line for the entire file. The top rows in the file look like: 2001,12,16,6,1333 2001,12,16,7,1130 2001,12,16,8,867 2001,12,16,9,553 2-1 SLP64 USER MANUAL 2001,12,16,10,308 The data are year, month, day, hour, sea-level-value (in time zone GMT and units of mm). When the month, day, or hour value is 1-9, it is only a single character. Also, the number of characters of the sea-level-value varies. For SLP64 convert.py to run, the columns need to be fixed for each field. Thus, one can use csv_to_fixed_columns.py to prepare the file for convert.py. A sample of the fixed-column CSV format (output of csv_to_fixed_columns.py): 2001,12,16,06,1333 2001,12,16,07,1130 2001,12,16,08,0867 2001,12,16,09,0553 2001,12,16,10,0308 Another example of an hourly format with Date-Time Value per row (or line) is the NOAA National Ocean Service format for hourly data. A sample is found in \SLP64\SAMP\kwaj2014.txt, which is Kwajalein Atoll hourly data for 2014. Here are a few rows 2014/01/01 2014/01/01 2014/01/01 2014/01/01 2014/01/01 00:00 1.114 01:00 1.53 02:00 1.96 03:00 2.299 04:00 2.463 - 1.269 1.671 2.09 2.548 Fields are year/month/day hour:min predicted-tide dash observed-tide. It is the last column with the observed hourly data that is desired. It is not fixed-column and needs to be corrected. Not clear above is the fact that tab characters separate some columns. If no data exists for the given Date-Time, then the missing flag is a dash. This complicated format is prepared for convert.py using the utility nos_to_fixed_columns.py. This example is chosen to encourage use of this Python program as a guideline to cater your own Python code to your situation. The code in the Python program has lots of explanations to help facilitate the modification of your own situation. Output of nos_to_fixed_columns.py can be viewed at \SLP64\SAMP\kwaj14fix.txt, which would become the input to convert.py. Prior to running convert.py, one must define the start and end columns of each field with file DTDCNV.DIN in directory \SLP64\DIN. When convert.py is executed, there are comments regarding options for the input file sea level value: 2-2 SLP64 USER MANUAL Code for sea level data format and missing data flag, if used 1: 5-digits in mm, range -9999 to 9998, flag 9999 e.g. -1223 or 3214 (read in fortran F5.0 or I5) 2: 6-positions in cm, range -999.9 to 999.8, flag 999.9 e.g. -122.3 or 321.4 (read in fortran F6.1) 3: 7-positions in m, range -99.999 to 99.998, flag 99.999 e.g. -12.823 or 3.214 (read in fortran F7.3) 4: 5-positions in m, range -.999 to 9.998, flag 9.999 e.g. 2.823 or 3.214 (read in fortran F5.3) For each of these options, there are sample files and pre-defined DTDCNV files given in \SLP64\DIN. See the 1readme.txt file for more notes. This gives the user the option to mirror your originator format into one of these 4 options. Additional comments about convert.py: The dates must be numeric. The year must be complete (4 characters, e.g. 1998). Date can be yyyy mm dd hh or any combination, for example, hh mm dd yyyy Delimiter positions between the dates or values can be any character (comma, space,etc.) This input file must not have headers. Multiple years within the input file are acceptable. If missing data, then two options: a) leave out record b) missing data flag Output in millimeters (mm) and placed in \SLP64\DAT Convert.py reads \SLP64\DIN\DTDCNV.DIN so if you want to use one of the 4 options defined in DTDCNV.OP[1-4], then copy the desired one to DTDCNV.DIN 2.2 Convert Units It is recommended to use millimeters as the scientific units while processing data with SLP64. However, one can change to other units if desired. The options allowed by the SLP64 formats are: 1. feet (in hundredths, e.g. UHSLC format 132 = 1.32 feet) 2. millimeters (in whole digits, 1141 = 1141.0 mm) 3. centimeters (in whole digits, 556 = 556.0 cm) One can convert among these three forms using the program, CHUNIT. To run the program, go to directory \SLP64\UTIL with Windows Explorer and click on CHUNIT (Optionally, one could open a Command Prompt window, cd \SLP64\UTIL, and type CHUNIT). 2-3 SLP64 USER MANUAL The program is interactive- you will be prompted to supply information about the input and output files and other options. 2.3 List Missing Data As described in Section 4, the next step in SLP64 processing will be to choose a time span of observed hourly data for tidal analysis. The output from GAPCOU.EXE is helpful for this task since one prefers a time span of minimal missing data values (gaps). One can obtain a list of gaps in an observed hourly data file using program GAPCOU on directory \SLP64\UTIL. To run the program, execute GAPCOU The program is interactive and prompts the user for file name attributes. The output is placed in file MISsssv.TXT (where sss is the station number and v is the version) in the same directory from which the program was executed. This utility is also useful for identifying short gaps (see Section 5.4) and for maintaining a log of missing data. 2.4 Make Blocks of Missing Data Flags Some programs, such as the time shifting program (Section 2.5) and the filtering program (Section 6) require the hourly data files to have either data or missing data flags for each hour of the year. To facilitate the creating of missing data flags, one can use the program FILLMONTHS.BAT. To run the interactive program, go to directory \SLP64\UTIL and execute: FILLMONTHS The output is placed in the directory \SLP64\DAT within file, FILL.DAT. One must then use an editor to paste the missing data flags into the appropriate place within the hourly data file. The batch job FILLMONTHS.BAT is used to run Fortran program FILLVM. The batch job ensures that if FILL.DAT already exists in \SLP64\DAT, then it is deleted before executing FILLVM. 2-4 SLP64 USER MANUAL 2.5 Shift Hourly Data in Time by Increments of An Hour The hourly data files can be moved forward or backward in time (maximum of 12 hours in each direction) by exact increments of an hour using the utility, TSALL. This program is applied to change the time zone reference of the data or to correct timing errors, as will be discussed in Section 5.3. This package does NOT provide a utility to shift the data by fractional hours in time. For example one can NOT shift in time by 15 minutes. If a time zone is not exactly on the hour, for example, locations in India have half-hourly zones, one cannot use this program to shift the data to GMT. The program can run on multiple consecutive years. However, one must ensure that each observed hourly data file has either a data flag or a data value for each hour of each year. To create blocks of missing data flags, see utility FILLMONTHS (Section 2.4). To run this interactive program, execute: TSALL The following prompts will be given: File Version. Give the version of the input observed data file. For multiple years, each file must have the same version. Station Number. Give the station number; for example, 003. Start Year. Provide complete 4-byte year; e.g. 1998. Number of Years. Give number of years. Shift Hours. Supply number of hours to shift (from -12 to +12). File Version of Output. Give the file version for the output file, which must be different from the input file version. Be sure a file version for this station and year does not already exist. Otherwise, the program will abort. If so, remove or rename those output versions or rerun TSALL and select a different file version. Update Monthly Header in Output. If the time zone is being changed, choose option N then provide the new time zone, which will be automatically placed in the monthly headers of the output files. If one is using TSALL to correct a short-lived timing error, then choose option Y, since one does not need to update the headers of the output. Obtain Extra Hours from Other Years. If you are shifting the data forward (positive), if the data start in January, and if December of the previous year is available (and the file has the same file version), one has the option to grab hours from this file so the output will not have missing data flags for the first few shifted hours. Reply Y or N. Conversely, if the data are shifted backwards (negative), if the last year ends in December, and if January of the following year is available (and has the same file version), then one can optionally grab the first hours off the following year. The output is placed in subdirectory \SLP64\DAT. 2-5 SLP64 USER MANUAL 2.6 Convert SLP64 Format to CSV Most contemporary data analysis is performed with packages such as MS Excel or Matlab. When SLP64 processing is complete, the users will likely want to convert the format. To facilitate this task, there are programs for the hourly, daily, and monthly SLP64 format conversions as HSL2CSV, DSL2CSV, AND MSL2CSV, respectively. 2.7 Add Constant to Each Hourly Value In some cases, one would like to add a constant level adjustment to all hours within a given series. This is possible with program ADDVAL. 2-6 SLP64 USER MANUAL 3 Plotting Routines Plotting programs supplement many of the core functions in this package. The routines are introduced here, but one will refer back to them many times in the sections that follow. The plotting programs are found in subdirectory \SLP64\PYPLOT. The plotting routines are all coded in Python/Matplotlib. It is assumed the input files have scientific units of millimeters. Output figures are placed in \SLP64\PYPLOT\PLOTS with a root name based on the input data file and file extension JPG (jpeg image). 3.1 Hourly Files Grouped by Year One of the first tasks after conversion of your local format to the SLP64 format is to plot the observed hourly data. The hourly data plot programs, HPLOT.PY and HPLOT_YRS.PY, are the most commonly used programs of this package and supplements the tidal analysis and prediction routines, as well as the quality control routines. These programs can plot either the observed hourly values or the residuals. To run this program, execute: HPLOT Carefully answer the interactive input options. The input data files are read from \SLP64\DAT. One can either read files in the SLP64 processing format (*.DAT files) or in CSV format (*.CSV). The CSV files must have fields as follows: year, month, day, hour, value. HPLOT plots one year of hourly data for one station at a time. The output graph appears on the computer screen and is saved in \SLP64\PYPLOT\PLOTS. Alternatively, one can run HPLOTS_YRS, which allows multiple consecutive years to be plotted. In this case, graphs are not sent to the computer screen. The output figures are placed in in \SLP64\PYPLOT\PLOTS. If one desires to plot a year of predicted tides: 1. Copy the data file from \SLP64\PRD to \SLP64\DAT. 2. Edit the file and take off the additional days at the top and bottom of the file that occur before and after the pertinent year. Make sure you do not take off the last two delimiter rows at the end of December (all 9s). 3. Use HPLOT (it does not work for HPLOT_YRS). 3-1 SLP64 USER MANUAL 3.2 Expanded Residual Plot SLPR2 provided a plot program to expand a section of time of the residuals to look more closely at details. This program is not provided with SLP64, since the figure that is sent to the screen with HPLOT allows the user to zoom. 3.3 Scatter Diagram for Tide Staff-Gauge Pairs A basic scatter diagram of the tide staff and tide gauge pairs (Appendix D.4) (found in \SLP64\CAL) can be plotted by typing: SCAT Further details are described in \SLP64\CAL\1readme.txt as well as in Section 5.2. 3.4 Daily and Monthly Data Plots The daily and monthly plot programs are DPLOT and MPLOT, respectively, and work similarly. Both are interactive and require information regarding file version and station number. As with HPLOT, the user can input SLP64 processing format (*.DAT) or CSV if fields are year, month, day, daily-value or year, month, monthly-value. These programs allow one to plot all years in the daily or monthly data files, to plot select years, or to plot a single year, which is the smallest time increment for plotting. Once successfully executed, a figure is sent to the computer screen and saved to a file within \SLP64\PYPLOT\PLOTS. The filename is based on the input data file name. There are new programs with SLP64 that did not exist with SLPR2. These are D2DIFF and M2DIFF, for daily and monthly data, respectively. Each execution produces two graphs in a single figure. The first graph allows two series to be plotted simultaneously. This is useful for comparing neighboring sea level stations. The mean is removed for easier comparisons. The second graph is the difference between two series. More discussion is given in Section 6. 3-2 SLP64 USER MANUAL 4 Tidal Analysis and Prediction The goal of this package is to quality control the hourly data files. The techniques have been well documented (Caldwell and Kilonsky, 1992; IOC, 1992). The basis for quality control is the inspection of plots of residuals, defined as observed data minus predicted tides. In the following sections, the tidal analysis and prediction routines, the plotting programs, and the various quality control procedures are outlined. Lastly, a filter routine for the creation of daily and monthly means from the hourly data is described. Many of these routines must be performed in a cyclical fashion; thus, one will need to refer backwards and forwards within the manual as necessary. In this manual, a majority of the routines are fixed to work with the SLP64 hourly formats grouped by year (one year per file). For one section of the manual (Section 5.2), the software works with hourly data grouped by month (one month per file). This is for a special case for those that have tide staff observations available to calibrate the tide gauge values. If so, please jump to this section and review the material. When calibration is complete, these monthly files in turn are concatenated to form SLP64 hourly files grouped by year, on which the remaining routines can be applied. This software package utilizes the tidal height analysis and prediction programs of Dr. M. Foreman of the Institute of Ocean Sciences, Victoria, British Colombia, Canada (Foreman, 1977). Most recent news and versions are listed at http://www.pac.dfompo.gc.ca/science/oceans/tidal-marees/index-eng.html. Various notes regarding his procedures are found in Appendix F. The analysis program applies a linear least squares fit for specified harmonic constituents (astronomical arguments), which are based on earth, moon, and sun astronomical configurations. The output of the analysis is a set of harmonic constants (Appendix G). These in turn become the input for the tidal prediction function, which produce predicted tides for any year of any century. The UHSLC/NODC package has facilitated the use of the Foreman routines by SLP64 formats to (from) the Foreman programs. 4.1 Tidal Analysis This program analyzes the hourly tide gauge data for a given period of time. Amplitudes and Greenwich phase lags are calculated via a least squares fit method coupled with nodal modulation for only those constituents that can be resolved over the length of the input record. The output is a file of harmonic constants which become input for the tidal height prediction program. 4-1 SLP64 USER MANUAL The first step in use of the tidal analysis is to determine a suitable time span as input. One picks a time span with minimal gaps. To obtain a listing of missing data, use \SLP64\UTIL\GAPCOU. Another means of inspecting the hourly data is plotting with \SLP64\PYPLOT\HPLOT or HPLOT_YRS. Plot and inspect the available years of observed data. This package is fixed to allow a maximum of 68 constituents (Appendices F and G), which can be achieved through analysis of a record length of 366 days. The program will not work (it will abort) if an input record greater than 13 months is requested. If one would like to apply Foreman software for longer periods of analysis, then one has the liberty to use the Foreman routines, but you will need to provide input in the Foreman format (Foreman, 1977). The program will function for record lengths less than 366 days; however, the number of output harmonic constituents will be fewer and subsequently the quality of the constituents will be less. For instance, a record length of 30 days will produce 30 constituents and a record length of 14 days will produce 11 constituents. From the hourly plots choose a time span of at least 366 consecutive days for a period with apparently good data (no obvious spikes or corruption) and with minimal data gaps. From the example shown in the Figures 1 and 2, a time span from the first hour of 01 April, 1985 to the last hour of 01 April, 1986 would satisfy the criteria. The Foreman Analysis program can handle data gaps, but it is preferable to choose a period without them. The time span can begin on any hour of any day and can end on any hour of any day in the following year, as long as one does not exceed 13 months. One restriction is that the time span CAN NOT cross the century boundary; that is, the input span cannot include the consecutive months of December 1999 and January 2000. Once a time period for analysis has been selected, one is ready to run the Foreman program. The procedures are found in subdirectory, \SLP64\TIDE\ANA. Go to this directory and execute the TIDEANL.BAT batch job (via Windows Explorer or Command Prompt window). The program is interactive. Carefully provide correct responses. Notes: Review STAINFO.DIN Parameters. The program will display information found in the STAINFO.DIN file from the \SLP64\DIN subdirectory. If this information is incorrect, hit CONTROL-C to abort the program. Then correct the STAINFO.DIN file and rerun the TIDEANL job. Time Zone of Observed Hourly Data. One must choose if harmonic constants will be created relative to the local time meridian or relative to Greenwich Mean Time (GMT). In turn the predicted tides will be relative to the time zone referenced in the harmonic constants file. The time zone of your observed hourly data files can be changed using the utility TSALL (see Section 2.5). The time zone of the input (observed hourly data) 4-2 SLP64 USER MANUAL determines the time zone reference of the harmonic constants. If you would like to change the time zone of the hourly data, abort the program by typing CONTROL-C, then run TSALL. If you are ready to continue, choose either Y for data relative to local time or N for data relative to GMT. Start of Analysis Period. One must enter the start time and date of the analysis period in form HHDDMMYYYY where HH is hour (using the British convention, that is, the first hour of the day is 01 and the last hour of the day is 24), DD is day, MM is month, and YYYY is 4-byte year (e.g. 1978). For example: 0101011978 (1st hour of 01 January 1978). End of Analysis Period. Same as above for end of the period. It can be in the same or the following year, but do not exceed 13 months. For example 2401011979 (last hour of 01 Jan 1979). File Version. Version of observed hourly data file; e.g. A. The output is placed in subdirectory \SLP64\TIDE\HARM in files INPsss.PRD (where sss denotes the three-digit station number), and a file HARMsss.LIS (Appendices F and G). The amplitude units are in centimeters and the phase units in degrees. File INPsss.PRD is used for the tidal prediction program, and file HARMsss.LIS contains a listing of the harmonic constants formatted for a hardcopy printout. If the tidal analysis program is ran again for the same station, these files will be replaced. Thus if one wants to save different versions of the harmonic constants, then one will have to rename them. For instance, one could have a version of the harmonic constituents relative to GMT by running the analysis on a time period of data relative to GMT. Then rename INPsss.PRD to INPsss.GMT. Conversely, for local time, one could rename the output to INPsss.LOC. Then one can control the time zone of predictions by copying the desired harmonic constant file to INPsss.PRD, which is input to the Foreman prediction program, prior to running the prediction. If a grossly bad value exists in the input hourly data file within the time span provided to the TIDEANL job, then the Foreman program will abort. A useful way to find the input record with the bad data is to do the following. 1) Edit the TIDEANL job. Place a PAUSE command on a line by itself after the line with only the following: FITTIDE. 2) Rerun the TIDEANL job. When the new PAUSE command is reached (just after execution of FITTIDE), hit CNTRL-C, then answer Y to exit the batch job. 3) Edit file TGOUTPUT. If a bad record was encountered, it will be identified. 4) Go to the input data file on subdirectory \SLP64\DAT and replace the questionable value with a missing data flag, 9999. If the questionable record is not identifiable, replace all values on that line (12 values) with 9999s. 5) Rerun TIDEANL. 4-3 SLP64 USER MANUAL 4.2 Tidal Prediction Tidal prediction utilizes the harmonic constituents calculated by the tidal analysis (Section 4.1) to compute predictions in the form of either equally-spaced (in time) hourly values in the SLP64 processing format or high-low heights in the form of a tide table. Prior to running the Foreman tidal prediction program, one may need to alter some default settings that are set in file PRDVP.DIN located in the \SLP64\DIN subdirectory: 1) *units of equally-spaced predictions 2) units of harmonic constants 3) start month of predictions 4) interval between predictions 5) time scheme (0:1-24 1:0-23) = 2 = 3 = 12 = 60 = 1 *units code 1:feet (hundredths) 2:millimeters (whole digits) 3:centimeters (whole digits). For example, PRDVP.DIN contains one record in the following format: 2 3 12 60 1 As seen above for parameter 1, the scientific units for the predictions defaults to millimeters. For the quality control routines that follow, one should choose the units of the predicted tides to be the same as the units of your observed hourly data. Thus, if you do not have your units as millimeters, YOU MUST MODIFY PRDVP.DIN. It is recommended processing your data in millimeters. The parameters 2 - 4 above should not be altered. An explanation for obtaining predicted tides for equally-spaced intervals less than one hour is given in Appendix H. If the user's data files have a time scheme convention of 1 to 24 hours, then parameter 5 should be changed to a value of 0, so that the raw data files and the predicted tides files will be compatible. One can run \SLP64\TIDE\PRD\TIDEPRD.BAT (single year prediction per run) or \SLP64\TIDE\PYPRD\TIDEPRD.PY (multiple, consecutive years of prediction files). Interactive Parameters: - file version for output predictions file (A-Z) station number (i.e. 005) year to be predicted (i.e. 2014) end year of prediction (PYPRD routine only) time reference meridian of the predicted data, which should coincide with the reference meridian of the raw hourly data file that is input into the tidal analysis program (note the hemisphere (E or W) is not needed) (i.e. 075 or 000 for GMT) 4-4 SLP64 USER MANUAL - numerical code for the form of predicted data (0:high-low 1:equally-spaced) The output of the program will be placed in subdirectory \SLP64\PRD either file CVSSSYY.DAT (Section 1.5) or HLsssyy.DAT for equally-spaced (hourly) or tide tables, respectively. For convenience, the file name specifics are repeated below: C: century (O:1800-1899, P:1900-1999, Q:2000-2099) V: file version (letters A-Z) SSS: station number YY: the last two digits of the year For example, PA00385.DAT is the predicted tides file (version A) for station 003 and year 1985. QA00385.DAT would be the same for year 2085. HL00385.LIS is the tide table for station 003 in year 1885, 1985, or 2085. The filename for the tide table option does not distinguish century although the actual predictions do. Special consideration must be given to the change of century years. If one is running this program for the last year of a century (eg, 1999), then the TIDEPRD batch job will send a message to the screen showing the last hour of that year. This value should be written down and saved. Then when making predictions for the following year (e.g. 2000), the value just mentioned should be hand-edited into the year 2000 predictions to replace the missing data flag (9999) of the first hour of January. This step is essential for use of the gap interpolation programs (Section 5.4). 4-5 SLP64 USER MANUAL 5 Quality Control Quality control ensures the scientific validity of the data. Three main aspects are emphasized: 1) the linking of the data to a reference level (tidal datum), 2) the inspection of the timing quality, and 3) the replacement of short gaps and spikes. Technical aspects of quality control procedures have been well documented (Caldwell and Kilonsky, 1992; IOC, 1992). The tidal analysis and prediction programs (Section 4) and the plotting routines (Section 3.1) are an integral part of the quality control task. Many of the routines run in a cyclical fashion. In the discussion below, one will be referred to various sections in this manual. Prior to initiating quality control of the data, make copies of the original observed data files. It is not uncommon to make mistakes during editing; thus, it is very important to preserve the original version of the data. Also, keep a log of modifications as you proceed. It is good practice to keep editing notes in a text file, which can be saved with the final version of the data to clearly indicate what changes were made to the original measurements. 5.1 Hourly Residuals The basis of quality control is the inspection of residuals, defined as observed data minus predicted tides. Residuals are made from program, RESIDM, in subdirectory \SLP64\QC. The interactive program will ask the following questions: Input Observed File Version. This is the version of the observed hourly data file in the UHSLC processing format (Section 1.5). Input Predicted Tides Version. This is the version of the predicted tides. Normally there is no need for more than one version. Output Residuals Version. This is the version of the output. Normally one uses the same version as used for the input observed file. Station Number. Provide three-digit number, e.g. 003. Start Year. Provide complete 4-byte year; e.g. 1998. Number of Years. Give number of years over which to calculate residuals. There must be a file of predictions and a file of observations for each desired year. The output is placed in subdirectory \SLP64\DAT. Make a residuals file for each year of observed data. Next, the residuals are visually inspected through use of the HPLOT or HPLOT_YRS plotting programs (Section 3.1). 5-1 SLP64 USER MANUAL If the observed data is of good quality and the tidal analysis estimated well the tidal species at the site (ie, good predictions), then the residual plots will have a smooth appearance (Figure 3). In this case, there are no obvious problems with the data. If the observed data has erroneous values but the predicted tides are accurate, then the residuals plot will clearly show the faulty values (Figure 4). In this case, some periods within the plot are smooth, while others show significant fluctuations, gaps, and spikes. Techniques for correcting these problems are discussed in Sections 5.3 and 5.4. In locations with highly non-linear tides due to shallow waters, influence of rivers or estuaries, or complex basin configurations, the Foreman tidal analysis routines will not be able to completely resolve all the tidal species. At such a location, assuming the observed data are of good quality, the residuals will show the tidal species that have not been resolved by the tidal analysis. This situation is shown in Figure 8 for a location on the western coast of Malaysia. Thus, quality control is limited at this site. Only grossly extreme spikes in the residuals could potentially be identified in the observed data file. For most locations, the harmonic constants, which were produced by the tidal analysis, are applicable for prediction of any year. However, at some locations, the tidal species are not completely resolved and thus the quality of the predictions decreases as the predicted year moves further away in time from the time span used in the tidal analysis. Note the residuals in Figures 9 and 10. The first figure shows residuals of 1995 observed data based on a tidal analysis of December 1991 - December 1992. Note the significant fluctuations. In the second figure, the residuals are based on the same year as plotted. In this case, the residuals are smoother. Since the goal of the residuals is to identify potential errors in the data, then for some sites with tidal species similar to this example, it would be best to refresh the harmonic constants file using time spans of tidal analysis close to the year in review. It is also possible that the harmonic constants could have been created from a time span that is less ideal than another available time span of the same time series. For instance, assume one creates a series of residual plots for various years, and notes in the residual plots that the time period selected for the tidal analysis had timing errors and other data problems. In this case, it would be best to recreate a harmonic constants file based on a more suitable time span as input to the tidal analysis. It is good practice to recreate a harmonic constants file after quality control is complete so that one has the best possible set of constituents to base future inspections. 5.2 Reference Level Stability The calibration of sea level data is necessary to relate the measurements recorded by a sensor to a stable datum (Mitchum et al., 1994). Often called the tide gauge datum, this 5-2 SLP64 USER MANUAL relationship defines the horizontal plane from which heights are measured at the tide gauge. For most float and stilling well gauges using analog or punched paper tape recorders, the tide staff zero is the fundamental calibration point, which is geodetically linked to a primary Tide Gauge Bench Mark and several auxiliary bench marks. The procedures for the establishment of a tide staff and the associated network of vertical control points are well described (Hicks et al., 1987). Visual staff readings are used in the calculations of the tide gauge datum as shown below. Many modern acoustic gauges have internal calibration systems so the following steps are unnecessary (IOC, 1992). Also, automated calibration devices, such as the UHSLC switch, provide supplementary calibration information. Use of switch data is not discussed here (see Mitchum et al., 1994)). Improper calibration leads to reference level shifts. Most shifts are readily identifiable in hourly residual plots (see Figure 4 at end of December). They can also be seen in plots of the daily and monthly data (Section 6) and in plots of differences of daily or monthly values with nearby tide stations or with redundant sensors at the site in question. If a shift is identified, the proper means of correction is through analysis of the tide staff readings and corresponding tide gauge values. Normally the link to the tide staff is made with the observed data of the highest frequency of samples; for instance, most data are digitized in 4-, 6-, 10-, 15-, or 60-minute intervals. For punched paper tape recorders, the tide staff reading is made exactly at the time of the punch. For analog data, the tide staff reading is taken exactly on the hour. Since the tide staff readings and most digital and analog punch paper tape or strip chart rolls come in on a monthly basis, this procedure is performed each month. For this reason the UHLSC processing format is designed for files consisting of either one month of data or up to one year of data in monthly increments. The monitoring of the reference level stability is most naturally performed on monthly files. These monthly files (Section 1.5.2) in turn are concatenated into yearly files at a later date, at which time the core functions of this package can be applied. These monthly files of hourly data must be kept on \SLP64\DAT. An important field in these files is the REF= in the header. The value given will be the value that must be subtracted from each hourly value in order for the series to be linked to the station tide gauge datum. The reference level quality control program SCAT.PY is found on \SLP64\PYPLOT. The method for calibration using tide staff readings is as follows. Build PAIR File. Within subdirectory \SLPR2\CAL, construct a "PAIR" file containing paired tide staff readings and the corresponding tide gauge measurements (Appendix D.4 and Section 1.5.2). No utility is provided to build this file. One option is to hand input the values into the PAIR file using a text editor. An example is found in \SLP64\SAMP\x0038807.dat. 5-3 SLP64 USER MANUAL Plot PAIR values. Plot the pairs using program SCAT (Section 3.3). Summary statistics are calculated within SCAT and placed in \SLP64\CAL in a file with the same root name as the PAIR file, but with file extension OUT (e.g. X0038807.OUT). A plot is sent to the computer screen and also placed in a file of the same root name with file extension JPG (jpeg image). One uses this output to help identify outliers by inspecting the last column (DIFF-the difference among pairs). Inspect PAIR values. If the tide staff readings were accurate and the tide gauge was functioning well, then the points on the graph will be nearly linear with slope equal to 1. It is common for a few points to be well off the linear trend due to bad staff readings (Figure 5). Note the one extreme outlier. If the gauge was malfunctioning, for instance, due to a clogged stilling well, then the scatter will be random (Figure 6). If the gauge zero was altered by a technician, then a reference level shift would be evident in the scatter (Figure 7). Correct PAIR File. If outliers were evident in the scatter diagram, then remove them from the PAIR file using a text editor. Next, replot the scatter diagram (SCAT) to verify correction. File Organization. If an identifiable reference level shift was found (as in Figure 7), create two PAIR files separating the time periods before and after the shift. There is no convenient file naming convention associated with this package to do this. A suggestion would be to make temporary files. For instance, rename the PAIR file (Xsssyymm.DAT) to PAIR.ORI (i.e. ORI denotes Original). Then use an editor to create two files, BEFORE Bsssyymm.DAT and AFTER Asssyymm.DAT, corresponding to the tide staff/gauge pairs before and after the shift. Be sure to keep of log of temporary filenames so not to be confused. Run the program SCAT (discussed immediately below) separately for the BEFORE and AFTER files. More instructions will be given later in this section in regards to applying a correction for a level shift. Compute Statistics for Pairs. Run SCAT for each section. The output is placed in the \SLP64\CAL with filename same as input but file extension .out. These files should be saved. If a level shift occurred, one will have to rename the output prior to executing the program for the second input file. Determine Preliminary Calibration Constant. The last column of the last row of the *.out file is the Average Difference, which is the best approximation of the output of the MEAN ZERO REFERENCE LEVEL CORECTION VALUE, or zero level correction for the given month. It has the same units as the data in the PAIR file. It is a preliminary correction value only. This is a preliminary calibration constant. The final calibration constant is determined after an extended time period of no changes to the instrument, staff or the foundation. Update Calibration Log. Build a file, CALsss.LOG (sss as station number), in the \SLP64\CAL directory with the form as shown in Appendix D.5. A sample is given in \SLP64\SAMP\CAL003.LOG. Make note of the MEAN ZERO REFERENCE LEVEL CORRECTION VALUE and the NUMBER OF OBSERVATIONS (tide staff/gauge pairs) for the given time period. This log forms the basic reference text for monitoring the long term stability of the datum. 5-4 SLP64 USER MANUAL One does not apply the calibration constant to the data unless there is a clearly evident reason, such as the resetting of the gauge zero by a technician, a malfunction of the gauge (e.g. the slippage of the gauge cable), or the vertical displacement of the tide staff. The basic assumption is that the gauge is stable and month to month variations in the MEAN ZERO REFERENCE LEVEL CORRECTION VALUE are caused by noise in the tide staff readings. At the UHSLC, annual reviews of the monthly calibration constants are made, and one value is assigned as the tide gauge datum correction for an extended period over which no obvious changes occurred to the instrumentation or the tide staff. These extended periods normally coincide with the periods between technical visits. Thus, one avoids introducing noise into the sea level time series due to month to month changes in the MEAN ZERO REFERENCE CORRECTION LEVEL VALUE. As a rule of thumb, for float/well analog and punched paper type gauges, the UHSLC considers changes greater than 0.05 feet (1.5 cm) as significant and worthy of an investigation into the level stability. Most acoustic gauges have greater accuracy. With the float/well systems, the UHSLC employs redundant gauges (Mitchum et al., 1994) at a given tide station to bring the accuracy of datum correction to finer levels. If a reference level shift has occurred, follow these guidelines: Identify Date-Time of shift. Verify the date that the shift occurred by looking at a plot of residuals (see Section 5.1) for the time period of interest. Unfortunately, the residual and plotting programs require yearly input files in the SLP64 hourly format. If one is presently working with the monthly files of hourly data, concatenate these into a file using the yearly file naming convention. Then make and plot residuals. An example of a shift is seen in Figure 4) at the end of December. Compute the Correction. Calculate the difference between the old zero reference level correction and the new zero reference level correction based on information provided by the statistics of the staff/gauge pairs. The old zero reference level correction can be found in the file header of the monthly file of hourly values (Section 1.5.2). For example, if this value is 1030 and the new zero reference level correction is 1050, then the difference is 20. Apply Correction to Data. Go to subdirectory \SLP64\QC and run program, COREF, to apply this difference to the individual monthly UHSLC file of hourly data. In the example used above, provide the value 20. This means that each hour in the file will be increased by 20. Note this program is only for hourly data files in monthly blocks. Also, this program corrects the entire month to the desired reference level. The output is placed in the \SLP64\DAT directory in file COREF.OUT. Use Editor to Apply Final Correction. Go to the \SLP64\DAT directory. When a reference level change has occurred in the middle of a month, the UHSLC recommends that the data for that month be corrected to the new zero reference level. A text editor is used to cut from the output of COREF and paste into the monthly file of hourly values. Be 5-5 SLP64 USER MANUAL sure a copy of the original monthly file is preserved. This is very important. As an example, assume the reference level shift occurs on the 1200 hour of the 12th day, replace all the data BEFORE this point in time in the monthly file of hourly values with the corrected data from the file COREF.OUT, and replace the old zero reference level value (REF=) in the header record with the new one. As in the example of the previous paragraph, replace REF=1030 with REF=1050. When editing is complete, delete COREF.OUT prior to executing the program again; otherwise, an error will result. Linking the observed data to a datum is one of the most important steps in quality control. The following sections describe other important aspects of controlling the quality of the data. 5.3 Timing Errors Timing errors are introduced into the data due to mistakes during data processing, to incorrect setting of the timer on the tide gauge, or to inaccuracies in the gauge time clock. The errors are evident in the plot of residuals as periodic fluctuations. An example can be seen in Figure 4. Note the large fluctuations in November and December, and the mild fluctuations during other months. As discussed in Section 5.1, residual plots can reflect inaccuracies in either the observed data or the tide predictions. The ability for the tidal analysis to fully model the tidal species depends greatly on the location. For gauges in ports adjacent to deep waters, the analysis is usually very good. For gauges in regions with influence of rivers, shallow coastal shelves, narrow basins, or complex basin bathymetry, the analysis is of low to moderate quality. In this case, assuming the observed data are good, the residuals with show periodic fluctuations for tidal species that were not resolved by the tidal analysis (Figure 8). With this software package, one can only adjust the timing of observed hourly data files by exact increments of an hour. Once a timing error is identified in the residuals, there are three ways to determine the magnitude of the timing error: Inspect Original Data Tables. If the data were digitized from analog rolls, then inspect the data tables in comparison with the observed hourly data file to ensure that an hour was not accidentally added or removed during digitization. Compare Residual to Observed Data Plot. Place a paper copy of a residual plot on top of a paper copy of an observed hourly data plot. One can use a light table or hold both papers, with borders aligned, up to a light. Note if the peaks in the fluctuations of the residuals is to the left or to the right of the observed hourly peaks. If the residuals are to the right, then assume a negative shift is necessary. One will have to guess, but start 5-6 SLP64 USER MANUAL with a guess of magnitude one. If that does not work, increment to greater values (2, 3, etc). Compare Observed to Predicted Files. Using a split screen editor (or two windows), place the observed hourly data file centered on the time span with the shift in one window, and the corresponding predicted tides file in the other window (Appendix I). Identify for a given semi-diurnal (or diurnal) tide cycle the hour with maximum amplitude in both files for the same day. Note if the peak (hour with largest value) in the observed file is behind or ahead in time of the corresponding peak in the predicted tides file. If the observed peak is before the predicted peak, then the observed file will need to be shifted forward by the number of hours of lag. Conversely, if the observed peak follows the predicted peak, then the observed file will need to be shifted backwards (negative). One may want to examine peaks in both files for a separate tidal cycle to verify the magnitude. Once one has determined the magnitude and direction of the shift, one uses the program TSALL, as described in Section 2.5. This program is in the \SLP64\UTIL directory. The program shifts the entire year and places the output in a file with a different file version in directory \SLP64\DAT. For example, choose the output file version "T", so if the input was VA00386.DAT, the output would be VT00386.DAT. To verify if the estimated shift is correct, use the RESIDM program (Section 5.1) to calculate residuals for the shifted file. Next, use the HPLOT or HPLOT_YRS plot programs (Section 3.1) to graph the residuals. As an example, see the timing error in November 1986 of Figure 4 and the inspection of magnitude in Appendix I. After applying the steps above for a positive one hour shift, the residuals are shown in Figure 11. Note that the November period is now smooth. Thus the choice of a positive one hour shift was correct. Also note in Figure 11 that other time periods now have significant periodic fluctuations. These can be ignored. To know exactly which hour the shift began, one can look at the residual file (RA00386.DAT) of the original data file (VA00386.DAT). For the example given above (shift in November), there are gaps on both sides of the shift, so it is easy to identify where the time period with the shift begins and ends. When there is not a gap adjacent to a shift, then on uses the plot (Figure 11) to guide you to the day of the shift. For instance, there is a timing error on December 4-7 of this same example. It is obvious where the timing error begins because there is a gap preceding it. But it is unclear on which hour the timing error ends. Looking at the residual file for the 6-8 December (first 11 columns of each line of file removed below to fit in text): 198612 198612 198612 198612 198612 198612 61-1784-1449 -774 62-1807-1602-1028 71-1566-1645-1339 72-1438-1601-1404 81 371 -149 -160 82 -135 -151 -158 54 822 1335 -209 646 1336 -733 0 675 -882 -161 577 -159 -137 -109 -154 -135 -106 1468 1689 1120 1176 -88 -77 1188 571 -243-1039-1609 1629 1177 447 -387-1116 1224 975 441 -254 -947 1478 1415 1006 361 -372 -75 -73 -81 -101 -124 -55 -41 -30 -33 -47 5-7 SLP64 USER MANUAL Note the dramatic difference from value to value during the period with the timing error, and the small difference between adjacent values for the periods without a timing error. In the example above, it appears that the timing error ends on hour 00 of 08 December. Make a note of the hour and date of the start and end of the period with the timing error. The next step is to make a copy of the original file. For instance, if the file is VA00386.DAT, then copy it to VB00386.DAT. This will preserve the original in case the following edits need to be removed. As pointed out, original file versions must be preserved. Use a text editor to modify the copy of the original (VB00386.DAT). One deletes the hourly values for the time period with the timing error. In a separate window, load the output of TSALL (for example, VT00386.DAT). Cut the corresponding period with the corrected shift from VT00386.DAT and paste into VB00386.DAT for the deleted time period. At UHSLC, we normally replace the start and end hour of the shift with a missing data flag (9999) since the exact hour of when the timing error began or ended is usually not clear. The missing data can later be interpolated as discussed in Section 5.4. To verify the correction, create residuals for the edited file (create RB00386.DAT) and plot. If timing errors are still present, repeat the process above. It is sometimes trial and error in discovering the magnitude and direction of a shift. One may have to iterate through the steps above several times before a correction is found. And recall, this package can only correct shifts of exact increments of an hour. If timing errors exist due to shifts of partial hours (such as 15 minutes, etc.), then this package is not applicable. Note in Figure 4 the small periodic fluctuations throughout January to October. It is suspected that these fluctuations are due to minor timing inaccuracies of the tide gauge clock (note how the amplitudes of fluctuations increase with time). This package cannot correct this timing error. A note regarding these timing uncertainties would be made in the final document file that accompanies the data in the final archive. If the originally sampled data are at intervals less than an hour (6-, 10-, or 15-minutes are common), then one should try the technique mentioned but for exact increments of the sampling interval of those data. Unfortunately, one will have to develop their own routines, since this package is geared to hourly values. 5.4 Short Gaps and Data Spikes Short gaps and data spikes are a common problem in most sea level time series records (See example in Figure 4 in December). This section describes a technique for using interpolation to correct such problems. 5-8 SLP64 USER MANUAL Program GAPCOU (Section 2.3) is a utility that lists the length (hours) and dates of missing data. Run this program to obtain this information. The best procedure for filling gaps is to replace the missing data flags (i.e. 9999) with quality controlled data from an auxiliary sea level gauge that is linked to the same datum. If a redundant sensor in not available, then another good method is linear interpolation via the predicted tide method through application of batch file GAPFALL.BAT (program FALLPTM) on directory \SLP64\QC. Technical aspects of running this program will be discussed shortly, but first, a note about the methodology. The predicted tide method for filling gaps requires yearly files of observed and corresponding predicted data. The predicted tides are shifted in time to match the timing characteristics of the observed series. The residuals between the predicted tides and the observed data are calculated. Then, a linear interpolation between the end points of the gap in the residual series is performed and each interpolation constant is added to the shifted predicted tides over the span of the gap. The UHSLC recommends using this procedure only for gaps less than or equal to 24 hours. This is essential for the integrity of the daily data which can be calculated from the hourly data (Section 6). To run the program, go to the \SLP64\QC directory (mandatory), and enter: GAPFALL The program will query the user for the following information: - station number (three-digits, e.g. 003) the file version of the observed data file with the gap (i.e. A), the file version for the predicted data (typically A), (ensure the scientific units of the observed data and predicted tides are the same) the file version for the output data with the gap filled, which must be different from the original version (i.e. B, C, etc.), the year with the gap in form yyyy (i.e. 1986), if a gap occurs on last few days of December, then one can optionally acquire data from January of the following year. However, the observed and predicted files of the following year must exist, and have the same file versions. The batch job GAPFALL.BAT is used to perform file management to ensure existing files cause the program to abort. The following output is produced: 5-9 SLP64 USER MANUAL Interpolated Data. The program places the interpolated data file in directory \SLP64\DAT. List of Interpolated Data. The GAPsssyyyy.TXT (mentioned above) is output to directory \SLP64\QC\GAPNOTES, where sss is station number and yyyy is year. At the UHSLC, interpolated gaps greater than six hours are listed in the station document file which accompanies the data in the final archive. Timing Drift. An output file, TIMING.TXT, is placed in directory \SLP64\QC\GAPNOTES. This file lists the drift (fractional hours) in timing between the observed data and predicted tides for each month. For each run of GAPFALL.BAT for a given station, the output is placed in TIMING.TXT. This file has little. Extra Data. File EXTRA.DAT is placed in directory \SLP64\DAT. It contains data for the first two days of the following year relative to the input of GAPFALL.BAT. This is for a special case: missing data at first hour(s) of a year. If data from the end of the previous year are available, then this gap can be interpolated. One runs GAPFALL.BAT for the preceding year, choosing affirmative (1) the option that data from the following year exist (step 5 above). Then go to \SLP64\DAT and replace the missing data flags at the start of the year (year after which the program was executed) with the output of EXTRA.DAT. The predicted tide method for filling gaps can also be used to correct data spikes and glitches. A data spike is an obviously wrong data point. A glitch is one or more, but less than or equal to 24, consecutive obviously wrong data points. These features are easily identified on plots of residuals (Figure 4). To correct, first identify the date and time by looking in the residuals data file. The spike should stand out as an outlier in the data. Second, use a text editor and replace the corresponding value in the observed data file for the date and time of the spike with a missing data flag, 9999. Finally, run the gap filling program as described above. One should verify the output of the gap interpolation by making and plotting residuals. 5-10 SLP64 USER MANUAL 6 Filtering Software is provided for obtaining daily values from hourly sea level data by use of a sophisticated, two-step filter and for obtaining monthly values from the daily values with a simple average. Plotting software is included for the daily and monthly data. Also, programs are available for calculating the difference between two daily or monthly files. A subdirectory \SLP64\FILT has been created for these purposes. An overview of the filtering method, the input and output files, the plotting routines, and other important items are discussed below. 6.1 The Filter Daily values are obtained using a two-step filtering operation. First, the dominant diurnal and semi-diurnal tidal components are removed from the quality controlled hourly values. Secondly, a 119-point convolution filter (Bloomfield, 1976) centered on noon is applied to remove the remaining high frequency energy and to prevent aliasing when the data are computed to daily values. The 95, 50, and 5% amplitude points are 124.0, 60.2, and 40.2 hours, respectively. The Nyquist frequency of the daily data is at a period of 48 hours which has a response of about 5% amplitude, thus, aliasing is minimal. The primary tidal periods have a response of less than 0.1% amplitude. The filtering operation incorporates an objective procedure to handle gaps. This objective technique simply replaces the filter weight at any missing observation with a zero and renormalizes the sum of the modified weight function to unity. This technique is equivalent to interpolating the missing observation with an estimate of the local mean of the time series. The local mean is defined as the mean of a given segment of length equal to the length of the filter. The error associated with this technique can be estimated objectively and is used as a criterion for accepting or rejecting a daily value computed in an area of the time series which contains a gap or gaps. This error depends on the ratio of the standard deviations of the input (hourly) and the output (daily) data. Thus in order to keep the ratio low, it is essential to apply this technique to the residual series as defined above. The monthly values are calculated from the daily data with a simple average of all the daily values in a month. If seven or fewer values are missing, the monthly value is calculated. The monthly value is stored in the file with the daily data and in a separate file as noted below. 6-1 SLP64 USER MANUAL 6.2 Running the Filter Program The input files for this program are in the UHSLC processing format (Appendix D.1). A requirement of the filtering program is that each month have 12 consecutive months of input hourly data or missing data flags in the SLP64 processing format. If some months are missing in a year, one may use program FILLMONTHS.BAT in the \SLP64\UTIL directory (Section 2.4) to create complete months of missing data flags, which can then be inserted by hand using a text editor into the data file so that the file has twelve consecutive months. The filter program will ask the user for the start year and number of years to be filtered. This defines the time window of the data to be filtered. The program also asks if data prior to and after the time window are available. This is because the filter length is 5 days long; thus, the first and last day of the time window cannot be calculated without information prior and post of the time window. For example, one runs the program with a time window of 1982 to 1984. Thus if data (not missing data flags) begin on 1 January, 1982 and data are available for 30-31 December, 1981, then the program can automatically access this data and thus produce a filtered daily value for 1 January, 1982. Note for the prior and post year files, the SLP64 processing format must also be used with the same file version yet the file DOES NOT have to have 12 consecutive months (an exception to the requirement above). To run the program, go to the \SLP64\FILT directory and type: FILTHR The output of the program is in the UHSLC processing format for daily data (Appendix D.2) and monthly data (Appendix D.3) and is placed in directory \SLP64\DAT. File names are Dvsss.DAT and Mvsss.DAT (v:version, s:station) for the daily and monthly data, respectively. All years for a given station are kept in the same file. The output daily and monthly file formats can be changed to a more contemporary format (Date one-value-per-row) in CSV text format with \SLP64\DSL2CSV and MSL2CSV (Section 2.6). 6.3 Examine the Output Python/Matplotlib programs DPLOT and MPLOT (Section 3.4) on the \SLP64\PYPLOT directory are provided to plot the daily and monthly data, respectively. The daily and monthly values are useful for studying the low frequency variability of sea level. In addition, they provide another opportunity to judge the quality of the reference level 6-2 SLP64 USER MANUAL stability (Section 5.2). Large level shifts are clearly apparent in plots of daily and monthly values as step-functions. A common way to analyze oceanic variability or to judge the quality of the reference levels is to compare one tide gauge with another gauge at the same or nearby site. Separate plot programs, D2DIFF and M2DIFF on \SLP64\PYPLOT. The programs simply prompt the user for the station numbers and file versions of the two stations for the input data (daily or monthly) files. The program automatically displays the time span of each station. One must choose an overlapping set of years. The program generates two graphs per figure. Figure 12 shows the daily difference plots, figure 13 shows the monthly difference plots. The top plot depicts both series, which individually have had their average subtracted (de-meaned). The bottom graph plots the difference. If a step-function signature is seen in the difference, then there is a possibility that a reference level shift occurred at one of the stations. One must return to the procedures discussed previously (Section 5.2) to investigate and correct the problem. 6-3 SLP64 USER MANUAL 7 Final Comment The JASL Sea Level Processing and Quality Control Software package for 64-bit Microsoft operating systems is available for enhancing the use and quality of sea level data. More data centers and oceanographic institutes now have access to personal computers and the JASL would like to share its software products and data processing techniques. The goal is to support international capacity building and to enhance data quality. Data collectors are encouraged to share their hourly sea level holdings with the Global Sea Level Observing System (GLOSS). The JASL is a GLOSS archive. Your contributions are welcome. Any questions or comments regarding this manual or software should be directed to: Mr. Patrick Caldwell Joint Archive for Sea Level Dept. of Oceanography University of Hawaii at Manoa 1000 Pope Rd. MSB 318 Honolulu, Hawaii 96822 USA phone: 808-956-4105 fax: 808-956-2352 Internet: caldwell@hawaii.edu Web: http://ilikai.soest.hawaii.edu/UHSLC/jasl.html 7.1 Acknowledgements I acknowledge Mike Foreman of Fisheries and Oceans, Canada, Institute of Ocean Sciences, for making his tidal analysis and prediction programs publically available. Also, thanks go out to Shikiko Nakahara, Bernie Kilonsky, and Brent Miyamoto of the University of Hawaii Sea Level Center for their contributions to the development of some routines in this package. Thanks also to Brent for his comments regarding calibration. For helpful examples in learning Python/Matplotlib, thanks are given to Jules Hummon of the University of Hawaii. Appreciation is addressed to Glen Rowe of Land Information New Zealand who provided edits to this manual. The views expressed herein are those of the author and do not necessarily reflect the view of NOAA or any of its sub-agencies. 7-1 SLP64 USER MANUAL 7.2 References Bloomfield, P. 1976. Fourier Analysis of Time Series: An Introduction. New York: John Wiley and Sons. pp 129-137. Caldwell, P., and B. Kilonsky, 1992. Data processing and quality control at the TOGA Sea Level Center. Joint IAPSO-IOC Workshop on Sea Level Measurements and Quality Control, Paris, 12-13 October, 1992. IOC Workshop Report No. 81, UNESCO. pp. 122-135. Caldwell, P. 1998. Sea Level Data Processing on IBM-PC Compatible Computers, Version 3.0 (Year 2000 Compliant). JIMAR Contribution No. 319, SOEST, University of Hawaii, 72 pp. Caldwell, P. and M. Merrifield, 2013. Joint Archive for Sea Level: Annual Report 2013. JIMAR Contribution No. 13-385, Data Report No. 23, SOEST, University of Hawaii, 73 pp. Foreman, M.G.G., 1977. Manual for Tidal Heights Analysis and Prediction. Pacific Marine Science Report 77-10, Institute of Ocean Sciences, Patricia Bay, Sidney, B.C., 97 pp. Hicks, D. and P. Morris, H. Lippincott, M. O'Hargan, 1987. User's Guide for the Installation of Bench Marks and Leveling Requirements for Water Levels. National Ocean Service, National Oceanographic and Atmospheric Administration. US. Dept. of Commerce. Intergovernmental Oceanographic Commission, 1992. Joint IAPSO-IOC Workshop on Sea Level Measurments and Quality Control. Workshop Report No. 81. Paris, 12-13 October, 1992. page 16. International TOGA Project Office, 1992. TOGA international implementation plan. Geneva, Switzerland. Fourth Edition, 01 October, 1992. ITPO-No. 1, 73 pp. Kilonsky, B. J. and P. Caldwell, 1991. In pursuit of high- quality sea level data. IEEE Oceans Proceedings. Vol. 2, October 1-3, 1991. Mitchum, G.T., B.J. Kilonsky, and B.M. Miyamoto, 1994. Methods for Maintaining a Stable Datum in a Sea Level Monitoring System. IEEE Oceans Proceedings. 0-7803-2056-5, 1994. UNESCO, 1993. IOC Training Course for the Global Sea Level Observing System (GLOSS) directed to the African and South American Portuguese- and Spanish-speaking Countries. Sao Paulo, Brazil, 1-19 February, 1993. IOC Training Course Reports No. 20. UNESCO, 1995. IOC/GLOSS-GOOS Training Workshop on Sea-Level Data Analysis. Geodetic and Research Branch, Survey of India. Dehra Dun, India. 21 November - 1 December, 1995. IOC Training Course Reports No. 39, 17 p. Wyrtki, K., 1979. The Response of Sea Surface Topography to the 1976 El Nino, J. Phys. Oceanogr., 9, pp. 1223-1231. 7-2 SLP64 USER MANUAL Appendices A Summary of Upgrades from SLPR2 to SLP64 - 32 to 64 bit executable programs SLPR2 is the year 2000 compatible version of the JASL Sea Level Processing (Caldwell, 1998). It is composed of Fortran programs compiled under the 32-bit Microsoft (MS) operating system. The new SLP64 Fortran programs are compiled under the 64-bit MS operating system. - Not Fixed to C: Drive Earlier versions had the c:\ drive written into the Fortran and BAT code to specifically define the hard drive. In SLP64, the paths in the Fortran, BAT, and Python code refer to \SLP64, which is placed on the root directory of any drive; thus, allowing flexibility in drive designation. - SLP64 Operated From Windows Explorer Within SLPR2, there are some BATCH (BAT) jobs that must be executed from an MS command prompt window. The new package can be entirely operated from a Windows Explorer page (directories and files) by clicking on a program to run it or on a file to view or edit it. There is no need to use the command prompt window, though it still works via that method. - Year 1900 Bug Fixed In SLPR2, year 1900 was mistakenly taken as a leap year. The new SLP64 correctly identifies 1900 as not a leap year. - Easier Conversion from Originator’s Format to SLP64 In SLP64, a set of Python programs are available to more easily convert hourly data into the SLP64 hourly sea level processing format. These tools allow a means of converting CSVformatted hourly files of the UHSLC data set and the text-tabbed formatted hourly sets of the NOAA National Ocean Service COOPS Water Levels Data Access System to SLP64 A-1 SLP64 USER MANUAL - Multiple Consecutive Years from Single Execution The Foreman Tidal Prediction task was executed one year at a time for SLPR2. For SLP64, a Python program facilitates execution of multiple years. Similarly, the plotting programs for the hourly observations or residuals can run over multiple consecutive years. - Enhanced Plotting Programs SLP64 uses Python/Matplotlib software for plotting. A new program was added for comparing daily and monthly series between two stations. It creates two plots per page. The first plot has the de-meaned daily or monthly series of two stations for overlapping years on the same graph and the second plot depicts the difference. A-2 SLP64 USER MANUAL B Access to Package Web site: http://ilikai.soest.hawaii.edu/UHSLC/jasl.html Under Software, select SLP64 B-1 SLP64 USER MANUAL C Directory Tree C:\SLP64 | |--\DIN | |--\DAT | |--\CAL | |--\manual | |--\FILT | |--\SRC | |--\pyPLOT | |--\plots | |--\SRC | |--\PRD | |--\QC | |--\SRC | |--\SAMP | |--\TIDE | |---\ANA | |---\PRD | |---\pyPRD | |---\HARM | |---\SRC | |--\UTIL | |--\SRC | {data information files} {data files} {calibration data files} {manual} {filtering software} {source code} {Python/Matplotlib plotting software} {holds plots generated from pyPLOT programs} {ancillary Fortran programs} {predicted tide files} {quality control programs} {source code} {sample data} {tidal analysis} {tidal prediction} {allows multiple years} {harmonic constituents} {source code} {utility programs} {source code} C-1 SLP64 USER MANUAL D Formats D.1 Hourly Data Processing Format Each observed hourly data file is given a name, CVSSSYY.DAT, where C: century (U:1800-1899, V:1900-1999, W:2000-2099) V: file version (letters A-Z) SSS: station number YY: the last two digits of the year Example: VA00385.DAT is the observed hourly data file (version A) for station 003 and year 1985. The file contains header records, data records and terminator records. Each month begins with a one-record header and ends with two terminator records. Each data record contains one half day of data or 12 hourly values. The time scheme convention centers the data on hours 0-11 and 12-23 for the respective data records. The header for each month is coded as: field ----------------station id station name bytes ------1-3 4-14 7-9 latitude 19-26 longitude time zone 33-41 50-55 reference offset 61-65 time interval month year units 67-68 70-72 74-75 77 # days in month 79-80 comment --------------------------------station number - must be 3 digits station name - note special cases: file type; PRD for predicted and RES for residual latitude in degrees, minutes, tenths of a minute, and hemisphere longitude as above time meridian reference for data, i.e. GMT or 090W constant value that is subtracted from each data point in the month in order for the data to be relative to the station tide staff zero time in minutes between each data point first 3 characters of month, i.e. JAN year without century indication, i.e. 78 scientific units of data as: F : hundredths of feet (1121 = 11.21 ft) M : integer millimeters(1121 = 1121.0 mm) C : integer centimeters(1121 = 1121.0 cm) not a mandantory field For example (follows on next page): # of days in month D-1 SLP64 USER MANUAL STATION time interval between values | ID ref level offset | units| | STATION time zone | | year | | | | latitude and longitude | | | month| | | __|_______| ___________________________|____________:_________|__|___|__|_|_| 003BALTRA LAT=00 26.8S LONG=090 17.2W TMZONE=GMT REF=00000 60 APR 85 M 30 The data records are coded as: field ---------------station id station name year month day record count data values bytes ------1-3 4-10 12-15 16-17 18-19 20 21-80 comment --------------------------------station number - must be 3 digits station name year i.e. 1988 numerical month, ie 12 numerical day, ie 31 1 for 1st record; 2 for 2nd record of day Fortran format 12i5 For example: STATION YEAR ID | MONTH | STATION | | DAY | | | | | RECORD # HOURLY VALUES IN feet __|_______|___|_|_|_|__________________________________________________________| 003BALTRA 1985 4 11 1492 1668 1879 2054 2152 2161 2067 1890 1663 1445 1285 1218 003BALTRA 1985 4 12 1266 1431 1667 1921 2123 2235 2230 2097 1893 1650 1432 1299 An extraction from an hourly file for Baltra is shown below. 003BALTRA LAT=00 26.8S LONG=090 17.2W TMZONE=GMT REF=00000 60 APR 85 M 30 003BALTRA 1985 4 11 1492 1668 1879 2054 2152 2161 2067 1890 1663 1445 1285 1218 003BALTRA 1985 4 12 1266 1431 1667 1921 2123 2235 2230 2097 1893 1650 1432 1299 003BALTRA 1985 4 21 1277 1385 1604 1868 2101 2268 2303 2201 1974 1681 1397 1185 003BALTRA 1985 4 22 1103 1180 1392 1695 2023 2295 2430 2401 2239 1971 1634 1337 003BALTRA 1985 4 31 1165 1141 1288 1581 1922 2236 2426 2451 2306 2027 1663 1303 003BALTRA 1985 4 32 1057 987 1112 1413 1813 2216 2533 2681 2604 2347 1961 1537 . . . . . . . . . . . . 003BALTRA 1985 4291 1823 2014 2145 2191 2124 1964 1740 1519 1345 1260 1294 1432 003BALTRA 1985 4292 1649 1896 2110 2245 2274 2195 2020 1787 1561 1392 1333 1387 003BALTRA 1985 4301 1549 1773 1997 2166 2231 2183 2032 1796 1545 1341 1234 1252 003BALTRA 1985 4302 1396 1638 1930 2186 2359 2402 2314 2108 1828 1550 1341 1251 99999999999999999999999999999999999999999999999999999999999999999999999999999999 99999999999999999999999999999999999999999999999999999999999999999999999999999999 003BALTRA LAT=00 26.8S LONG=090 17.2W TMZONE=GMT REF=00000 60 MAY 85 M 31 003BALTRA 1985 5 11 1301 1480 1742 2021 2242 2344 2302 2117 1840 1537 1278 1142 003BALTRA 1985 5 12 1161 1338 1638 1983 2302 2510 2555 2423 2147 1796 1456 1200 003BALTRA 1985 5 21 1093 1167 1405 1736 2076 2345 2460 2400 2173 1834 1470 1175 003BALTRA 1985 5 22 1024 1069 1307 1676 2094 2457 2679 2704 2524 2174 1736 1322 . . . . . . . . D-2 SLP64 USER MANUAL D.2 Daily Data Processing Format The filename consists of the letter "d" followed by the file version and the station number. For example, for station 027, file version a, the file name is "da027.dat". All years for the station are in this file. Header Record: NNNxxxxxxxxxxxx Daily values 119-point filter where NNN is the station number xxxxxxxxxxxx is station name The output data record format reserves the first eleven spaces for station identification, the next four for the year of the data record, the next two spaces for the month of the data record, the next space for a sequential number that positions the record within the month. The next space is a blank. The next 55 spaces contain the daily filtered values in groups of five spaces per observation, 11 per record. The data is units of mm. The averages are centered on the 12th hour GMT of the day. There are three records per month, and the last record contains the monthly average in the 11th data record group. Missing data is flagged by four nines (9999). i.e. station id year | | month | | | indicator for 1ST, 2ND, or 3RD part of the month | | | | | | | | |_________________________data_mm______________________ _|________|___|_|_|____|____|____|____|____|____|____|____|____|____|____ 01PONAPE 1983 21 538 562 574 567 573 547 555 558 548 532 538 01PONAPE 1983 22 582 600 586 587 590 606 620 592 589 592 569 01PONAPE 1983 23 574 581 590 614 638 638 580 ^ | monthly average D.3 Monthly Data Processing Format The filename consists of the letter "m" followed by the file version and the station number. For example, for station 027, file version a, the file name is "ma027.dat". All years for the station are in this file. Header Record: Consists of the station number, name, and starting year (bytes 55-58). Example. 003Baltra Data record MONTHLY MEANS 1985- D-3 SLP64 USER MANUAL Contain a record header (station number and abbrevated name in columns 1-10), year (columns 11-14), a fixed "M" in column 15, and 12 sea level values of four-bytes each in the remaining columns. Missing data given by 9999. Last Record: string of 9s. Example. station id year | | fixed | | | | | | | | ||_________________________data_mm______________________ _|________|__||___|___|___|___|___|___|___|___|___|___|___|___ 003Baltr 1985M999999991785179317791801177017551771175517541746 003Baltr 1986M180417981802183318141842182817921813185218971831 9999999999999999999999999999999999999999999999999999999999999999999999999999999 D.4 Tide Staff-Gauge Pair: File Format PAIR filenaming convention: Xsssyymm.DAT where X: fixed (always "X") sss is station number, e.g. 003 yy is the year (last two digits, e.g. 78) mm is the month The PAIR file holds the tide staff readings (highs and lows) and corresponding gauge observations. Record 1: Header with Station name and time span Record 2 TO N: Example: STATION 250 250 330 330 . . 570 570 360 360 byte contents --------- ----------------------------------1-4 highest readings 5-8 lowest readings Note if only one reading; both the same 9-13 gauge value 30 SANTA CRUZ 1281 1373 01 JUL - 01 AUG 1988 1602 1400 D-4 SLP64 USER MANUAL D.5 Calibration Log File STATION = Santa Cruz, 003 GAUGE = Fischer and Porter Analog-to-Digital Recorder (ADR) TIDE DATUM = TIDE STAFF ZERO TIDE GAUGE BENCH MARK = BM UH 1 AUXILLARY BENCH MARKS = BM UH 2, BM UH3, BM II, BM III, BM 4, BM 5, BM INOCAR 1, BM WS2 DATES -------------------01 JAN - 01 FEB 1988 01 FEB - 01 MAR 1988 01 MAR - 01 APR 1988 01 APR - 01 MAY 1988 01 MAY - 01 JUN 1988 01 JUN - 01 JUL 1988 01 JUL - 01 AUG 1988 MEAN ZERO REFERENCE LEVEL VALUE (units=feet*) ------------------1036 1037 1035 1036 1037 1035 1036 NUMBER OF OBSERVATIONS -------------------32 28 26 30 32 31 32 *units in feet to implied hundredths, ie 1036 = 10.36'. (the numbers above are made up--this is only an example) D-5 SLP64 USER MANUAL E Station Information File The station information file contains specifications for a given station that can be utilized by application programs. Each recipient of the UHSLC/NODC package has the liberty of using one’s own station numbers with the limit of three digits per number. The first record is a header to define the fields. The fields are: field --------------### bytes ---------4-6 station name 8-27 latitu 29-34 longitu 36-42 mer 44-46 S 48 strtdate 50-57 country 59-73 P 75-75 Example: ### station name 001 Ponape 002 Tarawa,Betio 003 Baltra 004 Nauru comment ----------------------------------station number - must be exactly 3 digits station name - limit to 20 bytes latitude in degrees and minutes longitude " " " hour meridian of station time scheme (0:1-24 1:0-23) (Ignore) start date of station data history (Ignore) country plot factor (obsolete, ignore) latitu 06 59N 01 22N 00 27S 00 32S longitu 158 14E 172 56E 090 17W 166 54E mer 165 180 090 165 S 1 1 1 1 strtdate Apr 1974 May 1974 Aug 1974 May 1974 country Micronesia Kiribati Ecuador Nauru P 2 2 2 2 E-1 SLP64 USER MANUAL F Notes on Foreman Tidal Analysis Program 1) Interpretation of the output file of the tidal analysis program. For each tidal component there are four columns (after the time span for analysis which is repeated on every line) corresponding to A, G, AL, and GL. The last two are the fitted amplitude (AL) and phase (GL) direct from the least squares analysis. The phase is relative to a reference time origin at 0000 01 Jan 1976. The first two are the amplitude (A) and phase (G) after applying the so-called nodal corrections (see next paragraph). Applying nodal corrections allows the fitted components to be used further from the actual time period used to fit the components. Each fitted component is in reality a sum of a major component and several satellite components that are too nearby in frequency to resolve in the least squares analysis. Therefore the AL and GL values do not accurately reflect the amplitude and phase of the major component, but show a modulation due to the satellites. By assuming that the ratio is equal to the ratio of the equilibrium components, it is possible to correct the major component to obtain the amplitude and phase that would be fit if the satellites were resolved. 2) Foreman updated his programs in 1992 to comply with change of century and to allow analysis and prediction for previous centuries. Two main changes have been made: 1) the calendar was extended for calculations outside of the 20th century, and 2) the position of the moon and sun (i.e., the astronomical arguments) are now approximated by higher order polynomials (in time) so that sufficient accuracy is maintained for dates well outside the 20th century. 3) Doodson number conventions (from Geoff Lennon) The Doodson numbers define the tidal component frequency and also determine the astronomical argument. The astronomical argument is the phase of the equilibrium tide at some reference time origin. Unfortunately, the Doodson numbers for a number of low frequency components and a few high frequency ones are not the same as those used in other programs. This causes the fit to be done at a slightly different frequency and with a different reference phase (the astronomical argument). This will not affect prediction, but will cause disparities between our analysis and those done at the IOS (UK) and the IHO. There is no simple fix for this situation since we do not know exactly which components are affected and what the preferred Doodson numbers are for each of them. F-1 SLP64 USER MANUAL G Harmonic Constants: Analysis Output 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 003 Baltra Z0 .00000000 SA .00011407 SSA .00022816 MSM .00130978 MM .00151215 MSF .00282193 MF .00305009 ALP1 .03439657 2Q1 .03570635 SIG1 .03590872 Q1 .03721850 RHO1 .03742087 O1 .03873065 TAU1 .03895881 BET1 .04004043 NO1 .04026859 CHI1 .04047097 PI1 .04143851 P1 .04155259 S1 .04166667 K1 .04178075 PSI1 .04189482 PHI1 .04200891 THE1 .04309053 J1 .04329290 SO1 .04460268 OO1 .04483084 UPS1 .04634299 OQ2 .07597494 EPS2 .07617731 2N2 .07748710 MU2 .07768947 N2 .07899925 NU2 .07920162 H1 .08039733 M2 .08051140 H2 .08062547 MKS2 .08073957 LDA2 .08182118 L2 .08202355 T2 .08321926 S2 .08333334 R2 .08344740 K2 .08356149 MSN2 .08484548 ETA2 .08507364 MO3 .11924210 M3 .12076710 SO3 .12206400 MK3 .12229210 SK3 .12511410 MN4 .15951060 M4 .16102280 SN4 .16233260 MS4 .16384470 MK4 .16407290 S4 .16666670 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 26S 090 785/ 786178.6955 785/ 786 3.7802 785/ 786 .3527 785/ 786 .7544 785/ 786 1.0036 785/ 786 .6430 785/ 786 2.0005 785/ 786 .0856 785/ 786 .1114 785/ 786 .0962 785/ 786 .3442 785/ 786 .1030 785/ 786 1.0145 785/ 786 .3383 785/ 786 .0398 785/ 786 .2466 785/ 786 .0600 785/ 786 .1035 785/ 786 2.6462 785/ 786 1.3144 785/ 786 7.8056 785/ 786 .1121 785/ 786 .1356 785/ 786 .1683 785/ 786 .5376 785/ 786 .2183 785/ 786 .6323 785/ 786 .1186 785/ 786 .1910 785/ 786 .5028 785/ 786 1.7823 785/ 786 2.5555 785/ 786 14.9349 785/ 786 3.0899 785/ 786 .7249 785/ 786 70.0426 785/ 786 .6465 785/ 786 .2083 785/ 786 .5082 785/ 786 1.5764 785/ 786 1.3464 785/ 786 18.7974 785/ 786 .3483 785/ 786 4.9431 785/ 786 .0401 785/ 786 .2544 785/ 786 .0436 785/ 786 .1659 785/ 786 .0158 785/ 786 .0193 785/ 786 .0507 785/ 786 .0261 785/ 786 .0837 785/ 786 .0304 785/ 786 .0394 785/ 786 .0116 785/ 786 .0614 17W .00 109.26 117.53 62.29 13.21 25.45 32.02 107.06 122.13 148.30 170.27 150.67 47.51 24.71 310.27 344.70 339.19 64.26 51.36 16.16 49.21 202.07 40.90 52.94 69.84 167.59 87.87 113.25 173.92 164.52 188.99 193.17 217.48 222.69 86.90 246.43 79.72 84.68 269.73 280.41 301.44 292.37 284.44 286.66 202.46 289.95 218.71 356.64 274.13 109.26 14.73 359.37 64.54 322.51 116.72 100.18 209.11 178.6955 3.7802 .3527 .7544 1.0036 .6430 2.0005 .1100 .1575 .1145 .4370 .1169 1.1655 .2187 .0474 .2599 .0698 .1028 2.6214 .8941 8.5533 .1138 .1264 .1881 .8075 .2512 .9812 .2156 .1815 .4839 1.6968 2.4735 14.4488 2.9874 .6907 67.9384 .6352 .2550 .4899 1.6704 1.3464 18.8300 .4250 6.2274 .0377 .3487 .0486 .1585 .0182 .0205 .0557 .0245 .0787 .0295 .0383 .0142 .0616 .00 112.59 278.78 239.82 99.16 288.92 96.75 214.71 42.03 344.99 179.94 74.11 148.29 113.63 47.30 311.91 159.43 70.57 61.10 349.24 44.58 196.65 192.38 231.27 165.32 66.73 171.10 278.08 14.83 273.35 115.44 27.63 229.16 142.91 358.86 344.16 180.32 353.46 5.34 303.40 298.11 292.29 102.53 97.63 288.43 189.60 57.22 143.42 14.83 202.36 10.01 108.78 260.00 334.11 214.37 8.89 208.95 G-1 SLP64 USER MANUAL 58 59 60 61 62 63 64 65 66 67 68 SK4 2MK5 2SK5 2MN6 M6 2MS6 2MK6 2SM6 MSK6 3MK7 M8 .16689480 .20280360 .20844740 .24002200 .24153420 .24435610 .24458430 .24717810 .24740620 .28331490 .32204560 0 0 0 0 0 0 0 0 0 0 0 785/ 785/ 785/ 785/ 785/ 785/ 785/ 785/ 785/ 785/ 785/ 786 786 786 786 786 786 786 786 786 786 786 .0182 .0124 .0068 .0311 .0407 .0819 .0306 .0326 .0212 .0023 .0205 114.89 143.05 238.65 357.93 337.75 37.54 17.82 64.01 108.36 348.03 302.46 .0230 .0128 .0075 .0283 .0371 .0771 .0362 .0317 .0259 .0023 .0182 285.78 333.88 233.86 205.08 270.95 232.93 24.25 161.58 16.98 276.60 333.39 G-2 SLP64 USER MANUAL H Make Predictions other than Hourly 1) edit tideprd.bat just before tidep.exe (before run tide prediction) place a "pause" command. 2) run tideprd.bat hit cntrl-C when pause is hit (the one just before tidep) 3) edit TIDEDATA on last non-blank record, in field just after EQUI, enter fraction of hour (default is hourly = 1.000) e.g. for 15 minutes change it to 0.25000) its fortran so be byte specific (don’t delete any zeros). 4) now just type TIDEP, which runs prediction. Your output will be in eqsprd.dat in the Foreman format. H-1 SLP64 USER MANUAL I Test for Magnitude of Timing Error One way to determine the magnitude of a timing error is to compare observed data to predicted tides. In separate windows for the same time span with the suspect timing error, place the observed data and predicted files. For example: Window 1: Observed File, VA00386.DAT, 14-16 November, 1996 003BALTRA 003BALTRA 003BALTRA 003BALTRA 003BALTRA 003BALTRA 198611141 198611142 198611151 198611152 198611161 198611162 1243 1120 1258 1129 1357 1239 1389 1166 1269 1050 1227 1029 1684 1399 1463 1177 1301 1026 2055 1751 1793 1477 1547 1230 2416 2131 2176 1867 1905 1573 2683 2454 2523 2262 2291 1977 2767 2637 2730 2564 2596 2351 2645 2627 2748 2683 2733 2592 2360 2437 2559 2604 2660 2651 ^ | Note high tide at hour 20 of 1983 2124 2211 2356 2402 2511 1591 1761 1799 2002 2016 2209 1274 1439 1405 1638 1596 1841 11-16-1986 Window 2. Predicted Tides File, PA00386.DAT, 14-16 November, 1996 003Baltra 003Baltra 003Baltra 003Baltra 003Baltra 003Baltra 198611141 198611142 198611151 198611152 198611161 198611162 1277 1219 1402 1372 1607 1592 1238 1068 1223 1093 1314 1222 1384 1120 1235 1010 1190 1011 1682 1358 1437 1140 1268 1014 2059 1718 1778 1446 1528 1226 2420 2110 2174 1850 1903 1588 2675 2439 2525 2254 2300 2014 2752 2621 2737 2559 2615 2401 2627 2608 2747 2685 2759 2651 2333 1939 1536 2412 2088 1720 2552 2199 1773 2605 2348 1985 2689 2423 2027 2701 2547 2236 ^ | Note high tide at hour 21 of 11-16-1986 As can be seen above, it appears that the observed data file lags the predicted tides by one hour. Thus, one can apply the timing error correction technique to shift the observed file for this time span by +1 hour. I-1 SLP64 USER MANUAL J Figures Figure 1 Observed hourly data showing typical semi-diurnal tides. J-1 SLP64 USER MANUAL Figure 2 Observed hourly data showing data gaps and suspicious spikes in December. In choosing a time span for tidal analysis, it is best to choose 366 days of continuous gap-free, apparently good data. In review of the previous plot for 1985 in addition to this plot, a suitable span for analysis would be from 00Z April 1, 1985 to 23Z April 1, 1986. J-2 SLP64 USER MANUAL Figure 3 Residuals are computed from observed minus predicted values. This plot shows smooth residuals, which indicates a good tidal analysis and clean data. Minor phase shifts in time are responsible for the small fluctuations during April and May. J-3 SLP64 USER MANUAL Figure 4 Residuals revealing timing errors of varying magnitude, which are especially obvious in November and December. Also, short gaps, data spikes, and a reference level shift are apparent in December. J-4 SLP64 USER MANUAL Figure 5 Scatter diagram for tide staff readings and tide gauge measurements. In this case, the pairs are good except for one outlier due to a bad staff reading. This point will be removed from the pair file prior to computing the mean difference. J-5 SLP64 USER MANUAL Figure 6 Scatter diagram for tide staff readings and tide gauge measurements for case with clogged stilling well. J-6 SLP64 USER MANUAL Figure 7 Scatter diagram for tide staff readings and tide gauge measurements. In this case, a reference level shift occurred due to resetting of the gauge zero during a site maintenance visit. J-7 SLP64 USER MANUAL Figure 8 Residuals of hourly data for Kelang, Malaysia. At this site, the tides are highly complex due to the shallow shelf and narrow basin configuration. The tidal analysis was unable to resolve many of the higher frequency tidal species. The questionable fluctuations in the residuals are caused by poor predicted tides. The observed data are of good quality. J-8 SLP64 USER MANUAL Figure 9 Residuals based on tidal analysis of observed data from 12/1991-12/1992. The fluctuations are due to poor tide predictions. J-9 SLP64 USER MANUAL Figure 10 Residuals based on tidal analysis of observed data from 1/1995-1/1996. In this case, the residuals are smoother because the tidal analysis is modelling the same time period as plotted above. It appears that the complex tides at this site are not fully resolved by the tidal anaysis; thus, the quality of the predicted tides drops as the predicted year moves away from the analyzed year. This site requires frequent updates of the harmonic constants. J-10 SLP64 USER MANUAL Figure 11 The observed hourly data file of Figure 4 was shifted by positive one hour (Appendix I above) using program TSALL. (The program shifts the entire year). The resultant residuals are shown above. The time spans of the original data file without a timing error now have fluctuations, but the time span of the suspect timing shift in November is now smooth. J-11 SLP64 USER MANUAL Figure 12 Daily difference plots between version/station a076 (Taranaki) and r071(Wellington). The good agreement suggests high data quality in the calibration of each series to its reference zero. There is also interesting oceanic variability seen on various time scales in the difference plot. J-12 SLP64 USER MANUAL Figure 13 Monthly difference plots between version/station a076 (Taranaki) and r071(Wellington). J-13