ABB Technical Support OperateIT B.0 Trouble Shooting Guide (ABB Technical Support) General Information The module goal is to provide the ABB Employees basic information to assist in the process of trouble shooting and/or the investigation of OperateIT problems. Description This module will focus on the process of verifying the configurations, operation, view log files and using utilities and tools available in troubleshooting problems. This module will also focus on the collection of information that may be required by Technical Support and Engineering in the investigation of problems in OperateIT. Areas covered: Software Installation Replication Alarm Management Historian RTDSs OperateIT Diagnostics Server Diagnostics Registry Objectives Upon completion of the Trouble Shooting module the student will be able to: Verify OperateIT software installation. Use of Server Diagnostic Use the OperateIT Diagnostic Tool Determine what files contain appropriate information for troubleshooting problems. 533583860 Version A1 1/18/2002 1-1 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Table of Contents OperateIT B.0 Trouble Shooting Guide (ABB Technical Support) ............................................................1 1.1 General Information .........................................................................................................................1 1.1.1 Description ................................................................................................................................1 1.1.2 Objectives ..................................................................................................................................1 1.1.3 Table of Contents ......................................................................................................................2 1.2 Trouble Shooting Guide ...................................................................................................................5 1.3 Base Trouble Shooting .....................................................................................................................6 1.3.1 Installation .................................................................................................................................6 1.3.2 Navigator ...................................................................................................................................7 1.3.3 Navigator Launch Problems .................................................................................................... 10 1.3.4 Line 46 error ............................................................................................................................ 10 1.4 Alarm Management ........................................................................................................................ 11 1.4.1 Historical Events ..................................................................................................................... 11 1.4.2 Event Bar Behavior ................................................................................................................. 12 1.4.3 Tones ....................................................................................................................................... 13 1.4.4 Annunciator Tone Configuration (TB20000092A) ................................................................. 20 1.4.5 ADP Annunciator Device and Panel ....................................................................................... 22 1.4.6 Alarm Quality.......................................................................................................................... 25 1.5 Data Capture Procedures ................................................................................................................ 26 1.5.1 Items to Collect for Navigator Problems ................................................................................. 26 1.5.1.1 Turning on OPW Console Window ................................................................................. 29 1.5.1.2 Running the DBWin32 Application ................................................................................. 30 1.5.2 Items to Collect for Historian Problems .................................................................................. 31 1.5.3 1 - Items to Collect for RTDS Problems ................................................................................. 32 1.5.3.1 2 - Items to Collect for Subscription Problems ................................................................ 32 1.5.3.2 RTDS/Subscription Data Collection Procedures .............................................................. 33 1.5.4 Data Capture for ProduceIT problems .................................................................................... 35 1.6 Diagnostics ..................................................................................................................................... 36 1.6.1 ABB Diagnostic (Hidden) ....................................................................................................... 36 1.6.1.1 Diagnostic Options ........................................................................................................... 37 1.6.1.2 Diagnostic Output............................................................................................................. 38 1.6.1.3 Display Invocator ............................................................................................................. 39 1.6.1.4 Execute Script .................................................................................................................. 43 1.6.1.5 Mirror Database................................................................................................................ 43 1.6.1.6 Navigator Options ............................................................................................................ 46 1.6.1.7 Operator Window ............................................................................................................. 46 1.6.1.8 Session Variables ............................................................................................................. 47 1.6.1.9 Tone Diagnostics .............................................................................................................. 49 1.6.1.10 Verb Cycler .................................................................................................................... 49 1.6.1.11 Window Manager Test ................................................................................................... 50 1.6.1.12 Window Test .................................................................................................................. 50 1.6.1.13 Windows Scripting Host Test ......................................................................................... 51 1.6.1.14 Windows Scripting Host Test (Server) ........................................................................... 52 1.6.2 OperateIT Diagnostics .............................................................................................................. 53 1.6.2.1 Application Events ........................................................................................................... 54 1.6.2.2 DBWin32 ......................................................................................................................... 54 1.6.2.3 Diagnostic Browser Utility ............................................................................................... 54 1.6.2.4 Diagnostic Dump Utility .................................................................................................. 54 1-2 533583860 Version A1 ABB Technical Support 1.6.2.5 Event Counts .................................................................................................................... 54 1.6.2.6 Event Group Bar .............................................................................................................. 54 1.6.2.7 Event List Comparison .................................................................................................... 55 1.6.2.8 Event Page ....................................................................................................................... 55 1.6.2.9 Event Stream from Concentrator ..................................................................................... 55 1.6.2.10 Event Stream from IEbEventServer ............................................................................... 56 1.6.2.11 Get Client ....................................................................................................................... 58 1.6.2.12 OPC Alarm Client .......................................................................................................... 59 1.6.2.13 Put Client ....................................................................................................................... 59 1.6.2.14 Security Diagnostic ........................................................................................................ 61 1.6.2.15 Sequence Event Bar ....................................................................................................... 64 1.6.2.16 Server Broker List .......................................................................................................... 64 1.6.2.17 Server Explorer .............................................................................................................. 65 1.6.2.18 Stream Client ................................................................................................................. 67 1.6.2.19 TimeSync Diagnostics ................................................................................................... 68 1.6.2.19.1 Time Sync Messages ............................................................................................... 71 1.7 Databases ....................................................................................................................................... 72 1.7.1 Mirror and RTDS databases keep growing ............................................................................. 72 1.8 Event View Logs ........................................................................................................................... 72 1.9 Harmony ........................................................................................................................................ 77 1.9.1 AtomWEB .............................................................................................................................. 80 1.9.2 ICI Error.................................................................................................................................. 80 1.9.3 Operation Parameters .............................................................................................................. 81 1.9.3.1 General Page .................................................................................................................... 82 1.9.3.2 Harmony Page .................................................................................................................. 83 1.9.3.3 Station Page ..................................................................................................................... 84 1.9.3.4 Substitution Page ............................................................................................................. 89 1.9.3.5 Red Tag ............................................................................................................................ 90 1.9.4 Block Details........................................................................................................................... 92 1.9.4.1 Specification Tab ............................................................................................................. 93 1.9.4.2 Output Tab ....................................................................................................................... 96 1.9.4.3 Description Tab ................................................................................................................ 97 1.9.5 Module Status ......................................................................................................................... 99 1.9.5.1 General Tab.................................................................................................................... 100 1.9.5.2 Status Tab ...................................................................................................................... 101 1.9.5.3 Problem Tab ................................................................................................................... 102 1.9.5.4 Technical Tab ................................................................................................................ 103 1.9.6 Tag Summaries ..................................................................................................................... 105 1.9.7 I90Server failure ................................................................................................................... 109 1.9.8 Unable to Load Tags ............................................................................................................. 110 1.10 Help File .................................................................................................................................... 111 1.11 Historian .................................................................................................................................... 112 1.11.1 Server Explorer Historian Diagnostics................................................................................ 112 1.11.2 Historian Syslog.log ............................................................................................................ 115 1.11.2.1 Historian Syslog entries for UTC time error. ............................................................... 120 1.11.2.2 Syslog buffer allocated vs buffer used ......................................................................... 120 1.11.3 Archive Syslog .................................................................................................................... 120 1.11.4 SQL Enterprise Manager .................................................................................................... 124 1.11.5 Historian Master ................................................................................................................. 127 1.11.5.1 Tags Table.................................................................................................................... 128 1.11.5.2 Atoms Table ................................................................................................................. 128 533583860 Version A1 1/18/2002 1-3 ABB Technical Support Trouble Shooting Guide Operate IT B.0 1.11.5.3 Event Policy Table ....................................................................................................... 129 1.11.5.4 File Policy Table .......................................................................................................... 130 1.11.6 ConfigServer ....................................................................................................................... 130 1.11.6.1 Pull_Time Table ........................................................................................................... 131 1.11.6.2 Tags_Modify ................................................................................................................ 131 1.11.6.3 Atoms_Modify ............................................................................................................. 133 1.11.6.4 Event_Policy_Modify .................................................................................................. 133 1.11.6.5 Event_Policy_Modify .................................................................................................. 134 1.11.6.6 “Delete” Procedure ....................................................................................................... 134 1.11.6.7 SPLOADALLMANUAL.SQL ..................................................................................... 135 1.11.7 Historian_1 .......................................................................................................................... 136 1.11.8 SQL Query Analyzer ........................................................................................................... 137 1.11.9 Check Script Files ............................................................................................................... 146 1.11.10 Historian RAW/BCBS Historization check ...................................................................... 150 1.11.11 RowSetViewer .................................................................................................................. 151 1.11.11.1 Tag_Atom Data .......................................................................................................... 154 1.11.11.2 Event Data .................................................................................................................. 157 1.11.11.3 Filter Call on EventPoint ............................................................................................ 161 1.11.11.4 File Data ..................................................................................................................... 163 1.11.12 Provider Example.txt File.................................................................................................. 164 1.11.13 Historian Databases – Suspect Error ................................................................................. 166 1.11.13.1 Historian Server always Establishing In Server Explorer screen ............................... 167 1.11.13.2 Historian Ebsp_Config_Pull_All ................................................................................ 167 1.11.13.3 Historian data rates ..................................................................................................... 168 1.11.14 Performance Monitoring ................................................................................................... 168 1.11.15 Historian Performance Calculation ................................................................................... 174 1.11.16 Historian Diskspace Calculation ....................................................................................... 176 1.11.17 Historian Faceplate show RTDS Failure ........................................................................... 177 1.12 Licenses ...................................................................................................................................... 177 1.12.1 Dr. Watson Error on License Entry ..................................................................................... 179 1.13 Lock Object ................................................................................................................................ 179 1.14 Network ...................................................................................................................................... 181 1.14.1 Multiple Network Card Applications .................................................................................. 181 1.15 Performance ............................................................................................................................... 182 1.15.1 OperateIT slower after Mirror and RTDS Databases grow. ................................................ 182 1.16 Registry ...................................................................................................................................... 182 1.16.1 Installed Products ................................................................................................................ 183 1.16.2 Symphony Common ............................................................................................................ 186 1.16.3 Server Broker ...................................................................................................................... 191 1.16.4 HostList ............................................................................................................................... 192 1.16.5 Symphony Historian ............................................................................................................ 197 1.16.6 Service Pack 4 Installation Identification ............................................................................ 198 1.17 Replication ................................................................................................................................. 199 1.18 Reports ....................................................................................................................................... 201 1.19 Security ...................................................................................................................................... 204 1.19.1 Microsoft Security Accounts ............................................................................................... 204 1.19.2 Security Diagnostic (See OperateIT Diagnostics) ............................................................... 205 1.19.3 SQL Database Access ......................................................................................................... 205 1.20 SQL Per Seat Configuration ....................................................................................................... 208 1.21 Task Manager ............................................................................................................................. 209 1.22 Time Syncing ............................................................................................................................. 211 1-4 533583860 Version A1 ABB Technical Support 1.22.1 Ethernet Daemon ................................................................................................................ 211 1.22.2 Harmony RTDS Time Master ............................................................................................. 212 1.22.3 OPC Time Sync .................................................................................................................. 215 1.22.4 System Time Adjustment .................................................................................................... 216 1.22.5 Set Operate IT Time............................................................................................................ 218 1.22.6 Time Sync Diagnostic ......................................................................................................... 219 1.23 Trends ........................................................................................................................................ 220 1.23.1 Trend Update Times ........................................................................................................... 221 1.24 User Dump ................................................................................................................................. 222 1.25 Windows NT Troubleshooting Tools ........................................................................................ 223 1.25.1 Dr Watson Log.................................................................................................................... 223 1.25.2 Blue Screen Dump .............................................................................................................. 224 1.25.3 Windows NT Diagnostics ................................................................................................... 225 1.25.4 Performance Monitor .......................................................................................................... 226 1.25.5 Event Viewer ...................................................................................................................... 227 1.25.6 Task Manager ..................................................................................................................... 228 1.25.7 Network Monitor ................................................................................................................ 229 1.25.8 Additional Resources .......................................................................................................... 230 1.26 Apendix A EbHresults .............................................................................................................. 231 Trouble Shooting Guide The purpose of this module is to assist the user in the verification of the system’s operation, the investigation of system problems and collection of data for further investigation by Technical Support and Engineering. The document is structured major topics in alphabetical order. Throughout the document references may be spelled out in ( ). Ex (See License). This indicates the additional information or detailed information on the topic is located in the section by the indicated name. IN those cases, use the table of contents to locate the topic. 533583860 Version A1 1/18/2002 1-5 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Base Trouble Shooting Most issues with Operate IT occur during the initial Installation and Startup of Operate IT. The first section of the document will cover initial troubleshooting for new or recent installations and Navigator issues. These items are a good foundation for verifying basic configurations and setups prior to going into specific intensive investigation of a problem for systems that are not in the initial load and startup phases. Installation For new installation a large number of problems can be contributed to the Installation of the OperateIT software. The following is a checklist of things to check to verify the installation. 1. Internet Explorer (Properties Configuration) Verify the configuration of the Internet Explorer. Reference the Introduction and Installation manual WBPEEUI220793B0. 2. DCOM (Configuration) Verify the configuration of DCOM. Reference the Introduction and Installation manual WBPEEUI220793B0. 3. SQL Server is configured for “Per Seat” (See SQL Server Per Seat Configuration) 4. Mirror and RTDS Site configurations for each Node. (See Navigator) 5. Check Microsoft Security configuration for (See Security): OperateService Act as part of the operation system Generate security audits Log on as a service LocalUserGroup Generate security audits 6. Verify OperateIT licenses are loaded (See License). 1-6 533583860 Version A1 ABB Technical Support Navigator Most Navigator problems initially are the result of the configuration issues with: Internet Explorer DCOM Mirror Site Internet Explorer Configuration Verify the configuration of the Internet Explorer. Reference the Introduction and Installation manual WBPEEUI220793B0. DCOM Configuration Verify the configuration of DCOM. Reference the Introduction and Installation manual WBPEEUI220793B0. Mirror Site The OperateIT Navigator will not startup if the PC does not have or find a Mirror Site database. Verify that a mirror site exist and if the PC has a current Mirror Site Database. There are three things to check: 1. SysSystemNode table in the ConfigServer Database The SymSystemNode should have an entry for each node in the system. Some nodes may be listed more than once. This would be the result if there is more than one life-cycle for a node object (Design, Out of Server, Release, Running). 533583860 Version A1 1/18/2002 1-7 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Under the Mirror column should be “1” for each node that is configured to have a Mirror Site. 2. Use the Replication Manager in SQL Enterprise Manager to verify if the Node has been replicated a mirror site. SQL Replication Monitor Check the ConfigServer_Mirror Agents for a listing to the PC having a problem with the navigator not starting up. It should be listed, have a status of Succeeded, and a recent date. 1-8 533583860 Version A1 ABB Technical Support 3. Use NT Explorer to verify a database exists on the Node. \Mssql7\Data 533583860 Version A1 1/18/2002 1-9 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Navigator Launch Problems Start Error Press “Launch Navigator” to continue message will appear if the Access Rights for a local user group has not been configured. If the you see the navigator screen shown below, check in SQL Enterprise Manager the ACCESS RIGHTS configuration shown in the Introduction and Installation Manual page 5-16 for the Computer on which you experience this problem. If the Local User group that the logged in user is a member is not configured for access to the databases on that machine, you will get the error shown below. Line 46 error Occasionally the following error message is seen when launching the Navigator screen. To correct the problem try going into the Internet Explorer and click on Language and then OK. If this fails then remove English and reinstall. 1-10 533583860 Version A1 ABB Technical Support Alarm Management Alarm Management controls Event and the action of Tones and ADP Panels. This section provides information in regards to Historical Events, Event Bar Behavior and troubleshooting of Tone and ADP problems. Historical Events Built in the Event Page of OperateIT is a Historical Event Page. The Historical Event page displays information that has been configured to be historized. Also, each server in OperateIT has a set of server initiated events that can also be historized and can be helpful in diagnosing system problems. Entries in the Event Page of interest will be events by servers that may relate to a given problem. For example if a problem is occurring in an RTDS. Look for event entries with the sender being the RTDS server. Then review the event for any information that may be helpful to the user. This section provides information to create a Diagnostic event page for troubleshooting purposes. Diagnostic Event Page Log To generate a diagnostic (DG) Event page performs the following steps. 1) Create an Event Group for Diagnostic and System Events. On this page under the Event Types heading, click on DG and SY. 2) Create a Historian Event Storage Policy. Go to the Event Collection tab and in the Filter Group input field type in the name of the Event Group from Step 1 above. Also, under this tab, you must assign the Historian Storage Policy. 3) Create a Display Event Page. In the Relations input field enter the Event Group name from Step 1. 4) The Historian Server Object has an EVENT Policy field that requires an entry for the historian to historize events. IF this field is blank the diagnostic log may not work. This completes the configuration section of this process. To view the event page, call up the Display Event Page that you created in Step 3 above. Once the page is on the display, click on History. This will bring up the Event Page that contains Diagnostic and System Event data. Bear in mind hat this display will not dynamically updated. If you want to see new data, you must Refresh the display. 533583860 Version A1 1/18/2002 1-11 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Event Bar Behavior An Event Group Event Bar has a numeric indicator in each of the Event Group Boxes, which represents a priority for an event in the associated Event Group. The function of the boxes and the priority within the boxes are dependent on three factors and prioritizes in the following order: 1. Unacknowledged state of any event in the associated event group 2. Highest priority event within the group (1 being the highest) 3. Most recent time. The Event Group box will flash if there is at least one unacknowledged event in the associated Event Group. If any event is unacknowledged the color and the priority number in the numeric indicator box will represent the unacknowledged event. If more than one event is unacknowledged, then the priority states are compared and the highest priority number and the active color for the event will be displayed. If more than one event exist for the highest priority found, then the time stamps are compared and the priority number and color for the event with the most recent time will be displayed. If all events in the associated Event Group have been acknowledged the numeric indication and color will be based on factors 2 and 3. The Aspects view for the active event associated to the numeric indicator is displayed by right clicking on the numeric indicator box. The Aspects view for the associated Event Group is displayed by right clicking on the area surrounding the numeric indicator box. The Primary Verb of the associated Event Group is launched by clicking on either the numeric indicator box or surrounding area, provided a primary verb is configured for the associated Event Group. 1-12 533583860 Version A1 ABB Technical Support Tones Tones Problems The following view of faceplates and of an event page is an example of objects to display in the Navigator view to assist in troubleshooting tone problems. When troubleshooting tone problems it is very helpful to be able to view the different objects related to the tone experiencing problems or for testing tones. In this view starting from the top right is the Tone object. The tone object is triggered by a Trigger object, which is to the left of the tone object. The trigger object it triggered by the Event Filter Function block which is to the left of the trigger object. The Event Filter Function block is controlled by an Event Group (Not shown) which, defines what tag objects are associated to the Event Filter block and tones. 533583860 Version A1 1/18/2002 1-13 ABB Technical Support Trouble Shooting Guide Operate IT B.0 The RCM object is a test block for this example. The RCM is an object within the Event Group that can initiate an event to test the logic. The Event Page is to view new event that occur for the Event Group that should trigger a tone. When the RCM is toggled or when a new event occurs in the Event Page, the following things should occur. 1. The Event Filter Function block should register an increase in the number of Event Counts and Unack Counts. The Event and Event Unack fields should be “1” state if any event exist and if any of the event are not Ack. 2. The Trigger object should receive a trigger from the Event Filter Function block. Which in turn should set the Output field to “ON”. 3. The Tone object should generate a tone and the Sound Loop should be set to “1”. Verify the Tone generation by either initiating an event that should trigger a tone or have the system generate the event if the Real Time objects are active. The first thing to watch is the status of the Event Filter Function Block. This faceplate should display information indicating if there are any active events and if they are periodically occurring. If the process is stable, then perform a page acknowledge on the event page to clear some events. When the event start to reoccur, the Event Count and Unack Counts should increase. If the counts change, then monitor the Trigger block to see if the Output field goes to “ON” State. If not, then the problem is between the Event Filter block and the Trigger block. Check the Input Trigger tag.atom configuration for the Trigger block. If the Trigger Block is Ok but still no tone, check the configuration of the Tone object and test the tone object. 1-14 533583860 Version A1 ABB Technical Support Tone Testing When a tone is not working the first thing to test is the Tone Object. Call up the Faceplate for the tone and test the tone. The Tone Faceplate displays the Sound File assigned to the tag object. In the bottom of the faceplate for the Tone tag is Tone Enable button. This is the long button on the left side of the faceplate. Enable Button states. “0” Disable tone Enable Tone Test “1” Enable tones Disable Tone Test If the Tone Enable button is set to “0” the user can click on the “1” state button (Right most) and then click the enter key (Arrow) and the tone should sound if working. If the Tone Enable button is set to “1” the Operation System has control of the tone tag and the Test Function is disabled. 533583860 Version A1 1/18/2002 1-15 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Sound File Test If the tone fails to sound the next step is to try and initiate a tone by double clicking on the Sound file using NT Explorer. \Symphony\default\Maestro NT\Navigation\Sound Locate and double click a sound file. If the sound occurs then the sound driver is OK and the problem is elsewhere. If the sound file does not play, a message may appear indication that there is not a sound driver installed on the PC or the sound configuration needs to be verified. Sometimes the volume may be set very low and or the speakers (if external) may not have power or the volume control is turn down all the way. 1-16 533583860 Version A1 ABB Technical Support If the Sound file operate ok, then look at the System Server object. SystemServer Diagnostic View The System Server controls the tone object. When a tone is triggered and should be sounding the SOUND/EN should be a “1” and the SOUND/FILE atom should list a Sound file (ex. Foghorm.wav). 533583860 Version A1 1/18/2002 1-17 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Trigger Object If the tone object checks out Ok and tones are still not being generated by events, check the Trigger object configuration and operation. Monitor the Trigger Faceplate for the Trigger Object. If the Output field in the Faceplate is not going to the “1” state on an event check the Trigger input. The Function Block Server controls the Event Filter Function block and Trigger object. Call up the Function Block Server Diagnostic and verify the Event Filter Function Block object and Trigger objects are listing in the Tags tab of the Diagnostic view. If running redundant Fbservers verify status and listing of both servers. FBServer Diagnostic View If all configuration appear to be in order for both the System Server an Function Block Server, try Failing over the Redundant servers and/or restart the PC on which, the servers reside. 1-18 533583860 Version A1 ABB Technical Support The System Server should reside on the PC that the tones are to occur. The Function Block Server can be located on any machine. Check the Server Explorer of Object configurations to determine the location of the servers. Event Filter Function Block Operation 533583860 Version A1 1/18/2002 1-19 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Annunciator Tone Configuration (TB20000092A) The following information is in regards to the configuration of the Event Filter Function block for Tones in the Tone Priority scheme shown below. This dialog explains a recommended configuration scheme using the Event_Unack output of the Event Filter Function Block and other associated configurations to properly set up the alarm tone function. (TB20000092A) Annunciator Tone Configuration Notes To generate audible tones based on process alarms, it is recommended that an Annunciator Tone object be driven by the output signal of a FB Trigger object, which in turn is driven by the acknowledge output of a FB Event Filter object. Specifically, the Annunciator Tone’s source trigger atom should be configured to refer to the OUTPUT/SIG atom of a FB Trigger object and that FB Trigger’s input should be configured to refer to the EVENT_UNACK atom of a FB Event Filter object. This will cause unacknowledged events, which match the FB Event Filter’s filter criteria, to generate audible tones. Since it is not usually desirable to have status events, inhibited tags, or inactive alarm conditions (i.e. return-to-normal conditions) trigger audible tones it is strongly recommended that such conditions be appropriately filtered out. This is accomplished by properly configuring the FB Event Filter’s associated Event Group filter object. Specifically, the Event Group’s UNINHB, ACT and ALMEN state checkboxes should be checked, and the INHB, INACT and STATUS state checkboxes should be cleared. This will cause only uninhibited, active alarm events to be processed by the associated FB Event Filter object. If inactive alarm conditions are not filtered out, as describe above, event-point state transitions from active to inactive and back to active will not cause silenced or expired Annunciator Tones to automatically re-trigger. In other words, event-point state changes do not automatically cause FB Trigger outputs to “fire” in this configuration scenario. An event’s acknowledge state and whether or not the event matches the associated Event Group’s filter criteria determines when the FB Trigger output is “fired”. To prioritize audible tones, it is recommended that the FB Trigger objects be chained or cascaded together to achieve the desired results. When the INHIBIT_OUT atom is connected to the INHIBIT_IN atom of a cascaded block, a higher “priority” tone will activate, deactivating any lower or cascaded tone. A similar action is achieved when the RESET_OUT atom is connected to the RESET_IN atom. In this case when a silence is done to the highest FB Trigger object, the signal is passed through the chain silencing the playing tone. Tone Priority Configuration 1-20 533583860 Version A1 ABB Technical Support 533583860 Version A1 1/18/2002 1-21 ABB Technical Support Trouble Shooting Guide Operate IT B.0 ADP Annunciator Device and Panel The Annunciator Device is an IIMKM02 board that is connected via a serial port to a PC. Up to four hard ADP Panels can be connected to the IIMKM02 assembly. Within the ADP Panels is an address switch which, is set for the device address. If more than one panel is connected to the IIMKM02 board each panel needs to have a unique address (1,2,3,4). Annunciator Device There are no special configuration within the PC that needs to be performed after connecting up the Annunciator Device to the PC. Providing the Annunciator Device object has been configured for the correct serial port, all that should be required is restarting the PC that the device is connected to. The Annunciator Device Object is called during the startup of the PC. Once the Client Machine is up and the navigator is available use the Annunciator Device Faceplate to test the ADP panel operation. 1-22 533583860 Version A1 ABB Technical Support Faceplate The Faceplate for the Annunciator Device provides the ability to test both the Contact relays and ADP panels associated to the Annunciator Device. ADP Panel Test Click on the PT button and the Click Enter to start the panel test. The lights on the ADP panels connected to the Annunciator Device should sequence through a series of patterns. Contact Relay Test The Faceplate has six Relay Enable buttons (Long Button on the Left Side) to control the state of the six relays within the IIMKM02 board. The “1” state enable the relay for external operation (OperateIT). The “0” state disables the relay and puts the relay in the Test mode which allow the user to manually activate the relay. Note: While testing the relay any external device connected tot he relay will be activated during the test. Disconnect field wiring for safety reasons if necessary. Enable Button states “0” Disable Relay Enable Relay Test “1” 533583860 Version A1 1/18/2002 Enable Relay 1-23 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Disable Relay Test The two buttons next to the Enable button are to select the state to be tested. “0” De-energize the relay “1” Energize the relay The Diagnostic from the Server Explorer can be used to look at the Relay States for the Annunciator Device. The Diagnostic will be under the SystemServer for the PC that is connected to the Annunciator device. Diagnostic View Each Relay will have three associated Atoms: 1-24 OUTPUT1/EN Output Enable state. OUTPUT1/REF Tag.Atom driving the contact relay OUTPUT1/SIG Relay output state. 533583860 Version A1 ABB Technical Support Alarm Quality Alarm Quality Indicators Character Quality Description Bad Quality Strings * Bad, Non-specific, Device failure, or Sensor failure . Bad, Configuration error ? Bad, Not connected / Bad, Last know value @ Bad, Communication failure Bad, Out of service Uncertain Quality Strings ! Uncertain, non-specific, Last usable vale, Sensor not accurate, Engineering units exceeded, or Sub-normal. Good Quality Strings Good, Non-specific & Good, Local override _ Default String Increasing/decreasing indicator Strings Not Increasing, Not decreasing, Not amended, Not annotation ^ Value increasing Value decreasing + Amended # Annotation _ Default String 533583860 Version A1 1/18/2002 1-25 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Data Capture Procedures Items to Collect for Navigator Problems For Navigator problems, please collect the following information: 1. Capture a bitmap of Internet Explorer. If you think that the Navigator may be hung, capture this before bringing up other windows (such as Task Manager) on top of the Navigator. a) Press the Print Screen key on the keyboard to capture a bitmap of the screen. b) Run the Microsoft Paint application. Display the Start menu by pressing the Windows key on the keyboard, which is located between the Ctrl and Alt keys. From the Start menu, select the Programs menu and then the Accessories menu, and finally click on the Paint item. Alternatively, the Paint application can be launched by clicking on the Run... item in the Start menu, and then typing MSPAINT and clicking on the OK button. c) Paste the bitmap into the Paint application. The Paste command is available from the Edit menu in Paint. d) Save the file by using the Save item from Paint's File menu. 2. Capture a bitmap of Task Manager showing all processes. a) Launch Task Manager. Press the Windows key on the keyboard to display the Start menu and the Task Bar. Click on the Task Bar to hide the Start Menu. Right click on the Task Bar and select the Task Manager... item from the menu that appears. b) Click on the Processes tab in Task Manager to display the list of processes that are currently running. c) Make the Task Manager window large enough so that no scroll bars are displayed. If this is not possible then you will need to capture multiple bitmaps to show all of the processes. d) Make sure that the Task Manager window has focus (i.e. it's title bar has a blue background). e) Hold down the Alt key on the keyboard and press the Print Screen key. Using the Alt key with the Print Screen key captures only the window that has focus instead of the entire screen, resulting in a smaller bitmap. f) Run the Microsoft Paint application as detailed in the first item above. g) Paste the bitmap into the Paint application. The Paste command is available from the Edit menu in Paint. h) Save the file by using the Save item from Paint's File menu. 3. For a Navigator hang, obtain a process dump of Internet Explorer using the userdump utility. a) Open a Command Prompt window. Display the Start menu by pressing the Windows key on the keyboard, which is located between the Ctrl and Alt keys. From the Start menu, select the Programs menu and then click on the Command Prompt item. Alternatively, a Command Prompt window can be launched by clicking on the Run... item in the Start menu, and then typing CMD and clicking on the OK button. b) Change to the directory containing the userdump program. This may be the C:\User dump directory. c) Launch the Task Manager if it is not still open. Press the Windows key on the keyboard to display the Start menu and the Task Bar. Click on the Task Bar to hide the Start Menu. Right click on the Task Bar and select the Task Manager... item from the menu that appears. d) Click on the Processes tab in Task Manager to display the list of processes. e) Click on the Image Name column header to sort the display by executable file name. f) Locate the IEXPLORE.EXE process and note the number in the PID column. You will need it for the next step. g) In the Command Prompt window, type the following command userdump PID where PID is replaced by the number from the PID column in Task Manager for IEXPLORE.EXE 1-26 533583860 Version A1 ABB Technical Support Example: userdump 1024 User Mode Process Dumper (Version 1.0) Copyright (c) 1999 Microsoft Corp. All rights reserved. Dumping process 1024 (IEXPLORE.EXE) to C:\User dump\IEXPLORE.dmp... The process was dumped successfully. Note that is may take several minutes before you see the last line. h) The IEXPLORE.dmp file is the userdump of the Internet Explorer process. Rename this file if you need to run userdump again. 4. For a Navigator hang, use the Diagnostic Browser utility to determine if any process is locked up in addition to Internet Explorer. a) Launch the Diagnostic Browser utility. Display the Start menu by pressing the Windows key on the keyboard. Select the Programs menu, the ABB Automation menu, the Operate IT menu, the Diagnostics menu, and finally click on the Diagnostic Browser Utility item. b) The Diagnostic Browser Utility has a tree view on the left side. You will probably see numerous entries of the form GetSpecification-Pppp-Tttt, PutSpecification-Pppp-Tttt, or StreamSpecification-Pppp-Tttt, where 'ppp' and 'ttt' are replaced by a number. The 'ppp' value is actually the PID of the process that is using the Get, Put, or Sream tag atom subscription. c) Click on each GetSpecification-Pppp-Tttt item in the tree view, other than the one that contains the PID of the IEXPLORE.EXE process. Also, expand each item one level. d) If the Diagnostic Browser Utility locks up when clicking on or expanding an item, that indicates that the process associated with that item is locked up. Follow the steps in item #3 above to obtain a userdump of the process, using the PID specified for the GetSpecification-Pppp-Tttt item instead of the PID for the IEXPLORE.EXE process. After the process is dumped, kill the process in the Task Manager by locating the process with the appropriate PID, right clicking on the image name, and selecting End Process from the menu that appears. Note if the Navigator resumes operation or not. e) Click on the Applications tab in Task Manager. f) Click on the AbbDiagnosticBrowser task. g) Click on the End Task button to stop the hung AbbDiagnosticBrowser task. h) Resume step c above, continuing from where you left off until you have clicked on each GetSpecification-Pppp-Tttt item. 5. For a problem with calling up a trend display, obtain process dumps of all OPW processes. Follow the procedure detailed in step 3 above to run the userdump utility, running it for each OPW.EXE process instead of for the IEXPLORE.EXE process. Note that you will need to specify a name for the dump file when running userdump or all of the OPW dump files will have the same name. So, for example, if an OPW process has a PID of 123, issue the following command: Userdump 123 opw-123.dmp 6. If you have enabled the OPW Console window, capture its output The procedures for enabling the OPW Console window and capturing its output are described in the Turning on OPW Console Window section below. 7. If you have DBWin32 application running, capture its output The procedures for running the DBWin32 application and capturing its output are described in the Running the DBWin32 Application section below. 8. Perform a Subscription Diagnostic Dump. Follow the steps outlined in RTDS/Subscription Data Collection Procedures, A - Doing a diagnostic dump. 9. Copy the last few days of Web server log files. 533583860 Version A1 1/18/2002 1-27 ABB Technical Support Trouble Shooting Guide Operate IT B.0 a) Launch the Windows Explorer. Display the Start menu by pressing the Windows key on the keyboard. Click on the Windows Explorer item. b) From the Windows Explorer's View menu, click on the Details item. c) Browse to the C:\winnt\system32\LogFiles\W3svc1 folder. d) In the pane on the right, click on the Modified column heading to sort the files by the date they were modified. e) Copy the three files that were most recently modified. f) If configuration pages were being brought up in the Navigator around the time the problem occurred, also copy the Web server logs from the configuration server machine. 10. Save the Windows Application and System Event Logs. a) Launch the Windows Event Viewer. Display the Start menu by pressing the Windows key on the keyboard. Select the Program menu, then the Administrative Tools menu, and finally click on the Event Viewer item. b) From the Event Viewer's Log menu, click on the Application item. c) From the Log menu, click on the Save As... item. Browse to the desired directory and save the file, making sure that you include the word App in the file name (this is because the type of log must be known to reopen it). b) From the Event Viewer's Log menu, click on the System item. c) From the Log menu, click on the Save As... item. Browse to the desired directory and save the file, making sure that you include the word Sys in the file name (this is because the type of log must be known to reopen it). 11. If a problem occurs with a trend display, see the section Items to Collect for Historian Problems. 12. If a problem occurs with a Produce IT display, see the section Data Capture for ProduceIT problems. CAUTION: When Internet Explorer is in full screen mode, as normally is the case when running the Navigator, the Windows Task Bar is covered by Internet Explorer. Typically, moving the mouse pointer down to the bottom of the screen displays the Task Bar. When the Navigator is hung, the Task Bar may be visible at the bottom of the screen, but it may not respond to the mouse. In this case, try pressing the Windows key on the keyboard to activate the Start menu and the Task Bar. (What is really happening is that the Task Bar is really covered by the Internet Explorer window, but since Internet Explorer is hung it doesn't redraw the area where the Task Bar was displayed.) 1-28 533583860 Version A1 ABB Technical Support Turning on OPW Console Window Enabling the OPW Debug Flag will cause a DOS window to be launched when the first OPW process is launched. All OPW processes will write some diagnostic messages to this DOS window. This may help in diagnosing OPW related issues, such as a trend that is launched and appears as an all black window. 1. Bring up the System Properties (Start Menu->Settings->Control Panel->System) dialog and select the Environment tab. 2. Make sure nothing is selected. If something is selected, in the white space in ‘User Variables for <Who is currently logged in>’ right mouse button click. 3. In the field at the bottom labeled ‘Variable’ type in FP_SHOW_CONSOLE. 4. In the field at the bottom labeled ‘Value’ type in a 1. 5. Press the Set button. 6. Close Navigator and log off and back on. 7. Start Navigator. 8. Bring up a graphic, trend or faceplate. 9. A DOS window should appear. 10. Place the mouse on the Title Bar for the DOS window. 11. Right mouse button click and select Properties. 12. Select the Layout tab. 13. Set Screen Buffer Size Width to 150 and Height to 1000. 14. Press OK. 15. In the Apply Properties dialog which will appear, Select the Modify shortcut which started this window or Save properties for future windows with same title option. (This will retain settings if navigator restarted). Note to disable the debug window delete the FP_SHOW_CONSOLE and logoff and back on. Capture data from DOS window 1. 2. 3. 4. 5. Select DOS window and place cursor in Title Bar area. Right mouse button click select Edit->Mark. Scroll up to the top of the information and right mouse button click in the DOS window at the start. Hold right mouse button down and drag the mouse down until all data is selected and then press the Enter key. The window will scroll when the mouse reaches the bottom of the DOS window. Paste data into Notepad. 533583860 Version A1 1/18/2002 1-29 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Running the DBWin32 Application OPW writes some diagnostic messages to the DBWin32 application when it is running. Running this application may help in diagnosing OPW related issues, such as a trend that is launched and appears as an all black window. Note that if this application is left running for several days it can consume more resources than is desired, so this application should be stopped and restarted once a day. If the machine is rebooted the DBWin32 application will have to be launched again. 1. Launch the DBWin32 application. Display the Start menu by pressing the Windows key on the keyboard, which is located between the Ctrl and Alt keys. From the Start menu, select Programs, ABB Automation, Operate IT, Diagnostics, and finally click on the DBWin32 item. 2. Bring up the Options menu in DBWin32 and make sure that the following items are checked: Output to Window Display PID Read Only Make sure that the Always on Top item is not checked. Capture data from DBWin32 1. 2. 3. 1-30 Select the DBWin32 application window. Choose Save Buffer As from the File menu. Browse to the desired folder and save with an appropriate name, such as dbwin.txt. The messages can now be cleared from the window by bringing up the Edit menu and selecting Clear Buffer. 533583860 Version A1 ABB Technical Support Items to Collect for Historian Problems Signs of a historian problem: A Trend stops updating – usually all trends Trends take a long time to be displayed (1 Min or more) Historical event page does not display Trends display flat lines that do not match the process (i.e. the value is changing) To correct problem for customer: Determine which historian is active (from server explore) (save this information via a screen dump) Start the data collection items listed below (1,3,4) (Don’t have to wait for it to finish before failing over, just give it a few minutes) Make a screen copy of the problem (trend with flat lines or blank event page) Failover the historian – this should allow the plant to retrieve data, then collect data for Wickliffe Items to Collect: From the historian that was not working: 1. Run d:\mssql\binn\sqldiag.exe. It will create a file SQLdiag.txt in the d:\mssql\log directory. Save this file) 2. Collect any DrWatson files (c:\winnt\drwtsn32.log) 3. On the ConfigServer computer, two query windows should be open – one for each historian computer. If a query window is not open attempt to open one to the historian server that is having the problem. Load the following file d:\abb\performance\ blocking_script_70_detailed12.sql (it may already be loaded in he window). Run this script – it will run for hours, but after 5 or more minutes click in the output window and hit the save button. 4. Again on the configuration server, open a performance monitor window and load the workspace from the d:\abb\performance\ historianproblem_Hist_1.pmw or d:\abb\performance\ historianproblem_Hist_2.pmw (depending on which machine is having a problem). Allow this to run for at least 5-10 minutes. 5. Attempt to open a new query window to the SQL server. If this window does not open or takes a very long time to open (1 - 2 minutes) then the SQL server process may be stuck. The sqlservr.exe process should be dumped (Follow step 3 under Navigator except change use the process SQLSERVR.EXE instead of IEXPLORE.EXE in item f) and g)). After completing the dump and collecting all other information, stop the Historian server (with the plants approval) and verify that query windows still exhibit the same behavior. If so, repeat the dump. Close all query windows to SQL server and again repeat the dump. ( We are looking for three dumps, with decrease in SQL server load between each) From both Historians servers save the following items: The NT application and system event logs The syslog.log and syslog.old.log files (in c:\temp) Call up the server explorer; select the historian servers and select server diagnostics. Record if the following atoms are changing Events_written, event_buffer, event_rate, values_lost, Values received, values written. The easiest way to do this is to screen copy the display and wait a few minutes and copy it again. Also record what the tag_atom_count is. After all data has been collect and the plant has approved - reboot the failed historian. (note: it may be worthwhile to allow items 3 and 4 to run for 30 minutes.) 533583860 Version A1 1/18/2002 1-31 ABB Technical Support Trouble Shooting Guide Operate IT B.0 1 - Items to Collect for RTDS Problems 1a. RTDS crash This scenario is detected by the fact that the RTDS is shown as 'unavailable' in the Server Explorer. RTDS failover should have happened, and clients should still be able to access data. Frequently, there will also be a "Dr. Watson" dialog open on the screen of the RTDS computer. If no Dr. Watson dialog is present, the I90Server is probably not visible in the task list on the RTDS node (if it is still there, then perform the steps from 1b instead). Steps: Acknowledge the Dr. Watson dialog on the RTDS computer, by pressing the OK button. Don't select the cancel button as this may cause the diagnostic data to be lost Check to make sure that no client nodes are misbehaving (screens are coming up okay, no lockups, points are good, etc.). If any are found to be misbehaving, perform the steps outlined in 2. Restart the RTDS from the ServerExplorer. If this doesn't work, restart the RTDS node. Collect the Dr. Watson information (see B) and NT Event log information from the affected RTDS. 1b. RTDS lockup This scenario is detected by the fact that client applications are having trouble getting data from one or both RTDS's. No failover has occurred. There is no Dr. Watson and the I90Server process is still present in the task list of both redundant RTDS nodes. Steps: Force a userdump of the RTDS on the active node (see C) Once the userdump is completed, kill the I90Server process from the tasks list. Within several seconds of this, the redundant RTDS should take over, and clients should regain their ability to communicate. Restart the RTDS from the ServerExplorer. If this doesn't work, restart the RTDS node. Verify that all clients are working again. If any are still exhibiting problems, perform the steps outlined in 2. Collect the userdump file, along with the NT Event log information. 2 - Items to Collect for Subscription Problems The first thing to do here is determine whether this is a problems with all clients, or just a single client. If all clients are experiencing the same problem, but the problem is isolated to a single tag or subset of tags, then the problem is probably not in the RTDS. If all tags are showing the same problems, execute the steps in 1b. If a single client is experiencing the problem, it is probably a problem on the client itself. Steps: Do a diagnostic dump (see A) of the affected node. Force a userdump of the Navigator (iexpore.exe) on the affected node (see C) Force a userdump of each opw.exe process Shutdown the Navigator Make sure that all opw.exe processes stop. If they don't, kill the ones that continue to run Restart the Navigator. 1-32 533583860 Version A1 ABB Technical Support Collect the userdumps, NT Event logs and diagnostic dumps. RTDS/Subscription Data Collection Procedures A - Doing a diagnostic dump To run a diagnostic dump use the Diagnostic Dump Utility located in the Start menu (Start -> Programs -> ABB Automation -> OperateIT -> Diagnostics). To use the utility, specify a file location to store generated dump file. Specify the name of the computer that you want to generate a dump for (the default is the local node). Press the 'Generate Diagnostics' to generate the diagnostic file. During the processing of the diagnostic file the progress bar will begin to fill in. Some system problems may cause the utility to lockup, causing the progress bar to freeze. If this appears to happen check the Task Manager to see if the utility is consuming CPU time, if it is then it is probably working on a large list of tags, wait a couple of minutes to see if the progress bar moves. If no CPU is being consumed then one of the processes that the utility is trying to diagnose has 'locked up'. You need to identify which process this is and then add its process id to the 'Ignore PIDs' field on the Diagnostic Dump Utility. Multiple PIDs should be separated by a comma. To recover from the lockup you need to slay the ABBDiagnosticDump utility using the Task Manager. To determine which process is causing the Diagnostic Dump Utility to lockup you need to run the Diagnostic Browser Utility (located in the same menu as the dump utility). When the Diagnostic Browser Utility opens you will see a tree control on the left side of the application window. Systematically click on each entry in the tree control to see if it fills in the list on the right side of the application window. If you click on an entry that causes the browser to lockup then you have found the entry that causes the Diagnostic Dump Utility to lockup. Note the PID number. The PID number is part of the name listed in the tree control. The name begins with the type of entry, followed by the process id (this is the number you are looking for), followed by the thread id. For example, one entry might look like 'GetSpecification-P1283-T45'. The process id for this entry is 1283. Now try and run the DiagnosticDumpUtility again and enter 1283 (for example) in the 'Ignore PIDs' field. If the dump utility locks up again then there is more than one process locked up. Use the browser to locate any additional 'locked' processes. Once the browser or dump utility locks up you need to kill it using the Task Manager. B - Collecting Dr. Watson information If the Dr Watson dialog is still up, select the OK button. NOTE: Don't select the cancel button as this may cause the diagnostic data to be lost Two files are created by the Dr Watson application, drwtsn32.log and user.dmp. By default these are in the WINNT folder. Sorting the folder with the most recent files at the top will typically show the files. If the files cannot be found then use the Windows Explorer to search for these files Move both files to a folder created for them e.g. 10Mar01-I90RTDS-Crash. You can also rename the files. The files must be renamed or moved since another Dr Watson will override the user.dmp file on the next crash and its data will be lost. These two files should then be zipped up and set to developers for analysis C - Doing a user dump 533583860 Version A1 1/18/2002 1-33 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Identify the PID of the process you want to dump. If a RTDS, ServerBroker or DSS is hung, this may be done from the "Diagnostic Browser Utility" by stepping through the tasks until one does not return. The entries in the browser utility are of the form: "AppName-Pxxx-Tyyy" where xxx is the PID. It may also be done by opening the Windows Task Manager and finding the process under the "Processes" tab. Open a DOS window i.e. Programs/Command Prompt (or use the Run window (Start\Run)) to call the userdump application Type the following without the quotes "userdump PID c:\temp\processname.dmp", where PID is the process ID identified in the steps above. The "c:\temp\processname.dmp" is the name and directory where the dump will be stored. If running this causes a "userdump is not recognized as an internal or external command etc " message then you mostly likely don't have userdump installed or it is not on the current path. In the case where userdump is not installed, copy the userdump.exe, userdump.sys & userdump.cpl to the root of the C: drive and try the command again. These files have been installed on numerous computers at Nova. All RTDS computers should have them. 1-34 533583860 Version A1 ABB Technical Support Data Capture for ProduceIT problems 1. Provide a thorough description of the symptoms, in plain English. Make sure to include the exact time of the event, as well as an account of any unusual events before or after the problem, and any other circumstances that may have affected the situation. 2. Capture all files starting with “Syslog” in “d:\Program Files\ElsagBailey\Conductor“ from the PC with the fault and from the ProduceIT primary server. 4. Check virtual memory and CPU utilization via the Task Manager on the ProduceIT primary server. If either indicator is high, capture the “processes” tab. 5. If appropriate, save screen images using the PrintScreen key and Paint or WordPad. Multiple screen shots may be required to get all the data. 6. If a Dr. Watson crash occurred, make sure to retrieve the file “WINNT\drwtsn32.log” from the PC where it happened. If the Dr. Watson dialog is still on display, don’t close it; get the process ID of the affected process from the Task Manager, and do a userdump for that ID. 7. If a Navigator lockup has occurred, get the process ID of Iexplorer.exe and do a userdump for that ID. In general, do a userdump for any other process that is locked up, but be aware that it is not very easy to tell whether a process is locked up. 8. If batches are affected, capture batch events from the “Events” tab on the batch information dialog (or the batch history information dialog, if the batch has been historized). Be sure to get events for all batches involved. 9. If a phase-related problem has occurred, run the OperateIT “Diagnostic Dump Utility” and capture the “Clientdump.xml”. 10. If an RTDS-related problem is suspected, capture events for the exact time frame using the “Display Event Page” object Diag_DisplayEventPage, via the “history” button. 11. After gathering all the data, see if the problem can be reproduced. 533583860 Version A1 1/18/2002 1-35 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Diagnostics ABB Diagnostic (Hidden) Within OperateIT are a number of Hidden Diagnostic. These diagnostics are primarily for development purposes and typically not used by the end user. This section will identify some of the diagnostics that fall into this category. To access the diagnostics the user need to hold the CTRL, ALT keys and then click on the ABB logo in the Navigator window. Diagnostic Menu 1-36 533583860 Version A1 ABB Technical Support Diagnostic Menu Continue Diagnostic Options 533583860 Version A1 1/18/2002 1-37 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Diagnostic Output The Diagnostic Output is a Log files for system messages as defined by the Diagnostic Option view. 1-38 533583860 Version A1 ABB Technical Support Display Invocator The Display Invocator is a utility to generate a script file that will simulate operator actions in the navigator window. The user can write a script to cycle through a series of OperateIT objects such as faceplate and displays call ups. The user needs to create a batch file before the Invocator is started. Display Invocator The Display invocator is a utility that enables the user to call up series of function, displays, etc up on the Maestro console for testing purposes. To use this feature the user needs to create a text(.txt) file to hold the commands for the procedure for calling up the desired functions. Use NOTEPAD or similar utility to create the file. The following procedure will call up a faceplate, an eventbar, a graphic and a trend display with 5 second delay after each element callup. The entires for commands and qualifiers are case sensitive. After each verb enter the object name to be called up. By default the faceplates for tags will be launched. To select an aspects of a tag, add the aspects (verb) after the object name. (verb FIC-454, eventpage) other aspects are trend, overview, section, help, configuration, etc. 533583860 Version A1 1/18/2002 1-39 ABB Technical Support Trouble Shooting Guide Operate IT B.0 verb Tag_name pause 5000 verb Tag_name, section pause 5000 verb eventbar_name pause 5000 verb graphic_name pause 5000 verb trend_display_name pause 5000 When all objects have been configured in the file, save the file to harddisk. Then execute the procedure by entering a url Address using the Internet Explorer. Maestro://Symphony/Default/testpages/Navigation/Displayinvocator.htm?c:\user_txt_file.txt Enter the file location on the hard drive after the “?”. In this example the C:\user_txt_file.txt This will call up the Display Invocator window and then press the START button. The function defined in the text file should be executed in the order they appear in the file along with the configured delays. The procedure should repeat automatically. To get a listing of aspects qualifiers run the following script in the Navigator.Mirror diagnostic. SELECT qrySymObject.NAME, qrySymObject.DESCRIPTION, qrySymObject.USERTEXT, qrySymObject.LANGID, qrySymContextMenu.ACTION, qrySymContextMenu.ATTRIBUTE, qrySymContextMenu.PRIMARYVERB, qrySymContextMenu.COMMAND, qrySymContextMenu.HELP FROM qrySymObject INNER JOIN qrySymContextMenu ON qrySymObject.OBJECTID = qrySymContextMenu.ITEMID The column labeled ATTRIBUTE in the results view are valid aspects or qualifiers. 1-40 533583860 Version A1 ABB Technical Support Sample text file with Display Invocator Scripts This file contains a series of verb scripts to call up various objects. Use this as a guide for creating new scripts. The script file can be created using NOTEPAD and can be saved anywhere on disk. C:\ is a convenient location. ProgramFiles ElsagBailey Symphony Default Testpages Navigation DisplayInvocator.txt DisplayInvocataor.txt file open /Symphony/Default/MaestroNT/Navigation/MntOpWin.asp?EbClass=FreeGraphic&EbObject={7722CD2DCCBA-11D2-9456-00C04F8069A3}&EbModel=Amm_Steam_Distribution pause 10000 open /Symphony/Default/MaestroNT/Navigation/MntOpWin.asp?EbClass=FreeGraphic&EbObject={E6EEC42CCCA8-11D2-9452-00C04F8069A3}&EbModel=Amm_Feed_Gas_Compression pause 10000 open /Symphony/Default/MaestroNT/Navigation/MntOpWin.asp?EbClass=FreeGraphic&EbObject={E6EEC52BCCA8-11D2-9452-00C04F8069A3}&EbModel=Amm_Master_Control_Group pause 10000 open /Symphony/Default/MaestroNT/Navigation/MntOpWin.asp?EbClass=FreeGraphic&EbObject={169D2B3ECCAA-11D2-9452-00C04F8069A3}&EbModel=Amm_NewConv pause 10000 open /Symphony/Default/MaestroNT/Navigation/MntOpWin.asp?EbClass=FreeGraphic&EbObject={E6EEC42CCCA8-11D2-9452-00C04F8069A3}&EbModel=Amm_Synthesis_loop_Overview pause 5000 open /Symphony/Default/MaestroNT/Navigation/MntOpWin.asp?EbClass=Faceplate&EbObject={47BE4C18-EF1E11D1-9F51-0000C0E7F2E7}&EbModel=Harm_ANALOG pause 5000 open /Symphony/Default/MaestroNT/Navigation/MntOpWin.asp?EbClass=Faceplate&EbObject={47BE49B5-EF1E11D1-9F51-0000C0E7F2E7}&EbModel=Harm_ANALOG pause 10000 //close /Symphony/Default/MaestroNT/Navigation/MntOpWin.asp?EbClass=Faceplate&EbObject={47BE4C18EF1E-11D1-9F51-0000C0E7F2E7}&EbModel=Harm_ANALOG //close /Symphony/Default/MaestroNT/Navigation/MntOpWin.asp?EbClass=Faceplate&EbObject={47BE49B5EF1E-11D1-9F51-0000C0E7F2E7}&EbModel=Harm_ANALOG //pause 10000 verb FI197, Faceplate pause 10000 verb FI197, Point pause 10000 533583860 Version A1 1/18/2002 1-41 ABB Technical Support Trouble Shooting Guide Operate IT B.0 verb FI197, Trend pause 10000 verb FI197, Overview pause 10000 verb FI197, Section pause 10000 verb FI197, Help pause 10000 verb FI197, Configuration pause 10000 exec EbPage.SetStatusText("The " + EbPage.WindowManager.GetFocusWindow().Name + " window has focus."); pause 10000 1-42 533583860 Version A1 ABB Technical Support Execute Script No Information Mirror Database The Mirror Database Diagnostic page allow the user to make SQL queries to the Mirror database. Use the Access Mirror Database pull down to locate and select Navigator.Mirror. 533583860 Version A1 1/18/2002 1-43 ABB Technical Support Trouble Shooting Guide Operate IT B.0 OperateIT has a Query File that provides a series of sample query that can help familiarize the user with this utility named Queries.txt. Access the file with notepad. Pragram Files > ElsagBailey > Symphony > Default > Testpages > Navigator > Script > Queries.txt To list known SystemNodes in the database cut and paste the System Node query highlighted above into the Mirror Database Query field. 1-44 533583860 Version A1 ABB Technical Support The Mirror Database To execute the query, click the following option in the indicated order. 1. 2. 3. 4. Open Database Open RecordSet Execute Query Show Results The show Results may take a little time and then the results are displays similar to the results shown in the view above. To running another query Start and Stop logging. 533583860 Version A1 1/18/2002 1-45 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Navigator Options Operator Window 1-46 533583860 Version A1 ABB Technical Support Session Variables 533583860 Version A1 1/18/2002 1-47 ABB Technical Support Trouble Shooting Guide Operate IT B.0 1-48 533583860 Version A1 ABB Technical Support Tone Diagnostics Tone Diagnostic Verb Cycler 533583860 Version A1 1/18/2002 1-49 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Window Manager Test Window Test 1-50 533583860 Version A1 ABB Technical Support Windows Scripting Host Test 533583860 Version A1 1/18/2002 1-51 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Windows Scripting Host Test (Server) 1-52 533583860 Version A1 ABB Technical Support OperateIT Diagnostics Access to the OperateIT diagnostic is as follows: Start > Programs > ABB Automation > Operate IT > Diagnostic The OperateIT diagnostics provide a number of diagnostics to aid in the troubleshooting of problems. This section will provide information for those diagnostics which information is available as to it’s use at this time. 533583860 Version A1 1/18/2002 1-53 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Application Events (Future) DBWin32 (Future) Diagnostic Browser Utility (Future) Diagnostic Dump Utility (Future) Event Counts The Event Count displays the number of event in the system and their status. Event Group Bar (Future) 1-54 533583860 Version A1 ABB Technical Support Event List Comparison (Future) Event Page (Future) Event Stream from Concentrator 533583860 Version A1 1/18/2002 1-55 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Event Stream from IEbEventServer The Event Stream from IebEventServer diagnostic will display event information from the Different Servers within OperateIT. This section will demonstrate the use of this diagnostic with a Function Block Server. The opening dialog window prompt the user to select a server within the OperateIT system. Enter a “n” to skip the listed server and to go onto the next server. Enter a “y” when the desired server is listed and press the Enter Key at the filter prompt to skip the filter. In the diagnostic view are messages that pertain to the activities of the server. Each server will display a unique set of messages pertaining to the servers functionality. This messages can be helpful in diagnosing system problems. 1-56 533583860 Version A1 ABB Technical Support When an event diagnostic program prompts for a “Filter: ”, what should be entered? To specify that the program should process all events, simply press the “Enter” key on the PC keyboard. To specify an actual event filter, enter a SQL expression that one would expect to see in the “where” clause of a SQL statement. For example: NAME LIKE ‘Harmony%’ OR ACT = 1 The following table lists the keywords and data types that are recognized by the Event System: Keyword Data Type Description SENDER NAME EVENTPT OBJECTID String String String String AREA UNIT INHB EQUIPMENT ETYP String String Integer String Integer ECLA ECAT Integer Integer ALMEN PRI DUAL ACT ACKED UNACKEP ACKR USERID SERVERUUID Integer Integer Integer Integer Integer Integer Integer String String Full tag.atom expression (e.g. ANA-100.LOW) Tag/Object name (e.g. ANA-100) Event-point name (e.g. LOW) Tag/Object UUID (e.g. ED831E80-8EA7-11D1-99A00000C0B7520F) Plant area description Plant unit description Tag is inhibited flag (0=False, 1=True) Batch equipment identifier Event type (1=PE, 2=MT, 3=PP, 4=PR, 5=OA, 6=IO, 7=SY, 8=FV, 9=EC, 10=AC, 11=DB, 12=MG) Event class (1=Simple, 2=Condition, 4=Tracking) Event category (0=None, 1=Boolean Changes, 2=Limit Events, 3=Deviation Events, 4=Gradient Events) Event is an alarm flag (0=False, 1=True) Event priority Dual event flag (0=False, 1=True) Event-point is active flag (0=False, 1=True) Event-point is acknowledged flag (0=False, 1=True) Event-point is not acknowledged flag (0=False, 1=True) Event requires acknowledgment flag (0=False, 1=True) Operator/Login name Server UUID of event source 533583860 Version A1 1/18/2002 1-57 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Get Client Get Client diagnostic is used to subscribe to a specific RTDS Tag.Atom for the purpose to verify if a subscription can be generated to return a value and or quality value from the Tag.Atom. On occasion an application may not be able to subscribe or loses a subscription to a tag.atom. This diagnostic can check the system ability to subscribe to trouble sum tag. The Get Client diagnostic allow the entry of up to 5 Tag.Atoms. Enter a Tag.Atom and then click on Add to add the tag to the subscription list. Then Click Subscribe to initiate the request. The Current Value, Quality and Time Stamp fields will dynamically update with the requested information for those Tag.Atom listed. 1-58 533583860 Version A1 ABB Technical Support OPC Alarm Client (Future) Put Client Put Client diagnostic is used to Put a value to a Putable Tag Object for a specific RTDS Tag.Atom. On occasion an application may not be able to Put a value to a specific tag object in an RTDS. This diagnostic can check the system ability to perform a Put command. The Put Client diagnostic allow the entry of up to 5 Tag.Atoms. Enter a Tag.Atom and then click on Add to add the tag to the subscription list. Enter a value and value type and then click Subscribe to initiate the request. To use the Put Client call up a faceplate or graphic to monitor the change and then launch the PutClient diagnostic. Procedure to perform the PUT. Enter the Tag.Atom, and then click on Add. Click on Subscribe button. 533583860 Version A1 1/18/2002 1-59 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Then enter a put value in the value field and then click the Put button. Note the Value type field is not used. The default VT_BSTR as is. 1-60 533583860 Version A1 ABB Technical Support Security Diagnostic Security Diagnostic (SecDiagnostics.exe) can be used to diagnose security problems with the configured system. An HTML report will be generated including a security summary and access rights for the configured users and groups. Click on the Generate Diagnostics to create a report. It may take a little time to generate the report, when the following message occurs click Yes to display the report. After generating diagnostics, object security access can also be tested by selecting a specific tag atom and lifecycle. 533583860 Version A1 1/18/2002 1-61 ABB Technical Support Trouble Shooting Guide Operate IT B.0 The top section of the generated report will list the OperateIT specific data. The first second of information pertains to Security Groups. In this example the security group listed is SymSecGroup and the member of the group is Everyone. The table will list the Access rights in OperateIT and the configured right for the security group, Allow or Deny. The NT User Groups displays the Group that is a member of the Security Group. 1-62 533583860 Version A1 ABB Technical Support Security Report Continued Logged in user group membership and Access rights granted and via what Trustee. 533583860 Version A1 1/18/2002 1-63 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Sequence Event Bar (Future) Server Broker List 1-64 533583860 Version A1 ABB Technical Support Server Explorer Server Explorer diagnostic provides the same view that is available from the Navigator Tool menu or the Web Based Server Explorer option under Base products. The Server Explorer diagnostic provides a view of OperateIT servers and their status. Server Explorer Diagnostic From the Server Explorer view the user can access the submenu items associated to the selected server. 533583860 Version A1 1/18/2002 1-65 ABB Technical Support Trouble Shooting Guide Operate IT B.0 In this example for the selected Harmony Server, the submenu options are: 1-66 Diagnostic Block Details Module Details Tag Summary 533583860 Version A1 ABB Technical Support Stream Client 533583860 Version A1 1/18/2002 1-67 ABB Technical Support Trouble Shooting Guide Operate IT B.0 TimeSync Diagnostics OperateIT has a TimeSync Diagnostic program to determine what Time Sync Daemon in the system is the time master. The diagnostic also provides a log file to provide information in regards to time adjustments. TimeSync Diagnostic To access help enter “?” or “Help” TimeSync Diagnostic The help menu includes the following commands. help msite quit trace The “msite command will return the machine name of the PC that is acting as the Time Sync Daemon master and it’s priority settings. 1-68 533583860 Version A1 ABB Technical Support TimeSync Diagnostics In this example the machine name id is ts_cnt_pri_2000 with a priority of 7. The Trace command will create a log file named tsyncdd.log for the logging of time sync related messages. The file is located in the C:\Tem directory. To enable the trace function, enter TRACE ON. To disable the trace function, enter TRACE OFF. TimeSync Diagnostics 533583860 Version A1 1/18/2002 1-69 ABB Technical Support Trouble Shooting Guide Operate IT B.0 An example of a tsyncdd.log file is as follows. TSYNCDD.LOG The following information is displayed in the view: 1-70 The name of the Time Sync Daemon machine (ts-cnt-pri_2000) Time adjustments ( –33 and –32 milliseconds) Reported Drifting (-0.03336 and –0.031136 sec) 533583860 Version A1 ABB Technical Support 1.1.1.1.1 Time Sync Messages The Time Sync System will generate a number of messages depending on the situation. Each time an RTDS broadcasts a time adjustment request a diagnostic (DG) event message to be generated by OperateIT Alarm Management system. An example would be “Console time adjustment by 2 seconds”. The message only indicates a request was made but does not indicate that the adjustment has been made. The Time Sync Diagnostic trace function will log message to the tsyncdd.log file. The following table list the events written to the NT Application Event Log by the TimeSync Daemon (tsyncdd.exe). Event Id 533583860 Version A1 1/18/2002 Description 100 The %1 service was installed. 101 The %1 service was removed. 102 The %1 service could not be removed. 103 The control handler could not be installed. 104 The initialization process failed. 105 The Time Synchronization daemon service was started 106 The service received an unsupported request. 107 Debug: %1 108 The Time synchronization daemon service was stopped. 109 Date changed by %1 from: %2 110 Assuming mastership. Date changed by previous master %1 from: %2. 1-71 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Databases Mirror and RTDS databases keep growing User reported issue. In several projects it was recognized that after some time of normal operation the size of configserver_RTDS and configserver_Mirror was significantly increased. That means the size is 3 or 4 times of a newly replicated database. When the database was deleted on a PC and rebooted, they were recreated with the smaller size. Then the start of Operate IT applications seems to be much faster and also the reaction of this navigator seemed to be faster. How can these replicated databases grow up in this way. the configserver_RTDS database was greater than the whole ConfigServer database. Answer SQL-Server may grow the file in order to store transactions as they are being replicated. The space is not automatically reclaimed. If space on the hard disk is an issue, then they can use the auto-shrink feature to periodically reduce its size. Another option is to create a maintenance plan to periodically do this. Use the same settings as the ConfigServer Event View Logs The event viewer logs are a key source of information in troubleshooting problems in OperateIT. The Microsoft Event Viewer utility is used to access these files. Access to Event Viewer files 1-72 533583860 Version A1 ABB Technical Support Application log The application log contains events logged by applications or programs. Security log The security log can record security events such as valid and invalid logon attempts as well as events related to resource use such as creating, opening, or deleting files. An administrator can specify what events are recorded in the security log. System log The system log contains events logged by the Windows system components. For example, the failure of a driver or other system component to load during startup is recorded in the system log. The event types logged by system components are predetermined by Windows. 533583860 Version A1 1/18/2002 1-73 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Select either the System Log or Application Log file from the menu. Application Log The Application log will typically contain messages from the various Operate IT and/or third party applications. Pay close attention to VBRuntime errors. These type of error will typically occur if an application is having a problem that will seriously impact the operation of OperateIT. 1-74 533583860 Version A1 ABB Technical Support System Log The System Log will typically contain DCOM type errors. In this example the host PC can not talk to the IIT2 computer. This may be due to the PC being offline. 533583860 Version A1 1/18/2002 1-75 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Saving the EventView files. Under the Log menu item is the Save As options. When saving the Event Viewer log there are typically two options that are used. Event Log Files (*.EVT) and Text Files (*.TXT). The *.EVT format requires the user to use the Event Viewer Utility on the PC that the user is using to look at the files. The *.TXT format allows the user to view the file with any text editor application. Which is usually a little easier to user, plus the user can view multiple records at a time. 1-76 533583860 Version A1 ABB Technical Support Harmony Harmony Server Faceplate The Harmony Server Faceplate provides server status, Loop and Node number and the server status as to Active or Inactive Server. If running Redundant servers the status of the redundant server will be displayed. Right Click the Station faceplate to view the Harmony Server Diagnostic Utility. 533583860 Version A1 1/18/2002 1-77 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Server Diagnostic access 1-78 533583860 Version A1 ABB Technical Support Harmony Server Diagnostics The Diagnostic Utility for the Harmony Server will list the Number of tags assigned to the Server. The Tags tab list all the tags and allows the user to view the values for the tag specific atoms. 533583860 Version A1 1/18/2002 1-79 ABB Technical Support Trouble Shooting Guide Operate IT B.0 AtomWEB OperateIT has an AtomWeb utility, which allows access to a database that provides reference information on the System Tag Atoms. The AtomWeb utility is accessed by entering the following Address in the Internet Explorer address field. HTTP://LocalHost/Symphony/AtomWeb/ ICI Error The following error has been seen due to an ICIC Event Stream Diagnostics for Harmony Server had the following error. 56 <MD> Error Create Server Process The ICICONFIG configuration for Name/Address field for Node had an IP address for an HNCC but for a COM1 or COM2 CIU this should be the name of the Host PC. The user does not need to reboot after an ICICONFIG change is performed. 1-80 533583860 Version A1 ABB Technical Support Operation Parameters The Operating Parameters application allows monitoring and changing the status of tags maintained by the Harmony Server. To launch the Operating Parameters, from an open Harmony tag object Right Click, select Server and then Operating Parameter. 533583860 Version A1 1/18/2002 1-81 ABB Technical Support Trouble Shooting Guide Operate IT B.0 General Page General Page Description Fields Field Tag name Tag type Quality Tag Description Tag ID Security Group Tag Inhibition Inhibited 1-82 Description Unique identifier for the tag in the system. Tag type (i.e., Harmony station). Information reported from the system about the quality of the tag. Information entered during configuration that further describes the function of the tag in the system. Tag ID assigned by the system. Security group that the tag is assigned. Information about the inhibit status of the tag. A tag inhibit disables event reporting for the tag. Checked if tag is inhibited. Auto Inhibited Checked if tag is automatically inhibited by another tag in the system. Manual Inhibit Enable to manually inhibited event reporting for the tag. Inhibitor Overall Alarm Status Area Tag that inhibited the tag. Checked if the tag is reporting an alarm. This is a single overall alarm status indication. Area structure the tag is assigned to. 533583860 Version A1 ABB Technical Support Unit Equipment Unit structure the tag is assigned to. Equipment structure the tag is assigned to. Harmony Page Harmony Page Descriptions Fields Field Loop Node Module Block Suspended Disestablished Substituted Value Data Events Received at 533583860 Version A1 1/18/2002 Description Loop number from 0 to 250 that identifies the communication network (loop) of the Harmony Control Unit. Node number from 0 to 31 that identifies the interconnection point on the communication network. Module number from 1 to 9,998 that identifies the device within the Harmony Control Unit. Function block location from 1 – 30000 that identifies the location of a specific function block in the Harmony Control Unit. Enable to have the system ignore the tag. Checked if the tag is disestablished in the Cnet-to-Computer interface. A tag becomes disestablished when a substitute value has been entered. Value of tag is manually substituted. Number of events related to this tag. Time when the information was last updated. 1-83 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Station Page Station Page Descriptions Fields Field Value Time Quality Mode Control Output Set Point Ratio Index High Alarm limit Low Alarm limit Deviation limit Station Type Tuning Block Computer Control Mode Locked Control Out of Calibration 1-84 Description Value being reported from the control System. Time that the value was reported. Quality of the value reported. Current operating mode of the tag. Current control output setting for the tag. Current set point value for the tag. Current ratio index value for the tag. High alarm limit configured for the tag. Low alarm limit configured for the tag. Allowed deviation between the output and the set point configured for the tag. Shows the station type (basic, cascade, or ratio) Block address of the process variable track signal. Checked if the station is under computer control. Checked if the mode is locked. Checked if the reported value is out of calibration. 533583860 Version A1 ABB Technical Support Operating Parameters Tag Type Descriptions The following tables list the Operation Parameters Tag Pages descriptions for Harmony tags. Analog and RMSC Page Field Analog Value Time Quality High limit Low limit Span Out of calibration Description Analog value being reported from the control system. Time that the last value was reported. Quality of the value reported. High limit configured for the tag. Low limit configured for the tag. Span of the process variable in percentage based on the high and low limits. Checked if he reported value is out of calibration. ASCII String Page Field Text Field Time Quality Sequence Number Original string length Receiving string length Block string length Console string length Accept mode change Data rejected Accepted data Remote Truncation occurred. 533583860 Version A1 1/18/2002 Description ASCII string being reported by the control system. Time that the last value was reported. Quality of the value reported. Maintained at the system level by the Harmony controller to assist in multiple user data packet management and user data packet identification. Original string length of the message in bytes. Received string length of the message in bytes. Block string length limit in bytes Console string length limit in bytes. Checked if the mode change was accepted by the controller. Checked if the data was rejected by the controller. Checked if the data was accepted by the controller. Checked if the data was remotely truncated. 1-85 ABB Technical Support Trouble Shooting Guide Operate IT B.0 DAANG Page Field Value Time Quality Mode Next higher Next lower High constraint Low constraint High reference Center reference Low reference Out of calibration Description Value being reported from the control system. Time that the last value was reported. Quality of the value reported. Current operation mode of the tag. Next higher alarm setting. Next lower alarm setting. High constraint value configured for this tag. The input will be constrained when it exceeds this configured value. Low constraint value configured for this tag. The input will be constrained when below this configured value. Maximum positive reference of the monitored value. Center reference bidirectional bar chart elements on consoles. Defines the positive and negative segment for the display. Maximum negative reference of the monitored value. Checked if he reported value is out of calibration. DADIG Page Field Current state Time Quality Mode Current priority Current comment Quality overwritten Description Current state being reported by the control system. Time that the last value was reported. Quality of the value reported. Current operating mode of the tag. Current priority number of the tag. Comment associated with the priority. Checked if the quality has been overwritten Digital, Device Driver, RMCB, and RCM Pages Field Current state Time Quality Current priority Current comment 1-86 Description Current state being reported by the control system. Time that the last value was reported. Quality of the value reported. Current priority number of the tag. Comment associated with the priority. 533583860 Version A1 ABB Technical Support Enhanced pages Field Override enabled Override value Description Checked if the override value is in use. Value of the override. MSDD Page Field Current state Time Quality Zero state One state Two state Three state Description Current state being reported by the control system. Time that the last value was reported. Quality of the value reported. Feedback state zero being reported by the control system. Feedback state one being reported by the control system. Feedback state two being reported by the control system. Feedback state three being reported by the control system. Module Page Field Controller type Module mode Node type Status bytes ICI type Controller offline Module error Communication errors System errors Local I/O bad Remote I/O bad 533583860 Version A1 1/18/2002 Description Displays the type of controller. Displays mode the controller is operating in. Displays the node type. Displays the status bytes reported from the controller. Checked if it is an ICI type. Checked if the controller is offline. Displays any module errors Displays any communication errors. Displays any system errors. Checked if he local I/O is bad. Checked if the remote I/O is bad. 1-87 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Server Page Field Loop Node Restarts ICI type ICI mode ICI revision 1-88 Description Loop that the server is connected to. Node number of the server. Number of restarts performed on the server. Type of Cnet-to-Computer interface that the server is connected to. Mode of the Cnet-to-Computer interface that the server is connected to. Revision of the Cnet-to-Computer interface that the server is connected to. 533583860 Version A1 ABB Technical Support Substitution Page Substituted Value Substitute can be used to enter a substituted value for a tag. When a substitution is applied, scan is automatically turned off. For a process value, use the up/down arrow buttons beside the value to enter a new value directly in the field. The Increment Step selection determines the percentage of increment for each click of the arrow button: 0.04% of span or 4.0 % of the span 533583860 Version A1 1/18/2002 1-89 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Red Tag Red Tag Page The Red Tag Page is available on controllable tag type that supports red tagging. Red tagging is a way of placing a tag out of service (for maintenance purposes) in a way that prevents it from being put back into service by unauthorized users. Typically only a limited number of users are permitted to red tag a tag. To Red Tag a tag, enter an alphanumeric string in the New Key field and click Add. The faceplate will display a lock icon with a red background to indicate the tag has been red tagged. Up to three users can Red Tag a tag. This tag is not put back into service until all Red Tag keys are deleted by the users who added them. To remove a red tag key, click Delete next to the key 1-90 533583860 Version A1 ABB Technical Support Red Tag Station Faceplate In this example a Station Faceplate has a Red Tag Lock indication as shown. 533583860 Version A1 1/18/2002 1-91 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Block Details Block Details function allows inspecting Function Blocks that are contained within a selected controller. It displays the specifications and the outputs of a selected function block and additional information related to the function code. The specifications of the selected function block can also be tuned. To launch Block Details, from an open Harmony tag object Right Click, select Server and then Block Details 1-92 533583860 Version A1 ABB Technical Support Specification Tab Block Details Specification Tab The Specification tab displays a list of function code specification records. Columns: Spec: The specification number. Value: The current value of the specification. Description: The description of the meaning of the specification. Icons under the Spec column are used to show additional information about each specification. Indicates a specification, which references another function block output. Indicates a specification whose value can be tuned 533583860 Version A1 1/18/2002 1-93 ABB Technical Support Trouble Shooting Guide Operate IT B.0 To tune a block, right-click and select Tune from the menu or double click on the note icon. Tunable Blocks 1-94 533583860 Version A1 ABB Technical Support Specifications whose value can be changed while the controller is in Execute Mode are said to be tunable. To tune a block, enter the desired value and click Ok or Apply. The Use Default button returns the value to the default value. Tune Properties View The properties view can be accessed by right clicking the Tunable Spec and selecting. The Properties view display the tuning properites parameters. 533583860 Version A1 1/18/2002 1-95 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Output Tab Block Details Output Tab The Outputs tab display function block output for the selected function code. Columns: 1-96 Output: Output Number Block: Block number associated with the output number. Type: Data type of the block output. Value: Value of he block output. Description: Description of the meaning of the output value. 533583860 Version A1 ABB Technical Support Description Tab Block Details Description Tab The Description tab contains a description of the function code that is currently on display. The lower portion of the page provides additional notes on the function code. The Block Navigation button allows the user to select a different block address or to page to the next or previous block. Only the bold selection are valid at any given time. 533583860 Version A1 1/18/2002 1-97 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Address Button Upon clicking on the Address selection the following Block Address window will appear. Enter the new address and click Ok to continue. 1-98 533583860 Version A1 ABB Technical Support Module Status Module Status application provides detailed information about the operational status of a selected Harmony controller. To launch Module Status, from an open Harmony tag object Right Click, select Server and then Module Status. 533583860 Version A1 1/18/2002 1-99 ABB Technical Support Trouble Shooting Guide Operate IT B.0 General Tab General Tab The General tab displays the address (Loop, PCU, Module, Block), Type, Revision firmware, and current mode of the selected controller. Operations modes: 1-100 Execute Online and in execute mode. Configure Online and in configure mode. Error Online and in error mode, this indication occurs when a controller configuration error exists. Failed Offline due to the existence of an internal problem 533583860 Version A1 ABB Technical Support Module Status Address button Status Tab Status Tab The Status tab displays a list of status error messages that apply to the selected controller at the time of the last scan. 533583860 Version A1 1/18/2002 1-101 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Problem Tab Problem Tab The Problem tab displays a list of problem report messages that apply to the selected controller at the time of the last scan. 1-102 533583860 Version A1 ABB Technical Support Technical Tab Technical Tab The Technical tab displays status bytes. The number of status bytes depends on the type of controller. Some controllers return five status bytes while other return 16 bytes. Of these bytes, the first two are common to most controllers. The following table provides interpretation of the status bytes. Bytes 1 2 ES Bit 7 ES FTX 6 5 MODE BAC RIO 4 3 LIO CFG 2 1 TYPE NVF NVI 0 DSS Error Summary 0 = Ok, 1 = error exists If there is an error indicated in the status information, the bit is set to one. MODE Controller mode: 00 = Configure 01 = Failed 10 = Error 11 = Execute 533583860 Version A1 1/18/2002 1-103 ABB Technical Support Trouble Shooting Guide Operate IT B.0 TYPE Controller type. Each Harmony controller has a different type identifier. FTX First time in execute mode: 0 = No, 1 = Yes BAC Summary of Backup status: 0 = Ok, 1 = Bad RIO Summary of Remote input status: 0 = Ok, 1 = Bad LIO Summary of Local input status: 0 = Ok, 1 = Bad CFG Online Configuration changes being made: 0 = No, 1 = Yes NVF Nonvolatile memory failure: 0 = No, 1 = Yes NVI Nonvolatile memory initialized: 0 = No, 1 = Yes DDS Digital Station status: 0 = Ok, 1 = Bad 1-104 533583860 Version A1 ABB Technical Support Tag Summaries The Tag Summaries application provides an interface for querying tag information and producing tag summary reports. The query is viewed in one of three ways. Design view SQL View Datasheet View Access Tag Summaries as follows: From the Harmony Server entry in the Server Explorer window, right click, select Service and then select Tag Summaries. Select Query The Select Query window contains a number of predefined queries for tag summaries. To view a tag summary, select the desired summary and click OK. 533583860 Version A1 1/18/2002 1-105 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Alarm Summary Query The Alarm Query Displays the predefined query to display the Alarm Summary reports. This is the view that is displayed if the user clicks on the SQL button in the title bar. Click on the Datasheet button to launch the report. Alarm Datasheet The Datasheet for Alarm list the tags in alarm. 1-106 533583860 Version A1 ABB Technical Support Design a Query The Design view allows the user to design a new query for a custom report. To design a new query, click on the New Query entry in the Tag Summary’s Select View window or click on the Design button in the title bar. Design View This view shows the configuration for a report to display the Tag Name, Alarm Description and Quality for tags in the database. 533583860 Version A1 1/18/2002 1-107 ABB Technical Support Trouble Shooting Guide Operate IT B.0 The SQL view displays the query generated from the Design view. The Datasheet for the design view is shown below. Datasheet view For further details on Tag Summaries reference Section 16 in the OperateIT Configuration Manual WBPEEUI220790B0. 1-108 533583860 Version A1 ABB Technical Support I90Server failure The EBI90Server dies (removed itself) in the task manager. The following error is seen in the Event Viewer Application Log . Check to see if the node has a valid license for the I90HarmServer. 533583860 Version A1 1/18/2002 1-109 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Unable to Load Tags The following window may appear during system startup. This error can occur if the number of defined tags in the Harmony Add-On exceeds the number of tags licensed. The EBI90Server will abort, drop out of the task manager. And the Event Viewer log will have a message indicating the EBI90server has been destroyed. Note: That if the user add a unlimited temporary license, the permanent license needs to be removed. The permanent license override a temporary license. This can also occur if SQL does not have access to enough RAM memory during startup of servers. SQL allows the amount of memory it can access during loading of SQL software. EBI90Server destroyed. The following Event Viewer application log entry can be seen if the number of tags in the database exceed the licensed number of tabs. Or if the Primary RTDS has a Redundant license or the Redundant RTDS has a Primary license. 1-110 533583860 Version A1 ABB Technical Support Help File Diagnostic Help File The Diagnostic Help file describes (Future) 533583860 Version A1 1/18/2002 1-111 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Historian This section will present some troubleshooting aids to assist in analyzing and determine problems with the Historian. The main focus in the Historian Module is the process of monitoring the operation and the correction of initial Historian Configurations. This section is not intended to address problems that may occur during normal system operation. Addition information on Historian Troubleshooting will be identified in the OperateIT Troubleshooting Guide. The order in which to troubleshoot a problem will vary depending on the type of problem being experienced Server Explorer Historian Diagnostics To determine is the Historian is function, check the following Historian Server atoms from the Server Explorer. Launch the Server Explorer and find the Historian Server listed in the Server Explorer and right click on the entry. This needs to be performed in the right plane. A Server Diagnostic Popup should appear. Select Diagnostic and then Select the General tab in the popup window The following Tag.Atoms listed within the popup window should periodically change if the historian is historizing data. Values_Lost Values_Received Values_Written Values_Receive_Rate Values_Write_Rate Rates are in Values/Sec Rates greater than 200-250 values/sec are considered high. In the General tab go to the bottom of the listing of atoms and values and find the atoms listed starting with VALUE. Historian Server Diagnostic 1-112 533583860 Version A1 ABB Technical Support With the exception of Values_Lost the values for these atoms should be changing or counting periodically indicating that the historian is historizing data. If the values are Zero or not updating the Historian is not historizing data. To verify if event data is being historized by the historian check the following atoms in the Historian Server diagnostic window. Event_Written Event_Buffer Event_Rate Rates are in Events/Sec Rates greater than 60 Events/sec are considered high. 533583860 Version A1 1/18/2002 1-113 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Historian Server Diagnostic These atoms should periodically change if the historian is historizing event data. 1-114 533583860 Version A1 ABB Technical Support Historian Syslog.log SYSLOG.LOG is a message file that is updated by the historian for logging Historian Activities. The data contained in this file can be operational information and/or problem related information. The historian maintains the SYSLOG.LOG file in the Temp directory (usually C:\temp). This location may also contain a similar file SYSLOGOLD.LOG. This file is created when the SYSLOG.LOG file becomes full. When the file becomes full, the data is copied to the SYSLOGOLD.LOG and the SYSLOG.LOG file is cleared for new incoming messages. SYSLOG.LOG file Size The SYSLOG.LOG file size can be modified. Using NotePad, open the SYSLOG.DEF file located in the same directory as the SYSLOG.LOG. Find the Syslog Size entry and increase the size shown. SYSLOG.DEF A user can modify the information that is logged to this file. It is not recommended that a user change these setting unless directed to do so my either ABB Technical Support or ABB engineering. Program Files > ElsagBailey > MaestroNT > Base Find SysLogModify Double click the file to launch the utility. 533583860 Version A1 1/18/2002 1-115 ABB Technical Support Trouble Shooting Guide Operate IT B.0 SysLogModify Screen Check the items to be included in the SYSLOG.LOG file. A user should not modify this screen unless directed to do so by either engineering or technical services. Click on the Set Mask button to active the selection made. 1-116 533583860 Version A1 ABB Technical Support SYSLOG.LOG file The SYSLOG.LOG file is dynamically updated by the historian. The log file will contain both historian activities as well as error messages. The following information shows examples of various SYSLOG.LOG messages with an explanation to their meaning. [Startup Messages] ============= Server debug definition(s) found ========== 20.01.2000 11:14:22.948 ============= New server attached to syslog ========== Name: 'EbHistorianServer.exe' Debug flags: 0 ======================== 20.01.2000 11:50:55.538 <EbHistorianServer.exe|247|350>: EB_ERR0 Event_Storage::Setup 'Setup complete ' ======================== 20.01.2000 11:50:56.394 <EbHistorianServer.exe|247|350>: EB_ERR0 Proc_Events::Proc_Events 'Event Main Connection to Event Concentrator ' [Informational Message} [Exception processing report: Every hour, can't be turned off] [If Lost not equal to 0 - incoming Exceptions are being lost] (Info also in Server diagnostic fields) – 05.01.2000 01:33:08.099 <EbHistorianServer.exe|385|527>: EB_ERR0 WRITE::STATISTIC ' In=59886527, Lost=0, Checkp=995424(1/checkp), Wrt=59903437(27.7229/write), TabSwt=23851' 05.01.2000 01:33:08.099 <EbHistorianServer.exe|385|527>: EB_ERR0 BUFFER::STATISTIC 'Max granted: 145959, in use: 97306' [ERRORS] [Usually Missing Tables] 14.01.2000 08:57:51.064 <EbHistorianServer.exe|308|859>: EB_ERR0 CEbHistSqlUpdViaShortIdx::CEbHistSqlUpdViaShortIdx 'Short index 299: Cannot update timestamp in Historian_1..BI_493' 14.01.2000 08:57:51.073 <EbHistorianServer.exe|308|859>: EB_ERR0 CEbHistToSqlSrv::Msg 'Call=CEbHistSqlUpdViaShortIdx, Err=42S02, ErrNo=208, ErrMsg='[Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'Historian_1..BI_493'.'' [Subscription failures - Status Codes can be useful - usually recovers on its own] 14.01.2000 10:17:16.990 <EbHistTagAtomToRtds.dll|308|315>: EB_ERR0 CEbHistTagAtomIf::raw_StreamValuesArrived '0x6 unreported VT_ERRORs received' ======================== 17.01.2000 15:08:33.665 <EbHistTagAtomToRtds.dll|279|319>: EB_ERR0 CEbHistTagAtomIf::raw_OperationsStatusArrived '0x4 unreported ItemStatus errors received' 17.01.2000 15:08:33.665 <EbHistTagAtomToRtds.dll|279|319>: EB_ERR0 CEbHistTagAtomIf::raw_OperationStatusArrived 'Error Condition, Status = 8004eb08' 17.01.2000 15:08:33.665 <EbHistTagAtomToRtds.dll|279|319>: EB_ERR0 CEbHistTagAtomIf::raw_OperationsStatusArrived '0x7 unreported ItemStatus errors received' 533583860 Version A1 1/18/2002 1-117 ABB Technical Support Trouble Shooting Guide Operate IT B.0 17.01.2000 15:08:33.675 <EbHistTagAtomToRtds.dll|279|319>: EB_ERR0 CEbHistTagAtomIf::raw_OperationStatusArrived 'Error Condition, Status = 8004eb27' ======================== 17.01.2000 16:59:58.957 <EbHistTagAtomToRtds.dll|279|325>: EB_ERR0 CEbHistTagAtomIf::raw_StreamValuesArrived 'Error Condition VT_ERROR 4eb08' 17.01.2000 16:59:58.957 <EbHistTagAtomToRtds.dll|279|325>: EB_ERR0 CEbHistTagAtomIf::raw_StreamValuesArrived '0x3 unreported VT_ERRORs received' [Received Event Data older than the period covered by the last two open event files.] [If just occasionally, event lost - too old to store, if frequent; check time sync of source] 17.01.2000 15:06:09.976 <EbHistorianServer.exe|279|376>: EB_ERR0 Event_Interface::OnEventNotification 'Error Time to old for Buffers: 1bf60fc229fdb30' [File Error - Not Critical - will happen every hour should be filtered out] 17.01.2000 15:06:58.228 <EbHistorianServer.exe|279|283>: EB_DB40 CEbFileToFromSqlSrv::Connect 'Own machine name is 'BHB_LAB_14'' 17.01.2000 15:06:58.259 <EbHistorianServer.exe|279|283>: EB_DB40 CEbFileToFromSqlSrv::Msg 'Call=Connect, Info=IM006, InfoNo=0, InfoMsg='[Microsoft][ODBC Driver Manager] Driver's SQLSetConnectAttr failed'' 17.01.2000 15:06:58.259 <EbHistorianServer.exe|279|283>: EB_DB40 CEbFileToFromSqlSrv::Connect 'Connect done, info='DRIVER=SQL Server;SERVER=BHB_LAB_14;UID=;APP=Conductor NT;WSID=BHB_LAB_14;Trusted_Connection=yes'' 17.01.2000 15:06:58.582 <EbHistorianServer.exe|279|283>: EB_DB40 CEbFileToFromSqlSrv::DisConnect 'Disconnect from sql server done' [Event Errors - OK if only one - Parameter out of Range - one Event skipped] 17.01.2000 19:21:00.314 <EbHistorianServer.exe|279|389>: EB_ERR0 BufferBlock::Write_Events 'Event Write Error - 1, Params - 1, Perr - 0 ' [Event Errors - Rate/Error because of overflow - some events lost] 17.01.2000 19:21:05.865 <EbHistorianServer.exe|279|380>: EB_ERR0 Event_Interface::OnEventNotification 'Error not enough space in buffer, dwCount = 1a, difference = 1 [Event Concentrator connection lost and recovered - Events may have been lost] 17.01.2000 19:21:06.313 <EbHistorianServer.exe|279|389>: EB_ERR0 Proc_Events::Proc_Events 'Event Main Connection Lost ' 17.01.2000 19:21:06.813 <EbHistorianServer.exe|279|389>: EB_ERR0 Proc_Events::Proc_Events 'Event Main Connection Restored ' [May occur following startup then stop, Ok - if occurs later, may be SQL problem] 18.01.2000 07:23:25.721 <EbHistorianServer.exe|279|447>: EB_ERR0 CEbHistToSqlSrv::Msg 'Call=Insert, Err=42000, ErrNo=913, ErrMsg='[Microsoft][ODBC SQL Server Driver][SQL Server]Could not find database ID 7. Database may not be activated yet or may be in transition.'' 18.01.2000 07:23:25.771 <EbHistorianServer.exe|279|447>: EB_ERR7 CEbHistInsert::Insert 'Cannot write checkpoint table' 1-118 533583860 Version A1 ABB Technical Support 18.01.2000 07:23:26.202 <EbHistorianServer.exe|279|221>: EB_ERR0 Event_Policy_Def::NewFile 'Event New Table Error - -1' 18.01.2000 07:23:31.750 <EbHistorianServer.exe|279|389>: EB_ERR0 BufferBlock::Write_Events 'Event Write Error - 1, Params - 20, Perr - 0 ' 18.01.2000 07:23:31.760 <EbHistorianServer.exe|279|171>: EB_ERR0 CEbHistToSqlSrv::Msg 'Call=Insert, Err=01000, ErrNo=109, ErrMsg='[Microsoft][ODBC SQL Server Driver][Named Pipes]ConnectionTransact (GetOverLappedResult()).'' 18.01.2000 07:23:31.760 <EbHistorianServer.exe|279|389>: EB_ERR0 BufferBlock::Write_Events 'Event Write Error - 1, Params - 10, Perr - 0 ' 18.01.2000 07:23:31.770 <EbHistorianServer.exe|279|171>: EB_ERR7 CEbHistInsert::Insert 'Cannot write checkpoint table' 18.01.2000 07:23:31.770 <EbHistorianServer.exe|279|389>: EB_ERR0 BufferBlock::Write_Events 'Event Write Error - 1, Params - 10, Perr - 0 ' 18.01.2000 07:23:31.770 <EbHistorianServer.exe|279|171>: EB_ERR0 CEbHistToSqlSrv::Msg 'Call=Insert, Err=08S01, ErrNo=0, ErrMsg='[Microsoft][ODBC SQL Server Driver]Communication link failure'' 18.01.2000 07:23:31.770 <EbHistorianServer.exe|279|171>: EB_ERR7 CEbHistInsert::Insert 'Cannot write data table Historian_1..B_3071' 18.01.2000 07:23:31.780 <EbHistorianServer.exe|279|171>: EB_ERR7 CEbHistTagAtom::Truncate 'Obj 2101605 <TIC600.XAY/LOW(uuid=4939388D-8253-11D2-ACF9-00805F8F724F)>: Write to table Historian_1..R_3112 failed for compr type -1' 18.01.2000 07:23:31.860 <EbHistorianServer.exe|279|70>: EB_ERR0 CEbHistToSqlSrv::Msg 'Call=Insert, Err=01000, ErrNo=109, ErrMsg='[Microsoft][ODBC SQL Server Driver][Named Pipes]ConnectionTransact (GetOverLappedResult()).'' [Cannot buffer anything more - loosing data] 18.01.2000 07:23:31.860 <EbHistorianServer.exe|279|171>: EB_ERR7 CEbHistTagAtom::Truncate 'Obj 2101606 <TIC600.XAY/SIG(uuid=4939388D-8253-11D2-ACF9-00805F8F724F)>: Write to table Historian_1..B_3064 failed for compr type -2' 18.01.2000 07:23:31.860 <EbHistorianServer.exe|279|70>: EB_ERR7 CEbHistInsert::Insert 'Cannot write checkpoint table' 18.01.2000 07:23:31.860 <EbHistorianServer.exe|279|171>: EB_ERR7 CEbHistTagAtom::Truncate 'Obj 2101607 <TIC600.XAY/UNITS(uuid=4939388D-8253-11D2-ACF9-00805F8F724F)>: Write to table Historian_1..R_3113 failed for compr type -1' 18.01.2000 07:23:31.870 <EbHistorianServer.exe|279|70>: EB_ERR0 CEbHistToSqlSrv::Msg 'Call=Insert, Err=08S01, ErrNo=0, ErrMsg='[Microsoft][ODBC SQL Server Driver]Communication link failure'' [SQL Database may be full? - write errors from multiple sources] 18.01.2000 07:23:32.451 <EbHistorianServer.exe|279|171>: EB_ERR7 CEbHistTagAtomStore::WriteData 'Incomplete write to sql server: 0 of 5' 18.01.2000 07:23:32.461 <EbHistorianServer.exe|279|70>: EB_ERR7 CEbHistTagAtomStore::WriteData 'Incomplete write to sql server: 0 of 5' 18.01.2000 07:23:32.571 <EbHistorianServer.exe|279|448>: EB_ERR0 Event_Policy_Def::NewFile 'Event New Table Error - -1' 18.01.2000 07:23:33.101 <EbHistorianServer.exe|279|389>: EB_ERR0 BufferBlock::Write_Events 'Event Write Error - 1, Params - 10, Perr - 0 ' Note: Some syslog messages are SQL scripts which can be copied from some of the SYSLOG messages into the SQL Query window and run, to help the user determine some specific problems. 533583860 Version A1 1/18/2002 1-119 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Historian Syslog entries for UTC time error. 31.07.2001 15:16:16.981 <EbHistorianServer.exe|357|620>: EB_ERR0 CEbHistComprRaw::CalculateValue 'Invalid Time Received (future) = UTC:31.07.2001 23:16:16.702 <01C11A16CC953DE0>, delta = UTC:01.01.1601 03:59:59.721 <0000002186E70D90>, table = 2, index = 1040 ' 31.07.2001 15:16:16.981 <EbHistorianServer.exe|357|620>: EB_ERR0 CEbHistComprRaw::CalculateValue 'Invalid Time Received (future) = UTC:31.07.2001 23:16:16.702 <01C11A16CC953DE0>, delta = UTC:01.01.1601 03:59:59.721 <0000002186E70D90>, table = 2, index = 1056 ' 31.07.2001 15:16:16.981 <EbHistorianServer.exe|357|620>: EB_ERR0 CEbHistComprRaw::CalculateValue 'Invalid Time Received (future) = UTC:31.07.2001 23:16:16.702 <01C11A16CC953DE0>, delta = UTC:01.01.1601 03:59:59.721<0000002186E70D90>, table = 2, index = 1057 ' These errors seem to be caused by the Module Time Stamp field in the Harmony Server object being checked. Syslog buffer allocated vs buffer used In the Syslog the allocated buffer and buffer used values are dynamic value. It should go up and down as buffer space is allocated and de-allocated. I'm assuming the value doesn't keep going down until it hits zero. If it goes up and not down that is a problem. It would mean a memory leak. If there is anything else let me know. Archive Syslog SYSLOG.LOG is a message file that is updated by the historian for logging Historian activities. The data contained in this file can be operational information and/or problem related information. This the same syslog files used by the historian for storage related messages. The historian maintains the SYSLOG.LOG file in the Temp directory (usually C:\temp). This location may also contain a similar file SYSLOGOLD.LOG. This file is created when the SYSLOG.LOG fill becomes full. When the file becomes full, the data is copied to the SYSLOGOLD.LOG and the syslog file is cleared for new incoming messages. SYSLOG.LOG file Size The SYSLOG.LOG file size can be modified. Using NotePad, open the SYSLOG.DEF file located in the same directory as the SYSLOG.LOG. [ARCHIVE SERVER STARTUP MESSAGES] 1-120 533583860 Version A1 ABB Technical Support [Normal Delete Process] 26.12.1999 12:10:15.663 <EbArchiveServer.exe|1060|1055>: EB_DB61 EbArchiveServer::EbHARptError 'EbArchiveServer STARTED with input params: arg1-9: < delete, , , , , , , , >' 26.12.1999 12:10:15.663 <EbArchiveServer.exe|1060|1055>: EB_DB63 EbArchiveServer::Main 'EbArchiveServer errors are reported to NT's APPLICATION Event Log' [Normal Archive Process] 26.12.1999 12:10:15.643 <EbArchiveServer.exe|1060|1055>: EB_DB63 EbArchiveServer::Main 'EbArchiverServer Start' 26.12.1999 12:10:15.663 <EbArchiveServer.exe|1060|1055>: EB_DB61 EbArchiveServer::EbHARptError 'EbArchiveServer STARTED with input params: arg1-9: < archive, , , , , , , , >' 26.12.1999 12:10:15.663 <EbArchiveServer.exe|1060|1055>: EB_DB63 EbArchiveServer::Main 'EbArchiveServer errors are reported to NT's APPLICATION Event Log' [EMERGENCY Delete Process: Delete all closed tables up to now and NOT scheduled for Archival] 01.01.2000 16:38:06.024 <EbArchiveServer.exe|1712|1691>: EB_DB63 EbArchiveServer::Main 'EbArchiverServer Start' 01.01.2000 16:38:06.024 <EbArchiveServer.exe|1712|1691>: EB_DB61 EbArchiveServer::EbHARptError 'EbArchiveServer STARTED with input params: arg1-9: < delete, historian_1, 1, , , , , , >' [EMERGENCY Archive Process: Archives all closed tables marked for archival up to NOW instead of up to scheduled time] 26.12.1999 12:10:15.643 <EbArchiveServer.exe|1060|1055>: EB_DB63 EbArchiveServer::Main 'EbArchiverServer Start' 26.12.1999 12:10:15.663 <EbArchiveServer.exe|1060|1055>: EB_DB61 EbArchiveServer::EbHARptError 'EbArchiveServer STARTED with input params: arg1-9: < archive, historian_1, 2, , , , , , >’ 26.12.1999 12:10:15.663 <EbArchiveServer.exe|1060|1055>: EB_DB63 EbArchiveServer::Main 'EbArchiveServer errors are reported to NT's APPLICATION Event Log' [DOUBLE EMERGENCY Delete Process: Delete anything not currently open including archivable tables] 01.01.2000 16:38:06.024 <EbArchiveServer.exe|1712|1691>: EB_DB63 EbArchiveServer::Main 'EbArchiverServer Start' 01.01.2000 16:38:06.024 <EbArchiveServer.exe|1712|1691>: EB_DB61 EbArchiveServer::EbHARptError 'EbArchiveServer STARTED with input params: arg1-9: < delete, historian_1, 3, , , , , , >' [Typical Delete process reporting if enabled] 26.12.1999 12:10:15.643 ============= New server attached to syslog ========== Name: 'EbArchiveServer.exe' Debug flags: AA80840C00000000 26.12.1999 12:10:15.643 <EbArchiveServer.exe|1060|1055>: EB_DB63 EbArchiveServer::Main 'EbArchiverServer Start' 26.12.1999 12:10:15.663 <EbArchiveServer.exe|1060|1055>: EB_DB61 EbArchiveServer::EbHARptError 'EbArchiveServer STARTED with input params: arg1-9: < delete, , , , , , , , >' 26.12.1999 12:10:15.663 <EbArchiveServer.exe|1060|1055>: EB_DB63 EbArchiveServer::Main 'EbArchiveServer errors are reported to NT's APPLICATION Event Log' 26.12.1999 12:10:15.963 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables '# of storage policies> 7' 26.12.1999 12:10:15.963 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'policy id> 1, Period: 1 weeks(5), Increment: 1 days(4)' 533583860 Version A1 1/18/2002 1-121 ABB Technical Support Trouble Shooting Guide Operate IT B.0 26.12.1999 12:10:15.963 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'cmd > historian_master..ebsp_delete_storage_select 1, 0x1bf49b3fe0e8000' 26.12.1999 12:10:16.244 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables '# of Tables found> 0' 26.12.1999 12:10:16.244 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'policy id> 2, Period: 1 months(6), Increment: 1 weeks(5)' 26.12.1999 12:10:16.244 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'cmd > historian_master..ebsp_delete_storage_select 2, 0x1bf347c84e74000' 26.12.1999 12:10:16.284 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables '# of Tables found> 0' 26.12.1999 12:10:16.284 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'policy id> 3, Period: 1 weeks(5), Increment: 12 hours(3)' 26.12.1999 12:10:16.284 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'cmd > historian_master..ebsp_delete_storage_select 3, 0x1bf4a1893436000' 26.12.1999 12:10:16.304 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables '# of Tables found> 0' 26.12.1999 12:10:16.304 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'policy id> 4, Period: 1 days(4), Increment: 1 hours(3)' 26.12.1999 12:10:16.304 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'cmd > historian_master..ebsp_delete_storage_select 4, 0x1bf4ef97a93e800' 26.12.1999 12:10:16.314 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables '# of Tables found> 0' 26.12.1999 12:10:16.314 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'policy id> 5, Period: 1 days(4), Increment: 4 hours(3)' 26.12.1999 12:10:16.314 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'cmd > historian_master..ebsp_delete_storage_select 5, 0x1bf4ef118cf8000' 26.12.1999 12:10:16.364 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables '# of Tables found> 6' 26.12.1999 12:10:16.364 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables ' Delete Table info > R_754, RI_754, Historian_1 754, 1' 26.12.1999 12:10:18.207 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'policy id> 6, Period: 1 days(4), Increment: 1 days(4)' 26.12.1999 12:10:18.207 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'cmd > historian_master..ebsp_delete_storage_select 6, 0x1bf4e6afc890000' 26.12.1999 12:10:18.227 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables '# of Tables found> 0' 26.12.1999 12:10:18.227 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'policy id> 7, Period: 1 days(4), Increment: 1 hours(3)' 26.12.1999 12:10:18.227 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables 'cmd > historian_master..ebsp_delete_storage_select 7, 0x1bf4ef97a93e800' 26.12.1999 12:10:18.247 <EbArchiveServer.exe|1060|1055>: EB_DB55 CEbHADeleteStorage::HADeleteExpiredTables '# of Tables found> 0' 26.12.1999 12:10:18.247 <EbArchiveServer.exe|1060|1055>: EB_DB63 EbArchiveServer::Main 'EbArchiverServer Exit' [Archive Server ERRORS] All errors are reported to the OperateIT Event System through the Historian Server and the Windows NT event log. Typical information fields are: 1-122 533583860 Version A1 ABB Technical Support Event fields Sender: EP: DSCRP: STATE: CMNT: Historian Server Tag Event Point Event Description Event State Event Comment Non-Event fields: Extra: Contains detailed error info sent to Nt Event Application Log and SysLog [In correct setup of DCOM security] 01.01.2000 12:42:11.584 <EbArchiveServer.exe|1680|1688>: EB_DB61 EbArchiveServer::EbHARptError '[HAArchiveStorage::HAPrepDatabase] EP: ARCSVR_DATA DSCRP: Archive Server Data Access STATE: Failed CMNT: Archive Server Data Access Extra: SQL Command> master..sp_create_removable HA_Test_Event_126,HA_Test_Event_126_sys,'e:\ha_historian_archive_event_1\HA_Test_Event_126_sys.mdf',2, HA_Test_Event_126_log,'f:\HA_ArchiveLogs\HA_Test_Event_126_log.ldf',1, HA_Test_Event_126_data,'e:\ha_historian_archive_event_1\HA_Test_Event_126_data.ndf',173 >> Microsoft SQL-DMO (ODBC SQLState: 42000) returned error 15003: [Microsoft][ODBC SQL Server Driver][SQL Server] Only members of the sysadmin role can execute this stored procedure.' 533583860 Version A1 1/18/2002 1-123 ABB Technical Support Trouble Shooting Guide Operate IT B.0 SQL Enterprise Manager The Historian Server is based on SQL for database purposes. Within SQL is the SQL Enterprise Manager (SEM) utility. SQL Enterprise Manager allows the user to view the SQL databases and operation of SQL as well as other aspects of OperateIT. Use the following sequence to access the SQL Enterprise Manager: Start > Programs > Microsoft SQL Server 7.0 > Enterprise Manager SQL Enterprise Manager Expanded SQL Server Group. 1-124 533583860 Version A1 ABB Technical Support Listed in the SEM view are a number of folders under the TS_Operate server. Databases Data Transformation Services Management Replication Monitor Security Support Services Further expansion of the primary folders 533583860 Version A1 1/18/2002 1-125 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Databases The database folder contains all the databases residing in the SQL Server in this example are for the TS_OPERATE machine. Notable databases are: ConfigServer ConfigServer_Mirror ConfigServer_RTDS Archive_Master Historian_1 Historian_Master The Database specifically of interest for the historian are: ConfigServer Archive_Master Historian_1 Historian_Master. ConfigServer The ConfigServer database contains and is the source for all the configured Historian Objects. The objects from the ConfigServer database need to be Pulled by the Historian to determine the historians requirements for collecting, storing and arching data. Historian_Master The Historian_Master database objects that defined the collection and storage requirements. Archive_Master The Archive_Master database contains the archive related objects to define the archiving requirements Historian_1 The Historian_1 database contains all the tables for data stored on the hard disk. In the expanded view of any of the database, a Table object is listed. By highlighting the tables object a listing of the tables for the selected database will be displayed. 1-126 533583860 Version A1 ABB Technical Support The view in the diagram above is for a Historian_1 Databases. In the view on the right are tables labeled B_x, F_x, R_x, and E_x, etc. these are representative of data tables stored data. B-x Compressed data F_x File data R_x Raw data E_x Event data In this case the display of these tables in an indication of the historian historizing data. Historian Master If no data or a specific data type is not being historized the user should look at the tables in the Historian_Master database. The following tables should be looked into to verify what data and/or policies have been configured for the historian. TAGS ATOMS Event_Policy File_Policy If no data is found in any of these tables this can be an indication that some configuration needs to be made or the configuration in the ConfigServer database was not be pulled into the Historian_Master database. If the Tags and or Atoms tables are empty then most likely tag objects have not been configured under the Historian tab in the tag object for assignment to a Historian Atom Group. If the Event_Policy or File_Policy tables are empty, then most likely no policy has been configured and in the RUN mode. 533583860 Version A1 1/18/2002 1-127 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Tags Table The TAGS table will contain a listing of Tags configured to be historization. The following diagram show a TAGS table containing four tags configured to be historized. Tag Table Find and right click on the TAGS Table. Open Table and Return all rows. Atoms Table The ATOMS table will contain a listing of Atoms configured to be historized. The following table shows an ATOMS table contain a list of atoms configured to be historized. Notice that there are several instances of the same atom such as ALARM, PV/SIG, QUALITY etc. Each atom for every tag configured to be historized should appear in this table. All the atoms that relate to a common tag will have a common tag_uuid number in the first column. 1-128 533583860 Version A1 ABB Technical Support Atom Table Event Policy Table The Event_Policy table contains a listing of all configured Historian Event Storage policies. If the table is empty check the Event_Policy_Modify table in the ConfigServer database Event_Policy Table 533583860 Version A1 1/18/2002 1-129 ABB Technical Support Trouble Shooting Guide Operate IT B.0 File Policy Table The File_ Policy table contains a listing of all configured Historian Event Storage policies. If the table is empty check the ConfigServer database File_Policy_Modify table. This example is empty and the user in this case should the check the File_Policy_Modify table in the ConfigServer database. File_Policy Table ConfigServer Open the SQL Enterprise Manager (SEM) on the ConfigServer PC. Find and open the Databases folder under the SQL Server Group for the ConfigServer. Expand the ConfigServer Database and highlight the Tables entry. If any of the tables in the examples above for the Historian_Master databases, look into the corresponding table in the ConfigServer database. If not of the tables contained data, then there is the possibility that the database was not pulled by the Historian. In this case check the Pull_Time table first. Otherwise check the following corresponding tables. 1-130 Tags_Modify Atoms_Modify Event_Policy_Modify File_Policy_Modify 533583860 Version A1 ABB Technical Support Pull_Time Table The Pull_Time table contains the time and date the historian server last pull a database from the ConfigServer. If the last change made to the historian was three days ago, the pull time would be for the time and date three days ago and not the present time. Open the Pull_Times table and view the data within this file. Pull_Times Table The server_name should field should display the Historian Name and the chtime field. The chtime field should display the time the database was last pulled from the ConfigServer. Tags_Modify Tags_Modify Table 533583860 Version A1 1/18/2002 1-131 ABB Technical Support Trouble Shooting Guide Operate IT B.0 This table should open a window with a listing of tags if tags have been configured for historization. Also notice the field tags_modify_type should have the word “update”. 1-132 533583860 Version A1 ABB Technical Support Atoms_Modify Atoms_Modify Table This table should open a window with a listing of Tag Atoms if tags have been configured for historization. Also notice the field Atom_Modify_type should have the word “update”. Event_Policy_Modify Event_Policy_Modify Table 533583860 Version A1 1/18/2002 1-133 ABB Technical Support Trouble Shooting Guide Operate IT B.0 This table should open a window with a listing of event policies if event policies have been configured for historization. Also notice the field event_modify_type should have the word “update”. Note the DELETE in the event_policy_modify column. See the Delete procedure and the end of this section. Event_Policy_Modify File_Policy_Modify Table This table should open a window with a listing of file policies if file policies have been configured for historization. Also notice the field file_modify_type should have the word “update”. “Delete” Procedure If any of these modify tables have entries that start with “Delete” instead of “Update” perform the following procedure as a possible workaround. There was a known problem in Maestro 3.1 with the Historian pulling an initial configuration (First time a configuration is pulled) in which “Delete” entries exist. This a hot-fix to work around the problem. Perform the following SQL commands from the SQL Query Analyzer window. This will be applied to the ConfigServer Database and will remove the “Delete” entries and should allow you to startup. Delete from file_policy_modify where file_policy_modify_type = ‘Delete’ Delete from event_policy_modify where event_policy_modify_type = ‘Delete’ Delete from tags_modify where tags_modify_type = ‘Delete’ Delete from atoms_modify where atoms_modify_type = ‘Delete’ 1-134 533583860 Version A1 ABB Technical Support SPLOADALLMANUAL.SQL The situation can arise where not all the policies from the ConfigServer are copied to the Historian. If this is true or suspect, run the SPLOADALLMANUAL SQL script in the Query Analyzer. Call up the SQL Query Analyzer up on the ConfigServer machine. Once in the SQL Query Analyzer select the ConfigServer database for the pull-down menu. SPLOADALLMANUAL When the script completes, a dialog will appear in the bottom pane display the results of the query. The script takes about a minute to run. The Pull_Time table entry will be deleted and this will force a new pull. The Historian computer will require a restart when the query has completed. 533583860 Version A1 1/18/2002 1-135 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Historian_1 Open SQL Enterprise Manger (SEM) on the Historian PC or the ConfigServer is the Historian SQL database is registered on the ConfigServer PC. Find and open the Databases folder under SQL Server Group for the Historian. Expand the Historian_1 folder and click on the Tables entry. Data being historized should appear in this folder. Look for files such as B_x were x is a number, R_x, BI_x, E_x, etc. These are various files for the different data types being historized. Historian_1 Tables 1-136 533583860 Version A1 ABB Technical Support SQL Query Analyzer In SQL there is a Query Analyzer utility available to the user to look into the system database. The Query analyzer can be used as a troubleshooting tool to help the user determine what historian objects have been configured and to determine what data has been historized and/or archived. This section will demonstrate the use of some simple Query Analyzer commands and scripts to help the user become more familiar with this tool. The following scripts can be entered into the Query Analyzer utility and executed. When using the query analyzer, make sure you have selected a valid Database in the “DB” pull down. If you get an error message after running the script, double check your syntax, and verify that you have the correct database selected. Run the query analyzer from the server machine that contains the database being queried. The ConfigServer database is on the ConfigServer machine and all Historian databases are on the historian machine. To access the query analyzer, use the following sequence of steps shown below. Start > Programs > SQL 7.0 > Query Analyzer The SQL Server should be (local) and Connection Information set for Use Windows NT authentication. Click on OK to continue. 533583860 Version A1 1/18/2002 1-137 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Query Analyzer Upon launching the Query Analyzer, select a database that you want to query. The following are valid choices for OperateIT if the default database names were used during Software Installation. Choices: ConfigServer Historian_1 Historian_Master Archive_Master The following example uses the Historian_Master database. Each SQL script is shown in Italic characters. After entering the query script, move the cursor so that it is located in the first line of the entered script or highlight the entire script. Then click the green arrow in the menu bar to execute the script. The user can enter multiple queries in the work spaced. Highlight the query you want to execute and then click on the green arrow. 1-138 533583860 Version A1 ABB Technical Support TAGS Table The following script will display the contents of the TAGS table. This table will contain the tags within the system’s database that have been configured to be historized. If this table is empty, then no tags are going to be historized. SELECT * FROM TAGS TAGNAME Table The following script will display the same information that was displayed in the first script except the data shown will be sorted in alphanumeric order by tagname. SELECT * FROM TAGS ORDER BY TAGNAME 533583860 Version A1 1/18/2002 1-139 ABB Technical Support Trouble Shooting Guide Operate IT B.0 ATOMS Table The following script will display the contents of the ATOMS table. This table will contain the Atoms within the system’s database that have been configured to be historized. If this table is empty, then the historian is not aware of any Atoms being configured to be historized by the historian and no atoms are going to be historized. SELECT * FROM ATOMS 1-140 533583860 Version A1 ABB Technical Support The following script will display the same information that was displayed above except the data shown will be sorted in alphanumeric order by Atom name. SELECT * FROM ATOMS ORDER BY ATOM 533583860 Version A1 1/18/2002 1-141 ABB Technical Support Trouble Shooting Guide Operate IT B.0 LOOKUP Table Display the contents of the lookup table. The lookup table will contain the list of tables that contain historized data. This is a good reference table to look at to see what tables have been created by the historian. SELECT * FROM LOOKUP 1-142 533583860 Version A1 ABB Technical Support The following script will display the same information that was displayed above except the data shown will be sorted in alphanumeric order by the Lookup_ID SELECT * FROM LOOKUP ORDER BY LOOKUP_ID 533583860 Version A1 1/18/2002 1-143 ABB Technical Support Trouble Shooting Guide Operate IT B.0 TAGS & TAGNAME Table A user looking at the Atoms table may not be able to determine what tags the listed atoms represent. To display the information in the atoms table along with the tagname for the displayed atoms, the INNER SQL command can be used. The following query will generate a listing of Tags, TagAtoms, Data_Type and the Retired status for all atoms configured to be historized. This query references two tables in this script, the Atoms and Tags table. Since both tables use the TAG_UUID property, the two tables can be linked together by the TAG_UUID field to display the tagname along with the atom information. SELECT TAGS.TAGNAME, ATOMS.ATOM, ATOMS.DATA_TYPE, ATOMS.RETIRED FROM ATOMS INNER JOIN TAGS ON ATOMS.TAG_UUID=TAGS.TAG_UUID This last query uses a common key “tag_uuid” to link the two tables together so the user can match the tagname found in the Tags table with corresponding Atoms names found in the Atoms table even thought the atoms table did not have a tagname column reference. 1-144 533583860 Version A1 ABB Technical Support The tag_uuid will be the same for both table for the same tags listed and will allow the user to list columns from both table in one listing. To run the same query for a specific tag.atom pair run the following script. SELECT TAGS.TAGNAME, ATOMS.ATOM, ATOMS.DATA_TYPE, ATOMS.RETIRED FROM ATOMS INNER JOIN TAGS ON ATOMS.TAG_UUID=TAGS.TAG_UUID WHERE TAGNAME = ‘tag_name’ Note: ‘tag_name’ = the tag being queried 533583860 Version A1 1/18/2002 1-145 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Check Script Files Historian Script files The following CHECK script files can used to view some parts of the configuration of the historian. Program Files/ElsagBailey/Symphony Configuration/Historian Scripts ConfigServer_Check.SQL ConfigServer_Check2.SQL ConfigServer_Check3.SQL ConfigServer_Check4.SQL Report_Hist_Config_All.SQL Report_Hist_Config_Default.SQL Running Scripts To run one of the listed scripts, open the SQL Query Analyzer. Access to the SQL Query Analyzer as follows. Start > Program > Microsoft SQL Server 7.0 > Query Analyzer SQL Server field should be (local) and set the Connection Information to “Use Windows NT authentication. In the SQL Query Analyzer window select ConfigServer from the DB pull-down list. 1-146 533583860 Version A1 ABB Technical Support DB Selection Click on the Open icon (Folder) and Browse to the SQL script file to run. All the check scripts are located in Program Files/ElsagBailey/Symphony Configuration/ConfigServer Database Creation/Historian Scripts Historian Scripts location In this case the Report_Hist_Config_all.sql is displayed. By Selecting and opening this file the following Query Analyzer view will appear. 533583860 Version A1 1/18/2002 1-147 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Script File Run the script by clicking on the Green Arrow pointer. The follow result view will appear. 1-148 533583860 Version A1 ABB Technical Support Script Results The information in the lower section is the result of the query. This script file displays a summary of all the Historian object configurations. This query process can be used for any of the other listed scripts above or any other SQL script available to the user. 533583860 Version A1 1/18/2002 1-149 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Historian RAW/BCBS Historization check If changing or configuring atom to be historized, compare the Atom_Modify table in the ConfigServer database to the Atom table in the Historian Master database. The ConfigServer database is the base, which should be copied to the Atom table in the Historian Master database. Compare these two tables to verify if the configurations have be copied to the Historian. Raw data parameters raw_collection_id raw_lookup_id BCBS data parameters bcbs_collection_id bcbs_lookup_id The parameters for either the Raw or BCBS parameters should be none zero if configured to be historized. Atom_Modify (ConfigServer Database) Atom (Historian Master) 1-150 533583860 Version A1 ABB Technical Support RowSetViewer RowSetViewer is a Microsoft Application to view data from various servers, and used in OperateIT to view stored Historian Data. There is user’s guide located in OperateIT under Stored Procedures folder under the Symphony Historian folder called Provider Examples.txt. Program Files > ElsagBailey > Symphony Historian > Stored Procedures Provider Examples.txt The executable file RowSetViewer.exe to launch this utility is only located only on a historian node. Program Files > ElsagBailey > Symphony Historian RowSetViewer.exe Launch RowSetViewer by double clicking on the file. RowSetViewer Click on File in the RowSetViewer window. 533583860 Version A1 1/18/2002 1-151 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Select Full Connect or Ctrl/F Full Connect Provider dialog window. Connection configuration Saved Config Default Provider MNT_Historian Click OK to continue. 1-152 533583860 Version A1 ABB Technical Support RowSetViewer window The command dialog field is the white area just below the menu bar. In the RowSetViewer utility, there are three primary command that will normally be using with OperateIT. Tag.Atom Event File When these commands are used in the simplest form the command by its self, use one of the following two commands to execute. Go to Session menu pull down Select OpenRowset Enter Ctrl/O The first example is the user of a Tag.Atom name. Enter a known tag.atom name in the command dialog window and then execute the command. Go to Session menu pull down Select OpenRowset 533583860 Version A1 1/18/2002 1-153 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Note: If the Enter key has been entered or the cursor is not on the same line as the command an error message may occur when executing the entered command. Tag_Atom Data Tag Atom data can be retrieved in the RowSetView by entering the Tag.Atom name in the command window. Tag.Atom command results The results of the Tag.Atom command displays a number of entries for the specified tag.atom pair. The results will return the time, value and quality for each sample displayed. To collect TAG Data perform the following steps: Enter a valid Tag Name, in this example a tag named K_SECOND.PV/SIG is being used and entered into the Dialog window There are two methods to launch the collection process. 1-154 533583860 Version A1 ABB Technical Support 1. Go to Session pull down Select OpenRowset 2. Enter Ctrl/O Refer to the Tag Data Table below for an example of returned data. Columns not shown in Tag Data Table are as follows: Operator Application Type Web Mime Type File Policy File Status File Size File ID File Index The Tag.Atom, Event, and File commands can be expanded with addition parameters to specify data for a specific period or based on a specific attribute of a tag or event. When using any of the three commands with additional parameters, or SQL scripting such as start and end times, Compression, interval units etc. use one of the following commands to execute there variations of the command. Go to Command menu pull down Select Icommand Execute Enter Ctrl/E To use a tag.atom command and request data for a specific time period enter the following command in the command dialog window and then execute the command. 533583860 Version A1 1/18/2002 1-155 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Tag.Atom by time period 1-156 533583860 Version A1 ABB Technical Support Event Data To request Event data, enter EVENT in the command dialog window and then execute the command. Go to Session menu pull down Select IOpenRowset OpenRowset Event Command This should return a listing of tag data saved to the historian. Starting at the present time and going back in time. There is a limit to the buffer and the listing will only show the amount of data the buffer can hold. The user can specify start and end time of the data to be collected. 533583860 Version A1 1/18/2002 1-157 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Note: The figure above only displays a subset of the total number of column in the returned event table. The are 36 columns. The following is a list of all columns within the actual returned event table. 1-158 Row Handle ObjectName ObjectID ServerID GroupID Area Description Unit EngUnit State Inhibited Equipment EventPoint Category TypeID Type Class Comment Quality EventTime EventTimePrecision TimeInvalid ViolaatedLimit Enabled Alarm 533583860 Version A1 ABB Technical Support Priority Dual Active ActiveTime ActiveTimePrecision InactiveTime InactiveTimePrecision Acknowledged AckRequired PageAckEnabled User 533583860 Version A1 1/18/2002 1-159 ABB Technical Support Trouble Shooting Guide Operate IT B.0 To request Event data for a specific period of time enter the following command and execute. Event Data by time period 1-160 533583860 Version A1 ABB Technical Support Filter Call on EventPoint A query to collect the history of a tag atom can be performed, as in the case of the following EventPage. Event Page In the EventPage is a sender K_RCM_1.OUT. The following example will return data for a tag OUT EventPoint for an RCM tag for a 10 minute period. The following parameters are used in the Event Command. Sender K_RCM.OUT ObjectName = K_RCM EventPoint = OUT EVENT, 10/24/00 08:30:00 AM, 10/24/00 08:40:00 AM, (OBJECTNAME like ‘K_RCM’ AND EVENTPOINT like ‘OUT’),, ETIME 533583860 Version A1 1/18/2002 1-161 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Command Results 1-162 533583860 Version A1 ABB Technical Support File Data To collect File Data perform the following steps: Enter FILE in the Dialog window (top portion on the MNT_Historian window) There are two methods to launch the collection process. 1. Go to Session pull down Select OpenRowset 2. Enter Ctrl/O The table will appear containing retrieved data from the historian. The example below shows only the first five columns. The actual table contains 13 columns of data. File Command 533583860 Version A1 1/18/2002 1-163 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Provider Example.txt File The following information is a copy of the Provider Example.txt file, which is found on a Historian node at the following location. Program Files > ElsagBailey > Symphony Historian > Stored Procedures Provider Examples.txt This document provides information on the syntax for RowSetViewer. Provider Example.txt This text file provides sample OLE-DB commands for accessing the Historian, using ADO, or the RowSetViewer. The following commands can be used to access data via the IOpenRowset interface [For a Tag.Atom] GC-FIC-459.PV/SIG [For Event Data] EVENT [To Obtain a list of all files stored] FILE [To retrieve a single file (where 1.1 is the File_id . File_Index obtain in the file list command)] FILE-1.1 The Following Commands can be used to access data via the ICommand Interface. [For Tag.Atom data - Start time, End Time, Compression type (0-Raw, 1-Avg, 2-Max, 3-Min, 4-Sum, 5-Int, 6-Range, 7-First, 8-Last), time interval for Compression, time interval units (1-Sec, 2-Min, 3-Hour, 4-Day, 5-Week, 6-Month, 7-Year, 8-Shift, 9,-HalfDay, 10-TwoMonth, 11-Quarter, 12-HalfYear ). Data will be retrieved from start time to end time (start time does not have to be less then end time) with the requested Compression function (i.e. 1 hour averages)] GC-FIC-457.PV/SIG, 12 Feb 1999 10:56:00, 12 Feb 1999 2:56:00, 1, 1, 3 [For Event data - Start time of data, End time of data, filter, Server, Sort order. The start time must be less then the end time. The filter (optional) can be any valid SQL expression, The Server parameter optional) limits events to the events collected on a single server. The sort order information (optional) can be any 1-164 533583860 Version A1 ABB Technical Support field(s) and the ASC (Ascending) or DESC (descending) key words] EVENT,2/18/99 7:45:00.000,2/18/99 7:46:00.000,NOT ("NAME" LIKE 'PROFANG1') AND NOT (EVENTPT LIKE '%.BAD'),Historian1 ,ETIME EVENT,2/1/99 11:00:00.000,2/1/99 11:00:10.000 EVENT,1/29/99 7:45:00.000,1/29/99 8:45:00.000,NOT ("NAME" LIKE '%Server%') AND NOT (EVENTPT LIKE '%.BAD'), ,ETIME EVENT,2/18/99 7:45:00.000,2/18/99 7:46:00.000,NOT ("OBJECTNAME" LIKE 'PROFANG1') AND NOT (EventPoint LIKE '%.BAD'), ,EventTime EVENT,2/18/99 7:45:00.000,2/18/99 7:46:00.000,NOT ("OBJECTNAME" LIKE 'PROFANG1'), , OBJECTNAME ASC,EventTime DESC [For File data - Start time of data, End Time of data, filter] FILE,2/18/99 7:45:00.000,2/18/99 7:46:00.000, File_name like '%symevent%' and Operator like '%lab%' [For File retrieval -The File_Id.File_Index obtained from the file list command ] FILE, 1.1 [For file Storage - The file name, the time to associate with the file, A file Policy to associated with the file (defines storage parameters - see configuration), the directory which contains the file (must be vailable via share to the Historian), the operator, the application type and a description. The file name, file policy and file location are required. The time field if not provided will default to Current time. perator will default to the security context of the connected user. Application will default to "Manual". escription will default to a blank ] FILE, Lab6SymEventReport1, 2/18/99 7:45:00.000, 1, "Report Description" \\share_machine\share_name, SymReport, Report, [Using German Date Format:] EVENT,17.02.99 14:30:00.000,17.02.99 15:30:00.000,NOT ("NAME" LIKE '%Server%') AND NOT (EVENTPT LIKE '%.BAD'), ,ETIME 533583860 Version A1 1/18/2002 1-165 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Historian Databases – Suspect Error When the Historian Machine boots up it performs an integrity check on the various databases. Historian_1 Historian_Master Archive_Master If the integrity test find the database as being suspect the SQL Enterprise manager will gray out the various suspect database. They become in-accessible. Those database will be marked accordingly as SUSPECT. The user should run the DBCC CHECKDB routine to analyze the problem. Run the DBCC CHECKDB in the SQL Query Analyzer. Select the database in the DB pull down menu to select db to analyze. The CHECKDB may provide information and the option to make repairs. If repairs are not an option, the Historian machine will need to be reloaded. The OperateIT software may only need to be reloaded. If running redundancy, the historian database may be (again maybe) be backed up from the good historian and restored to the bad machine. If this is performed, the Historian must be shut down. Stop the Historian server, and backup the database. Then restore the backups to the bad machine. If this does not correct the problem reload. Possible cause 1-166 Power failure User playing with SQL 533583860 Version A1 ABB Technical Support Historian Server always Establishing In Server Explorer screen Sometimes the Historian Server will show up in the Server Explorer hung in the Establishing mode. This typically would occur after a Software load procedure. Check the Historian Server Object under the Historian General tab and verify that the correct Historian Host Name and SQL Server Name. Also check the General_Config_Modify in the ConfigServer table and compare this to the General_Config table in the Historian_Master table using SQL Enterprise Manager. These two tables should match and the Historian_Name column should have the Historian Machine name. Check to see if the Historian_1 database owner is SA instead of OperateServer. If SA then you need to reload software. Appears that the user did not pick NT Authentication during software load. Historian Ebsp_Config_Pull_All To pull the database from the ConfigServer to the Historian Master is typically performed by using the SPLOADALLMANUAL.SQL script. The Pull_Time can be reset by using the following command which will perform the same function as the SPLOADALLMANUAL, but does not perform all the function of the SPLOADALLMANUAL which is not necessary when just trying force a pull database function. In the Query Analyzer, on the Historian_Master database, perform the following command which includes three parameters. Exec Ebsp_Config_Pull_ALL ‘ConfigServer_Database_Name‘, ConfigServer_Machine_Name’, ‘Historian_Machine_Name’ Each parameter should be enclosed in single quote ‘ ‘. Should not have to restart after executing the script. Note: If ebsp_config_pull_all does not exist, it is supposed to be created when the Pull_Config.SQL (Historian Configuration/Stored Procedures) script is executed during installation. It can be manually run by opening the file in the SQL Query Analyzer. It will select the database it needs when it is run. 533583860 Version A1 1/18/2002 1-167 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Historian data rates Historian Data rates. The supported event rate is 40/sec. The supported exception rate is 100/sec, but Paul says it can do 200 with no problem. Performance Monitoring The use of Microsoft Performance Monitor can be helpful to investigate Historian performance issues. The following list is a set of Performance objects and counters that may be helpful during this process. Disk counters are inactive by default and need to be activated. Activation of the disk counters will cause the system to use additional resources and the disk counters should be de-activated when the performance monitor activity is completed. It should be noted that the selection of counters used in this example is only an initial guide line and should not be considered a fixed collection of counters. There are numerous options available for investigating problems with Performance. To check to see if the disk counters are active from the DOS Prompt enter the following command “DISKPERF”. 1-168 533583860 Version A1 ABB Technical Support The message returned will indicate if the disk counters are set to start or never to start as indicated in the above window. To enable or disable the disk counters, use one of the following commands in the DOS Prompt window. To enable the counter to start at boot, type “DISKPERF –y”. To disable the counter to start at boot, type “DISKPERF –n”. Enable Diskperf The PC will require a reboot to enable or disable the counter after executing either of the two commands. Performance Monitor Performance Monitor is launch by the following procedure. Start > Programs > Administrative Tool (Common) > Performance Monitor 533583860 Version A1 1/18/2002 1-169 ABB Technical Support Trouble Shooting Guide Operate IT B.0 The performance monitor screen will come up with a blank window as shown above. The user will need to configure the specific counters to be monitored. The following is a basic list of counter for monitoring the Historian. Objects Counters Processor % Processor Time SQLServer:Databases Transactions/sec SQLServer:SQL Statistics SQL Compilations/sec SQLServer:Buffer Manager Page Writes/sec SQLServer:Buffer Manager Page Reads/sec LogicalDisk Disk Reads/sec LogicalDisk Disk Writes/sec Description of the listed Object Counters Processor: % Processor Time 1-170 533583860 Version A1 ABB Technical Support Processor Time is expressed as a percentage of the elapsed time that a processor is busy executing a non-Idle thread. It can be viewed as the fraction of the time spent doing useful work. Each processor is assigned an Idle thread in the Idle process which consumes those unproductive processor cycles not used by any other threads. SQLServer:Databases: Transactions/sec Number of transactions started for the database. SQLServer:SQL Statistics: SQL Compilations/sec Number of SQL compilations. SQLServer:Buffer Manager: Page Writes/sec Number of physical database page writes issued. SQLServer:Buffer Manager: Page Reads/sec Number of physical database page reads issued. LogicalDisk: Disk Reads/sec Disk Reads/sec is the rate of read operations on the disk. LogicalDisk: Disk Writes/sec Disk Writes/sec is the rate of write operations on the disk. The following window is a view of the Performance Monitor configured for the listed counters. Performance Monitor 533583860 Version A1 1/18/2002 1-171 ABB Technical Support Trouble Shooting Guide Operate IT B.0 To monitor these counters using Performance Monitor, click on the Edit function in the menu bar and select Add A Chart, and the following popup window will appear. Add to Chart Computer 1-172 Contain the PC host name on the PC running the performance monitor. 533583860 Version A1 ABB Technical Support Object Objects identify a system resource. A pull-down menu is provided to get a list of valid object. Each object will have a unique list of counters. Counter A listing of valid counters for the listed object. Instance The instance field will display all instances of a counter in the system. A total instance will represent the total count for all instances. To add a counter, select an object type from the pull down menu, select a Counter and then click the Add button. Repeat these steps for all counters to be configured into the monitor screen. When done click on the DONE button, which will replace the CANCEL button as counters are being added. Users can save the configuration setting for a session. Click on File in the menu bar and select SAVE CHART SETTINGS AS. Enter a name and a .PMC file will be created and can be used again at a later time. 533583860 Version A1 1/18/2002 1-173 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Historian Performance Calculation The Historian Performance Calculation is a tool to assist a user in determining machine requirements for the historian. The calculation file is found on OperateIT as follows. Start > Program Files > ElsagBailey > Symphony Historian > Stored Procedures > Performance.xls The performance numbers are for single historical data disk systems. If more then one data disk is provided, the number of I/O operations for each disk would have to be calculated. The memory requirements are only 1-174 533583860 Version A1 ABB Technical Support approximations, more memory will never hurt, less memory could effect performance, but unless paging occurs the effect could be minor. These calculations do not account for other activities - i.e. long term archival or backup operations. Both of these operations will use all available bandwidth while they are active, as such they should be schedule for periods of low activity (Saturday nights/Sunday mornings). Reports and Historical Event Page callups are not included in this estimate. 533583860 Version A1 1/18/2002 1-175 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Historian Diskspace Calculation The Historian Diskspace Calculation is a tool to assist a user in determining machine requirements for the historian. The calculation file is found on OperateIT as follows. Start > Program Files > ElsagBailey > Symphony Historian > Stored Procedures > Diskspace.xls 1-176 533583860 Version A1 ABB Technical Support Historian Faceplate show RTDS Failure This is typically because a subscription to a tag.atom may not be valid. This could be because the Historian ATOM Group is looking for an invalid tag.atom. Check the syslog.log file for error message for the tag in question. Check the event viewer logs for subscription errors. Licenses Access OperateIT License entry: License entry window 533583860 Version A1 1/18/2002 1-177 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Verify that the user specific license is listed. SLA file: NT Explorer 1-178 533583860 Version A1 ABB Technical Support Dr. Watson Error on License Entry A Dr. Watson log occurs when launching the License Entry feature. Two reports have come in that have been caused by an on board (motherboard network driver). Disabling the on board drivers and setting up a Network board corrected the problem. Lock Object Sometimes an error message appears during the configuration of an object in the Configuration Browser that certain records are locked by a user, but there is no user logged in using those records. This can occur if a user is in the Configuration Browser or in the Bulk Configuration Manager and improperly exits the function. If the situation should occur that an object is locked and cannot be unlocked by the BCM. The user can to go into the SQL Enterprise Manager to unlock the objects. Procedure to Unlock an object: 1. Open SQL Enterprise Manager 2. Open the ConfigServer Database 3. Open the Tables folder 4. Locate and right click on the SymLock table. 5. Return all rows SymLock Table The table will contain an entry for each object that is lock by a user. Highlight the entry to be unlocked and then press the Delete key on the PC Keyboard. This will delete the object from the SymLock Table and release the object to users. If there are a large number of objects listed and the user only wants to unlock a specific object, lockup the Tag object names in the SymObjectLang Table. Locate and record the 533583860 Version A1 1/18/2002 1-179 ABB Technical Support Trouble Shooting Guide Operate IT B.0 OBJECTID for the locked object. Then return to the SymLock table and find the entry with the recorded OBJECTID number. 1-180 533583860 Version A1 ABB Technical Support Network Multiple Network Card Applications OperateIT supports more than one network card. The network card for the OperateIT network connection must be the first card in the binding order for all services and protocols in the Windows NT network properties. In addition the adapter connected to the OperateIT network must be used in the route table entry for the limited broadcast address (255.255.255.255). This is always the adapter that was introduced to the network properties on the Windows NT last. The following procedure can be used to assure time synchronization works: 1. Add the adapter for the OperateIT network last to Windows NT. If the adapter was already added, remove it, restart the machine, and add the adapter back. 2. Change the binding order in the network properties so that the adapter for the OperateIT network is first in the binding order for all services and protocols. 3. Restart the Machine Note: You must follow these directions on all machines that have multiple network cards. If not, the machines will not be synchronized because it will either not listen for time synchronization broadcast messages on the right network or will send broadcasts to the wrong network. 533583860 Version A1 1/18/2002 1-181 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Performance OperateIT slower after Mirror and RTDS Databases grow. Why does performance decrease sometimes over time This is actually a bug that is fixed in the next release. Overtime the data and indexes in the database become fragmented. This can be worked around by creating a maintenance plan that periodically optimizes the mirror database. Registry Within the registry are a number of entries that can be useful in troubleshooting problems. Launch Regedit from the Run Utility. The prime focus is the ElsagBailey hive. 1-182 533583860 Version A1 ABB Technical Support Installed Products 533583860 Version A1 1/18/2002 1-183 ABB Technical Support Trouble Shooting Guide Operate IT B.0 The Installed Product folder displays the OperateIT software that has been loaded onto the host PC. In this example the following software has been loaded: Harmony Add-On MaestroNT Client OPC Add-On Configuration Server Historian RTDS Product The Harmony RTDS Add-On is highlighted in the view and there is a corresponding view of key in the right pane. By looking at the RTDS key, in the right pane, the data field is “Yes”. Yes indicates that the Real Time RTDS component (Server software) has been loaded. A “No” would indicate that RTDS components have been loaded for Harmony Object support but the PC is no a RTDS Server. 1-184 533583860 Version A1 ABB Technical Support Historian Product The Historian Product is highlighted in this example and the right pane displays key relating to the Historian. RTDS key indicates that this machine is the Historian Server. SQL Historian Master_Master Data partition is located on theD drive. SQL Historian_1 Data Partition is located on the D drive. SQL Historian data base is named Historian_1 (Default). Reloading Software Products During the reload of software, typically a menu will appear that will prompt the user to either user the Previous Setting during the initial load, or to add a New product. The occasion may occur that this prompt does not appear. Or sometime when reloading software, the Software install shield may not provide the user options during the install to reload all the components of a server or add-on or it will not allow the user to initialize the ConfigServer or Historian Databases. In these situations, the user can delete the Product key for the specific product being reloaded. Then when running the software install the prompt will appear and the option to initialize database should appear. The user can delete all entries under the Install Products folder if desired prior to a reload of software. It is recommended that the user does not delete the Install Folder itself. 533583860 Version A1 1/18/2002 1-185 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Symphony Common Key DatabaseCommandTimeout DatabaseConnectionTimeout EnableLogging EnableLogging Install Path MaximumExpectedFailoverTime MaximumServerResponceTime MaxSyncWaitTime TagData File TagData Path 1-186 Default Value 0 Location All Indicates how long (in seconds) to wait while executing a SQL command before terminating the attempt and generating an error. 0 = no timeout Indicates how long (in seconds) to wait while establishing a connection to the database before terminating the attempt and generating an error. 0 = no timeout 0 1 = allows the RTDS to log a summary to the NT application log of online changes submitted by the ConfigServer DSS. The DSS lock component also logs a message whenever it obtains and releases a lock prior to replication. 0 All All All 1 = allows the RTDS to log a summary to the NT application log of online changes submitted by the ConfigServer DSS. The DSS lock component also logs a message whenever it obtains and releases a lock prior to replication. <drive>:\Program Files\ElsagBailey Software installation path. 30000 (30 seconds) Maximum time to wait for failover to complete before issuing an error. 45000 (45 seconds) Maximum time to wait for a server to respond to a request. 30000 (30 seconds) Default timeout for synchronous calls (ie. PutSync, SubscribeSync) TagData.mdb No longer applicable. Cache$ All All All All All All 533583860 Version A1 ABB Technical Support Name of the cache share on the RTDS computer used to locate the presist.mdb file. The share usually represents the "Program Files\ElsagBailey\Maestro NT\Base\Cache" folder. Provider=SQLOLEDB;Integrated Security=SSPI;Data Source=(local);Initial Catalog=ConfigServer_RTDS;OLE DB Services=-2 Connection string used to access the RTDS database TagDataDSN All Symphony Common Folder A problem can occur when loading software, where the user may have entered the incorrect Name for the ConfigServer or the incorrect Name of the ConfigServer database. Check he TagDataDSN key in the right pane and verify the computer name is correct. In this example the computer name is (local). This is correct for a ConfigServer machine. On any other machine the name should be the Network name of the ConfigServer machine. OperateIT Server folders Under the Symphony Common folder is a listing of OperateIT servers resident on the host PC. When making configuration changes to the system Definition and/or Server Objects, the Server Explorer may encounter problems display the server within the OperateIT system. Sometime there may be an incorrect domain name for a specific server for an older server object or from an old database that has been overwritten by a new database. To correct the server explorer, the user may need to delete the server entries in the registry, delete all the Out Of Server objects from the ConfigServer database and restart the PC to correct the Server Explorer. Do Not delete anything other than the Server folders. In this example, they are the folders starting with the names of OPERATEIT………. Key ActiveSyncTimeout(s) Default Value 1 Location RTDS Internal redundancy timer in seconds. Specifies that cycle time of the redundancy system before all tags have been synchronized. This is the period at which tags are checked to see if they have indicated that they require synchronization. This is a faster period that the "IdleSyncTimeout(s)" registry setting AutoActivate 1 Not Used RTDS Description DiagnosticMessageThreshold GroupX\NodeX\ServerX RTDS RTDS 0 This is the level below which diagnostic messages are not logged. 0 = no limit EnableOnlineChanges Enable RTDS "Enable" or "Disable" the processing of online changes for a specific RTDS server. IdleSyncTimeout(s) 60 RTDS Internal redundancy timer in seconds. Specifies that cycle time of the 533583860 Version A1 1/18/2002 1-187 ABB Technical Support Trouble Shooting Guide Operate IT B.0 redundancy system when all tags have been synchronized. This is the period at which tags are checked to see if they have indicated that they require synchronization. LookOutTime(s) MaxInactiveTime(s) MaxSyncTags PingSensitivity(sec) SyncSensitivity(s) 1-188 30 RTDS Internal redundancy timer in seconds. Specifies the maximum time that a RTDS waits at load time, to see if its partner is also starting up. 10 Internal redundancy timer in seconds. Specifies the maximum time that both RTDS will both be in the inactive state before they negotiate and one of them goes active. 250 Total numbers of tags in a batch when an inactive RTDS synchronizes its tags with its active partner. (ie. 1000 tags will be synchronized in 4 batches of 250 tags) 10000I nternal RTDS redundancy timer in milliseconds. Both RTDS's in a redundancy partnership ping each other to ensure that redundancy is functioning correctly. This timer specifies the maximum time that a RTDS will wait for the return from a ping call before assuming that the called RTDS has failed. This will cause the callee to the assume active redundancy state. Modify with caution. If this is made to short it will result in "False Failover" i.e. a RTDS assumes that its partner has failed when the only problem is a network glitch. Making this timer to long results in a time delay from when a RTDS fails to when its partner takes over. Time allowed for the <MaxSyncTags> batch of tags to synchronized. CAUTION: balance MaxSyncTags and SyncSensitivity to allow the tags to synchronize in a reasonable time. (ie. increase SyncSensitivity in proportion to MaxSyncTags) NOTE: tags are locked during this process. Avoid increasing SyncSensitivity to more than 5 sec. RTDS RTDS RTDS RTDS 533583860 Version A1 ABB Technical Support DataSyncService Field DataSyncTime(secs) Default Value 25 Location ConfigServer The interval at which the ConfigServer DSS processes online changes. EnableDataReplication 1 ConfigServer Enable (1) or disable (0) processing of online changes on the ConfigServer EnableFileReplication 1 All Enable (1) or disable (0) file replication. Files to be replicated are listed in text files on the configuration server in the "Program Files\ElsagBailey\Symphony Configuration\Source\SyncFiles" directory. These files contain the source and destination directory to be replicated. EnableMirrorReplication 1 All Enable (1) or disable (0) mirror replication EnableRTDSReplication 1 All Enable (1) or disable (0) RTDS replication. EnableSchedReplication FileSyncTime MirrorQuality 1 Enable (1) or disable (0) scheduled replication. 5 (minutes) File replication interval (in minutes) 1 Quality status of mirror database indicates whether this node can be used as a mirror site. 533583860 Version A1 1/18/2002 ConfigServer All Mirror 1-189 ABB Technical Support Trouble Shooting Guide Operate IT B.0 SchedReplicationTime ShutDownServers StartServers SymLockTimeout(s) 1-190 1 = good 0 = bad 02:00:00 Time at which a forced replication will occur to all subscribed RTDS and mirror nodes. Time is in 24 hour format - default is 2 AM. 1 Enable (1) or disable (0) shutdown of servers when shutting down the datasync service. 1 Enable (1) or disable (0) startup of servers when starting the datasync service. 16 The number of seconds allowed for a remote DSS to ping the ConfigServer DSS to indicate the lock is still valid. This is used to avoid lock ups in the remote DSS from preventing other nodes from replicating. Before a remote DSS begins replication it obtains a lock from the ConfigServer DSS. In order to keep the lock valid, the remote DSS has to ping the ConfigServer DSS within the 16 sec period. If the remote DSS fails to ping within this period the ConfigServer DSS will force the lock to be released and granted to the next DSS in the queue waiting to obtain a lock. ConfigServer All All ConfigServer 533583860 Version A1 ABB Technical Support Server Broker Field Default Values QueryPingTimeOut 20000 (20 seconds) All Indicates how long (in milliseconds) to wait while contacting other server brokers on the network 0 All 1 indicates that the server broker will not update it's registry values with the ConfigServer, will use the current registry settings (ie. Hostlist, MirrorSiteList, QualityStrings, SystemProperties) 20000 (20 seconds) All Total number of seconds allowed to validate an RTDS pointer. If RTDS validation does not occur within the specified amount of time that server is assumed to be unavailable. SkipConfigServer ValidateListTimeOut 533583860 Version A1 1/18/2002 Location All All All 1-191 ABB Technical Support Trouble Shooting Guide Operate IT B.0 HostList Field ListList Default Values <computer name> All List of configured RTDS nodes on the OperateIT system. 1 = supports RTDS database - 0 = RTDS node does not support RTDS database Location All The Hostlist displays the OperateIT host known to OperateIT. This list should contain all OperateIT nodes. If a node is not listed the OperateIT ConfigServer does not know if it’s existence and will not replicate databases to the missing Node. If a node is suppose to be a Hostlist and is not listed in the right pane, check the configuration of the node in question and also check replication using the SQL Enterprise Manager. 1-192 533583860 Version A1 ABB Technical Support MirrorSiteList Field Default Values MirrorSiteList <computer name> All List of configured mirror nodes on the OperateIT system. value is not applicable Location All The MirrorSiteList displays the known Mirror site Nodes. Not all nodes need to be on this list to operate. A node that is not a Mirror Site can get a Mirror Database from another OperateIT machine in the system. If a node is suppose to be a Mirror Site and is not listed in the right pane, check the configuration of the node in question and also check replication using the SQL Enterprise Manager. 533583860 Version A1 1/18/2002 1-193 ABB Technical Support Trouble Shooting Guide Operate IT B.0 MirrorPluggableProtocalInfo Field Default Values MirrorNode . <computer name> All The current closest mirror node chosen by the server broker. ProtocolName http All Connection protocol used when connecting to the mirror site Location All All The MirrorPluggableProtocalInfo displays the MirrorNode being used by the host PC to obtain a mirror database. This does not necessarily have to be the host computer. Typically it will be the host PC if it is a mirror site. But if a PC cannot get in communication to the ConfigServer PC during startup or fails to find a Mirror Database on it’s own PC, the Node will attempt to establish a communication to another PC to obtain access to a mirror site. 1-194 533583860 Version A1 ABB Technical Support Symphony Configuration Key Default Value ConfigServerHost The name of the computer where the configuration server software is installed and running. ConfigServer The name of the configuration server database. This is the name of the configuration intranet web site. The default value is the computer name specified during installation. However, the value can be changed to an internet web site name such as www.somesite.com. If this key is removed, the ConfigServerHost key is used instead. Provider=SQLOLEDB; Integrated Security=SSPI; Data Source(local); Initial Catalog=<database>; <database> is the name of the configuration server projects database. By default this is the configuration server database specified during installation (typically ConfigServer). This key defines the default project the software uses if no project is specified. The value is an OLE-DB provider string that specifies the connection string the software uses to communicate with a configuration database. Provider=SQLOLEDB; Integrated Security=SSPI; Data Source=(local);Initial Catalog=<database>; <database> is the name of the configuration server projects database. By default, this is the configuration server database specified during installation (typically ConfigServer). This key allows a separate project database to be specified. This allows an ABB engineer to setup multiple configuration server projects in a single SQLS. The project database requires the SymProject and SymProjectUsers tables ConfigServerName ConfigServerWeb Default Project DSN Project DSN 533583860 Version A1 1/18/2002 Location All All All All All 1-195 ABB Technical Support Trouble Shooting Guide Operate IT B.0 DistributorHostName IsStandalone MirrorBackupCompleted RTDSBackupCompleted, MirrorDropSubscription to be present. The value is an OLE-DB provider string that specifies the connection string the software uses to communicate with the project database. <computer name> By default, the configuration server SQL Server machine is the replication distributor. In case of heavy loading, a SQL Server on another machine can be designated as the distributor. This key allows you to specify that other machine name. No Allows the configuration server software to be installed on NT workstation using SQL Server desktop edition. This edition of SQL Server does not support publishing of replication articles. When this key is set to 'Yes', replication is disabled. The configuration server database is used directly for RTDS and Mirror data. This key should only be set to 'Yes' in a standalone (single computer) scenario. It is most likely used for demonstrations on NT workstation running on a laptop. No Indicates the completion of the mirror database backup. Indicates the completion of the RTDS database backup. No All All Mirror, RTDS Mirror,RTDS Mirror,RTDS Forces a re-subscription to the ConfigServer database by dropping the current subscription. RTDSDropSubscription Mirror,RTDS No Forces a re-subscription to the ConfigServer database by dropping the current subscription. MirrorUpdateQueries Mirror,RTDS No When a replica database is created, the views and stored-procedures are obtained from the \\<ConfigServer>\Source$\RTDS or \\<configserver>\Source$\Mirror directories. Changing this key to 'Yes' will refresh the views and stored-procedures in the replica database the next time replication occurs. RTDSUpdateQueries NonRecoverableErrors ShowFailureCodes Mirror,RTDS No When a replica database is created, the views and stored-procedures are obtained from the \\<ConfigServer>\Source$\RTDS or \\<configserver>\Source$\Mirror directories. Changing this key to 'Yes' will refresh the views and stored-procedures in the replica database the next time replication occurs. These error codes are used to determine which replication errors will force a subscription to be rebuilt. 0 All All Enable or disable failure code output to the application log. To enable set this key to 'Yes' TruncateAlerts 1 1 = Delete all entries in the sysreplicationalerts table to prevent the MSDB database from growing in size. This table is used by SQL Server replication to log status and history information. UseExternalScript No All Uses an alternate replication scheme by running the "Synchronize Subscriber.vbs" script file. 1-196 533583860 Version A1 ABB Technical Support Note: When a user incorrectly enter a computer name or database name during software installation, the problem can sometimes be corrected by enter the correct information in the Symphony Configuration and Symphony Common folders. Symphony Historian The Symphony Historian folder contains a couple keys that effect the Historian. DELETEPERCENT 4 When the disk space for historized data falls below this %, emergency deletion takes place. MAXROWCOUNT 500000 The number of rows a table can have before a warning message is generated by the historian. MAXROWCOUNTEVNT 500000 The number of rows a table can have before a warning message is generated by the historian. WARNINGPERCENT 10 A warning message is generated by the Historian when diskspace falls below this %. 533583860 Version A1 1/18/2002 1-197 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Service Pack 4 Installation Identification The Installation of Service Pack 4 does not update the Build data displayed in the ABB Logo feature. The Build number can be seen in the registry and should be 3.02.133. Registry Location ElsagBailey > Installed Products > Maestro NT > Updates > OPW Patch 1-198 533583860 Version A1 ABB Technical Support Replication Access to SQL Enterprise Manager ConfigServer_Mirror The is a view of the Replication Monitor in SQL Enterprise manager for Mirror database. 533583860 Version A1 1/18/2002 1-199 ABB Technical Support Trouble Shooting Guide Operate IT B.0 The Information in the right pane shows the Nodes that have received a Mirror Database from the ConfigServer. A Snapshot is periodically created by the ConfigServer, and its Status should indicated Succeeded. If the Snapshot status does not indicate succeeded, the user can force a new snapshot by right clicking on the snapshot entry and clicking on START. Each node listed in the right pane should also have a status indicating Succeeded. To replicate the new snapshot, either cycle the DataSyncService service or restart the PC. ConfigServer_RTDS This is a view of the Replication Monitor in SQL Enterprise manager for RTDS database. The Information in the right pane shows the Nodes that have received an RTDS Database from the ConfigServer. A Snapshot is periodically created by the ConfigServer, and its Status should indicated Succeeded. If the Snapshot status does not indicate succeeded, the user can force a new snapshot by right clicking on the snapshot entry and clicking on START. Each node listed in the right pane should also have a status indicating Succeeded. To replicate the new snapshot, either cycle the DataSyncService service or restart the PC. 1-200 533583860 Version A1 ABB Technical Support Reports Function Block Server A Function Block Server is required in the system to provide the ability to run function blocks giving console related functionality like report generation and event triggers. The Application Function Block defines the condition (by referring to a Tag.Atom) by which a report will be generated. A Schedule Timer Function Block can be used to define a trigger condition on a cyclic basis. This can be used to schedule reports to print at a certain day and time. EBReport.asp processing 1. Establish file paths (must take remote mirrors into account) 533583860 Version A1 1/18/2002 1-201 ABB Technical Support Trouble Shooting Guide Operate IT B.0 2. Create ActiveX objects Excel.Application Scripting.FilesystemObject Wscipt.Shell 3. Read Install Path from the Registry 4. See if the Add-In (EBReportFunctions.xla) is registered. If it is not- register it. 5. Load Add-in. 6. Check the Report Timestamp. If it is not supplied- use the current system time. 7. Create Navigate.Mirror object (This is used to retrieve the Mirror database information) 8. Query the Mirror database for the Report Object information based on the Report Name. 9. Use the Excel Template in the Report Object information to create a new Workbook. 10. Use.asp arguments and Report Object information to enter data into named calls in Workbook. 11. If the Report Object information and Report Object has Event_Filter defined, perform the Event processing. See if “File” parameter was passed (Historic Event Page print request). Insert filter data into Named Cell in the workbook. Call EBAccessEvents (code is in the Add-in) this will retrieve the Event data and format report as defined in the template is being used. 12. If the Report Object has Tag.Atoms defined, perform the Tag.Atom processing. Insert Tag.Atoms associated with the Report Object into Named Cells. Insert Suffix processing associated with the Tag.Atoms into Named Cells. Call EBAccessTagAtoms (the code is in the Add-in) will retrieve Tag.Atom data and format report as defined in the template being used. 13. Check the Report destination A. If not OnScreen… 1-202 Verify the existence of the printer associated with the Report Object. Print Report If the report is to be stored in the Historian, perform the Historian processing. 533583860 Version A1 ABB Technical Support Create an empty workbook and transfer the report data and formatting into it (not the formula or all named cells). Save the new Workbook Call the EBHistorianSend (the code is in the Add-in) will submit the file from the previous step to the Historian. If the Report Template has a Routing Slip is defined, email the Report based on the Routing Slip. B. If OnScreen… Create an empty Workbook and transfer the report data and the formatting into it. (Not the formulas or all the named cells). 14. Do a ‘clean up’ (destroy the used components). 15. Insert error status in the Navigator window title bar. 533583860 Version A1 1/18/2002 1-203 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Security Microsoft Security Accounts OperateIT has two initial accounts that are required during software installation. The OperateServer and a Local User Group. Each account requires one of more user rights. OperateService account requires: Act as part of the operating system Generate security Audits Log on as a service User groups accounts require: Generate security Audits Microsoft Security Policies User Rights Configuration 1-204 533583860 Version A1 ABB Technical Support Security Diagnostic (See OperateIT Diagnostics) SQL Database Access During the Installation of OperateIT software the user should perform Access Right Configuration for the Different Database the a user may require access to on a machine by machine bases. This configuration is often overlooked by the installer. The noticeable side effect is the failure of a logged in user to be able to launch the Navigator window on the PC. To verify database access, use the SQL Enterprise Manager and select the Security folder. SQL Enterprise Manager Security Login 533583860 Version A1 1/18/2002 1-205 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Listed in the right pane in the Security Login are those users and user groups configured to have access to the database located on the local machine. Verify that User Groups configured by the installer are listed and properly configured. SQL Server Login Properties 1-206 533583860 Version A1 ABB Technical Support Verify the Database Access tab for each group is correctly configured. Make sure the following databases are check and the db_datareader box is check for the following databases. ConfigServer (only available on the ConfigServer Machine) ConfigServer_Mirror ConfigServer_RTDS Model Reference the Introduction and Installation manual for additional information. 533583860 Version A1 1/18/2002 1-207 ABB Technical Support Trouble Shooting Guide Operate IT B.0 SQL Per Seat Configuration SQL Server for the ConfigServer node must be configured Per Seat. Access the License configuration by: Start > Programs > Administration Tools (Common) > License Manager > Server Browser Locate the ConfigServer Node and double click on the entry. The License Mode must be set for Per Seat. 1-208 533583860 Version A1 ABB Technical Support Task Manager Task Manager 533583860 Version A1 1/18/2002 1-209 ABB Technical Support Trouble Shooting Guide Operate IT B.0 When experiencing problems with OperateIT, verify the task for the problem server is listed. Threads: Shows statistics for all active threads on the system Memory Detail: Shows memory-oriented information for each process, rather than the default of CPUoriented information Processes, Memory, Threads: Shows CPU, memory and thread information for each of the processes specified : EbServerBroker EbDataSyncService EbSystemServer EbFBServer EbI90Server EbEventconServer EbOPCServer EbEconDiag Process ID numbers can be obtained from the PID column of the Task Manager. 1-210 533583860 Version A1 ABB Technical Support Time Syncing Time sync in OperateIT is a combination of Ethernet Time synchronization and an RTDS Time Synchronization. The RTDS time synchronization is determined on which RTDSs are in the system and enabled to become time sync master. Ethernet Daemon The Time Sync Daemon master controls the Ethernet Time Sync. The Time Sync Daemon should be started automatically during the PC Startup. This service can be checked in the Control Panel. Control Panel Within the Control Panel window is listed the Time Synchronization Daemon service. Make sure the service is started. 533583860 Version A1 1/18/2002 1-211 ABB Technical Support Trouble Shooting Guide Operate IT B.0 The Time Sync Daemon can also be viewed in the Task Manager. Time Synchronization Daemon task The task name is tsyncdd.exe as seen in the following view. Harmony RTDS Time Master The Time Synchronization for a Harmony RTDS is controlled by the Harmony Server Object. Harmony Server Object The Time Synchronization field must be checked for the defined RTDS to be able to take Time Sync Mastership. 1-212 533583860 Version A1 ABB Technical Support If time synchronization is checked the RTDS can then take Time Sync Mastership if it is the Highest Accuracy Harmony node in the loop. Time Sync Accuracy for a Harmony RTDS can be set for one of the following accuracies. Time Sync Accuracy List If time syncing is checked the server will attempt to take over time mastership if the server does not receive a time sync message in the Message Wait Period with is 1200 seconds (20 minutes default). Both the Message Wait Period and the Master Update Periods can be modified is necessary. The Server Explorer has a diagnostic for the Harmony Server Object that provides time sync information. The view below displays the Time Sync atoms relating to the Harmony Server. Harmony Server Diagnostic View 533583860 Version A1 1/18/2002 1-213 ABB Technical Support Trouble Shooting Guide Operate IT B.0 The atoms include: 1-214 TS/ACC Time sync Accuracy TS/ENABLE Time sync enabled, indication that the RTDS can be time sync master. TS/MSTR If the RTDS the time master. TS/MSTR_ACC The Accuracy of the current Time Sync Master on the loop. TS/MSTR_LOOP The loop address of the current Time Sync Master on the loop. TS/MSTR_NODE The node address of the current Time Sync Node on the loop. TS/UPDATE The RTDS Master Update Period. TS/WAIT The RTDS Message Wait Period. 533583860 Version A1 ABB Technical Support OPC Time Sync The OPC RTDS object controls time synchronization for OPC RTDS. OPC RTDS Object TimeSyncEnbled: Enables or disables time syncing within OPC. TimeMaster: Enable Time Synchronization of OPC DA Server from the OPC RTDS. Enable = OPC RTDS sends the current time from the OPC RTDS clock (local node) by writing to the OPC ItemID specified in the OUT:DATE field when requested by the OPC DA Server. The OPC DA Server makes the requires through he OPC ItemID specified in the IN:BOOL field. Disable = OPC RTDS reads the time from the OPC DA Server. The OPC DA server writes the time and triggers a read using the IN:{Date|Trigger} field. SyncWhenInactive: Time Synchronization of OPC DA Server from Inactive OPC RTDS Enable or disables writes to the OPC ItemID in the OUT:DATE field when the OPC RTDS is in its inactive state. The TimeMaster field must also be enabled. 533583860 Version A1 1/18/2002 1-215 ABB Technical Support Trouble Shooting Guide Operate IT B.0 OUT:DATE: OPC ItemID of a DATE item that the current time (UTC) will be written to from the OPC RTDS when the OPC DA Server requests the time. The TimeMaster field must be enabled for this to occur. IN:BOOL: Boolean OPC ItemID that is read to trigger the OPC RTDS to write to the OPC ItemID specified in the OUT:DATE field. The TimeMaster field must be enabled for this to occur. IN:{DATE|Trigger} OPC ItemID of a DATE item that contains the current time when the OPC DA server knows its current time is correct. This time is used when the TimeMaster field is disabled. System Time Adjustment OperateIT has a Time Adjustment utility to allow a user to adjust the system time. The maximum adjustment that can be made is Plus or Minus 5 minutes. After a time adjustment is made to the system a broadcast message is made to the PC Network. Once the Time Sync Daemon receives the message it will begin adjusting the PC clock on all the OperateIT nodes. The Time Sync Daemon will adjust the time in increments based on the amount of change requested. <= 1 second: 4ms/sec. ( 1 second change will take 4 minute to adjust) > 1 second: 40 ms/sec (1 hour change will take at least 25 hours to adjust) Forward/Backward Jumps Time jumps forward and backwards can be performed without waiting for the time sync daemon to ramp. Large changes without the ramping process of the Time Sync Daemon can have a negative effect on the following servers. 1-216 Historian 533583860 Version A1 ABB Technical Support Batch system Event Concentrators The historian may stop collecting data, the Event Concentrators may discard new events if their generation times are older than events already in the Event list. The Batch system may skip a formula step(s) if the time is jumped forward. With this being said, the following procedure will allow the user to jump time forward or backwards. 1. Determine which PC host the master Time-Sync Daemon. 2. On that Node, set it’s Windows NT Adjust Date/Time Function to the desired time. In a couple minutes all the PC clocks will adjust to the new time. 533583860 Version A1 1/18/2002 1-217 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Set Operate IT Time The Set Operate IT Time allows the user to make small time adjustments up to 5 minutes. Start > Program > ABB Automation > Operate IT > Base Products > Set Operate IT Time 1-218 533583860 Version A1 ABB Technical Support System Time Adjust Active: When blank no time adjustment operation in progress. Time: Displays current system time. Target: Target time of the new adjustment operation. Deviation: Enter a time adjustment. Duration: Indicates the approximate length of time in minutes needed to complete the new time adjustment. Adjusts the deviation field plus/minus the number of seconds to change and then click on Apply or OK to start the adjustment. The duration field will display a time frame in which the change should take to synchronize. Time Sync Diagnostic See Operate IT Diagnostic Section for Time Sync Diagnostic. 533583860 Version A1 1/18/2002 1-219 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Trends No Trace The most common problem seen with trendviews is that the traces do not appear for Historical Trends. The problem is usually the result of the configuration for resolution. Resolution defines the number of points for a trace from 1 to 1280. Depending on the selected time span, one point represents an interval of: Interval = Time Span / Resolution Note: The configured resolution must be less than the time span value (in seconds) and should be an even multiple of the time span value. Value precision In the grid box at the bottom of the trend display is the value column. Along with the value is a precision indicator. Due to differences in collection rates, it is possible that no points were collected on the precise point in time indicated by the trend readout arrow. In this case, the value displayed is an interpolated value. The value actually shown may have been collected slightly before or after the point in the time indicated by the trend readout arrow. The value precision indicator is a one-character field that follows the value at the arrow. Indicators are listed in the following table. Historical Trend Value Precision Indicator Indicator x < > i 1-220 Description Indicates a precise, actual value is displayed Indicates that the time-stamp of the value displayed is actually prior to (older than) that indicated on the time at the arrow button; or the value cannot be interpolated. Indicates that the time-stamp of the value displayed is actually after (newer than) that indicated on the time at arrow button. Indicates an interpolate value calculate using the closest value before and after that indicated on the time at arrow button. 533583860 Version A1 ABB Technical Support Trend Update Times Update times are not configurable on OperateIT. Real Trends 1-2 seconds Historical Trends 10 seconds Can be modified in ASP file not recommended because of performance issue with Historian. B.1 is to historize real time data. 533583860 Version A1 1/18/2002 1-221 ABB Technical Support Trouble Shooting Guide Operate IT B.0 User Dump The Userdump application creates a dump of processes in Windows NT. The Userdump utility can be helpful in trying to investigate questionable processes. This utility should only be run under the direction of technical support and/or engineering. The Userdump file created by the utility will require the analysis of engineering. The Userdump utility can be loaded anywhere on a PC and is activated in a DOS prompt. Down load the userdump.zip file from the ABB Automation FTP site. Unzip the file into a folder on the machine on with the utility is to be used. The userdump.zip file is available for download from the ABB Automation Technical Services Web Site. Navigate to the Web Site at http://192.231.53.7 Click on "Automation Services FTP Site" to enter the FTP site. Using the Utility The user will be instructed Technical Support or Engineering to determine the Process ID for the task in question. This is determined by calling up the Task Manager and looking into the Processes tab. Locate the service or task to be monitored under the Image Name column. Next to the Image Name column is the PID column. This column will contain the Process ID required by the Userdump utility. Another method to determine the PID is to run the Userdump utility command followed by -P. This will list all the tasks and their associated PID number. Example C:\<path>\userdump.exe -P To create Userdump, at the DOS Prompt enter the path, filename, and PID and then press the enter key to execute. Example: C:\<path>\userdump.exe 132 where 132 is the PID for the task being monitored. The resulting dump file will be created and stored in the directory where the command was executed. The name of the file will match the task or utility name be monitored with a .dmp file extension. The FIND utility can also be used to help locate the file if necessary. This file should be sent to the person or organization requesting the information. 1-222 533583860 Version A1 ABB Technical Support Windows NT Troubleshooting Tools Dr Watson Log - An application error debugger, which is a program that detects application errors, diagnoses the error, and logs the diagnostic information. The log DRWTSN32.LOG file is created in the form of a text file. When an application error occurs, Dr Watson will start automatically. To start the debugger when no error occurs or to access the main dialog box, type DRWATSON.exe at the command prompt or in the RUN dialog box. Modifying the Dr Watson application: C:\WINNT\system32\drwtsn32.exe Log file path Crash dump location Name of a wave file for sound notification Number of instructions to disassemble Number of application errors to save in the event log Options: Dump symbol table Dump all thread contents Append to existing log file Visual notification Sound notification Create crash dump file Data from the Dr Watson Log Log File Description The file will always start with the following line: Application exception occurred: That line is always followed by exception (error) information. The exception number listed corresponds to the exception generated by the system. The next part of the log file contains system information about the user and the computer on which the application error occurred. This part of the log file contains the list of tasks that were running on the system at the time that the application error occurred. 533583860 Version A1 1/18/2002 1-223 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Blue Screen Dump A severe error (called STOP error, or fatal system error or blue screen) Configuring recovery Control Panel> System> Start-up/Shutdown Tab On Windows NT Workstations this is set-up by default. On Windows NT Server this is not configured. Write an event to the system log Send an administrative alert to users and computer specified. Restart the system automatically, instead of a manual reboot. Write a debug file to containing a dump of system memory to a specified file location. * Very important for troubleshooting A program called Savedump.exe writes the entire contents of memory to the pagefile. Pagefile must be at least as large as the amount of physical memory Pagefile must reside on the partition that contains the systemroot folder. The first four lines of the stop screen contain the most critical information about the problem. Blue Screen Crashes To log Blue Screen crashes (Blue Screen of Death in Windows NT) go the Control Panel and set up System Startup/Shutdown ‘Recovery’ options for saving crash data. Verify that the write debugging option is checked. Write debugging information to: %SystemRoot%/Memory.DMP 1-224 533583860 Version A1 ABB Technical Support Windows NT Diagnostics (winmsd.exe) a tool that shows the computer hardware and operating system stored in the Windows NT registry. Location: Start> Programs> Administrative Tools> Windows NT Diagnostics Tabs contain: Services- services and devices with state, or status, of running or stopped Resources- Displays system resources in use (IRQs, I/O ports, DMA channels, memory allocation, and device drivers) Environment- Environment variables Network- network configuration information Version- OS information with version numbers, such as the build and Service Pack information System- Displays BIOS, HAL, and CPU information Display- information about the video adapter, driver, and display settings. Drives- All available drives and their types. Also information such as the volume label, serial number, file system, and disk usage for each drive. 533583860 Version A1 1/18/2002 1-225 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Performance Monitor Enables you to look at a resource use for specific components and application processes through a dynamically updating display. These charts can then be saved as logs or reports. Location: Start> Programs> Administrative Tools> Performance Monitor. Some data which can be monitored: % Processor time- shows processor activity Interrupts/Sec counter- measures the rate of service requests from I/O devices. Processors queue length- the number of threads shown by the processor queue length is an indicator of system performance because each thread requires a certain number of processor cycles. 1-226 533583860 Version A1 ABB Technical Support Event Viewer Displays events and error messages generated by the Windows NT system, services, applications, or user actions. System log- contains events logged by the Windows NT system components and device drivers. Security log- can contain valid and invalid logon attempts, as well as events related to the resource use, such as opening, creating, and deleting files or other objects. A security audit must be started to log information. This is done through the Administrative Tools, User Manager. Under Policies and Audit can be started to monitor successes and failures. Application log- contains events logged by applications. A database program might record a file error in the application log. Application developers decide which events to monitor. The logs can and should be saved and sent in when a problem occurs. A log can be saved by clicking on Log and Save As. The file can be saved as an EVT file or a txt file. An EVT file can only be viewed with the Event Viewer application. 533583860 Version A1 1/18/2002 1-227 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Task Manager Windows NT utility that provides information about the current processes running in Windows NT. Monitors, stops, and starts active applications and processes. 3 Tabs in the Dialog Box: Applications- shows the status of the programs or tasks that are currently running on the computer Processes- shows the information about the processes that are currently running on your computer Performance- monitors the computer’s system performance. Changing the Priority of a running application. On the Processes tab, right-click on an instance of an executable. Point to Set Priority, then select the priority. A warning will appear. The only process suggestion for this is taskmgr.exe. If a machine is locking up in an inoperable state, raising the priority of the taskmanager may allow access after a lock-up. To modify the data presented on the Processes Tab With the Processes Tab selected, Click on View. Click Select Columns A dialog box appears allowing the selection of additional columns, which can appear on the Processes tab of the Task Manager. Page faults Non-paged faults Virtual Memory Size Handle Count Thread Count 1-228 533583860 Version A1 ABB Technical Support Network Monitor Part of Windows NT Server. Use this network traffic and protocol analysis to view the conversation between two hosts on the network. View individual packets to determine the root cause of a problem. Typical of a network sniffer- real-time statistics are displayed as activity is captured on the network. Installing: Log in as the Administrator Right-click Network Neighborhood, then click on Properties. Click on Services, then click Add. Select the Network Monitor Tools and Agent, then click OK. Enter the path to the Windows NT CD ROM installation directory. (You will need the Windows NT Server CD) When all the files are copied, click Close. When prompted, restart your PC. Capture window- menus and toolbars are available to control the use of the Network Monitor Graph Session statistics Total statistics Station statistics 533583860 Version A1 1/18/2002 1-229 ABB Technical Support Trouble Shooting Guide Operate IT B.0 Additional Resources TechNet- a compact disc that provides information for supporting Microsoft products. The TechNet disc includes: A complete set of online Microsoft Resource Kits The entire Microsoft Knowledge Base TechNet Supplemental (Drivers and Patches) compac disc Online Services- accessible through the following location: http://www.microsoft.com/support/ Resources available: Frequently Asked Questions Software library Knowledge Base Peer-to-Peer Newsgroups Microsoft Technical Support Tech Chat View support opinions by product NT Resource Kit 1-230 533583860 Version A1 ABB Technical Support Apendix A EbHresults EbHresult Error Codes are sometimes displayed in various applications and/or files. The following table lists, a series errors codes that pertain to OperateIT. HRESULT Descriptions Codes – Names - Description 0x0004EB01 EB_S_SERVERREFFAILURE An internal server ref object is not accessible; results for other servers may still be valid. (Most like cause is a single server has shut down, but internal object not yet cleaned up.) 0x0004EB02 EB_S_NOUPDATEREQUIRED An update operation was not performed because no change occurred, or a Get operation is returning an empty value set because no changes were detected. 0x0004EB03 EB_S_DOMAINLOCATED Cannot add leaf because leaf's domain is present in domain tree hierarchy as a non-leaf. 0x0004EB04 EB_S_UNSUPPORTEDIID The specified IID is not supported for this operation. 0x0004EB05 EB_S_ABORTED The blocking call on a server context was aborted by the client. 0x0004EB06 EB_S_SOMEFAILED One or more items supplied to a compound operation failed, but the operation did succeed for at least one item. 0x0004EB07 EB_S_UNSUPPORTED_LOCALE An Operation Status code and a Tag-Atom Item sentinel value. A subscription was successfully established on a Server, but the Server substituted a different locale than the one that was requested for that Tag-Atom. 0x0004EB08 EB_S_MISSING_TRANSITIONS A Stream sentinel value. A Stream Subscription on a Server has reached a size limit or had a problem keeping up with the data stream, resulting in one or more stream transitions to be lost. 0x0004EB09 EB_S_FAILOVER A Current Value or Stream Value Notification Status code. Some or all of the Servers providing data values for Tag-Atoms in a Subscription have successfully failed over to their backups. 0x0004EB0A EB_S_ALLFAILED All items supplied to a compound operation failed, but the operation was still a success. (e.g. CreateSubscription() failure when bAlwaysCreate=TRUE) 0x0004EB0B EB_S_NOTHINGTODO The operation was successful because no processing occurred. 0x0004EB0C EB_S_ACTIVE Returned by an RTDS on transition from the Inactive state to the Active state. 0x0004EB0D EB_S_INACTIVE Returned by an RTDS on transition from the Active state to the Inactive state. 0x0004EB0E EB_S_EVENTSHELDOFF Returned by an IEbEventStream object when the redundancy state of the server is uncertain i.e. both active, both 533583860 Version A1 1/18/2002 1-231 ABB Technical Support Trouble Shooting Guide Operate IT B.0 inactive or stanalone inactive 0x0004EB0F EB_S_INPROGRESS The operation has been started but is not yet complete. 0x0004EB10 EB_S_EVENTPOINT_ASSUMED_VALID The EventPoint cannot be validated with the OPC Event Server primarily because the Event server does not implement "IOPCEventServer::QuerySourceConditions" or "IOPCEventServer::QuerySubConditionNames". In this case the EventPoint is assumed to be valid and is registered as a FEP within the RTDS. 0x0004EB11 EB_S_SERVERDOWN A Current Value or Stream Value Notification Status code. Some or all of the Servers providing data values for Tag-Atoms in a Subscription have been shutdown (no backup available). Error values 0x8004EB01 EB_E_NOSTATEDATA State member pointer has not (yet) been allocated. 0x8004EB02 EB_E_ALREADYREGISTERED The interface for this object has already been registered (with the Server Broker, by calling RegisterStatusInterface). 0x8004EB03 EB_E_NOTREGISTERED The operation cannot proceed because the object has not registered itself (with the Server Broker, by calling RegisterStatusInterface). 0x8004EB04 EB_E_SERVERNOTREG The server broker has some data for the server, but the server has not registered its IUnknown pointer with the Server Broker. 0x8004EB05 EB_E_NOMATCHFOUND There were no servers in the domain tree which conformed to the requested filter. 0x8004EB06 EB_E_NOTNORMALRUNNINGSTATE The application is either starting up or shutting down (must likely the latter for the Server Broker ) and cannot return valid data. 0x8004EB07 EB_E_DUPLICATEITEM A supplied item was an illegal duplicate of a previously specified item. 0x8004EB08 EB_E_NO_SERVER An Operation Status code and a Tag-Atom Item sentinel value. A Subscription could not be established on any Server because no Server could be found that provides data for the Tag-Atom. 0x8004EB09 EB_E_UNSUPPORTED_DATA_TYPE An Operation Status code and a Tag-Atom Item sentinel value. A Server was found that can provide data for a TagAtom but a subscription could not be established on that Server because it could not provide data for that Tag-Atom in the requested data type. 0x8004EB0A EB_E_UNSUPPORTED_ATOM An Operation Status code and a Tag-Atom Item sentinel value. A Server was found that supports the given Tag but a subscription could not be established on that Server because the given Atom does not exist for that Tag. 0x8004EB0B EB_E_UNSUPPORTED_TAG The tag could not be found by a server (or by one or more servers) 0x8004EB0C EB_E_TAGATOM_NAME_FORMAT The name supplied could not be parsed to obtain identifiers for (server) tag and atom. 0x8004EB0D EB_E_SERVER_INACTIVE The server could not process the request because it is not currently active. This occurs when a redundant tag is 1-232 533583860 Version A1 ABB Technical Support found, but cannot be queried for atom information because the subscription request has stated that redundant tags be filtered out. 0x8004EB0E EB_E_CFG_REFERENCE_ATOM RTDS configuration error: The specified tag.atom is part of a reference atom that is configured incorrectly (eg, empty string so it doesn't refer to another atom) 0x8004EB0F EB_E_EVTPNTNOTFOUND IEbEventServer Error: The tag or event point specified in a event acknowledge counld not be found 0x8004EB10 EB_E_INVALIDTIME IEbEventServer Error: The active time specified as part of a acknowledge was longer valid 0x8004EB11 EB_E_EVENTSLOST IEbEventServer Error: Event in the RTDs have been lost 0x8004EB12 EB_E_ITEMSFAILED One or more items supplied to a compound operation failed. 0x8004EB13 EB_E_ITEMNOTFOUND The specified item's key could not be located. 0x8004EB14 EB_E_LOCALE The locale is invalid, unsupported, or the client-supplied locale for an item has changed since the last call for the same item. 0x8004EB15 EB_E_VERIFY General verification failure. 0x8004EB16 EB_E_PUT General put failure. 0x8004EB17 EB_E_ACCESSDENIED Client has been denied access to an attribute. 0x8004EB18 EB_E_ATTRIBUTENOTSECURED Client attempted to access an attribute that has not been secured. 0x8004EB19 EB_E_ACCESSRIGHTNOTFOUND Client tried to locate a non-existant access right 0x8004EB1A EB_E_SECURITYGROUPNOTFOUND Client tried to locate a non-existant security group 0x8004EB1B EB_E_OBJECTTYPENOTFOUND Client tried to locate a non-existant object type 0x8004EB1C EB_E_SECURITYINITFAILED Security component failed to initialize 0x8004EB1D EB_E_INVALIDPRECISION Precision is invalid 0x8004EB1E EB_E_LOCKED component is locked 0x8004EB1F EB_E_FILE_IO failed to open a file 0x8004EB20 EB_E_ADO_ERROR error related to msado.dll 0x8004EB21 EB_E_NEED_MORE_INFOR 533583860 Version A1 1/18/2002 1-233 ABB Technical Support Trouble Shooting Guide Operate IT B.0 need more information 0x8004EB22 EB_E_SUBSCRIPTION_TERMINATED Returned to the client in Item Values with a VT_ERROR Variant type to indicate that the previous data change sent to the Client was the last value for that Tag-Atom (due to Tag deletion or other change that causes the subscription for that Tag-Atom to be terminated by the RTDS). 0x8004EB23 EB_E_REPUT_AFTER_FAILOVER Returned to the client in an Operation Status object when a Put operation has failed due network or node failure and a backup RTDS is available but has not become Active yet. The Client can retry the Put after failover has successfully occured. 0x8004EB24 EB_E_INACTIVE Status of a tag atom item for which a Put operation failed when the server was inactive and the associated tag was flagged as redundant. The Client can retry the Put after failover has successfully occured. This result is also inserted into a Stream by the Client Components when a redundant Item has been subscribed on a Server that becomes Inactive. If the Server does not failover successfully or if the Server has no backup, then the Client will see this HRESULT as the value of a VT_ERROR Variant. 0x8004EB25 EB_E_ACKREJECT_INACTIVE IEbEventServer Error: A event point acknowledge was attempted on the inactive server for a event point that is redundant i.e. it's in a redundant tag 0x8004EB26 EB_E_DUPLICATESERVER The started server was an illegal duplicate of a running server. 0x8004EB27 EB_E_ATOM_NOT_PRESENT The atom exists, but is not defined for the tag (e.g., optional signal that is not present in current tag instance) 0x8004EB28 EB_E_OUT_OF_BOUNDS Specific verification failure code The value for a put is out of permissible bounds 0x8004EB29 EB_E_REDTAGGED Specific verification failure code The tag is red tagged, and hence can not be put to 0x8004EB2a EB_E_ATOM_STORE Failed to store the atom value 0x8004EB2b EB_E_DISESTABLISHED Failed to establish tag 0x8004EB2c EB_E_NAMESPACENOTSUPPORTED Namespace identified in prefix is not supported. 0x8004EB2D EB_E_OPCSERVERNOTRUNNING The Opc server configured for this RTDS is not currently running. 0x8004EB2E EB_E_SOURCE_DELETED This error code is used as the value of a Put subscription, for which the data source has been removed 0x8004EB2F EB_E_UNSUPPORTED_ARRAY_INDEX Used for array atoms. Returned when index out of range. 0x8004EB30 EB_E_ENUM_STATE_NOT_PUTTABLE Used for enumerated signals. This error will occur if the value being put is not in the control mask for that signal 0x8004EB31 EB_E_REPLICATION_FAILED Used by the DSS to indicate gerneral host.dll replication failure 0x8004EB32 EB_E_INVALID_ACK_TIME Reported on Ack of an EventPoint with a different active time. 1-234 533583860 Version A1 ABB Technical Support 0x8004EB33 EB_E_EVENTPOINT_NOT_VALID The Source, Condition and SubCondition for an EventPoint cannot be validated with the OPC Event Server. 0x8004EB34 EB_E_INVALIDARG An invalid argument was passed to the Harmony Namespace. Typically something wrong with the block address in the a Harmony Namespace subscription i.e. an invalid loop address 0x8004EB35 EB_E_DEFNOTLEAF The DEF pattern passed to the Harmony Namespace to not resolve to a leaf node. Defs must resolve to leaf nodes in the XML Block Schema 0x8004EB36 EB_E_INVALIDHARMONYITEM The block address is valid but the spec or output number is not support by the function code of the block in the Harmony Namespace 533583860 Version A1 1/18/2002 1-235