Troubleshooting
Techniques & Utilities
Staci Maund
Curtis Krol
Troubleshooting Methodologies
Hyland Troubleshooting Tools
9 INDISPENSABLE RULES OF
TROUBLESHOOTING
From Debugging by David J. Agans
ISBN # 0814474578
Rule #1
Understand the System
• Read the manual
• Know how the system is supposed to work
• Know your tools
Read the Manual
• Hyland Community
– Module Reference Guides
– Forum
• Training.OnBase
– Web Based Trainings
– Education Services Class Schedules
2013 ECM Training Conference
#dbwestECM
Know How it’s Supposed to Work
• Look up ALL the details on the issue
• System information
• Is it supposed to work?
• DO NOT ASSUME!
• Trust but verify
2013 ECM Training Conference
#dbwestECM
Know Your Tools
• Know what tools to use
• Know how to use them!
• Know the limitations too
• Should you use more than one tool?
Maybe…
2013 ECM Training Conference
#dbwestECM
Rule #2
Make It Fail
Reproduce the issue
Make It Fail
• Why reproduce The Issue?
– Observe the behavior
– Narrow down the cause
– Confirm the fix
2013 ECM Training Conference
#dbwestECM
Make It Fail
• What are good details of a problem?
• Detailed steps to reproduce the issue
• Obtain all facts when troubleshooting
• Does the problem happen after the same set of actions?
2013 ECM Training Conference
#dbwestECM
Make It Fail
• Intermittent Issues
– Find the signature of intermittent bugs
– Amplify testing
– Capture info on every attempt
– Never throw away a debugging tool
2013 ECM Training Conference
#dbwestECM
• Read the complete error message
• Get a screenshot
Rule #3
Quit Thinking & Look
Quit Thinking & Look
• See the actual failure
– Do not just assume and fix the wrong thing
• See the details - get complete info
• Guessing is okay, but only to focus your search
2013 ECM Training Conference
#dbwestECM
Narrow the search with
successive approximation
Rule #4
Divide & Conquer
Divide & Conquer
• Narrow the Search with Successive Approximation
• Use available info from:
– Step 1 – Understand the System
– Step 3 – Quit Thinking and Look
• Break it into smaller bites!
2013 ECM Training Conference
#dbwestECM
Divide & Conquer
• Test for an expected result
• Start at the problem and work backwards
• It could be more than one thing!
– Sometimes they are both the same issue
– Sometimes one fix causes another problem
• Is it noise or is it real?
• Widen the range if necessary
2013 ECM Training Conference
#dbwestECM
Evaluate the results of
each change
Rule #5
Change One Thing At a Time
Change One Thing At a Time
• Isolate the Key Factor
• Avoid Quick Fixes
• Change ONE piece at a time
– Test before the next change
• Determine what you changed since the last time it worked
2013 ECM Training Conference
#dbwestECM
Rule #6
Keep an Audit Trail
Log what you did, in what order,
and the outcome
Keep an Audit Trail
• Write down what you did (DETAILS)
– What order?
– What happened as a result?
• Understand that any details could be an important one
• Correlate Events
2013 ECM Training Conference
#dbwestECM
Rule #7
Check the Plug
“Assumptions often lead to trouble.”
– Doug Kupec
Check the Plug
• Do not ignore the obvious
• Question your assumptions
• Don’t start at square three
• Test the tool
2013 ECM Training Conference
#dbwestECM
• Ask for help
• Talk to the mannequin
Rule #8
Get a Fresh View
Get a Fresh View
• Ask for fresh insights
– Others are there to help
– Explaining the problem may help resolve it!
• Listen to the voice of experience
– Look at past experiences to resolve your problems
2013 ECM Training Conference
#dbwestECM
Get a Fresh View
• Report Symptoms – Not Theories!
– Reason to ask for help is that your theories are not getting you
anywhere
• Present ALL info even if it may not appear relevant
2013 ECM Training Conference
#dbwestECM
Problems never go away by
themselves
Rule #9
If You Don’t Fix It, It Ain’t Fixed!
If you Don’t Fix It, It Ain’t Fixed!
• Problems don’t occur for no reason and don’t fix themselves!
• Check that it’s really fixed
• Know the cause
• Verify that the fix is what fixed it!
2013 ECM Training Conference
#dbwestECM
OnBase Architecture
Database
Workstation
Disk Groups
Gateway
Caching
Server
HTTP
OnBase
Web Client
HTTP
(Remoting
or SOAP)
HTTP
OnBase
Web Client
OnBase
Web Server
Database
OnBase
Application
Server
HTTP
OnBase
Core Application
Disk Group
Application
Enabler
Workflow
Bar Code
Generator
Email
Integrations
Document
Composition
Unity
Client
CORE BASED APPLICATIONS
Report
Services
Disconnected
Scanning
Web Server
Application Server
OnBase Web Server Troubleshooting
• Web Diagnostics
– Test.gif
– aspnetcheck.aspx
– Diagnostics Web Page
• Diagnostics Console
Notes about the Core Apps
If a problem occurs in the Web Client,
check the Thick Client
If yes, probably configuration
Diagnostics Service
Diagnostics Console
What is the Diagnostics Console?
• A utility that gathers log information related to
OnBase Core based modules
• User interface that allows easy readability of the
logs
2013 ECM Training Conference
#dbwestECM
HYLAND DIAGNOSTICS CONSOLE
What is the Diagnostics Service?
• A service running on the Application Server and/or
Web Server
• Display messages to the Diagnostics Console or log
them to a file
• Allows for remote monitoring of servers
2013 ECM Training Conference
#dbwestECM
HYLAND DIAGNOSTICS SERVICE
Local vs. Remote Mode
•
Two ways to run the Diagnostics Console
– Local Mode
• Logs are gathered from the local workstation
• Will not work if Diagnostics Service is running
– Remote Mode
• Diagnostics Console connects to the Diagnostics Service
• Messages are written to the Diagnostics Service
• Must have rights to start the service
2013 ECM Training Conference
#dbwestECM
When to use it?
• Troubleshooting issues with Core based modules
– Web Server
– Application Server
• Any module which connects to the Application Server
• When running the OnBase Thick Client as a Windows Service
2013 ECM Training Conference
#dbwestECM
ENABLING DIAGNOSTICS SETTINGS
Diagnostics Web Logging Profile
Logging Profile option was added in 11.0.0.11
Logging Profile does not require an App Pool restart
Reset back to the Web.Config settings (defaults)
upon App Pool restart
ERRORS TAB
DATABASE TAB
LDAP / NT AUTHENTICATION TAB
FILE TAB
0
TIMER / DISTRIBUTION SERVICE
SYSTEM INFORMATION TAB
Enabling Thick Client Verbose
• Registry Settings
– 32-bit Workstations
• HKLM| SOFTWARE | Hyland | Diagnostics
– 64-bit Workstations
• HKLM| SOFTWARE | Wow6432Node | Hyland | Diagnostics
• Set Log Profile to decimal value of 64 (not HEX)
2013 ECM Training Conference
#dbwestECM
Best Practices
Log to file when logging a lot of data
Do not run the Diagnostics Console all the time
Best Practices
Only enable needed mailslots
– Utilize Web Diagnostics Logging Profile
Save all tabs when engaging support
Recommendations
Enable filtering
– If you don’t see what you are looking broaden
your filters
Enable the User Name column
Windows Event Viewer
WINDOWS EVENT VIEWER
What is logged here?
•
Messages from the OnBase Client closing unexpectedly
•
IIS (OnBase Web or Application Server)
– Application Pools not starting
– Platter Management Errors
– Active Directory Authentication
•
OnBase Integration issues related to:
– Office Business Application
– Outlook Integration
2013 ECM Training Conference
#dbwestECM
Tips on Windows Event Viewer
Errors typically correspond to messages in
Diagnostics Console
OnBase Thick Client
Verbose
What is Verbose?
•
Detailed log of OnBase activity
– Logs most SQL queries executed
– Used to see what is happening right before an error
– Can help pinpoint long running queries
2013 ECM Training Conference
#dbwestECM
Two ways to enable Verbose
•
Command Line Switch
– Available with OnBase Thick Client and Config
– Application launch issues
•
Tool Bar Option (OnBase Thick Client Only)
– Limits the amount of data captured
2013 ECM Training Conference
#dbwestECM
Command Line Verbose Options
• -V SQL,T
– Includes SQL with time stamps
• -V SQL,L
– Includes table lock codes
• -V SQL “itemdata”
– Includes only log statements that
– include the text of “itemdata”
2013 ECM Training Conference
#dbwestECM
Command Line Verbose Options
•
It is possible to use multiple command line
Verbose options together
•
Example:
-V SQL “update” “itemdata”,T,L
2013 ECM Training Conference
#dbwestECM
Enabling Verbose – Tool Bar option
Help | About OnBase Client | Verbose | chatty
2013 ECM Training Conference
#dbwestECM
What is the output?
• Verbose writes to the verbose.txt file
• Located in the TempParsePath (by default)
2013 ECM Training Conference
#dbwestECM
Tips on running Verbose
Capture the most relevant data
– Enable verbose immediately before action
– Truncate the log
– Disable afterward
Verbose files are always appended
– Most recent data at the bottom
Tips on running Verbose
Verbose files can become very large
– Limit the amount of data captured
– Don’t leave verbose switch on processes
RPT Files
What are RPT Files?
• Typically (but not always) generated when:
– OnBase unexpectedly terminates
– A debug assert error occurs
• Located in the same folder as the OnBase executables
– Obclnt32.rpt - RPT file for the OnBase Thick Client
– Obcfg32.rpt - RPT file for the OnBase Config
2013 ECM Training Conference
#dbwestECM
RPT Files
X
Always click Retry
Then click Abort
Don’t click Ignore
2013 ECM Training Conference
#dbwestECM
Tips on RPT Files
If the exception is recreateable, generate a
verbose along with the RPT file
Check to see if the date/times in the verbose
file match the RPT file
Tips on RPT Files
Open RPT files in an advanced text editor
– Notepad2, Notepad++, WordPad, etc.
RPT files are always appended
Tips on RPT Files
Where was the OnBase Thick Client opened?
– From a Network Share
– Locally on the workstation
Useful Information
Understanding OnBase Versioning
Let’s take a look…
2013 ECM Training Conference
#dbwestECM
Understanding OnBase Versioning
12.0.1.89
2013 ECM Training Conference
#dbwestECM
Understanding OnBase Versioning
12.
The first digit designates the version.
It corresponds to the year of release.
2013 ECM Training Conference
#dbwestECM
Understanding OnBase Versioning
0.
The second digit designates release or beta software.
Even numbers (0,2,4, etc.) are release software
Odd (1,3,5…) are beta software
These will only change if there is more than one full release in
a calendar year.
2013 ECM Training Conference
#dbwestECM
Understanding OnBase Versioning
89
The fourth digit designates individual builds within a version.
All builds of the Core start with 1 (1 – 499)
All builds of the Thick Client start with 500 (500-999)
The build will increase successively regardless of change in the
Service Pack (xx.x.#.xxx) value.
2013 ECM Training Conference
#dbwestECM
Understanding OnBase Versioning
1.
The third digit is for Service Pack versioning.
2013 ECM Training Conference
#dbwestECM
Check out the Hyland Community !!!
Product Information
Documentation (MRG’s)
Presentations & Films
Blogs
Forums
Technical Knowledge Base
Troubleshooting Checklist
and more…
HTTP://WWW.HYLAND.COM/COMMUNITY
HTTP://WWW.HYLAND.COM/COMMUNITY
Check out Training.OnBase.Com!!!
Class Schedules
Web Based Training
Premium Subscription
and more…
Engaging your
first line of Support
What information should I have?
•
Customer Name or HSI #
•
Level of priority for this issue
– Low, Medium, High or Critical
•
OnBase Environment of issue
– Production, Test or Disaster Recovery System
2013 ECM Training Conference
#dbwestECM
What questions will I be asked?
•
OnBase Environmental Information
– OnBase Version and Build
– Problematic OnBase Modules
– Database Platform and Version
2013 ECM Training Conference
#dbwestECM
What questions will I be asked?
•
Issue Specific Information
–
–
–
–
–
–
Can this issue be reproduced?
When did it start to occur?
Is it specific to a single user or all users?
Is this a new process or existing?
Have there been any changes?
Are there any error messages?
2013 ECM Training Conference
#dbwestECM
Useful Data
•
Screenshots
– Any errors or error messages
•
Log Files
– Verbose
– Diagnostics Console (Save all tabs!!)
2013 ECM Training Conference
#dbwestECM
Tips on gathering data
• Make sure the time stamps match
– Diagnostics Console
– Verbose file vs. RPT
– Third Party tools
2013 ECM Training Conference
#dbwestECM
Troubleshooting Techniques
&
Utilities
https://training.onbase.com/troubleshooting