Solving Problems with Autodesk® Vault Using Standard Troubleshooting Techniques. Richard Rankin – Autodesk (Assistant/Co-presenter optional) [Arial 10] DM6321 Starting with a general architectural overview of the Autodesk Vault client\server model, this class outlines the skills and resources needed to support Vault from a software vendor's perspective. Standard troubleshooting techniques will also be covered with a strong bias towards Autodesk Vault using real-life examples. We will conclude with best practices on planning and avoiding the worst-case scenarios that Autodesk software support staff find themselves facing. Learning Objectives At the end of this class, you will be able to: Know the core skills that you need to Support Vault Understand basic Vault Architecture Understand some of the core Structured Troubleshooting Techniques Understand the worst that can happen and plan for it About the Speaker Richard Rankin studied Mechanical Engineering before concentrating on the computerized systems that Engineers use. After gaining Microsoft certification, he began working for Autodesk Support in 2000 specialising in the then new product called Inventor. This was followed by a short stint working for an Autodesk Value Added Reseller before returning to Autodesk to support Vault in 2006. Since then he has been lucky enough to have fulfilled a number of roles, Data Management Support Specialist, DM Support Team Lead in EMEA and now the Worldwide DM Support Tech Lead. Richard is a regular contributor to the “Cracking the Vault” DM Support Blog Know the core skills that you need to Support Vault Skills Required: An understanding of:Server\Client software architecture Customers Processes Solving Problems with Autodesk® Vault Using Standard Troubleshooting Techniques. Vault Software CAD Software Networks, hardware and architecture Hardware requirements:Ability to restore large datasets (disks needs to be in the TB order of magnitude). Higher amount of Physical RAM (12GB+) Run different environments (Stick only with “Supported”) Operating Systems Qualifying CAD Products SQL version, level, & edition, Language In ADSK Support we use the following extensively VMWare (and various snapshots of different states) MSDN membership Webex (Remote support software e.g. GoToAssist) Understand Basic Vault Architecture ADMS Server consists of the Web server, Database server and the Vault file store. Vault clients consist of the Vault Client (Thick client), and various Vault add-ins for applications (Inventor, AutoCAD, Microsoft Office, etc). ADMS Logs:- 2 Solving Problems with Autodesk® Vault Using Standard Troubleshooting Techniques. Pre Vista™ C:\Documents And Settings\All Users\Application Data\Autodesk\VaultServer\Filestore Microsoft Vista™ - C:\ProgramData\Autodesk\VaultServer\Filestore Vlog-YYYYMMDD.txt Client and Server side logs ADMSConsoleLog-YYYYMMDD.txt Server –side Only. MS SQL Log Files:C:\Program Files\Microsoft SQL Server\MSSQL$AUTODESKVAULT\LOG IIS Log files:%systemroot%\system32\Logfiles C:\inetpub Job Processor Log Files C:\Program Files\Autodesk\Vault Professional 2012\Explorer Other Information Sources Search Engine (Google\Bing) “Cracking the Vault” Support Blog Autodesk KB Autodesk Forums Advanced Configuration Guide Readme_Vault (via “Cracking the Vault”) Vault White Papers site Implementation Guide (via “Cracking the Vault”) Vault Wiki Understand some of the core Structured Troubleshooting Techniques Challenges specific to Vault Backup sizes involved Business critical Multiple users will be affected Time sensitive 3 Solving Problems with Autodesk® Vault Using Standard Troubleshooting Techniques. One small problem gets multiplied by # of files in filestore Variety of supported configurations No two situations are the same Very first thing to check….. Is it supported? Check Vault Readme\Requirements for that release and for known issues The principles of troubleshooting Vault, are exactly the same as any other software. Step 1 Define the problem Step 2 Reproduce Step 3 Drill Down\Analyse Step 4 Identify Cause Step 5 Solve\Workaround “He who asks a question is a fool for five minutes, he who does not ask a question is a fool forever”. -Chinese proverb There are lots of questions you can ask, but it might help to have some structured technique behind it. From Our POV. If we’re not told what has been tried, we will assume it has not been tried. Example, if you don’t tell us that Update 1 was applied, we will ask. General Questions to ask yourself and help define the problem: How can you most efficiently define the problem using the smallest number of words? (imagine each word you use costs you $5) What information do you need to solve the problem? What information do you have? What are the benefits of solving the problem? (is it worth solving? Aside from the technical aspects, what are the business implications? What is not known? (Is there a bigger problem you are not being told about?) What do you not yet understand? (what do you need to know to) What is NOT the problem? (document what has already tried) 4 Solving Problems with Autodesk® Vault Using Standard Troubleshooting Techniques. Can you provide precise instructions on how to reproduce this? (if needed the files\backup as well?) Would drawing a diagram of the problem help explain it to someone else? Would taking a screenshot help explain it to someone else? Would recording a movie with Jing help explain it to someone else? What cannot be changed? (OS, SQL, Vault, time constraints, deadlines). Supportability Questions. Have you checked Autodesk.com for any relevant Vault\CAD Client Hotfixes that address this issue? Vault Inventor AutoCAD Mechanical AutoCAD For others See Product Support Have you searched for the error message? (google\bing etc) Is the problem reproducible after restarting the machine? (Client and Server) Have they run a maintenance plan? See Advanced Configuration Guide (ask for the log) Did it suddenly start? Or has it always been like this? When did it start? What changed? Suspect; Windows Update, Configuration changes (client\server), Group Policy, Network Configuration (driver settings) Is it a supported Environment? (See Vault System Requirements) Check Operating System version and language Check SQL version and language. Check Vault Server version and language (See TS1093327) Check CAD Client version and language(See TS1093327) Have they uninstalled\reinstalled all components? Vault 2010 Client (See TS1104567) Vault 2010 Server (See TS1104187) Vault 2011 Server (See TS15038912) Vault 2012 Client (See TS16662442) 5 Solving Problems with Autodesk® Vault Using Standard Troubleshooting Techniques. Vault 2012 Server (See TS16662284) Is the client and server all up to date with maintenance patches? Have they installed all the latest Updates on the server, CAD Applications and Vault Client? Can you reproduce it, if you restore the backup to a different server? (minus the backup) Troubleshooting Questions Is it a Server Or Client issue? Does it affect one, multiple or all clients? Can you reproduce it, if you create a new Vault on the same server? Is it reproducible in the latest release? Does it happen if you are NOT logged into Vault? Can you reproduce when you log in as another Windows user? Are you logged in as a member of the Local Admin, Power users or Normal Users? Can you reproduce when you log into Vault as another Vault user? (using Windows Authentication) Can you reproduce when you log into Vault as another Vault user? (NOT using Windows Authentication) Did you try deleting the local Vault user Profile? Did you try logging into Vault as a user with a different level of permissions? Did it install from the media or network deployment? (suspect a corrupt deployment) Can the problem be reproduced when Anti-Virus and the Firewall is disabled? (On the Client) (On the Server) (On both Client and Server) Can you provide all the log files? (vlogs, ADMSConsole logs, JobProcessor logs, Event Viewer Logs) See the Vault Support Checklist on “Cracking The Vault” 6 Solving Problems with Autodesk® Vault Using Standard Troubleshooting Techniques. Troubleshooting Techniques #1 If it helps, draw a diagram (Visio, Mindmanager, Paint) This can help you understand the problem and help others if you have to collaborate. You can plan the solution and work on one chunk at a time without getting totally lost. Troubleshooting Techniques #2 Half Split:- Think of the problem as a square representing a thousand possible causes. What one question could you ask that would narrow that down to 500? Then repeat…. Troubleshooting Techniques #3 Fishbone Diagrams Troubleshooting Techniques #4 Mind Maps example Troubleshooting Techniques #5 Dealing with Intermittent Problems. Some techniques to use include: Apply current patches or update to a new release. Consider any events with the same frequency Be prepared to gather more information Take everything off and put it back on again (TS16662284 or TS16662442) Troubleshooting Techniques #6 Identifying when you are stuck… 7 Solving Problems with Autodesk® Vault Using Standard Troubleshooting Techniques. You have been working on it for a while and are not making progress or are procrastinating You are going in circles, doing the same thing over and over again You miss a date you set with the customer You don’t know what to do next or don’t understand something. Actions to take Review the evidence Write things down Use diagramming Recheck basics (question\verify any assumptions) Use lateral thinking Use collaboration/escalation Understand the worst that can happen and plan for it Drawing Office Downtime Any activity that involves making Engineering Data unavailable should be planned and documented. Examples Autoloading Data Migrating to new release. Asses the risks involved. (What are the knowns and unknown?) Have a failover\contingency plan. Run a simulation in as near the “Live” environment as possible. If you encounter a problem during Migration\Upgrading, generally there are three possible outcomes. (Ordered by time to implement) You fix it quickly. (Search KB, Google etc, Service Pack) Roll back to the previous state. A codefix is required. (realistically; this could take a while depending on the complexity) 8 Solving Problems with Autodesk® Vault Using Standard Troubleshooting Techniques. Role of Support. “Get you back to a “healthy\productive state” in the shortest time possible, minimising data\production loss.“ A dummy run should be part of the process by default Example Dummy Upgrade ADMS 2011 to 2012. Stage 1 (Single Site) Install a new “TEST” server with the current Vault Server 2011. (plus Updates) Make sure you use the same version of SQL. (inc language and regional settings, OS not so important) (32\64 bitness is) Restore the latest 2011 ADMS Backup to “TEST”. Examine ADMSConsole\vlogs log for errors. Test 2011 Clients can connect. Upgrade to SQL 2008 SP2 (R2) (Express, Standard, or Enterprise edition) Install Vault Server 2012 over 2011. Research any errors (Document Errors & Solutions) Check “Documented Workflows” in new release for “Showstoppers”. Document solutions\new workflows. If no solutions are found online, contact support, before proceeding to Stage 2. Involve Stakeholders in go\no go decision. (Fair Process) Example Dummy Upgrade ADMS 2010 to 2011. Stage 2 (Single Site) In ADMS 2012 Console, delete Vaults and detach Vault Master Uninstall Vault Server 2012 Uninstall SQL 2008 Reinstall SQL <previousversion> Repeat Stage 1, Step 1-4 You now have a 2011 Backup server that can be switched to in case of emergency. Upgrade the Live server. 9 Solving Problems with Autodesk® Vault Using Standard Troubleshooting Techniques. Downtime Planning (Summary) Minimise the risks Plan the downtime. (Know what you are facing, how long it will take) Document the Plan. (draw a diagram if it helps) Pilot your Plan on a dummy server. Document any problems encountered during the test upgrade. (take screenshot of errors, what you did to overcome them). Understand how the old workflows work in the new release. If you encounter anything you cannot resolve yourself, log it with Autodesk Support. BEFORE taking production data offline. 10