RPC error troubleshooting
RPC is an acronym for Remote Procedure Call, and typically is used to
allow a program running on one computer to access services on another
networked computer. Microsoft added some of it's own extensions to the
standard RPC protocol and deeply intertwined RPC into the inter-process
communication of Windows operating system components. So if RPC
stops working, lots of other Windows features will fail as well.
An RPC error on a Cyberstation means that it has lost connection to the
database for some reason and you will need to restart Cyberstation in
order to recover.
There are a lot of reasons that RPC errors can happen. Below is a list of things to check when
dealing with RPC errors on CyberStation.
1. Multiple Cybers
Does the issue occur on multiple Cyberstations?
2. Database
Does the issue follow the database? i.e. load the database onto a test server or try a test blank
database. Were some new objects recently added to the database when this started happening?
Does reverting to a backup stop the errors?
3. Increase Virtual Memory
Right Click on My Computer> Properties> Advanced System Settings> Under Performance select
the Settings button> Advanced tab> Under Virtual Memory click the Change button
Recommended is 4096
4. Network disruption or slowness
a. Networking - Verify all aspects of your networked environment. This includes, but not limited
to, slow networks, possible problems with your network interface card (NIC), cabling issues,
switch/router issues, busy networks. As you can see from the above definition of what RPC is it
can be heavily reliant on the stability of the network and all it’s interfaces.
Continuum does not support DHCP
b. Multiple NIC cards – ensure there is only 1 NIC card enabled.
No all NIC cards are created equal. Connections issue can be caused by an integrated NIC card
of low quality. There can be settings within the NIC card itself that could affect consistent
communication. Check the manufacturers website for details.
Verify that there is no duplication of IP addresses on the network. Pull a cable and ping the
address, if you still get a response there are 2 devices with the same IP address.
c. WINS - Verify your WINS settings and make sure there are no issues with the WINS Server.
This is known by Microsoft to cause RPC errors.
d. DNS - Verify your DNS settings and make surer there are no issues with the DNS server. This
also is known by Microsoft to cause RPC errors.
e. Verify that the "TCP/IP NetBIOS Helper" service is running.
f. Check and see if “File and Print Sharing” is enabled. There are references in a lot of online
articles, on RPC, problems that turning on “File and Print sharing” resolves RPC errors.
g. Ping and SQL
Is the SQL database still available to Cyberstation? From a DOS prompt>
Ping IP (SQL Server) –t (continuous)
Ex. c:\> ping 192.168.1.5 –t
Note the number of responses as well as the time in milliseconds to respond.
A local SQL server would usually be under 100ms.
Check resources on the SQL Server (CPU, I/O Read/Write, memory usage, etc.)
h. Windows Server 2003 Network card setting
(This setting isn’t available in Windows Server 2008)
Shut Continuum down on the machine having the problem.
Start>Settings>Network Connections>Local Area Connections>Right Click and select
Properties> Click on File and Printer Sharing for Microsoft Networks>Click Properties button>
Select 'Maximize data throughput for network applications'.
(Default is Maximize data throughput for File Sharing)
Ok through all the screens and start Continuum and see if this makes a difference.
XP/Vista/Windows 7
Start> Settings> Network Connections> Local Area Connection> Properties>
Advanced tab> Windows Firewall - Click the Setting button> Click the
exceptions tab
Verify that File and Print Sharing is selected.
File and printer sharing are disabled by default with XP SP2.
There is no network throughput option in XP, Vista or Windows 7 like there is on Windows Server
2003.
5. Windows Event Viewer
Take a look at the Windows Event Viewer Application Log (Start> Run type ‘eventvwr’ or Launch
Windows Control Panel> Administrative Tools> Event Viewer
Examine the events and look for RPC error indications in the details. There may be a pattern of
which executable is causing the errors.
You may also see informational events that indicated momentary disconnects. A large number of
these can indicate an issue with the Cyberstation being able to connect to the database
consistently in a timely manner.
6. Monitoring tools
Run an acctrace.exe (C:\Program Files\Continuum) and a wireshark capture at the time of the
RPC error to see the Continuum and network processing going on at the Cyberstation.
Windows Process Monitor may also be helpful in seeing which process is having trouble.
http://technet.microsoft.com/en-us/sysinternals/bb896645
7. Check the system for SpyWare and/or Viruses.
8. Workgroup versus Domain connection
An excerpt from the Continuum System Admin manual
In a LAN Configuration, the Continuum CyberStation must be in a “Microsoft Domain “
type installation. This is because the Continuum CyberStation will use Named Pipes
Interprocess Communication System (IPC). This is accomplished with Windows remote
procedure call (RPC). In a Microsoft Domain model installation, once the session is
established between the Continuum Cyberstations Client(s) and the SQL server it is a
persistent connection once authentication is established.
In a Workgroup installation, the database connectivity session is re-authenticated and reestablished for each database transaction. This results in much slower performance and an
inconsistent connection.
9. If you are running a LAN system using Windows 2003 server SP2/R2 and SQL 2005 try
disabling the TCP Chimney Offload feature by following the below.
Disable the TCPchimney setting on the SQL server and application server
Windows 2003 SP2 comes inbuilt with a new network scalability pack which is supposed to
accelerate the Windows networking stack. But this setting may not be compatible with all the
network cards and it could pose connectivity issues as well. For more information you can refer to
http://support.microsoft.com/kb/912222
http://support.microsoft.com/kb/942861
To disable this feature please take the following steps on both the SQL Server and application
server.
Open up a command prompt on the machine.
Run the following command: "Netsh int ip set chimney DISABLED"
Change the values of the following registry keys to 0 a.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\
EnableRSS
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\
EnableTCPA
Reboot the machine after the above changes.
==========================================================
Disable SynAttackProtect on SQL server: Windows 2003 Sp1 and Sp2 introduces this new
feature which monitors processes getting high number of connections and if it detects that too
many connections are being made, the connections will be rejected.
a. Click Start, click Run, type regedit , and then click OK.
b. Locate and then click the following registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
c. On the Edit menu, point to New, and then click DWORD Value.
d. Type SynAttackProtect , and then press ENTER.
e. On the Edit menu, click Modify.
f. In the Value data box, type 00000000 . Click OK.
g. Quit Registry Editor.
10. Sound Card
In some cases it was found that the built in sound card was causing the problem. If your not using
WAV files for alarms then try disabling the integrated sound card in the BIOS. If WAV files are
being played then try disabling the integrated sound card and put in a PCI sound card.
11. There have also been problems with OEM versions of the Operating system of the client
machines. A lot of vendors, like Dell, add additional applications to the systems they ship to
assist them if they need to remote into the computer for troubleshooting purposes. It is strongly
recommended that these OEM versions of the Operating System is removed and install the
system from a genuine Microsoft installation CD.
12.) Go to Control Panel-> Administrative Tools -> services, click with right mouse button
Remote Procedure Call -> click properties. remote procedure call properties window is now open,
go to recovery and you´ll see: First failure, second failure, subsequent failures -> choose : restart
the service. In this setup the system will automaticaly restart RPC when something shuts it down.
13.) Check all your Firewall settings. If possible turn off the firewall for testing and see if the
problem goes away.
14.) Check to make sure there are no ports being blocked that the RPC Server uses.
See KB 908472 (http://support.microsoft.com/kb/908472) details the steps for specifying the RPC
port range.
Also, be aware of the consequences of reserving the RPC ports. This has the potential to break
other applications that assign ports in this range. KB 812873 discusses this, too.
(http://support.microsoft.com/kb/812873
15.) Make sure all Peripheral devices like Network cards, Video cards, Sound Cards, etc. have
the latest drivers for them installed.
16. http://support.microsoft.com/kb/913384
Control Panel > AdminTools >Services and selected the Properties for Remote Procedure Call
(RPC) In the Log On tab, they changed the Log on as: to the Local System Account. It had been
"This Account" NT Authority\Network Service.
17. Is an EventNotification configured to send a page on alarm? If a PC running Continuum did
not have a modem but on startup Continuum was trying to send an alarm with a page then
accdataservice was crashing causing an RPC.
18. Microsoft KB177446 article that describes in depth RPC troubleshooting procedures
http://support.microsoft.com/kb/177446