Secure Shell (SSH) in HP Systems Insight Manager How this paper is organized ................................................................................................................. 3 Introduction......................................................................................................................................... 3 Why SSH? ...................................................................................................................................... 4 Origins of SSH ................................................................................................................................ 4 Origins of OpenSSH ........................................................................................................................ 4 What is SSH ....................................................................................................................................... 5 Other SSH Implementations ............................................................................................................... 5 Reference........................................................................................................................................ 5 How does SSH work? .......................................................................................................................... 5 SSH Components ............................................................................................................................. 5 The SSH Connection Process ............................................................................................................. 6 Which SSH client does HP Systems Insight Manager use? ................................................................. 6 Which SSH server does HP Systems Insight Manager contact?........................................................... 6 SSH authentication mechanisms ......................................................................................................... 7 Server Authentication - verification of a managed system .................................................................. 7 User Authentication - verification of the username and password........................................................ 8 SSH Server on Windows – Differences ............................................................................................... 9 Cygwin mounts ............................................................................................................................ 9 Passwd and group for Windows Implementations .......................................................................... 10 Coexistence problems with other Cygwin installations..................................................................... 10 Documents and Settings directory on Windows installations ............................................................ 11 Installation Diagnostic File for SSH ............................................................................................... 11 Summary ...................................................................................................................................... 11 Supporting Documentation – SSH and HP Systems Insight Manager ........................................................ 12 SSH files ....................................................................................................................................... 12 SSH client configuration directory................................................................................................. 12 Directory location of various SSH files........................................................................................... 13 Known_hosts.............................................................................................................................. 13 Public/Private key pair................................................................................................................ 14 Authorized keys ......................................................................................................................... 15 Passwd and group files ............................................................................................................... 15 HP Systems Insight Manager Features requiring SSH .......................................................................... 17 The SSH process in HP Systems Insight Manager ............................................................................... 18 Installing SSH............................................................................................................................. 18 mxagentconfig ........................................................................................................................... 19 Use mxagentconfig to remove Systems from the known_hosts file .........................................20 Tool execution user (TDEF modification).............................................................................20 How does HP Systems Insight Manager use the known_hosts file? ................................................ 21 How to disallow new keys (for the highest level of security) ............................................................. 21 File locations of the SSH files ....................................................................................................... 22 HP Systems Insight Manager 4.2 SP2 Affects Need for SSH by ProLiant Tools – Windows CMS Only .. 22 How to change the Port used by SSH ............................................................................................... 23 HP Systems Insight Manager 4.2 and Plug-in Install Options................................................................ 24 Conclusion .................................................................................................................................... 25 Common Questions or tasks................................................................................................................ 25 Checklist to debug SSH on Windows ............................................................................................... 25 OpenSSH on a Windows CMS ....................................................................................................... 28 OpenSSH on a Managed System .................................................................................................... 29 Configuring SSH when the Administrator account is disabled or renamed ............................................ 30 Configuring the account to use..................................................................................................... 30 Modifying the HP Systems Insight Manager tools............................................................................ 31 Modifications Summary............................................................................................................... 31 Diagnostic Tool using mxagentconfig (Check SSH Setup) .................................................................... 32 Appendix A: Changes found in HP Systems Insight Manager 4.2 SP2 ..................................................... 33 Appendix B: Troubleshooting .............................................................................................................. 35 Problem: An MxAuthenticationException is generated when a tool is run, either from the GUI or the command line interface. .............................................................................................................. 35 Problem: mxagentconfig fails when trying to authorize a user on a Windows system that did not install OpenSSH. ................................................................................................................................. 36 Problem: When executing a task, the message Unknown OS is displayed......................................... 37 Problem: mxexec is not working with Windows runas command. ..................................................... 37 Problem: Windows 2003 does not allow the Local System account to have the privileges it needs to run the SSH service. .............................................................................................................................. 37 Problem: Standard Windows tools run on the CMS fail with authentication error. .............................. 38 Problem: mxagentconfig or command execution fails after reinstalling the openSSH server. ................ 38 Appendix C: Changing server properties.............................................................................................. 40 Appendix D: Tool examples ................................................................................................................ 41 MSA tools ..................................................................................................................................... 41 SSA tools ...................................................................................................................................... 41 Appendix E: Glossary ........................................................................................................................ 44 For more information.......................................................................................................................... 45 How this paper is organized This paper is organized into -four broad areas • An introduction to SSH and how it works • Supporting documentation about how HP Systems Insight Manager uses SSH • Common questions and tasks including a Checklist to debug SSH on Windows • Appendences covering Changes found in HP Systems Insight Manager 4.2 SP2, Troubleshooting, Changing server properties, Tool examples, and a Glossary The concentration in this paper is on the use of SSH in Windows environments because that is where there are the most challenges. Strip away the issues of installing HP Systems Insight Manager and which username is used in the security keys and which username is used in executing a tool and managing SSH becomes quite tame. Those who are new to the topic of SSH should start at the beginning. Scattered in this beginning section are insights about HP Systems Insight Manager which may assist you to better understand the latter sections that discuss how HP Systems Insight Manager uses this protocol. The supporting documentation section is important to those who need to support OpenSSH. Introduction This is an era of the security conscious IT administrator. User names and passwords are no longer passed in the clear over the intra or internet. We are providing more secure access into managed systems so that only authorized users have access to our most sensitive information, which is on the system itself. Telnet was one of the methods of the past for logging into a remote system and performing commands that would add, delete, and modify files. However, Telnet is not a secure method to use and does not protect network traffic. There is a small, unassuming, yet robust solution which is reasonably easy to use, inexpensive, and available for most of today’s operating systems. This solution is Secure Shell (SSH), which really is not a shell at all but a secure remote access protocol. The SSH protocol provides security on the network, authentication and data exchange with spook protections and encryption. HP Systems Insight Manager is a robust system management tool. Through use of the Distributed Task Facility (DTF) and SSH, HP Systems Insight Manager is able to securely log into remote systems on behalf of the HP Systems Insight Manager user of the Central Management Server (CMS) and make modifications, additions and run commands. The HP Systems Insight Manager terminology for these processes is called tasks. This white paper provides the following: • An overview of SSH and OpenSSH, which is an implementation of SSH • Demonstrates how SSH is used in HP Systems Insight Manager • Discusses some of the problems that can be encountered during its usage • Presents a recommended troubleshooting method when the SSH connection does not appear to be working Some of the programs installed into HP Systems Insight Manager, called plug-ins, also use the capabilities of HP Systems Insight Manager, and mention is made of plug-ins where appropriate. The custom commands and some command line tools (MSA) of HP Systems Insight Manager require that SSH is installed and configured on the CMS. Other tools (SSA) require SSH on the managed 3 system where they are to run. See Appendix D: Tool examples for examples of the MSA and SSA tools. Why SSH? SSH was chosen to be used with HP Systems Insight Manager for the following reasons: • To provide a way to securely execute commands and copy files to remote systems • To provide secure authentication mechanisms • To encrypts all data sent over the wire, unlike the traditional UNIX® r services • To provide a popular, non-proprietary protocol The main consideration for selecting SSH was to eliminate the necessity for a proprietary management agent. For example, HP Servicecontrol Manager’s mxagent to be installed on each managed system. Origins of SSH SSH and the SSH-1 protocol were developed in 1995 and in the same year it was documented as an Internet Engineering Task Force (IETF) Internet Draft standard. The protocol was popular and by 1997 the protocol was standardized and published as a new standardized protocol version, SSH 2.0, or SSH-2. This version was missing some of the SSH-1 features which continue to be more widely used even though SSH-2 is a more secure protocol. SSH was originally created to replace the UNIX r services remsh, rlogin, and rcp. These services provide remote shell, execution, and file copy. Unfortunately, all data passed between the communicating systems using r services is unencrypted clear text. Additionally, the authentication mechanisms are weak and vulnerable to attack. The rhost authentication is vulnerable to the man-inthe-middle attack, and passwords are passed over the network in clear text. SSH provides a mechanism to verify the identity of the remote system using key based host authentication, prevents password snooping by using over-the-wire encryption of all communications between the client and server, and provides stronger user authentication methods by way of public key authentication. HP Systems Insight Manager uses the version of SSH provided with the HP-UX and Linux media. On HP-UX 11.11 and 11.23 (verify), SSH must be installed or an init must be started. Various versions of Linux include SSH which is then installed automatically when the OS is installed. Testing of HP Systems Insight Manager has been with SSH-2 implementations. Origins of OpenSSH OpenSSH (http://www.openssh.com) is gaining prominence as an SSH implementation, developed under the auspices of the OpenBSD project (http://www.openbsd.org/) and freely available under the OpenBSD license. OpenSSH is based on the last free release of the original SSH, 1.2.12, and has developed rapidly as contributions have continued by many. It supports both SSH-1 and SSH-2 in a single set of programs, whereas SSH-1 and SSH-2 have separate executables. While OpenSSH was developed under OpenBSD, it has been ported successfully to Linux, Solaris, AIX and other operating systems. Active development has continued on OpenSSH and SSH-2 but has ceased for SSH-1. HP Systems Insight Manager uses and installs OpenSSH when it is installed on Windows systems. The SSH client used by HP Systems Insight Manager is compatible with any other implementation of SSH or OpenSSH installed on managed systems. Testing of HP Systems Insight Manager has been with SSH-1.5 and 2.0 implementations. OpenSSH uses Cygwin, only one Cygwin service can exist at a 4 time on a Windows platform, and the version provided with HP Systems Insight Manager has been modified to make it more secure than the open version. What is SSH SSH is not a product but rather a protocol for secure remote access based on public key encryption. It provides a specification of how to conduct secure communication over the network. Communications to and from the remote system uses the SSH protocol. This is conceptually similar to Secure Sockets Layer (SSL) which is another security protocol used in many internet communications such as Hypertext Transfer Protocol over Secure Socket Layer (HTTPS). The SSH suite includes: • ssh program which replaces rlogin and telnet • scp which replaces rcp • sftp which replaces ftp • sshd which is the server side of the package • Other basic utilities such as ssh-add, ssh-agent, ssh-keysign, ssh-keyscan, ssh-keygen and sftp-server Other SSH Implementations There are a number of other implementations of SSH, most notably commercial versions of SSH1 and SSH2 maintained and sold by F-Secure Corporation. The version of OpenSSH provided by HP Systems Insight Manager on Windows is not compatible with the F-Secure versions. Reference For power users we recommend the O'Reilly reference, SSH The Secure Shell by Daniel Barrett and Richard Silverman, 2001, as an excellent reference. How does SSH work? An SSH client, similar to the one used by HP Systems Insight Manager, initiates a connection to an SSH server on the managed system. The two compare keys, establish a connection, and the user is authenticated for the local login. If authentication is confirmed, then the client is logged into the managed system. This is similar to a telnet connection, however it is a very secure connection. SSH Components There are three components to the SSH protocol: • Authentication • Encryption • Integrity Each of these components is discussed further in the following sections. However, it is the Authentication component that is of most interest because this is where the installation and operational problems occur. There are two major application components of SSH that are of importance to us when discussing HP Systems Insight Manager: • The SSH client - The SSH client comes in many variations and may be specific to a particular program. HP Systems Insight Manager for instance, uses its own SSH client named J2SSH. Even though the SSH application protocol is installed on the platform where HP Systems Insight Manager is installed and has its own client, HP Systems Insight Manager uses its own SSH client when 5 connecting to an SSH server. More than one SSH client can be present on a platform at the same time. The SSH client is not a service. • The SSH Server - The SSH server is a specific implementation of the SSH application protocol installed on the target system. For instance, on an HP-UX system the SSH server is included with the operating system environment. On a Windows system, the version of OpenSSH provided by HP Systems Insight Manager can be installed. In our experience there can be only one SSH server installed on a system. HP Systems Insight Manager has not been tested with any of the F-Secure implementations of SSH. The SSH Connection Process The SSH protocol is utilized to establish a connection and then to log into the remote managed system. After a successful login, any packets which follow are encrypted using the SSH protocol. Each system where SSH or OpenSSH is installed natively includes an SSH client and an SSH server. Note: This white paper refers to SSH and OpenSSH as SSH. Let us take a common use of SSH such as connecting to another system using the command window. The command line to connect to another system: $ ssh –l smith host.example.com (-v turns on verbose mode) In this case, your SSH client is initiating a connection to the remote SSH server host called host.example.com using the login name of smith. You can use the –v option to turn on the verbose mode so the transaction can be followed. SSH client SSH server SSH client SSH server Which SSH client does HP Systems Insight Manager use? The SSH client is what initiates the SSH connection. On a platform where HP Systems Insight Manager with OpenSSH is installed, there are two clients available. • The OpenSSH client from the installation of OpenSSH • The SSH client library used by the application, which in this case is HP Systems Insight Manager HP Systems Insight Manager uses an SSH client named J2SSH. This client cannot be used by other programs as it is only for HP Systems Insight Manager to use when establishing a connection to a managed system. The J2SSH is called by the DTF process on HP Systems Insight Manager. Note: If from a command line on the platform where HP Systems Insight Manager is installed, you initiate an SSH log into the same or remote platform, all you are doing is demonstrating that the target SSH server is operational. This does not test that the HP Systems Insight Manager J2SSH client can login remotely. Which SSH server does HP Systems Insight Manager contact? Each platform where SSH or OpenSSH is installed has an SSH server. The HP Systems Insight Manager J2SSH client connects to the SSH server on the target system. At times, HP Systems Insight 6 Manager must log into the platform where HP Systems Insight Manager is installed. In this case, the SSH server on the CMS platform is contacted by the HP Systems Insight Manager J2SSH client. SSH authentication mechanisms Every SSH connection involves two authentications in the following order: 1. Server authentication - The SSH client verifies the identity of the SSH server. This ensures the SSH server is genuine and not an imposter. It also guards against an attacker redirecting the network connection to a different machine. This prevents a man-in-the-middle attack where an attacker positions itself between the client and server and is able to view and modify the communication. 2. User authentication - The SSH server verifies the identity of the username account requesting access. Server Authentication - verification of a managed system To establish a connection, the SSH client first contacts the remote system. Session keys are exchanged, and are used to encrypt all further communication between the client and server. The remote SSH server then sends its identity, known as the host key, to the SSH client for verification. The first time a connection is made between systems is the only time the connection is vulnerable to a man-in-the-middle attack. The identity of the remote system is unknown so there is nothing to compare. Generally, when invoking SSH from the command line, the SSH client indicates that the remote host is unknown, shows you the fingerprint of the host key, and asks if you want to accept it. If accepted, the host key of the remote system is stored in the /.ssh/known_hosts file for comparison in subsequent connections. Note: The known_hosts file in HP Systems Insight Manager can be pre-loaded thus bypassing the potential of a man-in-the-middle attack. With HP Systems Insight Manager 4.0 thru 4.1 SP1 the file can be created or modified on the fly, with HP Systems Insight Manager 4.2 the HP Systems Insight Manager service must be stopped then re-started after adding the hosts. The process used by the HP Systems Insight Manager SSH client is similar. However the progress of the connection is not visible to the user. Instead, if the HP Systems Insight Manager task (which includes establishing an SSH connection) fails, an error message appears in the task window. The following illustration displays the programmatic use of the HP Systems Insight Manager J2SSH client. CMS Client J2SSH SSH client requests login using HP Systems Insight Manager Server Authentication Port 22 Consults ~\Systems Insight Manager\config\sshtools\ known_hosts database. If matches then OK, if no host key - adds to database. If keys do not match, authentication fails. Sends twice encrypted session key Server authentication completed Managed System Server SSH Server identifies itself Sends its host key - Sends its server key Decrypts session key Sends confirming message First Point of Failure: Authentication fails – this occurs when the keys in the SSH client’s known_hosts file for the target system do not match the keys returned by the target system. The target system keys are found in the known_hosts file only if the SSH client had previously connected to the target. The SSH key miss-match occurs if the SSH application on the target system has been removed and re-installed, which creates a new set of keys. One way to resolve this issue is in the SSH client’s known_hosts file to edit and remove the keys for the target system. Use the following command for HP Systems Insight Manager 4.2 or higher: mxagentconfig –r <hostname or IP> 7 User Authentication - verification of the username and password Once the identity of the remote SSH server has been verified, the SSH client sends the username of the user who is requesting a login, along with any credentials (based on type of authentication) of the user to the target SSH server. The user is authenticated in one of three ways: • Using host-based authentication – using key files – automated method Note: This method is not supported by HP Systems Insight Manager 4.x. • Public key authentication – using key files – automated method • Password authentication – uses keyboard entry of the password Note: This method is not supported by HP Systems Insight Manager 4.x. While SSH refers to the second method as public key, it is the key for the specific user rather than the host or server. Each of the above are attempted by the SSH client in sequence until there is a successful user authentication or after the last one is tried with no response and results in a failure. In the case of host-based authentication, the SSH client sends its host key to the remote SSH server. The remote server then checks its list of trusted hosts and verifies if the SSH client is one of them. If the SSH client is one of them, the remote server trusts that the SSH client has already properly authenticated the user and allows the log into continue. If a password is sent, the remote SSH server simply uses the username and password information to try to authenticate the user. The only difference between the way SSH does this and the way r services does this is that, with SSH, the password is encrypted when it is transmitted over the network just like everything else sent over an SSH connection. However, with r services, the passwords are not encrypted which is one of the reasons for using SSH. Unlike password authentication, the public key authentication mechanism is unique to SSH, and it is the most secure way to login. This is the method implemented by HP Systems Insight Manager. A public key is harder to guess than a password, and the mechanism does not require the SSH server to trust that the SSH client has properly authenticated the user. In public key authentication, the SSH client sends the user’s public key along with the username. The SSH server then checks the list of authorized keys for the user, and if there is a match, it sends a message that is encrypted with the public key back to the client. The SSH client then decrypts the message, using its private key, and sends a return message to the server to prove it has the corresponding private key. Once the server receives this confirmation, the authentication is complete. The following illustration displays the details of User Authentication used with the HP Systems Insight Manager CMS. Managed System Server CMS Client J2SSH Sends request for CMS (administrator or root) log in User Authentication Alternate 1 Decrypts with dtfSshKey private key and returns encrypted message using public Key method CMS shows “Authentication failed” Alternate 2 Checks /etc/passwd for the CMS user, if not found – login denied access is returned If password entry found - looks in $Home/<usr>/.ssh/authorized_keys2. If yes, sends encrypted message with dtfSshKey.pub to verify if user has the private key. Verifies decryption and CMS is logged-in If password entry found, but it is not in $Home/<usr>/.ssh/authorized_keys2, then password is requested. Second Point of Failure: Denied access (target system denies access) – this occurs when the CMS user is not found in the target system’s password file. This is the step where the user name and 8 password provided by the SSH client is checked on the target system in the normal user authentication process. More about the passwd process later. In Alternate 1 above, after the username is verified by the target system, the SSH protocol interrogates the target system that the SSH client’s private keys are present by reading the <target>/.ssh/authorized_keys2 file. If the authorized_keys2 file contains a public key that corresponds to the SSH client’s private key, the user authentication process is complete. Third Point of Failure: Authentication failed – in the Alternate 2 above, after the username is verified by the target system, the SSH server attempts to load the username’s public key from the <username>/.ssh/authorized_keys2 file. If the username’s public key is not present, or if the client does not have the corresponding private key, the protocol asks the SSH client (in this case, HP Systems Insight Manager J2SSH) for the password (interactive method). HP Systems Insight Manager password authentication is not supported so the request is rejected and the login process is terminated and the task fails. In HP Systems Insight Manager, this is interpreted as Authentication failed. Once the username’s public key authentication has succeeded, the session can be used to copy files and execute commands. The HP Systems Insight Manager SSH client uses version 2 of the SSH protocol, which closes some weaknesses in the original protocol. HP uses RSA algorithms to generate public and private key pairs, names after the inventors Rivest, Shamir and Adleman. HP supplies the OpenSSH version of an SSH server for Windows systems, and use the SSH server built into other operating systems. Other SSH servers compliant with SSH-2 may work with HP Systems Insight Manager, but this has not been tested. As earlier stated, our testing shows that only one SSH server can co-exist on a managed system at a time. SSH Server on Windows – Differences While HP-UX and most Linux distributions usually ship with SSH or OpenSSH already installed, the same is not true of Windows-based operating systems. HP Systems Insight Manager provides a version of OpenSSH to be used with the HP Systems Insight Manager DTF on Windows systems. This is installed along with the rest of the HP Systems Insight Manager software when being installed on a Windows platform (thereafter called the CMS). For managed systems, it can be installed from the Management CD, downloaded from HP’s HP Systems Insight Manager website, or deployed from HP Systems Insight Manager to other Windows systems. Functionality has been added in HP Systems Insight Manager 4.2 SP2 for improved deployment to all Windows systems. Refer to Checklist to debug SSH on Windows for more information. SSH was originally implemented for UNIX-like operating system and is part of OpenBSD. OpenSSH is an outgrowth of that effort. To easily port it to be used on Windows systems, an emulation layer called Cygwin is used. Cygwin provides a UNIX emulation layer so that UNIX software can be easily ported to Windows. It also has some well-known security problems. For example, it creates worldreadable data structures to emulate UNIX processes. The potential exists for a non-administrator user on the managed system to interfere with any tasks run on that system. To make OpenSSH more secure, the version distributed with HP Systems Insight Manager contains a modified Cygwin compatibility layer that restricts access to these data structures to members of the Administrator’s group. HP Systems Insight Manager’s version of OpenSSH only allows Windows Administrators to log into the Windows system by way of SSH. Cygwin mounts To find certain OpenSSH files, you must first determine where they are stored. The UNIX files of concern are /etc/passwd, /etc/group, and /home/<username>. To see the complete listing for Linux and HP-UX and where they are located for Windows, refer to Directory location of various SSH files. Cygwin emulates a UNIX environment. To locate files such as /etc/passwd and /etc/group, and the user’s home directory (for example, /home/<username>), Cygwin sets up mount points. 9 In the registry, navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2 Under this registry key the following three mount points are defined: /, /home, and /usr/bin. The native key under each of these is set to the corresponding Windows directory. Therefore, to determine where /home maps to, look up the following and read the native key value: HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/home This mount point defaults to C:\Documents and Settings. Similarly, the root directory (/) defaults to C:\Program Files\OpenSSH. So, /etc/passwd is found in C:\Program Files\OpenSSH\etc\passwd and the /usr/bin directory defaults to C:\Program Files\OpenSSH\usr\bin. Passwd and group for Windows Implementations The passwd file contains an entry for each user who is allowed to use SSH. If a user who is not listed in the password file tries to login, the connection fails with an illegal user error. Note: The passwd file on Windows does not actually contain any passwords but it does contain the path of the user’s home directory. Installation of the HP Systems Insight Manager OpenSSH package sets up password entries for whoever is running the install, as well as Administrator. Administrator is set up because all of the preinstalled Windows command line tools run as Administrator. The /etc/group file is also created at install time, but this file should not need updating to add subsequent users. Also see additional information about the passwd and group files in the section Passwd and group for Windows Implementations. When troubleshooting a user, one thing to check is the capitalization of the home directory which is stored in the passwd file. OpenSSH is case-sensitive in this regard, so /home/HPsimUser is not the same as /home/hpsimuser. Check that the capitalization in the password file is the same as the directory to which it refers and that the directory is indeed the correct home directory for that user. The following is a sample passwd entry. The second-to-last field specifies the home directory: Note: This would occur on a single line with no new lines. Administrator:unused_by_nt/2000/xp:500:513:U-PCDLONG2\Administrator,S-15-21-3769691966-4004114397-3833753107500:/home/Administrator:/bin/switch Coexistence problems with other Cygwin installations Multiple Cygwin based programs can be installed but only one version of Cygwin is used. During the HP Systems Insight Manager installation, the OpenSSH installer checks in the registry for an existing Cygwin and if found the installation fails. There are certain registry settings that have to exist for Cygwin to function, namely the mount points defined above. The OpenSSH installer checks for the Cygwin registry keys and refuses to install if they exist. The installation also fails if the full Cygwin distribution or any other software that uses 10 Cygwin is installed. For example, Python distribution in WinCVS uses Cygwin. This is an unfortunate consequence of multiple Cygwin installations not being able to coexist. There are other products in the market that use Cygwin, and HP Systems Insight Manager’s OpenSSH distribution is not compatible with them. This includes other freely available OpenSSH distributions. If you are already using another version of OpenSSH and do not want to install the HP Systems Insight Manager version, that is fine. Keep in mind that the HP Systems Insight Manager version is the only version that restricts access to the Cygwin data structures. If you are having trouble getting the HP Systems Insight Manager OpenSSH package to install, search your system for the Cygwin registry keys, as well as the file cygwin1.dll. The location of the file might give you some idea of what software is installed that is conflicting with the OpenSSH installation. Documents and Settings directory on Windows installations When a user account is created on a Windows system, the home directory for the account is not created until the user logs into the system for the first time. If this account is also used for SSH access the lack of the account name under C:\Documents and Settings can cause problems. The user’s home directory must exist so that SSH has a place to put its files the first time an SSH client contacts the SSH server for that username. SSH creates a directory in the user’s home directory to place its known hosts and authorized keys files. For example, C:\Documents and Settings\user\.ssh\known_hosts. Therefore, if the username’s home directory has not been created, running mxagentconfig for that user fails because the authorized_keys2 file cannot be created. Therefore, be sure to login as the user on each managed system so the home directory is created and that the username is the one which is used to execute tasks requiring the use of HP Systems Insight Manager’s SSH client. This is discussed more in latter sections of this document. Installation Diagnostic File for SSH When installation issues for the SSH portion of the HP Systems Insight Manager installation occur, the first location you should look in is the initconfig.log file. This file is located C:\Program Files\HP\Systems Insight Manager\logs and may provide clues. Summary • HP Systems Insight Manager uses public key authentication, therefore requires its public key in the authorized_keys2 in the /<user>/.ssh directory on the managed system for each user that tasks are to run-as. • If the managed system’s public key in the HP Systems Insight Manager known_hosts file (on Windows C:\Program Files\HP\Systems Insight Manager\config\sshtools\ or on Linux or HP-UX in /etc/opt/mx/config/sshtools) is mismatched, the task fails. This might occur if the keys on the managed system have changed by uninstalling and re-installing SSH. • Using mxagentconfig is easiest way to populate the managed system’s /<username>/.ssh/authorized_keys2 file with the HP Systems Insight Manager public key. To resync, delete the authorized_keys2 file on the managed system, delete the managed system entries in the known_hosts on the CMS, re-run mxagentconfig and then run the HP Systems Insight Manager Identify task against that system. There are more details later in this document. • HP Systems Insight Manager uses the root or administrator account for all tasks except custom tools created by the user from the HP Systems Insight Manager menu. • Users running custom tools must have an entry in the passwd file on the managed system. • User accounts against which tasks will be run must exist before the mxagentconfig command is run for adding the authorized_keys2 file. 11 Supporting Documentation – SSH and HP Systems Insight Manager SSH files There are several important files involved in the mechanisms described previously. On the system where each SSH client resides, notice the known_hosts file, which contains the public and private key pair used for public key authentication. For example, the public key of the SSH servers. Where the SSH server application is located, there is an authorized public and private key pair for each user and the host key of the SSH server. SSH server SSH client • known_hosts • public/private keys • \home\.ssh • public key – each user • SSH server host key • authorized_keys2 Since HP Systems Insight Manager uses OpenSSH, the locations and filenames described below are specific to OpenSSH. SSH client configuration directory Each username who uses the standard OpenSSH client has a configuration directory that the client uses to store these files. On HP-UX and Linux, it is the hidden directory .ssh under the user’s home directory. For example, $HOME/root/.ssh. On Windows the directory is in the username Documents and Settings directory. For example, C:\Documents and Settings\<username>\.ssh. The .ssh directory is automatically created by SSH the first time a connection is made from an SSH client and results in the file known_hosts being created. When mxagentconfig in HP Systems Insight Manager is executed against a managed system to set up user authentication, the .ssh directory is required to be previously created so that the public key from the CMS can be placed in the \<username>\.ssh authorized keys file. The authorized_keys2 file name is a name that is chosen by OpenSSH - mxagentconfig simply populates it. 12 Directory location of various SSH files File type HP-UX Linux Windows OpenSSH install /etc/opt/ssh/ /etc/ssh/ C:\Program Files\OpenSSH\ OpenSSH keys /etc/opt/ssh/ /etc/ssh/ C:\Program Files\OpenSSH\ etc .ssh directory $HOME/<usr>/.ssh $HOME/<usr>/.ssh C:\Documents and Settings\<usr>\.ssh known_hosts ( for HP Systems Insight Manager) /etc/opt/mx/config/ssh tools/ /etc/opt/mx/config/sshtools/ C:\Program Files\HP\Systems Insight Manager\config\sshtools\ known_hosts (for OpenSSH) $HOME/<usr>/.ssh/ $HOME/<usr>/.ssh C:\Documents and Settings\<usr>\.ssh\ passwd and /etc/ /etc/ …\OpenSSH\etc\ group files CMS ssh keys .dtfSshKey and .~.pub authorized_keys2 (uses SID for Windows user) /etc/opt/mx/config/ssh tools/ etc/opt/mx/config/sshtools/ (hidden files) C:\Program Files\HP\Systems Insight Manager\ config\sshtools\ (hidden files) $HOME/<usr>/.ssh/ $HOME/<usr>/.ssh/ C:\Documents and Settings\<usr>\.ssh\ Known_hosts The list of known host keys can be found in the file known_hosts. It contains the public host keys of the SSH servers that the username has accepted. The known_hosts file is always associated with the use of a specific SSH client. The SSH client used by HP Systems Insight Manager has its own known_hosts file. Refer to the table above and notice the location of the known hosts for HP Systems Insight Manager and location of the known_hosts for OpenSSH clients. Next is an example of an HP Systems Insight Manager known_hosts file displaying two entries: 192.103.1.21 ssh-dss AAAAB3NzaC1kc3MAAACBAM8yDS/qQI8pqwavOcXat4ygJFSsX1SNqXvW7sKzYrYF0k1 wk9LyUyHdnTVI8MRRQYZpOsR+UFqhHz2/emADlNSvlL2mHtd5yUbz/QKWT6ikAX7lxe Pg1HtcDvFfLoPG8k0uENvQgb1Exfzbdf9+CpoyG0QFnrWns+xYzBW3FbpXAAAAFQCHO IKdwA0A1qNNHPKbbCCnzOg3+wAAAIAeGMN7NuaR72bHGe9pgBd9vBh3MX/Jdh8aptFR Tl0cj4U/0aMa5WU4z/dL9N/8/GmgGxHr1VAJjF4TaIyC0HsM7/t16TunDHr9OFddsWg RCP3UBA28xwLI/enCuORTwcyW0M+SMMOPcPgDd74OOGN+gK107sSstMNn9ooOAGnw5A AAAIBQwoqfiDV6Zmp+v0XO+TWr12Hta2u8ZeeWfoM1ZeQnSUyRuv0Cf1vcUFS6BeFlI X+b7+zqtZfFP3xQTgMHk7Uf3t1NJHBSr9kI4Te3Mdj2WLClcMnEMPPqoa5w5+5GGGBC +zPqT2t6ZZ8rqo3Hf8vJwUZvQfZrrWi5hGQa6/snnA== Ovpc129.rse.hp.com,192.87.137.243 ssh-dss AAAAB3NzaC1kc3MAAACBALuFgiIFPeNLJw7o4/wup7Qal8qZSRJWVe/oZb7BR9haLA9 oc5yhDv07a1xHgyAzkg3ghdoVk70QbMye44DTP4VHPzM1CQ4jSVRC8+l9sPvMPlCfAl am66c15GInVytqExcD5zTu1wSp44oJne0yFJ9XcGLcNGP+x4wa7D2C3Mr/AAAAFQDDV P1Kn8pJMvbq46/T86T1uMZ0QQAAAIEAk/qa4eyxlmWoPO2GxEPv9+LP1KNM2YzfZuJF AgV6XWTbaEHYh8uDsgpjddTDi4Yu49u5xIdS1+bFjb72WQKZj46EH4BTddUNTUYVHUp kGgwJDB8ie+jJCkqJg8wJexDJquK+EGAYYkitLpUoVUHKTFxXiX4DxfK7cv+IDZ7UAJ AAAACAcK7VcmEBCqcgGNJXhsj1laM1ujDfxXgCzXjMdotMkib8Ye1vp3hc2MuN6BVz7 OeJTsopFTEj2J86SoT9zIl9qPO/rm3FrCIm/8VuDVezcpVIS7TyrSQWbdQwVmeAJX/u TIJB48suUDrjlF/bsUfM1naU/kZFSwnMo09Pa+mJ/uI= 13 Whenever the SSH client connects to an SSH server for the first time using the command line SSH client, the SSH client informs you that it does not know the host, and asks if you want to continue. If you respond yes, the key from the remote host is added to the known_hosts file. $ ssh peanut The authenticity of host 'peanut (192.168.0.2)' can't be established. RSA key fingerprint is 31:d7:ce:aa:24:c3:42:fe:77:cd:48:80:f6:0e:34:b6. Are you sure you want to continue connecting (yes/no)? When you accept the request to continue connecting, an entry is added to C:\Documents and Settings\<username>\.ssh\known_hosts. If the host key of the SSH server ever changes, for example when the server is reinstalled, or if another system tries to impersonate that server, the given key does not match the known key and the SSH client does not allow the connection to continue: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is 31:d7:ce:aa:24:c3:42:fe:77:cd:48:80:f6:0e:34:b6. Please contact your system administrator. Add correct host key in /home/sshuser/.ssh/known_hosts to get rid of this message. Offending key in /home/sshuser/.ssh/known_hosts:1 RSA host key for peanut has changed and you have requested strict checking. Host key verification failed. Note: There can be more than one key for a system in the known_hosts file. It can be listed by IP address, short DNS name and fully qualified DNS name. Only the first one identified during the transaction is read. If incorrect, the connection is rejected. Public/Private key pair For public key authentication as used by HP Systems Insight Manager 4.x, a key pair is created and stored in the C:\Program Files\HP\Systems Insight Manager\config\sshtools directory. The private key never leaves the client. It is used during authentication to decode messages that the remote SSH server encodes with the matching public key. Below is an example of the key pairs that were generated when HP Systems Insight Manager was installed. Notice the location of these keys. Also notice the known_hosts file, which is used by HP Systems Insight Manager to record the keys from successful SSH connections with managed systems. 14 The public key is not used by the SSH client. It is stored in the \<username>\.ssh managed system configuration directory so it can be copied to remote systems. In fact, if this file is ever lost, it can be regenerated from the private key. Therefore, it mainly exists for convenience. The illustration below is an example of a list of keys that were generated on a Windows system when OpenSSH was installed. Key pairs are generally stored with names matching the type of key they are. The private key has no suffix, and the public key is the same name with .pub appended. For example, an OpenSSH DSA key pair is stored in the files id_dsa and id_dsa.pub. An RSA key pair is stored in id_rsa and id_rsa.pub, and so on. Authorized keys The last file in the \<username>\.ssh configuration directory that is discussed is the authorized keys file used by HP Systems Insight Manager, authorized_keys2. This is the list of keys that is checked by the SSH server when a remote login is being requested using public key authentication. If the key being presented by the remote SSH client is listed in the file, the SSH server uses it to encrypt a challenge by the remote SSH client and then allows it to login provided the response to the challenge is correct. If the public key is not present, the authentication fails. This file is generally maintained manually. You generate a key pair on the SSH client system (for the location of the CMS keys, view the table entry CMS ssh keys above), copy the public key to all of the systems you want to log into using password authentication, and then concatenate it to the end of your authorized_keys2 file on each of those systems. Alternatively, you could have your home directory NFS mounted on each of the systems and then you would only have to update one file. This can become tedious for a large number of systems, and it requires you to remotely log into each of the systems, copy the key over, and then issue some command to update the key file. Fortunately, HP Systems Insight Manager 4.x provides a tool, mxagentconfig, that helps simplify this process. This tool is also used by the Install OpenSSH tool (installssh.bat) that deploys OpenSSH onto a Windows system. mxagentconfig is discussed in the following section. Passwd and group files The passwd (password) file is located in the C:\Program Files\OpenSSH\etc directory. After the SSH session is established between the SSH client and the SSH server, the SSH client transmits the login username to the SSH server. For each username allowed to use SSH, there must be an entry in the password file (passwd). If a username is not listed in the password file but tries to login, the connection fails with an illegal user error. The entry for Administrator and rjones in the graphic which follows are actually on one line for each. 15 When the HP Systems Insight Manager 4.x OpenSSH package is installed on the Windows platform, a password entry for whatever username is running the install, as well as Administrator is created at C:\Program Files\OpenSSH\etc\passwd. The graphic above shows both passwd entries. The Administrator username is set up because all of the HP Systems Insight Manager pre-installed Windows command line tools run as Administrator. The \etc\group file is also created at install time, but this file should not need updating to add subsequent usernames. Additional usernames are authorized by creating an entry for them. The entry actually contains a SID and the password remains internal to Windows. The entry is created using the mkpasswd command (C:\Program Files\OpenSSH\bin\mkpasswd.exe). This command looks in the Users settings and extracts relevant information (SID) for the username; the result of mkpasswd is concantated to the end of the passwd file. For example, if you want to verify that <hpsimuser> is an allowed SSH user. First, use an editor to view the contents of the passwd file. If the username entry is not there, then use the mkpasswd command to pull the user and passwd entries for <hpsimuser> and add to the C:\Program Files\OpenSSH\etc\passwd file. To verify if <hpsimuser> is an allowed SSH user: 1. Open a DOS window and navigate to C:\Program Files\OpenSSH\etc. 2. Execute the command: mkpasswd –l –u hpsimuser >> passwd. Alternatively, if it is a domain user follow step 3. 3. Execute the command: mkpasswd –d –u hpsimuser domain >> passwd. Use mkpasswd to add authorized system users into the passwd file. For local users, use the -l switch. For domain users, use the -d switch. Some users have suggested that for domain and local, it is best to run the command twice. 16 Note: Remember to use >>, not >. If you use domain and local, make sure to edit the file to remove any duplicate entries. mkpasswd -l [-u <username>] >> ..\etc\passwd (local users) mkpasswd -d [-u <username>] >> ..\etc\passwd (domain users) Note: To add users from a domain that is not the primary domain of the system, add the domain name after the username. Note: Omitting the username switch adds ALL users from the machine or domain, including service accounts and the Guest account. One of these commands might return an error. The error is acceptable as error output prints out on the screen and is not redirected to the file. If the mkpasswd command cannot be found, navigate to the bin directory of the OpenSSH installation, which is generally C:\Program Files\OpenSSH\bin\mkpasswd.exe. Once the passwd entry has been created, the user should be able to login using SSH. When troubleshooting a user, another thing to check is the capitalization of the home directory. OpenSSH is case-sensitive in this regard, so /home/HPsimUser is not the same as /home/hpsimuser. Check that the capitalization in the password file is the same as the directory to which it refers. Also verify that the directory name really matches that of the user’s home directory as sometimes Windows uses different names for users. The following is another sample passwd entry: Note: The following example would appear on a single line with no new lines. Note: The second-to-last field specifies the home directory. Administrator:unused_by_nt/2000/xp:500:513:U-PCDLONG2\Administrator,S-15-21-3769691966-4004114397-3833753107500:/home/Administrator:/bin/switch Note: With HP Systems Insight Manager 4.1, 4.2 and 4.2.0.1, Deploy is used to install OpenSSH on any Windows platform (DeployÆDrivers, Firmware and AgentsÆInstall OpenSSH) and might add a domain user to the passwd file making a best guess as to the domain information. For example, if the user’s home directory is \Documents and Settings\<username.mydomain> the OpenSSH install adds /home/<username> to the passwd file which is incorrect. However, with the version of Deploy included with HP Systems Insight Manager 4.2.0.2 SP2, the install queries the registry and extracts the correct wording of the domain /home/<username.mydomain> into the passwd file. If you are using an earlier version of HP Systems Insight Manager to deploy OpenSSH, or if you have concerns about having the correct domain username in the passwd file, edit the file manually and make the corrections to the username path by appending the .domainname after the username. HP Systems Insight Manager Features requiring SSH All command line tools in HP Systems Insight Manager are executed by the distributed task facility (DTF) using SSH including those executing on the CMS platform itself. Also see HP Systems Insight Manager 4.2 SP2 Affects Need for SSH by ProLiant Tools – Windows CMS Only for changes in HP SIM 4.2 SP2. Remember, HP Systems Insight Manager considers the platform where it is installed as a managed system. SSH is used for execution on the CMS for platform independence and multiple native methods are not needed to support Linux, HP-UX, and Windows. Tasks can be run the same way across all platforms and they are always executed through SSH. Custom commands, or application launch tools, are executed on the CMS platform from HP Systems Insight Manager. When you select a custom command to be executed against a set of managed systems, the HP Systems Insight Manager custom command process logs into the platform using SSH and the current HP Systems Insight Manager login, then the process is executed on the CMS platform. 17 The list of systems is passed to the DTF through an environmental variable. The custom command then does what it was written to do against each target system. It is not necessary for the target systems to be running SSH to function properly. The custom command could operate though another protocol that, for example, network switches understand. Unlike most command line tools, only the CMS platform has to be running an SSH server to enable custom commands. This is true with some of the HP Systems Insight Manager plug-ins such as VPM, RDP 2.0, PMP and OSEM. Command line tool execution is a powerful capability inherited from HP Servicecontrol Manager. There are two styles: • Single-system aware (SSA) • Multi-system aware (MSA) MSA tools function similar to custom commands in that the tool is run on an execution system, which is usually the CMS platform, and the target systems are passed by using an environmental variable. The tool is responsible for communicating with the managed systems using whatever protocol it uses. Software Distributor for HP-UX is an example of an MSA tool. The execution system is the system running the Software Distributor service. SSH is required to be running on that system so that the CMS can contact it with information about the software to install and the managed systems on which to install it. Unlike custom commands and command line tools, SSA tools are run directly on the managed system. The DTF opens an SSH client connection with each of the target systems, executes the command over the SSH protocol, and stores any output, including valid command output as well as error messages, in the HP Systems Insight Manager database. This process occurs on each target system that you selected which requires each target system to be running an SSH server. Examples of both MSA and SSA command line tools that ship with HP Systems Insight Manager can be found in Appendix D: Tool examples. To summarize, the CMS must have an SSH server installed (also see HP Systems Insight Manager 4.2 SP2 Affects Need for SSH by ProLiant Tools – Windows CMS Only) and configured to run any custom commands and most MSA command line tools. In addition, each managed system that you want to select as a target for a SSA command line tool must be running a properly configured SSH server. Now that you are aware of the features that require SSH, the following discussion indicates how the protocol itself is used within HP Systems Insight Manager. The SSH process in HP Systems Insight Manager Now that we have gone over the HP Systems Insight Manager features that require SSH, as well as an overview of the SSH protocol, we can discuss how these pieces fit together. In the last section we discussed a lot about SSH clients and SSH servers. All SSH client actions are performed on behalf of the Distributed Task Facility (DTF) which uses the built-in SSH client. You never see the DTF as it is embedded within the code base for HP Systems Insight Manager. The DTF contains an SSH client (J2SSH in HP Systems Insight Manager 4.x) that uses the SSH version 2 protocol to perform all of its actions on managed systems. These actions include opening password authenticated sessions for installing the SSH public key (.dtfSshKey.pub) in each execution user’s authorized keys file, executing management commands on the managed systems, and collecting output from them. Installing SSH There are five methods for installing OpenSSH on Windows systems depending upon if the system is the CMS or a system managed from the CMS: • When HP Systems Insight Manager is installed on a Windows system using the Typical install option, OpenSSH is installed at the same time. In HP Systems Insight Manager 4.2.0.2 (SP2), the credentials supplied during this process are automatically entered and work on Windows XP, 2000 18 and 2003 systems. See Checklist to debug SSH on Windows, step 8 for more information. Prior versions of HP Systems Insight Manager on Windows 2003 may require additional manual steps as identified in the referenced link. • Using the Custom installation option, OpenSSH can be de-selected and OpenSSH is not be installed. Beginning with HP SIM 4.2 SP2, if OpenSSH install is selected, the user will be asked to supply user credentials. In addition, the OpenSSH install now displays a security warning, "NOTE: The local security policy will be modified to give this user the following rights: log on as a service, create a token object, and replace a process level token. See the README file for more details." Note: The OpenSSH install (with Custom install options) does not prompt for service credentials on Windows 2000 systems; only Windows XP and Windows 2003 display this dialog. • A downloaded or extracted Windows HP SIM bundle has an OpenSSH directory containing the self-extracting file OpenSSH.exe. This file can be copied to a target system and run to install the software. However, mxagentconfig will not run resulting in the authorized_keys2 file not being created. The mxagentconfig command needs to be run from the HP SIM CMS. The OpenSSH.exe file included with the HP SIM 4.2 SP2 contains all of the updates. • OpenSSH for a Windows system can be installed from HP Systems Insight Manager using the menu DeployÆ Drivers, Firmware, and AgentsÆ Install OpenSSH. This process copies the OpenSSH program to the system, installs it, runs mxagentconfig to create the authorized_keys2 file, and then completes the operation. • Another closely related process is when an Initial ProLiant Support Pack is selected to be deployed as one option includes installing OpenSSH. Note: HP Systems Insight Manager 4.2 SP2 fixes installation issues previously experienced with installations on Windows 2003 including the following: • The OpenSSH install new displays a security warning (above). • The OpenSSH Install now correctly looks to the local system or the domain controller for the SID of the given user account (installing user, service user and local administrator). • The OpenSSH Install now only adds entries to the passwd file if they do not already exist; the OpenSSH Install is able to distinguish between local and domain users by the same name during this check. If an entry is to be added, it will be added to the start of the passwd file. • Checks were made to ensure that the OpenSSH Install continued to function correctly when installed on Windows NT 4.0, Windows 2000, Windows XP and Windows 2003; mxagentconfig Now we can examine what happens when you set up a managed system using mxagentconfig. The mxagentconfig is used for two purposes: • To obtain and store the host key of the target system in the on the CMS’ known_hosts file • To place the public key (.dtfSshKey.pub) of the HP Systems Insight Manager CMS in the user’s authorized_keys2 file so that future connections can be made by HP Systems Insight Manager using public key authentication First, mxagentconfig opens an SSH connection to the specified managed system. This means that SSH has to be already installed on the managed system and the \<username>\ directory present for that username. The managed system replies with its host key, which is verified against the list of known hosts on the CMS. If the host key is unknown, it is added to the list. If a host key is already stored for that system, the key that was sent during this connection is compared to it. If the keys match, the connection is allowed to continue. If it does not match, the connection fails. This check prevents man-in-the-middle attacks, except for the first time when the host key of the managed system is unknown. 19 Once the SSH connection has been established, mxagentconfig authenticates the specified username using password authentication on the managed system. A secure ftp (sftp) channel is then opened. This is used to look for the \<username>\.ssh directory in the user’s home directory. If it does not exist, it is created. Then mxagentconfig checks for the existence of the authorized keys file, which is authorized_keys2. If it exists, mxagentconfig appends the public key of the DTF (.dtfSshKey.pub) to the user’s authorized keys file which is named authorized_keys2. If the file does not exist, the authorized keys file is created for the username with the public key of the DTF as its first entry. At this point, the user is configured for public key authentication on the managed system. Use mxagentconfig to remove Systems from the known_hosts file Beginning with HP Systems Insight Manager version 4.2, the contents of the HP Systems Insight Manager known_hosts file is placed in cache when the HP Systems Insight Manager service is started. It remains in memory until the service is stopped. If the SSH server key of a system ever changes, such as after a re-install of OpenSSH, then the known_hosts file may have an incorrect key which prevents communication with that system. In this case you should remove the old entries from the known_hosts file. In previous versions, the known_hosts file could be edited or even removed while the HP Systems Insight Manager service was running, then you could have HP Systems Insight Manager re-identify all of the systems. This would cause the SSH public keys from the systems with SSH to be re-registered and re-populate the known_hosts file. With HP Systems Insight Manager 4.2, there are two methods to remove systems from the HP Systems Insight Manager known_hosts file: • Run mxagentconfig –r <system name>. This command contacts the DNS server for all versions of the system name and recursively remove the entries from the in-memory copy of the known_hosts file. The memory copy is re-written to the flat file. Also see How does HP Systems Insight Manager use the known_hosts file? • Stop the HP Systems Insight Manager service, modify or delete the known_hosts file, and re-start the HP Systems Insight Manager service. Run system identification for the systems removed from the known_hosts file. Tool execution user (TDEF modification) You use mxagentconfig to set up public key authentication so the DTF can execute tasks for a particular username. But how do you decide which usernames to set up? Tools in HP Systems Insight Manager have the concept of an execution user, which is the user who runs a tool when it is executed. This value in the TDEF files can be changed. If this user (<executeas-user>Administrator</execute-as-user>) is not specified in the tool definition file (TDEF), it defaults to the username logged into HP Systems Insight Manager. Therefore, if you log into a Windows CMS as rjones, for example, any tool that you run that does not specify an execution user, such as custom tools, attempts to run using the rjones username. This is most often a concern when running cross-platform tools. If you log into a Windows CMS and run an RPM query against a Red Hat Linux server, the tool should run as root, not as Administrator. For this reason, the tool TDEF files delivered with HP Systems Insight Manager generally specify root for Linux and HP-UX tools, and Administrator for Windows tools. The general guideline is that mxagentconfig should be run for root on Linux and HP-UX managed systems, and Administrator on Windows managed systems. 20 The following are the tools that contain an <execute-as-user>Administrator</execute-as-user> tag line in the file: The concept of execution user is most important with tools that do not specify who in which to run. Since these tools run as whoever is logged in, mxagentconfig must be run to set up keys for each user who wants to run the tool. In other words, if a certain tool runs as the logged in user rjones, and wants to be able to execute the tool, mxagentconfig must be run for rjones on each managed system the tool is to be run on including the CMS platform. This is an important concept in troubleshooting. If you are getting an authentication exception trying to run a tool, be sure that the keys have been set up for administrator or root, as well as for the user having trouble executing a command. For more information on execution user, please refer to the online help or manpage for the mxtool file. After modifying a tool XML file (TDEF), you must re-register the tool with HP Systems Insight Manager. This is performed from the command window running the mxtool command. See Modifying the HP Systems Insight Manager tools for an example. How does HP Systems Insight Manager use the known_hosts file? During the discovery process, HP Systems Insight Manager makes an SSH logon query to systems which have already been discovered. As you learned previously, the remote SSH server responds with its public key. This key is added to the C:\Program Files\HP\Systems Insight Manager\config\sshtools\known_hosts file and further connection to the remote sever is not pursued. The discovery causes SSH connections to be made against each machine to determine what version, if any, of SSH the managed system is running. As the CMS encounters new SSH servers, it automatically adds them to the list of known hosts. Subsequent connections are verified using the stored host key so that it can be checked during future connections. When an SSH server is reinstalled on a managed system, the host key changes and SSH connections from the CMS fails. This occurs because the remote system’s old public key is in the HP Systems Insight Manager known_hosts file. This can be resolved by using mxagentconfig –r –n <systemname> to delete the key from the known_hosts file. Note: Beginning with HP Systems Insight Manager 4.2, when the HP Systems Insight Manager service is started, the content of the known_hosts file is placed into active memory. Since the CMS keeps a copy of known_hosts in memory, simply editing and removing known_hosts file while CMS is running does not have impact and the changes are ignored. Alternatively, you could stop the HP Systems Insight Manager service, remove the entry form the known_hosts then re-start the service. How to disallow new keys (for the highest level of security) In some situations, the system administrator might decide that allowing the CMS to automatically add keys to the known_hosts file is unacceptable. In this case, add the following line to the mx.properties file: MX_SSH_ADD_UNKNOWN_HOSTS=false 21 Note: On Windows systems, the file is located in the C:\Program Files\HP\Systems Insight Manager\config\ directory. For more information on changing CMS properties, refer to Appendix C: Changing server properties. With this option set, the CMS no longer adds keys to the known_hosts file and it refuses to connect to an unknown system. There are two ways to use this capability: • You can run an initial discovery to create the known_hosts file and then set the option • You can set the option before initial discovery and create the known_hosts file manually. The easiest way to create a known_hosts file manually is to log into each system using SSH, from the command line. Create a CMS known_hosts file manually from the command line: 1. Stop the CMS. 2. Delete the existing CMS known_hosts file: C:\Program Files\HP\Systems Insight 3. 4. 5. 6. Manager\config\sshtools\known_hosts Delete the Administrator’s known_hosts file: C:\Documents and Settings\Administrator\.ssh known_hosts Log into each system, including the CMS system itself, from a command window using the command ssh –l <username> <host.example.com>. This adds the remote system’s keys to the user’s known_hosts file. Be sure that you make a connection using each system’s long name. For example, a long name may be name.domain.com, or a short name may be name, and the IP address may be 15.1.48.11. Using the three methods ensures the known_hosts file contains all three variations as HP SIM discovery may use either of the addresses. Copy the user’s known_hosts file into the CMS known_hosts file location. Start the CMS service This process can, unfortunately, have the same vulnerability as allowing the keys to be added automatically as there is a remote possibility of the man-in-the-middle attack. The only absolutely secure way to create entries for the known_hosts file is to physically go to each system and copy the key from there. To do this, repeat the process above, but only log into the local system through SSH. Collect the individual known_hosts entry from each machine this way, and then concatenate them together. Refer to SSH client configuration directory for more information. File locations of the SSH files The files used in the SSH process can be found in a previous section Directory location of various SSH files. HP Systems Insight Manager 4.2 SP2 Affects Need for SSH by ProLiant Tools – Windows CMS Only New code functionality has been added to HP Systems Insight Manager 4.2 SP2 for certain categories of tasks to bypass execution through SSH. This is enabled by HP Systems Insight Manager 4.2 SP2 for a new installation and optional if the separate SP2 is applied to an existing HP Systems Insight Manager 4.2 or HP Systems Insight Manager 4.2 SP1. This option allows the CMS to run local tools without using SSH. This new mechanism is used for all tools that use SSH locally on the CMS to target the CMS and that execute using the Administrator account. This functionality improves the operation of the CMS. The usage of the bypass feature is automatic and applies to plug-in operations including the following tools: • HP ProLiant Essentials Vulnerability and Patch Management Pack (VPM) • HP ProLiant Essentials Virtual Machine Management Pack (VMM) • Install OpenSSH • Initial ProLiant Support Pack Install • Configure or Repair Agents 22 • Custom Commands, if the login user is added to a special property. When enabled on Windows using HP Systems Insight Manager 4.2 SP2 or the SP2 patch, only tools of type msa-command-tool and app-launch-tool (review the XML files in the tools directory to find these entries) run by usernames listed in the mx_dtf_ssh_bypass_user option below bypasses SSH and locally run tasks without using SSH. The DTF determines if tasks are run directly through SSH or using native OS methods. MSA tools, are run as the <execute-as user> as specified in the TDEF file. If the <execute-as user> is not defined, then the task is run as the current login user. For a user created custom command, the current login user is used. If two users are using the same tool, one might see the execution go though SSH and another might not. Tasks run with the bypass feature actually run on the CMS as the same account used by HP SIM service and this account has full administrative capabilities on the CMS. Therefore only administrative users should be added to the bypass property. Two system properties are added to the C:\Program Files\HP\Systems Insight Manager\config\globalsettings.props file to enable the by-pass feature. The following entries are made into the file: #turn on the bypass feature mx_dtf_enable_ssh_bypass=true #Multiple ssh bypass usernames can be added if separated by "," #The user must have administrators privilege to avoid security risk mx_dtf_ssh_bypass_user=Administrator Account names should be separated by a comma, with no spaces. Domain accounts require two backslashes between the domain name and the user name, such as domain\\user. This feature can be disabled entirely by setting mx_dtf_enable_ssh_bypass=false in the same properties file. After making these entries in the globalsettings.props file, implementation occurs only if the HP Systems Insight Manager service is restarted. How to change the Port used by SSH Normally, SSH servers listen on TCP port 22. If, for some reason, this needs to be changed, the SSH port that HP Systems Insight Manager uses is configurable. To configure the port that the SSH client for the CMS uses to initiate SSH sessions: 1. The SSH port used by HP Systems Insight Manager is set by changing MX_SSH_PORT in the C:\Program Files\HP\Systems Insight Manager\config\mx.properties file. 2. For example, to change the port to 6450, add the following line to mx.properties: MX_SSH_PORT=6450. 3. Every SSH server on each managed system and the CMS must be configured to listen on that port as well. Changing the port on the SSH servers is accomplished easily by the following method. a. Go to the managed system. b. Open the C:\Program Files\OpenSSH\etc directory. c. Edit the file sshd_config using Notepad or similar editor. 23 d. Change the port number entry to the one used on the CMS and save the file. For more information on changing CMS properties, refer to Appendix C: Changing server properties. HP Systems Insight Manager 4.2 and Plug-in Install Options (Where software is run, updated 26 Jan 2005 – valid to HP Systems Insight Manager v 4.2.0.1.) The chart below can be used to determine the coexistence of HP Systems Insight Manager and various ProLiant Essentials plug-ins including if the plug-in needs SSH. Be sure to read the footnotes. HP Systems Insight Manager VPM VMM RDP 2.0 PMP 3.1 OSEM Install on/as Windows server Yes Yes Yes Yes Yes Yes Linux server Yes No No No No No HP-UX server Yes No No No No No Separate server linked to HP Systems Insight Manager N/A Yes (VPM plug-in) No Yes No Yes HP Systems Insight Manager and Product on same server? N/A Yes Yes Yes Requires a specific install order 1 Yes Yes If OS is a Windows virtual machine Yes Yes No Yes Yes Currently not qualified 4 If OS is a Linux virtual machine No N/A No No No No Needs SSH Yes No Yes, 5, No No No 6 3,6 (RDP plug-in) 2,6 RDP Install order on same server: If MSDE is used, install RDP and its MSDE first, install HP Systems Insight Manager, and then install the RDP plug-in (connector). If MSSQL is used, then install both RDP and HP Systems Insight Manager in any order, and then install the RDP plug-in (connector) last. See RDP guides for more information. 1 24 The RDP plug-in (connector) is installed on the HP Systems Insight Manager server which uses SSH. SSH is not used between HP Systems Insight Manager and the RDP server. 2 The VPM plug-in (connector) is installed on the HP Systems Insight Manager server which uses SSH. SSH is not used between HP Systems Insight Manager and the VPM server. VPM targets may or may not use SSH between the VPM server and the target. See the VPM User Guide for more information. 3 OSEM 1.3.4 has been installed into a Windows based Virtual Machine (VM), but has not been formally qualified. 4 5 SSH required on VM host system to deploy VMM agent. 6 SSH not required when using HP SIM 4.2 SP2 bypass feature. Conclusion HP Systems Insight Manager uses the SSH-2 protocol to execute tasks on managed systems. This requires an SSH server to be running and accepting requests on each managed system on which tasks are to be executed. Features of HP Systems Insight Manager that require SSH being installed and configured include custom commands and command line tools including MSA and SSA. HP Systems Insight Manager provides an OpenSSH package to be installed on Windows-based managed systems, as well as a key management tool (mxagentconfig) for setting up a user with the public key of the DTF. The information contained here gives you an idea of the topology of remote task execution in HP Systems Insight Manager—and also gives you an idea of where to start troubleshooting when there is a problem. Common Questions or tasks See Configuring SSH when the Administrator account is disabled or renamed. Checklist to debug SSH on Windows The standard installation of HP Systems Insight Manager on a Windows platform assumes a local account called Administrator. This is a short checklist to follow when debugging SSH issues. It starts with some basic questions then gets more specific. 1. Start by recording some basic information about the HP Systems Insight Manager CMS installation. a. What is the OS? a. Windows 2000, Windows XP b. Windows 2003 b. Has the local administrator account renamed or disabled? HP Systems Insight Manager 4.2 SP2 detects a renamed administrator account and correctly configures the passwd and authorized_keys2 files for the renamed user. Earlier versions require manual modification. c. Is there a home directory for the local administrator? Situations exist where the local administrator has never logged into the system. In these situations, the home directory is not created in Documents and Settings, as it is only created when the first login occurs. If this is the account to be used for SSH login, the installation of OpenSSH by the HP Systems Insight Manager install process fails. 25 2. If Windows 2003 a. Check what login account SSH is using, but do not check all the Windows 2003 unique configurations unless you have issues getting SSH working. b. Also see step 8 below for customization information. 3. In situations where the local administrator account has been renamed. For example: • The local Administrator account has been renamed or disabled but the home directory is still Administrator • The local Administrator account and its home directory have both been renamed In these circumstances there are several steps you need to take: • Ensure that the account to be added has actually logged in to the system at least once, in order to create a home directory for this user. • The passwd file must contain the correct account that is to be used for SSH (mkpasswd command). You may need to edit the passwd file to include the correct home directory for this account. See Passwd and group files and Passwd and group for Windows Implementations and Modifying the HP Systems Insight Manager tools . • Run mxagentconfig for this account to set up the authentication key. • Modify the XML TDEF tool files to use the correct account. See Modifications Summary, and Tool execution user (TDEF modification) for more information. After modification, the XML files must be re-registered using mxtool. For example, run the following command: mxtool –m –f toolname.xml –x force 4. Use a command window from the CMS platform and perform a basic log into test SSH to verify the sshd is running correctly. For example, ssh administrator@localhost. You may need to change to the directory where ssh.exe exists, which is C:\Programs and Files\OpenSSH\bin\ssh.exe, to run the command. A successful execution logs you into the local platform using an SSH connection. 5. Look in the user’s home directory (for example, C:\Documents and Settings\Administrator) for the .ssh folder and the authorized_keys2 file. The HP Systems Insight Manager OpenSSH installer actually puts these in place for the installing user and local administrative account 6. If the authorized_keys2 file is not present then run mxagentconfig from the CMS. See Authorized keys and mxagentconfig. If it succeeds, then go on to testing the SIM commands. For HP Systems Insight Manager 4.2 SP2 also see Modifications Summary. If mxagentconfig fails you can copy the keys manually using the steps below: a. On the Windows CMS: 1. Go to C:\Program Files\HP\Systems Insight Manager\config\sshtools. 2. Enter the following: type .dtfSshKey.pub >> authorized_keys2 3. Log into the remote system and change to the username’s home directory. 4. If not present, create an .ssh directory. 5. Copy the authorized_keys2 file from the CMS into the .ssh directory b. On Linux and HP-UX – similar process as above, except as noted: I. Login as root on the CMS platform II. Create an authorized_keys2 file for copying to other systems by entering the command: cat /etc/opt/mx/config/sshtools/.dtfSshkey.pub >> /<user’s home directory>/.ssh/authorized_keys2 III. The above file is now manually copied to other systems into the $HOME/.ssh directory. IV. Verify that the permissions on the $HOME, $HOME/.ssh directory, and the authorized_keys2 file are owner write only and there are no write permissions for group or everyone. 7. Windows 2003 customization (HP Systems Insight Manager 4.1, 4.2, 4.2.0.1). Windows 2003 does not allow the Local System account to have the privileges it needs to run the OpenSSH (SSH) service. This workaround involves configuring the service to run as a real administrative user. 26 Note: The following five sub-steps are performed by the OpenSSH install on Windows 2003 systems in HP Systems Insight Manager 4.2.0.2 (SP2). The sixth sub-step is performed by the HP Systems Insight Manager install after OpenSSH is installed, by calling mxagentconfig. a. Stopping the service: 1. Go to Start MenuÆControl PanelÆAdministrative ToolsÆ Services. The services window appears. 2. Find the OpenSSH Server service and stop it. b. Changing the Log On As User I. In the same window, right-click the OpenSSH Server service, and select Properties. II. Select the Log On tab. III. Click this account, and enter .\Administrator or other administrative account that is to be used for this service. Local or domain accounts which are members of the Administrators group may be used. IV. Enter the password for this account and click OK. c. Setting file permissions I. Right-click the Start Menu button and select Explore to open a file explorer window. II. Navigate to C:\Program Files\OpenSSH\var\log, and delete any files you find in that directory. III. Navigate to C:\Program Files\OpenSSH\etc and select the files ssh_host_dsa_key, ssh_host_key, and ssh_host_rsa_key by holding down Ctrl and clicking on them. IV. Right-click one of the files, and select PropertiesÆSecurity. V. Click Advanced. VI. Select the Owner tab, and click Other Users or Groups to change the owner to Administrators for all of the files, and then click OK. d. Setting user privileges: Windows 2003 has added an extra security policy and it is possible that policy changes have been made for the Administrators account. I. Open the Control PanelÆAdministrative ToolsÆLocal Security Policy to open the security policy window. II. Find the policies for Create a token object, Replace a process level token and Log on as a service. Administrator, or other administrative account, must be added to this group and can be accomplished by double-clicking each of these privileges in turn and adding administrative account. The administrative account used for OpenSSH must have all three of these policies. III. After double-clicking, click Add User or Group, and enter Administrator or alternate administrative account in the Enter the object names to select field. IV. Click Check Names to verify the entry and then click OK. e. Starting the service: I. At this point, the service Log On As User is set to Administrator or alternate administrative account, and this account has been granted Create a token object, Log on as a service, and Replace a process-level token privileges. II. Return to the Services window, and restart the OpenSSH service. f. Re-installing the HP Systems Insight Manager SSH keys. I. To ensure that SSH authentication is correctly set up for running command line and to ensure custom tasks work in HP Systems Insight Manager, re-run mxagentconfig for Administrator or alternate administrative account if HP Systems Insight Manager was installed by someone other than Administrator. II. To do this, run: mxagentconfig -a -n <cms> -u Administrator -p <pwd> Where <cms> is the name of your management server and <pwd> is the password for the administrator account. Replace Administrator with the name of an alternative administrative account that is to be used to run SIM tools. 27 III. If mxagentconfig fails it is possible that the SSH host key has changed; this happens if OpenSSH has been re-installed. Run the following command to remove the old host key and then repeat step II: mxagentconfig -r -n <cms> 8. Test the basic SIM SSH functionality: I. Open a command window on the CMS. II. To check ssh to see if it is configured for HPSIM: mxagentconfig –c –u <administrator> -n <HPSIM CMS> Note: The –c option is for HP SIM 4.2 SP2 and does not work on earlier versions. III. The command should run and report success IV. Run mxexec -t netstat -n <HP SIM CMS>. V. The command runs and a listing displays after a short wait. 9. If this works, then you are finished. If not, then time for more troubleshooting. I. If the error is not authentication failed, check all the Windows 2003 unique configurations and also if SSH has been reinstalled. These instructions must be followed very carefully. II. You may have to run mxagentconfig -r for the CMS system. See mxagentconfig. III. Then log into the CMS and run Identify systems for the CMS from the OptionsÆDiscoveryÆIdentify menu. This re-adds the CMS to the HP Systems Insight Manager known_hosts file. IV. Although the correct directory permissions are critical it is rare that any issues are found with them. 10. At this point SSH should be working. It usually does not matter which account for HP Systems Insight Manager is used to run the SSH service as long as it is a member of the local Administrators group and has the user rights described above. OpenSSH on a Windows CMS This section explains how SSH is configured after installing HP Systems Insight Manager with OpenSSH on your Windows CMS. The configuration varies depending on your account naming. The settings described below are set up for a new installation of HP SIM version 4.2 SP2. Some additional manual configuration may be required if an earlier version is installed. Two accounts are important for SSH configuration: • The local administrator account. This is called Administrator by default, but may be renamed or disabled in accordance with your security policy. • The account used to install HP SIM. This account must be an administrator on the CMS (a member of the Administrators group), and may be a local or domain account. If HP SIM is installed by the local administrator account then this account is the same as the account above. OpenSSH is installed as a service running as the installing user for Windows XP and 2003. This account must have the appropriate user rights assigned in the Local Security Policy tool. OpenSSH runs as the local system account on Windows 2000 and NT4 installations. The following rights are automatically added during OpenSSH installation: • Log on as a service • Create a token object • Replace a process level token Note: Sometimes a domain policy may prevent HP SIM from adding these rights to the installing user. HP SIM must be installed by a user who can have these rights. Next HP SIM attempts to configure SSH for the two accounts above. 28 To configure SSH: 1. The usernames are added to the passwd file, including the path to their home directory. Domain users are referenced in this file without the domain name. For example, mydomain\myuser is referenced as myuser. 2. An .ssh directory is created in the user’s home directory if it does not exist 3. The authorized_keys2 file is created in the user’s .ssh directory if it does not exist and the CMS public key is appended to the end of this file. In normal operation both the passwd file and authorized_keys2 file are correctly configured on the CMS. However, some cases may require manual configuration: • The user’s home directory does not exist if the user has never logged in to the CMS. Obviously, this cannot apply to the installing user but may apply to the administrator account. In this case, the administrator is not correctly configured during installation. The workaround is to log in to the CMS as administrator and run mxagentconfig to add the administrator. Alternatively, the SIM tools may be modified to run as a different user, such as the installing user, by modifying the TDEF files. Whatever account is chosen for running SSH tools must exist on all managed systems. For more information, see Tool execution user (TDEF modification). • If tools are to run as any user other than Administrator, then tool files must be updated with the correct <execute-as-user> user name. Typically the renamed administrative account is used, but the installing user is an option. As above, whatever account is chosen for running SSH tools must exist on all managed systems. Note: If domain accounts are used, the <execute-as-user> user name should include the domain portion of the name domain\username. For more information, see Tool execution user (TDEF modification). Versions of HP SIM prior to 4.2 SP2 may require additional manual configuration if SSH is not working correctly: • The user account running the OpenSSH service may not have sufficient user rights, as defined by the local security policy. Run the administrative tool Local Security Policy and select the User Rights Assignment, then add the installing user to each of the three policies: o Log on as a service o Create a token object o Replace a process level token • The passwd file may not have the correct home directory of the users. Edit the file to ensure that the correct home directory is referenced. For more information, see the section Passwd and group for Windows Implementations. • The .ssh directory and authorized_keys2 file may not have been correctly configured. Run mxagentconfig for the installing user to correct this. OpenSSH on a Managed System HP SIM can install OpenSSH on managed nodes using the Install OpenSSH tool in the Deploy Drivers, Firmware and Agents menu. This tool requests a username and password to be used to connect to the remote system which is the installing account. The local administrative account is recommended here, although other administrative accounts including domain accounts can be used. As described above for the CMS, the local security policy on the managed system is modified to add user rights for this username, and the OpenSSH service runs as this username. The passwd file is updated to include this username and its corresponding home directory, and the .ssh\authorized_keys2 file is created in the user’s home directory. 29 In addition to the installing user, the local administrative account is also configured in the passwd and authorized_keys2 file. If this username has never logged in to the managed system then this configuration fails. HP SIM versions prior to 4.2 SP2 may require additional manual configuration after OpenSSH is installed. See the section above on the CMS installation. If SSH is not working correctly on a managed system then the simplest solution may be to uninstall OpenSSH, remove the host key from the CMS (with mxagentconfig –r –n <systemname>), and then re-install OpenSSH from HP SIM. Configuring SSH when the Administrator account is disabled or renamed The standard installation of HP Systems Insight Manager assumes a local account called Administrator is available on Windows, and this account is used when running standard tools such as ToolsÆCommand Line ToolsÆWindowsÆdel. What do you do if there is no local account named Administrator? There are some additional steps you must take if you do not have a local account with this name. This includes choosing and configuring an account to use, and update the HP Systems Insight Manager tools to use the correct account name. Configuring the account to use 1. Select a user account that is to be used to run tools on Windows systems, including managed systems and the CMS. The username has to have administrative rights on these managed systems1. The username can be the same account used to install HP Systems Insight Manager provided that account is valid on the managed systems. If this same account is used to install OpenSSH on the managed systems then the managed system is correctly configured. The user can be a domain account or a local account with the same name on each system. If this same user account is to be used to manage Linux or HP-UX systems, the account name must be no longer than eight characters. Take the following steps if the account you want to use is not the one you used to install HP Systems Insight Manager: a. Create the account in Windows if it does not already exist, then login to Windows on the CMS using this account to ensure this user’s home directory is created. b. Enable SSH access for this user by adding the user to the OpenSSH passwd file: I. Navigate to C:\Program Files\OpenSSH\etc II. If a local account is to be used, run mkpasswd –l –u <username> >> passwd The user must be an administrator if the OpenSSH server supplied by HP SIM is used. If another SSH server is used then this may not need be an administrator, provided the chosen user has sufficient right to run the desired tools on the managed system. 1 30 III. Or if you have chosen a domain account, run mkpasswd –d –u <username> <domain> >>passwd c. Verify that the correct home directory is referenced in the passwd file. If not, edit the passwd file to reference the correct directory. d. Add this user account to HP Systems Insight Manager with full configuration rights and authorizations on all systems using the GUI or the following command: mxuser –a <domain>\<username> -p full –C Administrator Modifying the HP Systems Insight Manager tools 1. Modify the Windows HP Systems Insight Manager tools to use the new user account: a. Navigate to the tools directory. For example, C:\Program Files\HP\Systems Insight Manager\tools. b. Search the tools directory for all files that execute tools as Administrator – i.e. all files containing <execute-as-user> Administrator. c. Edit mx-tools.xml for example, using Notepad. d. Find each execute-as-user line in the XML file and change Administrator to the account specified in step 1: <execute-as-user>Administrator</execute-as-user> e. Make changes to reflect the following: <execute-as-user>username</execute-as-user> f. Run mxtool –m –f toolname.xml –x force. g. Repeat these steps for the other XML tools that use the Administrator account: openssh-install.xml, proliant-msa-tools.xml, repair-msatools.xml, wbemsubscriptions.xml, including any ProLiant Essentials specific XML files such as vmmtools.xml. 2. Configure each of your managed systems that is to run tools with this user account: a. If this user account was used to install OpenSSH then the managed node should be correctly configured. If a different account is used then continue with these following steps. b. Ensure that the user has logged into each managed system at some point, creating a home directory. c. Add the user to the passwd file on each managed system. The user is already configured if SSH was installed using that user account. The commands used are the same as those used on the CMS in step 1b above. d. Run mxagentconfig on the CMS to copy the authentication keys for this user to each managed system: mxagentconfig –a –u <username> -p <password> -n <system> Modifications Summary 1. Passwd file: change the home directory for user newname to reference the correct home directory, which is usually Administrator when this account has been renamed. For Example: /home/newname to /home/Administrator 2. TDEF files: Change the username Administrator in the tag <execute-as- user>Administrator</execute-as-user> to the name you recorded in line D above. In this case, the line would read: <execute-as-user>newname</execute-as-user> There are a number of files that should be changed. Basically, it is any XML file in the \tools directory that have the tag line <execute-as-user>Administrator</execute-as-user>.Examples of file names to change are in the graphic below where a search was made for this tag: 31 After modification, the XML files need to be re-registered using mxtool. For example, run the following command: mxtool –m –f toolname.xml –x force Diagnostic Tool using mxagentconfig (Check SSH Setup) HP Systems Insight Manager 4.2 SP2 has added new functionality to the mxagentconfig tool that can aid in diagnosing the creation or copying of the HP Systems Insight Manager public key into the authorized_keys2 file. This tool cannot be used with earlier versions of HP Systems Insight Manager. The new option added for mxagentconfig checks whether SSH on the target system is configured properly to use with HP Systems Insight Manager. This option verifies whether the SSH connection can be successfully authenticated for the specified user using the HP Systems Insight Manager public key authentication methods. If the authentication is successful, HP Systems Insight Manager is able to execute tools on the target system. The syntax to use from the CMS platform is the following: mxagentconfig -c –u username –n targetsystemname 32 Appendix A: Changes found in HP Systems Insight Manager 4.2 SP2 Status # Issue Description of code change or documentation update Found in this document 1 known_hosts Doc - remove system address from the HP Systems Insight Manager known_hosts file using mxagentconfig -r Which SSH client does HP Systems Insight Manager use? Doc and Code - Changes to the SSH Installation regarding local administrator on Windows systems Checklist to debug SSH on Windows, see step 8, and Installing SSH SIM 4.2 + 2 Windows 2003 Installation issues 3 passwd file Doc - How to check and modify the username in the SSH passwd file due to “Authentication failed” error message Passwd and group files and Passwd and group for Windows Implementations 4 Renamed administrator Doc - Discusses modifying the TDEF files and changing the SSH passwd for the /home entry. Tool execution user (TDEF modification) and Passwd and group files 5 Disabled administrator Doc – Discusses modifying TDEF files and reregistering the XML using mxtool. Verify the new user is in the SSH passwd file of the managed system. Tool execution user (TDEF modification) and Passwd and group files 6 Wrong home directory Doc - Passwd file verification of the username in the/home entry. Passwd and group files, and Cygwin mounts 7 Incorrect file permissions Doc – Checking file permissions feature disabled in the windows installation of HP Systems Insight Manager OpenSSH in 4.2 or earlier (strict mode off) in the sshd_config (chmod yes or no), but not in user provided SSH. Checklist to debug SSH on Windows steps 6, 8 and 11. 8 Domain accounts Doc and Code – OpenSSH install has been repackaged and is properly configured SSH on Windows XP, 2000 and 2003 platforms, including when deployed to remote systems using HP Systems Insight Manager. This includes changes to the passwd file and all of the Windows 2003 workarounds. Checklist to debug SSH on Windows step 8. 9 No diagnostic tools for SSH Doc and Code - A new way to diagnose the deployment of SSH keys is implemented by using mxagentconfig –c. Available only in HP Systems Insight Manager 4.2.0.2 (sp2) Diagnostic Tool using mxagentconfig (Check SSH Setup) 10 Improved DTF performance Doc and Code - See #15 below. CMS performance improved when using the SSH bypass options for local CMS operations. HP Systems Insight Manager 4.2 SP2 Affects Need for SSH by ProLiant Tools – Windows CMS Only 11 Domain controller support Part of #8 12 Missing home directory Doc and Code – The revised OpenSSH install package makes a home directory for installing user if not there. This might occur on a ghosted copy of an OS where the Administrator has not yet logged in or where an administrative username has been created and user has never logged-in. Checklist to debug SSH on Windows, see steps 1 and 8, also Installing SSH 13 Installation errors hidden Doc -– check the initconfig.log for SSH installation Installation Diagnostic File for 33 error messages SSH 14 Unsupported policy Doc and Code – see #8 above – requires security policy changes to add extra security privileges in Win 2003. Checklist to debug SSH on Windows, see step 8. 15 Local task execution SSH bypass Doc and Code - Option for bypassing the use of SSH on the Windows CMS for running any SSH command locally. Performance improvement has resulted. Usage of the bypass feature would include VMM and VPM. When implemented only on Windows HP Systems Insight Manager 4.2.0.2, all MSA tools bypass SSH and locally run tasks run as administrator. Two properties are added in the globalsettings.props (not in GUI). Use of the CLI – none. Windows CMS only. Loss of security – none. Bypass users must have system administrator’s privileges to keep security intact. HP Systems Insight Manager 4.2 SP2 Affects Need for SSH by ProLiant Tools – Windows CMS Only Initconfig.log – Shows SSH install errors, but no user message because of no command line install. 34 Appendix B: Troubleshooting When you have a problem executing a task, one of the following might be the cause: • The SSH server on the managed system on which you are trying the command is not available • The user running the command is not authorized to loginthrough SSH to the managed system • The user trying to run the command does not have the HP Systems Insight Manager authorizations to run this tool on that managed system In general, make sure that SSH is available by trying to login outside of HP Systems Insight Manager. Then, make sure the user is able to login through SSH using password authentication, again using some method outside of HP Systems Insight Manager. And finally, check the user’s authorizations in HP Systems Insight Manager, and make sure mxagentconfig has been run for that user against that managed system. Most importantly, make sure the user trying to run the command is the correct user. Sometimes the tool is designed to be run by a particular user such as root or Administrator. Other tools are designed to be run by the user who is logged into the CMS. Initconfig.log – Shows SSH install errors, but no user message because of no command line install. Problem: An MxAuthenticationException is generated when a tool is run, either from the GUI or the command line interface. Solution: Several things can cause authentication exceptions: • The user might not have the privileges needed to run the tool • The user might not be set up with the public key of the DTF To resolve this issue, HP recommends: 1. Make sure that the user you are trying to run as has privileges to run that tool on that system. Refer to the HP Systems Insight Manager online help to check and grant authorizations. 2. Make sure that the SSH server is accessible on the target system. From the CMS, attempt to connect to the target system using an SSH command line tool. There is no need to login, but make sure that you can connect. Try to login as the administrative user to a Windows system, and as root to an HP-UX/Linux system. From an HP-UX/Linux CMS: ssh root@<HP-UX/Linux system> or ssh Administrator@<Windows system> From a Windows CMS: <OpenSSH directory>\bin\ssh root@<HP-UX/Linux system> <OpenSSH directory>\bin\ssh Administrator@<Windows system> If you are prompted to accept a host key or enter a password, then the SSH server is accessible. 3. Re-run mxagentconfig to make sure that the keys are transferred: mxagentconfig -a -n <system name or IP> -u <user> -p <password> With HP SIM 4.2 SP2, you can instead use mxagentconfig –c as discussed earlier. If the return is a success, there would be no need to run the command above. 4. On the system you are attempting to run the HP SIM tools, check the permissions of some directories. 35 Check the current login account permission on the home directory of which you are trying to run the tool. • The home directory should have permissions: drwxr-xr-x (755) • The .ssh directory within the home directory should have permissions: drwxr-xr-x (755) • The authorized_keys2 file in the .ssh directory should have permissions: -rw-r--r-- or -rwxr-xr-x (644 or 755) To check these permissions: On Windows: <OpenSSH Install Directory>\bin\ls -ld <File or directory name> On HP-UX/Linux: ls -ld <File or directory name> a. To change permissions: On Windows: <OpenSSH Install Directory>\bin\chmod <Permission number><File or directory name> On HP-UX/Linux: chmod <Permission number> <File or directory name> (Permission number is the number above, for example, 644/755) When the command is run, the Execute-as user is listed in the status. You are running mxagentconfig for this user. 5. If execution has worked in the past and now is failing, verify that SSH has been reinstalled on the target system. Reinstalling SSH causes the system to have a different host key. Therefore, SSH is not able to verify that the target system is the one that it is trying to contact. If SSH has been reinstalled, then use mxagentconfig to modify known_hosts file: mxagentconfig –r –n <systemname> Note: mxagentconfig is the only tool to remove the entry from known_hosts file while CMS is running. Manually editing file while CMS is running does not have any impact Alternately, you can also remove the entire known_hosts file when CMS is not running, which means that SSH re-registers the keys of every system next time it contacts them. This could be a security problem until each system has been contacted. 6. Remove the .ssh directory from the home directory of the user on the managed system. This removes any old keys or old permissions that could cause mxagentconfig to fail. 7. Run mxagentconfig again. Problem: mxagentconfig fails when trying to authorize a user on a Windows system that did not install OpenSSH. Solution: The user is probably not authorized to use SSH on that system. 1. If trying to run as a Domain User, that user must log into the system prior to running mxagentconfig. The user’s Documents and Settings directory does not exist until the user logs in, and if the user's Documents and Settings directory does not exist, then mxagentconfig fails. 2. As an administrative user on the system, run: c:\Program Files\OpenSSH\bin\mkpasswd -l -u <username> >> “c:\Program Files\OpenSSH\etc\passwd” and c:\Program Files\OpenSSH\bin\mkpasswd -d -u <username> <Domain name> >> “c:\Program Files\OpenSSH\etc\passwd” 36 Note: One of these might exit with an error, depending on the user. This is acceptable and expected. 3. Re-run mxagentconfig. If mxagentconfig still fails, make sure SSH is running by following the steps outlined above. 4. Make sure that the username being sent to mxagentconfig does not include the domain. Use myusername instead of mydomain\myusername. 5. Remove the .ssh directory from the home directory of the user on the managed system. This removes any old keys or old permissions that could cause mxagentconfig to fail. 6. If none of these work, then manually copy over the key. Transfer the file .dtfSshKey.pub to the managed system. The file can be found in the sshtools configuration directory. Linux and HP-UX: /etc/opt/mx/config/sshtools/.dtfSshKey.pub Windows: <HP SIM Install Directory>\config\sshtools\.dtfSshKey.pub On Windows: type <location of .pub file> >> <user's home directory>\.ssh\authorized_keys2 On Linux and HP-UX: cat <location of .pub file> >> ~/.ssh/authorized_keys2 Problem: When executing a task, the message Unknown OS is displayed. Solution: The installation might not have been completed properly. 1. If you are trying to execute a task on a Windows system, make sure that it was rebooted after installation of SSH. A reboot is required to complete the installation. 2. Enable DMI, WBEM, or SNMP on the system so the type of operating system can be determined, then run data collection to update the HP Systems Insight Manager database. 3. Make sure that commands to determine the operating system are working. For Windows, type: ver For HP-UX and Linux, type: uname Problem: mxexec is not working with Windows runas command. Solution: A user who does not have full configuration rights cannot run the command line interface tools. This is expected behavior. Problem: Windows 2003 does not allow the Local System account to have the privileges it needs to run the SSH service. Solution: Configure the service to run as a real administrative user. For more information, see Checklist to debug SSH on Windows. 1. To stop the OpenSSH Server service, go to Start Menu Î Control Panel Î Administrative Tools Î Services to bring up the services window. Find the service labeled OpenSSH Server and stop it. 2. Change the Log On As user: a. In the same window, right-click OpenSSH Server service and select Properties. b. Select the Log On tab. 37 c. Select the This account radio button, and enter .\Administrator. Enter Administrator’s password and click OK. 3. Set file permissions: a. Open a file explorer window by right-clicking the Start menu button, and selecting Explore. Navigate to C:\Program Files\OpenSSH\var\log. Delete any files you find in that directory. b. Navigate to C:\Program Files\OpenSSH\etc and select the files ssh_host_dsa_key, ssh_host_key, and ssh_host_rsa_key by holding down Ctrl and left-clicking on them. c. Right-click on one of the files, select Properties. d. Select the Security tab. e. Click Advanced. f. Select the Owner tab. g. Click Other Users or Groups and change the owner to Administrators. 4. Set user privileges: a. Select Start Menu Î Control Panel Î Administrative Tools Î Local Security Policy to open the security policy window. b. Find the Policies for Create a Token Object and Replace a Process Level Token. c. Add Administrator to this group by double-clicking the appropriate privilege. d. Click Add User or Group, enter Administrator in the Enter the Object Names to Select field e. Click Check Names to verify the entry. f. Click OK. 5. Start the OpenSSH Server service: a. Go to Start Menu Î Control Panel Î Administrative Tools Î Services to bring up the services window. b. Find the service labeled OpenSSH Server and start it. c. At this point, the service Log On As user is set to Administrator, and Administrator has been granted Create a Token Object and Replace a Process Level Token privileges. Return to the Services window and start the service. 6. Reinstall Systems Insight Manager SSH keys: OpenSSH is properly configured to work under Windows 2003. To get command line and custom tasks to work in HP Systems Insight Manager, you must re-run mxagentconfig for Administrator if HP Systems Insight Manager was installed by someone other than Administrator. To re-run mxagentconfig for Administrator from a command window: mxagentconfig –a –u Administrator –p <Administrator’s password> -n <cms machine name> Alternately, run mxagentconfig from the command line with no parameters and enter the data into the GUI. Problem: Standard Windows tools run on the CMS fail with authentication error. Solution: The Administrator account might not be correctly configured on the CMS to run SSH tools. Run mxagentconfig to configure the Administrator: mxagentconfig –a –u Administrator –p <Administrator’s password> –n <cms machine name> Problem: mxagentconfig or command execution fails after reinstalling the openSSH server. Solution: The known_hosts file contains the signature of the old SSH server, and does not allow connections to a server at the same address but with a different key. Edit the known_hosts file under 38 <install dir>/config/sshtools/known_hosts to remove all the lines containing the target hostname and IP address. The new key is added automatically unless adding unknown hosts has been disabled. Refer to How does HP Systems Insight Manager use the known_hosts file? for more details. Note: When HP SIM 4.2 SP2 has been installed, the mxagentconfig –r command should be used to remove hosts from the known_hosts file. 39 Appendix C: Changing server properties The vast majority of users do not need to change any of the default server properties. Please change these values only if absolutely necessary. The HP Systems Insight Manager system daemons read server properties at startup time. To change one of these properties, it is necessary to stop the system daemons, set the property in mx.properties, and restart the daemons. 1. Stop the system daemons. On HP-UX and Linux: /opt/mx/bin/mxstop On Windows: a. Select Start Î Control Panel Î Administrative Tools Î Services. The services window appears. b. Locate the service that begins with HP Systems Insight Manager and double-click the service and click Stop to stop the service. 2. Edit the property. On HP-UX and Linux, edit the file: /etc/opt/mx/config/mx.properties On Windows, edit the file: C:\Program Files\HP Systems Insight Manager\config\mx.properties If the property you want to change does not exist in the property file, add it. Otherwise, edit the property with the desired value. 3. Restart the system daemons. On HP-UX and Linux: /opt/mx/bin/mxstart On Windows: a. Select Start Î Control Panel Î Administrative Tools Î Services. The services window appears. b. Locate the service that begins with HP Systems Insight Manager and double-click the service and click Start to start it. It might take some time for the daemons to initialize and the system to begin responding again. 40 Appendix D: Tool examples This section provides examples of MSA and SSA tools available in HP Systems Insight Manager. MSA tools Category Tool Name Description Command Line Tools PostgreSQL DB Backup Back up the Systems Insight Manager PostgreSQL database. Configuration Tool Subscribe to WBEM Events, Unsubscribe to WBEM Events Configure a managed system to send WBEM indications to HP Systems Insight Manager. Category Tool Name Description Configure Configure DMI Access Set DMI access on selected systems. Configure Configure SNMP Access Set SNMP access on selected systems. General Tools Install RPM Install RPM Package Manager package(s.) General Tools Query RPM Query installed RPM Package Manager package(s) version. General Tools Uninstall RPM Uninstall RPM Package Manager package(s.) General Tools Verify RPM Verify installed RPM Package Manager package(s.) General Tools bdf Report free disk space on files or filesystems. General Tools cat Display the contents of a file. General Tools copy Copy one or more files to another location. General Tools cp Copy file or files to a destination file or directory. General Tools del Delete one or more files (or all files in specified directories.) General Tools df Report free disk space on files or filesystems. General Tools dir Display list of files and subdirectories in a directory. General Tools find Recursively descend a directory hierarchy. General Tools ls List files or directories. General Tools mv Move file or files to a destination. General Tools net Display Windows System and Network information. General Tools netstat Display active network connections. General Tools ps List system processes. General Tools rm Remove files or directory trees. General Tools rmdir Remove a directory and all its contents. General Tools type Display the contents of one or more text files. Partition Management Create Partition Start the Create Partition dialog on the selected system in the complex. Partition Management Partition Manager Start the Partition Manager graphical user interface on the selected system in the complex. SSA tools 41 Category Tool Name Description Partition Management Show Complex Details Start the Show Complex Details dialog on the selected system in the complex. Partition Management View Partition Manager Log Start the Log Viewer dialog on the selected system in the complex. Resource Management Display Resource Usage Display the current Process Resource Manager resource usage. Resource Management Event Monitoring Service Configure and view resource monitoring requests on the managed system. Resource Management List Resource Availability List Process Resource Manager resources available. Resource Management Process Resource Manager Console Run the Process Resource Manager for managing system resources. Software Management CLI List Software Example tool that runs Software Distributor (SD) swlist command on each system. Software Management CLI Preview Install Example tool that runs Software Distributor (SD) swinstall -x match_target=true command on each system. Software Management CLI Verify Software Example tool that runs swverify command on each system. Software Management Set SD Access Set Software Distributor (SD) access to the target system by way of the appropriate SD access control lists (ACLs.) Software Management Software Distributor Daemon Log Display the tail end of the Software Distributor (SD) daemon log. Software Management View Depot Software Start the Software Distributor (SD) graphical user interface to view depot software and depot logfile. Software Management View Installed Software Start the Software Distributor (SD) graphical user interface to view installed software and agent logfile. Software Management View Software Distributor Agent Log Display the tail end of the Software Distributor (SD) agent log. System Administration Accounts for Users and Groups Start the HP-UX SAM Accounts for Users and Groups functional area. System Administration Auditing Start the HP-UX SAM Auditing functional area. System Administration Disks and File Systems Start the HP-UX SAM Disks and File Systems functional area. System Administration Kernel Configuration Start the HP-UX SAM Kernel Configuration functional area. System Administration Peripheral Devices Start the HP-UX SAM Peripheral Devices functional area. System Administration Printers and Plotters Start the HP-UX SAM Printers and Plotters functional area. System Administration System Properties Start the HP-UX SAM System Properties functional area. System Administration System Security Policies Start the HP-UX SAM System Security Policies functional area. System Administration Verified Commands Start the HP-UX SAM Verified Commands functional area. System View SAM Log Start the HP-UX SAM Log Viewer X application. 42 Category Tool Name Description Administration 43 Appendix E: Glossary API— application programming interface. An interface provided for programs to execute services provided by a piece of software, vs. a human executing those services by way of the command line or a GUI. CMS— central management server. The system on which HP Systems Insight Manager is installed. Cygwin— a UNIX compatibility layer that is used to port some UNIX utilities to Windows. DSA— digital signature algorithm. A public key algorithm used by SSH. GUI— graphical user interface. For example, the Web-based portal interface to HP Systems Insight Manager. Host key— the public key that proves the identity of a particular host. IETF— Internet Engineering Task Force. From the IETF Web page: “The Internet Engineering Task Force (IETF) is a large open international community of network designers, operators, vendors, and researchers concerned with the evolution of the Internet architecture and the smooth operation of the Internet.” Managed system— any system on the network being managed by HP Systems Insight Manager, including the CMS itself. Mount point— maps a physical file system name to a logical name, which can then be used for convenience. MSA tool— multi-system aware tool. This is a tool that executed on a certain system called the execution system, and then performs tasks against the target systems. Target systems are provided to the tool by an environment variable. OpenBSD— a free, Berkeley Software Division (BSD) 4.4–based UNIX-like operating system. Their implementation of the SSH protocol is OpenSSH. OpenSSH— a free version of the SSH protocol suite, implemented and supported by the OpenBSD project. Private key— the private half of a public/private key pair. The private key is stored in and owner read-only file (for example, only the owner can view it) on a particular system. The private key is never transmitted to another system. Public key— the public half of a public/private key pair. The public key can be freely distributed without fear that it can be used to impersonate the user. It can only be used for authentication in conjunction with a private key. Remote task— a task initiated on the CMS, and executed on a managed system. RSA— Rivest-Shamir-Adleman. A public key algorithm used by SSH. SFTP— Secure File Transfer Protocol. It is the part of the SSH protocol used to transfer files between systems. This protocol is performed with the same server as command execution. SSA Tool—single-system aware tool. This type of tool is executed by way of SSH on the target system. SSH— Secure Shell. An IETF recommendation. There are two protocols: the original SSH version 1 protocol (SSH-1) and the current SSH version 2 (SSH-2.) Whenever SSH is mentioned in this document, it refers to the SSH-2 protocol. SSH client— connects to SSH servers to perform remote task execution and file copy. SSH server— listens for and services requests coming in on the proper TCP/IP port, usually port 22. Target system— the system selected for a tool to run on. TDEF— tool definition file. It defines parameters of a tool, its execution user, tool box, etc. in XML format. 44 For more information • HP Systems Insight Manager www.hp.com/go/hpsim • IETF secsh working group home page www.ietf.org/html.charters/secsh-charter.html • OpenSSH www.openssh.org 45 © 2004, 2005 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Java is a U.S. trademark of Sun Microsystems, Inc. Linux is a U.S. registered trademark of Linus Torvalds. UNIX is a registered trademark of The Open Group. Windows is a U.S. registered trademark of Microsoft Corporation. 5982-4832EN, 4/2005 46