[SPI-392] Nightly test nodes are not declared in Kerberos database (passwordless ssh fails) Created: 20/Dec/13 Updated: 05/Feb/14 Resolved: 31/Jan/14
Status:
Project:
Component/s:
Affects
Version/s:
Fix Version/s:
Resolved
SPI
None
None
Type:
Reporter:
Resolution:
Labels:
Remaining
Estimate:
Time Spent:
Original
Estimate:
Bug
Andrea Valassi
Completed
None
Not Specified
None
Priority:
Assignee:
Votes:
High
Benedikt Hegner
0
Not Specified
Not Specified
Description
Hi, I finally managed to understand why the CORAL 'network glitch' nightly test intermittently
fails on some servers. The test relies on password-less ssh and this fails if the server is not
declared in the Kerberos database:
[avalassi@ec-slc6-x86-64-spi-8 tcsh] ~ $ ssh -vvv -o
PasswordAuthentication=no -o UserKnownHostsFile=/dev/null -o
StrictHostKeyChecking=no $HOST
OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010
debug1: Reading configuration data
/afs/cern.ch/user/a/avalassi/.ssh/config
debug1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to ec-slc6-x86-64-spi-8 [128.142.153.96]
port 22.
debug1: Connection established.
debug1: identity file
/afs/cern.ch/user/a/avalassi/.ssh/identity type 0
debug3: Not a RSA1 key file
/afs/cern.ch/user/a/avalassi/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /afs/cern.ch/user/a/avalassi/.ssh/id_rsa
type 1
debug3: Not a RSA1 key file
/afs/cern.ch/user/a/avalassi/.ssh/id_dsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespacedebug3: key_read: missing
whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /afs/cern.ch/user/a/avalassi/.ssh/id_dsa
type 2
debug1: Remote protocol version 2.0, remote software version
OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 792 bytes for a total of 813
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchangesha256,diffie-hellman-group-exchange-sha1,diffie-hellmangroup14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfishcbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndaelcbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfishcbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndaelcbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmacsha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmacsha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchangesha256,diffie-hellman-group-exchange-sha1,diffie-hellmangroup14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfishcbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndaelcbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfishcbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndaelcbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmacsha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmacsha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 nonedebug1:
SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 837
debug2: dh_gen_key: priv key bits set: 126/256
debug2: bits set: 507/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 144 bytes for a total of 981
debug3: check_host_in_hostfile: filename /dev/null
debug3: check_host_in_hostfile: filename
/etc/ssh/ssh_known_hosts
debug3: check_host_in_hostfile: filename /dev/null
debug3: check_host_in_hostfile: filename
/etc/ssh/ssh_known_hosts
Warning: Permanently added 'ec-slc6-x86-64-spi8,128.142.153.96' (RSA) to the list of known hosts.
debug2: bits set: 513/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 997
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 48 bytes for a total of 1045
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /afs/cern.ch/user/a/avalassi/.ssh/id_rsa
(0x7f9b126a17f0)
debug2: key: /afs/cern.ch/user/a/avalassi/.ssh/id_dsa
(0x7f9b126a1770)
debug3: Wrote 64 bytes for a total of 1109
debug1: Authentications that can continue: publickey,gssapikeyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapikeyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-withmic,publickey,keyboard-interactive
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-withmic,publickey,keyboard-interactive
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: Trying to reverse map address 128.142.153.96.
debug1: Unspecified GSS failure. Minor code may provide more
information
Server not found in Kerberos database
debug1: Unspecified GSS failure. Minor code may provide more
information
Server not found in Kerberos database
debug1: Unspecified GSS failure.
information
Minor code may provide more
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: Wrote 96 bytes for a total of 1205
debug1: Authentications that can continue: publickey,gssapikeyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key:
/afs/cern.ch/user/a/avalassi/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 240 bytes for a total of 1445
debug1: Authentications that can continue: publickey,gssapikeyex,gssapi-with-mic,password
debug1: Offering public key:
/afs/cern.ch/user/a/avalassi/.ssh/id_dsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 528 bytes for a total of 1973
debug1: Authentications that can continue: publickey,gssapikeyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-withmic,password).
This causes the network glitch test to fail in the nightlies, and also in manual mode on these
nodes:
[avalassi@ec-slc6-x86-64-spi-8 tcsh]
~/myLCG/CORAL_HEAD/src/Tests/PyCoral_NetworkGlitch $ python
NetworkTunnel.py
__main__ Start
__main__ Instantiate NetworkTunnel manager
__main__ Check processes
NetworkTunnel (21915) Check if any network tunnels are open
NetworkTunnel (21915) Execute: 'ps -ef | grep avalassi | grep
/tmp/avalassi/sshTunnel45000 | grep -v grep'
NetworkTunnel (21915) No network tunnel is open
__main__ Open tunnel
NetworkTunnel (21915) Open a new network tunnel from
localhost:45000 to test21-v.cern.ch:10121 through ec-slc6-x8664-spi-8
NetworkTunnel (21915) Execute: '/tmp/avalassi/sshTunnel45000 k -o PasswordAuthentication=no -o UserKnownHostsFile=/dev/null
-o StrictHostKeyChecking=no -N -L 45000:test21-v.cern.ch:10121
ec-slc6-x86-64-spi-8 2>&1 | grep ssh'
NetworkTunnel (21915) Command executed
NetworkTunnel (21915) Sleep 1s
WARNING! Could not open network tunnel (yet?)
WARNING! ps returns 0 tunnels (one expected): []
WARNING! Sleep 3s and run ps again
FATAL ERROR! Could not open network tunnel
FATAL ERROR! ps returns 0 tunnels (one expected): []
FATAL ERROR! print error on stderr and exit
FATAL ERROR! Could not open network tunnel
I imagine this is something to be configured in the AI for these nodes?
Note that, of course, this also means that I cannot use password-less ssh from another node:
[avalassi@slc6pf01 tcsh] ~ $ ssh -vvv ec-slc6-x86-64-spi-8
OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010
debug1: Reading configuration data
/afs/cern.ch/user/a/avalassi/.ssh/config
debug1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to ec-slc6-x86-64-spi-8 [128.142.153.96]
port 22.
debug1: Connection established.
debug1: identity file
/afs/cern.ch/user/a/avalassi/.ssh/identity type 0
debug1: identity file
/afs/cern.ch/user/a/avalassi/.ssh/identity-cert type -1
debug3: Not a RSA1 key file
/afs/cern.ch/user/a/avalassi/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /afs/cern.ch/user/a/avalassi/.ssh/id_rsa
type 1
debug1: identity file
/afs/cern.ch/user/a/avalassi/.ssh/id_rsa-cert type -1
debug3: Not a RSA1 key file
/afs/cern.ch/user/a/avalassi/.ssh/id_dsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespacedebug3: key_read: missing
whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /afs/cern.ch/user/a/avalassi/.ssh/id_dsa
type 2
debug1: identity file
/afs/cern.ch/user/a/avalassi/.ssh/id_dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version
OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 960 bytes for a total of 981
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchangesha256,diffie-hellman-group-exchange-sha1,diffie-hellmangroup14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,sshdss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dsscert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfishcbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndaelcbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-
cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndaelcbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmacripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmacripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchangesha256,diffie-hellman-group-exchange-sha1,diffie-hellmangroup14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfishcbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndaelcbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfishcbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndaelcbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmacsha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmacsha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 1005
debug2: dh_gen_key: priv key bits set: 113/256
debug2: bits set: 521/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 144 bytes for a total of 1149
debug3: check_host_in_hostfile: host ec-slc6-x86-64-spi-8
filename /afs/cern.ch/user/a/avalassi/.ssh/known_hosts
debug3: check_host_in_hostfile: host ec-slc6-x86-64-spi-8
filename /afs/cern.ch/user/a/avalassi/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 679
debug3: check_host_in_hostfile: host 128.142.153.96 filename
/afs/cern.ch/user/a/avalassi/.ssh/known_hosts
debug3: check_host_in_hostfile: host 128.142.153.96 filename
/afs/cern.ch/user/a/avalassi/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 679
debug1: Host 'ec-slc6-x86-64-spi-8' is known and matches the
RSA host key.
debug1: Found key in
/afs/cern.ch/user/a/avalassi/.ssh/known_hosts:679
debug2: bits set: 505/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1165
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 48 bytes for a total of 1213
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /afs/cern.ch/user/a/avalassi/.ssh/id_rsa
(0x7f3054fc96b0)
debug2: key: /afs/cern.ch/user/a/avalassi/.ssh/id_dsa
(0x7f3054fc96e0)
debug3: Wrote 64 bytes for a total of 1277
debug1: Authentications that can continue: publickey,gssapikeyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapikeyex,gssapi-with-mic,passworddebug3: preferred gssapikeyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-withmic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboardinteractive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: Trying to reverse map address 128.142.153.96.
debug1: Unspecified GSS failure. Minor code may provide more
information
Server not found in Kerberos database
debug1: Unspecified GSS failure. Minor code may provide more
information
Server not found in Kerberos database
debug1: Unspecified GSS failure.
information
Minor code may provide more
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: Wrote 96 bytes for a total of 1373
debug1: Authentications that can continue: publickey,gssapikeyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key:
/afs/cern.ch/user/a/avalassi/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 240 bytes for a total of 1613
debug1: Authentications that can continue: publickey,gssapikeyex,gssapi-with-mic,password
debug1: Offering public key:
/afs/cern.ch/user/a/avalassi/.ssh/id_dsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 528 bytes for a total of 2141
debug1: Authentications that can continue: publickey,gssapikeyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
avalassi@ec-slc6-x86-64-spi-8's password:
Can this be fixed? Thanks
Andrea
Comments
Comment by Benedikt Hegner [ 20/Dec/13 ]
Thanks for this detailed analysis. I'll look into that once back to CERN next year.
Comment by Andrea Valassi [ 28/Jan/14 ]
I had another look at this issue because it is the blocker for network glitch tests in the CORAL
nightlies.
There can be many causes for the "Server not found in Kerberos database" error.
For instance, http://technet.microsoft.com/en-us/library/bb463167.aspx suggests that declaring
the hostnames as "hostname.example.com" or simply as "hostname" may have an impact.
However I went through a few nightly nodes and this does not seem to matter. For instance I am
able to ssh ec-slc6-x86-64-spi-2 without a password.
Also the contents of /etc/sysconfig/network do not seem to matter. On that same node ec-slc6x86-64-spi-2 this file is probably misconfigured (HOSTNAME=ec-slc6-test) but I can ssh with
no password anyway.
So I concentrated on the difference between nodes where I can and cannot ssh without
password. The difference seems to be that /etc/krb5.keytab is present in the former and absent in
the latter. For instance: this accepts passwordless login
ssh ec-slc6-x86-64-spi-2
Last login: Tue Jan 28 12:05:35 2014 from slc6pf01.cern.ch
[avalassi@ec-slc6-x86-64-spi-2 tcsh] ~ $ ls -l
/etc/krb5.keytab
-rw-------. 1 root root 472 Dec 6 05:17 /etc/krb5.keytab
while this does not
ssh ec-slc6-x86-64-spi-7
avalassi@ec-slc6-x86-64-spi-7's password:
Last login: Tue Jan 28 11:23:39 2014 from slc6pf01.cern.ch
[avalassi@ec-slc6-x86-64-spi-7 tcsh] ~ $ ls -l
/etc/krb5.keytab
ls: cannot access /etc/krb5.keytab: No such file or directory
In summary, I would suggest to try and recreate /etc/krb5.keytab on all nightly nodes. This may
be enough to solve this issue. There was a message by Jarek Polok beginning 2013 about how to
do this. His message was more or less saying:
Generate the keytab as root:
# cern-get-keytab (--verbose)
Wait few seconds, then verify the resulting new keytab file by
running as root:
# klist -kt
# kinit -k
# klist
You should see valid Kerberos credentials for your system as:
Ticket cache: FILE:/tmp/krb5cc_0_XXXXXXXX Default principal:
host/XXXXXX.cern.ch@CERN.CH
A (non-exhaustive?) list of nodes where this should be done is



ec-slc6-x86-64-spi-7
ec-slc6-x86-64-spi-8
ec-slc6-x86-64-spi-9
On other nodes I can ssh without password:


ec-slc6-x86-64-spi-2
ec-slc6-x86-64-spi-3
Other nodes still seem to be declared in the Kerberos database, but probably have user
authentication (for myself) not configured, ie :




ec-slc6-x86-64-spi-1
ec-slc6-x86-64-spi-4
ec-slc6-x86-64-spi-5
ec-slc6-i686-spi-1
Can you please run cern-get-keytab on those three nodes 7,8,9?
Thanks
Andrea
Comment by Benedikt Hegner [ 30/Jan/14 ]
All the nodes should have access for [a]valassi and the proper keytab now.
Comment by Benedikt Hegner [ 31/Jan/14 ]
As the keytab file was added on the nodes now, I consider this item as resolved.
Comment by Andrea Valassi [ 05/Feb/14 ]
Hi Benedikt,
thanks it looks much better now.
Andrea
Generated at Wed Feb 10 07:26:43 CET 2016 using JIRA 6.4.9#64024sha1:1f1084e06c9893c77549621edbccfecfaa68be5d.