Debug issues in SSH key-based authentication

SSH prompting for a password despite getting all sshd_config settings correct? Read on!

SSH clients will not be able to give much helpful information on login failures. Hence the server is the right place to start.

The best way to debug any login issues is to start the SSHD on a different port in DEBUG mode on the server and try to connect to this SSHD from a client machine.

Run on server:

# Enable 2222 port to accept connections in iptables:
sudo iptables -A INPUT -p tcp --dport 2222 -j ACCEPT
# Run the sshd
/usr/sbin/sshd -d -p 2222

On the client, log in to the host using the new SSH port (where our server is in DEBUG output mode):

ssh -p 2222 root@<hostname>

Notice the server prints in a problematic scenario:

Server listening on :: port 2222.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 10.104.71.200 port 65321 on 10.196.15.135 port 2222
debug1: Client protocol version 2.0; client software version OpenSSH_8.6
debug1: match: OpenSSH_8.6 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: permanently_set_uid: 50/50 [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: curve25519-sha256 [preauth]
debug1: kex: host key algorithm: ssh-ed25519 [preauth]
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none [preauth]
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user root service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
reprocess config line 123: Deprecated option RhostsRSAAuthentication
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "10.104.71.200"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth_send_banner: sent [preauth]
debug1: userauth-request for user root service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: userauth_pubkey: test pkalg rsa-sha2-512 pkblob RSA SHA256:6Z24v/h/Zb1GyRvXZ0n363UyXiis/1eSmZL940FnIWI [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
Authentication refused: bad ownership or modes for file /root/.ssh/authorized_keys
debug1: restore_uid: 0/0
Failed publickey for root from 10.104.71.200 port 65321 ssh2: RSA SHA256:6Z24v/h/Zb1GyRvXZ0n363UyXiis/1eSmZL940FnIWI

The real problem :

Authentication refused: bad ownership or modes for file /root/.ssh/authorized_keys

Now correct the permissions :

chmod 644 ~/.ssh/authorized_keys

The login will now go through:

/etc/ssh/sshd_config line 122: Deprecated option UsePrivilegeSeparation
/etc/ssh/sshd_config line 123: Deprecated option RhostsRSAAuthentication
debug1: sshd version OpenSSH_7.8, OpenSSL 1.0.2ze-fips  3 May 2022
debug1: private host key #0: ssh-rsa SHA256:4dpQTOE/Ioo+VdUium5OF8rUsxMJP95pEB5d2OTRr+U
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:SHnZyW2FZYwwgqqDuOYrNvnmsJF1troSj0a/bNri59c
debug1: private host key #2: ssh-ed25519 SHA256:uK/8PoIM3s06/l2Nm4x8gZ5ArHCeOY+qz+yiiBhEsSY
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='2222'
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 10.104.71.200 port 65266 on 10.196.15.135 port 2222
debug1: Client protocol version 2.0; client software version OpenSSH_8.6
debug1: match: OpenSSH_8.6 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: permanently_set_uid: 50/50 [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: curve25519-sha256 [preauth]
debug1: kex: host key algorithm: ssh-ed25519 [preauth]
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none [preauth]
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user root service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
reprocess config line 123: Deprecated option RhostsRSAAuthentication
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "10.104.71.200"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth_send_banner: sent [preauth]
debug1: userauth-request for user root service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: userauth_pubkey: test pkalg rsa-sha2-512 pkblob RSA SHA256:6Z24v/h/Zb1GyRvXZ0n363UyXiis/1eSmZL940FnIWI [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: /root/.ssh/authorized_keys:1: matching key found: RSA SHA256:6Z24v/h/Zb1GyRvXZ0n363UyXiis/1eSmZL940FnIWI
debug1: /root/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Accepted key RSA SHA256:6Z24v/h/Zb1GyRvXZ0n363UyXiis/1eSmZL940FnIWI found at /root/.ssh/authorized_keys:1
debug1: restore_uid: 0/0
Postponed publickey for root from 10.104.71.200 port 65266 ssh2 [preauth]
debug1: userauth-request for user root service ssh-connection method publickey [preauth]
debug1: attempt 2 failures 0 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: /root/.ssh/authorized_keys:1: matching key found: RSA SHA256:6Z24v/h/Zb1GyRvXZ0n363UyXiis/1eSmZL940FnIWI
debug1: /root/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Accepted key RSA SHA256:6Z24v/h/Zb1GyRvXZ0n363UyXiis/1eSmZL940FnIWI found at /root/.ssh/authorized_keys:1
debug1: restore_uid: 0/0
debug1: auth_activate_options: setting new authentication options
debug1: do_pam_account: called
Accepted publickey for root from 10.104.71.200 port 65266 ssh2: RSA SHA256:6Z24v/h/Zb1GyRvXZ0n363UyXiis/1eSmZL940FnIWI
debug1: monitor_child_preauth: root has been authenticated by privileged process
debug1: auth_activate_options: setting new authentication options [preauth]
debug1: monitor_read_log: child log fd closed
debug1: PAM: establishing credentials
debug1: rekey after 4294967296 blocks
debug1: rekey after 4294967296 blocks
debug1: ssh_packet_set_postauth: called
debug1: active: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch
debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_pty_req: session 0 alloc /dev/pts/1
debug1: Ignoring unsupported tty mode opcode 11 (0xb)
debug1: Ignoring unsupported tty mode opcode 17 (0x11)
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
Starting session: shell on pts/1 for root from 10.104.71.200 port 65266 id 0
debug1: Setting controlling tty using TIOCSCTTY.