SSH agent stopped working after macOS migration

altano

Problem:

1Password and the SSH agent were working fine. I migrated macs by migrating my entire user over and 1Password's agent stopped working, and I can't seem to fix it. Note that while migrating macs I may have changed OS version and made other incidental software changes as I did not try ssh immediately before and after the migration.

I can SSH fine unless it needs my 1Password key, and then I get the error:

$ ssh red
root@red: Permission denied (publickey).

More information:

My key is generated by 1Password
My key is in my Private vault and is NOT archived

I can see the key added to the agent:

$ ssh-add -l
256 SHA256: SSH Key - Alan (1P) (ED25519)

~/.1password/agent.sock is a symlink to the agent.sock:

$ ls -l ~/.1password
... agent.sock -> /Users/alan/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock

I've ruled out my ssh config as causing the problem (by connecting without it):

$ ssh -F none \
-o "PreferredAuthentications publickey" \
-o "IdentityAgent ~/.1password/agent.sock" \
-o "IdentitiesOnly yes" \
red
alan@red: Permission denied (publickey).

Here's a log with ssh -vvvv ...:

OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/alan/.ssh/config
debug1: /Users/alan/.ssh/config line 50: Applying options for red
debug1: /Users/alan/.ssh/config line 188: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/alan/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/alan/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to red port 22.
debug1: Connection established.
debug1: identity file /Users/alan/.ssh/id_rsa type -1
debug1: identity file /Users/alan/.ssh/id_rsa-cert type -1
debug1: identity file /Users/alan/.ssh/id_ecdsa type -1
debug1: identity file /Users/alan/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/alan/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/alan/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/alan/.ssh/id_ed25519 type -1
debug1: identity file /Users/alan/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/alan/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/alan/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/alan/.ssh/id_xmss type -1
debug1: identity file /Users/alan/.ssh/id_xmss-cert type -1
debug1: identity file /Users/alan/.ssh/id_dsa type -1
debug1: identity file /Users/alan/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.4p1 Debian-5+deb11u1
debug1: compat_banner: match: OpenSSH_8.4p1 Debian-5+deb11u1 pat OpenSSH* compat 0x04000000
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to red:22 as 'root'
debug3: record_hostkey: found key type ECDSA in file /Users/alan/.ssh/known_hosts:118
debug3: load_hostkeys_file: loaded 1 keys from red
debug1: load_hostkeys: fopen /Users/alan/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp256
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp256,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:
debug3: record_hostkey: found key type ECDSA in file /Users/alan/.ssh/known_hosts:118
debug3: load_hostkeys_file: loaded 1 keys from red
debug1: load_hostkeys: fopen /Users/alan/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'red' is known and matches the ECDSA host key.
debug1: Found key in /Users/alan/.ssh/known_hosts:118
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug2: get_agent_identities: ssh_agent_bind_hostkey: agent refused operation
debug1: get_agent_identities: agent returned 1 keys
debug1: Will attempt key: /Users/alan/.ssh/id_rsa
debug1: Will attempt key: /Users/alan/.ssh/id_ecdsa
debug1: Will attempt key: /Users/alan/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/alan/.ssh/id_ed25519
debug1: Will attempt key: /Users/alan/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/alan/.ssh/id_xmss
debug1: Will attempt key: /Users/alan/.ssh/id_dsa
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred:
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/alan/.ssh/id_rsa
debug3: no such identity: /Users/alan/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /Users/alan/.ssh/id_ecdsa
debug3: no such identity: /Users/alan/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /Users/alan/.ssh/id_ecdsa_sk
debug3: no such identity: /Users/alan/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /Users/alan/.ssh/id_ed25519
debug3: no such identity: /Users/alan/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /Users/alan/.ssh/id_ed25519_sk
debug3: no such identity: /Users/alan/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /Users/alan/.ssh/id_xmss
debug3: no such identity: /Users/alan/.ssh/id_xmss: No such file or directory
debug1: Trying private key: /Users/alan/.ssh/id_dsa
debug3: no such identity: /Users/alan/.ssh/id_dsa: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
root@red: Permission denied (publickey).

Are these lines interesting?:

debug2: get_agent_identities: ssh_agent_bind_hostkey: agent refused operation
debug1: get_agent_identities: agent returned 1 keys

The log file at ~/Library/Group\ Containers/2BUA8C4S2C.com.1password/Library/Application\ Support/1Password/Data/logs/1Password_rCURRENT.log does not have any output while I am attempt to use ssh.

I confirmed that my PC has the same 1Password key, that the PC can connect fine, and that there are no other authorized keys on the destination host other than the 1Password public key.

---

1Password Version: 1Password for Mac 8.10.1 (81001001)
Extension Version: Not Provided
OS Version: macOS 13.2
Browser:_ Firefox

floris_1P

I've ruled out my ssh config as causing the problem (by connecting without it):
$ ssh -F none \
-o "PreferredAuthentications publickey" \
-o "IdentityAgent ~/.1password/agent.sock" \
-o "IdentitiesOnly yes" \
red
alan@red: Permission denied (publickey).

What happens if you try without IdentitiesOnly yes? Or if you add IdentityFile ~/path/to/public-key?

altano

What happens if you try without IdentitiesOnly yes?

Well that fixed it. What on earth, I thought my config was working before? Also, why would IdentitiesOnly not work with IdentityAgent? Does it only work with IdentityFile?

Anyway, please consider this matter closed. I'll go learn ssh_config some more :)

Thank you so much for your help, even with my not-at-all-a-1password-issue issue.

floris_1P

Glad to hear that it's working for you now! Though still a bit of mystery how a macOS migration/upgrade could impact this.

About IdentitiesOnly: it actually does work with IdentityAgent, as long as you also specify IdentityFile (with a public key). IdentitiesOnly basically tells SSH clients: don't try every key you can find out there, but only use what's set as -i / IdentityFile for this command or host. So if there's no IdentityFile specified, the command will fail.