ssh-agent Compatibility with Ruby Net::SSH Desired

SeanSith
SeanSith
Community Member
edited May 2022 in SSH

I am highly interested in the new 1P8 ssh-agent functionality and I'm hoping this message is received as attempting to help make it better.

Problem:
When attempting to pass authentication from a remote host through a Forwarded Agent setup, 1P8's ssh-agent does not seem to respond to Ruby gem Net::SSH's request to negotiate a protocol version.

Configuration and Background:

  • I am using keys which have been previously generated (and have been working with a macOS's default ssh-agent) and imported into 1P8 beta.
  • I have SSH_AUTH_SOCK configured to point to ~/.1password/agent.sock as suggested here.
  • Basic SSH functionality works using the 1P8 ssh-agent in combination with command-line ssh. I can do interactive and non-interactive sessions perfectly fine.
  • The company I work for uses Capistrano to deploy many of its products. This tool uses Net::SSH to make the connections to remote servers and clone code using git from a remote repository (e.g. GitHub).
  • The remote server used in testing is running Ubuntu 18.04 with a vanilla OpenSSH configuration as provided by the default AWS EC2 Ubuntu 18.04 AMI.
  • I am running Ruby 2.7.5p203 locally with Net::SSH 6.1.0.

Example:
Given the code:

#!/usr/bin/env ruby

require 'net/ssh'
require 'pp'

Net::SSH.start(ARGV[0], ENV['USER'], verbose: Logger::DEBUG,  use_agent: true) do |ssh|
  pp ssh.exec!("ssh -T git@github.com")
end

An attempt to use a remote Linux host described above to authenticate to GitHub results in this abbreviated stack trace:

...
D, [2022-02-17T17:37:22.571198 #39032] DEBUG -- net.ssh.authentication.agent[294]: connecting to ssh-agent
D, [2022-02-17T17:37:22.571297 #39032] DEBUG -- net.ssh.authentication.agent[294]: sending agent request 1 len 46
Traceback (most recent call last):
    12: from ./stub.rb:6:in `<main>'
    11: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh.rb:255:in `start'
    10: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/session.rb:72:in `authenticate'
     9: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/session.rb:72:in `each'
     8: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/session.rb:86:in `block in authenticate'
     7: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/methods/publickey.rb:19:in `authenticate'
     6: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/key_manager.rb:113:in `each_identity'
     5: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/key_manager.rb:204:in `agent'
     4: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/agent.rb:68:in `connect'
     3: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/agent.rb:105:in `negotiate!'
     2: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/agent.rb:221:in `send_and_wait'
     1: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/agent.rb:211:in `read_packet'
/.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/buffer.rb:147:in `append': can't modify frozen String: "" (FrozenError)

When using macOS's default ssh-agent with the code above, the relevant debug output looks as such:

D, [2022-02-17T17:37:13.379668 #38897] DEBUG -- net.ssh.authentication.agent[320]: sending agent request 1 len 46
D, [2022-02-17T17:37:13.379802 #38897] DEBUG -- net.ssh.authentication.agent[320]: received agent packet 5 len 1

This output shows that Net::SSH received a response to its inquiry from macOS's ssh-agent.

The relevant Net::SSH code is here, I believe.

I imagine this isn't exactly a Ruby, nor a Net::SSH, problem. My guess is that 1P8's ssh-agent currently lacks the ability to respond to an inquiry about what agent protocol the systems want to use. I'll apologize for my lack of knowledge in the SSH agent space to determine whether or not Net::SSH should make its inquiry optional. If I need to pursue that avenue, I can.

Thank you for taking your time to look through this report. I am very happy to be able to start moving one more of my authentication mechanisms into 1Password!

Comments

  • floris_1P
    floris_1P
    edited February 2022

    Thanks for the detailed report! We will investigate this. One question I have: do you only experience this issue with an agent forwarding setup or with any Net::SSH request?

  • SeanSith
    SeanSith
    Community Member

    Thanks for taking a look at this!

    Net::SSH works when I'm not executing a task which requires the agent for additional SSH authentication. It does the initial login just fine and I can change the command that's executed (line 7 of my script above) to something like hostname and get back expected output.

    I have ForwardAgent yes in my ~/.ssh/config enabled, so Agent Forwarding is always turned on, but I don't get back the 'frozen String' thing unless the situation calls for it.

  • SeanSith
    SeanSith
    Community Member

    @floris_1P Just wanted to provide an update as it seems something has changed in the behavior of 1P8's SSH beta, but not quite working, yet:

    D, [2022-03-09T11:17:20.822210 #59255] DEBUG -- net.ssh.authentication.agent[294]: sending agent request 13 len 1208
D, [2022-03-09T11:17:20.836267 #59255] DEBUG -- net.ssh.authentication.agent[294]: received agent packet 5 len 1
Traceback (most recent call last):
    12: from ./stub.rb:6:in `<main>'
    11: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh.rb:255:in `start'
    10: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/session.rb:72:in `authenticate'
     9: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/session.rb:72:in `each'
     8: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/session.rb:86:in `block in authenticate'
     7: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/methods/publickey.rb:19:in `authenticate'
     6: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/key_manager.rb:114:in `each_identity'
     5: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/key_manager.rb:114:in `each'
     4: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/key_manager.rb:122:in `block in each_identity'
     3: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/methods/publickey.rb:20:in `block in authenticate'
     2: from /.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/methods/publickey.rb:62:in `authenticate_with'
     1: from/.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/key_manager.rb:180:in `sign'
/Users/ssmith/.asdf/installs/ruby/2.7.5/lib/ruby/gems/2.7.0/gems/net-ssh-6.1.0/lib/net/ssh/authentication/agent.rb:155:in `sign': agent could not sign data with requested identity (Net::SSH::Authentication::AgentError)

    I'm still using the original configuration and code from the first post to test. It seems we're getting further along, but still missing something with regard to request signing according to the code. I'm not sure how to provide additional information, so if there's something else you'd like me to try, please let me know.

    Thanks for making progress on this!

  • floris_1P
    floris_1P

    @SeanSith anything you're seeing in the logs the moment you invoke the SSH request? ~/Library/Group Containers/2BUA8C4S2C.com.1password/Library/Application Support/1Password/Data/logs/1Password_rCURRENT.log on Mac.

  • SeanSith
    SeanSith
    Community Member
    edited March 2022

    @floris_1P When I made the request, here is what I received:

    WARN  2022-03-15T13:47:35.052 tokio-runtime-worker(ThreadId(2)) [1P:ssh/op-ssh-agent/src/codec.rs:48] failed to decode agent message
WARN  2022-03-15T13:47:35.052 tokio-runtime-worker(ThreadId(2)) [1P:ssh/op-ssh-agent/src/lib.rs:252] failed to receive agent request (Protocol), replying with SSH_AGENT_FAILURE
ERROR 2022-03-15T13:47:53.540 tokio-runtime-worker(ThreadId(6)) [1P:/Users/builder/builds/BhfSvM9x/0/dev/core/core/ssh/op-ssh-agent/src/lib.rs:377] Error handling sign request: Key(signing with ssh-rsa is unsupported; SHA-1 may be insecure)
  • floris_1P
    floris_1P

    Ah yes, the agent currently doesn't support SHA1 signatures. I'm not sure if Capistrano lets you configure that it should use a more modern host key algorithm like rsa-sha2-256 and that your server supports that too, but if not you could also consider switching to Ed25519 keys for now.

  • SeanSith
    SeanSith
    Community Member

    @floris_1P Understood. I'd started some other testing after reviewing the logs you requested earlier. I archived my existing RSA keys in 1Password, leaving only my ed25519 key active, and the test script above started working (as does a test deployment using Capistrano as mentioned in my original post).

    As you pointed out, the issue is SHA1 signing, and apparently Net::SSH 6.1.x doesn't support rsa-sha2-256/512, yet, but support is coming in the next major release. I suppose leaving my RSA keys disabled for now until SHA2-256/512 signing is implemented is the best course.

    Thanks for helping me get this far and to you and the team for fixing up the agent response issue which began the initial thread. The SSH support is very welcome and will hopefully be a good selling point in expanding our 1Password usage.

    One last thing, though. Thanks to you showing me the 1P8 log, I saw these errors when successfully using my ed25519 key during the testing above:

    WARN  2022-03-15T17:08:47.993 tokio-runtime-worker(ThreadId(7)) [1P:ssh/op-ssh-agent/src/codec.rs:48] failed to decode agent message
WARN  2022-03-15T17:08:47.994 tokio-runtime-worker(ThreadId(7)) [1P:ssh/op-ssh-agent/src/lib.rs:252] failed to receive agent request (Protocol), replying with SSH_AGENT_FAILURE
WARN  2022-03-15T17:08:48.787 tokio-runtime-worker(ThreadId(10)) [1P:ssh/op-ssh-agent/src/codec.rs:48] failed to decode agent message
WARN  2022-03-15T17:08:48.788 tokio-runtime-worker(ThreadId(10)) [1P:ssh/op-ssh-agent/src/lib.rs:252] failed to receive agent request (Protocol), replying with SSH_AGENT_FAILURE
WARN  2022-03-15T17:08:48.973 tokio-runtime-worker(ThreadId(5)) [1P:ssh/op-ssh-agent/src/codec.rs:48] failed to decode agent message
WARN  2022-03-15T17:08:48.973 tokio-runtime-worker(ThreadId(5)) [1P:ssh/op-ssh-agent/src/lib.rs:252] failed to receive agent request (Protocol), replying with SSH_AGENT_FAILURE

    The authentication and subsequent pass-through was, in fact, successful. Didn't know if that was something that you wanted to look at?

  • floris_1P
    floris_1P

    @SeanSith An update on ssh-rsa: the latest 1Password beta now supports this!

  • RISCfuture
    RISCfuture
    Community Member

    @SeanSith I am running into this same issue with my Capistrano deploys. I have SHA256 Ed25519 key and 1p ssh-agent configured, but it's failing to pass the credential, with the same error in your latest post. Is my understanding correct that this is not a 1p issue but a ruby Net::SSH issue (specifically, Net::SSH does not support SHA256 keys)? Is the solution to revert to using a SHA1 key until Net::SSH is updated? (Or is there a better solution with Capistrano?)

  • RISCfuture
    RISCfuture
    Community Member

    Update, this fixed it:

    diff --git a/Gemfile b/Gemfile
index 66eacd0..6eac71d 100644
--- a/Gemfile
+++ b/Gemfile
@@ -51,6 +51,8 @@ group :development do
   gem 'capistrano-rvm', require: false
   gem 'capistrano-sidekiq', require: false
   gem 'capistrano-webpacker-precompile', require: false
+  gem 'ed25519', require: false
+  gem 'bcrypt_pbkdf', require: false
 end

 group :test do
This discussion has been closed.