CLI calls reliability

mickael

Hi,

I have noticed that the performance of the CLI seems to be worse. The call of op commands randomly fails with the error:

(ERROR) Get https://MYSUBDOMAIN.1password.eu/api/v1/account/keysets: net/http: TLS handshake timeout

This becomes really annoying. Randomly, I need to retry 4-5 times op commands to create a single entry.

Do you have some plans to improve the situation ?

We struggled to adapt our script around the bad performance of the CLI but it looks like we have reached a point where we can't do anything on our side now and we can't execute our scripts that rely on op calls without expecting random errors :(

---

1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

graham_1P

Hi @mickael,

That is indeed a problem. op should not be consistently throwing errors in regular usage. Can you tell me a little more about how you are getting these errors? I would like to reproduce the issue you are seeing.

Graham

mickael

Hi @graham_1P ,

Thanks for coming back to me quickly.

Sadly, I don't run op commands directly, but rather use some of our wrappers available here: https://github.com/mickaelperrin/onepassword-tools.

Yesterday, I needed to deploy a new website and wanted to create a new server credential by using the following command:

op-tools new-server-account --username="mynewaccount" --hostname="myserver.com" --return-field=UUID

This script doesn't handle authentication. Authentication is done manually in another command. It only spawns op create item commands with the appropriate parameters.

It looks like there are intermittent issues reaching your .eu hosting or your hosting have from time to time load pikes that prevent them to process requests correctly.

mickael

I forgot to say that this also randomly happens when trying to authenticate with simple op login commands. This morning I got again the issue. This is annoying but less problematic in my case because the process is done interactively not in scripts.

Enter the password for EMAIL at TEAM.1password.eu:
[LOG] 2019/08/21 10:08:09 (ERROR) Get https://TEAM.1password.eu/api/v2/auth/EMAIL/A3/SOMECHARS/CHARS/CHARSAGAIN: net/http: TLS handshake timeout

@graham_1P, So this morning I retried 7 times and gave up. The service looks like it was simply unavaible.

[LOG] 2019/08/21 10:08:09 (ERROR) Get https://TEAM.1password.eu/api/v2/auth/EMAIL/A3/XXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxx: net/http: TLS handshake timeout
[LOG] 2019/08/21 10:11:55 (ERROR) Get https://TEAM.1password.eu/api/v2/auth/EMAIL/A3/XXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxx: net/http: TLS handshake timeout
[LOG] 2019/08/21 10:12:15 (ERROR) Get https://TEAM.1password.eu/api/v2/auth/EMAIL/A3/XXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxx: net/http: TLS handshake timeout
[LOG] 2019/08/21 10:12:37 (ERROR) Get https://TEAM.1password.eu/api/v2/auth/EMAIL/A3/XXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxx: net/http: TLS handshake timeout
[LOG] 2019/08/21 10:12:56 (ERROR) Get https://TEAM.1password.eu/api/v2/auth/EMAIL/A3/XXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxx: net/http: TLS handshake timeout
[LOG] 2019/08/21 10:13:18 (ERROR) Get https://TEAM.1password.eu/api/v2/auth/EMAIL/A3/XXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxx: net/http: TLS handshake timeout
[LOG] 2019/08/21 10:13:38 (ERROR) Get https://TEAM.1password.eu/api/v2/auth/EMAIL/A3/XXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxx: net/http: TLS handshake timeout

I finally successfully logged in at 10:26, it looks like the service was unavailable for at least 18 minutes even if nothing is reported on https://1password.statuspage.io

graham_1P

Thanks for the email with more details mickael.

We are looking into this on our end, and hopefully will come back with some positive results.

mickael

I am currently stucked with that issue. I can't do anything with the CLI anymore. I have around 50% of chance to have a single op command to fail.

All the scripts and helpers I have wrote fails constantly because they are performing multiple op commands one after one, and when this is not the first op call that fails this is the second, or third.

I am in a really bad situation regarding this.

I saw that a previous user of the forum posted a similar issue https://discussions.agilebits.com/discussion/99219/error-net-http-tls-handshake-timeout, it was last year in december.

Like him, I use a lot of custom certificates:

• 12 in session
• 669 in system
• 127 in system root

But, until the beginning of this post, I never encountered that issue and especially during the months of march/april when I wrote all my stuff around your CLI . The CLI was slow but never fails any request.

Version 0.6 was released a few days after the opening of this post. Could it be possible something was upgraded on your servers to prepare the release that causes the issue ?

As I continue to generate news certificates on a regular basis, could it be possible that I reach a point where there are too much local certificates ?

mickael

So, I deleted all my custom trusted certificates and guess what ?

the few tests I did, the op worked fast and reliable.

The performance of the op client is directly impacted by the number of certificates you have in your keychain.

I guess I reached a point where the performance was so bad, it started to timeout.

I hope this is something you could fix in a future release.

In all cases, this is something that should be tagged on your knowledge tool with TLS timeout and performance issues.

graham_1P

Thanks for the follow ups Michael. These posts combined with your emails give good troubleshooting steps to the community.

It is interesting in what you mentioned regarding prior to 0.6.0 there were no failures. Doing a diff between versions, many things were changed between 0.5.7 and 0.6.0 including updates to many of our imported libraries. This is another good data point.

Like in our email conversation, for anyone else reading this try our new op docker image. Considering it is in a more isolated environment, it may remove the certificate issue.

mickael

To be precise, I updated the client only yesterday. Until then I was still on the 0.5.6 version.

But maybe there was some changes deployed in production on your servers just before the 0.6 release.

cohix

@mickael I'll echo what Graham suggested, trying out the Docker version may just resolve your issue in this case.