CLI is so/too slow

Also running into this on macOS. I have 16 environment variables containing tokens on the start-up of a new shell session. Prepending each of my export VAR calls as time export VAR, I was able to get the real timing. Fetching them one-by-one takes 1–2 seconds, plus the time for my Apple Watch to tap my wrist asking for permission (≈5 seconds).

• (15 × 1) + 5 = 20 seconds (best case)
• (15 × 2) + 5 = 35 seconds (worst case)

Of course, this is unacceptable. I was able to improve the efficiency by grouping them with the same tag, and only querying the 1P backend once, and using the --cache flag:

OP_DATA="$(op item list --tags "type:env" --format=json --cache | op item get - --format=json --cache)"
export ARTIFACTORY_ACCESS_TOKEN="$(echo "$OP_DATA" | jq -Mr '. | select(.id == "{id-value}") | .fields[] | select(.id == "credential") | .value')"
export HOMEBREW_GITHUB_API_TOKEN="$(echo "$OP_DATA" | jq -Mr '. | select(.id == "{id-value}") | .fields[] | select(.id == "credential") | .value')"
export GITHUB_TOKEN="$(echo "$OP_DATA" | jq -Mr '. | select(.id == "{id-value}") | .fields[] | select(.id == "credential") | .value')"
export PAGERDUTY_TOKEN="$(echo "$OP_DATA" | jq -Mr '. | select(.id == "{id-value}") | .fields[] | select(.id == "credential") | .value')"
export AC_PASSWORD="$(echo "$OP_DATA" | jq -Mr '. | select(.id == "{id-value}") | .fields[] | select(.id == "credential") | .value')"

unset OP_DATA

This saves time, but is very verbose. Sure, I can wrap it in a function, but still. Doing things this way (fetch once), each jq query takes between 0.04–0.11 seconds. Adding 5 seconds to double-click my Apple Watch for the initial request took (adding the real values from time):

6.461 seconds.

Still a little slow, but MUCH better. This also has the benefit of the environment variables being hydrated from a secure data store, rather than from plain text files on-disk. Cool, but this can still be better.

On macOS, Keychain Access.app is the primary OS-level credential storage solution. Windows and Linux have their own, but I'm going to talk about macOS for the moment. The CLI tool that allows you to interact with Keychain Access.app is called security.

$ time security find-generic-password -a testy -s api_credential -w
my_secret

real    0m0.094s
user    0m0.026s
sys 0m0.043s

Running this repeatedly, I'm seeing the same kinds of times as the jq commands above: 0.04–0.11 seconds. But the benefit is that this works for all macOS users, and is less complex to add support for in your shell session.

The best solution would be something similar-in-spirit to AWS Vault. Tokens are fetched on-demand, passed to a sub-shell where a command executes, then after it's done, the sub-shell ends taking the secret values with it. Something in the spirit of:

$ secrets exec github-personal -- curl -XPOST -H "Authorization: Bearer ${GITHUB_TOKEN}" https://github.com/...
$ secrets exec github-work -- curl -XPOST -H "Authorization: Bearer ${GITHUB_TOKEN}" https://github.enterprise.internal/...

There should probably be a config file (like ~/.aws/config) so that you can configure which environment variable should receive the value of the credential, the name of the field to fetch, etc., but it's a one-time setup sort of thing.

And it has the benefit that — for long-term, interactive shell sessions — the environment variables don't hang out in memory for very long, enabling a rogue script or a poorly-written VSCode extension to write the values of your environment variables to disk as a log file (which happened to me recently).