(feat) Hydrated option for `op list items`

tyltot

Per another issue, we have a vault with 100+ items and are trying to write the vault's items to a JSON file. In doing so, we are calling op list items --vault=<OUR_VAULT> and then op get item <ITEM.UUID> for each item to get a fully hydrated object. However, we've found that it isn't very scalable as we add a larger set of vault items. It would be great if there were another argument to listing items to specify that you want a fully hydrated object, i.e. op list items --vault=<OUR_VAULT> --hydrate.

---

1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

mickael

@tyltot To resolve performance issues in my workflow, I created that tool which uses the local sqlite database of the Desktop client.

I doubt it could be usable straight away in your case, but maybe it could be helpful.

graham_1P

Hi @tyltot

This is a good feature request. Having a per vault/per user bulk fetch with fully inflated items would be handy for your CI/CD scenario, as well as for auditing and other purposes. Plus it would limit the load on our servers. I'll chat with my co-workers regarding the possibility of adding it to a future release.

rickfillion

@tyltot : I would love to better understand what you're trying to achieve. I get that you're looking for the complete info, but for what purposes? And why do you need the whole vault at once?

Rick

tyltot

Hey @rickfillion

We're looking for the complete info so that we can normalize it into our own format for consumption. In the vault items, we not only have credentials, but flags and other information to augment testing. This helps maintain a lot of flexibility as we can treat the vault items as a single source of truth and not have to resort to a bunch of other hacks we've had to do in the past.

We pull the vault all at once for a couple reasons. First, quick access to our own format for the vault items. Second, it's able to keep the workflow and code simpler by separating any setup from test execution. Third, we have a variety of tests - some that run every four hours that are meant to be under twenty minutes and some that run weekly that run for over two hours. In my testing, I've had session issues while maintaining a long running session over a longer testing interval. Finally and most importantly, having the full data is necessary for our legacy system without a refactor effort. Previously, we used a self service mechanism for maintaining credentials and the the framework was built on having the full set of data. Moving to 1password has been a big improvement, but we've struggled with this compatibility piece.

Basic CI/CD Flow

• Spin up testing box
• Initialize 1pass
• Download vault and normalize to our format
• Run variety of tests against integrations
• Aggregate results and notify stakeholders
• Tear down testing box

rickfillion

Thanks for the additional info @tyltot. That really helps.

Hrmm... I thought I had come up with a much better way for you to do this, but it turns out that we aren't exposing a vault's content version with the CLI. Ideally you should be caching that data locally. I suppose you could use the item's itemVersion and only refetch the item if the local cached version is less than the current one. You could use GPG to encrypt your data locally, and use an item in the vault to store the GPG passphrase. This would require persistent storage on your testing box.

I don't see a future where we'd make op list item return fully hydrated items, but I can see the usefulness of having a way to return a full vault in an efficient manner. We have ways of doing that which are much more efficient than fetch-for-every-item, but we still consider them to be relatively expensive.

Rick

mickael

@tyltot If you don't mind syncing the sqlite database of a 1password desktop client to your testing infrastructure, you could check https://github.com/mickaelperrin/onepassword-local-search. I have implemented the --format support for list and it allows you to format the output of the list command with any decrypted field you want.

cohix

Nice!