Is there a faster way to empty a vault using CLI2?

agrpcali · November 2022

Hi, I have a requirement to empty certain vaults periodically and then refill them. I'd rather not delete the vault and recreate it and all the permissions if possible.
For a vault with 500 items in it is taking about 20 mins to do this using CLI 2.7 like this:

for item_name in $(op item list --vault $VAULT_NAME --format=json | jq -r '.[].id')
do
op item delete $item_name --vault $VAULT_NAME
done

I also tried the below but it caused a runtime out of memory error when performed on a vault with 500 items:

op item list --vault $VAULT_NAME –format json | op item delete - --vault $VAULT_NAME

I also tried the below but the syntax isn't valid:
op item list --vault $VAULT_NAME --format=json | jq -r '.[].id' | op item delete - --vault $VAULT_NAME

I wonder what the best/most efficient/quickest way is to do this via CLI? Otherwise I have to do manually via the 1Password windows app.

Thankyou, Andrew

1Password Version: CLI 2.7.1
Extension Version: Not Provided
OS Version: Oracle Linux 7
Browser:_ Not Provided

zcutlip · November 2022

If doing this in python is an option for you, you could use pyonepassword for this. Tried it out just now, and have the (relatively minor) changes required implemented in a local branch. If useful. I could cut a release supporting this.

Example:

from pyonepassword import OP

vault = "Test Vault"
op = OP()
item_list = op.item_list(vault=vault)
for item in item_list:
    op.item_delete(item.unique_id, vault=vault)

Worth noting, we're deleting one item at a time, so it's about ~1 sec./item, so not super fast. Also I don't know if you'll hit any throttling issues from 1Password; each item_delete() operation is actually two operations: a get (to resolve the unique ID) followed by a delete.

With moderately more effort, I could implement batch deletion where you could, say, delete 25 (or whatever number ends up working) items at a time.

So, similar to above, but maybe (this isn't implemented yet):

from pyonepassword import OP

vault = "Test Vault"
op = OP()
item_list = op.item_list(vault=vault)

# probably have a default batch size, but override with a kwarg? 
op.item_delete_batch(item_list, vault=vault)

Let me know if either of those sounds useful.

The first option I could probably get out as soon as tonight. The second, wouldn't be too bad, but might take a week or so.

Zach

zcutlip · November 2022

Here's a screenshot of it actually working for me (I had far fewer items than you of course).

Screenshot of multiple item delete in python

agrpcali · November 2022

Zach, thankyou for your post. Your first script is probably deleting about the same speed as my for loop if it's around 1 per second. Mine's perhaps a bit over 1 per sec which is quite a long time for a large vault of 500 items. We wouldn't be allowed to use your API for security reasons but I will still have a look to see how you've done it.

I wonder how the 1Password windows app does a bulk deletion under the hood because I can highlight all 500 items in my vault, click delete and they're gone in seconds or at least appear to be. If they could implement this in CLI it would be very useful to me.

zcutlip · November 2022

We wouldn't be allowed to use your API for security reasons

That's unfortunate as pyonepassword completely open source and can be inspected & verified. But I know how these things go sometimes

I'll still probably push the update. I think I can speed it up a fair bit as well.

zcutlip · January 2023

I know this issue is a couple months old, but for anyone who comes along, I started working on it in pyonepassword . You can follow along here:

https://github.com/zcutlip/pyonepassword/issues/82

zcutlip · January 2023

Again, for anyone who comes along, I have this working in a branch:
https://github.com/zcutlip/pyonepassword/tree/dev/82-support-batch-deletion

The API is:

def item_delete_multiple(self, vault, categories=[], include_archive=False,
                              tags=[], archive=False, name_glob=None, batch_size=25)

In testing, deleting 100 items with the default batch size of 25 takes about 42 seconds, vs 140 seconds looping over the vault's item list deleting one at a time (pyonepassword first does an item_get() for each item deleted, which adds some overhead).

Jack.P_1P · January 2023

Hi @zcutlip:

Thanks for sharing!

Jack

agrpcali · February 2023

I wonder if anyone can share the code behind the multiple items deletion you can do from the 1Password Windows application which takes me about 20 secs for 500 items. This is much faster than any method I know of using the CLI.

As an enhancement request it would be very useful to have a CLI option for emptying a vault (not delete a vault) in a fast efficient way.

zcutlip · March 2023

Just wanted to update this issue with the release of pyonepassword 3.6.0. The main feature in this release is a result of this community question: OP.item_delete_multiple().

OP.item_delete_mutiple() gives you a number of selection criteria, starting with vault but including tags, categories, and title_glob, which is a shell-style glob-pattern

Deleting via item_delete_multiple() is substantially faster than looping over a list of items and deleting one at a time.

It defaults to a batch size of 25 items, which is to say 100 items gets deleted in four passes. It's worth experimenting with the batch_size kwarg to see if there's a more optimal size for your use case.

You can install it from PyPI and read about it here:
https://pypi.org/project/pyonepassword/3.6.0/

Jack.P_1P · March 2023

Great work @zcutlip!

Jack

Is there a faster way to empty a vault using CLI2?

Comments