OP CLI - Unable to build a list of item categories

raggy

Hey everyone,

after updating the OP CLI to 2.7.0 and switching to a other docker image I'm getting after some successful op create login attempts the message:

Unable to build a list of item categories

I use "op vault list" and "op item delete" as well and never get an error or something. Only when using op item create...

My progress:
Getting all vaults (~30)
Loop over each vault
delete every item in one vault
create about 60 new "login" items
( Around 2k requests are sent in a short time - This worked before the changes)

I guess the error message has is some kind of rate limit? Because the first item creations works and then suddenly all subsequent don't?
Even with the example from the docs.

op item create \
    --category login \
    --title "Netflix" \
    --vault MyVaultHere \
    --url 'https://www.netflix.com/login' \
    --generate-password='letters,digits,symbols,32' \
    --tags netflix,login,family,share

OP_DEVICE is set to the same value every time and I'm using the --session parameter
Switching back to an older OP CLI Version 1.9.X doesn't work either.

What does the message (Unable to build a list of item categories) mean?

---

1Password Version: 8.9.4
Extension Version: 2.4.4
OS Version: Linux
Browser:_ Chrome

cronekorkn

Same here.

[ERROR] 2022/09/09 13:04:01 Unable to build a list of item categories.

cronekorkn

Just checked Version 2.6.2 and getting same error after some operations. I do one op item list operation und in total 298 op item create and op item delete operations, before the error appears. Is there a limit of 300 operations per x time?

andi.t_1P

Hi @cronekorkn and @raggy! Thanks for bringing this up. At the first glance, this seems as a rate limit issue coming specifically from op item create, as you were also guessing. However, I have a few questions in order to be 100% sure that we are correct, and if we are, to help us design a solution for this:
0. How many op item create operations do you do in total? (do not count the other operations)
1. What is the exact command you are doing for creating all these items? Do you create them using a for loop to repeat a op item create operation or do you use piped input to tell the creation command what items to create (i.e. op item list | op item create -, for example)?

All the best,
Andi

cronekorkn

Hi Andi,

thanks for looking into this.

I can confirm it being a rate limiting issue. When i get the Unable to build a list of item categories error and run something like op item template get login afterwards, i actually get an appropriate error message:

[ERROR] 2022/09/09 18:07:33 (429) Too Many Requests: Too many requests. Your client has been rate-limited.

0.) After a break of multiple hours, i ran it again and managed to create 167 items. My script creates and deletes items by turns and after the create actions start to fail, the delete still works. So delete is indeed not affected by that rate limit.

1.) The command is a loop calling something like this (no pipe):

op item create \
  --category 'login' \
  --title 'some title' \
  --url 'http://some.url' \
  --tags 'sometag,some:othertag' \
  --account '************************' \
  --vault '**********************' \
    username='api' \
    password='*******************'

I also tried avoiding the category lookup by writing the login template to a temporary file and using --template 'temp/file/path' instead of --category 'login', but to no avail.

Thanks for you help,
Martin

raggy

Hi,

glad I'm not the only one having this issue.
As cronekorkn mentioned (0.) the create works for the first X items, then breaks due to the limit and the delete does still work.

I'm calling op item create around 800-1000 times within 10mins.
op item create --category=login --title="Project Stage" --tags="Shop: Project Stage" --url="https://project.de/backend/" --vault="<vaultId>" username="<email>" password="<randomPw>" --format=json --session="<sessionId>"

Thanks in advance,
Marco

andi.t_1P

Hey all, thanks for providing further info on this. So just to triple confirm, this is a rate limit issue. I cannot disclose any information as to what are the thresholds for rate limiting but after some internal talks with have come up with a solution for this and filed an internal issue. What this means for you: in the future, you will be able to avoid being rate limited like this on op item create but only if you switch to the 'piped input' approach. This means that instead of using a for loop to run op item create repeatedly, you will have to get items/item templates locally and then run <command for listing the bulk of item you'd like to create> | op item create - .

All the best,
Andi

cronekorkn

Hi,

alright, i will switch to piped input and wait for the rate limiting changes to take effect. Thank you very much for addressing this issue.

Greetz,
Martin

raggy

Hi Andi,

thanks a lot for your help.
Do you have a real world example how to create a bulk of item with the piped input approach?

This one gives me an internal server conflict 409:
echo "[{\"title\":\"1234\", \"category\":\"LOGIN\"}, {\"title\":\"567\", \"category\":\"LOGIN\"}]" | op item create --vault "VaultName" -

Greetings,
Marco

andi.t_1P

For creating one item through piped input, like in your example, you could use op item template get LOGIN | op item create -. Alternatively you could save the output of the first command to a file and edit with the desired fields: op item template get LOGIN --out-file my1PasswordTemplate.json. After you have created multiple such files you can do something like find . -maxdepth 1 -name '*my1PasswordTemplate*' -exec awk 'FNR' {} + | op item create -. The first command searches all files with my1PasswordTemplate in their name and outputs their contents and then the pipe will pass these contents into the create command for creating multiple items.

Hope this helps a bit,
Andi

cronekorkn

@andi.t_1P Unfortunately, its even worse with piped input:

[ERROR] 2022/12/22 16:36:24 unable to process element 101: Unable to build a list of item categories.

It ratelimits after 100 items. I tried making batches of 100 each, but then it fails soon after with

[ERROR] 2022/12/22 16:43:17 unable to process element 1: Unable to build a list of item categories.

Is there any way to get a reasonable api-limit for our corporate account?

raggy

Same here. Would be awesome to get this to work with more then 100 batch login items