Apparent show stopper for my CLI scripts effort

HippoMan

I found what seems to be a flaw in the CLI software which appears to be a show-stopper for my effort to write CLI utility scripts (https://discussions.agilebits.com/discussion/90284/utility-scripts-for-the-cli).

The basic principle of my procedure is to use op list items and search for uuid's in order to see whether any given item exists outside of the Trash.

However, this methodology falls apart in the following manner:

Suppose I delete all the entries in my vault and then create a new secure note with the title "abcdefg" via op create item "Secure Note" ..., and suppose its uuid is "7i7gmvbmbemnmygkf2njchhf6a".

After doing this, I can do an op get item "abcdefg", and it correctly shows uuid="7i7gmvbmbemnmygkf2njchhf6a".
Furthermore, I can do an op list items, and it also shows uuid="7i7gmvbmbemnmygkf2njchhf6a".

Then, if I do an op create item "Secure Note" ... with the same title and different data followed by op delete item "7i7gmvbmbemnmygkf2njchhf6a", the original item properly goes into Trash.

Then, I can do an op get item "abcdefg", and it shows a new uuid, for example, "snj266t5yytufjprfmr63k574y". Doing an op list items also shows this new uuid, and the Trashed item does not appear in that list ... and when I go to the 1Password web site, I indeed see that one deleted item in the Trash.

So far, so good. However, if I repeat this process, this is where the problem occurs.

If I once again do an op create item "Secure Note" ... with the same title and different data, doing an op get item "abcdefg" indeed shows a new uuid, for example, "olmjqwedln6drvfcardbjt3ldy". But now, if I follow that with op delete item "snj266t5yytufjprfmr63k574y" and an op list items, the op list items still shows a single entry with "snj266t5yytufjprfmr63k574y" as the uuid, even though op get item "abcdefg" shows the new uuid="olmjqwedln6drvfcardbjt3ldy".

This problem op get list seems to occur when there are two or more items in the Trash with the same title, and if that is the same title of an already-existing, non-Trashed Secure Note.

If I then go and manually empty the Trash, the problem goes away, and once again, op list items and op get item "abcdefg" are back in sync.

Because of this inconsistent op get list behavior, my methodology cannot be utilized.

I'm now stopping work on my CLI utility scripts, and I will wait until enough CLI issues get fixed so I can go forward with this effort again.

Again, I think that if we were simply provided with some way to empty the Trash via the CLI, I could then write all these scripts and they would work more reliably.

---

1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

jpgoldberg

That's weird.

(I haven't yet tried to reproduce this, but it definitely sounds like a bug.)

cohix

@HippoMan well now that is strange. I've put this on my list to look at ASAP. I've been following along with your other threads as well and I assure you we're taking it all to heart :) Hoping to incorporate some of your suggestions/issues into the next development cycle.

Thanks for putting the tool through its paces so well, it's people like you that make building this tool worth it!

HippoMan

Well, thanks to both of you for looking this over. Perhaps I'm doing something wrong on my end that I missed. I'd love to find out that this is an error in my own code ... because then I can fix it. :)

Anyway, I'm glad to be doing this work.

cohix

@HippoMan Alright so I dug into this just now. I was not able to reproduce what you're seeing, but I'd like to show my work in case I'm getting something wrong.

Firstly, I'm running this on macOS latest stable.

I started by doing your steps manually, which came up with the behaviour I expected, which is the UUID of the last created item shows up in op get item and op list items.

So naturally, I said "well, let's not be lazy, HippoMaster is a scripting guru, and it's entirely possible that doing this in < 1 sec will yield different results than doing it manually", so I wrote a script (please don't judge me, I am not the ZSH wizard that you are):

UUID1=$(op create item "Secure Note" $(cat note.json | op encode) --title=testnote | jq -r '.uuid')

sed -i.bak "s/asdfghj/qwertyu/g" ./note.json
UUID2=$(op create item "Secure Note" $(cat note.json | op encode) --title=testnote | jq -r '.uuid')
op delete item $UUID1

sed -i.bak "s/qwertyu/zxcvbnm/g" ./note.json
UUID3=$(op create item "Secure Note" $(cat note.json | op encode) --title=testnote | jq -r '.uuid')
op delete item $UUID2

sed -i.bak "s/zxcvbnm/dfghjkkll/g" ./note.json
UUID4=$(op create item "Secure Note" $(cat note.json | op encode) --title=testnote | jq -r '.uuid')
op delete item $UUID3

op list items --vault=qebeow3ahuj5u6u7mcogxb4x4m | jq .

echo $UUID1
echo $UUID2
echo $UUID3
echo $UUID4

And that yielded the output:

./testnotes.sh
[
  {
    "uuid": "v7jydi3w5b5qrsztocuuxebzpi",
    "templateUuid": "003",
    "trashed": "N",
    "createdAt": "2018-05-28T22:57:41Z",
    "updatedAt": "2018-05-28T22:57:41Z",
    "changerUuid": "FFTR4EMAJBE3FKHWV4NGGCZD4M",
    "itemVersion": 1,
    "vaultUuid": "qebeow3ahuj5u6u7mcogxb4x4m",
    "overview": {
      "ainfo": "",
      "title": "testnote"
    }
  }
]
btzymlq2lj5ptt27jf2afbairu
nlggkqvn2qxa4lc46ujak6qf6q
tgzdqnndx73ayydhu7rfcs545u
v7jydi3w5b5qrsztocuuxebzpi

Which, as far as I can tell, is exactly what's supposed to happen.

And, of note, I ran this with an empty vault and an empty trash. The contents of note.json is just the output of op get template "Secure Note" with the gibberish strings that sed changes each time for me.

Please let me know if this is at all matching what you see, and please run my script to see if you get the same results.

HippoMan

Thank you for all this work!

I'm going to check this tomorrow ... it's already past my bedtime :)

Your tests look promising, and I'm hoping that this indeed shows that my issues were just due to programming errors on my part.

Stay tuned ...

HippoMan

I ran your test code, and its output is the same under my linux environment. However, there are still discrepancies that exist between using op get note with a uuid argument and using it with a title argument. The title argument gives inconsistent results. I cleared my vault and Trash, and I ran this variation of your script ...

# For consistency upon retesting, always start with the same original note
cp orignote.json note.json

vaultid=vixcgqmctfuqklpkyrtq4mg4fm

echo ''

UUID1=$(op create item "Secure Note" $(cat note.json | op encode) --title=testnote --vault=${vaultid} | jq -r '.uuid')
echo "for UUID1: from create item:       UUID=$UUID1"
echo "for UUID1: from get item testnote: UUID=$(op get item testnote --vault=${vaultid} | jq .uuid -r)"
echo "for UUID1: from get item UUID1:    UUID=$(op get item ${UUID1} --vault=${vaultid} | jq .uuid -r)"
echo ''

sed -i.bak "s/asdfghj/qwertyu/g" ./note.json
UUID2=$(op create item "Secure Note" $(cat note.json | op encode) --title=testnote --vault=${vaultid} | jq -r '.uuid')
op delete item $UUID1
echo "for UUID2: from create item:       UUID=$UUID2"
echo "for UUID2: from get item testnote: UUID=$(op get item testnote --vault=${vaultid} | jq .uuid -r)"
echo "for UUID2: from get item UUID2:    UUID=$(op get item ${UUID2} --vault=${vaultid} | jq .uuid -r)"
echo ''

sed -i.bak "s/qwertyu/zxcvbnm/g" ./note.json
UUID3=$(op create item "Secure Note" $(cat note.json | op encode) --title=testnote --vault=${vaultid} | jq -r '.uuid')
op delete item $UUID2
echo "for UUID3: from create item:       UUID=$UUID3"
echo "for UUID3: from get item testnote: UUID=$(op get item testnote --vault=${vaultid} | jq .uuid -r)"
echo "for UUID3: from get item UUID3:    UUID=$(op get item ${UUID3} --vault=${vaultid} | jq .uuid -r)"
echo ''

sed -i.bak "s/zxcvbnm/dfghjkkll/g" ./note.json
UUID4=$(op create item "Secure Note" $(cat note.json | op encode) --title=testnote --vault=${vaultid} | jq -r '.uuid')
op delete item $UUID3
echo "for UUID4: from create item:       UUID=$UUID4"
echo "for UUID4: from get item testnote: UUID=$(op get item testnote --vault=${vaultid} | jq .uuid -r)"
echo "for UUID4: from get item UUID4:    UUID=$(op get item ${UUID4} --vault=${vaultid} | jq .uuid -r)"
echo ''

op list items --vault=${vaultid} | jq . -M

Here's the output ...

for UUID1: from create item:       UUID=upfr552tdu2efwitznnxf6hwxa
for UUID1: from get item testnote: UUID=upfr552tdu2efwitznnxf6hwxa
for UUID1: from get item UUID1:    UUID=upfr552tdu2efwitznnxf6hwxa

for UUID2: from create item:       UUID=6gcvbvy4f57b5x6gdfujqt5y3u
for UUID2: from get item testnote: UUID=6gcvbvy4f57b5x6gdfujqt5y3u
for UUID2: from get item UUID2:    UUID=6gcvbvy4f57b5x6gdfujqt5y3u

for UUID3: from create item:       UUID=mqvxm4p5pxnzrzt3nhpoqjqoza
for UUID3: from get item testnote: UUID=6gcvbvy4f57b5x6gdfujqt5y3u
for UUID3: from get item UUID3:    UUID=mqvxm4p5pxnzrzt3nhpoqjqoza

for UUID4: from create item:       UUID=otcs5u5b6yc2pgroqo74mxydzy
for UUID4: from get item testnote: UUID=6gcvbvy4f57b5x6gdfujqt5y3u
for UUID4: from get item UUID4:    UUID=otcs5u5b6yc2pgroqo74mxydzy

[
  {
    "uuid": "otcs5u5b6yc2pgroqo74mxydzy",
    "vaultUuid": "vixcgqmctfuqklpkyrtq4mg4fm",
    "trashed": "N",
    "templateUuid": "003",
    "overview": {
      "ainfo": "",
      "title": "testnote"
    }
  }
]

Note that this starts going south starting with UUID3. I believe it's because the op get item TITLE --vault=VAULT_UUID command gets confused by the items that are already in the Trash with the same TITLE.

I realize that as long as I'm always doing op get item by UUID, everything works fine. However, my limited brain can only remember my secure notes via meaningful titles, not random 26-character ID's. :)

PS: and also note that this occurs even when I always explicitly specify the vault ID on all of the op commands.

HippoMan

In addition, I get the same op get list TITLE inconsitencies when I add --vault=VAULT_UUID to all the op delete item UUID commands.

Also, if you run this script a second time without clearing the Trash, you'll see the incorrect op get list TITLE results starting with UUID1. This further supports my conjecture that op get list TITLE --vault=VAULT_UUID gets confused by the existing items in the Trash.

HippoMan

NOTE: I incorrectly wrote op get list above, when I actually meant to write op get item.

With this op get item problem in mind, I wrote an op-title-exists function that doesn't utilize op get item at all. It traverses the results of op list items instead. I'm not sure, but op list items seems to not suffer the same problem as op get item.

Here is the code. I will test it further, and if I don't come up with any more issues, I will go back to my full scripting effort during my spare time.

#!/bin/zsh -f                   

prog=${0##*/}
op=/usr/local/bin/op
jq=/usr/bin/jq
tmp=/tmp/.${prog}.$$

quit() {
  /bin/rm -f ${tmp}
  exit ${1}
}

trap 'quit 1' 1 2 3 15

[[ $# -lt 1 ]] && {
  print - "
usage: ${prog} 'title' [ optional 'op get item' arguments ]

  returns 0 if an item with the title exists, otherwise non-zero

  optional arguments could contain '--vault=XXX' or
  '--tag=FOOBAR'; but don't specify '--title=XXX'
"
  quit -2
}

title="${1}"
shift

# Use temp files so that we can catch the `op` return code
# before passing any results to `jq`, and likewise for
# catching the `jq` return code before checking its results.

${op} list items "${@}" >${tmp} || quit 1
[[ -s "${tmp}" ]] || quit 1

found=0
saveUuid=''
filter="select(.[] | .overview.title == \"${title}\") | .[] | .uuid"
${jq} "${filter}" --raw-output <${tmp} \
| while read uuid dummy
  do
    (( found = found + 1 ))
    saveUuid="${uuid}"
  done

case "${found}" in
(1)
  # Found one item. Print its uuid to stdout with a 0 return code
  print - "${saveUuid}"
  quit 0
  ;;
(0)
  # No items found
  quit -1
  ;;
(*)
  # Found more than one item.
  # The return code is the number of items.
  print - "${found} matches for ${item} ${*}"
  quit ${found}
  ;;
esac

cohix

@HippoMan well yes I think you've figured it out. The get by name does seem to get confused whereas the get by UUID does not. I believe I've figured out why this is. Let me take another look a little later today and I'll get back to you with what I find.

HippoMan

OK, thank you!

But in any case, I now have a work-around that utilizes op list items, and that seems to work fine ... at least so far.

rickfillion

Filed as issue 441 here.

Rick

HippoMan

I have now realized that I accidentally ran my test program with an older version of op that was still on my system. When I properly test with the latest 0.4.1 version, the problem went away.

I'm sorry for the false alarm.

So, you might want to remove issue 441

cohix

@HippoMan that's great! I was a little confused just now when I ran your modified version of my script and it all came out normally :)

Glad to see everything is working again!