CLI secure-note utilities written in python

HippoMan

After finally figuring out how to get the CLI working via shell-script wrappers, I have now written some python scripts for managing Secure Notes via the CLI ... currently, these work in version 0.4.1.

I chose python because it offers more programming capabilities than shell languages, such as a json module that eliminates the need for hacking around with jq. Also, I didn't choose go, because I have years of experience with python, but am still learning go.

These scripts can create, view, update, and delete Secure Notes. The only caveat is that by "delete", I mean "put them in the Trash". The Trash will still have to be emptied manually via the 1Password UI. Once the CLI offers an empty-the-Trash capability (please!!!), I can update these python programs so they don't leave lots of dangling garbage sitting around in the Trash.

As soon as I finish writing this message, I will post the scripts here, one by one, in this thread. Here's a summary ...

1. op-item-exists -- python program: tests for the existence of any item in any vault (used internally in the other programs)
2. op-get-note -- python program: retrieves the contents of any Secure Note
3. op-put-note -- python program: creates or updates a Secure Note
4. op-rm-note -- python program: deletes a Secure Note
5. op-create-note -- shell script: very thin wrapper around op-put-note which only does creation of Secure Notes
6. op-update-note -- shell script: very thin wrapper around op-put-note which only does updating of Secure Notes

---

1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

HippoMan

op-item-exists -- check for item existence

usage: op-item-exists [ -qQvV | --verbose | --quiet ] item [ optional 'op create item' arguments ]

  tests for the existence of an item, and prints out its uuid if found

  -v, -V, or --verbose mean to print out some error messages
  -q, -Q, or --quiet   mean to not print out the item's uuid if found
  note that --verbose and --quiet are independent; both can exist

  returns  0 if the item exists,
           N if it finds N (> 1) matched items,
          -1 if the item doesn't exist,
           1 in all other cases

Python code ...

#!/usr/bin/python3

import sys
sys.dont_write_bytecode = True

import os
import re
import json
import getopt

sys.path.insert(0, '/usr/local/etc/python')
from u import invoke

prog    = os.path.basename(sys.argv[0])
op      = '/usr/local/bin/op'
multpat = re.compile(r'\(ERROR\)\s+multiple\s+items\b', re.M | re.DOTALL)
nonepat = re.compile(r'\(ERROR\)\s+\bitem\s+.+?\s+not\s+found\b', re.M | re.DOTALL)

def main():
    try:
        opts, args = getopt.getopt(sys.argv[1:], 'vVqQ', ['verbose', 'quiet'])
    except getopt.GetoptError as e:
        print('{}'.format(e))
        return 1

    quiet   = False
    verbose = False
    for o, a in opts:
        if o in ('-v', '-V', '--verbose'):
            verbose = True
        elif o in ('-q', '-Q', '--quiet'):
            quiet   = True
        else:
            return usage('invalid option: {}'.format(o))

if len(args) < 1:
    return usage()

item = args[0]

rc, o, e = invoke([ op, 'get', 'item', item ] + args[1:])
if rc == 0:
    resp = json.loads(o)
    if not quiet:
        print(resp['uuid'])
    return 0
elif multpat.search(e):
    ndups = len(o.split('\n')) - 1
    if verbose:
        print('{} matching items: {}'.format(ndups, item))
    return ndups
elif nonepat.search(e):
    if verbose:
        print('no matching items: {}'.format(item))
    return -1
else:
    return rc

def usage(msg=None):
    if msg:
        print('\n{}'.format(msg))
    print('''
usage: {} [ -qQvV | --verbose | --quiet ] item [ optional 'op create item' arguments ]

  tests for the existence of an item, and prints out its uuid if found

  -v, -V, or --verbose mean to print out some error messages
  -q, -Q, or --quiet   mean to not print out the item's uuid if found
  note that --verbose and --quiet are independent; both can exist

  returns  0 if the item exists,
           N if it finds N (> 1) matched items,
          -1 if the item doesn't exist,
           1 in all other cases
'''.format(prog))
    return -2

if __name__ == '__main__':
    sys.exit(main())

HippoMan

op-get-note - retrieve Secure Note

usage: op-get-note [ -qQ | --quiet ] item [ optional 'op get item' arguments ]

  -q, -Q, or --quiet mean to print no error messages

  if the item exists, returns 0 and prints the item to stdout,
  otherwise returns non-zero

  optional arguments could contain '--vault=XXX', for example

Python code ...

#!/usr/bin/python3

import sys
sys.dont_write_bytecode = True

import os
import json
import getopt

sys.path.insert(0, '/usr/local/etc/python')
from u import invoke

prog       = os.path.basename(sys.argv[0])
op         = '/usr/local/bin/op'
itemexists = '/usr/local/bin/op-item-exists'

def main():

    try:
        opts, args = getopt.getopt(sys.argv[1:], 'qQ', ['quiet'])
    except getopt.GetoptError as e:
        print('{}'.format(e))
        return 1

    verbose = True
    for o, a in opts:
        if o in ('-q', '-Q', '--quiet'):
            verbose = False
        else:
            return usage('invalid option: {}'.format(o))

    if len(args) < 1:
        return usage()

    name = args[0]
    base = os.path.basename(name)
    rc, o, e = invoke([ itemexists, base ] + args[1:])
    if rc == 255:
        if verbose:
            print('not found: {}'.format(name))
        return -1
    elif rc > 1:
        if verbose:
            print('{} items found: {}'.format(rc, name))
    elif rc:
        return -1

    rc, o, e = invoke([ op, 'get', 'item', o.rstrip() ] + args[1:])
    if rc:
        if verbose:
            print(e)
        return rc

    jsonerror = 'invalid json: {}'.format(name)
    try:
        data = json.loads(o)
    except:
        if verbose:
            print(jsonerror)
        return 1

    details = data.get('details', None)
    if not details:
        if verbose:
            print(jsonerror)
        return 1

    notes = details.get('notesPlain', None)
    if notes is None:
        if verbose:
            print('invalid json: {}'.format(name))
        return 1

    sys.stdout.write('{}\n'.format(notes))
    sys.stdout.flush()

    return 0

def usage(msg=None):
    if msg:
        print('\n{}'.format(msg))
    print('''
usage: {} [ -qQ | --quiet ] item [ optional 'op get item' arguments ]

  -q, -Q, or --quiet mean to print no error messages

  if the item exists, returns 0 and prints the item to stdout,
  otherwise returns non-zero

  optional arguments could contain '--vault=XXX', for example
'''.format(prog))
    return -2

if __name__ == '__main__':
    sys.exit(main())

HippoMan

op-put-note - create or update a Secure Note

usage: op-put-note [ -iIcCuU | --stdin | --create | --update ] item-name [ args ... ]

  'args' are optional 'op create item "Secure Note"' arguments
  which could contain '--vault=XXX' or '--tag=FOOBAR';
  but don't use '--title=XXX'

  returns 0 if the item has been created or updated, otherwise non-zero

  option -i, -I, or --stdin  means take item contents from stdin;
  option -c, -C, or --create means only create when item doesn't exist;
  option -u, -U, or --update means only update when item does exist;
  with neither --create nor --update, decide which to do based upon existence;
  --create and --update cannot both appear

Python code ...

#!/usr/bin/python3

import sys
sys.dont_write_bytecode = True

import os
import re
import json
import getopt

sys.path.insert(0, '/usr/local/etc/python')
from u import invoke, utf8decode

prog       = os.path.basename(sys.argv[0])
op         = '/usr/local/bin/op'
itemexists = '/usr/local/bin/op-item-exists'
titlepat   = re.compile(r'^--title=', re.I)

def main():

    try:
        opts, args = getopt.getopt(sys.argv[1:], 'cCiIuU', ['create', 'stdin', 'update'])
    except getopt.GetoptError as e:
        print('{}'.format(e))
        return 1

    fromstdin   = False
    forcecreate = False
    forceupdate = False
    for o, a in opts:
        if o in ('-i', '-I', '--stdin'):
            fromstdin = True
        elif o in ('-c', '-C', '--create'):
            forcecreate = True
        elif o in ('-u', '-U', '--update'):
            forceupdate = True
        else:
            return usage('invalid option: {}'.format(o))

    if (forcecreate and forceupdate) or len(args) < 1:
        return usage()

    for a in args:
        if titlepat.search(a):
            return usage('illegal argument: {}'.format(a))

    item   = args[0]
    params = args[1:]
    base   = os.path.basename(item)
    cmd    = None
    data   = None
    exists = None
    uuid   = None

    rc, o, e = invoke([ itemexists, base ] + params)
    if rc == 0:
        if forcecreate:
            return 1
        exists = True
        uuid   = o.rstrip()
    elif rc == 255:
        if forceupdate:
            return 1
        exists = False
    else:
        print(e)
        return rc

    try:
        if fromstdin:
            data = utf8decode(sys.stdin.read())
        else:
            with open(item, 'r') as f:
                data = utf8decode(f.read())
    except:
        return -1

    note = {
        'notesPlain': data,
        'sections':   []
    }

    jnote = json.dumps(note).rstrip()

    rc, o, e = invoke([ op, 'encode' ], input=jnote)
    if rc:
        return rc

    encoded = utf8decode(o.rstrip())

    rc, o, e = invoke([ op, 'create', 'item', 'Secure Note', encoded, '--title={}'.format(base) ] + params)
    if rc:
        return rc

    if exists:
        rc, o, e = invoke([ op, 'delete', 'item', uuid ] + params)

    return rc

def usage(msg=None):
    if msg:
        print('\n{}'.format(msg))
    print('''
usage: {} [ -iIcCuU | --stdin | --create | --update ] item-name [ args ... ]

  'args' are optional 'op create item \"Secure Note\"' arguments
  which could contain '--vault=XXX' or '--tag=FOOBAR';
  but don't use '--title=XXX'

  returns 0 if the item has been created or updated, otherwise non-zero

  option -i, -I, or --stdin  means take item contents from stdin;
  option -c, -C, or --create means only create when item doesn't exist;
  option -u, -U, or --update means only update when item does exist;
  with neither --create nor --update, decide which to do based upon existence;
  --create and --update cannot both appear
'''.format(prog))
    return -2

if __name__ == '__main__':
    sys.exit(main())

HippoMan

op-rm-note - delete a Secure Note

usage: op-rm-note item [ optional 'op delete item' arguments ]

  returns 0 if the item is deleted, otherwise non-zero

  optional arguments could contain '--vault=XXX'

Python code ...

#!/usr/bin/python3

import sys
sys.dont_write_bytecode = True

import os

sys.path.insert(0, '/usr/local/etc/python')
from u import invoke

prog       = os.path.basename(sys.argv[0])
op         = '/usr/local/bin/op'
itemexists = '/usr/local/bin/op-item-exists'

def main():

    if len(sys.argv) < 2:
        return usage()

    name   = sys.argv[1]
    base   = os.path.basename(name)
    params = sys.argv[2:]

    rc, o, e = invoke([ itemexists, base ] + params)
    if rc:
        return rc

    uuid = o.rstrip()

    rc, o, e = invoke([ op, 'delete', 'item', uuid ] + params)
    return rc

def usage(msg=None):
    if msg:
        print('\n{}'.format(msg))
    print('''
usage: {} item [ optional 'op delete item' arguments ]

  returns 0 if the item is deleted, otherwise non-zero

  optional arguments could contain '--vault=XXX'
'''.format(prog))
    return -2

if __name__ == '__main__':
    sys.exit(main())

HippoMan

op-create-note - thin shell wrapper around op-put-note for creating Secure Notes ...

#!/bin/zsh -f
prog=${0##*/}
exec -a ${prog} /usr/local/bin/op-put-note --create "${@}"
exit -1

op-update-note - thin shell wrapper around op-put-note for updating Secure Notes ...

#!/bin/zsh -f
prog=${0##*/}
exec -a ${prog} /usr/local/bin/op-put-note --update "${@}"
exit -1

Note: exec -a ${prog} is a zsh construct to force argv[0] to be set in the executed program.

HippoMan

Note that these python programs use some functions from a module which I called u. Here's the __init__.py code for this u module, wherein these functions are defined ...

#!/usr/bin/python3

import sys
sys.dont_write_bytecode = True

import os
import warnings
import unidecode
import subprocess

def utf8decode(text, rstripnull=True):
    if not text:
        return ''
    if type(text) is bytes:
        if rstripnull:
            try:
                text = text.rstrip(b'\0')
                if not text:
                    return ''
            except:
                pass
        try:
            text = text.decode('utf8')
        except:
            text = text.decode('utf8', errors='replace')
        if not text:
            return ''
    try:
        with warnings.catch_warnings() as w:
            warnings.simplefilter('ignore')
            text = unidecode.unidecode(text)
    except:
        text = ''
    return text

def _communicate(subproc, inputstr=None):
    if inputstr is None:
        return subproc.communicate()
    if not inputstr:
        inputstr = ''
    if sys.version_info >= (3,):
        return subproc.communicate(inputstr.encode())
    else:
        return subproc.communicate(inputstr)

def invoke(cmd, input=None, evars=None):
    e = os.environ.copy()
    if evars:
        for k,v in evars.items():
            e[k] = v
    p = subprocess.Popen([str(x) for x in cmd], shell=False, bufsize=0, env=e, close_fds=True,
                         stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    try:
        out, err = _communicate(p, inputstr=input)
    except Exception as e:
        out = ''
        err = ''
    rc = p.returncode
    return (rc, utf8decode(out), utf8decode(err))

rickfillion

This is incredible. You should throw that stuff onto a github repo to make it easily sharable with others.

Wow you're awesome. :)

Rick

HippoMan

Thank you!

Yes, I plan to github-ize it soon.

AGAlumB

Please do! SO great! Secure Notes are my FAVOURITE. :chuffed:

cohix

@HippoMan This is damn fantastic. If you create a GitHub repo for this, I would love to fork and contribute comparable Go wrappers. It would be amazing to start a community-driven repository of scripts.

Seriously, thank you for all the hard work you've been doing over the past few weeks.

HippoMan

Thank you!

I started a new job and have less spare time now, so it might be a couple weeks before I can get the github repo set up. But I will definitely do it.

HippoMan

Also, here are some proposed enhancements of the CLI code base which could make these python wrapper efforts more efficient:

• The ability in the CLI to query items (documents, secure notes, etc.) in the Trash and also to be able to delete them from the Trash and restore them from the Trash to non-Trash locations.
• Return codes from op list items which give us information about how many items are found, so we don't have to parse error strings.
• The ability in the CLI to rename items (documents, secure notes, etc.)
• The ability in the CLI to update items (documents, secure notes, etc.)
• Some sort of op status command which will return information about whether the current session is valid ... via return codes, not simply by means of error strings which would then need to be parsed.

With these capabilities (especially the first one), the CLI wrappers will be a lot more efficient. And if the first four of these are implemented, the wrappers might not even be needed any more.

And as I mentioned elsewhere, I'm willing to sign a non-disclosure agreement and work on making these changes in a copy of the official code base. I'm learning go, and I'm sure I could implement these enhancements.

rickfillion

The ability in the CLI to query items (documents, secure notes, etc.) in the Trash

Does the --include-trash flag on op get item and op list items not do what you're looking for? Can you help me understand what you're trying to do?

to be able to delete them from the Trash

The next release should have a op delete trash command to empty the trash. Hopefully this will do the trick for you.

restore them from the Trash to non-Trash locations.

Interesting. Makes sense. Filed as issue 447 in our tracker so that we can consider that.

Return codes from op list items which give us information about how many items are found, so we don't have to parse error strings.

I think you might mean op get item here? Counting items from op list items should be pretty easy as it's an array of objects. Error codes in general need some rethinking in our tool though. We seem to want them to mean a few different things and there isn't enough consistency.

The ability in the CLI to [rename, update] items (documents, secure notes, etc.)

We'll get there. I suspect both renaming and general updating will be done via the same command (op edit item) when we roll that out.

Some sort of op status command

That'd be good, but it feels like a bit of a hack to me. I'd much rather we build something where you didn't need to worry about the status of your session. If the session dies for whatever reason the client should be able to re-negotiate a new one with the server. Something like eval $(op signin agilebits --longlived) and from then on you can assume your session is always available until you close that terminal. Otherwise you'd be stuck running op status between every command to possibly renegotiate a new session if needed (the server reserves the right to invalidate a session for any reason it sees fit).

Rick

HippoMan

Sorry for taking so long to reply. I've been busy at my new job.

First of all, THANK YOU (!!!) for op delete trash! This plus --include-trash will allow me to greatly simplify the python wrappers, and it makes the CLI much more usable.

Yes, I meant op get item. If that call fails, the only way we can know that the failure is due to there being multiple matched items is to parse the error messages. It's highly inefficient to always have to do op list items in conjunction with every op get item failure, just to be able to find out whether the failure was due to a multiple-item match. It would be cleaner and more efficient if we could know of this multiple-match case via a simple op get item return code.

Also, when using --include-trash, is it possible to distinguish between Trash items and other items? I seem to recall that all returned items have a trash or similar JSON attribute set to "N", even for items in the Trash. Perhaps I'm wrong about this, however (I can't test this at the moment, because I'm away from my desktop machine and posting via the mobile app).

Anyway, thank you for these enhancements, and especially for the upcoming op delete trash!

rickfillion

It would be cleaner and more efficient if we could know of this multiple-match case via a simple op get item return code

Yes. We need to get that error to you somehow.

Also, when using --include-trash, is it possible to distinguish between Trash items and other items? I seem to recall that all returned items have a trash or similar JSON attribute set to "N", even for items in the Trash.

I'm not quite sure I get what you're asking here. All items will have a trashed attribute which should be either Y or N depending on whether it's in the trash or not.

Anyway, thank you for these enhancements, and especially for the upcoming op delete trash!

You're welcome. Hopefully we can get that release out soon. We've been busy working on the SCIM bridge lately. We need more hours in a day!

Rick

HippoMan

I will double-check when I get home, but I seem to recall that trashed is always returned as "N" in all cases, even for items in the Trash. Or perhaps I'm thinking of an older CLI version. I'll report back.

AGAlumB

If so, that definitely sounds like a bug. Let us know what you find!

HippoMan

I have checked, and trashed now seems to indeed be set correctly. I believe I was thinking about an earlier CLI version, and I'm sorry for the false alarm.

AGAlumB

No worries! Thanks for checking. I don't recall an issue with that, but a lot has happened this past year too, so you may be right. :)