Framework/tool to simulate `op` in automated testing

zcutlip

Hello,

I've been working on a project to compliment the op CLI tool that I'm pretty excited about. I wanted to share, in hopes it would be useful to others.

Here's a link: https://github.com/zcutlip/mock-op

It's a project called mock-op that's for simulating the op command in automated testing. If you're using op in a scripted environment, and you want to do any sort of automated regression testing etc., you probably want to do so without interacting with an actual 1Password cloud account. mock-op does that by using your command line arguments to look up and play back recorded responses and exit statuses.

It has several parts that may be useful depending on your situation.

First off is the mock-op CLI tool. Provided a directory of responses, mock-op will simulate several different types of op queries. Here are a couple of examples:

Getting item "Example Login 1" from vault "Test Data":

$ mock-op get item "Example Login 1" --vault "Test Data"
{"uuid":"nnotgv5xwrhjbdj6bt3rugrijy","templateUuid":"001","trashed":"N","createdAt":"2020-12-04T00:50:48Z","updatedAt":"2020-12-04T01:21:24Z","changerUuid":"RAXCWKNRRNGL7I3KSZOH5ERLHI","itemVersion":2,"vaultUuid":"yhdg6ovhkjcfhn3u25cp2bnl6e","details":{"fields":[{"designation":"username","name":"username","type":"T","value":"johndoe1999"},{"designation":"password","name":"password","type":"P","value":"W9bZ@ZwGpRXCqnWt"}],"notesPlain":"","passwordHistory":[],"sections":[{"name":"linked items","title":"Related Items"}]},"overview":{"URLs":[{"l":"website","u":"https://example.cheeseburger/login.php"}],"ainfo":"johndoe1999","pbe":94.353515625,"pgrng":true,"ps":100,"title":"Example Login 1","url":"https://example.cheeseburger/login.php"}}

Above we see the JSON response to a successful query written to standard out.

Getting non-existant item "Invalid Item"

$ mock-op get item "Invalid Item"
[ERROR] 2021/02/05 14:56:31 "Invalid Item" doesn't seem to be an item. Specify the item with its UUID, name, or domain.
$ echo $?
1

Above we see an error being logged to standard error with an exit status of 1, in response to an invalid query.

The mock-op command just needs a directory of response contexts, consisting of a JSON dictionary, and a set of standard out/standard error response files.

Here's an example of the response-directory JSON:

{
  "meta": {
    "response_dir": "tests/config/mock-op/responses"
  },
  "commands": {
    "get|item|Example Login 1|--vault|Test Data": {
      "exit_status": 0,
      "stdout": "output",
      "stderr": "error_output",
      "name": "get-item-example-login-1-vault-test-data"
    },
    "get|item|Invalid Item": {
      "exit_status": 1,
      "stdout": "output",
      "stderr": "error_output",
      "name": "get-item-invalid-item"
    }
  }
}

And then here's the corresponding directory tree containing the responses:

$ tree responses
responses
├── get-item-by-uuid-example-login-2
│   ├── error_output
│   └── output
├── get-item-example-login-1-vault-test-data
│   ├── error_output
│   └── output
├── get-item-example-login-vault-archive
│   ├── error_output
│   └── output
└── get-item-invalid-item
    ├── error_output
    └── output

4 directories, 8 files

Response Generation

I designed the file & directory structure to be fairly straightforward so that one could create it by hand or easily script it. However, mock-op comes with a tool to generate responses. You provide it a configuration file, and it'll sign in to your 1Password account (using the real op tool), perform the queries, and record the responses.

Note: response generation requires you install my pyonepassword Python package. It can be found in PyPI and installed via pip.

Here's an example configuraiton file for generating responses:

[DEFAULT]
config-path = ./tests/config/mock-op
response-path = responses
response_dir_file = response-directory.json

[get-item-example-login-1-vault-test-data]
type=get-item
item_identifier = Example Login 1
vault = Test Data

[get-item-invalid-item]
type = get-item
item_identifier = Invalid Item

Then you can run response-generator and have it create your response directory:

$ response-generator ./response-generation.cfg
1Password master password:

Using account shorthand found in op config: my_onepassword_login
Doing normal (non-initial) 1Password sign-in

Currently the mock-op command only supports the op's get command and the item and document subcommands. But more are coming.

Simulating sign-in

It also can partially simulate successful and unsuccessful signing in. It cannot simulate initial signin, however. You can set an environment variable to inform mock-op whether to succeed or fail on the signin command:

Set MOCK_OP_SIGNIN_SUCCEED=1 to tell mock-op to simulate success. Note you can use the --raw option as well:

$ export MOCK_OP_SIGNIN_SUCCEED=1
$ mock-op signin no_such_user
export OP_SESSION_no_such_user="Ch2X7IOmPTQDYpUb9DhL7p4krRZPYd8taSmW8YuhAJY"
# This command is meant to be used with your shell's eval function.
# Run 'eval $(op signin no_such_user)' to sign in to your 1Password account.
# Use the --raw flag to only output the session token.
$ mock-op signin no_such_user --raw
zPL4YT8IU9WGfdl27QscoUbae5XFuKn5SwyVY9YdhZn

Also note that mock-op generates a unique (fake!) token each time.

To simulate signin-failure, set MOCK_OP_SIGNIN_SUCCEED=0

$ export MOCK_OP_SIGNIN_SUCCEED=0
$ mock-op signin no_such_user
[ERROR] 2021/02/19 17:03:57 Authentication: DB: 401: Unauthorized

API

If the mock-op and the response-generator utilities are too limiting, there is also API for each.

For response generation, there is the OPResponseGenerator class.

For response simulation, there is the MockOP class which can be extended to respond understand arbitrary command line arguments.

I won't go into use APIs here. The mock-op project has a detailed readme along with examples.

Notes:
1. As OPResponseGenerator uses pyonepassword to programmatically query your 1Password account, it can only generate responses to queries thatpyonepassword knows how to perform. If you need other queries, it is recommended to script the response generation on your own.
2. Even by extending MockOP to create a custom op simulator, only read commands are supported. This is because the underlying framework doesn't support saving/modifying state across command invocations

You can see it in use in my own pyonepassword project's tests (which I've just started adding):
https://github.com/zcutlip/pyonepassword

$ (pyonepassword) pytest -s
=========================== test session starts ===========================
platform darwin -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/zach/Sync/Projects/src/py-onepassword
collected 4 items

tests/test_get_item.py Doing normal (non-initial) 1Password sign-in
.Doing normal (non-initial) 1Password sign-in
.Doing normal (non-initial) 1Password sign-in
.Doing normal (non-initial) 1Password sign-in
1Password 'get item' failed.
.

============================ 4 passed in 0.71s ============================

Anyway, I hope this is useful to someone!

Cheers,
Zach

---

1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

ag_yaron

Hey @zcutlip !
Wow, that is super. Thank you very much for sharing! Hopefully this will indeed be useful to other users. :+1: :+1:

zcutlip

Thank you very much for sharing!

Yeah you bet! I do hope it's useful. I know it's fairly limited right now, but as I grow pyonepassword, I plan to grow the testing tool along with it

ag_yaron

Sounds awesome.
Looking forward to see your project grow :)