OPVault implementation details

System

This discussion was created from comments split from: Timeline for Multiple Vaults on Android.

rnml

I'm writing a Linux command line tool for reading and manipulating opvault data. So far I've implemented the decryption part necessary for reading out all the encrypted JSON data.

My question is how to get from the internal json representation of one-time-password (a pair of base64 encoded blobs) to the time series of 6 digit one-time passwords

AGAlumB

@rnml: Unless I'm misunderstanding you, this seems like more of a TOTP question than an OPVault question, since the secret used to generate the code is just adhering to the TOTP specification (RFC-6238). After all, if you've already got decryption working, you're just looking at the stored data, and that's the algorithm being used, not something we invented. Does that help?

rnml

Thank you @brenty. That does help. My problem is that I didn't know to use TOTP.

More generally, is there any specification of the JSON data one gets after decrypting the OPVault?

More specifically, I'm still having trouble understanding how to apply RFC-6238 to the data I see. In particular, I see strings of the form TOTP_{16-octet-hexadecimal-string} in the decrypted vault.

In the terminology of RFC-6238, I expect one half of those 16 bytes (the first 8 bytes?) to be the "shared secret" K and the other half (the second 8 bytes?) to be the "initial counter time" T0. However, T0 is supposed to be a value in Unix time (seconds since epoch) and I assumed it should be a value in the past so that the computation of (current_time - T0) doesn't involve any questions of how to represent negative values. When interpreted as Unix times, neither the first nor last 8 byte sequences is a time in the past, whether I interpret them as big-endian or little-endian.

I feel I'm probably missing the bigger picture again.

Jasper

Hi @rnml,

TOTP_{16-octet-hexadecimal-string} is just an identifier/UUID for the field.

The actual data you need is in field's v value. It can be an otpauth:// URL, such as:

otpauth://totp/Dropbox:wendyappleseed@dropbox.com?secret=ZXBKPQIPPPFOZFKWXNWHXZ5ZAY&issuer=Dropbox

Or simply a TOTP secret:

ZXBKPQIPPPFOZFKWXNWHXZ5ZAY

You generate the one time password from that secret. I'm not sure which language you're using, but there are a number of TOTP generator libraries out there which you may be able to use, or at least look at for reference.

rnml

Ok, now I understand. My code finally works. Thanks for the help!

@JasperP, is there any reference online on how to interpret the JSON blobs that come out of OPVault? For example: What is the difference between k, n, v, and t fields when those come bundled together? I see some older entries with value, name, designation, and type all together -- is that related? What to "ps" and "ainfo" field mean in an overview? What are "backupKeys" and how does one decode them? There appear to be UUIDs all over the place -- what are those used for? How stable is this format? Is it constantly evolving, and if so, does old data get upgraded to the newer conventions automatically?

As you can see, I have lots of questions. I'd appreciate any pointers you can give me.

rickfillion

Hi @rnml,

Unfortunately we don't have public facing documentation for the JSON formats at that level. We have internal documentation that we use to make sure our clients are compatible with one another, and one day I'd love for us to clean that up do publish it publicly.

The format used within OPVault is considered stable in the sense that new clients should be able to read older data fine, and older clients shouldn't much issues with newer data we put in. We haven't had to modify OPVault much since its creation as it's fairly open ended.

To answer about the ones you've asked about...

• k : field type
• n : field identifier (often a uuid, but if it's something we need to fill it's typically a known string to us)
• t : field title
• v : field value
• ps : password strength (0-100). Newer items created in Mac/iOS will have pbe (bits entropy) and pbrng (true when generated from the random number generator).
• ainfo : additional info, typically used as a subtitle for the item in the list
• backupKeys : set of keys that could be used for the attachments. Decrypting of attachments can be ugly... they'll be encrypted with either the item key itself (the one stored in details.k), or one of the backup keys. I could write a loonnngggg post about all of that.

I think it's awesome that you're writing code that reads OPVault. But I would be careful about writing code that writes to OPVault unless you're only planning on using it personally. It's easy to create bad data.

Rick

rnml

Hi! My wife and I are happy 1Password Pro users. We keep our 1Password data synced between our phones using my Dropbox account. Neither of us use the Windows or Mac desktop clients for 1Password. 1Password has definitely increased the information security of our family, so thank you for that.

I'm a computer programmer by trade, and am currently writing a command line interface for my 1Password data for my own personal use in a Linux environment.

Could one of your developers please tell me what I need to know as a fellow developer in order to reimplement the same thing that has already been implemented in the Mac and Windows clients, namely adding and syncing additional vaults? The excellent opvault spec you've posted gives me enough information to create a new vault, but I'm not sure how to put it into my Dropbox in such a way that I can then get the iPhone app to recognize it and start syncing it as well as the default/primary vault.

As a test, I tried copying the sample "freddy" vault into my Dropbox in a new subdirectory alongside "default" in Apps/1Password/1Password.opvault/ and went through the iPhone app's Settings > Vaults > Sync Additional Vault dialog to start syncing this additional vault, but I get a "Vault Exists" error message since I'm already syncing with that Dropbox account at the same path.

Then I realized, these two vaults have different master passwords, so I'm not sure how this could have worked anyway with the "freddy" vault. (In a multiple vault setting, do all the vaults have to have the same master password? I would expect so given how the UI requests this once up front.)

Any guidance on this matter would be greatly appreciated. I could go ahead and create a new opvault and try again but I expect I have some wrong assumptions about how this ought to work, so that probably isn't as fruitful as just asking y'all.

rudy

@rnml,

They don't need to share the same password, when you initially load that vault in 1Password it will prompt you for the vault's password and then save access keys so that when you unlock 1Password using your master password it will unlock the secondary vault as well.

You actually want to put your freddy vault in a folder along side 1Password.opvault.

So you should have Apps/1Password/1Password.opvault and Apps/1Password/Freddy.opvault as your two vaults. The actual data files go inside of those opvault folders.

Rudy

rnml

Oh great! That worked! Thanks a bunch @rudy !

Does the profile name ever show up in the UI? Right now my primary and secondary vaults each have only a single profile named "default". Is that always the case, or are there ever multiple profiles in play for a single vault?

Also, where are the secondary vault "access keys" stored between runs of the app? Do they go into the primary vault? If so, by syncing with a secondary vault have I implicitly shared its keys with anyone who also syncs with my primary vault?

AGAlumB

@rnml: I hope you don't mind, but I've split off your comments from the other thread and merged them into a single discussion on your project. :pirate:

Does the profile name ever show up in the UI? Right now my primary and secondary vaults each have only a single profile named "default". Is that always the case, or are there ever multiple profiles in play for a single vault?

The "profile name" is shown in the apps in some contexts, primarily the desktop versions. But it's not something we've ever emphasized. The filenames are more often used, since they're needed to track down the vault in Dropbox for syncing, for example.

Also, where are the secondary vault "access keys" stored between runs of the app? Do they go into the primary vault? If so, by syncing with a secondary vault have I implicitly shared its keys with anyone who also syncs with my primary vault?

Wow! Great questions. I've asked Rick to jump in here since I think he'll do a better job of explaining how this works.

rickfillion

Could one of your developers please tell me what I need to know as a fellow developer in order to reimplement the same thing that has already been implemented in the Mac and Windows clients, namely adding and syncing additional vaults?

Unfortunately we can't do that. Simply because it's a massive amount of code and we'd be sitting here forever. Doing multiple vault sync requires a unified data storage beyond the sync vaults... which is why we use sqlite to back the apps themselves. Then there's the fact that each vault has its own encryption key and password, and you want to be able to unlock the app with a single password as opposed to unlocking each individual vault so then that means you need to build a bit of a tree of keys. You may want to read a blog post I wrote a couple years ago about Master Password sync as it explains some of the concepts that are needed for this:
https://blog.agilebits.com/2015/04/28/how-1password-syncs-changes-to-your-master-password/

Also, where are the secondary vault "access keys" stored between runs of the app?

You should get your hands on a sqlite file produced by our Mac and iOS apps. It has a table called profiles which really means vaults, and this table has a column called attributes_data. This column contains basically all of the vault attributes, encrypted with the primary vault's encryption key. This data never gets synced anywhere and is used solely for the purposes of creating a hierarchy that enables unlocking the app with a single password.

You've noticed that your vaults are all named default. default is the name the apps give the first vault created in the app, and the apps know to translate it to "Primary" whenever they see it. Unfortunately we've never gotten around to building the ability to sync vault names so we only import them from the sync vault when initially creating a vault based on a sync vault. Same with icons and colors. The data format is missing a couple attributes that would be needed to actually sync that. It's one of those details that I'd love for us to get right.

I hope this helps.

Rick

rnml

@brenty, no I don't mind at all that you split off this discussion. It saves me having to provide context again and again. Thank you.

AGAlumB

No problem. :) :+1:

rnml

Thanks for all the useful information. It's starting to take shape in
my mind but I'm still absorbing it ...

I do have some questions though.

I hear the term "vault" used to mean two different things that
seem orthogonal. One meaning is tied to the location where the
vault is stored (i.e. local vault versus sync vault) and the other is
tied to the organization of items stored therein (primary vault versus
other vaults created via the Mac/Windows client).

Is there a more precise terminology that people use to distinguish
between these two concepts? Does the OPVault notion of "profile"
correspond to the equivalence-class-of-items notion of a vault?

@rickfillion you mentioned that local vaults contain a "tree of keys"
and a "hierarchy of vault attributes". Those sound like two descriptions of
the same thing, perhaps. Can you say more about the relationship between a
parent and child in this tree?

rickfillion

The two "vaults" you describe may seem different to you, but they're actually the same.

An AgileKeychain is a vault. An OPVault is a vault. A "profile" in the sqlite database is a vault. They get setup initially then sync keeps them the same. Outside of the syncer code, 1Password for Mac doesn't know what an AgileKeychain or OPVault is. It just knows about the ones in its sqlite database, and it knows when to fire up the syncer which knows how to read every format and move data back/forth and merge changes. At its most basic, a vault consists of an encryption key. That encryption key is most commonly used to encrypt items which we mark as being related to the vault (either by foreign key in a database, or by storing the files in the right locations for AgileKeychain/OPVault).

The tree of keys concept is due to being able to unlock with a single password. If every vault has its own password which is used to derive a key which is used to encrypt the encryption key (that's a mouth-full!)... you should need to know and enter all passwords to derive all keys to decrypt all keys. No one wants to do that.

So if you have:

Primary Vault : Password1 : EncryptionKey1 (stored as encrypt(EncryptionKey1, deriveKey(Password1)) )
Secondary Vault1 : Password2 : EncryptionKey2 (stored as encrypt(EncryptionKey2, deriveKey(Password2)) )
Secondary Vault2 : Password3 : EncryptionKey3 (stored as encrypt(EncryptionKey3, deriveKey(Password3)) )

Everything is independent and you'd need Password1, Password2, Password3 to access EncryptionKey1, EncryptionKey2, EncryptionKey3.

What I described earlier is a bit of an implementation detail and it's easier to think of it at a slightly higher level. We store what you can think of as this:

Primary Vault : Password1 : EncryptionKey1 (stored as encrypt(EncryptionKey1, deriveKey(Password1)) )
Secondary Vault1 : Password2 : EncryptionKey2 (stored as encrypt(EncryptionKey2, deriveKey(Password2)) ) : EncryptionKey2 (stored as encrypt(EncryptionKey2, EncryptionKey1) )
Secondary Vault2 : Password3 : EncryptionKey3 (stored as encrypt(EncryptionKey3, deriveKey(Password3)) ) : EncryptionKey3 (stored as encrypt(EncryptionKey3, EncryptionKey1) )

From Password1 you can deriveKey(Password1) which will allow you to decrypt EncryptionKey1. Now that we have EncryptionKey1, we can decrypt EncryptionKey2 and 3 from the alternate storage where we've encrypted it with EncryptionKey1.

So in this case the tree would have EncryptionKey1 as the root node, and two leaf nodes of EncryptionKey2 and EncryptionKey3.

It's not a very deep tree and when you're talking local vaults only it's never deeper than that. But when you get 1Password.com accounts it can get arbitrarily deep.

I hope this actually clears things up and doesn't just muddle everything around. :)

Rick

rnml

Thanks for taking the time to explain all this @rickfillion

Is the SQLite database considered a vault? It sounds like it isn't given that it contains multiple vaults. But then again it sounds like profile is a synonym for vault and an OPVault can contain multiple profiles. So is a multi-profile OPVault considered a single vault?

I'm not a 1Password account holder, so this is mostly just curiosity, but how does the key/vault tree get deeper than 2 levels? What does this mean in the user interface? Do users ever think of this tree?

rickfillion

Is the SQLite database considered a vault?

No. It's just a database into which we store vaults.

But then again it sounds like profile is a synonym for vault

Yes. Naming is hard. :)

an OPVault can contain multiple profiles.

Not really. In theory a single OPVault file could possibly contain multiple vaults/profiles, but in reality that's not the case.

So is a multi-profile OPVault considered a single vault?

Just to make sure that I understand what you mean by this... can you be more specific? Do you mean an opvault folder that contains more than the 'default' folder? If so I don't think any of our apps actually read the other folders.

I'm not a 1Password account holder, so this is mostly just curiosity, but how does the key/vault tree get deeper than 2 levels? What does this mean in the user interface? Do users ever think of this tree?

1Password.com accounts introduce two new concepts here : user keys (every user has a public/private key as well as a symmetric key), and group keys (same). The tree would consist of User -> [Groups | Vaults], and Groups can point to [Groups | Vaults]. The user never has to think about this and it doesn't show up anywhere in the client app's UI. But this is how access control is done in 1Password Teams.. you give a group access to a vault by encrypting the vault key with the group's public key. Then give a user access to a group by encrypting the group keys with the user's public key. Technically groups can have access to groups, so it gets arbitrarily deep.

It's fun stuff. :)

Rick

rnml

Ok, it sounds like the OPVault spec was slightly over-engineered in the sense that it allows for profiles other than "default", but this never really panned out. Whatever. It happens. I think I now get what a vault is and isn't. Thanks.

As for the key tree, if user U is a member of group A and A is itself a "member" of another group B, then somewhere KeyB is stored encrypted with KeyA and somewhere KeyA is stored encrypted with KeyU. How do these parent-child linkages make it into my sqlite DB? Presumably someone besides me could have been the one that set up these U->A->B relationships. So how did I end up with these key-encrypted-keys? Did they come to me by way of some vault? Or maybe this is why 1P account holders also end up subscribed to 1P's in-house cloud storage service -- that way you have an additional channel for syncing not just vault data but also these sorts of permissions relationships.

rickfillion

@rnml : I haven't read the spec for OPVault in a long time, I didn't even think of that. It's easier for me to look at the code, so I just always jump to that.

As for the key tree, if user U is a member of group A and A is itself a "member" of another group B, then somewhere KeyB is stored encrypted with KeyA and somewhere KeyA is stored encrypted with KeyU.

Exactly right.

How do these parent-child linkages make it into my sqlite DB?

This lives in the other database, the one that's specific to 1Password accounts : B5.sqlite. The table in question here would be the keysets table along with vault_access. Vault keys are stored, encrypted, in the vault_access table... encrypted by a key that resides, encrypted, in the keysets table. Those keys are encrypted either by another key in keysets, or by what we call the Master Unlock Key for the account (the key derived from the password of the account).

So how did I end up with these key-encrypted-keys?

By someone (maybe yourself) configuring it via the 1Password.com webapp. This stuff is all based on public/private key encryption, and every 1Password.com user (and group) has a public key. Which means I can give you access to something by encrypting that something's key with your public key.

Did they come to me by way of some vault?

No... this all resides outside of vaults, and needs to do so because of the fact that it's what gives you access to the vaults.

Or maybe this is why 1P account holders also end up subscribed to 1P's in-house cloud storage service -- that way you have an additional channel for syncing not just vault data but also these sorts of permissions relationships.

Exactly right. Everything was redesigned for 1Password.com. We even gave it its own database locally, with a different schema (at the items level it's pretty similar to OPVault) to support everything we need. Sync was redesigned from scratch, and for the first time ever we now have a server component that can help coordinate sync across all of the clients. This greatly simplified things. In a lot of ways up until now we've just been laying down the groundwork. We have some features we want to roll out that would have been near impossible to implement without the centralized service.

The fact that 1Password.com and OPVault share an item structure is by design. For one... we're still really happy with the flexibility that the OPVault structure provides. It's rather open-ended compared to AgileKeychain. But for another... we're not done adding features to standalone vaults. Having the same structure makes that a lot easier.

Have yourself a great weekend.

Rick

rnml

Hello again. I'm happy to report my tool now supports creation of new opvaults. I'm just plopping them into Dropbox and syncing them with my iPhone for now. I don't do anything like the SQLite database you guys use. I'm just manually adding the secondary vault master passwords to my primary vault.

My setup is pretty simple: just my wife and I sharing passwords with each other. The reason I wanted multiple vaults is so that we could each have one for our own work related items that don't matter to the other.

Now that I'm writing as well as reading opvault data, it would be helpful to know more about the synchronization logic -- the Sync actor @rickfillion mentioned. How does it reconcile conflicts?

There's a backstory to my question. I'm now saving past states of my vaults in a hg repo as a simple backup scheme. This morning I had reason to try to revert my primary vault back to an earlier state, so I overwrote the appropriate Dropbox subdirectory with the backup vault I had saved away. However, the changes didn't make it over to my iPhone app, and instead I ended up with "conflict" files in Dropbox. I was able to get back into the right state by decrypting my backup vault and manually copying over the changes into the iPhone app, but that process would have been much more painful if I had to do it for more than a handful of items.

So my real question is how can I write a backup copy of a vault into Dropbox in such a way that the reconciliation logic in the Sync-er always "rules in favor" of my backup vault's view of things. My guess is that I should freshen some of the time stamps (but which ones?) before overwriting the Dropbox subdirectory to indicate my desire for the old item states to once again be considered the most up to date item states. Also I wonder if I could help things along by somehow modifying the lastUpdatedBy field in profile.js.

Thanks in advance for any light you can shed on how to solve this problem. Also thanks for 1Password and for publishing the details of the OPVault format! It has been very satisfying to be able to program additional functionality for myself. I find myself becoming a more committed 1Password user with every line of code I write!

rickfillion

@rnml In the event that you have to recover from backup, your best bet is to disable sync on all the devices, roll back the changes to the opvault, remove the local vaults entirely on the devices, then set them up again. 1Password tries its darned best to never drop data and it'll try to merge the records in a bunch of cases (which isn't what you want in this case).

To try to answer your question for the fun of it... the items in opvault have a "txTimestamp" value. The app has a table where it stores the local txTimestamp (unix timestamp of modification time of item) and the txTimestamp that it last saw in the OPVault file at the last sync. With this data it can know if there are changes locally (item.txTimestamp != syncStateForItem.localTxTimestamp) or if there are changes remotely (opvaultItem.txTimestamp != syncStateForItem.sourceTxTimestamp) or both. If we have local changes, we push those up. If there are remote changes we pull those in. If there are both then we pull in and merge the item.

The app also knows how to clean up the dropbox conflict files by merging the records from the original file and the conflict file then removing the conflict file.

Cheers.

Rick

rnml

Got it. it seems like disconnecting and wiping the iPhone is the way to go.

rnml

I have a question about the hmac field in encrypted opvault items.

https://support.1password.com/opvault-design/#hmac

My uncertain understanding of the pseudo code listed there is that the hmac is the running hash

hmac.add(field_name_1)
hmac.add(field_value_1)
hmac.add(field_name_2)
hmac.add(field_value_2)
...

where the fields are visited in sorted order of the field names. My question is what to do when some field value isn't a string, for example the three integer valued timestamp fields. The pseudocode seems to say that it isn't the value but the value "description" that gets hashed. So what does that mean?

For example, if an encrypted item contains the entry

"updated":1488466140

what is the "description" of that timestamp? Is it the string "1488466140", perhaps?

rnml

@rickfillion does this question make sense? I'm blocked on adding additional vault-changing functionality until I work out this one detail.

To be extremely concrete, what sequence of strings should I feed the HMAC computation for the following item that I pulled from a dummy vault I created for testing.

{"C76F71B43D2146F3A85319526B1D0EA4":{"uuid":"C76F71B43D2146F3A85319526B1D0EA4","category":"001","o":"b3BkYXRhMDGMAAAAAAAAAJ2qw+O6s9uNh2M9kbINx\/fsZlDTgvm4ihi++em0J3m6FvD59JvKNXwZB0i+TGVz5PhIMMFkHvqNdTcRg\/xrgvGAyPeh5CuryyzGF9PSdsOjRp1EniN0Lvw1XdPCLqsRp0PV7A1StCGRKKxHMPdM4jH4srdyY3nzubWTFwZX9KtlqvfCusdRETlOU5aRGLvuWMXUN07jaTWx9eTy3CH3jwBsz0Un4+OTZuu1oSf0AytWLfDMgV8qgUJUa44W0O\/W0A==","hmac":"Pj2ReMSzhxbkDGEYvGB1Df8duiNBmfuK45Lcyn60xGM=","updated":1494186682,"k":"QIrU4caemsvK3OjVmypbQKSqPj9JfK7Cr3qseA7tcAYVOWMBeaBe1fi7D5mCDoVhUcz3HhcqFM7sd9pLFIVD1w467FjLBxS2ZLvBzhXaON4ITNVHD4FINy4XGFm7QBDdddyCgGOnzUY28hhrDjohEA==","d":"b3BkYXRhMDH+AQAAAAAAAPHXaZ6v+mpA2DtQWOtVyWBYnk1T3VN5V0\/ydbEFGrHSyU3cGCfLs0\/Q8+KhVZikwy843n6NOYJyPWL2gLHVcw2c+LrQUEmw3LyetMDfsYoSfKM95y6YoR8Nb\/WF6KxHKwcuT8PnBHQ36n0xijbgDVkHk1SYYhKWPmWDZu1HXdCpA3NlJef4QEwaNTioMick6+8e32CFTNWNpOHC8MW4s5Uuu5GDDL55S6sBZaXUdA3GDAW3NYbqujRRm8LXKLIlEueZJFXoxbNkVGg0C68ejifbifl04neN0RQIMrxPXibPKamAd1NV3ukxo42t9Tm\/EFSRaM9y4ZJHh0VZ2cH5nM6E9e40kI7XqSKmtp+z6PnoJGknGpUl5tPZaLl\/23Y8aGTL77kDnzoRrcPiZxgsEc\/VE1MbP9dWvYYFnlyQUUgTr\/61oDLrL22WgfL7TJCFlFf8N0xtE+PZMh59wskwQ8aS2Ex2FbuxBNYtvN6rZYWonKhz0tHONN2EuHj230ULFwHcipaEQYC2jLDaSHkIFpakQ0+2biAPG8RWIZx4zM\/2i6\/SOCkgxiiBFNfuJpryd3PbdDvzIjGmdf\/uRnCYGxrmr62LzfgRAOAziSmt7th6TM0gL9FFvuQMTo18zVFCj3o8NTD5PWUP0uyFhsr40uzASvoM00mJz5wzOoMA8ix53PgRp1qcBNoiEsvEat1s8sMr9Clra0LP3kXyIQGTwmSgUJ1h56zfiT1jziyORCb1","created":1494186648,"tx":1494186682}}

rickfillion

@rnml : That question makes perfect sense, yes. I'm very familiar with this, as I filed OPM-3200 (not yet fixed) against this implementation a while ago.

In the general JSON sense, the value could be of type string, array, dictionary, number, etc.. That would cause a problem because something like "dictionary" has many string representations that could come out so it would need be be more formally defined. This is why I filed OPM-3200... we need to further define what should happen here.

Luckily though (and the reason it's still not improved) is that in reality there should only be two types for values here: strings and numbers. The strings should be used as is (in UTF-8), and the numbers simply get translated into string versions of their base 10 representation. (e.g. 1494186648 -> "1494186648")

Hope this helps.

Rick

rnml

@rickfillion , I'm almost there! I had a couple small bugs that were feeding subtly wrong strings into the hmac. ugh.

The only items my code breaks on now are trashed ones. Are there any special rules for dealing with items containing trashed:true? I've tried everything I can think of but I always get the wrong hmac. My first thought was to serialize true as "true". Does the "trashed" entry get sorted in some non-standard way -- placed somewhere else in the sequence other than where it would appear if only sorting by field names?

Nathan

rickfillion

@rnml : boolean fields get encoded as decimal numbers still... so it'd be "1" there.

Rick

rnml

Ah, I should have known. :)

Thanks, @rickfillion

rickfillion

You're welcome.

Rick

rnml

Hooray, now I can produce the hmac of an item!