Geeky under-the-hood question: What is the export_keys table for?
So...I'm updating my notes for how 1Password works under the hood, for a "lunch and learn" talk here at my company... And I noticed a new table and data item that I didn't recognize. I assume it came in with 1Password7, but I'm not positive.
I see an "export_keys" table with two entries. Both have the same UUID, one claims to be encrypted by "mp" and one by "mk". (BTW -- it's confusing as all heck that sometimes it seems the "kid" element identifies the present key itself, and sometimes seems to identify the key that encrypts it) (or maybe not. enough to say that it just constantly messes with my head...).
Anyway...there are two entries. One of them (the "encrypted by mk" key) I can decrypt using the "mp" from the primary account's first keyset (that is, my primary account MUK decrypts enc_sym_key for the 1st keyset, the key within that is the "mp" key, and that decrypts this first "encrypted by mk" key in export_tables). This decrypted key then decrypts a new item in "accounts" (with my name, email, account name and server), and also seems to decrypt Watchtower data in ~/Libra
ry. (that data seemed to maybe be test data, as the hashes in there were things I don't use). (unless...can watchtower also search in shared vaults within the company? anyway, none showed up in the watchtower pane in the UI).
The other one I can't decrypt. It's also much more interesting, as it seems to provide different 2SKD parameters than what's used elsewhere. It's still PBES2g-HS256, but the iteration count is now 500,000 instead of 100k. Using that parameter to derive a different MUK using these parameters didn't let me decrypt this field, and also the previous one that I just decrypted didn't work either
So what's all this stuff for? I found an AccountExports folder in ~/Library with zip files, containing just that 2nd key info... so I'm guessing it's to support a future data export function?
Thanks for any insights you can give me!
david.
1Password Version: 7
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided
Comments
-
Good morning, @darthnull!
Awesome sleuthing job! You're absolutely right. That new table holds keys related to the encrypted account exports you're seeing.
Right now the Mac app is creating these automated exports/backups, but there's no import functionality available yet. It's on our todo list to create an opensource tool that will allow of decryption of these exports. It looks like you may beat us to that. :)
The key management for the exports is really fun. For kicks you should go through a Master Password change in 1Password, and you'll notice that you'll now have 3 rows in that table. Basically we create a chain of keys, and an MP change would cause a link to be added to the chain. As the new key gets added to the chain, all edits to items get saved with the new key.
The process was designed to be able to be safely run while 1Password is locked. It was a ton of fun to work with @jpgoldberg on its design.
The two keys you're seeing in the db are actually the same key, encrypted with different things (the MP, and the Primary vault's key). If you can decrypt one of them, you should then be able to use the resulting key to decrypt data in the encrypted export.
Rick
0 -
I'll have to try that MP change to see what happens...later.
I'm still not sure how the password-based key is encrypted. It looks like it's supposed to be the normal 2SKD, but with 500,000 iterations. But I'm not able to get it decrypted..so I was wondering if maybe some of the other data changed or if it's not really 2SKD (but instead something more akin to the windows EMK process).
Also, I don't see anything in the actual export .zip file -- other than the aforementioned PBES2g-HS256 key and the export_enc_details for my secondary account. I'm guessing it's not actually exporting anything yet...? (I'm definitely intrigued by your "safely run while 1Password is locked" comment, too -- anytime someone says that a design was fun, I'm intrigued... :)
david.
0 -
It looks like it's supposed to be the normal 2SKD, but with 500,000 iterations.
It's PBES2g-HS256 but there's no Secret Key component, so it's not 2SKD. It's a key derived from a single secret: the Master Password used to unlock the app. I'd have to go diving into the code to remember how that ends up changing things.
I'm guessing it's not actually exporting anything yet...?
It will only export vaults where you have the export permission. Is it possible that you don't have that for any vaults? You can check this by seeing if the File > Export menu item is enabled when specifically viewing that vault. If you have export permission and you're not seeing vault UUIDs in a dir structure in the export then you may actually be seeing a bug. The feature is still in development (there is no visible importer yet as you've [not] seen).
I'm definitely intrigued by your "safely run while 1Password is locked" comment, too
Automated backups/exports are tricky: when do you run them? If you've got a large database, it could take a fair amount of time. So when would it make sense to trigger it if the assumption is that you need to have the data unlocked/decryptable? There's no great answer there other than "shortly after unlock, I guess?" So that means periodically the user's going to get a slower app as we start doing a whole pile of work right after they unlock in hopes that we can complete that work before they lock the app again. Yuck.
Instead the goal was to be able to have it run while locked. This allows for it to run at any arbitrary time. We could trigger it at 2am every night when the user isn't using the app. Or we could specifically wait for them to lock them app (which generally means they're done using it) in order to have things run at a time that won't affect the user. That's useful for the user, but makes the process much more technically challenging.
With standalone vaults, our automated backup process is really simple: it just takes the whole database and associated files and zips that up. Restoration of that backup is a matter of unzipping that and putting the old database in place.
We wanted something much better with the backups of 1Password.com vaults. You shouldn't want to restore a whole database there because if you were to do that, next time you connect to the internet it'll want to update itself to whatever latest information is on the server. Odds are if you're needing to find something in a backup, this means you really don't want that happening.
The backup process in this case is actually split into two parts. As you interact with 1Password while unlocked, by say editing an item or downloading a new version from the server... 1Password will encrypt that item with its original vault key as it always has. Then it will encrypt another copy of that item with another key (the backup key you're trying to decrypt). If you look at the vault items table you should see two new columns that weren't there previously, one to store the encrypted overview and the other to store the encrypted details with the new key. The other half of the backup process is the part that runs while locked. It goes through each account/vault and can do a copy of the encrypted data out to the export file without needing to do any encryption. Both of these steps are relatively simple... where the fun/complexity came into play was when to generate and regenerate the export key. And how to build things such that when the export key changed we didn't need to re-encrypt all of the data again.
Once the feature is actually done I'd love for us to write something up about how it all works and provide as much information as humanly possible about the export format so that others can read it. The end goal is to help put you in control of your data.
Rick
0 -
I didn't even notice the new fields in items. I see some stuff there now, but not for all vaults -- haven't bothered to figure out which vaults did and which did not include export data. But nothing's getting saved to the .zip files. Again, though, unfinished feature, so not really a surprise that the zips are mostly empty.
And now that you've explained it, it makes sense. In some ways, it's almost a live-updated journal. "Export-on-Change" as it were. And it certainly makes decrypting the archive a lot easier than computing the MUK and going through all the various symmetric and asymmetric keys and stuff... (which I've already got working, just haven't had the time to clean it up or write a blog post explaining everything yet. some day. there's a distinct lack of tools to decrypt the newest vault formats on GitHub...)
I tried changing my master password and, yes, the old export key is there now, encrypted with the new export key, which is itself encrypted by both the new mk (MUK-derived), and also separately encrypted by the master-password-derived key we were just discussing. Which looks like it's using 'export@1password.com' as part of the input data. I needed to break out Hopper to get that email part figured out. But seems logical, esp. since the backup may contain data from multiple accounts with different email addresses. (I'd tried with blank emails, blank algorithms, algo in place of email, and all kinds of other permutations. This solution makes a lot more sense.)
Interestingly, the "old" export key (again, now encrypted with the new export key) still has the PBES2g parameters attached in the enc_sym_key structure, even though it's no longer encrypted by a derived key.
Anyway, thanks so much for explaining all this. It's fantastic how open Agile is with the system (and great that you'd take the time to help a rando snoop around a not-even-finished-feature).
BTW, what's the "g" in "PBES2g-HS256" mean? Is this an internal designation, just to distinguish it from "pure" PBES2-HS256? Was there an "a" through "f" which didn't cut it?
Maybe next month I'll ask about the opdata01 structures in .plist files I've found lying around... :)
david.
0 -
I hope you don't mind, I've moved this discussion over to the Lounge since its more of a technical discussion than most of the ones going on in the Mac forum. I think @jpgoldberg wanted to add something as well, but he might be drafting a larger response.
0
