Opvault vault and 1Master password syncing technical details

Trishaelwood

Hi Sir,

Am trying to study 1password for its security and had some doubts for opvault design and key derivation as well as master password sync.

1. The derived key which we get from the master password has the key size of 64bytes? 256bit for encryption key and 256 for mac
2. When you unlock 1password do you keep the derived key in memory to decrypt master key and overview key? Or do you decrypt master and overview key once 1password unlocks and keep them in memory for further use?
3. The salt that is contained in the profile.js is that the only salt 1password uses through out the application? i.e say to get the derived key form master password we have to use the iterations and salt mentioned in the profile.js file. Now to decrypt the masterkey and overview key which are kind of opdata object what is the encryption salt and hmac salt used? can you explain in bit more detail how can we decrypt master and overview key with the help of derived key?

Had some doubts on master password sync with regards to below link
https://blog.agilebits.com/2015/04/28/how-1password-syncs-changes-to-your-master-password/

Say I have 2 devices d1 (MasterPassword : A ) and d2(MasterPassword : B ) both having their own vault and AES key. Now say i start sync process from d1, here the sync vault would have a new AES key with d1 master password right. Now say it syncs to device D2 so we decrypt the sync AES key with D1 password and encrypt it again with D2 AES key.

1. When we sync new data from D2 to sync vault, here does it get decrypted first with d1 AES key and re-encrypted with sync vault AES key?
   i.e How do you sync this with iCloudkit ? is the process same or has it changed ?
2. Its mentioned that sync vault AES is encrypted with local vault AES key then when D2 master password changes only thing changing will be the encryption key of local vault AES key and not the key itself, why do we decrypt and re-encrypt sync vault AES key with master password?

and if we decrypt and re-encrypt sync AES key then

1. Dn't you have to maintain when to unlock sync AES key with master password and when to unlock it with AES key?
2. When the changed sync vault AES(encryption changes only) key propagates to another devices how is it possible that the sync process does not get affected. bcoz now the other device would not be able to decrypt the sync vault AES key with its own local AES key.

rickfillion

Hi @Trishaelwood,

Thanks for writing in, and being taking the time to look into this stuff. Let me see if I can clarify things for you.

First your questions...

The derived key which we get from the master password has the key size of 64bytes? 256bit for encryption key and 256 for mac

Correct.

When you unlock 1password do you keep the derived key in memory to decrypt master key and overview key? Or do you decrypt master and overview key once 1password unlocks and keep them in memory for further use?

The derived key is not stored beyond the unlock process. The process looks like:

• Create derived key based on the password, salt, and iterations.
• Use derived key to attempt to decrypt master key. If it fails, unlock fails.
• Use derived key to attempt to decrypt overview key. If it fails, unlock fails.
• Unlock succeeds.

The salt that is contained in the profile.js is that the only salt 1password uses through out the application? i.e say to get the derived key form master password we have to use the iterations and salt mentioned in the profile.js file. Now to decrypt the masterkey and overview key which are kind of opdata object what is the encryption salt and hmac salt used? can you explain in bit more detail how can we decrypt master and overview key with the help of derived key?

I think what you're looking for is here the initialization vectors for decryption. You can read up on the format of opdata and how its header contains the initialization vector needed to decrypt the piece of data on our OPVault Design page.

Now lets talk about password changes and sync. I think you might have a misunderstanding about how sync encryption works. Let's see if I can correct that.

You have devices 1 and 2 (d1, d2), and they're syncing via OPVault (opv). Let's go over what each has:

Device 1 (d1) has:

• password (d1-p) : A
• derived key (d1-dk)
• master key (d1-mk) encrypted with d1-dk
• overview key (d1-ok) encrypted with d1-dk
• Items, encrypted with d1-mk/d1-ok
• Copy OPVault's (opv) master key, and overview key, encrypted with d1-mk
• Copy of d1-mk + d1-ok, encrypted with opv-mk

Device 2 (d2) has:

• password (d2-p) : B
• derived key (d2-dk)
• master key (d2-mk) encrypted with d2-dk
• overview key (d2-ok) encrypted with d2-dk
• Items, encrypted with d2-mk/d2-ok
• Copy of OPVault's (opv) master key, and overview key, encrypted with d1-mk
• Copy of d2-mk + d2-ok, encrypted with opv-mk

OPVault (opv) has:

• password (opv-p) : A
• derived key (opv-dk)
• master key (opv-mk), encrypted with opv-dk
• overview key (opv-ok), encrypted with opv-dk
• Items, encrypted with opv-mk/opv-ok

This means that on Device 1, you have two avenues to unlocking the kingdom : through d1-p, or through opv-p. The same is true for Device 2 through d2-p or opv-p.

When we sync new data from D2 to sync vault, here does it get decrypted first with d1 AES key and re-encrypted with sync vault AES key? i.e How do you sync this with iCloudkit ? is the process same or has it changed ?

Correct. Items get decrypted then re-encrypted before being stored into the OPVault. The same is true for CloudKit sync. CloudKit sync can really be thought of as OPVault without band files where the storage is CloudKit.

Its mentioned that sync vault AES is encrypted with local vault AES key then when D2 master password changes only thing changing will be the encryption key of local vault AES key and not the key itself, why do we decrypt and re-encrypt sync vault AES key with master password?

Let's take a look at a concrete example with the above data. d1 and opv have the same master password: 'A'. And d2 has a master password of 'B'. Now if on d2 you were to use the 'A' master password, here's what would happen:

• Compute a possible d2-p based on 'A' and the salt/iterations of d2
• Attempt to use d2-p to decrypt d2-mk. It fails. So we know that this is not the local vault's Master Password.
• Look to see if you're syncing, see that you're seeing with OPVault opv, so we'll try with that.
• Compute opv-p based on 'A' and the salt/iterations of opv
• Attempt to use d2-p to decrypt opv-mk. It succeeds. So now we know that this is the OPVault's Master Password, and from this we're going to infer that the user wants to switch their local Master Password to match what's in the OPVault.
• Use opv-mk to decrypt d2-mk + d2-ok, which unlocks the local vault
• Create d2-p based on 'B' and the salt/iterations of d2
• Use d2-p to encrypt d2-mk + d2-ok, and write those to disk, which effectively changes the local vault's Master Password to 'A'

When the changed sync vault AES(encryption changes only) key propagates to another devices how is it possible that the sync process does not get affected. bcoz now the other device would not be able to decrypt the sync vault AES key with its own local AES key.

The keys themselves never get changed, only encrypted forms of keys. A vault's key in this format is static.

The sync process never gets affected because each device keeps a copy of the opv's master and overview keys, encrypted with its own master key. Which means that even though we can't derive the correct derived key for the opv, we always maintain access to the data inside of it because we can skip that step.

I hope this answers all of your questions. We try to be as open as possible about how all of this works. Let me know if you have any other questions.

Rick

Trishaelwood

Hi thanks for the information rick. To summerise the encryption process here is what happens can you please correct if anything is wrong

Opdata Format

8 - Opdata01
8 - length of plaintext Uint
16 - IV

Next is padding as explained in opvault design overview

Now the whole thing is encrypted with AES-256 CBC mode. On resulted cipher text SHA-512 is applied and the MAC is appended to the cipher text. So considered the last 32 bytes are the MAC.

Information Given
Profile.js
Masterkey - base64 encoded
OverView key - base64 encoded
Iterations for pbkdf2 - which are not less the 10k on any OS
salt - this must be used to get the derived key from master password

MasterPassword to Derived Key
When user Enters master password(MP) it is converted to UTF8 null terminated string and a derived key is obtained using

CCKeyDerivationPBKDF( Pbkdf2, MP, MP.length,
salt, salt.length,
HMAC-SHA512, Iterations,
uint8_t *derivedKey, 64)

Where salt and iterations are given in profile.js

Derived Key to MasterKey

The 64 byte derived key is then divided into 32bytes encryption key and 32 bytes MAC key.

Derived To Mac
1. The overview key and master key are base64 encoded so first we need to decode them. Then extract the mac i.e last 32 bytes for (256 bit mac) let's call it overviewMac. The remaining part is the cipherText so we need to apply HMAC-SHA-512 on it with derived MAC key the result must be compared with overviewMAC.

but here when we convert overviewmac to plaintext it always provides nil. i.e the base64string is not getting converted to plaintext. Are we missing something over here.

Also, can you explain the decryption portion in bit more details. From derived to item decryption. I did read the opvault design overview but am having few doubts.

jpgoldberg

1. The overview key and master key are base64 encoded so first we need to decode them.

Correct. This should give you a much of binary data.

Then extract the mac i.e last 32 bytes for (256 bit mac) let's call it overviewMac.

Let me double check that we use the full 32 bytes for the MAC. Yes.

The remaining part is the cipherText so we need to apply HMAC-SHA-512 on it with derived MAC key the result must be compared with overviewMAC. [emphasis added]

No. This should be HMAC-SHA-256 here.

but here when we convert overviewmac to plaintext it always provides nil. i.e the base64string is not getting converted to plaintext. Are we missing something over here.

I'm not entirely sure what you mean by converting the MAC to plaintext. That MAC is always an array¹ 32 random-ish bytes. It doesn't really have either a "plaintext" (because it is not ciphertext) nor a "plain text" (because it is not something that makes sense in ASCII) counterpart.

I hope that this is of some help. I may have misunderstood where you got stuck so I may have answered the wrong question. Please let me know.

---

1. Or however you are representing these sorts of things in what you are doing. In Objective-C, we are using NSData. ↩︎

Trishaelwood

Thanks jpgoldberg. After applying HMAC-SHA-256 the mac derived from overviewData and the mac separated from overview matches. On further studying had confusion on 2 places.

Convension
1. overviewKeyBase64 = Given Data from profile.js
2. overviewKeyDecoded = Base 64 decoded data from overviewKeyBase64
3. overviewkey (Kind of Opdata) = overviewKeyDecoded.length-32
4. overviewkeyMAC = Remaining last 32 bytes.

Opdata
1. opdata01 = overview key first 8 bytes : NSRange(0,8) - convert it to UTF8 and given text is fetched - Done
2. length of plaintext = overviewKey next 8 bytes : NSRange(8,8) - Little endian conversion and it gives length - Done (in this case 64 bytes)
3. IV = overviewKey 16 bytes after length : NSRange(16,16)

Confusion1:
It is mentioned that "The data, including the IV and prepended padding, is encrypted using AES in CBC mode with a 256-bit encryption key."

Question : Should the header be removed from data while decrypting? Are the below steps correct?

We must decrypt the overview key data except the header i.e first 16bytes which are opdata01 string and length. so the resulted data to be decrypted is

dataToDeCrypt = overviewkey.subdatawithrange.NSRange(16,overviewkey.length-16)

Note: Temporary using CCCrypt instead of CCCryptorCreate

    ccStatus = CCCrypt(Decrypt,
                           kCCAlgorithmAES128,
                           0, //Since you are not using kCCOptionPKCS7Padding
                           derivedKey.bytes,
                           derivedKey.length,
                           IV.bytes, //Which we get from overview key
                           dataToDeCrypt.bytes,
                           dataToDeCrypt.length,
                           dataOut.mutableBytes,
                           dataOut.length,
                           &cryptBytes);

The result must be separated as below:
1. iv : NSrange(0,16) IV
2. padding : NSrange(16,16) (even data bytes so 1 block of random data prepended must be removed)
3. OverviewKeyDecrypted (64 bytes) = NSrange(32, result.legth-32)  

Confusion 2:
Its Mentioned : " The contents of overviewKey is verified and decrypted with the derived keys; the result is then hashed with SHA-512; the first 32 bytes of the hash output will be the overview encryption key; the last 32 bytes of the hash output will be the overview hmac key."

According to above statement the resulted "OverviewKeyDecrypted" must be hashed with SHA-512 so here what should be the hash Key ?

Once hashed the OverviewKeyDecrypted must be divided into 2 parts (overviewItemkey(32) + overviewiTemMacKey(32)). overviewiTemMacKey must be used to verify item overview.

1. ItemOverviewBase64Encoded : BandFile the data with key 'o'
2. ItemOverview : Base64 Decode 
3. ItemOverview = ItemOverviewData + ItemOverviewMac (last 32 bytes)
4. ItemOverViewDerived =  ItemOverviewData HMACSHA-256 with overviewiTemMacKey

if (ItemOverviewMac == ItemOverViewDerived ) //We are not getting this equal
Process till now is correct

Trishaelwood

Hi thanks. please ignore the above comment all confusions have been cleared. Thanks for the support though.

Trishaelwood

Hi sorry again had few questions. They are as below

1. When we decrypt value for 'd' from profile.js, along with password you also get something like backup keys, can you explain what are these and when are they used?
2. Whenever a new vault is created its new overview key and master key are generated right then with which master password are they encrypted ? i.e since you do not store master password anywhere then how are these encrypted when we create a custom vault inside master password. (we think we have misunderstood something here)

jpgoldberg

When we decrypt value for 'd' from profile.js, along with password you also get something like backup keys, can you explain what are these and when are they used?

Can you tell me more about how your OPVault that you are playing with was created? d is not a typical field in a profile.js. (Note that if you are looking at the SQLite local data formats, there is stuff in there that isn't in a .opvault, but I need to know what you are looking at to be able to answer.)

Whenever a new vault is created its new overview key and master key are generated right then with which master password are they encrypted ?

Unless you are talking about 1Password for Teams, a Master Password for a new vault does need to be specified when you create it.

i.e since you do not store master password anywhere then how are these encrypted when we create a custom vault inside master password. (we think we have misunderstood something here)

A vault Master Password is needed when the vault is created. But the keys to the new vault will be stored in the primary vault on the device that you created the vault with. When you first use that vault on a different device, you will need to enter the password for that vault.

I'm not sure that I've actually answered your question, but I hope that this helps anyway.

Trishaelwood

Hi jpgoldberg, thanks for the reply. We have 1password for mac, when we go to 1password -> preferences -> sync -> sync Primary vault with folder option, a opvault is generated at that location. On finder show contents options we can see the profile.js and few band files.

For the first question we mistyped the file, we meant band file (e.g. band_2.js).
1. When we decrypt value for 'd' from band_2.js, along with password you also get something like backup keys, can you explain what are these and when are they used?

Following is the data we get after decrypting. In this we needed to know regarding the key named backupKeys.
{"backupKeys":["EC7KGKyrlkMruLyI515aOBdy8mgjlwsH9X6fvgb+sgh3pEuuSlOyFgD6JK5G5eO3VhjndgVneb5pg0dL4C8Q7ow=="],"password":"xxxx"}

jpgoldberg

OK. That makes more sense. You are finding d in an item, not in profile.js.

These backupKeys are added to resolve a rare problem that can happen with synching and conflicts.

An item in an OPVault will only be encrypted with its item key, but that same item may have a different item key in the local formats of on different platforms. And indeed if a new or different OPVault is created with the same item, it may end up with a different item key.

We like to be able to stage various parts of synching even when 1Password is locked. Very roughly you can think of OPVault as something that 1Password exports to and imports from (which is separate from transmitting the files). So when you unlock 1Password on your Mac you are getting both the keys for your local vault, but also the keys for the OPVault. When 1Password needs to merge an item in your local vault with something in your OPVault it needs to be able to decrypt both and needs both keys.

But now imagine if an item has been deleted on one system but not another. We want to be extra careful to keep track of all of the keys it has been encrypted under so that it could be fixed up during a sync even if one side of the sync has lost its own item key and has to just rely on the item in the OPVault. So what we do is we keep these keys in the details, so that if you can decrypt an item in one place (say in an OPVault, you can get the item key that you might need in your own SQLite file.

They don't reveal anything because you can already decrypt the details to get at them, but they make it easier to automatically recover from some synching mishaps.

Synching is hard. Indeed, it is harder than crypto; so I might have to come back here and correct things I might have misstated.

Rad

Hi Trishaelwood,

1Password item attachments are encrypted with the item’s key. The item’s key can change over time: Master Password has changed, the item’s vault password has changed, etc. When this happens, we store the previous keys to ensure we can unlock attachments.

Hope that this helps :wink:

Trishaelwood

Hi jpgoldberg,

**I would like to know more about the sync process, how the item encryption and decryption works between a device and sync vault (cloudkit) ? How are conflict resolved ? **

This is what I have understood currently. Taking example of 2 iOS devices (D1, D2) and Sync Vault.

As explained by rick
1. D1 has MP-A
a) d1-mk, d1-ok encrypted by D1-dk
b) syncMK, syncOk encrypted with d1-mk
c) d1-mk, d1-ok encrypted with syncMK (Question : Where are these stored in sqlite?)
d) D1-Item1 encrypted with d1-mk/d1-ok (Question : Where are these stored in sqlite?)
we have b) and c) because we can unlock other if we have mp of anyone of them.

2. D2 has MP-B
a) d2-mk, d2-ok encrypted by D2-dk
b) syncMK, syncOk encrypted with d2-mk
c) d2-mk, d2-ok encrypted with syncMK
d) D2-Item2 encrypted with d2-mk/d2-ok

3. Sync Vault(Cloudkit)
a) syncMK
b) syncOk

Lets see the sync process:
When D1 syncs with SyncVault.
1. A sync vault(Its the first time we are syncing so) is created with syncMK and syncOK keys.
2. Now we export item to sync vault so the item first gets decrypted with D1-MK/D1-OK and then encrypted with syncMK/syncOK and is stored in cloud kit Db
3. D1 stores the copy of syncMK/syncOK encrypted with its d1-mk and a copy of d1-mk and d1-ok encrypted with sync-mk for future sync use.

When D2 syncs with SyncVault.
1. MP for for sync vault is asked once SyncMK/SyncOk are decrypted, a copy of SyncMK/SyncOk encrypted with D2 is stored in D2 vice versa.
2. Now for items, if going according to apple's suggested way(Local data upload, resolve conflict, fetch cloud changes, store in local), first local items i.e D2-Item2 is uploaded internal process being decrypting D2-Item2 with D2-MK/D2-OK and encrypting it with SyncMK/SyncOK then its uploaded to cloud kit.
3. Now next step being importing items from sync vault, D1-Item1 is imported to D2 internal process being decrypting D1-Item1 with syncMK/SyncOK and re-ecrypting it with D2-MK/D2-OK and storing it in local.

So have I understood it properly?

Few Questions though
1. How do you handle cloud account changes ?
2. "But now imagine if an item has been deleted on one system but not another. We want to be extra careful to keep track of all of the keys it has been encrypted under so that it could be fixed up during a sync even if one side of the sync has lost its own item key and has to just rely on the item in the OPVault. So what we do is we keep these keys in the details, so that if you can decrypt an item in one place (say in an OPVault, you can get the item key that you might need in your own SQLite file."

For the above scenario
1. D1 deleted its item
2. SyncVault and D2 has that item yet.
3. Now when D1 deletes the item, and syncs then the item must be deleted from sync vault too for that we can identify the item from its UDID we do not need to decrypt it. So why backup keys?
4. And even in case the opvault do have that item we can always decrypt the item from opvault and create a new item and insert it in our sqlite file with new item key encrypted with device mk/ok. Am still not clear on backup key utilization.

@ rad "1Password item attachments are encrypted with the item’s key. The item’s key can change over time: Master Password has changed, the item’s vault password has changed, etc. When this happens, we store the previous keys to ensure we can unlock attachments."

Why does the item key change?

If i have understood the process clearly enough then
_Masterpassword-> DerivedKey ->MasterKey -> ItemEncryptionKey _ so even if master password changes only the encrypted form of master key would change why is their a change in itemEncryption key?

Example :
(Note: Masterkey will be randomly generated data, here only for the sake of example we are considering in a more understandable form)
a) master password = "A"
b) MasterKey(non-Encrypted) = "abc"
c) MasterKey(Encrypted with derivedkey) = "XXX"
d) ItemEncryptionKey (Encrypted with MasterKey(non-encrypted form)) = "ZZZ"
e) user changes master password from "A" to "B"
f) MasterKey(Encrypted with derivedkey) = "YYY" (since master password changes so does derived key and so the encrypted form of masterkey).
g) MasterKey(non-Encrypted) = "abc" (this remains same)
h) ItemEncryptionKey (Encrypted with MasterKey(non-encrypted form)) = "ZZZ"

The g) and h) remains same since their is only a change in encrypted form of the key and not the key itself.

https://blog.agilebits.com/2015/04/28/how-1password-syncs-changes-to-your-master-password

@ jpgoldberg :
1. Would also like to know more about the sync process in detail. Issue faced and conflict resolution etc.

note : I noticed that you provide user an option to choose which copy of the data to keep and not decide on your own based on modified date. why is that?

Ben

Hi @Trishaelwood,

As these forums are primarily intended for troubleshooting technical difficulties and addressing pre/post sales questions, I'd like to move this conversation to email so we can have the appropriate resources interact more directly.

Could you please drop us an email to support+forums@agilebits.com with a link to this thread?

I'm going to close this conversation here.

Thanks.

Ben