What does "Rebuild Data File" actually do?

m_w

Hi - I was playing around with different keychains and at one point I deleted all records from the active keychain which had 776 records, deleted the trash and created just a couple of test secure notes. I saw the keychain bundle shrink from around 32 MB to below 2 MB. Then I performed a "clear cache" and "rebuild data file" operation and was surprised that during the rebuilding it was processing 914 records. I repeated this process a few times but it always kept processing 914 records. So I had a look in the ~/Dropbox/1Password/1Password.agilekeychain/data/default folder and found 917 items in there. And yet the keychain only contained 2 secure notes. Later I restore all records using the "File > Restore" command and the keychain was back at 32 MB. Rebuilding the data file again processed 917 files and in the default folder I still had 917 records.

Therefore my question: what does "rebuild data file" actually do?
And in addition: is it right to have 917 items in the keychain default folder if the keychain contains only 2 items?

Cheers
Michael

MikeT

Hi Michael,

That is normal. When you empty your trash, the items are not gone but simply tombstoned. In other words, the data values are deleted for good but the metadata is left behind. The reason is because it's used to sync permanent deletions to other endpoints such as your iOS app, browser extensions and so on. If we deleted the file itself, the endpoints would think it's missing and syncs the data back to the data file.

Rebuilding basically reads the data, copy it and recreate it into a new item that's exactly like the first one, it's a way to refresh the items to get rid of any stale stuff. This also does the same for the tombstoned items.

Here's an example of a tombstoned item if you look at it:

{
    "uuid": "0B3DD335F48E432BB8A0D47307A4D7C3",
    "updatedAt": 1360806914,
    "locationKey": "",
    "openContents": {
        "contentsHash": "fe62a730"
    },
    "keyID": "E4AFA4BA5A814571B85938DADE701C1C",
    "trashed": true,
    "title": "",
    "location": "",
    "encrypted": "U2FsdGVkX18OUZakhn9MBGtJc48Pj/HKTqX7uonStyE=\u0000",
    "createdAt": 1278191192,
    "typeName": "system.Tombstone"
}

m_w

Thanks for your explanation Mike - that makes sense. However, does this mean that the data folder will grow indefinitely? Or will it start pruning those tombs once they synced across all devices (how would it know that all devices have been synced though) or after a certain time?

Cheers
Michael

MikeT

Hi Michael,

Yes, it can grow over time despite the tombs being very small. We're trying to find a better way to handle this. This is one of the reasons why the sync process can be the most difficult part of many apps, it's in their nature to make the entire app 10x more complex.

Right now, the cleanest way to get rid of those tombs are to export the data and import it in a new data file, which is probably something we can do in the future with a hidden option. The thing about doing this, is that you'll have a separate data file ID, which kicks in a security alarm in the 1Password Helper to block the sync between the browser extensions and the main 1Password app. You'd have to reinstall the extensions to allow the new IDs.

The security policy is in place to prevent 1Password from accidentally syncing two different data files into the same file.

m_w

Understood - thanks Mike.

MikeT

You're welcome!

chuckycheese

How do I move my data to a new file to get rid of the tombstoned files? My keychain is large from years of tombstoned credentials and I'd like to shed them to keep my keychain as small as possible.

khad

They shouldn't really be taking up much of any space at all, but you can:

1. Backup your data file (File > Backup).
2. Export all your data to 1PIF (File > Export All > 1Password Interchange File).
3. Create a new data file.
4. Import the file you created in step one (File > Import).

To securely delete the unencrypted 1PIF file in Finder:

1. Empty your trash (Finder > Empty Trash). (This will ensure that only the 1PIF is securely deleted since the process takes a long time and there is likely no need to securely delete all of the items in your trash.)
2. Move the 1PIF file to the trash (File > Move to Trash).
3. Securely empty your trash (Finder > Secure Empty Trash).

If we can be of further assistance, please let us know. We are always here to help!

m_w

I amended the steps slightly to adapt to my situation:

1. Backup your data file (File > Backup).
2. Stop time machine or other backup processes that may trigger
3. Stop dropbox syncing (on all computers accessing the same 1Password.agilekeychain)
4. Export all your data to 1PIF (File > Export All > 1Password Interchange File)
5. Create a new data file. (also stop browsers and 1Passsword running on other computers)
6. Import the file you created in step one (File > Import).
7. Securely delete the unencrypted 1PIF
8. Start dropbox syncing; wait until synced
9. Start time machine or other backup processes

I performed these steps but can't see any reduction of tombs:

Dropbox/1Password/1Password.agilekeychain/a
before 30,946,247 bytes (38.4 MB on disk)
after 30,479,363 bytes (38.1 MB on disk)

Dropbox/1Password/1Password.agilekeychain/config
before 520 bytes (16 KB on disk)
after 6 bytes (8 KB on disk)

1Password/1Password.agilekeychain/data
before 1,272,110 bytes (4 MB on disk), 943 items in default folder
after 1,254,691 bytes (4 MB on disk), 943 items in default folder

I would have expected the number of items on the data folder to go down to 788. Mind you maybe we need to add another step before 4 to empty the 1Password Trash before exporting the data. I can see 129 items in the Trash which leaves a total of 917 items.

When I setup the new keychain file I also gave it a new master password. This works on my iMac and laptop now. But when I start 1Password on my iPhone I can only log in with the old master password and then the sync tells me that the data does't match.

Cheers
Michael

m_w

Even deleting the app from my iPhone and reinstalling it doesn't help. It still only accepts the old master password.

khad

m w, I'm really sorry for any confusion. My post was a reply to chuckycheese above and not intended as steps for you to follow for any issue you are having with unlocking 1Password on your iOS device.

If you're still having trouble, could you please send us a Diagnostics Report?

In 1Password 3.8, just select Help > Troubleshooting > Diagnostics Report from the menu bar.

If you are using 1Password 3.9 (from the Mac App Store), you'll need to download the 1Password Troubleshooting utility.

Then attach the entire file to an email to us: support@ agilebits .com

Please do not post your Diagnostics Report in the forums, but please do include a link to this thread in your email so that we can "connect the dots" when we see your Diagnostics Report in our inbox.

Once we see the report we should be able to better assist you. Thanks in advance!

m_w

Thanks khad - I followed the steps to get rid of the tombs albeit not quite with the expected result.

The problem with the password on my iPhone started only after setting 1Password up from fresh. I had changed the master password in the process and would have expected that the iPhone client would ask for the new password, especially after deleting the app and its data from the iPhone. But it kept rejecting the new master password and only let me sign in with the old password. I repeated deleting the app a few times and eventually it brought up Dropbox asking me to allow access. It then asked for the new master password and 1Password is now running fine on my iPhone with the new master password.

Back to the original question of this thread: any idea why the tombs were not removed when I cleared my keychain?

Cheers
Michael

khad

It may not be possible to remove them. I thought that the export/import dance would have pleased the tombstone gods, but I'm double-checking with the developers (RM4346).

khad

It sounds like you could export to 1PIF, manually remove the tombstones and then recreate a data file from the 1PIF. But that seems like a lot of work for something that doesn't really offer any gain. I'm still not sure I understand why you would want to do this. The amount of space taken up by tombstones is likely on the magnitude of a fraction of a fraction of a percent of your entire data file size. We're talking kilobytes here. :)

pervel

I think the reason for wanting to delete the tombstones completely is something entirely different than data size. It's a matter of having control of your data. If I delete something, I want all traces of that something to be gone. Ideally I don't even want there to be any trace left that there once was an item.

It appears that this conflicts with the needs to sync data. So to ease my mind a bit, I would like to know if there is anything left in the tombstone from the original entry other than pure metadata? That is, would it be possible for somebody to recreate things like title or location or even just the date of creation of the entry?

Jeff Shiner

The tombstone contains almost no data at all. The title, contents have all disappeared for good and can not be reconstructed. The one piece you mention that does remain is the createdAt timestamp. We can certainly look at removing even that moving forward. The tombstone data portion in the export would look like the below. You can see that title, location, secure contents and even the item type are gone.

"trashed":true,"title":"","location":"","secureContents":{},"createdAt":1353034574,"typeName":"system.Tombstone"

m_w

I agree with pervel; indeed, it's not about gaining disk space but about control of your data.

Seems there is a little more details in each tomb though (see also MikeT's post further up). I found this data in one of my tombs:

{"uuid":"some data","updatedAt":1336941378,"locationKey":"","openContents":{"securityLevel":"SL5","contentsHash":"some data"},"keyID":"some data","trashed":true,"title":"","location":"","encrypted":"some data","createdAt":1336941372,"typeName":"system.Tombstone"}

I replaced some parts with "some data" to avoid posting anything sensitive.

khad

The data that is specific to you — data you have entered yourself or that 1Password automatically recorded when saving a Login item via the browser extension — is removed: title, location, etc. All the data you are seeing is 1Password metadata: UUID, keyID, etc. There is not a way for anyone to reconstruct anything useful from it.

I'm not sure I understand the scenario you are concerned about. Could you provide more details as to how you think this could be used for ill?