Metadata is not encrypted

fargo888

I just got 1password from the appstore. Just clickin around a bit I found that inside the 1Password.agilekeychain are numerous files that actually contain the login URL and the username used in PLAIN TEXT. Seriously?

If somebody gets a hold of the 1Password.agilekeychain file they will be able to know which sites I usually log in and which username I use. Is that intentional?

Steven

khad

Hi Steven,

Thanks for taking the time to contact us. Usernames are not stored in the clear. The only information that is stored in the clear is the metadata about your sensitive data not the sensitive data itself. You can easily see a list of what is stored in the clear in the Agile Keychain Format by simply looking at the View > Columns menu in 1Password for Mac. Here is the list for your convenience:

• Icon
• Title
• Location
• Type
• Modified Date
• Created Date
• Folder
• Tag

Password strength used to be included in that list as well, but that was changed way back in November 2011.

This is outlined in a few different places in the User Guide. From the Agile Keychain Design document:

The Agile Keychain is nearly identical to the Mac OS X keychain in terms of what is kept encrypted and what is left open in plain text. The distinction is an important trade-off between security and convenience. The more that is encrypted, the less a would-be thief can access, but it is also necessary to leave enough open to allow applications to freely access certain items without needing to decrypt every single entry each time. The Mac OS X keychain nicely balances security and convenience, so the Agile Keychain follows suit.

>

Here is an example entry from the Agile Keychain:

@{
 "title" : "dave @ AWS login",
 "locationKey" : "perfora.net",
 "encrypted" : "...",
 "typeName" : "webforms.WebForm",
 "securityLevel" : "SL5",
 "openContents" : {
   "createdAt" : 1216012929,
   "updatedAt" : 1216012929,
   "usernameHash" : "...",
 },
 "location" : "https://webmailcluster.perfora.net:443/xml/webmail/Login",
 "uuid" : "0A522DFCAE6442D991145BC76E55D343",
 "folderUuid" : "A90D66D1A4E34481BDF03DDEA9F511AC"
}

As you can see, not all the information is encrypted. Most notably, the name/title of each entry (i.e. dave @ AWS login) and the location/URL are open. Having these open allows 1Password to organize your data and display it without suffering the performance hit of needing to decrypt every single item. All the truly confidential information is stored in the encrypted section of the file.

The original form of the Agile Keychain left its assessment of password strength among the unencrypted data. This was removed in 2011.

The above file format is based on JSON (JavaScript Object Notation). It is a lightweight notation for structuring data without the overhead associated with formats like XML. As a side benefit, these JSON files can be loaded directly into a web browser. The name of the file is based on the UUID (Universally Unique Identifier) of the item. This guarantees the filename is unique and will stay the same even when items are renamed.

You can read more about this in the "Unlocked vaults or unlocked boxes" section of the Security of storing 1Password data in the Cloud document.

As hinted at in the aforelinked Defending against 1Password harvesters, the new Cloud Keychain format already in use for iCloud syncing encrypts or well-obfuscates even the metadata. You can read about the Cloud Keychain format here:

1Password 4 Cloud Keychain design

As we move forward, the Cloud Keychain format will be used in more places.

If we can be of further assistance, please let us know. We are always here to help!

fargo888

Thanks for your answer :)
I hope the 1Password 4 Cloud Keychain Design will be implemented very soon as I strongly believe that it is part of good privacy considerations that the websites I visit (and I own some of them) could not be easily read by a potential attacker.

Steven

khad

It has already been implemented. :) The Cloud Keychain format is used for iCloud syncing. Additionally, the sandboxed browser extension databases also encrypt and obfuscate the metadata.

We will be rolling out Cloud Keychain more widely soon.

David

I've been using 1Password for years and was unpleasantly surprised to find that some fields (such as the title!) are unencrypted. To me this is an unacceptable breach of trust in a password manager.

Has this problem been fixed in 1Password version 4? I spent 5 minutes looking for the benefits of upgrading from v3 to v4 and couldn't find them on your website.

khad

Hi @David,

I merged your post with this existing thread which has more up to date information. Please read my post above and let me know if you still have any other questions for concerns. I would be more than happy to address them.

Thanks!

jpgoldberg

Hi @David and @fargo88 (and others whom I've missed)

I'm really concerned that you felt betrayed in any way. If you can think of ways that we could better document that fact that in the Agile Keychain information such as Title and URL are not encrypted, please let us know. As previous comments here have shown, we have

1. Documented this from day one in our description of our data format
2. We have posted several blog articles that make specific reference to this. (There are more, I'm sure, but I got tired to searching for them.)
3. We have several other security documents that discuss this
4. This has been openly discussed in our discussion forums since day one (which is why we merge all these threads into one place)
5. I don't think that any other commercial password manager has gone to the lengths that we have to make the details their security design as public as we have.

It would be possible for us to lightly obfuscate that sort of data. Doing so would provide no meaningful security, but it would mean that people looking casually at the data may not realize that it is properly encrypted. We try to avoid that kind of "security theater", but as a consequence casual examination of the data does transparently reveal what isn't encrypted. So we could have concealed the fact that that data isn't encrypted by employing some minor obfuscation, but such a move would only conceal from users (and not attackers) that the data isn't encrypted. That isn't our style.

As for 1Password 4, you can read about the security changes, including the fact that for the data format that will be used for synching and storage "in the cloud" Location and Title are encrypted.

But please do go and look at Khad's earlier post. We've been having this discussion for a long time, with new people joining in over the years.

Cheers,

-j

–-
Jeffrey Goldberg
Chief Defender Against the Dark Arts @ AgileBits
http://agilebits.com

David

@khad and @jpgoldberg, thank you for responding to my message.

I like your transparency about some pieces of information not being encrypted, as you enumerated in your post above. The problem is that the casual user, like myself, is not going to look in those places before using the program. I launch the program, see an animation of ALL the information being obscured by a door requiring a password, and assume that (like the GUI implies) all the information is protected.

Here are some design guidelines:

• Busy users don't read manuals (nor product blogs, nor security documents, nor forum messages).
• Users learn about program state/behavior via GUI hints. This is what makes GUIs so powerful. When my Trash icon in Mac OS looks empty, I assume that the directory ~/.Trash is empty. I don't go reading Mac OS blogs, documents, forum messages, etc., to learn whether this is really true or not.
• You are very overtly using GUI hints. A full-window graphic of a sliding security door is a massive hint. It implies that everything behind the door is protected. My assumption, similar to the one above about Trash, is that the underlying data in the file Dropbox/1Password.agilekeychain/data/default/0A1E4BDFEB884B458B6582717252ED71.1password is similarly protected and that Dropbox, Inc., (and Amazon) don't have access to the knowledge that I have a password with the title "BofA safe deposit box".

Here's what I would like you to do differently:

1. Treat security as your #1 priority and everything else (ease of use, etc.) as secondary. In a security product (which 1Password is), security = trust. I'm a software engineer and I know that if I can encrypt my whole hard disk and still have good performance and ease of use, then you can find a way to encrypt (not obfuscate) all of the data behind that door.
2. If you're going to imply that the data is secure with GUI hints, then create new GUI hints to show which data ISN'T secure.

David

jpgoldberg

Hi David,

Thank you for comments and suggestions.

Security is our #1 priority. The decisions and trade offs we make are often "security/security" tradeoffs. That is, we are balancing security in one domain with security in another. You will see this discussed in far more detail in some of the documents I linked to. In particular this one.

The particular issue also provides a great example for your other suggestion, namely that the model or metaphor we present to the user correctly present what is going on under the hood. You will see that (for security reasons) we only decrypt one item at a time. Yet what is presented to the user presents an all or nothing view. Should we really have our visual metaphor reflect that?

We also say things like "encrypted with your Master Password" and everything we present reflects that. Should our GUI try to convey that data is encrypted with a key that is the result of hashing a random hunk of data (created at key chain creation time) that is decrypted by a key and initialization vector which is derived from an iterative process of scrambling your Master Password combined with a random salt?

All of that is relevant to security. In fact, really just encrypting data with the Master Password (instead of doing the kind of stuff we do) would be a terrible idea. Yet at the same time, it is most useful for the large majority of people for them to just think of their Master Password as a key.

Now there are cases where the differences between the simple model presented through the GUI and the underlying stuff can lead to problems and are tough decisions. These sorts of decisions are never taken lightly. But on the whole, our approach is to present something simple and intuitive while also providing full details for those who do want to understand how things "really" work.

I don't think that you will agree with all of the particular decisions that we've made along those lines, but I hope that you will agree that presenting a picture of how things really work would be utterly confusing. If you need a lesson in why directly encrypting data with the Master Password would be a bad idea just to be able to use 1Password, then 1Password would be providing security for far far fewer people.

As an aside, there actually are some visual hints. I think it's posted somewhere much earlier in this thread, but if you look at a 1Password 3 open on a particular item, the stuff in the white rectangle is that is encrypted. The stuff above (Title and location) and the stuff below (creation data, modify date) is not.

Of course this is growing ever more moot as we continue to roll out the new data format.

Although you may never be happy with all of the various security decisions we've made, I hope that you will recognize that these were made mindfully.

Cheers,

-j

–-
Jeffrey Goldberg
Chief Defender Against the Dark Arts @ AgileBits
http://agilebits.com

David

Security is our #1 priority. The decisions and trade offs we make are often "security/security" tradeoffs. That is, we are balancing security in one domain with security in another. You will see this discussed in far more detail in some of the documents I linked to. In particular this one.

I'm glad to hear that. Your linked document makes for some interesting reading, in particular this:

When you go to a login page, say http://www.example.com/Login.php, 1Password needs to find all of the boxes that could potentially be a Login for that location. It also needs to present you with a list of those potential Logins so that you can choose among them. Conceivably (but incorrectly), 1Password could go and unlock each box in the room looking through their contents to determine which ones are potential matches. But that would take a very long time. Opening a single box doesn’t take any noticeable time, but opening all of them would be prohibitively slow.

What we have done is put labels on the outside of each box. The labels contain, most importantly, the web location associated with that Login and the title that you gave to that Login. This way 1Password can scan the locations quickly without having to unlock any boxes. It can then present you with the titles of the ones that are potential matches. Once you select to fill with a particular login will 1Password unlock the particular box.

The downside of this is that 1Password must keep the titles and the web locations unencrypted in your data.

I disagree with the last sentence. You could, e.g., store an HMAC digest for the URL of each box. When the user goes to a login page, you could then compute the HMAC digest for that page's URL and then quickly scan for a match. I just did a test in Node.js on my computer and it took 145 ms to generate an HMAC-SHA256 for a worst case URL of 2,000 characters.

The particular issue also provides a great example for your other suggestion, namely that the model or metaphor we present to the user correctly present what is going on under the hood. You will see that (for security reasons) we only decrypt one item at a time. Yet what is presented to the user presents an all or nothing view. Should we really have our visual metaphor reflect that?

The GUI metaphor doesn't present all of the underlying mechanism to the user, otherwise it wouldn't be an abstraction. It provides only the information that is of value to the user. I would argue that in this case, no, it's not valuable to the user to know that you decrypt only one item at a time. As a user I don't care about that.

We also say things like "encrypted with your Master Password" and everything we present reflects that. Should our GUI try to convey that data is encrypted with a key that is the result of hashing a random hunk of data (created at key chain creation time) that is decrypted by a key and initialization vector which is derived from an iterative process of scrambling your Master Password combined with a random salt?

Again, no, that's not valuable to the user because the user cares about results, not implementation. For example, when I place a check into an envelope designed to obscure the contents, I know which information is protected and which isn't because I can see the plastic window reveal the address. I don't bother to look inside the envelope to see how the envelope implements the obfuscation of the rest.

If you need a lesson in why directly encrypting data with the Master Password would be a bad idea just to be able to use 1Password, then 1Password would be providing security for far far fewer people.

This is missing the point. As long as data is secured, users don't care how it's secured. They do care which data is secured and which isn't.

As an aside, there actually are some visual hints. I think it's posted somewhere much earlier in this thread, but if you look at a 1Password 3 open on a particular item, the stuff in the white rectangle is that is encrypted. The stuff above (Title and location) and the stuff below (creation data, modify date) is not.

Ah ha! Put a symbol of an unlocked padlock in the blue stuff above, a locked padlock in the white rectangle below, and we're there.

I hope that you will recognize that these were made mindfully.

You've clearly put a lot of thought and writing into it, so I do recognize that your security decisions were made mindfully. I want you to do better. :-)

David

jpgoldberg

You could, e.g., store an HMAC digest for the URL of each box. When the user goes to a login page, you could then compute the HMAC digest for that page's URL and then quickly scan for a match.

Very nice! We did look at solutions along those lines. In the end, we rejected it because it would require exact matches. We couldn't rank matches based on how well the stored location matched the current URL; instead it would be all all or nothing thing.

Of course there are still things that we could have done. We could have had multiple hashes per location. For example with a location like http://z.y.x.w.example/path1/path2/page.htm we could store HMAC(key, http://z.y.x.w.example), HMAC(key, z.y.x.w.example), HMAC(key, y.x.w.example), HMAC(x.w.example); and then do the same with path components added as well.

But to list potential matches in terms of best fit, we would also store (in plaintext) a score for each of these indicating how well it fit the full location, and do so in a way that would be commensurable across items. Or more simply, we could simply not try to find the "best" fit and only list all items that match.

We probably could have done something along these lines, but instead we focused our attention on developing a new data format. There were other things that we've needed to do, and so instead of trying to deal with piecemeal migration and changes to the Agile Keychain Format, we've been working on a format that is designed for the future. The data format you design today is going to be something you live with for a very long time.

Furthermore, with our new data format, we are able to have a new "minimum" requirement for the hardware. What was computationally prohibitive on the systems that were in wide use five years ago are doable now. When the Agile Keychain was first developed in 2007, there were 366 MHz 64MB iBooks still running the then latest version of OS X (Tiger, OS X 10.4).

1Password 4 for Mac will require (at least) 10.8, and 1Password 4 on iOS requires iOS 6. So the minimal phone (3GS) that runs 1Password 4 today has twice the processing speed and four times the memory along with hardware accelerated AES support. And the portion of users using that minimum is far lower today than the number of people near the minimum in 2007.

The trick that we use now to keep things like Title and Location encrypted, just wasn't practical back then. The trick is to encrypt the "overview information") with a separate overview key. All of the overview data gets decrypted when you unlock 1Password, but the details of each item remain encrypted during normal operation until you need that particular item.

It's not just the hardware that changes, but also the threat landscape. We designed the Agile Keychain Format for syncing, but I'm not sure that we fully appreciated the extent to which syncing and remote storage would play a role. And – I'm perfectly ready to admit this – our understanding of privacy concerns has grown over the years. When the Agile Keychain Format was first developed, it was typical for people to not be concerned about encrypting the contents of their web browser bookmarks. The unencrypted information in the Agile Keychain is little more than what is in browser bookmarks. Back in 2007, when the Agile Keychain was being developed, this was hardly an unreasonable position to take, particularly when we simply didn't have the tools to do otherwise without making other sacrifices.

You've clearly put a lot of thought and writing into it, so I do recognize that your security decisions were made mindfully. I want you to do better.

Enjoy the new data format!

Cheers,

-j

–-
Jeffrey Goldberg
Chief Defender Against the Dark Arts @ AgileBits
http://agilebits.com

David

Thank you for the historical context and your transparency.

jpgoldberg

Thank you, @David!

Valdemar1

I was curious as to how this would work: If I need to hide the metadata in my dropbox folder, I could simply drop the 1password.agilekeychain file into a sparse disk image. I don't use any smartphones and have two Macs that have access to the file in my dropbox folder on two different continents. I would merely open the disk image when I wanted to access my 1password data. Is this feasible?

jpgoldberg

Hi @Valdemar1,

The last time I tested, synching sparsebundles via Dropbox isn't very reliable. Also the Agile Keychain Format is very disk I/O intensive so you will probably have severe performance problems that may lead to hangs. So we do not support this approach.

However, if you promise to make and keep very good backups, you may want to give this a try. But anticipate data corruption, so make and keep good backups. Whether this works for you or not, please report back to let us know how well it works for you.

Don't attempt this unless you maintain excellent backups of your 1Password data. Again, this is unsupported, meaning we can't promise to help you get this to work.

Cheers,

-j

–-
Jeffrey Goldberg
Chief Defender Against the Dark Arts @ AgileBits
http://agilebits.com

Valdemar1

Yes, I promise to make excellent backups! I used Knox to set up the sparse disk image file at 100mb in size with 256 bit encryption which was created in my dropbox folder. After it was mounted on the desktop, I moved the 1password file into that drive. I proceeded to delete the original 1password file.

Upon restarting 1password, the program asked me to search for the file, which was easy enough to find in my newly created sparse disk drive on the desktop.

An immediate drawback is that my login items can no longer contain the 1password program. Instead, I have placed the sparse disk image file to open at login. This requires me to enter my password each time I restart my computer. A subsequent launch of the 1password program works flawlessly.

I notice no immediate difference when opening 1password and launching websites or browsing the program. I also notice no difference when logging into websites from my browser. I notice some activity from the dropbox menu, but all seems to be well.

When I travel to my other computer, I'm guessing I will need to be particularly mindful to shut my first computer down, or at least eject the sparse disk image drive!

Thanks for the tips!

Valdemar1

Regarding your comment about syncing sparse disk image files and their reliability, I wanted to add that I have several Knox files in my dropbox folder and have had them there for a couple of years.

In the beginning, I had several Macs sharing the dropbox folder, at work and at home. My biggest issue was when I failed to eject the sparse disk image drive of any of the Knox files that I had open. When I got home and opened the drive, I noticed that my dropbox was quite active. This made no sense, at least to the degree that it was occurring. Upon closer examination of the sparse disk image files (Knox files), I noticed many "conflicted copies" in the bands being created. I dragged the conflicted bands to the trash and deleted them, with no adverse effect, but it was then that I realized that it could turn out badly if I left a Knox drive open on a previous computer.

I have now made it a habit to eject all such drives from the desktop before traveling.

Valdemar1

One disconcerting thing has sprung up since I started my experiment. Even though my 1password file is safely tucked away in an unmounted sparse disk image file (the 1password program upon launching, asks me to create a new file or search for an existing file), the Safari extension still has all my passwords and information readily available. Testing in Firefox, the same phenomenon is apparent. How can this be? There is obviously a copy of all my 1password information stored somewhere on my computer which Firefox and Safari are accessing.

1password logs indicates that somewhere in library/application support/1password/extensions is being accessed.

Naturally, I don't want the browser extension to be functioning as long as the sparse disk image is not mounted. Please advise.

khad

The browser extensions each have their own encrypted data store within their respective sandboxes.

Naturally, I don't want the browser extension to be functioning as long as the sparse disk image is not mounted. Please advise.

Your 1Password data is protected by your Master Password in the main app as well as the separate, sandboxed browser extensions. There is no need to further encrypt it, and, as mentioned above, the storing of your data file in a disk image is an unsupported configuration. If you don't want the data in the extensions to be accessible when the disk image is not mounted you will need to uninstall them when you unmount the disk image.

Again, we don't recommend or support this. As Jeff mentioned above, we can't promise to help you get such a configuration to work properly.

Valdemar1

I was unaware that there were copies of my 1password database accessible to the browsers. I was concerned about the encryption of said extra passwords. Thank you for addressing that concern. I will continue my experiment.

khad

The Cloud Keychain Format is used for the browser extension's data stores. Among other things, the Cloud Keychain Format introduces AES256 and does not reveal any metadata which I presume to be your main concern based on the title of this thread in which we are posting.

Valdemar1

Yes, that was exactly my concern. Thank you for the update.

khad

Happy to help. Stay safe out there. :)

jpgoldberg

Hi @Valdemar1,

Metadata in 1Password 4 Cloud Keychain Format

Just to be completely up front, there is some metadata that is discernible from the 1Password 4 Cloud Keychain Format. The document that @Khad pointed you to spells these out:

Some metadata remains unencrypted: Which folder an item is in; what category (Login, Credit Card, …) an item belongs to; creation time; modify time; and last sync time.

The item UUIDs are fully available, which can be used to determine how many attachments, if any, an item has associated with it. The UUID of any folder an item belongs to is unencrypted, and thus an attacker can determine which items are in the same folder.

UUIDs are just arbitrary and randomly chosen numbers. The names of folders or items cannot be deduced from them.

1Password JavaScript Extension Data Format

We haven't carefully detailed the encryption and data format used by the browser extensions, because we change these more frequently. These are actually similar to what we have in the 1Password 4 Cloud Keychain Format, in that the "overview" information is encrypted and it uses 256-bit AES keys. In many ways, what we introduced in our 1Password JavaScript Extensions two years ago, was a testbed for things that we now use in the new synching format. Things like URL and Title are encrypted in the browser extensions.

There are some differences of course. Folders, attachments, and other things aren't included in the browser extensions. And because the extension data doesn't live on the cloud, we haven't (yet) brought authenticated encryption to it. And then there are just some annoying things. In the extensions, we used OpenSSL style salting, which we've finally weaned ourselves from with the 1Password 4 Cloud Keychain Format. But, as I said, we can tinker with and change those relatively quickly; so while we discuss and mention that database, we don't go to great lengths to document it.

I hope that this helps.

Cheers,

-j

–-
Jeffrey Goldberg
Chief Defender Against the Dark Arts @ AgileBits
http://agilebits.com

Valdemar1

Just to be completely up front, there is some metadata that is discernible from the 1Password 4 Cloud Keychain Format. The document that @Khad pointed you to spells these out:>

Yes, I read that as well. My biggest concerns, which this information addressed, are the availability of metadata related to URLs. Web-sites such as www.myfavoritepornsite.com or www.myhugeswissbankaccount.com are obviously not something that one wants others to get ahold of, particularly in this era of Prism (and spouse) spying. Not that I have such accounts, mind you. I'm just looking out for others.

khad

Not that I have such accounts, mind you. I'm just looking out for others.

Discussions of security and privacy bring out the altruism in all of us. :)