Tool for detecting and removing duplicates? [Available in 8.10.18]

Ability to sort or group by password would help with simple duplicate detection, since duplicate entries have duplicate passwords, so it would put duplicates next to each other, where I would be able to decide which entry is kept and which can be dropped. Sort by title is meaningless, because different (but duplicate) sources have different definition of what is a title (for some it's the url, for some it's the website, for some it's free text).

Sort or group by (first listed) url could also help with duplicate detection, since most entries will have only one url, but there is no possibility to sort by url.

In v7, searching for duplicates was a chore. In v8 it seems you can use the Watchtower/reused passwords view to group entries with identical passwords, but since I have cleaned up my database before using v8, I cannot say how this actually will work out.

Ben, if you bring back the duplicate cleanup tool that was in version 7, and provide the option to ignore some of the insignificant metadata, that would be very helpful. As you point out, the tool in version 7 was unnecessarily conservative. Since providing that in version 8 may require recoding, even if you simply add that ability into version 7, or as a separate tool available from Cust Service, it would be really helpful.
I think we who merged standalone vaults into the subscription service in when upgrading are the most severely affected.

Thanks.

Just fyi, when I imported my stand-alone vaults I wound up with several thousand duplicates, and no way to remove them except one-by-one. There was no warning at the time that duplicates would not be merged, with no ability to dedupe.

spinoza:

Just fyi, when I imported my stand-alone vaults I wound up with several thousand duplicates, and no way to remove them except one-by-one.

Make sure you take a look at the list of tags in the 1Password app: every round of import you performed should have been assigned a specific tag with the import date, so you can select all duplicates at once.

@ag_ana man, that would be useful if that were the case, but with my dupes, that isn't the case.

To spinoza's point: "we who merged standalone vaults into the subscription service in when upgrading are the most severely affected."

This is exactly right. I feel like I was punished for trying to migrate to the new subscription model — and now I'm stuck in a situation that seems I can't recover from.

@"Chris Messina"

That's a real bummer. I'm sorry. 😞 There are a number of different ways duplicates (or similar items) can occur, and as a result it is likely that more than one solution will be required.

1. For imports (v7), as Ana mentioned, a tag is created for the imported items. This makes it easy to identify which items were imported when, and if an import is accidentally performed twice it'd be possible to simply delete all of the items with one tag or the other. v8 does not yet support importing, but hopefully we'll do the same or similar there.
2. For migrations from standalone, we now write out a hidden "migration receipt," which tracks what has been migrated, in an attempt to prevent duplicate migrations. This does break down if that receipt, stored in each standalone vault, doesn't get synced between your devices, though.
3. The browser extension(s) used to lead to duplicate items being saved, though we've made improvements in this regard and I rarely see opportunity for duplicates since the in-line menu. It is possible though that there is still room for further improvement here.

One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place. We've made some good changes in this regard as outlined above, but it'd be great to do more. Of course that doesn't help folks that already have duplicates, and I hope we can address that as well. I don't know of anything definite planned at this time, but I will continue to push for a solution.

Ben

One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place.

Yes, that would be helpful.

Of course that doesn't help folks that already have duplicates, and I hope we can address that as well.

But no, it wouldn't help in my case. Such it is.

I will point out that I don't recall ever "importing" entries through "the front door" as it were ... instead I dragged and dropped thousands of entries from my local vault into my 1password.com vaults — but since I couldn't tell which ones I'd saved locally vs saved in my synced vaults, I assumed that 1Password would maintain internal UUIDs for each item, and catch the conflicts automatically and ask me what I wanted to do with them.

Clearly that didn't happen — so I just want to point out that I didn't "import" in any conventional way, but used drag and drop between vaults to fuck myself over.

One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place.

Yes, that would be helpful.

Of course that doesn't help folks that already have duplicates, and I hope we can address that as well.

But no, it wouldn't help in my case. Such it is.

I will point out that I don't recall ever "importing" entries through "the front door" as it were ... instead I dragged and dropped thousands of entries from my local vault into my 1password.com vaults — but since I couldn't tell which ones I'd saved locally vs saved in my synced vaults, I assumed that 1Password would maintain internal UUIDs for each item, and catch the conflicts automatically. That clearly didn't happen — and so when I dragged and dropped items from one vault into another, I screwed myself over in ways that I didn't anticipate.

Perhaps that's something that should be considered — and a warning or prompt added when >100 items are dragged from one vault to another... that duplicates might be created?

I agree, duplications (past and present) is definitely something that we can do better at.

Part of the big change to 1Password 8 was to stop developing separate apps with different feature-sets (see duplication tool only available in 1Password for Mac) and implement a common core - this allows us to implement a feature once and have it work effortlessly on all platforms at the same time.

I'm hopeful that as we keep working along that vein a more robust duplication tool can be implemented, but as Ben mentioned we are unfortunately unable to offer any promises for the future. Know that we're also very interested in improvements in this area, and we'll keep advocating for it on our end. :smile:

I came here because I, too, moved my Primary vault items to my Private folder. 850 items, all seemed to be duplicates. Fortunately, I'm still with 1Password 7 for Mac, so my removal tool took away about 550.

@"Chris Messina" , I think one method for you might be to Move everything over to a new Vault. Then, as you use items, move them back into your Private vault. This will cut out you seeing the duplicates as you slowly build your Private vault back up.
It's sort of like moving everything from your e-mail Inbox to a separate folder. Then work towards keeping your Inbox emptied. The stuff is all still there, but not right in front of your face causing havoc.
(Gosh I hope this makes sense.)