Tool for detecting and removing duplicates? [Available in 8.10.18]

Hello everyone! 👋

With the release of the latest beta for Mac, Windows, and Linux you can now see a list of your duplicate items in the app. Choose “Items with duplicates” in your Watchtower dashboard to see or delete all of your duplicates.

Make sure that you're running the latest beta to see the feature. As always, please let us know if you find the tool useful and if you have any suggestions for the team. 🙂

-Dave

My Duplicates occurred when I upgraded from V7 standalone on my least used machine to V8. I was already using V8 on two other Macs an iPhone and two iPads. That upgrade process was terrible. It wanted me to create a new 1Password membership (I already have a membership which there was no obvious way to connect to.). I ended up downloading V8 and installing it. It said I had to convert my V7 data to V8. It was not at all clear that I could have just skipped that step. Now I've got about 800 duplicates. The upgrade process needs to be more clear so duplicates are not created in the first place. We also need a duplicate finder/remover tool.

In an ideal world, nobody will duplicate data at import time, but actually it will happen. I will not discuss how it can happen or can be avoided, only that it will happen - proof is this thread and others. Duplicate items are not always user error. Switching to a decent password manager often means merging from multiple sources, then consolidating: import from previous password manager plus Chrome's plus Firefox's plus Safari's internal password managers.

One does import once directly after account creation, then never again. You at Agilebits do it never again, so you don't see really any need for postprocessing aid. However, for a new user it's crucial, it's a barrier.

With duplicate items I mean login items that contain the same userid+password combination.

What I propose you can do about this:
- give an overview how many items match above description, in a similar way Watchtower does this. Also create filters, so the user can see a list of only matching entries.
- also give an overview how many items match with url+userid, but different password. These might represent current and obsolete entries of the same login. The category could be something like "unable to merge", "merge conflict" or "password mismatch". Also make a filter for these.
- explain if duplicates are distributed over multiple vaults. If 2 vaults contain the same, because the user imported the same 2 times, the user should see this and the easy fix for the user is to just delete one of these vaults
- make it clear over how many different vaults how many duplicated entries are distributed
- add the opportunity to "merge" duplicate entries in one vault, in spirit similar to functions to merge different address entries in address books in other apps. Merge only items within the same vault.
- this merging would be available from the context menu you get if you select multiple entries. Merging candidates are only taken from the entries the user selected.
- This merging functionality could find login entries with same userid+password combination, then choose the oldest entry as "merged" entry. Then merge all URLs from the other entries to this entry, the same with any custom fields and sections with different values (don't add custom fields with same title+value multiple times). And merge the password history of course. Also skip merging for history entries with identical password+timestamp combinations. After merging, add the tag "merged " to the merged entry (so the user can find all processed entries and can check if all is ok). Then delete the items their content was merged into the merged item.

It's an investment in getting new users. I bet the turnover rate from ex-Lastpass users would be higher if you had such a functionality.
When I was new, I merged entries from Password Safe and Chrome. I almost gave up, because it was so tedious - after all, Chrome autofill wasn't that bad, and Password Safe for just a local vault also wasn't that bad for all non-browser login items. A month after the trial expired I gave it a second shot and decided to do the tedious work.

Awesome sauce!

Indeed!

That's hilarious. I JUST wrote a CLI tool to do this after it drove me crazy enough. I should have checked here first!
For posterity's sake, here it is: https://github.com/Taytay/1password_utils

Hi @"Chris Messina",

Thanks for this question - it's one, as a person who somehow ends up with multiple copies of everything, that is near to my heart.

Duplicates in 1Password 8 for Mac will, for now, need to be done manually. You can do this by sorting items by Title (you can find this option at the top of your item list, by clicking the downward-pointing arrow).

You can then remove duplicates by selecting the desired item (or Ctrl + click to select multiple items). Then, right click one of the selected items and choose Delete.

I should say that we have heard from customers that a duplicate detective 🕵️‍♀️ feature would be helpful in 1Password 8 for situations like this (I agree!), and hear your request for it here as well. I'm happy to add your request to our internal discussion around that issue. We're always interested in making the app better, and we act on customer feedback often. 🙂

Thanks for letting us know how this could help for you. I'll pass it along!

ref: dev/projects/customer-feature-requests/#422

Oy, man that sucks. I literally have thousands of items and hundreds of dupes. I definitely need a more automated way to get my account back into a usable place. 😓

Yes, please do add my request to your internal discussion. Thanks!

1Password has had this problem for many years, dating back to 1Password 6, and they refuse to fix it. A search of the forums will show a lot of history. Plus I have years of begging them to fix it.

Hey folks,

I don't think it is a refusal to address the problem, though I understand how having not come up with a better solution can appear to be and I suppose in some ways is effectively a refusal. The issue isn't so much that we wouldn't like to build a robust duplicate detection and clean up system, it is that we recognize doing so would be a heavy lift to do right, and other tasks & features have taken priority. We did actually build a duplicate cleanup tool into 1Password 7 for Mac, but it was incredibly conservative in what it considered to be a duplicate and as such wasn't super helpful in many cases.

I wish I could promise that a great solution is coming, but I'm not in a position to do that. What I can say is that I'll continue to advocate for one, and I'll also take this opportunity to make sure our product team is aware of the level of demand for such a feature.

Ben