Tool for detecting and removing duplicates? [Available in 8.10.18]

If not this, then what is on your roadmap that is more important?
From last release changelog -

We’ve added a tooltip to the delete buttons in the item editor.

We’ve fixed an issue that caused the scrollbar to display in dialogues where it should be hidden.

We’ve fixed an issue that caused signing in to accounts to be slow.

This is more important? Are you developing 1Password for your own pleasure it seems? Picking those random cosmetic issues to fix that nobody asked for. How about something people asking you about daily for a year?

We all who has this issue upgraded form 7 to 8 (that's what caused the duplication it seems), we paid you expecting at least same level of functionality as in 7. Not worse. And we put up for almost a year now not only with duplicates but way worse performance comparing to 7. Slow unlock was fixed just last week! But duplicates issue is way worse and should have been fixed first.

It seems that forum support being the messengers is not working. Perhaps actual manager of 1Password project has no idea about severity of our requests. Because if any sane product manager had this level of angst they would be fixing this top priority.


Obviously the forum support staff is not passing requests as they are made and not explaining to product manager what is actually happening here.

I am requesting to escalate this to your product manager please. Directly, not as you passing the message. we need them to see this.

Hi, I imported my passwords from Chrome and Apple Keychain so I could use 1password as my main, but now I've got hundreds of duplicates. Please make this feature happen! Even just a merge tool, for now, would be perfection!

Super inconvenient -- I upgraded to 1password8 and now have hundreds of duplicates. Please bring back the duplicate removal tool! This is very painful for people that have been using 1password for a long time. Please help us early adopters!

@ben Catching up on this thread, Its pretty bad. This is a complete fail.

Even worse, the lack of support for this is a sign the company is obviously making some bad choices. Is this what we get for being long term loyal customers? Yikes. Seems like time to move on.

More specifically this may be what moves me to one of the many many other solutions out there. Its hard to believe this has not been addressed yet. As a software developer this should be a simple fix: Allow the user to identify duplicate content, and remove it.

Duplicate content for me looks like:
- identical username + password + create/update dates.

A diff of the content would provide an exact match minus any internal IDs. Duplicate content. Why is this so difficult?

Allow the user to "remove duplicates" - and the discarded items could go in the trash (incase some accident). This should not take more than a day to fix, and will save your customers exponentially more time. As for how I got the duplicates, all I did was update to the latest version (a bad decision obviously).

If this isnt fixed soon I'm out -- along with my faith with the product, and my suggestion to anyone I know.

Hi folks,

Thank you for the feedback on how you ended up with duplicate entries. This feedback helps us understand how we can help prevent such situations in the future. We also understand how useful a deduplication feature would be as it would also solve future issues.

As Ben said in a previous reply, we'll continue to advocate for the feature to be created in the future. I also want to note that for anyone who hasn't previously had a vote filed, I've gone ahead and added your votes.

IDEA-I-615

+1 for me too. I used to be a fervent supporter of 1Password (ever since 2006). I have to admit that I dread sorting out my some 3300 entries. I was used to a much responsive company/developer that could be entrusted with mac users passwords. Now... I have serious doubts. Please 1password developers, do something about the duplicate issue. It's important to mac, iPhone, iPad users. We don't need fancy stuff, just a password manager we can rely on.

This has become very noticeable after migrating to 1Password8:
* The lack of any duplicate detection
* The lack of merging multiple records for the same site that contain some unique information in the various instances
* The strange way that two identical records with identical passwords can have different ratings for the password strength

The last would seem to qualify as an actual bug.
The first two would seem to qualify as much-requested and very useful features

Can anybody comment about the "best" known workflow for doing this manually?

Export to this format: __________
Use this tool: ___________
Filter in Excel
Re-import

I created a Perl script to process the exported CSV from 1Password to create a new CSV file sorted by duplicate count (in descending order so that the ones with duplicates appear above the ones without). You could then open this resultant CSV file in Numbers (or Excel, or similar) and then manually work your way down the list - searching for the given Title or Url and then examining (!) each entry to choose one to keep and then archive or delete the rest.
(! examine carefully because my code only looks at the first four fields: Title, Url, Username, and Password, it does NOT look at OTPAuth, Favorite, Archived, Tags, or Notes - so you might need to do some manual merging before archiving/deleting the redundant entries).

Save the script to your Downloads folder (e.g., ~/Downloads/1pecsv.pl), Export your 1Password data as CSV (e.g., ~/Downloads/1PasswordExport-1234858.csv) and you can then run it from the command line.

 cd ~/Downloads
 perl ./1pecsv.pl ./1PasswordExport-1234858.csv ./1pe.csv

In the end, you'd see:

 RESULTS IN: ./1pe.csv

If you then go to ~/Downloads in your Finder, you should be able to click on that file (1pe.csv), and it will open in whatever application you have associated with CSV files. From there, it's a manual back-and-forth between that file and 1Password.

Theoretically, one could modify the script to generate a unique listing, and then you could archive/delete everything you have in 1Password and Import the resultant CSV file - but I'm not willing to claim that would be safe.

If you don't have Perl installed on your computer you might have to download and install it - for that, I suggest you go through Google or whatever your web-browser-search-engine-of-choice is.

(NOTE: there was a typo in the previous posting - the original script is 1pcsv.pl not 1pecsv.pl - so the usage of the script has the wrong name in it, sorry)

Here's a modified version of the script that bases the duplications on the domain of the URL - i.e., it ignores:
* http:// vs https://
* www. at the beginning
* anything following the first '/' (after the http:// has been removed)

so things like (space added to prevent these as showing up as links):
* http:/ /www.gianteagle.com/pharmacy
* https:/ /www.gianteagle.com/pharmacy
* https:/ /www.gianteagle.com/Sign-In/?return=https%3A%2F%2Fwww.gianteagle.com%2F
* http:/ /www.gianteagle.com
* https:/ /gianteagle.com

Would all be treated as duplicates of one another for the "gianteagle.com" domain.

Not sure if this is better than the previous script or should be run before or after the previous script - but regardless, it provides another means of finding duplications that you might want to archive/delete.

Usage (after downloading the script and having your 1Password CSV exported data):

 `cd ~/Downloads`
 `perl ./1p2csv.pl ./1PasswordExport-1234858.csv ./1pe2.csv1

In the end, you'd see:

 `RESULTS IN: ./1pe2.csv`

Again, you can then load up the result file in Numbers (or Excel, etc.) to then manually process it, going back and forth between the spreadsheet and 1Password to archive/delete duplications.

Hope this helps (I'm still working through cleaning up my data)