Tool for detecting and removing duplicates? [Available in 8.10.18]

I was able to clean it up reasonably quickly (about 10 minutes), but I could imagine this being a lot worse for others. Ben - I know you're doing your best, and I imagine there are higher priority features out there, but I hope this does get resolved eventually.

That is just a band-aide on the issue. How about you folks just fix it? You are programmers, right? You broke it, why should we have to fix it?

Hi @dwabyick

Welcome to the 1Password Support Community.

This post contains the latest information we have on this subject. I'm sorry to hear you ended up with duplicated items in your vault. There are a number of ways 1Password can help you identify and clean up items that are duplicated:

• Watchtower's Reused Passwords checker
• Sort options: sort by title to spot similarly named items (e.g. multiple "Google" items)
• Creation and modification timestamps on item details

Please let us know if we can help you use any of these features. 😊

Ben

Hi team 1password -

I find this unacceptable that you don't have this feature.

I finally upgraded from 1password 6 to improve my experience, and in the process of the upgrading multiple devices it created duplicates in a single vault. Thus I have a degraded experience.

This is making me reconsider whether I want to continue my subscription to 1password. Unfortunate as I was planning on buying my wife a subscription too so we can share.

Could we get a target quarter where this feature will be available?

(NOTE: there was a typo in the previous posting - the original script is 1pcsv.pl not 1pecsv.pl - so the usage of the script has the wrong name in it, sorry)

Here's a modified version of the script that bases the duplications on the domain of the URL - i.e., it ignores:
* http:// vs https://
* www. at the beginning
* anything following the first '/' (after the http:// has been removed)

so things like (space added to prevent these as showing up as links):
* http:/ /www.gianteagle.com/pharmacy
* https:/ /www.gianteagle.com/pharmacy
* https:/ /www.gianteagle.com/Sign-In/?return=https%3A%2F%2Fwww.gianteagle.com%2F
* http:/ /www.gianteagle.com
* https:/ /gianteagle.com

Would all be treated as duplicates of one another for the "gianteagle.com" domain.

Not sure if this is better than the previous script or should be run before or after the previous script - but regardless, it provides another means of finding duplications that you might want to archive/delete.

Usage (after downloading the script and having your 1Password CSV exported data):

 `cd ~/Downloads`
 `perl ./1p2csv.pl ./1PasswordExport-1234858.csv ./1pe2.csv1

In the end, you'd see:

 `RESULTS IN: ./1pe2.csv`

Again, you can then load up the result file in Numbers (or Excel, etc.) to then manually process it, going back and forth between the spreadsheet and 1Password to archive/delete duplications.

Hope this helps (I'm still working through cleaning up my data)

I created a Perl script to process the exported CSV from 1Password to create a new CSV file sorted by duplicate count (in descending order so that the ones with duplicates appear above the ones without). You could then open this resultant CSV file in Numbers (or Excel, or similar) and then manually work your way down the list - searching for the given Title or Url and then examining (!) each entry to choose one to keep and then archive or delete the rest.
(! examine carefully because my code only looks at the first four fields: Title, Url, Username, and Password, it does NOT look at OTPAuth, Favorite, Archived, Tags, or Notes - so you might need to do some manual merging before archiving/deleting the redundant entries).

Save the script to your Downloads folder (e.g., ~/Downloads/1pecsv.pl), Export your 1Password data as CSV (e.g., ~/Downloads/1PasswordExport-1234858.csv) and you can then run it from the command line.

 cd ~/Downloads
 perl ./1pecsv.pl ./1PasswordExport-1234858.csv ./1pe.csv

In the end, you'd see:

 RESULTS IN: ./1pe.csv

If you then go to ~/Downloads in your Finder, you should be able to click on that file (1pe.csv), and it will open in whatever application you have associated with CSV files. From there, it's a manual back-and-forth between that file and 1Password.

Theoretically, one could modify the script to generate a unique listing, and then you could archive/delete everything you have in 1Password and Import the resultant CSV file - but I'm not willing to claim that would be safe.

If you don't have Perl installed on your computer you might have to download and install it - for that, I suggest you go through Google or whatever your web-browser-search-engine-of-choice is.

Can anybody comment about the "best" known workflow for doing this manually?

Export to this format: __________
Use this tool: ___________
Filter in Excel
Re-import

This has become very noticeable after migrating to 1Password8:
* The lack of any duplicate detection
* The lack of merging multiple records for the same site that contain some unique information in the various instances
* The strange way that two identical records with identical passwords can have different ratings for the password strength

The last would seem to qualify as an actual bug.
The first two would seem to qualify as much-requested and very useful features

+1 for me too. I used to be a fervent supporter of 1Password (ever since 2006). I have to admit that I dread sorting out my some 3300 entries. I was used to a much responsive company/developer that could be entrusted with mac users passwords. Now... I have serious doubts. Please 1password developers, do something about the duplicate issue. It's important to mac, iPhone, iPad users. We don't need fancy stuff, just a password manager we can rely on.

Hi folks,

Thank you for the feedback on how you ended up with duplicate entries. This feedback helps us understand how we can help prevent such situations in the future. We also understand how useful a deduplication feature would be as it would also solve future issues.

As Ben said in a previous reply, we'll continue to advocate for the feature to be created in the future. I also want to note that for anyone who hasn't previously had a vote filed, I've gone ahead and added your votes.

IDEA-I-615