Signing back into the Community for the first time? You'll need to reset your password to access your account. Find out more.
Forum Discussion
Former Member
4 years agoTool for detecting and removing duplicates? [Available in 8.10.18]
I have several logins that somehow have been duplicated in 1Password 8. This may have happened in different times when I've attempted to migrate my 1Password 7 data to my 1Password.com account.
In any case, I have a load of duplicates in 1Password 8 and it would be very helpful if there were an efficient tool or feature to help me merge/dedupe these entries (and ideally test them to make sure that I'm preserving the correct credentials).
Any thoughts/ideas for how to handle this situation?
1Password Version: 1Password for Mac 8.5.0 80500017, on BETA channel
Extension Version: 2.1.4
OS Version: macOS 12.0.1 (21A559)
- Former Member
I was able to clean it up reasonably quickly (about 10 minutes), but I could imagine this being a lot worse for others. Ben - I know you're doing your best, and I imagine there are higher priority features out there, but I hope this does get resolved eventually.
- ctphotoOccasional Contributor
That is just a band-aide on the issue. How about you folks just fix it? You are programmers, right? You broke it, why should we have to fix it?
- 1P_Ben
1Password Team
Hi @dwabyick
Welcome to the 1Password Support Community.
This post contains the latest information we have on this subject. I'm sorry to hear you ended up with duplicated items in your vault. There are a number of ways 1Password can help you identify and clean up items that are duplicated:
- Watchtower's Reused Passwords checker
- Sort options: sort by title to spot similarly named items (e.g. multiple "Google" items)
- Creation and modification timestamps on item details
Please let us know if we can help you use any of these features. 😊
Ben
- Former Member
Hi team 1password -
I find this unacceptable that you don't have this feature.
I finally upgraded from 1password 6 to improve my experience, and in the process of the upgrading multiple devices it created duplicates in a single vault. Thus I have a degraded experience.
This is making me reconsider whether I want to continue my subscription to 1password. Unfortunate as I was planning on buying my wife a subscription too so we can share.
Could we get a target quarter where this feature will be available?
- Former Member
(NOTE: there was a typo in the previous posting - the original script is 1pcsv.pl not 1pecsv.pl - so the usage of the script has the wrong name in it, sorry)
Here's a modified version of the script that bases the duplications on the domain of the URL - i.e., it ignores:
* http:// vs https://
* www. at the beginning
* anything following the first '/' (after the http:// has been removed)so things like (space added to prevent these as showing up as links):
* http:/ /www.gianteagle.com/pharmacy
* https:/ /www.gianteagle.com/pharmacy
* https:/ /www.gianteagle.com/Sign-In/?return=https%3A%2F%2Fwww.gianteagle.com%2F
* http:/ /www.gianteagle.com
* https:/ /gianteagle.comWould all be treated as duplicates of one another for the "gianteagle.com" domain.
Not sure if this is better than the previous script or should be run before or after the previous script - but regardless, it provides another means of finding duplications that you might want to archive/delete.
Usage (after downloading the script and having your 1Password CSV exported data):
`cd ~/Downloads` `perl ./1p2csv.pl ./1PasswordExport-1234858.csv ./1pe2.csv1
In the end, you'd see:
`RESULTS IN: ./1pe2.csv`
Again, you can then load up the result file in Numbers (or Excel, etc.) to then manually process it, going back and forth between the spreadsheet and 1Password to archive/delete duplications.
Hope this helps (I'm still working through cleaning up my data)
- Former Member
I created a Perl script to process the exported CSV from 1Password to create a new CSV file sorted by duplicate count (in descending order so that the ones with duplicates appear above the ones without). You could then open this resultant CSV file in Numbers (or Excel, or similar) and then manually work your way down the list - searching for the given Title or Url and then examining (!) each entry to choose one to keep and then archive or delete the rest.
(! examine carefully because my code only looks at the first four fields: Title, Url, Username, and Password, it does NOT look at OTPAuth, Favorite, Archived, Tags, or Notes - so you might need to do some manual merging before archiving/deleting the redundant entries).Save the script to your Downloads folder (e.g.,
~/Downloads/1pecsv.pl
), Export your 1Password data as CSV (e.g.,~/Downloads/1PasswordExport-1234858.csv
) and you can then run it from the command line.cd ~/Downloads perl ./1pecsv.pl ./1PasswordExport-1234858.csv ./1pe.csv
In the end, you'd see:
RESULTS IN: ./1pe.csv
If you then go to ~/Downloads in your Finder, you should be able to click on that file (
1pe.csv
), and it will open in whatever application you have associated with CSV files. From there, it's a manual back-and-forth between that file and 1Password.Theoretically, one could modify the script to generate a unique listing, and then you could archive/delete everything you have in 1Password and Import the resultant CSV file - but I'm not willing to claim that would be safe.
If you don't have Perl installed on your computer you might have to download and install it - for that, I suggest you go through Google or whatever your web-browser-search-engine-of-choice is.
- voltsSuper Contributor
Can anybody comment about the "best" known workflow for doing this manually?
Export to this format: __________
Use this tool: ___________
Filter in Excel
Re-import - Former Member
This has become very noticeable after migrating to 1Password8:
* The lack of any duplicate detection
* The lack of merging multiple records for the same site that contain some unique information in the various instances
* The strange way that two identical records with identical passwords can have different ratings for the password strengthThe last would seem to qualify as an actual bug.
The first two would seem to qualify as much-requested and very useful features - Former Member
+1 for me too. I used to be a fervent supporter of 1Password (ever since 2006). I have to admit that I dread sorting out my some 3300 entries. I was used to a much responsive company/developer that could be entrusted with mac users passwords. Now... I have serious doubts. Please 1password developers, do something about the duplicate issue. It's important to mac, iPhone, iPad users. We don't need fancy stuff, just a password manager we can rely on.
- FrankyO1P
1Password Team
Hi folks,
Thank you for the feedback on how you ended up with duplicate entries. This feedback helps us understand how we can help prevent such situations in the future. We also understand how useful a deduplication feature would be as it would also solve future issues.
As Ben said in a previous reply, we'll continue to advocate for the feature to be created in the future. I also want to note that for anyone who hasn't previously had a vote filed, I've gone ahead and added your votes.
IDEA-I-615