Tool for detecting and removing duplicates?

Chris MessinaChris Messina Member
Community Member

I have several logins that somehow have been duplicated in 1Password 8. This may have happened in different times when I've attempted to migrate my 1Password 7 data to my 1Password.com account.

In any case, I have a load of duplicates in 1Password 8 and it would be very helpful if there were an efficient tool or feature to help me merge/dedupe these entries (and ideally test them to make sure that I'm preserving the correct credentials).

Any thoughts/ideas for how to handle this situation?


1Password Version: 1Password for Mac 8.5.0 80500017, on BETA channel
Extension Version: 2.1.4
OS Version: macOS 12.0.1 (21A559)

View Approved Answer

«13

Comments

  • PeterG_1PPeterG_1P

    Team Member

    Hi @Chris Messina,

    Thanks for this question - it's one, as a person who somehow ends up with multiple copies of everything, that is near to my heart.

    Duplicates in 1Password 8 for Mac will, for now, need to be done manually. You can do this by sorting items by Title (you can find this option at the top of your item list, by clicking the downward-pointing arrow).

    You can then remove duplicates by selecting the desired item (or Ctrl + click to select multiple items). Then, right click one of the selected items and choose Delete.

    I should say that we have heard from customers that a duplicate detective 🕵️‍♀️ feature would be helpful in 1Password 8 for situations like this (I agree!), and hear your request for it here as well. I'm happy to add your request to our internal discussion around that issue. We're always interested in making the app better, and we act on customer feedback often. 🙂

    Thanks for letting us know how this could help for you. I'll pass it along!

    ref: dev/projects/customer-feature-requests/#422

  • Chris MessinaChris Messina Member
    Community Member

    Oy, man that sucks. I literally have thousands of items and hundreds of dupes. I definitely need a more automated way to get my account back into a usable place. 😓

    Yes, please do add my request to your internal discussion. Thanks!

  • spinozaspinoza Junior Member
    Community Member

    1Password has had this problem for many years, dating back to 1Password 6, and they refuse to fix it. A search of the forums will show a lot of history. Plus I have years of begging them to fix it.

  • BenBen AWS Team

    Team Member

    Hey folks,

    I don't think it is a refusal to address the problem, though I understand how having not come up with a better solution can appear to be and I suppose in some ways is effectively a refusal. The issue isn't so much that we wouldn't like to build a robust duplicate detection and clean up system, it is that we recognize doing so would be a heavy lift to do right, and other tasks & features have taken priority. We did actually build a duplicate cleanup tool into 1Password 7 for Mac, but it was incredibly conservative in what it considered to be a duplicate and as such wasn't super helpful in many cases.

    I wish I could promise that a great solution is coming, but I'm not in a position to do that. What I can say is that I'll continue to advocate for one, and I'll also take this opportunity to make sure our product team is aware of the level of demand for such a feature.

    Ben

  • Tertius3Tertius3
    Community Member

    Ability to sort or group by password would help with simple duplicate detection, since duplicate entries have duplicate passwords, so it would put duplicates next to each other, where I would be able to decide which entry is kept and which can be dropped. Sort by title is meaningless, because different (but duplicate) sources have different definition of what is a title (for some it's the url, for some it's the website, for some it's free text).

    Sort or group by (first listed) url could also help with duplicate detection, since most entries will have only one url, but there is no possibility to sort by url.

    In v7, searching for duplicates was a chore. In v8 it seems you can use the Watchtower/reused passwords view to group entries with identical passwords, but since I have cleaned up my database before using v8, I cannot say how this actually will work out.

  • spinozaspinoza Junior Member
    Community Member

    Ben, if you bring back the duplicate cleanup tool that was in version 7, and provide the option to ignore some of the insignificant metadata, that would be very helpful. As you point out, the tool in version 7 was unnecessarily conservative. Since providing that in version 8 may require recoding, even if you simply add that ability into version 7, or as a separate tool available from Cust Service, it would be really helpful.
    I think we who merged standalone vaults into the subscription service in when upgrading are the most severely affected.

    Thanks.

  • spinozaspinoza Junior Member
    Community Member

    Just fyi, when I imported my stand-alone vaults I wound up with several thousand duplicates, and no way to remove them except one-by-one. There was no warning at the time that duplicates would not be merged, with no ability to dedupe.

  • ag_anaag_ana
    1Password Alumni

    @spinoza:

    Just fyi, when I imported my stand-alone vaults I wound up with several thousand duplicates, and no way to remove them except one-by-one.

    Make sure you take a look at the list of tags in the 1Password app: every round of import you performed should have been assigned a specific tag with the import date, so you can select all duplicates at once.

  • Chris MessinaChris Messina Member
    Community Member

    @ag_ana man, that would be useful if that were the case, but with my dupes, that isn't the case.

    To @spinoza's point: "we who merged standalone vaults into the subscription service in when upgrading are the most severely affected."

    This is exactly right. I feel like I was punished for trying to migrate to the new subscription model — and now I'm stuck in a situation that seems I can't recover from.

  • BenBen AWS Team

    Team Member

    @Chris Messina

    That's a real bummer. I'm sorry. 😞 There are a number of different ways duplicates (or similar items) can occur, and as a result it is likely that more than one solution will be required.

    1. For imports (v7), as Ana mentioned, a tag is created for the imported items. This makes it easy to identify which items were imported when, and if an import is accidentally performed twice it'd be possible to simply delete all of the items with one tag or the other. v8 does not yet support importing, but hopefully we'll do the same or similar there.
    2. For migrations from standalone, we now write out a hidden "migration receipt," which tracks what has been migrated, in an attempt to prevent duplicate migrations. This does break down if that receipt, stored in each standalone vault, doesn't get synced between your devices, though.
    3. The browser extension(s) used to lead to duplicate items being saved, though we've made improvements in this regard and I rarely see opportunity for duplicates since the in-line menu. It is possible though that there is still room for further improvement here.

    One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place. We've made some good changes in this regard as outlined above, but it'd be great to do more. Of course that doesn't help folks that already have duplicates, and I hope we can address that as well. I don't know of anything definite planned at this time, but I will continue to push for a solution.

    Ben

  • Chris MessinaChris Messina Member
    Community Member

    One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place.

    Yes, that would be helpful.

    Of course that doesn't help folks that already have duplicates, and I hope we can address that as well.

    But no, it wouldn't help in my case. Such it is.

    I will point out that I don't recall ever "importing" entries through "the front door" as it were ... instead I dragged and dropped thousands of entries from my local vault into my 1password.com vaults — but since I couldn't tell which ones I'd saved locally vs saved in my synced vaults, I assumed that 1Password would maintain internal UUIDs for each item, and catch the conflicts automatically and ask me what I wanted to do with them.

    Clearly that didn't happen — so I just want to point out that I didn't "import" in any conventional way, but used drag and drop between vaults to fuck myself over.

  • Chris MessinaChris Messina Member
    Community Member

    One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place.

    Yes, that would be helpful.

    Of course that doesn't help folks that already have duplicates, and I hope we can address that as well.

    But no, it wouldn't help in my case. Such it is.

    I will point out that I don't recall ever "importing" entries through "the front door" as it were ... instead I dragged and dropped thousands of entries from my local vault into my 1password.com vaults — but since I couldn't tell which ones I'd saved locally vs saved in my synced vaults, I assumed that 1Password would maintain internal UUIDs for each item, and catch the conflicts automatically. That clearly didn't happen — and so when I dragged and dropped items from one vault into another, I screwed myself over in ways that I didn't anticipate.

    Perhaps that's something that should be considered — and a warning or prompt added when >100 items are dragged from one vault to another... that duplicates might be created?

  • Dayton_agDayton_ag

    Team Member

    I agree, duplications (past and present) is definitely something that we can do better at.

    Part of the big change to 1Password 8 was to stop developing separate apps with different feature-sets (see duplication tool only available in 1Password for Mac) and implement a common core - this allows us to implement a feature once and have it work effortlessly on all platforms at the same time.

    I'm hopeful that as we keep working along that vein a more robust duplication tool can be implemented, but as Ben mentioned we are unfortunately unable to offer any promises for the future. Know that we're also very interested in improvements in this area, and we'll keep advocating for it on our end. :smile:

  • WaltDittrichWaltDittrich
    Community Member

    I came here because I, too, moved my Primary vault items to my Private folder. 850 items, all seemed to be duplicates. Fortunately, I'm still with 1Password 7 for Mac, so my removal tool took away about 550.

    @Chris Messina , I think one method for you might be to Move everything over to a new Vault. Then, as you use items, move them back into your Private vault. This will cut out you seeing the duplicates as you slowly build your Private vault back up.
    It's sort of like moving everything from your e-mail Inbox to a separate folder. Then work towards keeping your Inbox emptied. The stuff is all still there, but not right in front of your face causing havoc.
    (Gosh I hope this makes sense.)

  • Chris MessinaChris Messina Member
    Community Member

    Hey @WaltDittrich — that makes sense but is a bit impractical given that I've got 6000+ items (with the dupes. That said, I have created a few new vaults (for Work and Family sharing), and those are indeed much cleaner!

    All in all, I can search and generally get to the items I need... the real bother is when I go to update a password and I have to pick the item I want to hold on to! It can be hard to sort through the recent items when all these dupes are floating around.

  • PeterG_1PPeterG_1P

    Team Member
    edited April 23

    Just wanted to say thanks for the information on your use cases here, folks. This is an area we hope to improve and we appreciate the chance to understand the scenarios you're running into!

    ref: IDEA-I-615

  • cornellncornelln
    Community Member
    edited May 27

    Several of the people here are in this situation due to some issue occurring on migration from 1Password 7 to 1Password 8. This issue on its own is a failure of a critical part of the migration function and seems unaddressed in the thread. The system quite simply be checking much better than it does to avoid that fate and recover from it.

    The fact that manual duplicate clean up is the solution on a database that for many can approach 1k or greater is simply not good enough. If migration issues occur the duplicate search should really be moved up your priority queue.

    Lastly I ended up just deleting everything from the account online and then re-importing. This would be my suggestion to anyone who identifies this problem before using the new data base too much.

    This issue alone mostly just got me thinking again about bailing on the product all together. After all it's simply database app at it's core function and if it can't be trusted w that what's the use?

    In general the migration process didn't feel well designed to me. For example why would the system not remind the user to go ahead and detach the other vault? This upgrade process is not good, keep working on that or just expect additional users in here over time complaining about that (well the one's that bother to give feedback).
    ...
    For example presently on iOS I have 2 Vaults in iOS 7. This is how the UI is confusing

    All Vaults Screen
    1. "Vault for Saving" was still set to iCloud... Not updated to save to the online account... So right there if I didn't change that I would have the issue of saving newly created passwords to a vault I am migrating off of.

    1. The old vault was still "On" so of course you end up w a confusing set of duplicates that way too

    2. The naming structure is just confusing to someone not working for the company.

    "Vaults included in all vaults"
    You have
    [FirstName LastName] ON
    Personal ON

    On iCloud
    Primary ON

    Let me ask you do you think that the average non technical user can tell you quickly which vault is which w this kind of naming structure? Or that one is their old vault from 7 vs where they are trying to migrate too?

    In short your UI and migration does the user very little favors to be honest to the point its borderline user hostile. What % of users hose their migration 10-20%?
    ...
    Another issue you have a promotional upsell function where users can get a discounted price - that's fine... But does this disappear after you pay and sign in with an account - no it doesn't. So is there no QA validation of something as simple as 'if user has paid account stop displaying the promotional notice'... Guess not? Seems like a simple nuance but it's not accounted for.

  • lyricnz2lyricnz2
    Community Member
    edited June 6

    Yes, this lack of function is unfortunate!
    If you have decent password hygiene, you might be able to find the duplicates easier in the Watchtower->Duplicate Passwords view.

  • jordazzlejordazzle
    Community Member

    @Chris Messina thanks for all your posts and bringing this to light. I'm in the exact same boat as you. I used the standalone version of 1password for many many years. And finally decided... I get enough value out of this that I don't mind paying a small monthly fee for the one software application that I use every single day, all day long.

    When we made the move to the new cloud subscription service everything seemed okay. But now as I'm going to use a login identity I'm seeing at least two versions of the exact same credentials.

    I recognize this may take some significant development to build a duplicate item cleanup tool, but I think it would really helpful to streamline our online identities.

  • Chris MessinaChris Messina Member
    Community Member

    Yep — and it's gotten worse as I update passwords... now I have TWO entries and especially when they sync to my phone, it's nearly IMPOSSIBLE to tell which is the correct/current credential! 😣

  • spinozaspinoza Junior Member
    Community Member

    We've been raising this problem for many years. 1Password refuses to allow a cleanup duplicates tool (they allowed a very limited one in 1Password 7, then abandoned it upon release of 8.) New staff members pop up here every few years, and say they will suggest it to the development team. I've been using 1Password since the beginning. I acquired 6,000 duplicates when they switched to subscription for 1Pass6, and have been manually deleting one-at-a-time ever since - down to 3,300 now!!

    I'm hoping to ditch this crappy company when I can -- they abandoned me years ago. Their goal is to lock you to their ecosystem, while pretending to be good guys :-( Luckily Apple and Google are stepping up their game.

  • PeterG_1PPeterG_1P

    Team Member

    Hi folks, thanks for your feedback here. This is certainly an issue I'm happy to advocate for on your behalf.

    To @spinoza's point: 1Password has grown incredibly fast in the last few years, and we've re-organized and re-thought the way we prioritize and triage issues along the way. What I mean by this is: just because something was requested before and hasn't happened, doesn't mean it's off the table. While I don't have anything specific to share on this feature request at the moment, I can tell you honestly that we in customer service are tallying the votes and that community demand does matter.

    So if this is something you feel would be useful for you, and you haven't let us know already, I welcome you to do so. I'll advocate for your use case in any way I can.

    Speaking of which, all the votes from this thread which were not previously added have now been tallied (and we already had votes for @Chris Messina, @spinoza, and @cornelln). 👍

  • czyzczyzczyzczyz
    Community Member

    Please add a vote for me. In the migration from 1password 7 to v8 I've ended up with thousands of duplicates. I've been using 1password since February of 2008 and have never run into this kind of frustration.

    I spent a half hour manually command-clicking and deleting duplicates and barely made a dent in it. The duplicates crowd my view of passwords whenever I try to use 1password to login to a site.

    The duplicates are all exact dupes, with all entries and content and the same modification and creation timestamps. There wouldn't be a need for a tool that does fancy comparisons between records -- just quickly scan through all of them and generate a hash for each, then delete one of each matching pair. Maybe I'll see if I can figure out a way to automate this with 1password-cli.

  • OlivierPOlivierP
    Community Member

    Hi,
    I just enrolled and onborded my colleagues in 1P8 and they faced the duplicate issue.
    For some who were using Edge and Chrome which the same credentials in each: they've got a lot of duplicates.
    Unfortunately, the only solution I can them is is to sort this by hand...
    Not a really good start, they would say.
    So our voice for a solution to this.

  • PeterG_1PPeterG_1P

    Team Member

    Hi @olivierP and @czyzczyz, I have added votes on the issue for each of you in turn. Thank you for taking the time to let us know why you find value in that feature and the specific circumstances you've run into. I'm passing it on to our development team now and appreciate your sharing this with us - and the thoughts on how a solution could be implemented.

    ref: IDEA-I-615

  • ctphotoctphoto
    Community Member

    Has this still not been fixed yet? Wasn't there any beta testing to discover the duplicate issue before it happened to us paying customers?

  • PeterG_1PPeterG_1P

    Team Member

    Hi @ctphoto, I don't have any updates to supply at the moment but can tell you that I'm happy to pass on your feedback - and have now done so. Thanks for letting us know this is a priority for you. 🙏

  • mikesilvamikesilva
    Community Member

    +1 on this issue for me.

  • ag_tommyag_tommy

    Team Member

    👍 ✏️

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file