Tool for detecting and removing duplicates?

I have several logins that somehow have been duplicated in 1Password 8. This may have happened in different times when I've attempted to migrate my 1Password 7 data to my 1Password.com account.

In any case, I have a load of duplicates in 1Password 8 and it would be very helpful if there were an efficient tool or feature to help me merge/dedupe these entries (and ideally test them to make sure that I'm preserving the correct credentials).

Any thoughts/ideas for how to handle this situation?


1Password Version: 1Password for Mac 8.5.0 80500017, on BETA channel
Extension Version: 2.1.4
OS Version: macOS 12.0.1 (21A559)

Comments

  • PeterG_1PPeterG_1P

    Team Member

    Hi @Chris Messina,

    Thanks for this question - it's one, as a person who somehow ends up with multiple copies of everything, that is near to my heart.

    Duplicates in 1Password 8 for Mac will, for now, need to be done manually. You can do this by sorting items by Title (you can find this option at the top of your item list, by clicking the downward-pointing arrow).

    You can then remove duplicates by selecting the desired item (or Ctrl + click to select multiple items). Then, right click one of the selected items and choose Delete.

    I should say that we have heard from customers that a duplicate detective πŸ•΅οΈβ€β™€οΈ feature would be helpful in 1Password 8 for situations like this (I agree!), and hear your request for it here as well. I'm happy to add your request to our internal discussion around that issue. We're always interested in making the app better, and we act on customer feedback often. πŸ™‚

    Thanks for letting us know how this could help for you. I'll pass it along!

    ref: dev/projects/customer-feature-requests/#422

  • Oy, man that sucks. I literally have thousands of items and hundreds of dupes. I definitely need a more automated way to get my account back into a usable place. πŸ˜“

    Yes, please do add my request to your internal discussion. Thanks!

  • spinozaspinoza Junior Member ✭

    1Password has had this problem for many years, dating back to 1Password 6, and they refuse to fix it. A search of the forums will show a lot of history. Plus I have years of begging them to fix it.

  • BenBen AWS Team

    Team Member

    Hey folks,

    I don't think it is a refusal to address the problem, though I understand how having not come up with a better solution can appear to be and I suppose in some ways is effectively a refusal. The issue isn't so much that we wouldn't like to build a robust duplicate detection and clean up system, it is that we recognize doing so would be a heavy lift to do right, and other tasks & features have taken priority. We did actually build a duplicate cleanup tool into 1Password 7 for Mac, but it was incredibly conservative in what it considered to be a duplicate and as such wasn't super helpful in many cases.

    I wish I could promise that a great solution is coming, but I'm not in a position to do that. What I can say is that I'll continue to advocate for one, and I'll also take this opportunity to make sure our product team is aware of the level of demand for such a feature.

    Ben

  • Ability to sort or group by password would help with simple duplicate detection, since duplicate entries have duplicate passwords, so it would put duplicates next to each other, where I would be able to decide which entry is kept and which can be dropped. Sort by title is meaningless, because different (but duplicate) sources have different definition of what is a title (for some it's the url, for some it's the website, for some it's free text).

    Sort or group by (first listed) url could also help with duplicate detection, since most entries will have only one url, but there is no possibility to sort by url.

    In v7, searching for duplicates was a chore. In v8 it seems you can use the Watchtower/reused passwords view to group entries with identical passwords, but since I have cleaned up my database before using v8, I cannot say how this actually will work out.

  • spinozaspinoza Junior Member ✭

    Ben, if you bring back the duplicate cleanup tool that was in version 7, and provide the option to ignore some of the insignificant metadata, that would be very helpful. As you point out, the tool in version 7 was unnecessarily conservative. Since providing that in version 8 may require recoding, even if you simply add that ability into version 7, or as a separate tool available from Cust Service, it would be really helpful.
    I think we who merged standalone vaults into the subscription service in when upgrading are the most severely affected.

    Thanks.

  • spinozaspinoza Junior Member ✭

    Just fyi, when I imported my stand-alone vaults I wound up with several thousand duplicates, and no way to remove them except one-by-one. There was no warning at the time that duplicates would not be merged, with no ability to dedupe.

  • ag_anaag_ana 1Password Alumni

    @spinoza:

    Just fyi, when I imported my stand-alone vaults I wound up with several thousand duplicates, and no way to remove them except one-by-one.

    Make sure you take a look at the list of tags in the 1Password app: every round of import you performed should have been assigned a specific tag with the import date, so you can select all duplicates at once.

  • @ag_ana man, that would be useful if that were the case, but with my dupes, that isn't the case.

    To @spinoza's point: "we who merged standalone vaults into the subscription service in when upgrading are the most severely affected."

    This is exactly right. I feel like I was punished for trying to migrate to the new subscription model β€” and now I'm stuck in a situation that seems I can't recover from.

  • BenBen AWS Team

    Team Member

    @Chris Messina

    That's a real bummer. I'm sorry. 😞 There are a number of different ways duplicates (or similar items) can occur, and as a result it is likely that more than one solution will be required.

    1. For imports (v7), as Ana mentioned, a tag is created for the imported items. This makes it easy to identify which items were imported when, and if an import is accidentally performed twice it'd be possible to simply delete all of the items with one tag or the other. v8 does not yet support importing, but hopefully we'll do the same or similar there.
    2. For migrations from standalone, we now write out a hidden "migration receipt," which tracks what has been migrated, in an attempt to prevent duplicate migrations. This does break down if that receipt, stored in each standalone vault, doesn't get synced between your devices, though.
    3. The browser extension(s) used to lead to duplicate items being saved, though we've made improvements in this regard and I rarely see opportunity for duplicates since the in-line menu. It is possible though that there is still room for further improvement here.

    One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place. We've made some good changes in this regard as outlined above, but it'd be great to do more. Of course that doesn't help folks that already have duplicates, and I hope we can address that as well. I don't know of anything definite planned at this time, but I will continue to push for a solution.

    Ben

  • One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place.

    Yes, that would be helpful.

    Of course that doesn't help folks that already have duplicates, and I hope we can address that as well.

    But no, it wouldn't help in my case. Such it is.

    I will point out that I don't recall ever "importing" entries through "the front door" as it were ... instead I dragged and dropped thousands of entries from my local vault into my 1password.com vaults β€” but since I couldn't tell which ones I'd saved locally vs saved in my synced vaults, I assumed that 1Password would maintain internal UUIDs for each item, and catch the conflicts automatically and ask me what I wanted to do with them.

    Clearly that didn't happen β€” so I just want to point out that I didn't "import" in any conventional way, but used drag and drop between vaults to fuck myself over.

  • One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place.

    Yes, that would be helpful.

    Of course that doesn't help folks that already have duplicates, and I hope we can address that as well.

    But no, it wouldn't help in my case. Such it is.

    I will point out that I don't recall ever "importing" entries through "the front door" as it were ... instead I dragged and dropped thousands of entries from my local vault into my 1password.com vaults β€” but since I couldn't tell which ones I'd saved locally vs saved in my synced vaults, I assumed that 1Password would maintain internal UUIDs for each item, and catch the conflicts automatically. That clearly didn't happen β€” and so when I dragged and dropped items from one vault into another, I screwed myself over in ways that I didn't anticipate.

    Perhaps that's something that should be considered β€” and a warning or prompt added when >100 items are dragged from one vault to another... that duplicates might be created?

  • Dayton_agDayton_ag

    Team Member

    I agree, duplications (past and present) is definitely something that we can do better at.

    Part of the big change to 1Password 8 was to stop developing separate apps with different feature-sets (see duplication tool only available in 1Password for Mac) and implement a common core - this allows us to implement a feature once and have it work effortlessly on all platforms at the same time.

    I'm hopeful that as we keep working along that vein a more robust duplication tool can be implemented, but as Ben mentioned we are unfortunately unable to offer any promises for the future. Know that we're also very interested in improvements in this area, and we'll keep advocating for it on our end. :smile:

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file