Tool for detecting and removing duplicates? [Available in 8.10.18]

Chris Messina

I have several logins that somehow have been duplicated in 1Password 8. This may have happened in different times when I've attempted to migrate my 1Password 7 data to my 1Password.com account.

In any case, I have a load of duplicates in 1Password 8 and it would be very helpful if there were an efficient tool or feature to help me merge/dedupe these entries (and ideally test them to make sure that I'm preserving the correct credentials).

Any thoughts/ideas for how to handle this situation?

---

1Password Version: 1Password for Mac 8.5.0 80500017, on BETA channel
Extension Version: 2.1.4
OS Version: macOS 12.0.1 (21A559)

PeterG_1P

Hi @Chris Messina,

Thanks for this question - it's one, as a person who somehow ends up with multiple copies of everything, that is near to my heart.

Duplicates in 1Password 8 for Mac will, for now, need to be done manually. You can do this by sorting items by Title (you can find this option at the top of your item list, by clicking the downward-pointing arrow).

You can then remove duplicates by selecting the desired item (or Ctrl + click to select multiple items). Then, right click one of the selected items and choose Delete.

I should say that we have heard from customers that a duplicate detective 🕵️‍♀️ feature would be helpful in 1Password 8 for situations like this (I agree!), and hear your request for it here as well. I'm happy to add your request to our internal discussion around that issue. We're always interested in making the app better, and we act on customer feedback often. 🙂

Thanks for letting us know how this could help for you. I'll pass it along!

ref: dev/projects/customer-feature-requests/#422

Chris Messina

Oy, man that sucks. I literally have thousands of items and hundreds of dupes. I definitely need a more automated way to get my account back into a usable place. 😓

Yes, please do add my request to your internal discussion. Thanks!

Unknown

This content has been removed.

Ben

Hey folks,

I don't think it is a refusal to address the problem, though I understand how having not come up with a better solution can appear to be and I suppose in some ways is effectively a refusal. The issue isn't so much that we wouldn't like to build a robust duplicate detection and clean up system, it is that we recognize doing so would be a heavy lift to do right, and other tasks & features have taken priority. We did actually build a duplicate cleanup tool into 1Password 7 for Mac, but it was incredibly conservative in what it considered to be a duplicate and as such wasn't super helpful in many cases.

I wish I could promise that a great solution is coming, but I'm not in a position to do that. What I can say is that I'll continue to advocate for one, and I'll also take this opportunity to make sure our product team is aware of the level of demand for such a feature.

Ben

Tertius3

Ability to sort or group by password would help with simple duplicate detection, since duplicate entries have duplicate passwords, so it would put duplicates next to each other, where I would be able to decide which entry is kept and which can be dropped. Sort by title is meaningless, because different (but duplicate) sources have different definition of what is a title (for some it's the url, for some it's the website, for some it's free text).

Sort or group by (first listed) url could also help with duplicate detection, since most entries will have only one url, but there is no possibility to sort by url.

In v7, searching for duplicates was a chore. In v8 it seems you can use the Watchtower/reused passwords view to group entries with identical passwords, but since I have cleaned up my database before using v8, I cannot say how this actually will work out.

Unknown

This content has been removed.

Unknown

This content has been removed.

ag_ana

@spinoza:

Just fyi, when I imported my stand-alone vaults I wound up with several thousand duplicates, and no way to remove them except one-by-one.

Make sure you take a look at the list of tags in the 1Password app: every round of import you performed should have been assigned a specific tag with the import date, so you can select all duplicates at once.

Chris Messina

@ag_ana man, that would be useful if that were the case, but with my dupes, that isn't the case.

To @spinoza's point: "we who merged standalone vaults into the subscription service in when upgrading are the most severely affected."

This is exactly right. I feel like I was punished for trying to migrate to the new subscription model — and now I'm stuck in a situation that seems I can't recover from.

Ben

@Chris Messina

That's a real bummer. I'm sorry. 😞 There are a number of different ways duplicates (or similar items) can occur, and as a result it is likely that more than one solution will be required.

1. For imports (v7), as Ana mentioned, a tag is created for the imported items. This makes it easy to identify which items were imported when, and if an import is accidentally performed twice it'd be possible to simply delete all of the items with one tag or the other. v8 does not yet support importing, but hopefully we'll do the same or similar there.
2. For migrations from standalone, we now write out a hidden "migration receipt," which tracks what has been migrated, in an attempt to prevent duplicate migrations. This does break down if that receipt, stored in each standalone vault, doesn't get synced between your devices, though.
3. The browser extension(s) used to lead to duplicate items being saved, though we've made improvements in this regard and I rarely see opportunity for duplicates since the in-line menu. It is possible though that there is still room for further improvement here.

One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place. We've made some good changes in this regard as outlined above, but it'd be great to do more. Of course that doesn't help folks that already have duplicates, and I hope we can address that as well. I don't know of anything definite planned at this time, but I will continue to push for a solution.

Ben

Chris Messina

One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place.

Yes, that would be helpful.

Of course that doesn't help folks that already have duplicates, and I hope we can address that as well.

But no, it wouldn't help in my case. Such it is.

I will point out that I don't recall ever "importing" entries through "the front door" as it were ... instead I dragged and dropped thousands of entries from my local vault into my 1password.com vaults — but since I couldn't tell which ones I'd saved locally vs saved in my synced vaults, I assumed that 1Password would maintain internal UUIDs for each item, and catch the conflicts automatically and ask me what I wanted to do with them.

Clearly that didn't happen — so I just want to point out that I didn't "import" in any conventional way, but used drag and drop between vaults to fuck myself over.

Chris Messina

One aspect of this I'd like to see us focus on is the prevention of the creation of duplicate items in the first place.

Yes, that would be helpful.

Of course that doesn't help folks that already have duplicates, and I hope we can address that as well.

But no, it wouldn't help in my case. Such it is.

I will point out that I don't recall ever "importing" entries through "the front door" as it were ... instead I dragged and dropped thousands of entries from my local vault into my 1password.com vaults — but since I couldn't tell which ones I'd saved locally vs saved in my synced vaults, I assumed that 1Password would maintain internal UUIDs for each item, and catch the conflicts automatically. That clearly didn't happen — and so when I dragged and dropped items from one vault into another, I screwed myself over in ways that I didn't anticipate.

Perhaps that's something that should be considered — and a warning or prompt added when >100 items are dragged from one vault to another... that duplicates might be created?

Dayton_ag

I agree, duplications (past and present) is definitely something that we can do better at.

Part of the big change to 1Password 8 was to stop developing separate apps with different feature-sets (see duplication tool only available in 1Password for Mac) and implement a common core - this allows us to implement a feature once and have it work effortlessly on all platforms at the same time.

I'm hopeful that as we keep working along that vein a more robust duplication tool can be implemented, but as Ben mentioned we are unfortunately unable to offer any promises for the future. Know that we're also very interested in improvements in this area, and we'll keep advocating for it on our end. :smile:

WaltDittrich

I came here because I, too, moved my Primary vault items to my Private folder. 850 items, all seemed to be duplicates. Fortunately, I'm still with 1Password 7 for Mac, so my removal tool took away about 550.

@Chris Messina , I think one method for you might be to Move everything over to a new Vault. Then, as you use items, move them back into your Private vault. This will cut out you seeing the duplicates as you slowly build your Private vault back up.
It's sort of like moving everything from your e-mail Inbox to a separate folder. Then work towards keeping your Inbox emptied. The stuff is all still there, but not right in front of your face causing havoc.
(Gosh I hope this makes sense.)

Chris Messina

Hey @WaltDittrich — that makes sense but is a bit impractical given that I've got 6000+ items (with the dupes. That said, I have created a few new vaults (for Work and Family sharing), and those are indeed much cleaner!

All in all, I can search and generally get to the items I need... the real bother is when I go to update a password and I have to pick the item I want to hold on to! It can be hard to sort through the recent items when all these dupes are floating around.

PeterG_1P

Just wanted to say thanks for the information on your use cases here, folks. This is an area we hope to improve and we appreciate the chance to understand the scenarios you're running into!

ref: IDEA-I-615

loyed256

Where is the tool to resolve duplicates in v 8?

There is no Help > Tools any more. In fact I don't see Tools anywhere or any command to help resolve duplicates. Somehow in the move to v8 many of my 1000+ entries duplicated.

---

1Password Version: 8.7.0
Extension Version: Not Provided
OS Version: Not Provided

PeterG_1P

Hi @loyed256, thank you for the question - and I'm sorry to learn you've run into this duplicate issue! 👀

I'd like to connect you with one of our specialists teams who will be best positioned to help with this. Can you send a brief email to support+tradein@1Password.com, with a link to this discussion? They should be able to advise on the best solution, given the current state of your data.

And while 1Password 8 doesn't currently have a duplicate removal feature, I have put in a request for this with our development team and will be happy to add a vote on your behalf to that issue.

I hope this is helpful - and we'll look forward to resolving this for you! 👍

Unknown

This content has been removed.

spiggott

Same issue here. Just upgraded to v8 and have many, many duplicates. What is the best way to resolve?

Lew

@PeterG_1P...

Your specialist offered the following to my request for assistance with duplicates:

• As your all your data is still in your account, the option we suggest is to manually sort the vaults as this is not something we can do for you on our end.

I've been a 1PW user since version 1 and have required very little support over the years. I find the current level of support very disappointing.

ag_tommy

@Lew

My apologies. It could have been a misunderstanding of the questions. I'll get this before a team lead to see how best to proceed. I appreciate your patience as I work to find you the support you need.

Unknown

This content has been removed.

PeterG_1P

Hi folks, sorry to learn that you have encountered an issue with duplicates as well. The best immediate solution will be to contact our specialist team at support+tradein@1Password.com, who can provide the most specific advice on a solution for you individually.

To add to that, I have cast votes on behalf of everyone in this thread for a Duplicate Detective🕵️ feature in 1Password 8. Thank you for letting us know that this is valuable to you!

ref: IDEA-I-615

Unknown

This content has been removed.

ag_tommy

@ctphoto

1Password 8 was a re write from the ground up. It's more of a not there, vs. removed situation. As Peter noted you have all been included in this request. I might add that I am in the request as well. Apologies for the delay. We are running full steam ahead trying to answer everyone as quickly as possible.

Unknown

This content has been removed.

kOoLiNuS

This is so disappointing…

Do I have to write an email with my official 1password email account or with the 1password Community email account?

ag_tommy

@kOoLiNuS

We always recommend contacting us with your accounts email address.

CFS

I recently updated to 1Password for Mac 8, and also have a "duplicates" issue. But they're not really duplicates. My Personal vault of 500-600 entries (I forget which) suddenly ballooned to over 1,000 entries, because most (not all) of the entries now show up as pairs. There is the current version, and then an entry which is some archived version that somehow made its way back into the vault. In many cases it's easy to tell which is which, but there are lots where it is almost impossible, without testing each entry to find out whether it's current or not, and then re-archiving the invalid entry. This will be, to put it mildly, a huge PITA -- and needless to say, a job I shouldn't have to be doing. Any suggestions for how to fix this problem? After a similar disaster I had with a recent updating of 1Password for Windows, where every entry went blank (thankfully, this was finally fixed), I don't have a lot of confidence that 1Password updates are fully tested before they are released.