Not identifying duplicates

benjilb

I had 2 vaults with some overlap. Knowing this i merged them together and tried the Duplicate Password tool.

It's found 1 dupe (ironically, not what i would call a dupe) but nothing else. I literally had a few hundred login items where the URL, username and password match but they are not being identified?

---

1Password Version: 7.6
Extension Version: Not Provided
OS Version: 10.15.6
Sync Type: Not Provided

Ben

Hey @benjilb

I'm a bit surprised to hear a non-dupe was identified. Would you be able to share any additional details about that? 1Password for Mac is currently incredibly conservative about what it considers a duplicate. Unless the items are exact matches they shouldn't be showing up.

Ben

benjilb

@Ben
My mistake - it was one of my many Slack logins and it is the same! I hadn't checked it properly.

I have loads of other login items though where the only difference i can see is the tag (where i imported a lot of logins from a previous password manager) and the tag 'import-date' is different.

Ben

My mistake - it was one of my many Slack logins and it is the same! I hadn't checked it properly.

Ah, no worries on that point @benjilb. Just wanted to make sure something didn't get caught accidentally.

I have loads of other login items though where the only difference i can see is the tag (where i imported a lot of logins from a previous password manager) and the tag 'import-date' is different.

Indeed. Unfortunately different tags would be enough to disqualify items from being 'duplicates' by 1Password for Mac, as it stands. We can perhaps look to broaden the scope in the future. :+1:

Ben

benjilb

ok, great. Thanks for the responses @Ben

ag_ana

On behalf of Ben, you are welcome @benjilb! If you have any other questions, please feel free to reach out anytime.

Have a wonderful day :)

rprice

A prompt to ignore a specific tag, e.g. "1PIF Import XX-YY-ZZ" when comparing duplicates would be greatly appreciated. Or a tool to say - why are these two not considered duplicates, so we can understand that the tag was the problem in the first place =)

rprice

I removed an import tag and attempted to find duplicates to no avail.
I then "Duplicate"d an item, renamed it to be identical, and then ran "Help > Tools > Clean up Duplicate Items", and nothing was found.
Is it comparing last modified dates as well? =/

Ben

@rprice

The check for duplicates is currently very precise, and additionally it does not check across vaults. Are these items in the same vault? Please let me know.

Ben

rprice

@Ben I only have one vault.

I compared the two by doing the following:
In 1password desktop application

1Password 7
Version 7.6 (70600006)
Mac App Store

Select both, export as CSV (with headers) to eg.csv

Then in python:

import csv, json
for i, item in enumerate(csv.DictReader(open('eg.csv'))):
    json.dump(item, open('eg_%d.json' % i, 'w'), indent=2, sort_keys=True)

Then in bash:

| diff eg_*.json 
| 9c9
| <   "Modified Date": "1600890148", 
| ---
| >   "Modified Date": "1492322620",

This leads me to believe the only difference is the modified date.

rprice

My comment was destroyed because I was trying to blockquote diff output with triple backslash and it failed with embedded greaterthan, which I tried to prefix with vertical-bar space to allow it, which caused it to crash and disappear.

rprice

@Ben I only have one vault.

I compared the two by doing the following:
In 1password desktop application

1Password 7
Version 7.6 (70600006)
Mac App Store

Select both, export as CSV (with headers) to eg.csv

Then in python:

import csv, json
for i, item in enumerate(csv.DictReader(open('eg.csv'))):
    json.dump(item, open('eg_%d.json' % i, 'w'), indent=2, sort_keys=True)

Then in bash (replaced gt/lt with curly for markdown crash):

diff eg_*.json 
9c9
{   "Modified Date": "1600890148", 
---
}   "Modified Date": "1492322620",

This leads me to believe the only difference is the modified date.

Ben

@rprice

Looked like the rapid edits triggered the spam filter. :) Fixed.

This leads me to believe the only difference is the modified date.

I believe you are correct on this but I will try to confirm with development. If that is the case, I will suggest we consider removing that as a requirement for the match.

Ben

rprice

Worst case I can try to use the CLI to force same-second noop updates to my entire vault, or I can export, de-dup, import, but I'd rather not ;)

AGAlumB

That may be best, as I'm not sure it's a good idea for the vast majority of users to throw out data as "duplicate" when it is not. Modified date is metadata that is useful and used by many people, and 1Password itself with regard to important Watchtower notices. So while you may not care about it in this particular case, it's probably a bad idea for 1Password to be making that kind of call instead of the user.