Not identifying duplicates

benjilb
benjilb
Community Member

I had 2 vaults with some overlap. Knowing this i merged them together and tried the Duplicate Password tool.

It's found 1 dupe (ironically, not what i would call a dupe) but nothing else. I literally had a few hundred login items where the URL, username and password match but they are not being identified?


1Password Version: 7.6
Extension Version: Not Provided
OS Version: 10.15.6
Sync Type: Not Provided

Comments

  • Hey @benjilb

    I'm a bit surprised to hear a non-dupe was identified. Would you be able to share any additional details about that? 1Password for Mac is currently incredibly conservative about what it considers a duplicate. Unless the items are exact matches they shouldn't be showing up.

    Ben

  • benjilb
    benjilb
    Community Member

    @Ben
    My mistake - it was one of my many Slack logins and it is the same! I hadn't checked it properly.

    I have loads of other login items though where the only difference i can see is the tag (where i imported a lot of logins from a previous password manager) and the tag 'import-date' is different.

  • My mistake - it was one of my many Slack logins and it is the same! I hadn't checked it properly.

    Ah, no worries on that point @benjilb. Just wanted to make sure something didn't get caught accidentally.

    I have loads of other login items though where the only difference i can see is the tag (where i imported a lot of logins from a previous password manager) and the tag 'import-date' is different.

    Indeed. Unfortunately different tags would be enough to disqualify items from being 'duplicates' by 1Password for Mac, as it stands. We can perhaps look to broaden the scope in the future. :+1:

    Ben

  • benjilb
    benjilb
    Community Member

    ok, great. Thanks for the responses @Ben

  • ag_ana
    ag_ana
    1Password Alumni

    On behalf of Ben, you are welcome @benjilb! If you have any other questions, please feel free to reach out anytime.

    Have a wonderful day :)

  • rprice
    rprice
    Community Member
    edited September 2020

    A prompt to ignore a specific tag, e.g. "1PIF Import XX-YY-ZZ" when comparing duplicates would be greatly appreciated. Or a tool to say - why are these two not considered duplicates, so we can understand that the tag was the problem in the first place =)

  • rprice
    rprice
    Community Member

    I removed an import tag and attempted to find duplicates to no avail.
    I then "Duplicate"d an item, renamed it to be identical, and then ran "Help > Tools > Clean up Duplicate Items", and nothing was found.
    Is it comparing last modified dates as well? =/

  • @rprice

    The check for duplicates is currently very precise, and additionally it does not check across vaults. Are these items in the same vault? Please let me know.

    Ben

  • rprice
    rprice
    Community Member
    edited September 2020

    @Ben I only have one vault.

    I compared the two by doing the following:
    In 1password desktop application

    1Password 7
    Version 7.6 (70600006)
    Mac App Store
    

    Select both, export as CSV (with headers) to eg.csv

    Then in python:

    import csv, json
    for i, item in enumerate(csv.DictReader(open('eg.csv'))):
        json.dump(item, open('eg_%d.json' % i, 'w'), indent=2, sort_keys=True)
    

    Then in bash:

    | diff eg_*.json 
    | 9c9
    | <   "Modified Date": "1600890148", 
    | ---
    | >   "Modified Date": "1492322620",
    

    This leads me to believe the only difference is the modified date.

  • rprice
    rprice
    Community Member

    My comment was destroyed because I was trying to blockquote diff output with triple backslash and it failed with embedded greaterthan, which I tried to prefix with vertical-bar space to allow it, which caused it to crash and disappear.

  • rprice
    rprice
    Community Member
    edited September 2020

    @Ben I only have one vault.

    I compared the two by doing the following:
    In 1password desktop application

    1Password 7
    Version 7.6 (70600006)
    Mac App Store
    

    Select both, export as CSV (with headers) to eg.csv

    Then in python:

    import csv, json
    for i, item in enumerate(csv.DictReader(open('eg.csv'))):
        json.dump(item, open('eg_%d.json' % i, 'w'), indent=2, sort_keys=True)
    

    Then in bash (replaced gt/lt with curly for markdown crash):

    diff eg_*.json 
    9c9
    {   "Modified Date": "1600890148", 
    ---
    }   "Modified Date": "1492322620",
    

    This leads me to believe the only difference is the modified date.

  • @rprice

    Looked like the rapid edits triggered the spam filter. :) Fixed.

    This leads me to believe the only difference is the modified date.

    I believe you are correct on this but I will try to confirm with development. If that is the case, I will suggest we consider removing that as a requirement for the match.

    Ben

  • rprice
    rprice
    Community Member

    Worst case I can try to use the CLI to force same-second noop updates to my entire vault, or I can export, de-dup, import, but I'd rather not ;)

  • AGAlumB
    AGAlumB
    1Password Alumni

    That may be best, as I'm not sure it's a good idea for the vast majority of users to throw out data as "duplicate" when it is not. Modified date is metadata that is useful and used by many people, and 1Password itself with regard to important Watchtower notices. So while you may not care about it in this particular case, it's probably a bad idea for 1Password to be making that kind of call instead of the user.

This discussion has been closed.