Feature request: Sanity checks before importing data

carlosreq

I finally got to import (via CSV file; more on that on a different post) my old firefox logins. Around 1000 logins, most of them from sites I've used only one. I'm trying to do clean up, and I'm finding that a lot of the work I'm doing manually could be done by the import system. Specifically:

• Don't import passwords for domains that no longer exist at all
• Allow filtering by last used (since that's s timestamp field that firefox exports) - probably no point in importing anything not used in the past year
• Find collisions with existing entries, most likely what 1pass already has is the correct info

---

1Password Version: 7.6.785
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

AGAlumB

1Password is not going to check the existence of websites for you, as it does not interact with them at all (it simply send the URL to the OS, which in turn sends it to the browser). It would be problematic for 1Password to check in with us for something like that, and the alternative of your machine individually contacting each itself would be resource intensive, in addition to being a weird thing to do when merely importing text records.

Otherwise, most users would not be okay with throwing out data just because they haven't needed it recently, and the individual is really the only one in a position to determine if some are redundant; those are not judgement calls 1Password should be making. @MrC ’s converter scripts could be used to help with some of that, but it does seem to me that it would be better to get the data out of plaintext and into 1Password sooner rather than later and then sort it out there, securely, and so you have it backed up along the way in case you make a change you later regret. But, keeping those concerns in mind, these are things we can evaluate as we create future versions of 1Password. Cheers! :)

MrC

@carlosreq / @brenty

I saw this request the other day, and thought about providing some comments, but decided not to. But now that the discussion is open, I'll add my views.

Don't import passwords for domains that no longer exist at all

This is a tough one, for not only the reason brenty mentioned, but also because temporary failures or timeouts in DNS can return false negatives. And when domains expire, registrars often point the now-vacant domain to a landing site, so those will be false positives. There are also redirects to consider.

Allow filtering by last used (since that's s timestamp field that firefox exports) - probably no point in importing anything not used in the past year

I won't address the 1Password filtering aspect, but the latter request to not import items unused for some duration of time.

Any date / time cutoff threshold would be rather arbitrary and binary. I personally have important logins that I don't use but once a year or two, but losing those would be problematic. I'm sure I'm not alone here. I think it is best left to users to decide on a case-by-case basis which items can / should be deleted (and this can be done on a lazy, as you encounter them, fashion). Doing this in 1Password is trivial enough.

Find collisions with existing entries, most likely what 1pass already has is the correct info

I've read many requests over the years for auto-magical duplicate detection / avoidance. I use the word auto-magical because simple ideas can be difficult to implement without creating additional issues that are only avoidable via magic or wishful thinking. When you think of duplication, you have to think about all the fields in a record, and these have to be compared against all the fields in a set of one or more possible target records. The titles, fields such as URL, username, password, etc. And things like the metadata, such as the various time stamps, password histories, favorite status, tags & folders, etc. What for example, should this automated sanity checking software do in the case where a 1Password entry exists, but is determined to be too old to import based on your Firefox export's last used time stamp? It's the edge cases, and there are many of them, that need to be coded, and a UI made, to present the user with choices so that any converter / importer / 1Password doesn't make mistakes. You as a human can make a quick, decisive decisions that an item is superfluous and is a candidate for deletion - software cannot know your mind, let alone the large set of 1Password customer's minds in aggregate (to do the right thing).

I chime in, as a user like you, but also as one who has written many converters for multiple platforms, and I've thought about these issues and how to make a tool to best help users.

The converter suite could, for example. do DNS lookups on Login URLs. This would introduce long delays in the conversion process, as each domain probe would need to possibly wait the 10 to 15 second DNS timeout period before making a decision. I would not code anything that used arbitrary cutoff dates to ignore entries (because I would not want to support users who feel they've lost data, not fully understanding the implications of such a heuristic). But the entry could be flagged accordingly with its domain lookup status (if negative).

Regarding Last Used, 1Password has the ability to sort by Last Used. But this date does not get imported from arbitrary CSV files. I don't know if this metadata field can be set by external software (either via op or during import of a 1PIF).

I'd thought about how to help users de-dup their many items. I rejected trying to compare items being converted to those already within 1Password (no access is available to these w/out exporting, or using op where it is available). And I ultimately rejected trying to do an item by item, field by field comparison, as I strongly suspect most cases will involve subtle differences not reliably easy to auto-reject, nor present to users in a comprehensive yet meaningful way to make decisions without error. So I, and I believe 1Password, ultimate error on the side of safety and precaution, and although perhaps somewhat tedious, let users clean their own house as they see fit.

kevink707b

I think collisions should be handled similar to how MacOS Contacts do it, you get a choice to keep both, keep old, keep new or merge. If you choose merge it does the best it can to keep data from both but magic isn't guaranteed.

MrC

This usage case typically is designed when adding / merging one or two items. But this item-by-item decision-based UI paradigm doesn't scale well for users with many records (often hundreds or thousands). Again, there are edge cases that users don't think about. Erring on the side of caution, and providing a good, simple UI, is probably best.

I don't mean to stamp out any discussion here. The topic has been discussed many times over the years, and I'm just summarizing my point of view, and the history of the past discussions.