Feature request: Sanity checks before importing data

@carlosreq / @brenty

I saw this request the other day, and thought about providing some comments, but decided not to. But now that the discussion is open, I'll add my views.

Don't import passwords for domains that no longer exist at all

This is a tough one, for not only the reason brenty mentioned, but also because temporary failures or timeouts in DNS can return false negatives. And when domains expire, registrars often point the now-vacant domain to a landing site, so those will be false positives. There are also redirects to consider.

Allow filtering by last used (since that's s timestamp field that firefox exports) - probably no point in importing anything not used in the past year

I won't address the 1Password filtering aspect, but the latter request to not import items unused for some duration of time.

Any date / time cutoff threshold would be rather arbitrary and binary. I personally have important logins that I don't use but once a year or two, but losing those would be problematic. I'm sure I'm not alone here. I think it is best left to users to decide on a case-by-case basis which items can / should be deleted (and this can be done on a lazy, as you encounter them, fashion). Doing this in 1Password is trivial enough.

Find collisions with existing entries, most likely what 1pass already has is the correct info

I've read many requests over the years for auto-magical duplicate detection / avoidance. I use the word auto-magical because simple ideas can be difficult to implement without creating additional issues that are only avoidable via magic or wishful thinking. When you think of duplication, you have to think about all the fields in a record, and these have to be compared against all the fields in a set of one or more possible target records. The titles, fields such as URL, username, password, etc. And things like the metadata, such as the various time stamps, password histories, favorite status, tags & folders, etc. What for example, should this automated sanity checking software do in the case where a 1Password entry exists, but is determined to be too old to import based on your Firefox export's last used time stamp? It's the edge cases, and there are many of them, that need to be coded, and a UI made, to present the user with choices so that any converter / importer / 1Password doesn't make mistakes. You as a human can make a quick, decisive decisions that an item is superfluous and is a candidate for deletion - software cannot know your mind, let alone the large set of 1Password customer's minds in aggregate (to do the right thing).

I chime in, as a user like you, but also as one who has written many converters for multiple platforms, and I've thought about these issues and how to make a tool to best help users.

The converter suite could, for example. do DNS lookups on Login URLs. This would introduce long delays in the conversion process, as each domain probe would need to possibly wait the 10 to 15 second DNS timeout period before making a decision. I would not code anything that used arbitrary cutoff dates to ignore entries (because I would not want to support users who feel they've lost data, not fully understanding the implications of such a heuristic). But the entry could be flagged accordingly with its domain lookup status (if negative).

Regarding Last Used, 1Password has the ability to sort by Last Used. But this date does not get imported from arbitrary CSV files. I don't know if this metadata field can be set by external software (either via op or during import of a 1PIF).

I'd thought about how to help users de-dup their many items. I rejected trying to compare items being converted to those already within 1Password (no access is available to these w/out exporting, or using op where it is available). And I ultimately rejected trying to do an item by item, field by field comparison, as I strongly suspect most cases will involve subtle differences not reliably easy to auto-reject, nor present to users in a comprehensive yet meaningful way to make decisions without error. So I, and I believe 1Password, ultimate error on the side of safety and precaution, and although perhaps somewhat tedious, let users clean their own house as they see fit.

This usage case typically is designed when adding / merging one or two items. But this item-by-item decision-based UI paradigm doesn't scale well for users with many records (often hundreds or thousands). Again, there are edge cases that users don't think about. Erring on the side of caution, and providing a good, simple UI, is probably best.

I don't mean to stamp out any discussion here. The topic has been discussed many times over the years, and I'm just summarizing my point of view, and the history of the past discussions.