1PUX - need some help
Hi 1Password folks,
I'm not sure if I'm posting this in the right place - feel feel free to move it if desired.
This past week, I've spent a fair amount of time reworking the mrc-converter-suite to import 1PUX. And I made great progress, but I've hit what I feel is a major stumbling block.
Feel free to say, "Hey MrC, we don't want your stinkin' converter suite any longer" and I'll stop, and you can stop reading...
Hopefully you are still reading...
Trying to determine the cause of failed imports is very difficult (the log is woefully inadequate for this). I've had to work via trial and error. Yesterday, I achieved my first successful import. It took a while to discover that certain attributes are required, or the import will fail. An item's createdAt and updatedAt were the first two surprises. This requirement is new to 1PUX.
Today's major problem occurred as I try to import stock template sections / fields. When any of the field attributes guarded, multiline, dontGenerate or inputTraits (or any of its attributes keyboard, correction, capitalization) are missing, the import will fail. The requirement of such attributes is also new to 1PUX.
And this is a major problem. First, it means that I have to know all the values for those attributes for every field. That's not an impossible task, but is a problem if a) you change any of those attributes, b) I get any wrong, which will create issues for you and users - they can delete a field that would ordinarily be guarded, or a password generator doesn't appear when it should, the wrong keyboard is used, etc. (users will have no way to fix such issues).
Previously, the field's "id" was sufficient for a 1PIF import to match the proper field within a category's template. I think the same should be true for a 1PUX import wrt. not requiring the previously mentioned attributes. 1Password's template values for the "id" should be used to set default attribute values, so that they are not required to be in the 1PUX. But if these attributes are present in custom fields, they should be used. Likewise, the same for createdAt and updatedAt, which should both default to the time of import when absent.
I think all the fields in yellow below should be optional - use if present, default values assigned on import if missing:
Other attributes aren't yet clear to me, such as state, and I presume indexAtSource is the field's index order within a section.
The 1PUX document has a few issues. I'm happy to provide details if you're interested.
Finally, I've been hoping to be able to use the CLI instead of 1PIF / 1PUX for this type of conversion / import. But it seems still not ready for this; I'm hoping it will be someday.
1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Browser:_ Not Provided
Comments
-
Hi @MrC. I hope you keep persevering on 1PUX.
I started to rant off-topic and decided to hold back, but here's the gist:
In another discussion, @Jack.P_1P says "At this time, the primary goal of the .1pux export is to be able to take one's data from 1Password to another product, and not explicitly as a backup."
@ag_kevin mentions "We expect to publish specifications for 1pux in the future. Note that 1pux is an unencrypted format and thus easy to read, so we expect other password managers and other developers will support this format soon."
So...you expect other password managers to support importing the 1PUX format that you haven't documented yet? (I found it documented, after some searching.) And it includes things
Just for kicks, I tried exporting from 1P8, and the only option I had was to export the entire account. That seems like a HUGE problem. It seems like the only thing I can do with a 1PUX file is import it into another 1P8 account. With the exception of Bitwarden, all the other PW managers that 1P can import from can import a CSV or sometimes 1PIF, but not 1PUX.
AgileBits has always made a point that our data is ours and can be exported to another PW manager. That was definitely try with 1P7, but apparently not in 1P8. 1PUX includes LOTS of info that other PW managers don't handle. The CSV option gets logins, but it leaves all the Secure Notes, Software Licenses, everything else stranded.
Your data is yours. If you want to export your data from 1Password for any reason, you can. The 1Password Unencrypted Export (1PUX) format allows you to export 1PUX files, so you can access your data outside of 1Password. In 1Password 8, you can export a 1PUX file for each of your accounts and use it to transition to another password manager.
From what little I've read in the forums this evening from AgileBits employees, all their logic/reasons stated behind 1PUX are contradicted or are missing significant bits of software to be written by outside parties.
Remembering back from Microsoft's "switch" to the Open Office Format (DOCX, XLSX, PPTX, etc), they continued to support the binary files (DOC, XLS, PPT) but also provided the ability to export to RFT, HTML, and OpenDocument (ODT). Excel had something the most comparable to the 1PUX vs 1PIF if it were only to support saving/exporting as CSV, thus losing all formatting, formulas, etc.
Microsoft went further by explicitly documenting the new XML-based formats and provided a way for older versions of Office (without native support of the XML formats) to read and write the new file formats.
AgileBits seems to have created "lock-in" with 1PUX in the guise of open-data access. I say that in part due to the lack of support for older formats and dropping support for their own older file-interchange format. In part, I say that because with the ability to have vaults with different people having access (I'm thinking of Families in particular), the "monolithic" 1PUX includes the entire account. The largest silo I'd want to export/import would be at the Vault level.
It's doubly ironic because the 1PUX format, being unencrypted, is wholly unsuited to exporting from 1 1P account to another, and the encrypted version, 1PEX, isn't available yet.
Since AgileBits doesn't encourage 1P7 and 1P8 coexistence on the same system, the web importer should be able to take up the slack, namely being able to import 1PIF, CSV files for more than logins and secure notes, and even 1PUX files (since the web interface gives the option to import or ignore individual items.
I'm a huge AgileBits fan and preach its merits every chance I get. However, there are some very significant shortcomings in 1P8 that is quickly moving out of "early access." As such, the lack of significant feature parity in places is a little disconcerting and very unlike AgileBits, IMHO.
I do love 1P8, and I can't wait for it to be more available for iOS. But yes, @MrC, it seems your utilities are still very much needed and the ability to handle 1PUX would be a great addition.
0 -
I'm happy to support 1PUX, but it will take some work on 1Password's part to eliminate the issues I mentioned above. I won't publish a 1PUX generator that creates a problematic import for a 1Password customer.
I've already reworked the code to be able to produce either format. But as I mentioned, I'm at a dead halt at this point.
Straying from the topic of my post... I'm not sure I'd go so far as to say there's any customer lock-in. 1PUX is very easy to read, parse, and is complete enough, so it's not a substantial hurdle for anyone to consume it, esp. other password managers. Customer demand will cause other password manager's to implement a 1PUX importer. A simple JSON file that is readily readable by anyone is not quite the same as the highly proprietary Microsoft formats of a few decades ago. Let's also keep in mind it was the open source threat that caused Microsoft (and Adobe) to move towards supporting open formats, and opening their formats; they had no choice ultimately (others had sufficiently revise engineered the format). And they had no choice in continuing to support their propriety formats (they could not just stop being able to read the billions of extant documents).
1Password's 1PIF export has been one of the best, and most comprehensive in the industry. 1PUX is better. And AgileBits / 1Password has historically been exceptionally responsive to my feedback and fixing issues with their imports. So, if they are still interested in having the converter available, and have the bandwidth to resolve the issues, I'm sure they will soon enough. Their code base is probably several orders of magnitude larger and more complex than it was a decade ago. I can appreciate that they may not be able to get to everything.
0 -
Hi, again, @MrC.
I'll happily help you with these.
For a little bit of context, especially with respect to error messages on missing fields, we're using a standard JSON parser for Rust. We declare the data structure that the parser should read into, and then the parser fails the first time it encounters a place where the available data doesn't match the structure itself. So on that front, we're going to be stuck with the very old style of "try, fix the first error found, try again".
But, the error messages we emit are, actually, very unhelpful. I'll see about improving them some, at least to the point of saying which field is missing. That feels insufficient, though, so I'll talk to our security folks to work out a plan for providing more information.
I'll research all of the other fields. I agree that some can be optional. The others are potentially optional, but I need to check for any extra issues that I may not be aware of.
What feedback would you like to provide with respect to the documentation? I'm too close to the system to understand the drawbacks of our current documentation, so I am definitely interested to know and to make things better.
Since I lead the Import project only a few weeks ago, I've let our CS team that I can be your point person for future technical issues here, too. I will often take a few days to respond as I have a lot of responsibilities, but I definitely want to help you with your work.
0 -
Lots of progress. The converter suite can now generate 1PIF or 1PUX. The default will be 1PUX.
I have a little more clean-up, testing and documentation to complete, but I'm really happy with the results so far.
How did I work around the issues I mention above?
I decided to let 1Password 8 work for me. I created sample records for each category, and a special record that contains the 1Password data types (this allows the convert driver or the individual converters to create custom fields). These records are exported to a 1PUX file, and it is used as input to a 1PUX tool I wrote. It can dump a 1PUX for reading, or it can be run in a generate mode that is a parser/code-builder. The parser/code-builder generates all the required PUX templates and data tables, including the necessary field attributes for all the stock fields and the basic data types. These field attributes are generated and stored in a simplified, space-efficient manner to make reading the PUX generator code as easy as reading the PIF generator code.
I'm hoping to release an update soon.
0
