Documentation for .1pif export format?
I understand that it's JSON, but is there any documentation for the structure?
1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided
Comments
-
Unfortunately I don't think we have a document like that at this point. I know we do for the OPVault format:
OPVault design - 1Password Support
But I'm not seeing anything similar for 1PIF. It should be mostly human readable once you've formatted the JSON with some white space etc. Were there any specific questions that I might be able to get answers to for you?
Ben
0 -
@Ben, can I try to persuade you guys that having a document would be really great and would be worth putting some energy into it? Here are my reasons.
We all wish that 1Password live for a really long time. I am a newcomer into it, I have just signed up and converted my data into 1Password, and it is all fine. I am reasonable happy with it. However, I am old enough to realize that this may not always be the case. I am a Bento refugee (for those of you who do not know, Bento was a great database system running on Mac and IOS, made for the lambda users, affordable, etc, but the product stopped), and I was really very happy that I could get my data out of Bento (into proper CSV files) that I could then import (more exactly massage and import) into other systems. That made my transfer into 1Password much easier, for example. I wish I would never be forced to do the same for 1Password, but I do not think anyone of us can predict what will happen in, say, 5-10 years down the line.
Thus: I would like to have a really easy way of exporting my 1Password data into something portable. Just in case.
1Password can save everything into CSV. With all due respect, it is close to unreadable and unmanageable the way it is, it is one giant table that, for example, my Atom editor cannot even handle (I have around 350 records). However, I get it that the 1Password team may not have the time to produce something better (e.g., a collection of csv files, one per category, with all essential data saved), but you may want to rely on the geek community out there to come to rescue. However, a decent documentation is really a prerequisite for that. Having had a first look at the 1.pf file, I could see someone coming up with a, say, node.js based tool to create those CSV files, but it is a pain to reverse engineer the content which is full of information that are irrelevant for pure export. I would even be happy playing with that myself but... removing the pain with a good documentation would really be helpful.
As an aside: @Michael Tsai: no, 1.pf is not JSON. It is a collection of small JSON snippets, separated by a string whose reason I do not know. As an aside, it would be a major improvement to generate a real JSON file that I could get into a Javascript directly instead of being forced to turn this into a genuine JSON. Would that be possible, @Ben?
(@Ben, I know about the utilities which does have one "print" exporter. I have not tried it out yet, I do not know whether it produces something one could use. I must admit it was a very long time ago when I used Perl, and I am not really ready to go back to it to massage it...)
0 -
@iherman ,
My personal point of view is that Agilebits will never document the 1PIF format for external consumption. That then ties them to this format, and it is not intended to be used for anything but transferring data to and from 1Password.
The converter suite's onepif exporter has been updated to read a 1PIF and produce various formats such as CSV and HTML. It now uses formatters to create the required export format. It can also create one file per 1Password category when it creates CSV, so it greatly simplifies reading the CSVs.
The record separator used in the 1PIF to separate the JSON record lines is trivial to remove programmatically, as is parsing and printing the JSON records for human readability. (The separator is simply a legacy implementation from long ago, and likely will never change.)
The 1PIF format is trivial to read and understand and convert, so you need not be concerned about being unable to understand it and gather your data. The intricacies in 1PIF are only relevant to programs like the converters which must create a 1PIF suitable for consumption by 1Password. If 1Password goes away in the future, nobody cares about these intricacies at that point anyway, right?
If you really want to dive into creating a tool for reading 1PIF, you can start by looking at the code in the converter suite - the module you want is Utils/PIF.pm. I'm happy to discuss any of it with you, and you'll likely have a better understanding of the details of 1PIF far before Agilebits documents it.
0 -
1Password can save everything into CSV.
That is unfortunately not true. The CSV export does not include user-added sections/fields or attachments. Custom fields aren't even combined into a blob of text; they're just silently dropped.
1.pf is not JSON. It is a collection of small JSON snippets, separated by a string whose reason I do not know.
Correct, although in practice it was easy for me to JSON-parse just the odd lines. It was straightforward to write a converter for my data, just being careful to make sure that it warned me about any keys it didn't understand so that I could investigate them. However, for a general purpose converter or importer it sure would be nice to have everything documented.
One specific question: what is the format for dates? It seems to be the number of seconds since the 1970 Unix epoch, however the time zone is not clear. It does not seem to be GMT because when I used NSDateFormatter to turn this back into a human readable string some of my dates were off by one.
0 -
@MrC : I looked at the converter suite, and I downloaded the latest release from GitHub. But I did not find the onepif exporter you are talking about (the list is on github, too...). I only found ** onepif2html** that creates an HTML file, but no CSV files.
As for the other remarks
- Yes, of course, it is trivial to turn PIF file into proper JSON. But that still means an extra (fairly unnecessary) steps of removing those lines and collate all JSON snippets into a proper JSON array (for example). But why this extra load when it would be trivial to produce a proper PIF file using those strings (some sort of an identification, I presume) as keys in a proper JSON object...
- Maybe our notion of 'trivial' is different. It is of course possible to reverse engineer the useful bits of the JSON output making either a CSV of (as a first step) a user facing version of the data in JSON. I would not call it trivial, however, it requires looking at a fair number of different records to get these in a proper order (see also @Michael Tsai's question on the dates).
Bottom line: I maintain that having a proper documentation would be something useful and would probably spawn a number of geeks out there to provide new codes to your infrastructure. It is a win-win in my view.
(@Michael Tsai is the code you have produced so far publicly available?)
0 -
@iherman Not currently.
0 -
@iherman ,
The onepif converter is in the 1.10 version, in Testing Bits, mentioned in the first post of the converter suite thread. It is a rewrite of the old onepif2html converter. My apologies that I forgot to mention this.
RE: 1. While you are correct that it should be easy to produce a single JSON object in the 1PIF, you're not considering that millions of installations already exist and do not know how to read this new 1PIF version you are proposing. All 1Password versions, old and new, would have to be updated and replaced at the same time essentially. The "extra load" (you mean, on the processor, I presume) is absolutely insignificant.
RE: 2. A 1PIF to CSV converter already exists: onepif. And the Utils/PIF.pm source code does describe the necessary details of the 1PIF format.
I am confused as to your personal, ultimate goal in this thread. I'm reading that mostly you want to not be locked in. But that goal is already met so long as you can export 1PIF, and you can already convert it to the more friendly CSV format.
You also mention node.js and from that I've presumed you are a developer of some skill, so my choice of word trivial was geared towards that skill set with respect to the problem at hand (parsing and converting the 1PIF data). If you are not a developer, then the mention of node.js and extra load seems tangential and irrelevant to your goal.
I'll reiterate that 1PIF was not and is not designed to be a documented, public data structure for users or developers. In other words, it is not produced or consumed via some public API that developers use. It is an opaque, private data structure, that just happens to be exportable to the file system for import/export convenience and capability. It is far from being a public data structure and API not simply because it is not documented... but also because an external tool cannot generate a 1PIF that 1Password will import in a well-defined manner.
Until there is an API that produces and consumes a 1PIF, with guaranteed results, documentation of 1PIF has essentially no value to 1Password customers as a whole, but Agilebits will incur a significant burden. And once there is an API, the documentation for 1PIF is yet again unnecessary, as it will still be an opaque, private data structure only produced and consumed by that API.
I've been writing my converters for over three years. In all that time, I've seen about 4 or 5 people ask for explanations of the 1PIF. The audience for this is very, very small.
0 -
thanks for the reference to the testing version of the converter suite. Found it, installed it, and got it to work (Mac OS El Capitan). And indeed it solves my original issue of being locked in. (Although, for 1Password's sake, I actually hope I will never really hit this issue in practice:-)
I realized that the script does not expert two categories yet, namely WiFi Routers and Reward Programs. I happen to use both, but I also presume that this is only a temporary miss, and they will be added when the this version of the converter becomes stable. Where will the new version be announced when the time comes?
(B.t.w., I take your point on the problem with deployed 1PIF files around. But with onepif working for CSV export my particular issue goes away anyway. )
0 -
Hi @iherman ,
Great, glad it helps you out.
I add code based on user requests; I take a lazy approach to code development (add what's needed, what's asked for), and because I get quick feedback that the code meets the need. I'll add WiFi Routers and Rewards for you tomorrow (Sunday, California time). Feel free to request other 1Password categories or feature requests.
I promote converters to "Stable Bits" periodically, as enough features are added (I don't want to burden Agilebits with too many code pull requests). The term "Testing Bits" is just a fun term I coined, and a place where rapid fixes and new features get posted as per user-requests like yours. When I promote the code, I update the converters thread and submit my code to AgileBits so they can pop it into their Git repository.
0 -
@Ben Could you get an answer to the question about the date format?
0 -
One specific question: what is the format for dates? It seems to be the number of seconds since the 1970 Unix epoch, however the time zone is not clear. It does not seem to be GMT because when I used NSDateFormatter to turn this back into a human readable string some of my dates were off by one.
The date value is now defined as being seconds since 1970, in UTC timezone. The user will see a date in UTC timezone.
We also define a 'legacy date' format which is: A legacy date value is defined as being seconds since 1970 UTC, plus the local timezone offset (which may be negative). Whatever timezone that may be.
At different points in time different 1Password products have used one or the other. The former is now the preferred method, but I can't say for certain that all platforms are currently using it.
Ben
0 -
@Ben Are saying that given a .1pif file, a field may be in the new or legacy format, and the file doesn't say which?
0 -
That is correct, a single 1pif file could have a mix of both the old and new timestamps.
Rudy
0 -
@rudy OK, that definitely explains why I couldn't get my script to format all the dates correctly. Not what I was hoping to hear, but at least I have an answer now. Thanks.
0 -
Unfortunately the only reliable way to get them to all match is to edit and save each item in 1P before exporting to 1pif.
Rudy
0 -
Hi @MrC,
my apologies, I seemed to have made a mistake when I tested the WiFi. At that point, for some reasons, it did not work. I looked at the list of allowed categories in the documentation (in the section "Exporting from your Password Manager") which did not list WiFi and Rewards, and I quickly concluded that this was simply a missing feature. I just tested it again and indeed works; apologies for the noise (although it may be a good idea to update the documentation:-).
(Oh and sorry for the late reply, I had a few days 100% off...)
0 -
No worries @iherman .
I think you must have been looking at the CSV section of Exporting from your Password Manager, but that's for the csv converter; it is not for the onepif converter which you are using (and it is the first section, named 1Password, under Exporting from your Password Manager).
0
