1pux export data will be wrong or old (and uncomplete)

Domenico

Hi,

I am using the 1password 8 app for windows, Iphone Ipad. They all are updated to last version (8,9,10).

I need to export the data I stored in 1Password, but I encounter many problems:

1) both the csv and the 1pux methods export only a very little subrange of the data I stored (with csv being the least complete one);

2) the data stored in the 1pux file (I am referring to the text file you get once you unzip the 1pux file) are very messed up:

• it reports also the entries I deleted in the past, which is very confusing (many times I deleted an item just to re-entry it in a different, better way)
• worse than all, some of the data are WRONG: they show the OLD passwords and username instead of the updated ones (updated by me when, for example, I change the password of an account). Just to give you an example, the username and password of my NAS are the ones I used more than 2 years ago.

I have been using 1Password for years, and this problem of the partial extraction (ref. point 1, I never really checked point 2 before today) has always been there (ref. all the tickets opened on this community and the complaints you can find by googling a bit).

Honestly and clearly speaking, it is not possible and I do not (and can not) believe that your software engineers haven't found a way to extract ALL the data customers insert in your software.

These partial extractions appear to be a clumsy and clear attempt to keep your customers renewing their licence every year.

Unfortunately, I have been unemployed for more than 2 years now, so I need to cut all the possible costs. Therefore I also would like not to renew my 1Password licence. But with this kind of "export" you make it impossible (unless I copy and paste every single line/entry, which would need hours and hours of work).

Can you help me? Have you planned to implement a FULL export soon?

Regards,
Domenico

---

1Password Version: 8.9.10
Extension Version: Not Provided
OS Version: windows 11
Browser:_ Chrome
Referrer: forum-search:export

MrC

@Domenico

I just did some testing w/ the Windows app (both released and beta versions).

Re: Deleted items appearing in the 1PUX. I'm not seeing this. I just deleted a Secure Note, it appears in Recently Deleted, and yet does not appear in a 1PUX export. I do, however, see Archived items, and these will have a "state" attribute with the value "archive":

"state": "archived",

Re: worse than all, some of the data are WRONG: they show the OLD passwords and username

I've done pretty extensive testing with 1PUX exporting for creating the converter suite. I have not seen this problem. Once the data is changed, the export should reflect this. I've seen no evidence of this not occurring. Can you provide a step by step to repeat it?

These partial extractions...

I'm confused by this statement. Your first item indicates too much data is being exported, not too litlte (or partial).

appear to be a clumsy and clear attempt to keep your customers renewing their licence every year.

I going to have to strongly disagree here. There may be bugs, and if you've encountered one, I'm sure the 1Password folks will recitify it. I've experienced a decade of them resolving import and export bugs/. They have demonstrated no interest in locking you in, or attempting to make it difficult for you to leave the platform.

Unfortunately, I have been unemployed for more than 2 years now, so...

That's a really unfortunate situation for you, and I'm sorry your experiencing such life troubles. You can use the converter suite's onepux converter to create one or more CSV files (monolithic, or per-category), and you can select which vault(s) should appear in the CSVs. Ask if you'd like help.

hardybird

I am currently testing 1Password as an alternative to Lastpass. I always want the ability to export my data into an easy-to-manipulate unencrypted format (preferably csv) for both portability and offline-backup purposes. I downloaded the MrC Converter Suite and read the instructions for Windows users. While I commend you @MrC for your excellent work, the fact that users have to jump through hoops just to extract ALL data to csv is not acceptable. Whether the 1Password folks intend it or not, the inability to easily export all data to a csv or other standard format (supported by other vendors) is a de facto affront to data portability. I am truly sorry this is the case and will now have to consider not using 1Password.

Domenico

@MrC :

I think what I wanted to say with my email was pretty clear, regardless my poor english (I am Italian).
@Happybird above explained it much better than me.

BUT there is more than that!
A lot of ERRORS in exportation.

Given the language difference and thinking that a video can show you much better what problems I find, I attach a few small videos.
They show you all the steps I take to extract and also a few example of errors.

I strongly suggest you see the videos from 0 to 5 (in this order), for better understanding.

P.S.: in the last video (number 5) I had to show a password, because it was the only way to show you that the password exported is NOT the actual password (it is a very old one). I am going to change that password immediately, so that every policy is satisfied.

As per your comment:
_These partial extractions...

I'm confused by this statement. Your first item indicates too much data is being exported, not too litlte (or partial)._

Data exported are INCOMPLETE: not all the fields are exported
Data exported are wrong:
1) I find data I deleted times ago
2) much worse, in the data that should be in the file, I find old passwords and/or usernames (i.e. which I changed in the past), not the actual ones.

Looking forward to your reponse (and a solution!!!).

Regard,
Domenico

P.S.: once finished this mail, I noticed wmv or mp4 formats are not allowed to be attached. Would you be so kind to suggest me a way to share the videos with you?

Regards,
Domenico

Domenico

If you want, you can send a private message to my email ([PII removed by moderator; this is a public forum]), with indications on how to send you the videos.

Tertius3

It comes to my mind you really may have old data in your vaults, for example you might have used 1Password 7 with a local vault and copied data from the cloud to your local vault, for example for backup purposes, and the vault data become stale because you actually edited the cloud data only.
With the upgrade to version 8, the client offered to upload any local vault to the cloud, and you accidentally uploaded your stale/old vault, and now you're seeing this in the export.

It also may be you used different accounts in the past for trial, and you exported some trial account with old testing data instead of the accout where you manage your current data. After all it's possible to be logged in with multiple 1Password accounts, so you might have chosen the wrong one to export.

Or you're exporting with an app that has broken cloud sync and you didn't realize this.

Domenico

Hi @Tertius3, thanks for the reply.

I answer in the following:

• I never used the cloud for backup or other purposes.
• The settings of my apps are the default (from installation) ones
• I did see the 8th version offer of the client to upload to the cloud, but I didn’t choose that option.
• I have always used same account, same username and password since the very beginning.

The fact that my Windows app, my iPhone app and my IPad app show exactly the same data and are synchronised almost in real time make me think that the cloud isn’t the problem.

Anyway, even if one of your guesses is the right one, I see my problem as a bug of the software: how it possible that I “ask” to an app with all my (correct) data to export its data, but I get something else?!?

Do you know if it exists some kind of remote support (for example with an operator working remotely on my own pc)?

I need to solve this issue, and I am clueless.

Domenico

P.S.: I don’t know how this communì works (I mean, if these messages are read also by 1Password software engineers)… but my offer to share the videos is still valid, and would clarify the problem more than 100 posts.

Domenico

@MrC
I just had an idea: could you repeat the tests you did, but starting from version 7, filling it in with some data, then delete some of them, and then upgrading to version 8?

This would replicate exactly what I did.

MrC

Hi @Domenico

No problem with the language issues, and certainly I can understand you're likely frustrated with the problem you're trying to understand and overcome.

I've created likely over a million items, deleted them, exported, and converted several hundred thousand times - without exaggeration. I've done this over the past 10 years using version 4 through 8. So I have a fairly good understanding of the exports.

Here's a Dropbox link to a movie (2.9Mb) I created that demonstrates what I'm understanding you are describing:

https://www.dropbox.com/s/iy01zadqmfxftdg/deletion-video.mp4?dl=0

The following description explains what is occurring in the video:

1. In 1Password 7 (7.9.8) a Login item titled "foobar" is shown in a vault, and it has a username "user@example.com".
2. The Login item is Deleted, and this deletion is confirmed.
3. The deleted Login item is shown to not be in Archive.
4. Jumping to 1Password 8 (8.9.12 beta), the Recently Deleted section is selected in the sidebar. The "foobar" item with username "user@example.com" appears there.
5. The account is exported to 1PUX, saved to the Desktop.
6. I use my 1pux_tool.pl to dump the contents of the 1PUX in an easy to read textual format, and send that content to the grep tool to look for the string "user@example.com". It does not appear in the text stream, so there is no output from that command sequence.

This confirms that the deleted item "foobar" with username "user@example.com" does not appear in the 1PUX.

I'm not doubting you are experiencing something that doesn't seem right to you. I just need an exact sequence of instructions to reproduce it to follow-up.

Domenico

Hi @MrC,

thanks for the reply.

I understand very well that you need the exact sequence of instructions to reproduce.

I hope the 6 videos linked below will give you enough info.
If not, please let me know what you need more and I'll provide it to you.

[ links removed - videos contain confidential information ]

Domenico

@MrC
I thought also the information that I have only one vault could be useful to you.
Here is the link:

[ link removed - video contains confidential information ]

Tertius3

@Domenico I looked into your "2-extracting-data-from-1pux" video. What you selected from the "files" subfolder is some attached file you attached to some item, probably a "secure notes" item. Probably some old file with some export from some other password manager. It cannot be your real 1Password export, because it isn't json format. The "files" folder contains custom icons you set and file attaches. You were probably mislead by the fact the word "password" is in that name, but that's only the name of the attached file.

Your actual current 1Password data is the file "export.data", which is in json format. Load this and you will find all current data.

MrC

@Domenico

It appears you have confidential information being exposed in the videos. I've removed the links, and you should securely delete these files from Dropbox.

Domenico

@Tertius3
opppps
Looking forward to look at the json data.

But... errr... how can I open "export.data" to make it readable?

I tried with a text editor weeks ago, but the result is something almost unreadable.
That's why I wrongly assumed the export file is the one you see in the video.

I am looking in internet right now, but it doesn't look to exist a "json file" app...

I

MrC

@Domenico

The export.data file is a nicely formatted JSON data file. It contains a structured formatting of your account information, the vaults contained in the account, the records in each vault, and the fields in each record. These are bracketed with curly and square braces. This is roughly, but not entirely accurately, described here:

https://support.1password.com/1pux-format/

You can open this file with a text editor and search for items. The structure is explained in the document I reference with the link above.

Domenico

@MrC

Opening the json file with a text editor works, but it is really really uncomfortable to read... even if the structure is clear.

Just coming back to what @Happybird wrote above, what a customer would like is a simple csv file. And, I add, a one-row-per-vault-item one (the metadata being in columns).

I am right now importing with Excel the json file.
It works, but the result is terrible:

• many rows per vault item instead than only one;
• funny enough, despite the fact that the metadata columns are 50 or more, username and password (the real basic reason for which apps like 1Password exist) are in the SAME COLUMN (!!!), in two rows. I mean, from 50 to 51 column doesn't change much, right? so why not making at least two separated columns for the two most important metadata?
  After working for almost an hour on it, deleting non necessary metadata columns and moving metadata columns in an easier to read order, I got an excel file that is very far from user friendly; I would say still hardly comprehensible. The most annoying thing is the fact that one single vault item is split in dozen of rows (the columns are not the problem, they are necessary having multiple metadata, but the multiple rows makes no sense).

Probably and hopefully I am so fresh to json files that there is a better way to convert them.

Therefore the big question is: have 1Passord created a "json to csv" converter?

Without such a tool I can't reach my goal: i.e. having a simple, easy and quick to use list of all items I input in the vault.
And such list should be, like common sense suggests, made by ONE ROW for each item of the vault,and one column for each of the metadata in the item itself.

In one of your messages above, you wrote:
I use my 1pux_tool.pl to dump the contents of the 1PUX in an easy to read textual format and send that content to the grep tool...

Is it possible to have this tools? Where can I find them?
Maybe they do what I am dreaming for...

MrC

@Domenico

If you want to review the contents of the 1PUX more easily, you can use the onepux converter from the converter suite. It can create a monolithic CSV or one per-category. See the README included in the suite, and you might have a look at the files in the suite's download location under Extra Help > Examples > 1PUX to CSV. There are also videos that show various conversions under Extra Help > Videos.

The 1pux_tool.pl tool won't be more help to you, since it just presents what you've already seen (plus a few more details about the vaults). The method mentioned above is what you want.

Ben

And such list should be, like common sense suggests, made by ONE ROW for each item of the vault,and one column for each of the metadata in the item itself.

Where we run into problems with this is the fact that we support custom sections and fields. Exporting those into CSV is unmanageable for large datasets that have taken advantage of the feature. We also have custom categories in beta for 1Password Business customers, which further compounds the situation. Every item could have hundreds of irrelevant columns, times thousands of items... At a minimum it would be incredibly inefficient and difficult to parse.

1PUX is a modern format that was built with these challenges in mind.

Ben

Domenico

@Ben: I am ready to use your converter suite.

I have done all the steps in the readme.html file.

But I don't know which converter I should use.

Would you please tell me which one?

MrC

@Domenico

Use the onepux converter. See the Examples mentioned above. They show how to run the conversion. The section 1Password 8 (1PUX to CSV) is the relevant converter in the README.

Please note, the converter suite is not written by 1Password. I'm just a user like you who wrote the set of tools over the years for other users like you and me.

Domenico

Thanks a lot guys for your support.

I am using this command line: perl convert.pl -v onepux --to_format csv --ignoretimestamps -o ..\boh.csv ..\export.data

but I get an error message saying "The onepif converter requires a 1PUX file"

Weird, being export.data a 1pux file...

MrC

@Domenico ,

Provide the path to the originally exported .1pux file, not the export.data file. The converter expects the original 1PUX format, not something you've unzipped.

You can rename the .1pux file from the rather longish 1PasswordExport-xxx-2023-0104-#####.1pux to something simpler such as 1Pexport.1pux. That is easier to type on the command line (in case you are not aware of the shortcuts for providing command line completion or drag-n-drop of files).

Eg:

Domenico

Thanks guys, it worked.

At the end is almost like I wanted it: just ONE ROW per item. The columuns are a lot, but that is normal, as @Ben said, in a database with lots of metadata.

One suggestion, if you allow me: transform all this into a "1-click" 1pux-to-csv converter app...
At the end of the day, your suite (congratulations, by the way, it must have not been easy to create it) does exactly that... but in a way I believe for the common user is a bit too complicated.

Thanks again,
Domenico

MrC

@Domenico

Great!

You can reduce the number of columns by using the --percategory option, as shown in the screenshot above.

In the end, any CSV export will have many columns, since you need one column per 1Password field.

I won't create a binary app, since users would be unable to examine the actual source code and verify no malicious code is involved. I don't want you to trust that I haven't planted some nasty code; I want you, along with others, to be able to verify this.

As to the complexity, I'm very aware of this. I wish I had a good answer.

There have indeed been a few hours involved building this set of tools. :-)

Domenico

@MrC: I understand it.

The solution would be to integrate the "app" I was talking about into the 1Password App, replacing the existing (very poor) csv export with your "real" one :-)

Best regards,
Domenico

MrC

@Domenico

Oh... if things were only so simple!

Imagine simply trying to make your airplane (you have one, right?) go faster by simply bolting-on more engines. It might seem like a good idea at first, until you realize that the extra air resistance, friction and weight added by the engines, stronger frame and extra fuel requirements defeats the original goal.

The converter is written in an entirely different language, so wouldn't be suited to this. It would either require a complete rewrite (and of course the 1Password folks are fully capable of this on their own; I'd add no value here), or would require a massive add-on wart of code (the runtime environment and libraries) for the sole purpose of doing a CSV export. So, I can safely say this won't happen (nor should it) - too much weight.

Anyway, I hope you have what you need. I'm here if you need more assistance.

Enjoy 1Password!

Unknown

This content has been removed.

Unknown

This content has been removed.