What is the point of the export feature?

burnwa

I recently needed to go back in time. I ended up using 1password.com for that. But, before I did, I thought to use my many 1pux and csv exports that I keep on an encrypted volume.

The csv was almost useless since very simple information was missing. For example, date modified, date created, and vault name were all missing. That made it impossible to figure out which entry was relevant since I keep various archive vaults of old entries which have the same names as entries in the active vaults. So it seems not much attention was given to making the csv export useful.

I then looked at the 1pux. I do understand how to get at the json inside it, but working directly with that is very hard. (I did start to use "jq" to query it.) I see no way to look at such an extract in 1Password. It seems I could import an exported 1pux, but I have no idea what that would do to my existing data.

I really can't see what utility these exports provide. What am I missing? Since they are being provided, I assume they are meant to add some value. Maybe the answer is they are only for transferring the data to a competing product.

---

1Password Version: 8.10.23
Extension Version: Not Provided
OS Version: Sonoma
Browser: Not Provided

MrC

@burnwa

An imported 1PUX imports date into new vaults. Your existing data is untouched.

CSV is simply not capable of carrying all the necessary information. Historically, password managers considered this to be the minimum requirement for user comfort level. The 1Password folks have clearly gone beyond this minimum. Not all password managers can import 1PUX. But 1Password can, and that’s what’s important.

1PUX is as close to lossless as you have. Passkey data is currently absent.

burnwa

Thanks for the response.

CSV is simply not capable of carrying all the necessary information.

CSV can encode anything. What makes sense to include depends on the intended use of the file. My intention is to open it in a spreadsheet program or text editor and read it. So, large or binary data would get in my way. But, vault name, created date, and modified date would not and they would add a lot of value.

I think vault name is a critical omission since the export is forced to include all vaults one has access to. It's exactly that omission that made my CSV export useless when I needed it. That discovery is what made me wonder what the point of the CSV export is.

An imported 1PUX imports date into new vaults. Your existing data is untouched.

That's good news. I'd like a bit more clarity since it's certainly not something I want to experiment with. Say I have 10 vaults. I think you're saying that after the import of the 1PUX I'll have 20 vaults. Will I be able to distinguish the original vaults from the imported vaults?

Any chance you have a reference where the 1PUX import is documented?

MrC

You're welcome @burnwa

Yes, of course anything can be encoded, but its not practical with custom fields and binary data. That's why an alternate packaging form is used. The JSON 1PUX is suitable. I won't disagree that additional data could be added to the CSV. It will never be comprehensive.

[EDIT: this is incorrect]
When you import a 1PUX, all vaults within that 1PUX are imported as new vaults, and prepend the name "Imported " in front of the vault names inside the 1PUX. So if you have vaults A, B and C, after an import, you'll have vaults "Imported A", "Imported B", and "Imported C". You can repeat the process, and you'll get additional vaults. The vault names are not relevant - there will be no data collisions, since a vault UUID is generated on import, and that's the important factor.
[END EDIT]

There is some documentation for 1PUX here, but it is far from comprehensive. I had to work out the specific details. You can also review the source to my converter suite. By the way, the converter suite will allow you to convert 1PUX into a comprehensive CSV. It supports selecting vaults by name / pattern, and records by tag names/patterns.

burnwa

Thanks again for your response. Hey! It's great you made that converter suite; very much appreciated.

I won't disagree that additional data could be added to the CSV. It will never be comprehensive.

But, I do think we disagree about what makes for a bare minimum CSV in order to support well a typical use case. The vault name is an attribute in the bare minimum set of attributes. It makes me think that the CSV export was not given much attention, but it would be unfair to assert that.

When you import a 1PUX, all vaults within that 1PUX are imported as new vaults, and prepend the name "Imported " in front of the vault names inside the 1PUX. So if you have vaults A, B and C, after an import, you'll have vaults "Imported A", "Imported B", and "Imported C". You can repeat the process, and you'll get additional vaults. The vault names are not relevant - there will be no data collisions, since a vault UUID is generated on import, and that's the important factor.

Thanks for that info.

So, if you import two different backup versions, you'll have two "Imported A" vaults. Even though they will have distinct UUID's, they will be indistinguishable in the UI. So after the first import and before the second, it would make sense to rename all the imported vaults in a way that associates them with the particular backup.

I guess, if the goal is to dig through backups, importing would be a very awkward approach.

There is some documentation for 1PUX here, but it is far from comprehensive.

I find the 1PUX mostly self-documenting; it's nicely done. The thing that's not documented is the 1PUX import, but your answers are a good substitute. Thanks for taking the time to provide them.

I think I might write a small program that uses a jq-like query and delivers the query's output for each 1PUX backup file in a folder. That will give me all the local history I need. I hadn't expected to need this to make the backups manageable during a crisis. I'd better do it ahead of time, before my hands are shaking in some panic of lost access.

MrC

@burnwa,

I'm so sorry. I'm actually entirely wrong about reimporting an exported 1PUX vault. It does not create new vaults - data in each vault gets re-added, thus duplicating the data.

burnwa

@MrC ,

Thanks for the quick retraction. That makes the 1PUX export quite the loaded gun. Does the application warn you about that before the import actually takes place?

MrC

@burnwa

There's no indication that the duplication will occur, and this caught me by surprise too. I tested this in my testing account, so no harm.

I had not noticed the duplication, since my converter allows 1Password to create the new UUIDs on import (I leave the vault UUID empty, so 1Password generates them). So multiple runs always get new vaults, and the vault's comments show the converter used and the date of conversion:

Created by the passwordboss converter in the MrC Converter Suite on Fri Dec 8 15:55:42 2023

I'd clearly not checked (or forgotten the results of) a 1Password 1PUX export - import round trip. Bad on me. The records themselves DO get new UUID values.

I do have a tool named 1pux_tool that presents the 1PUX data from the zip:

I haven't released it in the suite because it is a dual-purpose tool. It also builds code definitions from stock records. These are used to supply the converter with proper record and field attributes. And I didn't feel like documenting sufficiently for others.

Again, I'm really sorry for my screw-up on this info.

burnwa

Again, I'm really sorry for my screw-up on this info.

No need to apologize since you checked and retracted so quickly.

I think my original question still stands. If we put aside the desire to move 1Password data to a different program, I really don't know what the exports are for. If they are intended to provide utility to a non-programmer, then it seems to me they are half-baked.

The CSV export has data from every vault, but doesn't bother telling you what vault an item is from. It's like getting a list of cities without being told what states or countries they belong to. I don't think anyone was really serious about making the CSV useful.

The 1PUX export is a hand grenade. There's no explanation of what to do with it. One is easily lead to believe it's OK to just import it. But there are no warnings given and questions left unanswered. What if permissions have changed and some of the import is disallowed? Does it partially complete? Can you roll back a partial import? Can you roll back a bunch of vaults to before an import after you realize what damage it did?

The 1PUX is an enormous beast, containing the contents of every vault that you have access to in your account. Importing it in the same account that it was exported from has an enormous, undesirable effect and seems to be irreversible. Also, there's no way for a user like myself to learn what it does since I don't have access to a test account.

I'm left scratching my head.

MrC

@burnwa

I can't disagree with anything you've mentioned, or your reasoning.

Here's hoping these issues get improved.

Tertius3

@MrC
I just tested importing 1pux export of my regular family account into the beta account 1password provides for passkey test with the Windows desktop app, and all my vaults were actually imported as "Personal importiert", "Geteilt importiert". (names seem localized)
It seems your original claim was correct: an " importiert" is appended to each imported vault. Items are not mixed into existing vaults.
I imported the same file again, and again new vaults were created, however with the same names with "imported" appended: "Personal importiert", "Geteilt importiert"

So as far as I see, you can safely import any 1pux, as long as you don't already have any vaults with "importiert" as appendix, otherwise you're unable to distinguish between the existing and the new vaults.

I read @burnwa tried to read the json inside the 1pux with jq. I did this as well, and I used ChatGPT to help me creating a proper jq filter to get what I was looking for. I was successful with a quite complex filter (something like "give all entries with word xyz in the notice field that were created before date d and have the same url as another entry"), with absolutely no knowledge of jq before.

burnwa

@Tertius3
I hadn't realized we could get a test account. Thanks for that information.

I'm not sure what to make of the inconsistent results, you getting fresh vaults for the import and @MrC not. It seems like we might have a nondeterministic hand grenade.

That's pretty great that ChatGPT worked for you. I've only used it once for some code, to generate a pretty famous algorithm. The result ran fine, but after I studied it I saw that it was nonsense. It also wasn't an appropriate starting point for a real solution. But, I think it might be ideal for something like this task since it would be easy to check the query and result.

Tertius3

@burnwa
ChatGPT didn't create a fully functional filter for me, but it created a filter draft I was able to work with and refine. That's the most difficult part: find the start and the proper approach.

It's just an assistant in the end, not a full grown engineer. It's also highly dependent on what you're telling it and how you write your question. For my jq query, I extracted the core of the json and left single entries to tell of the document structure, and asked to generate a query to link some fields, apply some condition and extract some fields - a bit like how a SQL query with WHERE and JOIN is made. It was not a single question, it was quite a conversation with 5-10 questions/answers back and forth to set a context and make things more clear.

For me, this was the way to do mass/batch operations with my 1password data. I'm finished with all cleanup work, so I will probably not need this any more. I did create a 1pux file, extract the json and applied some jq query to perform what I want. Then I implanted the changed entries into an empty json skeleton that 1Password was able to import. The imported entries ended up in their own vault. I verified them, then replaced the original entries with the imported entries. As far as I remember, I also used tags to select items.

The 1Password staff will probably roll their eyes - that's what the 1password cli is for, which I will use for such batch/scripted updates in the future.

burnwa

@Tertius3

You're more brave than I am. I've concluded from this thread that I will never, ever import a 1PUX into my main account. The result is undocumented and inconsistent. And, it's not certain that I would be able to recover from a failure. The only exception would be if I wanted to start fresh from a backup after wiping all entries from my account.

Thanks so much for your input.

Unknown

This content has been removed.

MrC

@Tertius3

I just tested importing 1pux export of my regular family account into the beta account 1password provides for passkey test with the Windows desktop app, and all my vaults were actually imported as "Personal importiert", "Geteilt importiert". (names seem localized)
It seems your original claim was correct: an " importiert" is appended to each imported vault. Items are not mixed into existing vaults.

Thanks for checking.

That was my recollection of past experience too. But after my reply, I had a tingling that said I should re-test the round-trip again. I was mortified to see that I had the same 3 vaults, all with duplicate records. It was more important for a quick retraction to avoid others having issues, than causing suffering.

I don't know if this is a beta or specific version issue, or a change with the Mac app. But I don't like it, and it feels broken. I haven't had the time to determine any more specifics.

In my opinion, a re-import should do what we believe(d) it should do - new vaults, and the new vault's comments should include the import's date / time stamp.

I'm hoping someone from 1Password has been monitoring this thread.

Tertius3

@MrC I tested with the Windows desktop app, version "1Password for Windows 8.10.24 (81024029)" (nightly channel) just before I took that screenshot earlier today to make sure I'm writing about the current behavior. It is also my recollection from perhaps one year ago that this behavior was the same then.

However, importing duplicates is not as disastrous as it were. The current desktop app Watchtower includes a fine duplicates cleanup tool, so even if duplicates are created in an existing vault, you're able to clean that up without too much effort.

MrC

@Tertius3

I was hoping I wasn't having some age-related brain skid.

Let's note that one version of a dup will contain the item's history, the other will not. The 1PUX does not retain the item's history - that remains strictly the providence of the 1Password server.