MrC's Convert to 1Password Utility (mrc-converter-suite)

MrC

@echopecker ,

I'll see if I can take a look today at the issues brought about by High Sierra. I've been a bit busy the past week since doing the update, so haven't had a chance to look at this.

Back with you soon...

echopecker

Thanks so much! I’ve searched to the end of the internet to find a solution, but not being a coder makes it more difficult.

MrC

@echopecker ,

It appears this is a bug with AppleScript in High Sierra. Others have reported this, and as of today, there is no fix. I haven't found any workaround, so you may just have to manually press Allow many, many times. Sorry.

MrC

@echopecker ,

Ugh, it gets even worse. High Sierra was just updated today, and the new dialog presented by security requires a password for each entry now. From the release notes:

Description: A method existed for applications to bypass the keychain access prompt with a synthetic click. This was addressed by requiring the user password when prompting for keychain access.

echopecker

Ironic, as when I saw that there was an issue with Keychain today, my immediate reaction was "I gotta get this upgraded". Little did I know I was screwing myself. Even though I've upgraded my main machine today, if I get logged in on my Yosemite machine, so you think the AppleScript would work there?

Again, I appreciate the efforts.

MrC

@echopecker ,

Ironic, isnt it?!

The script shoudl work on Yosemite. And you should be able to export, as a keychain, yoru current keychain, import it into a separate keychain on Yosemite, to use the security tool on that imported keychain.

nean

Hi MrC,

as proposed, let's follow up the discussion from github, let me sum up :

The existing keepassx converter is not working -> (new version exports csv instead xml , see below)

$ perl convert_to_1p4.pl keepassx test.csv -v
Uncaught exception from user code:

    not well-formed (invalid token) at line 1, column 7, byte 7:
    "Group","Title","Username","Password","URL","Notes"
    ======^
    "Root","KeePassHttp Settings","","","",""
    "Root/sites","XXXXXXXXXXX,"XXXXXXXXX","XXXXXXXX","XXXXXXXX",""
     at /opt/local/lib/perl5/site_perl/5.24/darwin-thread-multi-2level/XML/Parser.pm line 187.
    XML::Parser::parse(XML::Parser=HASH(0x7fb9632e26c8), "\"Group\",\"Title\",\"Username\",\"Password\",\"URL\",\"Notes\"\x{a}\"Root\",\"K"...) called at /opt/local/lib/perl5/site_perl/5.24/XML/XPath/XMLParser.pm line 74
    XML::XPath::XMLParser::parse(XML::XPath::XMLParser=HASH(0x7fb9636066b0)) called at /opt/local/lib/perl5/site_perl/5.24/XML/XPath.pm line 154
    XML::XPath::find(XML::XPath=HASH(0x7fb9637b2868), "//group") called at Converters/Keepassx.pm line 62
    Converters::Keepassx::do_import("test.csv", undef) called at convert_to_1p4.pl line 125

There was a change in the export as of version 2, but the recommended csv converter does not work properly, see below why:

keepassx 2.0.3 -> csv export -> csv converter -> FAIL

$ perl convert_to_1p4.pl csv nean2x.csv -v
Imported 186 items (180 items expanded to 360 items)
Exported 180 login items
Exported 186 membership items
Exported 366 total items
You may now import the file 1P_import.1pif into 1Password

Because Asset is detected as membership cards, thats a reason why in the end I created a different one which is a modified version of the csv converter (the current name is Keepasxc, you can choose a better name like KeepassX2 or keepasscsv)

I've just played with the csv converter code, but what I did not get so far is the detection of the asset type (Membership,Notes,password,etc...) , for some reason the original csv converter is detecting almost everything as a membership card?!

I've successfully tested my module (keepasxc) with following keepassX(c) versions

keepassx 2.0.3 -> csv export-> OK

 $ perl convert_to_1p4.pl keepassxc test.csv -v
Imported 186 items
Exported 180 login items
Exported 6 note items
Exported 186 total items
You may now import the file 1P_import.1pif into 1Password

keepassxc 2.2.0 -> csv export -> OK

$ perl convert_to_1p4.pl keepassxc test.csv -v
Imported 186 items
Exported 180 login items
Exported 6 note items
Exported 186 total items
You may now import the file 1P_import.1pif into 1Password

Also I tried to convert keepass2 csv1.x export files, but it is not yet working, but can be fixed as well.

keepass 2.36 (via wine) -> csv 1.x export -> FAIL

$ perl convert_to_1p4.pl keepassxc test.csv -d
CSV column names do not match expected names

...,small adoption need there as well.

In the end we'd only need a csv based converter that converts csv eported files from all keepass 2 derivates (kbdx).
btw. isn't there a perl or python module available that can open kbdx vaults directly?

however, you have my code with the adoptions, please have a look, verify and let's go from there ;-)

cheers,
nean

MrC

Thanks @nean,

As I mentioned in our other conversation, only the XML format of KeepassX 1.0x is supported by the keepassx converter. I did this to preserve as much of the data as possible, and because the CSV export from KeepassX 2 isn't as rich.

The CSV exported from KeepassX 2.x may contain various record categories, and the csv converter requires only a single category (and a header) per .csv file. Since most folks used KeepassX for Logins, this has not been an issue.

I'll take a look at your converter, and look into incorporating it into the suite by the end of this weekend. I need to create some test records, understand the data formats exported, and know the stock vs. custom category capabilities.

Again,
Thanks for your time and help here!
MrC

captainflo22

Hello, i will get all my data from iCloud Keychain to an third party software because I will migrate from apple to windows,

I tried all things I found here but I got an error that look I'm alone here,

I created a new keychain with the same password like my local account but I got an error :
impossible to add an item to the actual keychain
a no valid value was detected

you know what happens? thank you :)

MrC

@captainflo22 ,

I don't have a clue - there may be trouble in translation, and I don't recognize the error message. It could be that you have to disable iCloud syncing on the device and KEEP all items local when you do. iCloud Keychain exporting is a pain and a mystery.

Sorry I don't have more information for you.

captainflo22

I already unlinked my iCloud account and keep data, maybe restart my Mac could solve it, but I must wait.
to go more in describing : I could continue to write the password, and have this popup, at the end, I got 2 or 3 items copied

MrC

@captainflo22 ,

Some item types just don't seem to copy, and I don't know why. Because its impossible to debug the issue, and I can't reproduce the circumstances anyway, I haven't spent much time on this.

captainflo22

ok, now I found what the problem
I just would transfer my secured notes that's the only problem

MrC

Hi @nean,

I've evaluated KeepassXC's export and it is really trivial CSV, no different from KeePassX, containing only 6 columns:

"Group","Title","Username","Password","URL","Notes"

These are easily imported using the existing csv converter into Logins by adding the appropriate header row:

"Tags","Title","Login Username","Login Password","Login URL","Notes"

I'm not going to recommend that people use this form of export since its export is quite lossy - you lose all the attributes, and other record-specific data not included in the 6 columns above. Since the database is keepass2 compatible, for users that want all the data, they should export using keepass2; for users that don't care, its trivial to export as CSV, add the header row and convert using the csv converter.

MrC

Note: I've updated the converter suite version 1.10 in Testing Bits to support Perl 5.26.

nean

Hi MrC,

as I mentioned above, in my case csv and keepasx both did'nt work as expected, i had to modify the cvs converter, also to get content of the "Group" field, divided by "/" (path) converted into multiple tags, which is quite nice.

But indeed, for users who want as much information as possible, they shall use "keepass2" converter, therefore a proper exported "Keepass XML (2.xx)" file is required.

My understanding is, that on macOS as well as for Linux, users most likely use native KeepassX or meanwhile KeepassXC for kbdx databases, just for ease of use. And the problem is, that those 2 native apps "only" supports "csv" export format, and no "Keepass XML (2.xx)" format.

To get original keepass2 app running on macOS or linux involves wine or mono, but this can be challenging for some users and its also resource eating, thats why native apps are more common, hence you'd see such users using csv export.

Other idea, kbdx format is open, do you thing it is feasible to convert kbdx files ootb/directly, without all the export step (which is bit of a hassle) ?
like via http://search.cpan.org/~rhandom/File-KeePass-2.03/lib/File/KeePass.pm
or
https://sourceforge.net/projects/kpcli/files/
...
just thinking,...

thanks,
nean

MrC

@nean,

The keepassx converter only works with the 1.0x version's export - an XML file. The README.pdf makes this clear.

The Group field in the CSV export of KeepassXC is a hierarchical path - the / is not a field separator. If you have Group paths such as Work/Financial and Personal/Financial, and you use / as a separator to create individual tags, then there is no way to quickly select only your personal financial records (since splitting these paths into individual tags would place all records under Financial).

I do fully realize the implications of the lack of XML exports in the KeepassX application; that's why I made the notations I did in the README.pdf document. (I used keepass2 on a Mac a long time ago; it and its mono-dependencies are simply awful).

Thanks for the links you provided for natively decoding the kdbx file; I'll take a look at the docs. I already do this for SPB Wallet converter.

Edit: UGH - the File::Keepass module has many dependencies, and this is problematic for OS X users. The installation of these may be more problematic than installing Keepass2 + Mono temporarily to do an export.

Wyndham

Help... I must be doing something wrong.

I've run the first few steps more than once and it may be that in itself is causing a problem.

This screenshot illustrates that install_modules is no longer working - do let me know what I should try next.

Thank you again for helping.

MrC

Hi @Wyndham ,

There are a couple of things wrong:

Please be sure you have version 1.10 in Testing Bits if you are using version 5.26 of Strawberry Perl. I'd recommend it generally, regardless since there have been many fixes, updates, improvements. Testing Bits is mentioned in the first post of the converter suite thread.

Wyndham

:) Thank you very much. I thought I had followed the script carefully so there you are.

Many thanks for the all effort you have and continue to put in. People like you represent the good that can be found on the internet...

echopecker

MrC,

Just wanted to give some feedback. After upgrading to High Sierra and being unable to pull my keychain out, I took your advice and installed Yosemite on a laptop and was able to get at my keychain. The two scripts worked really well.

Only issue I had was that with over 1000 items in my iCloud keychain, when I selected all of them, I'd get an error and nothing would paste into my new temporary keychain. My solution was to break it down and do it by 2-3 leading letters at a time, until I had the entire alphabet done. Took about 10 different attempts. Found a couple letters that were giving me fits and eventually figured out the bad entries.

From there I just follow the directions and it all worked well. I'd recommend editing the first post on this page to give the Sierra/High Sierra users warning that it doesn't work and let them know they might need to downgrade to Yosemite on another machine or virtual machine.

Thanks for answering my questions on this process.

MrC

@echopecker ,

First and foremost, I'm really happy to hear you were able to get the bulk of your data transferred. That's great news. The divide and conquer approach was perfect.

I sure wish I knew exactly why copying data doesn't work, and that there was more debugging info - alas, its a big, nasty black-box, so we're stuck guessing.

I started trying to workout the issues with High Sierra, then the update occurred, and I'd wasted enough time that I had to stop. Perhaps when I have some free time, I'll try to come up with something else.

I'll update the first post to caution about High Sierra. Thanks for the reminder.

MrC

@echopecker,

I've added a note to the first post of the converter suite thread. Thanks again for the reminder and feedback.

echopecker

@MrC,

Thanks for the script! You've saved me hundreds of hours in work.

tillg

Hi MRC, thanks for the great work! It's an impressive list of transformers that you have made!!
I got stuck when trying to create an HTML page of my passwords.
Windows 10 1Password 4.6.2.626 Strawberry Perl 5.24.3.1 (2017-09-25) Portable Edition

I exported my data from 1Paswword as 1pif file and then ran the following command:

perl convert_to_1p4.pl onepif2html3 -o test1.html test1.1pif

Here is what I get:

---

Uncaught exception from user code:
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "\x{bf}\x{bf}{"locati...") at Utils/PIF.pm line 931.
JSON::PP::decode_error("malformed JSON string, neither array, object, number, string "...) called at C:/myperl/perl/lib/JSON/PP.pm line 1061
JSON::PP::word() called at C:/myperl/perl/lib/JSON/PP.pm line 749
JSON::PP::value() called at C:/myperl/perl/lib/JSON/PP.pm line 714
JSON::PP::PP_decode_json(JSON::PP=HASH(0x3f269b8), "\x{bf}\x{bf}{\"location\":\"https://appleid.apple.com/account/home\",\"secur"..., 0) called at C:/myperl/perl/lib/JSON/PP.pm line 145
JSON::PP::decode(JSON::PP=HASH(0x3f269b8), "\x{bf}\x{bf}{\"location\":\"https://appleid.apple.com/account/home\",\"secur"...) called at C:/myperl/perl/lib/JSON/PP.pm line 110
JSON::PP::decode_json("\x{bf}\x{bf}{\"location\":\"https://appleid.apple.com/account/home\",\"secur"...) called at Utils/PIF.pm line 931
Utils::PIF::get_items_from_1pif("test1.1pif") called at Converters/Onepif2html3.pm line 76
Converters::Onepif2html3::do_import("test1.1pif", undef) called at convert_to_1p4.pl line 125

---

As I have no experinece with PERL I fear digging into your code... Any idea what my problem is and how I could fix it?
Thx a lot from Germany,
Till.

MrC

Hi @tillg,

Thanks. Sorry for the troubles.

The problem is that there is some unexpected Unicode in the PIF output. It is those two 0xbf characters in a row. They are appearing right before the "location" key in the JSON, and these are not valid Unicode characters or a sequence.

Can you tell me which version of 1Password this is?

Also, did you happen to open/edit that 1PIF file with some Text Editor?

H_1PwD

Hi @goneawol ,

You probably downloaded Perl version 5.26. See the last note on the first post of the converter suite thread. Use version 5.24.

Also, which conversion are you going to do (from what password manager)?

---

Mr C, your comment above helped me resolve my problem. Thanks!

jlynch3sf

Any hope for High Sierra users? My wife has over 1K entries in her iCloud Keychain!

tillg

Hi @MrC ,

You are damn fast! ;) And you helped me fix it!!

My plattform and versions:
Windows 10, 1Password 4.6.2.626, Strawberry Perl 5.24.3.1 (2017-09-25) Portable Edition

The problem was that the 1pif file was UTF-8 encoded, but the Perl program wants ANSI. So I had to change it to ANSI (I used this: https://stackoverflow.com/questions/3710374/get-encoding-of-a-file-in-windows). The only annoying thing was to change the German characters that are in my 1Password store. But this way I got my HTML file created!

Thx a lot, both for helping me and for producing such a nice tool in the 1st place.

MrC

@tillg ,

I just now realized you are using an out of date converter - try using the 1.10 version in Testing Bits, mentioned on the first post of the converter suite thread.

There were huge changes made to the converter you are using, and it is now named simply onepif. See the Changes file for the change notes. This version reads UTF-8, and strips any BOM (the leading characters you had in your file).

@H_1PwD FYI - the 1.10 version of the converter suite in Testing Bits now supports Perl 5.26.