MrC's Convert to 1Password Utility (mrc-converter-suite)

hoods

Hi,

I wanted to convert and import my keepass2 database into 1Password but unfortunately the convert_to_1p4 script from MrC fails with an error. I've tried conversion script 1.05 and 1.06 but both fails with:

root# perl5.16 convert_to_1p4.pl keepass2 export.kdbx -o test
Uncaught exception from user code:

    not well-formed (invalid token) at line 1, column 0, byte 0:

< redacted >

     at /System/Library/Perl/Extras/5.16/darwin-thread-multi-2level/XML/Parser.pm line 187.
    XML::Parser::parsefile('XML::Parser=HASH(0x7f88a286e178)', 'export.kdbx') called at Converters/Keepass2.pm line 71
    Converters::Keepass2::do_import('export.kdbx', undef) called at convert_to_1p4.pl line 116

Any help highly appreciated.

Thanks hoods

---

1Password Version: 5.3.2
Extension Version: Not Provided
OS Version: OS X 10.10.5
Sync Type: WLAN, Folder

MrC

Hi @hoods,

I'm betting the export.kdbx is not a KeePass 2 XML export, but is an encrypted KeePass 2 file. As per the KeePass 2 export instructions:

● KeePass 2

Launch KeePass 2, and export its database to an XML export file using the File > Export ... menu item, and select the KeePass XML (2.x) format. In the File: Export to: section at the bottom of the dialog, click the floppy disk icon to select the location. Select your Desktop folder, and in the File name area, enter the name pm_export.txt. Click Save, and you should now have your data exported as an XML file by the name above on your Desktop. You may now quit KeePass 2.

hoods

Omg, it can be so simple .... :(

I thought keepass2 is the option for the "encrypted" file and keepassx for the xml version .... thanks a lot for your quick answer !

So long, hoods

MrC

Version 1.06 has now been promoted to stable.

bahanx1

Sir,

I am still getting nowhere with this. I am using OS/X 10.11 and convert_to_1p4_1.06

Here is my output:

LANDMINE:~ r*******$ cd Desktop/convert_to_1p4
LANDMINE:convert_to_1p4 r*******$ perl5.16 convert_to_1p4.pl Keychain -v ../pm_export.txt
Keychain parse failed, after entry 0; unexpected:

Thanks for your assistance.

MrC

@bahanx1 ,

I asked in your other thread: "How did you generate that pm_export.txt file?"

The converter is failing immediately upon trying to read the first record of your data. That is indicating to me it was passed a file not containing Keychain data. So i need to know how you created the file. What was the exact command line you used? I showed mine in the other thread.

If you want some Remote help, I'm happy to do so. My email address is at the top of the script - send me an email, and I'll reply with instructions how we can do this.

411pass

Is it possible to import file attachments from KeePass2 into 1Password? I tried it and I don't see any attachments for some of the entries that I know have some, but they may have ended up in a different place.

Does the -modified flag work with the KeePass2 converter? When I tried it I got a parsing error (I don't have it handy right now).

Thank you for your help and your effort in putting together these converters!

MrC

Hi @411pass,

Let me take a look at the attachments that KeePass2 makes available.

The --modified flag is supported by this converter - it works for me, so if you have some case that doesn't work, please provide whatever details you can. You can use the -d and -v flags to help show you more info.

411pass

Here's how I called the converter:

perl ./convert_to_1p4.pl keepass2 -d -v -f -m --nowatchtower pm_export.txt

And here's how the --modified fails:

char_handler        :  **** Field: LastModificationTime: ==> '2014-09-20'
char_handler        :  **** Field: LastModificationTime: ==> 'T15:30:36Z'
Uncaught exception from user code:
    Error parsing time at /System/Library/Perl/5.18/darwin-thread-multi-2level/Time/Piece.pm line 469.
    XML::Parser::parsefile('XML::Parser=HASH(0x7fa00300bb20)', 'pm_export.txt') called at Converters/Keepass2.pm line 71
    Converters::Keepass2::do_import('pm_export.txt', undef) called at ./convert_to_1p4.pl line 116

Not sure why the field is broken up into 2, it doesn't seem to do that for any of the entries above the failed one and the line in the XML file seems OK:

    <Times>^M
       <CreationTime>2014-09-20T15:29:07Z</CreationTime>^M
       <LastModificationTime>2014-09-20T15:30:36Z</LastModificationTime>^M
       <LastAccessTime>2014-09-20T15:30:36Z</LastAccessTime>^M
       <ExpiryTime>2014-09-19T21:27:12Z</ExpiryTime>^M
       <Expires>False</Expires>^M
       <UsageCount>1</UsageCount>^M
       <LocationChanged>2014-09-20T15:29:07Z</LocationChanged>^M
    </Times>

MrC

Hi @411pass,

The attachments cannot be imported by the converter into 1Password currently. What I can do for you is convert the compressed, encoded attachment data back into files, and place those files into a KeePass Attachments folder. The files will be named with their original names, with each record's attachments placed in a sub-folder whose name will be based on the record's Title.

Regarding the --modified error you see. I'm going to change the converter to use a different XML parser. The parser I'm using can break some tokens in inopportune spots, and I'd rather use a simpler XPATH-based XML parser for a couple of reasons, including fixing this bug, and for dealing with the large attachment data.

Give me a day or two.

411pass

Thanks, it sounds good. The attachments are not a huge deal, I don't have a lot of them, I could fix them by hand with a bit of work, but I think it's good to fix it in general for others who come after me and want to do the same thing.
The last modified date I'd like to preserve if possible.
Thanks for your hard work.

MrC

@411pass,

I've placed version 1.07 of the converter suite in the _testing folder of my AgileBits Utilities folder. See the Changes.txt file for more information.

Let me know how it works!

sparky76

Generated an export file with Keepass 2.30 (keepass xml 2.0) and tried version 1.0.7 from https://dropbox.com/sh/a3skeey2zqimdlv/AAD9afRFP-otpL5G1eG4OrVTa/_testing/convert_to_1p4_1.07.zip?dl=0

I am using command line "perl convert_to_1p4.pl keepass2 -v --folders 1password.csv" and get error message:
Error: failed to load converter module 'keepass 2'
Can't locate XML/XPath.pm in @INC (you may need to install the XML::XPath module) (@INC contains: c:/Perl64/site/lib c:/Perl64/lib .) at Converters/Keepass2.pm line 25.
BEGIN failed--compilation aborted at Converters/Keepass2.pm line 25.
Compilation failed in require at convert_to_1p4.pl line 38.

I have no idea how to fix this to get my Keepass data imported.
I really really would like to see a simple Windows Converter program from agilebits :-(

MrC

Hi @sparky76,

Please see steps 3c, 3d, and 3e in the README.pdf. It instructs you to install some additional required modules into the Perl environment. You are missing the XML::XPath module, and that is step 3e.

I hear you about wanting a simple converter. Everyone wants one, and I'm fully aware of the burden the converters place on users (esp. in Windows). They are the way they are out of necessity, to provide you with the assurance that your data is not being transmitted anywhere, and so that they can run on various platforms. It would simply not be possible for me to create native apps to work on OS X and Windows and have so many converters available to users. Nor do I believe AgileBits would be able to afford the time required to do this either.

411pass

@MrC,
Thank you for the fixes, it works well now. Keep up the good work on the converters. They don't need to be pretty as long as they work well, they are a 1-time deal (but very useful).

AGAlumB

Keep up the good work on the converters. They don't need to be pretty as long as they work well, they are a 1-time deal (but very useful).

Couldn't have said it better myself. Cheers! :chuffed:

MrC

I've promoted version 1.07 to stable. Changes:

Version 1.07:
- New: The keepass2 converter will decode and convert an entry's attachments. They are placed in a folder named 1P4_Attachments in the same location that the 1P4_import.1pif file will be created. An entry's attachments are placed in a sub-directory named with the entry's Title.
- Fix: The keepass2 converter sometimes failed to parse an entry's modification time (and theoretically could have failed to parse other items correctly). To resolve this, and to robustly support attachments, the parser was rewritten, using different XML parsing modules.
- Fix: Essentialpim converter might have missed detection of the notes or title of a customized entry due to an off-by-one error iterating through a loop.
- New: CSV converter. This converter is just a basic CSV converter that currently only supports Logins and Notes. It will be customized or updated as necessary to accommodate specific user needs.
- Change: make UUID values all uppercase; 1Password was expecting these when using OPVault (used for creating band files).

fowlesp

Hi MrC

Ok, so RFM, and its works importing data from SplashID v7.2.0.751

Now I need instructions on how to customise the import functions to account for my custom fields...

MrC

@fowlesp,

Great!

Currently your custom fields are going into the Notes section of an entry, in Key: Value pairs.

The SplashID converter works by matching the stock label names for a given category against the list of labels in a table in the converter. When a match is found, the converter maps that field value to a 1Password field. When the converter cannot detect a match on a field, it will place items in the Notes section as mentioned.

Now, you can teach the converter your field names - if they map to a stock 1Password field, then all that needs to be done is to update that table. If, on the other hand, there is no natural mapping into 1Password's fields, you can add an entry to that table AND include a call to a function to tell the converter where to place the field and its value (you give it a Section name, field name, a field type, and a unique keyword).

Whether or not you want to spend time teaching the converter about your fields depends on how many different fields you have, and how consistent they are within a category.

SplashID supports up to 10 fields, any of which can be given custom labels. If you've routinely changed fields/labels somewhat arbitrarily, you might find it more trouble than it worth to customize the converter. But if your labels are consistent across your categories, and/or within categories, and you have many of these, it might make sense to do.

Do you have a sense for your data, with regards to what I just explained?

AnBeSG

Hi,
New to this and tried to convert my Handysafe file but looks like it does not work at all. In terminal window I get the following error:-bash: /usr/bin/perl5.16: No such file or directory

MrC

Hi @AnBeSG ,

Which version of OS X are you running?

Please show your entire command line.

AnBeSG

OS X 10.8.5
Anderss-MacBook-Air:~ andersberglund$ cd '/Users/andersberglund/Desktop/convert_to_1p4/' && /usr/bin/perl5.16 convert_to_1p4.pl handysafe '/Users/andersberglund/Desktop/pm_export.xml' -v
-bash: /usr/bin/perl5.16: No such file or directory
Anderss-MacBook-Air:convert_to_1p4 andersberglund$

MrC

@AnBeSG

For 10.8.5, use simply perl rather than /user/bin/perl5.16.

AnBeSG

OK, but this is automatically generated by the Applescript convertion helper....?

AnBeSG

Trying to copy the same command line but using only "perl", then I get an error saying that perl 5.14 is require and I only have 5.12.

Next I downloaded latest perl (5.20), now I get this error:
Error: failed to load converter module 'handysafe'
Can't locate XML/XPath.pm in @INC (you may need to install the XML::XPath module) (@INC contains: /Users/andersberglund/Library/ActivePerl-5.20/lib /usr/local/ActivePerl-5.20/site/lib /usr/local/ActivePerl-5.20/lib .) at Converters/Handysafe.pm line 22.
BEGIN failed--compilation aborted at Converters/Handysafe.pm line 22.
Compilation failed in require at convert_to_1p4.pl line 38.

Usage: convert_to_1p4.pl

converters:
clipperz csv dataguardian datavault essentialpim ewallet handysafe ironkeyim keepass2
keepassx keychain lastpass licensekeeper msecure nortonis onepif2html passpack
passwordagent passworddepot passwordwallet safeincloud safewallet spbwallet splashid
vcard

specify one of the converters above on the command line to see complete options

MrC

@AnBeSG

Sorry, I'll get that fixed. Just let the AppleScript do its thing and fail. Then, use up arrow to recall that command line and use the cursor to change the perl command,

fowlesp

@MrC

It sound like you've really though through this converter :-)

The problem you've identified (everything going into the notes field) is exactly what I'm seeing, so I assume this is default behaviour?

OK, my data is consistent with each category. The custom categories are for things like "Valuable Items" or different types of Email or FTP accounts etc.

I'd like to move from Splash to 1Password (having tried it), but of re-entering the data is a no-go, so I'm happy to teach the convertor the new types. I'm also happy to post my modifications (assuming I get them to work) as they may help others?

Best regards

MrC

Hi @AnBeSG,

First off, I'll apologize for the troubles you're having.

I haven't tested against Mountain Lion in a long time - its now 3.5 years old. In the past, I'd giving some users some suggestions such as reducing the minimum required Perl version inside the script. I don't recommend this, as certain Unicode bugs were fixed in 5.14.

If you are using ActivePerl, you'll need to install the modules via ActivePerl's PPM installer. I've never used ActivePerl on OS X. The command on Windows would be ppm install xml-xpath. See if that works.

AnBeSG

@MrC
Still the same problem... Here command line and error:
Anderss-MacBook-Air:convert_to_1p4 andersberglund$ perl convert_to_1p4.pl handysafe '/Users/andersberglund/Desktop/pm_export.xml' -v
Error: failed to load converter module 'handysafe'
Can't locate XML/XPath.pm in @INC (you may need to install the XML::XPath module) (@INC contains: /Users/andersberglund/Library/ActivePerl-5.20/lib /usr/local/ActivePerl-5.20/site/lib /usr/local/ActivePerl-5.20/lib .) at Converters/Handysafe.pm line 22.
BEGIN failed--compilation aborted at Converters/Handysafe.pm line 22.
Compilation failed in require at convert_to_1p4.pl line 38.

Usage: convert_to_1p4.pl

converters:
clipperz csv dataguardian datavault essentialpim ewallet handysafe ironkeyim keepass2
keepassx keychain lastpass licensekeeper msecure nortonis onepif2html passpack
passwordagent passworddepot passwordwallet safeincloud safewallet spbwallet splashid
vcard

specify one of the converters above on the command line to see complete options
Anderss-MacBook-Air:convert_to_1p4 andersberglund$

MrC

@fowlesp,

Thanks - the converter suite has grown over time to suit the needs of the various password managers.

To customize the converter, you have to modify the %card_field_specs table. it consists of a series of sections, each a category specific to a password manager. The SplashID converter has addresses, bankacct, clothes, etc. Within each of these categories is a list of field descriptions which specify the field key name to be used in 1Password, and a string value that is the field's name within SplashID. To add more fields to be recognized, you add an entry within the category.

To get that entry added as a custom field within a (possible custom) Section, you have to add some add_new_field() calls to the do_export() routine. The SplashID converter already has some there, for membership and bankacct categories. These specify the relevant category, the field key name from the %card_field_specs table, the section name (you can use the predefined values 1Password uses, or use your own section name), the data type, and finally the label for the field. Examine these entries in the Splashid.pm converter module. Also look as some other converters that also add new fields this way.

There's a reason I don't just arbitrarily add new fields and new sections - I don't want to overwhelm an entry with possibly dozens of custom fields, and users are likely going to want to change them anyway, so its easiest to get the data into Notes, and let users do what the want.

See if this helps.