MrC's Convert to 1Password Utility (mrc-converter-suite)

MrC

Hi @moku,

Let's see if we can get you going. Try this (copy and paste the command below into the Terminal window):

ls -F ~/Desktop

Do you see the convert_to_1p4 folder in the output?

moku

Yes, I do see the convert_to_1p4 folder in the output.

MrC

@moku,

If you see that directory, then the command:

cd ~/Desktop/convert_to_1p4

should work. Try copy / paste of the command above (that's a tilde at the front of the path).

moku

That got me farther along the road, but when I try to execute the perl script (using perl15.16 because I'm running Yosemite), this is what shows up in Terminal:

MarksNewiMac:convert_to_1p4 Mark$ perl15.16 convert_to_1p4.pl lastpass -v ../pm_export.txt
-bash: perl15.16: command not found

MrC

Note: perl15.16 is not what you want - you want perl5.16 (it is version 5.16 not version 15.16).

moku

That did it! Thanks so much for walking me through this process and for putting up with my ineptitude.

MrC

You're welcome!

Working command line stuff is a bit like building a house of cards - almost anyone can do it, but oh can it be painful, frustrating, and replete with retries for those who don't do it frequently!

Enjoy 1Password...

AGAlumB

Working command line stuff is a bit like building a house of cards - almost anyone can do it, but oh can it be painful, frustrating, and replete with retries for those who don't do it frequently!

:+1: :lol: :+1:

MrC

@ptr727 ,

If I say to replace, the entries in question appear to be blank, no details other than name.
If I respond no, then there are two entries, one a bank card and one a login card.

This is resolved in 4.3.2.BETA-566.

AGAlumB

@ptr: Indeed. MrC is correct. I'm sorry I missed your post here, but I see that svondutch was able to update the other thread as well. Cheers! :)

MrC

I've promoted version 1.04 to the stable release. The final change list:

Version 1.04:
- New: Data Guardian converter now available.
- New: Datavault converter now available.
- New: Passpack converter now available.
- New: PasswordWallet converter now available.
- New: Option --modified will set the 'last modified' field for a 1Password entry based on the exported modification
date provided by a password manager's export data. See 'Option: --modified' in the README file.
- Fix: An item's notes from OS X Keychain converter could contain a stringified ARRAY reference name.
- Fix: The modification and creation dates were missing from the notes section for the OS X Keychain converter.
- Internal: Remove extraneous, remnant argument to UUID generation call.

TechDolphin

For the KeePass 2 converter, would it be possible to add an option to create a new section (perhaps, called "Additional Info" or "KeePass Fields") and then map each custom field from KeePass into a text field with the corresponding name under that section?

For example, I typically have the following custom fields in KeePass:

• E-mail [and often E-mail2, E-mail3, etc.]
• Phone [and often Phone2, Phone3, etc.]
• Security Question #1 [and Security Question #2, etc.]
• Security Answer #1 [and Security Answer #2, etc.]
• PIN
• Account Number
• Capacity
• Two Factor Status
• Account Creation Date
• Encryption Key
• ...and many more

It would be REALLY nice to automatically import all of these in an organized fashion rather than having all of them added to the Notes section. ... Maybe there is already a way to do this and I just missed it. Thanks! :-)

MrC

Hi @TechDolphin ,

Additional fields could be added to an entry, and in fact, various converter modules do this via the add_new_field() function, called in do_export(). The issue with doing this for every custom field is one of sanity checking, and keeping constraints on how many fields are created for a 1Password entry. I wouldn't want to having some errant input data run amok and create a hundred fields in an entry. So, the converters only map fields they know about (based on the table defined by the %card_field_specs hash), and any additional fields that are obvious when converting from fields within a password manager's categories that don't directly map to 1Password fields - these fields are added programmatically to the table just prior to import (and its a little complicated to explain why they are added programmatically rather than just added to the table statically - it has to do with how that table is used to drive both parsing of the input data structure and in the determination of an entry's type).

I can show you how to add entries to the table and programmatically if that helps.

TechDolphin

Hey @MrC ,

Thanks for the quick reply. That makes sense. If it's not too much trouble, I would be grateful if you showed me how to add entries. I have a fairly large KeePass database, and the vast majority of the entries have some custom fields. Not having an easy way to import that information into 1Password in an organized manner is what has kept me from switching to 1Password.

MrC

Hi @TechDolphin,

Adding the additional fields is not too difficult - there are some caveats, but we'll skip those for now. Let's start with the basics.

The file Converters\Keepass2.pm needs to be modified in two locations, as mentioned above: 1) the %card_field_specs table, and 2) the do_export() function.

The %card_field_specs hash lists the types of records that a password manager has, and then lists that record type's known field names. Keepass2 only has Login entries, so the table is simple:

    login =>            { textname => undef, fields => [
       [ 'url',             1, qr/^URL$/, ],
       [ 'username',        1, qr/^UserName$/, ],
       [ 'password',        1, qr/^Password$/, ],
    }

The fields are 'url', 'username', and 'password', and these are detected by the converter when reading the file by looking for the unique strings in the export 'URL', 'UserName', and 'Password', respectively (the stuff inside the qr/.../ is a regular expression pattern, and it should be as strict as possible). So when the string 'URL' is found, its value is mapped to a field key 'url' for the entry (the converter maintains a list of found records, by key / value pairs). The 1 you see in the entries tells this convert that it is safe to set the type of the entry (in the case, its 'login'). Some patterns in may be found in various categories in other converters, and in those cases it would not by safe to assume the record type (aka category).

Now let's say you want to add new custom fields to your Login entries, and these are Account Number, and Custom Fld 1. To do this, we add the patterns to the table, and assign a unique field key:

    login =>            { textname => undef, fields => [
       [ 'url',             1, qr/^URL$/, ],
       [ 'username',        1, qr/^UserName$/, ],
       [ 'password',        1, qr/^Password$/, ],
       [ '_acctnum',        0, qr/^Account Number$/, ],
       [ '_custfld1',       0, qr/^Custom Fld 1$/, ],
    }

The custom field keys which are not stock keys, are prefixed with an underscore (my own preference), and then any unique name can be chosen. So when the converter is parsing the Keepass2 XML export, and finds the fields exactly named 'Account Number' and 'Custom Fld 1', it will map their values to '_acctnum' and '_custfld1'. So now the data is stored. And notice the 0 in the field defintion instead of a 1 this time - there is no reason to tell the converter to assume the record type to Login, simply from seeing the field Account Number, or Custom Fld 1. A 0 here means - don't set the record type when this field is found; a 1 means the type may be set.

Let's move onto getting the converter's exporter to know that those new custom keys should go into 1Password custom fields, and not into Notes. That's done in the do_export() function via a call to add_new_field(), and I'll show that function with the two new calls added:

sub do_export {
    add_new_field('login',   '_acctnum',   'other.Other Information',  $Utils::PIF::k_string, 'account number');
    add_new_field('login',   '_custfld1',  'other.Other Information',  $Utils::PIF::k_string, 'custom field 1');

    create_pif_file(@_);
}

These functions add new custom fields to the 'login' type, named '_acctnum' and '_custfld1', respectively, and they are to be stored in a Section named 'Other Information', and they are both of type 'string', and their custom field names will be 'account number' and 'custom field 1', respectively. And for simple things, that's all there is to it.

There are more complex things that can be done (such as exploding a record into multiple output records, modifying a records title based on some field, customized processing callback functions to massage the data between import and export, etc.). I suspect given the names of some of the fields you are using you might like to have some record splitting (e.g. to create both a Login and a Bank Account record from a single Keepass2 login record). But let's take it one step at a time.

Make the modifications above to the Keepass2.pm converter module (tailored to your own field names), re-run the conversion and import, and you should have your new custom fields within the Login record's custom fields and no longer within the Notes area.

TechDolphin

@MrC ,

Thank you! This is very helpful. Based on your explanation and taking a look at PIF.pm, it appears that your converter has the power to do a lot of sophisticated importing with some up front configuration/customization on my end. I look forward to playing around with this! Am I correct that I can change the field type in 1Password from standard text to something else (e.g., a secondary password field) by changing $Utils::PIF::k_string to something else (e.g., $Utils::PIF::$k_concealed)?

I can't find a way within 1Password's interface to convert a Login record to another type--for example, to convert a Login to a Wallet Item-->Bank Account. Am I correct that there is no way to do that within the 1Password program itself? If so, I am thinking it may be easier to explore doing those conversions through your converter utility. (It would be nice if 1Password allowed you to open more than one record at a time so you could have old Login open and then a new Bank Account record open next to it so as to facilitate copying and pasting from the old record to the new record.)

MrC

Hi @TechDolphin,

You're very welcome.

Yes, you are correct that you can change the kind of data (string, concealed, etc.). But please look at examples of how these data kinds are used in other converters and in Utils\PIF.pm. There are some special cases that have to be managed, but simple changes should not be a problem.

The only category conversion 1Password supports is from a Password to a Login field. So you'll want to get your categories created correctly in the 1PIF file (i.e. before import). And this limitation is one of the reasons I don't create too many custom fields - it is much easier to bulk copy from Notes all the field/value pairs into another new 1Password entry, than to copy/paste field name, field value, from one record to another.

One thing I could do to make adding new fields to the converter easier is to construct another table mechanism, which you could just fill with values, and the do_export() function will handle the calls to add_new_field() for you. Its a minor difference, but lends to easier customizations seeing the table at the top of the file and not having to worry as much about adding new lines of code.

After you've played around a bit with the simple customization we worked out above, let's continue this discussion via email for any of your additional customization needs. My email is at the top of the script files.

TechDolphin

@MrC,

That sounds like a good plan. I will be in touch via e-mail once I've had the time to sit down and experiment with the converter. Thanks!

Unknown

This content has been removed.

MrC

Hi @Sefer,

You're welcome!

I'm sending you a PM with instructions. Nothing confidential about them, I just haven't personally verified them yet for public consumption (others have used them).

Unknown

This content has been removed.

MrC

Thanks for the update @Sefer. I'm happy to hear your data is now converted and imported.

nicebeevor

Hi there, I have downloaded and unzipped the convert_to_lp4 folder and moved it to my desktop, but when I open the readme file it is blank so I can't find the instructions. Are they available anywhere else please?

MrC

Hi there - I just checked the README.pdf files in all three locations - the 1.04 version in my Dropbox area, the 1.05 version in the _testing folder, and the 1.04 version posted by AgileBits on GitHub. The README.pdf contains content in all locations. Some questions:

• Where exactly did you get the zip file from?
• What is the file's size (in bytes)?
• Which OS did you download/upzip on?
• Which PDF viewer are you using?

nicebeevor

Hi there MrC - thanks for your answer. I got it off the Dropbox/AgileBits Utilities folder; total size was 324KB; readme is 167 KB; unzipped via IOS 10.10.3; read with Adobe reader.

MrC

Can you try opening the the PDF with Preview instead?

nicebeevor

Result! Thanks for the tip.

MrC

You're welcome. I don't know why Reader is having trouble with the PDF generated (by the Marked program, from the .md file). I can let the author know about the issue. Thanks.

DJRiful

This tool save my time! Big thank you!

MrC

Another happy customer makes a great weekend for both of us!

You're welcome.