MrC's Convert to 1Password Utility (mrc-converter-suite)

jriga

By the way, full error is:
Can't locate Utils.PIF.pm in @INC (you may need to install the Utils::PIF module) (@INC contains: C:/myperl/perl/site/lib C:/myperl/perl\lib) at convert_to_1p4.pl line 15. BEGIN failed- -compilation aborted at convert_to_1p4.pl line 15.

MrC

Hi @jriga,

First, use the 1.10 version of the converter in Testing Bits, mentioned in the first post of the converter suite thread. It resolves the issue you mention in your second post above.

Password Safe isn't in the AppleScript helper, since I had not verified Password Safe converts reliably on a Mac. My quick tests shows it does, but I just don't recall how much testing I did. Go ahead and try it there - use the command line for now.

Make sure you put the folder convert_to_1p4 onto your Desktop - you have it inside onepassword-utilities. The instructions are based of the Desktop location - if you move it, or leave it elsewhere, you'll have to modify the cd and the path to your export appropriately.

jriga

@MrC OK 1.10 downloaded. To be clear: you want me to run this on the Mac now using the 1.10 converter and the Password Safe XML files generated on the PC?

MrC

@jriga ,

Since it's XML, it should work in either place. Try it on either platform, and let me know how it works out. Typically you should convert on the platform you export on, but some exports are agnostic.

If you see any issues converting on the Mac, let me know and I'll see if I can fix them right away.

jriga

Looks like it worked perfectly... thanks SO much. Do you have a Patreon or paid support model?

MrC

@jriga,

Outstanding!

Your "thanks" is my payment.

Enjoy 1Password!

ps. I'll add Password Safe to the AppleScript now.

Edit: 1.10 version in Testing Bits updated.

marc_laederach

Hi MrC

First of all thanks a lot for spending so much time and effort for a feature which I would expect to be part of the software itself!

I would like to convert an exported 1pif to HTML so I can print it and send the password information to a customer so they can keep them in a safe for emergency situations.
So I found your 1password converter utilities but unfortunately it's not working yet. Basically I followed the Readme very carefully but I couldn't get it to work. I ran into errors a few times, first I had to downgrade strawberry perl prior to v5.26, then I got uncaught user exceptions so I updated the converts to Test Bits 1.10.
I'm using the following command to convert 1pif to html: perl convert_to_1p4.pl onepif -d

And I'm getting the following error:

Uncaught exception from user code:
        malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "\x{bf}\x{bf}{"uuid":...") at C:/Users/Marc/Desktop/convert_to_1p4/./Utils/PIF.pm line 997.
        JSON::PP::decode_error("malformed JSON string, neither array, object, number, string "...) called at C:/Users/Marc/Desktop/convert_to_1p4/Modules/JSON/PP.pm line 1045
        JSON::PP::word() called at C:/Users/Marc/Desktop/convert_to_1p4/Modules/JSON/PP.pm line 728
        JSON::PP::value() called at C:/Users/Marc/Desktop/convert_to_1p4/Modules/JSON/PP.pm line 688
        JSON::PP::PP_decode_json(JSON::PP=HASH(0x3c0ef7c), "\x{bf}\x{bf}{\"uuid\":\"364127E0DE304AA6984E9C3D3B9D9DEB\",\"folderUuid\":\"5D"..., 0) called at C:/Users/Marc/Desktop/convert_to_1p4/Modules/JSON/PP.pm line 148
        JSON::PP::decode(JSON::PP=HASH(0x3c0ef7c), "\x{bf}\x{bf}{\"uuid\":\"364127E0DE304AA6984E9C3D3B9D9DEB\",\"folderUuid\":\"5D"...) called at C:/Users/Marc/Desktop/convert_to_1p4/Modules/JSON/PP.pm line 110
        JSON::PP::decode_json("\x{bf}\x{bf}{\"uuid\":\"364127E0DE304AA6984E9C3D3B9D9DEB\",\"folderUuid\":\"5D"...) called at C:/Users/Marc/Desktop/convert_to_1p4/./Utils/PIF.pm line 997
        Utils::PIF::get_items_from_1pif("C:\\Users\\Marc\\Desktop\\Kathand.1pif") called at C:/Users/Marc/Desktop/convert_to_1p4/./Converters/Onepif.pm line 92
        Converters::Onepif::do_import("C:\\Users\\Marc\\Desktop\\Kathand.1pif", undef) called at convert_to_1p4.pl line 140

I would be glad if you could help me to get this converter working.

Kind Regards
Marc

MrC

Hi @marc_laederach ,

Sorry for the late reply - for some reason, I didn't not receive a notification from the forum that someone had posted to this thread.

You'll want to use version 1.10, from Testing Bits, mentioned in the first post of the converter suite thread. It seems like you are using it, but I just want to make that clear. Strawberry Perl 5.26 and above is supported, but you need the 1.10 version of the converters.

The re-written onepif converter now uses various formatters - they specify the output format and style. These live in the Formatters directory, and they are specified by name without the suffix. You use the --format option, and the command line will look like:

perl convert_to_1p4.pl onepif --format html_simplelogins -v ~/Desktop/1P_export.1pif

and your resulting file will be on the Desktop named 1P_converted.html .

Example:

$ perl convert_to_1p4.pl onepif --format html_simplelogins -v ~/Desktop/1P_export.1pif
Imported 32 items
Exported 2 email items
Exported 2 passport items
Exported 2 driverslicense items
Exported 2 software items
Exported 2 bankacct items
Exported 2 creditcard items
Exported 2 outdoorlicense items
Exported 2 note items
Exported 2 identity items
Exported 2 socialsecurity items
Exported 2 wireless items
Exported 2 membership items
Exported 2 login items
Exported 2 database items
Exported 2 server items
Exported 2 rewards items
Exported 32 total items
Your output file is /Users/MrC/Desktop/1P_converted.html

Let's see if this helps you.

marc_laederach

Hi @MrC

Thanks a lot for your help. I really appreciate it. I am using Testing Bits 1.10 and Strawberry Perl 5.24 Portable 32-bit. Do I need 32 or 64-bit?
I used your command, but still getting the same uncaught exception errors.

MrC

Hi @marc_laederach ,

The 32-bit version is fine.

You are encountering the problem reported here, that I thought I'd fixed. I have to rely on users to let me know if a fix resolves their issue, since I don't have access to their data. I see now the OP didn't reply, since he resolved the issue on his own. So my fix went untested, and apparently isn't quite right.

Your 1PIF export has a "BOM" as the first two bytes. This normally would not be present in a file encoded as "UTF-8", but sometimes 1Password adds it, and I'm not handling it correctly for some 1PIF files.

Can you tell me which version of 1Password you are using, and which Windows version?

marc_laederach

Hi @MrC

Thanks for the background information why I have this issue. I'm having some issues with the encoding of text files since a few months, where it suddenly changes the encoding from UTF8 to UTF8-BOM. I don't know why and I don't know if it's linked with this issue.

I'm using version 4.6.2.626 of 1Password and Windows 10 Pro v1803 Build 17134.48.

MrC

@marc_laederach ,

Thanks. I have a sample file now. I'll have a fix for you tomorrow. Sorry for the troubles.

marc_laederach

@MrC Thank you so much!

Flip76

@MrC

thank you so much for the uncountable hours you are investing in supporting 1Password. When Agilebits introduced 1Password subscriptions and they didn't want to sell me a 1Password4 (Windows) license anymore, I switched over to Enpass. Now as they are bringing back the normal license with 1Password7 (Windows) I wanted to give it a try. But now I have to export my data from Enpass and import it to 1Password7. I just downloaded your testing version but I see, the enpass.pm only support english at this time... But I realized, there are some converter-modules supporting different languages (language-directory). My question: Is it possible to implement the language function to enpass.pm? I could do the translation (german) if you want?

Best regards,
Flip

MrC

Hi @Flip76

You're welcome.

I started looking into non-English languages for Enpass, but this is likely a dead-end, as Enpass does not have any translation strings available. To make my own, I created sample vaults (in German nontheless) for every possible entry type, with all the stock fields, and discovered that Enpass exports the same field name ambiguously, or in conflict, and often within the same type.

However, Enpass exports the field names based on the language configured. So, I realized a language-specific version was not necessary, since you can configure Enpass to export in English. Then the enpass converter should work. It may need an adjustment based on your specific needs. So, change its language to English, close the app, and re-launch and export.

Be sure to use the 1.10 version of the converter suite, mentioned in the first post in the thread.

MrC

@marc_laederach ,

I've posted an update to the 1.10 version in Testing Bits. Please give it a try.

This turned out to be much more work than I'd anticipated, for the simple reason (I now recall) that I had not tested the updated onepif converter with any of the new formatters using 1Password 4 for Windows. Its 1PIF export is entirely different than those for 1Password 6 and 7, which is where I'd done all of my testing.

It's possible I haven't caught all the glitches due to the differences in 1PIF format. Although I have several 1PIFs generated with many sample entries, it's hard to anticipate all the ways 1P4 could be different. Please let me know if you find any errors.

Flip76

Hi @MrC

thanks for your quick response. Unfortunately it seems like Enpass is not exporting the field names in the choosen language. I am using version 5.6.9 and the field names in the csv are mainly in german. Another proglem: Usernames for login-types are sometimes labeled as "email" and sometimes as "Login"...

Best regards,
Flip

MrC

@Flip76 ,

Hmmm, this wasn’t my experience on the Mac side with Enpass. Let me recheck in the morning.

Enpass’ export is an ambiguous mess.

Flip76

Thanks @MrC

I am using Enpass on Windows and I think the big problem is to handle the user-defined labels in Enpass - those wouldn't be translated at all.

Regards,
Flip

MrC

Hi @Flip76,

I tested out the language settings using Enpass portable for Windows. When I change the language, entirely quit the app, and relaunch it, the field labels are changed to the chosen language, as are the labels in the CSV export.

I'm confused by your results.

The converter works by matching a minimum number of labels in a record. The category/field definitions table in the converter defines a list of field names for a category, and the mapping to 1Password's category and its fields (and how to handle them). It generally is not a problem to add additional fields in a customized version of a converter, so this gives users the flexibility to convert their customized data.

Because the Enpass export contains no category identifier, only the field names are available candidates to use for category detection. Worse yet, Enpass will suppress empty field/value pairs, making positive category identification impossible in some cases. Even worse still, Enpass outputs the same field label within a record, and when it suppresses a field/value pair that came earlier in the record, it is impossible from the export to detect which it is. I describe these issues here.

To get reasonably good detection of record categories. I had to create and evaluate each and every record type, with its stock fields, and create tables to match a certain minimum number of those fields. This was done empirically using the English strings, and the converter deals with the ambiguities the best it can. E.g. if the converter sees a certain field such as CVV, and another credit card-specific field or two, it knows the record is (likely) a Credit Card. But it cannot determine a Login positively because the URL, Username and Password fields are available in just about every Enpass category - it must rule other all the other categories first.

Finally, Enpass' translations are poor, in that it will export two different string names for the exact same field label, make the category detection impossible in some cases. Inconsistent translations frustrate the techniques mentioned above - this technique utterly breaks-down when relying on poor language translations in the export when the field names are inconsistent or wrong.

Flip76

Hi @MrC

thank you so much for your detailed answer. I think I figured out the reason for my translation problem. My first wallet software was Safewallet. After they disappeared, I imported my data to mSecure. From mSecure to 1Password and finally to Enpass. Now back to 1Password. So it seems like all my export/import actions resulted in a total mess (mainly customized labels which will not be translated).

I spent all day on adjusting the enpass.pm to fullfill my needs (I hope you don‘t mind!) and finally I managed to get all my data imported to 1Password. This way I realized the big problem with the record categories. On of my big problems was that sometimes a record got extracted in different categories - e. g. Credit Cards: A part of the record appeared in Bank accounts and the other part in Credit Cards. Now I have a lot of data in the notes-field but anyway I need to cleanup to get rid of this mess...

Thanks a lot for all your support - I really appreciate it!

Best regards,
Flip

Flip76

Oh I forgot one last thing... Is there any function to change the date/time-format? E. g. for credit cards expiry date which is in mm/yyyy format (Enpass)...

MrC

@Flip76,

You're very welcome.

And by all means, the converters were designed so that it would be possible to customize them - I'm happy you worked your way through the category/field table successfully!

You are not the first user who has encountered issues due to jumping from one password manager to another. There's nothing software can do in these cases, in terms of automatic detection.

1Password works using a certain paradigm - that is, certain categories have specific meanings and utility. The converter tries to accommodate this by splitting source records where it makes sense, to naturally fit into 1Password's categories. This is why you see, for example, a Bank Card entry being split into three different types: Bank Account, Credit Card, and Login. This allows 1Password to form-fill on web sites. Otherwise, you'd have to manually add new entries, duplicating the information, or moving it from one record to another. You can disable this by editing the %card_field_specs entries - I'll show you how, if you are interested.

Enpass' date / time entry is non-validated, free-form garbage. So the converter cannot know what to do with date entries such as 01/01/01, or 2001,02,03, 02/03, 3/2, 3/18 (date/year or month/date?), or 02/31/2019 (invalid), or FOOBAR. Their developers are lazy.

If you are certain that all of your dates are in the mm/yyyy format, I can show you how to add date-validating functions and do the correct field assignments. 1Password has both Unix-style Epoch dates and MonthYear style dates, depending on the specific field.

Jul

Hello MrC, first thanks for your great job :-)

I try run "convert_to_1p4" to convert exported data from 1Password.
Here is what I obtain :

    main                : Runninng script from 'C:/Users/Julien/Desktop/convert_to_1p4'
    main                : Command Line: onepif -v -d ..\UNENCRYPTED_DATA.1pif
    main                : Output file: C:\Users\Julien\Desktop\1P_import.1pif
    print_fileinfo      : Export file info: "..\UNENCRYPTED_DATA.1pif"
    print_fileinfo      :   size: 256604
    Can't use an undefined value as an ARRAY reference at
            C:/Users/Julien/Desktop/convert_to_1p4/./Utils/PIF.pm line 1074, <$io> line 1 (#1)
        (F) A value used as either a hard reference or a symbolic reference must
        be a defined value.  This helps to delurk some insidious errors.

    Uncaught exception from user code:
            Can't use an undefined value as an ARRAY reference at C:/Users/Julien/Desktop/convert_to_1p4/./Utils/PIF.pm line 1074, <$io> line 1.
            Utils::PIF::get_items_from_1pif("..\\UNENCRYPTED_DATA.1pif") called at C:/Users/Julien/Desktop/convert_to_1p4/./Converters/Onepif.pm line 92
            Converters::Onepif::do_import("..\\UNENCRYPTED_DATA.1pif", undef) called at convert_to_1p4.pl line 140

Any idea how to solve that ? Thanks :-)

Jul

MrC

Hi @Jul

You need to specify the formatter to use with the onepif converter. See my response here.

Jul

Hi @MrC, thanks for your quick reply !

Unfortunately, I still have the same mistake :

    C:\Users\Julien\Desktop\convert_to_1p4>perl convert_to_1p4.pl onepif --format html_simplelogins -v ..\UNENCRYPTED_DATA.1pif
    Can't use an undefined value as an ARRAY reference at
            C:/Users/Julien/Desktop/convert_to_1p4/./Utils/PIF.pm line 1074, <$io> line 1 (#1)
        (F) A value used as either a hard reference or a symbolic reference must
        be a defined value.  This helps to delurk some insidious errors.

    Uncaught exception from user code:
            Can't use an undefined value as an ARRAY reference at C:/Users/Julien/Desktop/convert_to_1p4/./Utils/PIF.pm line 1074, <$io> line 1.
            Utils::PIF::get_items_from_1pif("..\\UNENCRYPTED_DATA.1pif") called at C:/Users/Julien/Desktop/convert_to_1p4/./Converters/Onepif.pm line 92
            Converters::Onepif::do_import("..\\UNENCRYPTED_DATA.1pif", undef) called at convert_to_1p4.pl line 140

MrC

@Jul,

Sorry about the issues - can you tell me which version of 1Password this is?

I see what the problem is, but don't know the circumstances to recreate the problem here so I can create test data, to be sure I have the fix correct.

Jul

@MrC ,

My 1Password version is the 4.6.2.626
Please contact me if needed.

Flip76

Hi @MrC

thanks again for your help. If you don‘t mind I wanted to ask you if you can give me a short explanation for the card_field_specs. Especially the number after the variable cause I couldn‘t figure out the functionality behind it...

Best regards,
Dietmar

Flip76

Oh abd regarding the date format...Unfortunately only credit card dates are formated mm/yyyy - german default would be dd.mm.yyyy