mSecure Converter for 1Password 4

MrC

I've created a script to convert your mSecure data into a format that can be imported by 1Password 4 and up.

Edit: this version of the converter has been obsoleted. The new and improved version is located here:

https://discussions.agilebits.com/discussion/30286/mrcs-convert-to-1password-utility/p1

Megan

Hi @MrC,

Thank you so much for all your help with importing - you're truly awesome!

ddsocks

Hey MrC
Just got 1password and have been using mSecure for a while now. So downloaded you script and followed the notes but the reply I get is

cd Desktop/msecure_to_1p4
Des-Retina-MacBook-Pro-777:msecure_to_1p4 dd$ perl msecure_to_1p4.pl ~/Desktop/msecure_export.csv
Failed to open CSV file: /Users/dd/Desktop/msecure_export.csv: No such file or directory
Stopped at msecure_to_1p4.pl line 148.
Des-Retina-MacBook-Pro-777:msecure_to_1p4 dd$ perl msecure_to_1p4.pl ~/Desktop/msecure_export.csv
Can't use an undefined value as an ARRAY reference at msecure_to_1p4.pl line 164, <$io> line 3.
Des-Retina-MacBook-Pro-777:msecure_to_1p4 dd$

any ideas what this might be.
Cheers D

MrC

The error is the script telling you that the file named "msecure_export.csv" doesn't exist on your Desktop. Confirm the exact file name you created when you exported from mSecure. I had suggested when you create it that you place it on the Desktop, so hopefully you see it there. It is possible that it was created in another folder.

ddsocks

I must be doing something wrong... here is a screen shot of the items on my desktop if that helps.

ddsocks

Everything seems to be in the right place, but still get the same message. I made sure to use the filename you suggested.

Enzo1953

Hi MrC

like others I just got 1password and have been using mSecure for a while (>200 passwords). I downloaded you script and followed the notes but the reply I get is:

Last login: Sat May 10 15:43:33 on ttys000

Enzos-MacBook:~ heinzgatschet$ cd Desktop/msecure_to_1p4

-bash: cd: Desktop/msecure_to_1p4: No such file or directory

Enzos-MacBook:~ heinzgatschet$ cd desktop

Enzos-MacBook:desktop heinzgatschet$ msecure_to_1p4

-bash: msecure_to_1p4: command not found

Enzos-MacBook:desktop heinzgatschet$

Both files (msecure_to_1p4.pl and mscure_export.csv) are on my desktop.

I am NOT a PC freak - just an "ordinary" user. therefore I am lost in space with the above comment nor do I know what to do next. I would very much appreciate your help in this case.

Thank you in advance for spending your time!

Enzo

MrC

@ddsocks. Sorry you are having trouble. Don't worry; I'll get you through the hump.

Unfortunately, I can't see your screenshot. I think there is something about posted files and images on the forums here. AgileBits can see them, but users can't. All I can see is a small thumbnail of the screenshot. If you want to email it to me, I'll take a look and we can work through whatever issues you are having (mike @ cappella dot us).

@Enzo1953 - You probably are using Safari, and when you clicked the link above, the zip file gets automatically downloaded into your Downloads folder, and is automatically unzipped for you. So you would have the folder "msecure_to_1p4" in your Downloads folder. You should just move that entire folder to your Desktop. Then the instructions will match. Feel free to use the email above if you need to.

Enzo1953

Hi MrC

thank you very much for your feedback. so far I am able to go to the right directory and have now the folder with all files at the right spot. nevertheless I get the following error message:

Last login: Sat May 10 20:56:30 on ttys000
Enzos-MacBook:~ heinzgatschet$
Enzos-MacBook:~ heinzgatschet$ cd Desktop/msecure_to_1p4
Enzos-MacBook:msecure_to_1p4 heinzgatschet$ perl msecure_to_1p4.pl ~/Desktop/msecure_export.csv
Can't use an undefined value as an ARRAY reference at msecure_to_1p4.pl line 164, <$io> line 2.
Enzos-MacBook:msecure_to_1p4 heinzgatschet$

any idea what's wrong ? Thank you for your support!

MrC

Great, thanks for the update. User ddsocks has the same error, so I suspect there might be a version difference with our mSecures. Can you tell me what version you are using?

The error you get would be caused if there was some empty lines in the CSV file. I've added some diagnostic code to help narrow the problem. If you send me an email, I'll reply with the updated version for you to test.

MrC

Hi folks. I've discovered what the problem is. The Naive Westerner that I am, I failed to consider your wallets being localized. I'm adding code now to the script that will accept your localized field names. Sorry about the troubles... be back shortly.

Enzo1953

and the Oscar for excellent user support goes to........ MrC

thank you for your outstanding job!

MrC

Thanks Enzo1953.

Enzo1953 and I worked through several iterations to narrow the problems, and we got them solved. The two main problems: 1) as post #11 mentions, I didn't realize card types would be in the user's native language, and 2) mSecure allows users to rename their card types, so they didn't match my card types table (which is needed for determining the meaning of the columns). These cards are pushed to type Secure Notes.

Enzo1953 is now able to import all the mSecure data!

I've posted the update on Dropbox above, so folks should re-download the new version and use it.

Problem #2 above could be addressed a little better by allow users to create a table mapping their current card name and its fields to one of the 1P4 import types. If this is something that might help users, I'm happy to implement a table mechanism to do this. I've avoided it because it may feel more complex that is necessary. Feedback welcome.

ddsocks

The latest version you posted worked......it must have been that I modified the " Card types “ in mSecure as you said. My language was set to English so the language issue wasn’t relevant maybe in my case.
Well done for working that out you must be a genius, especially when you didn’t even have the files or my computer with you ….. amazing!
Yes, there are some of those entries that I’ll need to sort out but thats a lot of it done for me.

Thanks so much for persevering … RESULT!

dwaldro1

Getting this error in terminal :

CSV_PP ERROR: 2025 - EIQ - Loose unescaped escape

MrC

Hi dwaldro1,

If you can open the msecure_to_1p4.pl file with TextEdit, and search for the the word "auto_diag", you'll find the line:

my $csv = Text::CSV->new ({ binary => 1, eol => ",\x{a}", allow_loose_quotes => 1, sep_char => ',', auto_diag => 2 });

Change the 2 value in:

auto_diag => 2

to a 1, so that the line is:

my $csv = Text::CSV->new ({ binary => 1, eol => ",\x{a}", allow_loose_quotes => 1, sep_char => ',', auto_diag => 1 });

and save the file. Then try to re-run the converter. You'll still see an warning message output, but the script won't die this time. The warning is indicating the input data is malformed, but the CSV parser is compensating.

Unfortunately my testing with mSecure revealed it had some problems producing properly formatted CSV, so I intentionally caused the script to die with an error like this if more problems where found. I think in this case, the CSV parser used by the script should handle the bad quoting. If the script converts your data successfully, be sure to check the data imported into 1P4.

Please report back your findings, and I'll update the script to be more lenient.

dwaldro1

So I followed the instructions, but coming up with this error:

Renamed card type 'Important Documents' is not a default type, and is being mapped to Secure Notes
Renamed card type 'Important Documents' is not a default type, and is being mapped to Secure Notes
Renamed card type 'Important Documents' is not a default type, and is being mapped to Secure Notes
Renamed card type 'Important Documents' is not a default type, and is being mapped to Secure Notes
Renamed card type 'Important Documents' is not a default type, and is being mapped to Secure Notes
Renamed card type 'Important Documents' is not a default type, and is being mapped to Secure Notes
Renamed card type 'Important Documents' is not a default type, and is being mapped to Secure Notes

CSV_PP ERROR: 2025 - EIQ - Loose unescaped escape

MrC

This:

Renamed card type 'Important Documents' is not a default type, and is being mapped to Secure Notes

is just informational. You have renamed stock card types to "Important Documents", and so these are being mapped to Secure Notes types.

This:

CSV_PP ERROR: 2025 - EIQ - Loose unescaped escape

should also just be informative now, if you made the change suggested above. And you should have the 1P4_import_* files on your desktop.

dwaldro1

MrC,

After making the change as suggested, I am getting the 3 new files on my desktop, but still see an error in the Terminal:

CSV_PP ERROR: 2025 - EIQ - Loose unescaped escape

Unexpected failure parsing CSV: row 159 at msecure_to_1p4.pl line 241, <$io> line 160.

MrC

Go ahead and try to import the 3 new files created, and check the entries.

The warning you are seeing occurs if there is junk at the end of the file that did not appear to be valid CSV. In order to debug this further, I'm going to need to see the problematic entries (with your private data obfuscated, of course, but we'll have to take care not to change any special characters such as double-quote or comma, since these would be the characters that are likely to create invalid CSV). The script is mentioning that like 160 in the export file is the problematic one, so if you import into 1P4 does not look correct, we'll look at lines around 158-162.

dwaldro1

MrC

I believe the error resulted in about 130 records from my mSecure file not being converted.

MrC

I'll send you a private message with my email address so we can resolve this offline.

wktsugue

MrC, you're really a good guy! I downloaded your perl scrip, and followed your precise instructions: it worked fine!! You saved a lot of time to do the same job handily. Very thank you for your help!! **You are big heart, magnanimous !!! **

KC1

I am getting the following error:

**** Hit mSecure CSV quoting bug on row 205: compensating...

'# CSV_PP ERROR: 4004 - EUF -

The first error ('quoting bug') repeats for every line, then it ends with the 4004 error. I am using version 3.5.3 of mSecure. If you have any suggestions, I would be grateful. Thank you!

MrC

Hi KC1,

The quoting bug is an inadequacy in mSecure's export to CSV, and the diagnostic message "Hit mSecure CSV quoting bug..." is just an informational diagnostic. You can most likely ignore these messages, and the data should export fine. See Note 1.

Since the mSecure export is untrustworthy, I have the script error on the side of caution and abort if it detects invalid CSV data. In your case, you can try disabling the auto-abort functionality of the CSV parsing. Edit the script with TextEdit, and find the term auto_diag => 2, and change the 2 to a 1, save the script, and re-run it. You'll still get the warning, but the script should continue and hopefully handle the malformed CSV data.

If you are still having troubles and not producing the 1P4 CSV import files on your Desktop, send me a PM and we can continue offline.

Note 1: I have the script inform you when mSecure is failing to do the proper quoting of the CSV data, and the script tries to compensate when it notices a row has been split into two or more rows. The row number listed in the warning diagnostic (row 205 in the case of the above error message) would be the row number in the exported CSV file. The error occurs typically in an notes column (which, for each output type, is the 4th column in mSecure's CSV output), where a note contains multi-line data and that data contains a double-quote and one or more commas within that note:

Line 1,
Line "two",
Line 3

So, does your notes data in the mentioned rows contain a double-quote and a comma?

Note 2: Regarding the 4004 error, I believe that's a case where one of the rows improperly ends with a double quote. The CSV parser tries to handle this case, so if it does handle it correctly, it is safe to proceed (the setting of auto_diag to 1 allows this). Just check your imported data for the noted rows.

KC1

MrC - I made the change as you suggested. The script ran successfully. I completed your remaining instructions and it worked great. Thanks for developing this and for your help this weekend!

agrinshtein

It appears this will only work for the MAC version. Will this also work for the windows version of 1password?

MrC

hi @agrinshtein,

I wrote the script to be cross-platform, so it might just work there, but I haven't tested it against the Windows version of mSecure. I'll go test it out now. Stay tuned...

MrC

MrC

Ok, with a few changes, I was able to make the conversion work. The CSV export from mSecure is fairly different than the one on the Mac.

Unfortunately, the Windows version of CSV importing is (sorry AgileBits), not very good at all.

So what I need to do is to add to the mSecure converter what I've done to other converters: add the 1PIF conversion format, and this will have much better importing into 1P4 for Windows.

Can you hold on for a day or two?

MrC

Hi @agrinshtein (and anyone else interested),

I have completed the update to the msecure_to_1p4 converter. It now works on OS X and Windows. I haven't finished updating the ReadMe file, but if you are interested in trying it out, I'd love to get your feedback.

The new converter abandons the CSV import, and instead uses the 1PIF format which imports with much better fidelity.

ocuk

Hi MrC

I would be interested in giving the new version a try, I have got approx 500 records of various types I need to convert some of which with weird notes and passwords.