MrC's Convert to 1Password Utility (mrc-converter-suite)

MrC

@EllenK ,

When items cannot be copied from an iCloud vault to a local one, it typically means there has been sync errors or corruption in the iCloud keychain. You might find that you can copy some items, but not others. See if you can copy some items, but not another (and I can't really provide any hints on which ones might work).

I've posted about trying to resolve these issues. It typically involves deleting the iCloud keychain and resyncing. But this can be tricky when you have multiple devices. And you certainly want to be sure you have the data in one place so you don't delete all copies. Some people have had luck getting this resolved. It's all a bit of black magic. I'll see if I can find the references.

EllenK

Thanks so much @MrC I'll give that a try! I gave that a try and it seems as if I can't copy or paste anything.

MrC

@EllenK

I found a few references, and copied a comment from another larger thread:

https://discussions.agilebits.com/discussion/comment/346252/#Comment_346252

https://discussions.agilebits.com/discussion/comment/319241/#Comment_319241

https://discussions.agilebits.com/discussion/comment/344616/#Comment_344616

The error about "unable to access item" is likely due to a corrupted iCloud Keychain, which never synced properly between iCloud and your device. A few users have encountered it, and I was able to reproduce it once. The only solution that I know about is to delete the iCloud keychain and resync with iCloud.

EllenK

@MrC Thanks so much for your guidance! I read some of the discussions you posted and I was finally able to get it to work. I don't think it pulled over everything but it pulled over the bulk of my logins. Thanks again!

MrC

@EllenK ,

You're very welcome, and I'm really happy to hear you got at least most of your logins transferred!

rharris

Hi there,

My mom is coming to 1password from Keeper Desktop version 11.0.0 on macOS. She has a newer version than was listed in the conversion utilities, but I still thought it would likely work. Unfortunately one of the things listed in the instructions, "Export to Excel", was not listed within the "Backup" section of the app -- our only choices were to export to PDF or to a text file (which appears to be some sort of CSV format without a header). I was able to export a text file, which was unable to be read by either the utilities or the CSV importer built into 1password.

I first tried this with the version of the utilities linked on the 1password.com Support page, and then tried it again with the Testing version that was available here on this forum thread after I became aware of it.

If anyone could give some advice on the best way to export data from this app at this point, that would be wonderful. Thank you!

MrC

Hi @rharris,

Let me took a look at version 11 / OS X now... hold tight.

MrC

@rharris ,

Ok, it appears now that Keeper 10 no longer exports in their "Excel" format (which was really HTML). It now exports as CSV. You won't be able to use the csv converter, because Keeper exports custom field data (just past the Notes column) as Label/Value pairs. So, I'll update the converter to handle this new CSV format. It will correctly manage the custom fields for you.

I'll work on it now...

rharris

@MrC Thank you for the prompt response! I understand that this will take some time to do and test properly, but I look forward to helping you test this out once it's time. (I would've looked at the code myself, but my perl is... very rusty, to say the least.)

Thanks again!! Your help is greatly appreciated. (If you look at the Keeper app too long, you might start to understand why I'm so excited to finally be getting her over to 1password :) )

MrC

Hi @rharris ,

I've updated version 1.10 in Testing Bits, to handle the CSV exported by Keeper Desktop version 10.

Use the Backup button, select the Export Now button adjacent to Export to Text File. After the file has been exported, and Keeper has opened up the Finder folder where it did the export, drag the export to your Desktop and use this file as the filename (no need to rename it).

Since Keeper exports the field names, if you want the custom fields to go into a custom section, add the option --addfields. Otherwise, the field/value pairs will go to the notes section.

Example command line usage:

perl convert_to_1p4.pl  keeper -v --addfields  ../20170815210201-keeper.txt

I'll update the README later. (EDIT: Done)

Let me know if you find any issues!

MrC

Hi @rharris ,

Just checking in... did the updated keeper converter work for you?

rharris

Hi there, our schedules haven't matched up the right way since the day we tried to do this initially but hopefully they will again tomorrow and I'll hopefully get to test it then! :) Sorry for the delay!

rharris

Sorry to double post, but couldn't quickly see how to edit my previous post...

@MrC, it looks like that version of the converter was able to import the data successfully! Because of time constraints we haven't yet been able to go through and make sure everything that was in Keeper is in there but I'll let you know if anything is amiss when we do. Thanks again for all your help!! It's greatly appreciated :)

MrC

Thanks for the update @rharris!

gautch

Hey @MrC
I'm jumping from True Key to 1Password and your converter seems to not be working.
Do you mind taking a look at it? Below is the debug results from terminal. It says a file (1P_import.1pif) is output to the desktop, but its not there.

`Last login: Thu Aug 31 16:12:10 on ttys000
J761618-OSX:~ j761618$ clear;clear

J761618-OSX:~ j761618$ cd '/Users/j761618/Desktop/convert_to_1p4' && /usr/bin/perl convert_to_1p4.pl truekey '/Users/j761618/Desktop/truekey-export.csv' -v --debug
main : Command Line: truekey /Users/j761618/Desktop/truekey-export.csv -v --debug
main : Output file: /Users/j761618/Desktop/1P_import.1pif
print_fileinfo : Export file info
print_fileinfo : size: 92722
print_fileinfo : kind: UTF-8 Unicode text, with very long lines
print_fileinfo : mime: text/plain; charset=utf-8
Uncaught exception from user code:
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "kind,autologin,cardh...") at Converters/Truekey.pm line 50.
JSON::PP::decode_error('malformed JSON string, neither array, object, number, string ...') called at /System/Library/Perl/5.18/JSON/PP.pm line 1045
JSON::PP::word() called at /System/Library/Perl/5.18/JSON/PP.pm line 728
JSON::PP::value() called at /System/Library/Perl/5.18/JSON/PP.pm line 688
JSON::PP::PP_decode_json('JSON::PP=HASH(0x7fff401f88a0)', 'kind,autologin,cardholder,city,company,country,dateOfBirth,de...', 0) called at /System/Library/Perl/5.18/JSON/PP.pm line 148
JSON::PP::decode('JSON::PP=HASH(0x7fff401f88a0)', 'kind,autologin,cardholder,city,company,country,dateOfBirth,de...') called at /System/Library/Perl/5.18/JSON/PP.pm line 110
JSON::PP::decode_json('kind,autologin,cardholder,city,company,country,dateOfBirth,de...') called at Converters/Truekey.pm line 50
Converters::Truekey::do_import('/Users/j761618/Desktop/truekey-export.csv', undef) called at convert_to_1p4.pl line 125
J761618-OSX:convert_to_1p4 j761618$
`

MrC

Hi @gautch ,

What version of True Key was this, and what platform was the data exported from?

This debug line:

malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "kind,autologin,cardh...") at Converters/Truekey.pm line 50.

indicates there is unexpected stuff before the JSON string that contains your formatted export data.

EDIT: Never mind. I just installed the latest version, and see Intel has changed the export format from JSON to CSV. I'll update the converter to handle this, and will post a reply when its ready later today.

MrC

@gautch ,

I've updated the truekey converter in version 1.10 to accommodate the change in the export format. You'll find it in Testing Bits, mentioned in the first post of the converter suite thread.

gautch

@MrC
You rock! It worked great!!

and131

Hi MrC,

I read your May 2015 comments (MSecure conversion category problems) on how to overcome category issues from mSecure using pearl and convert_to_1p4.pl and modifying Msecure.pm. I tried to follow, but I still ended up with over 400 secure notes rather than neatly organized 1Password records with sections and labels. I am a long time mSecure user running Windows desktop version 3.5.5. In the mSecure.pm file, I did the following:

replaced:textname => 'Web Logins'
with:textname => 'Website'
replaced:textname => 'Identity'
with:textname => 'Identification'
replaced:textname => 'Email Accounts'
with:textname => 'E-Mail Account'

Imported 503 items
Exported 59 login items
Exported 427 note items
Exported 10 email items
Exported 7 identity items
Exported 503 total items

Even the records that the log above suggests imported correctly did not. They were parsed into fields but the sections and labels are wrong.

In mSecure, I hardly ever used the stock categories without modifying them first, and this is surely the root cause. I always found a need to add more rows in a category like security questions and answers or I changed something else. 'Web Logins' is my largest custom category in mSecure. I also have 'Identification', 'E-Mail Accounts' and others. Also, I see most of the records had the following error 'Hit mSecure CSV quoting bug'.

Do you have any suggestions? Did I get the syntax wrong? Should I modify the msecure export file rather than the converter? Should I attempt to build my own CSV? I am estimating 5 minutes per record or 40 hours if I manually input the records into 1Password. Thanks.

MrC

Hi @and131 ,

mSecure certain did make this tough... they gave customers enough rope to hang themselves, and didn't tell you someday the floor would drop.

It would be best to work with your unmodified mSecure export, since modifying it can lead to other unintended consequences. And then we can work together to customize the convert to suit your categories.

First, it appears your export data is in English, so that will simplify the customization. Please confirm if this I am incorrect about this.

Second, you need to identify your mSecure categories and the fields you've added/modified. The export contains no information about the meaning of any given cell of the CSV data exported beyond the first 4.

Third, once we have this table of category names with their list of fields, we can customize the converter. I can either show you how that is done (for the trivial case), or help do more complex things for you.

Let's take this offline, like I've done with others, and we can summarize here when the conversion is done. My email is at the top of the convert_to_1p4.pl script.

In writing this, I just thought of a nice feature I can add to the msecure converter to help users like yourself accomplish the third item above.

hepabolu

Hi MrC,

Thanks for the wonderful converter.
I've exported my CSV file from TeamPass and managed to import it into 1Password with your converter script.

MrC

You're very welcome @hepabolu. Thanks for the kind feedback!

Tidewind

Hi! I am a brand new subscriber to 1Password for my Mac, iPhone, and iPad. After losing a month of data in the completely hosed mSecure 5 (and basically losing my sanity in the process), I have made the jump to 1Password. I'm writing to seek help on mapping my category types from mSecure to 1Password.

So far, I have used the convert_to_1p4 utility for macOS. After carefully reading and re-reading the README file, I proceeded. To my dismay, I found that about 400 records defaulted to Secure Notes, including renaming the record type names in mSecure 3.5 on my Mac to match the corresponding Categories in 1Password. No luck — my 400+ login records were still converted to secure notes.

Bummer!

I am willing to do the grunt work of manually recreating each of the records that I had tried to export and migrate to 1Password. Doing so will take several weeks and will be a huge pain in the butt-tocks. But is there any way to perform a Category mapping so that when I run the AppleScript, it will recognize and properly match the Category types from mSecure to the proper Category in 1Password? That would be an enormous help to me. I wish there was a utility for this that would ease the migration!

Please note that I was extremely careful to match the spelling of the Categories (aka, Types in mSecure) when I renamed the Types in mSecure so that they would not conflict. Still, the utility did not recognize them.

Separately, I will add suggestions for Category types that I hope AgileBits will consider adding to a future release of 1Password. I have been impressed so far with 1Password; the one enhancement that I hope to see in the future will be the option of either allowing users to add Custom Categories, or to see AgileBits greatly expand the possible Categories available to your customers/subscribers.

Thank you very much in advance for any help you might suggest. If I can successfully automate the migration of my data from mSecure to 1Password, and put the mSecure nightmare behind me, I will be eternally grateful. As it is, I will have to recover the password to my Social Security online account that I lost, thanks to mSecure. But I digress...

B3ndy

Hey everyone,

Firstly, wow, what a fantastic and active forum Agilbits has. Its great to see so many people commenting and getting involved, a sure sign of a good product.

Secondly, a cry for help with this keychain to 1P export / import process, its driving me crazy.

My set up is this, we are a Mac House, everyone in our company uses Apple products, including iCloud Keychain. However I want to bring in stronger password protection for our shared accounts, so I am looking at setting up 1P for all of us. The short is, I need to get this working if we are going to move, and I really want to move!

I've been following this thread with interest and have been playing with creating, copying and running apple scrips from the excellent @MrC (I am sure he was a DJ in 90s British group the Shamen). However I am running into one problem, every, damn, time. When I go to copy the keychain entries from my iCloud keychain to my local newly created one, you got it:

An error has occurred. Unable to add an item to the current keychain.
The contents of this item cannot be retrieved.

Super frustrating.

I've read that it could be an issue with keychain sync, so I've disabled it a number of times, no joy. I've even tried a brand new Mac, still nothing.

For what its worth my machine is running High Sierra (release candidate).

Has anyone else figured this out, or any other way of getting passwords out of iCloud?

Thanks for reading my post.

MrC

@B3ndy ,

Nope, no DJ for Shamen. Can't say I even know a single track by them.

Anyway, sorry for the troubles.

This error you receive typically means there is a sync problem between iCloud and the local store. The advice that works most often is to turn off iCloud syncing on the device, re-enable it, and resync. Then try your copy/paste again. I've found some items just will NOT copy. Sometimes I've had to disable sync on ALL the other devices - the working theory is that one or more devices is out of sync, and the local and remote records cannot be sync'd because of this. The idea is that you remove all but one copy.

The biggest challenge I've faced with all the converters is this iCloud Keychain copy issue, by far. There is just NO reliable way for me to reproduce, diagnose, or resolve the issues users have face. The Keychain is essentially a black box, and iCloud and its errors further confound attempts at wrestling it into submission. So I'm left with giving weak, trial-and-error advice, and I hate giving this kind of "advice" because its more like voodoo and black magic.

The "An error has occurred" has to be one of the lamest, most lazy error messages possible, and the lack of any diagnostic to help resolve the issue is pathetic. Shame on the developers for this.

Wyndham

Thank you for the great script. Unfortunately it failed when I ran it.

Do let me know if I can help.

MrC

@Wyndham ,

See the last note in the first post of this thread. You are likely using version 5.26 of Perl. Use 5.24.

Rob_

FWIW - I got the "Can't locate Utils" error with perl 5.26 and it just needed a tiny change to "convert_to_1p4.pl"

Add this line to use local libs

use lib './';

use Utils::PIF;
use Utils::Utils;
use Getopt::Long;
use File::Basename;

MrC

@Rob_ ,

The current directory was removed from Perl's default search path in 5.26 and there are a couple of ways to resolve the issue. I've just not had to the time to do and test it. A fix is coming soon.

Thanks for the info.

echopecker

1PW 6.8.2
OS X 10.13

MrC, I'm running into an AppleScript problem with conversion of my Mac keychain into a pm_export.txt file. I'm getting the same problem using the two different Scripts, so if I can solve one I think I can solve the other. The first is from the Script that allows you not to have to hit the Allow button each time when writing to the pm_export.txt file. Here is a screenshot of my error and script:

Now I only have about 200 entries here, so I can just click through, but my real problem is the next one. The Script for the dialog box to allow me to skip entering the PW for the iCloud chain gives this error:

I even tried the script in the Extra Help and got the same error. Any help is appreciated, as I have almost 900 entries for the iCloud chain.

Thanks,

Echo