MrC's Convert to 1Password Utility (mrc-converter-suite)

drtome · December 2014

Hi MrC,
Thank you for the fast support. That worked great.
Cheers,
Thomas

MrC · December 2014

That's great news. You're very welcome, and Happy New Year.

falconeye · December 2014

After upgrading to Yosemite, I decided to dump eWallet and give 1Password a try.

This has been made possible by the great work of MrC and his script. No way I could have reentered all my data manually...

Therefore, big KUDOS to MrC's outstanding work! Also, I like 1Password a lot so far.

However, while I appreciate to have found the link to MrC script on the entry HowTo section of Agilebits website, I do NOT appreciate to see no sign whatsoever about any compensation received by MrC for his work.

I herewith STRONGLY encourage to donate part of the revenue of my purchase to MrC. He more than deserves it.

MrC · December 2014

Hi @falconeye,

Thank you so much for you kind remarks and feedback. It's really great to hear that you've converted and imported successfully.

No compensation is necessary; I'm happy to help users like yourself enjoy 1Password as much as I have. AgileBits' Chief Eliminator of Obstacles Jeff Shiner and I had a nice conversation last July, so all is good.

Happy New Year!

falconeye · December 2014

Ok, after the above general comment, let me share my experience with the conversion script and a few workarounds ...

The documentation mentions three different toplevel folders under Desktop (Mac). It is no big issue as the precise folder only matters to those not understanding a file path. But it is inconsistent nonetheless.
I edited the ewallet export file to only include a single ewallet folder. Thereby, I noticed that the script would import zero cards if file isn't ending in two empty lines. Took me a while to debug ...
The script writes tag entries for each leaf-level folder name in ewallet. 1Password imports the tags but no folders. But within 1Password, all entries for a given tag can easily be moved into a manually created folder hierarchy. And tags can then be deleted within 1Password if desired. Using these steps, an existing ewallet fokder structure can be migrated to 1Password.
The HowTo though lacks a description of the workflow described in #3 and made me edit the export file into partial exports files per folder and debug them. A step new 1Password users could avoid with a better HowTo section.
Attachments are not migrated (eWallet ignores them when generating the txt file). Nothing the script could do about, but should be mentioned somewhere ...

Overall though, a great migration tool which made the purchase of 1Password possible for me.

falconeye · December 2014

One last comment ...

With upgrading to Yosemite, eWallet started to crash for me (and the "fix" described by iliumsoft is only temporary). Moreover, eWallet seems like stalled in comparison to 1Password, lacking iCloud support on Mac or Safari or TouchID plugins.

After migration, I can recommend an upgrade from eWallet to 1Password to most eWallet users. Thanks for such a great product.

MrC · December 2014

Comments are welcome.

The documentation mentions three different toplevel folders under Desktop (Mac)...

Which folders are you referring to?

I edited the ewallet export file to only include a single ewallet folder...

I suppose you're in somewhat of a no-man's land when manually editing an export. This seems rare enough that I'll leave this one as documented here in the thread.

The script writes tag entries for each leaf-level folder name in ewallet. 1Password imports the tags but no folders...

The creation of Folders is on my To Do list. Its a larger project that requires more attention than I've been able to give during the Holidays, and with some other large projects on my plate. Folder import will only support (if possible at all) vaults with no existing folders.

The HowTo though lacks a description of the workflow described in #3...

True. I described what is done with folders, tags, etc. in the README:

Groups, Tags, and Other Indicators

Some password management programs allow placement of items into groups or folders (flat or hierarchical). When available in the export, these will be recreated as colon-separated lists of 1Password Tags (note: these are not currently shown in the main UI of the Windows version of 1Password, however, the Tags do exist within the database file, and can be seen in the Tags section when editing an entry).

Likewise, any tags, favorite or star markers, when available for an entry, will become 1Password tags.

Finally, other specific indicators may be placed into the 1Password entries notes section (e.g. Color: Red).

but don't really describe any workflow for a user to migrate tagged items into folder items. If I get folder creation to work, this will be moot. Either way, I can mention how to migrate tags --> folders.

Attachments are not migrated ... but should be mentioned somewhere ...

Agreed. I'll add it to the README. There's only one indirect indication that attachments are not supported (in the SplashID section).

Thanks again!

falconeye · December 2014

@MrC, thanks for the comment that no compensation is necessary.
I see. But I would still like to see Agilebits donate back to you. After all, your wirk is generating a lot of revenue for them. Because many users out there already have a password manager locking them in w/o your enthusiasm.

falconeye · December 2014

Which folders are you referring to?

I am away from my Mac. From my memory:

onepassword-utilities
onepassword-utilities-master
convert_to_1_p4 folder must be placed on your Desktop (from the website's HowTo)

MrC · December 2014

Text Added:

3. Export Your Data

See your password manager's specific section under Exporting from your Password Manager below, and proceed to Step 4 when the data is exported. Note that some password managers support attachments and may or may not support exporting these attachments. Regardless, convert_to_1p4 does not support attachment conversion.

Text Added:

Groups, Tags, and Other Indicators

...

>
>

_To place a set of tagged 1Password entries into a folder, select a given tag and create an appropriately named folder for that tag. With the Folder list expanded, and the tagged items selected, drag the items onto the folder name.

_

From your earlier post:

With upgrading to Yosemite, eWallet started to crash for me (and the "fix" described by iliumsoft is only temporary). Moreover, eWallet seems like stalled in comparison to 1Password, lacking iCloud support on Mac or Safari or TouchID plugins.

Yup, eWallet crashes in Yosemite. The desktop versions are old programs, from back in the days of Palm Pilot. I can't imagine it would be cost-effective to rewrite them today, so they are stuck in the past and linger.

jonathanmorgan · December 2014

Hello MrC!

The script is working great. Turning folders into tags rather than having them in notes works like a charm (well, except for one thing).

I am not sure if you remember, but we worked through some issues with login items that did not have a username, and I tested out the 1pif work you were doing a while ago. Just migrated my wife from keepass to 1Password using the latest script (well, imported keepassx 0.4.3 *.kdb file into latest keepass for Windows - 2.28 - then exported as keepass XML v.2.0, then ran your script on that with type of "keepass2"), and it worked like a charm except for one thing: Ampersands in folder names in the keepass XML cause a name parsing problem, such that the first time you encounter an ampersand in a name, that folder name is converted to a tag like this: <part_before_ampersand> ::&:: <part_after_ampersand>. Then, each subsequent folder, even if it is not a child folder, is appended to the end of the "first part plus ampersand" part of that tag name value, so something like: <part_before_ampersand> ::&:: <folder_name_including_children_,_properly_delimited>. This also compounds each time you get an additional ampersand. So, say you have the following folders:

Design & Work
Important Stuff
More Stuff
Pinterest
Banking & Finance
Work
- work 1
- work 2

The tags I got were:

Design ::&:: Work
Design ::&::Important Stuff
Design ::&::More Stuff
Design ::&::Pinterest
Design ::&::Banking ::&:: Finance
Design ::&::Banking ::&::Work
Design ::&::Banking ::&::Work::work 1
Design ::&::Banking ::&::Work::work 2

So, the tags are preserved, but you have to understand what is going on to find and remove the stuff from the beginning and clean up the names of those folders/tags that have ampersands in them.

I did a little digging into the XML, and it looks like the problem might be related to how the text inside each <Name> element, nested inside each <Group> element, is escaped, and so related to how you or your XML parser are parsing the <Group><Name>. In the XML, ampersands are entities - & (since they don't put them in a CDATA wrapper). For example:

<Name>Design &amp; Work</Name>

So, you might check to see what a group name with an ampersand entity resolves to in your script, and that might tell you what is going on with the rest of this.

Let me know if there is anything I can do to help troubleshoot, or if you have trouble reproducing. If it would help, I could send you an anonymized sample keypass XML file, so you can play with it yourself.

Thanks,

Jonathan Morgan

jonathanmorgan · December 2014

Also, you might consider adding the actual values for the different types of converters to the README, either in a list in step 4 where you mention that you have to have the right converter, or in the list at the top that you refer to in step 4 that includes all the converters, but not the strings you'd use in the command line. It is good that running perl convert_to_1p4.pl --help does list out the actual string converter values that are valid, but it would be nice if they were also in the README since they are actually different values from the values that are in the list at the top to which you refer (in case and in spaces being excluded, at the least).

Again, this is a great script, the documentation is good, and this was the only suggestion I have for the doc.

MrC · December 2014

@jonathanmorgan‌,

Good to talk with you again. I recall our past email conversation well. Thanks so much for bringing the ampersand bug to my attention. It probably would have been triggered by a few more reserved XML characters too.

I've updated the converter to correct this bug; version 1.02 is in the _testing directory of the AgileBits Utilities folder (link in post #1). In addition, I've rewritten Step 4 in the README to accommodate your feedback above.

Perhaps you can give the new converter a test, and see if the README adequately addresses your suggestion.

MrC · December 2014

@falconeye,

The documentation mentions three different toplevel folders under Desktop (Mac). It is no big issue as the precise folder only matters to those not understanding a file path. But it is inconsistent nonetheless.

...

I am away from my Mac. From my memory:

>

onepassword-utilities
onepassword-utilities-master
convert_to_1_p4 folder must be placed on your Desktop (from the website's HowTo)

>

I see what you mean now. The main landing page on AgileBit's site is both inconsistent, and has changed with respect to the name of the onepassword-utilities folder and the zip archive name. I'll notify someone that the page needs to be updated. Thanks!

MrC · January 2015

Some good news. A new feature is coming. Folders.

The creation of Folders is on my To Do list. Its a larger project that requires more attention than I've been able to give during the Holidays, and with some other large projects on my plate. Folder import will only support (if possible at all) vaults with no existing folders.

I've written the code to convert a password manager's Folders / Groups to 1Password folders. It will be optional, off by default. This is because there is a limitation, as suggested above - the converter is unaware of existing folders in your vault. If you use this option, any existing Folder names are ignored, and the converter will create new Folders, possibly with names identical to those already in your vault. In addition, re-running the converter and re-importing will duplicate the Folder names as well, since new unique identifiers are created each time the converter is run. Obviously the best results will be on a fresh vault.

I'll post an update in the _testing area soon, and will update this post with an edit that the code has been posted.

Edit: Jan 4: I've posted the 1.03 version which includes folder support to the _testing area. See the Changes and README files, and the --folders option. I've tested this for the supported converters on OS X, but not Windows yet.

Edit: Feb 18: the 1.03 version has been promoted as stable and is posted.

rkempton · January 2015

THANK YOU for the help, MrC.....I am only slightly technical - here is what my terminal session says...can you tell me how to respond? (thank you)

Last login: Sun Jan 4 13:07:12 on ttys000
Defaults-iMac-2:~ User$ cd Desktop/convert_to_1p4
Defaults-iMac-2:convert_to_1p4 User$ cpan Text::CSV

CPAN.pm requires configuration, but most of it can be done automatically.
If you answer 'no' below, you will enter an interactive dialog for each
configuration option instead.

Would you like to configure as much as possible automatically? [yes]

MrC · January 2015

Hi @rkempton‌,

On OS X, you don't need CPAN - that's only for Windows. OS X comes with all the necessary software. Just be sure to follow the OS X portion of the README, and ignore the Windows instructions.

Anthony Main · January 2015

FYI Everyone - the folders feature is looking good on Windows from v1.03, but it only supports a single level of folders, not subfolders

MrC · January 2015

Thanks for the Folder feedback.

Apparently sub-folder importing is not yet supported in 1Password for Windows. The folder data is present in the 1PIF; however, the full folder hierarchy is not presented. Only the final folder is shown.

I have pinged the Windows folks and am awaiting a reply...

Edit: the bug has been fixed in the upcoming version 4.1.0.BETA-535.

nicky · January 2015

I somehow get an error "installing" with the following command cpan Date::Calc, the other commands get a --OK but this one doesn't.
When I run the command I get this result, can someone please help me with this, because I don't understand what I'm doing wrong:

c:\myperl>cpan Date::Calc
CPAN: CPAN::SQLite loaded ok (v0.202)
Database was generated on Mon, 12 Jan 2015 14:24:37 GMT
Running install for module 'Date::Calc'
Running make for S/ST/STBEY/Date-Calc-6.3.tar.gz
CPAN: Digest::SHA loaded ok (v5.84)
CPAN: Compress::Zlib loaded ok (v2.06)
Checksum for C:\myperl\cpan\sources\authors\id\S\ST\STBEY\Date-Calc-6.3.tar.gz o
k
CPAN: Archive::Tar loaded ok (v1.90)
CPAN: File::Temp loaded ok (v0.22)
CPAN: CPAN::Meta loaded ok (v2.120921)

  CPAN.pm: Building S/ST/STBEY/Date-Calc-6.3.tar.gz


*************************************************************
****** BEWARE: Use "make install UNINST=1" to install! ******
*************************************************************

Checking if your kit is complete...
Looks good
Writing Makefile for Date::Calc
Writing MYMETA.yml and MYMETA.json
CPAN: Module::CoreList loaded ok (v2.83)
cp lib/Date/Calendar.pm blib\lib\Date\Calendar.pm
cp lib/Date/Calc/Object.pod blib\lib\Date\Calc\Object.pod
cp lib/Date/Calendar/Year.pm blib\lib\Date\Calendar\Year.pm
cp lib/Date/Calc/PP.pod blib\lib\Date\Calc\PP.pod
cp lib/Date/Calendar/Profiles.pm blib\lib\Date\Calendar\Profiles.pm
cp lib/Date/Calendar.pod blib\lib\Date\Calendar.pod
cp lib/Date/Calc/PP.pm blib\lib\Date\Calc\PP.pm
cp lib/Date/Calc.pod blib\lib\Date\Calc.pod
cp lib/Date/Calc.pm blib\lib\Date\Calc.pm
cp lib/Date/Calc/Object.pm blib\lib\Date\Calc\Object.pm
cp lib/Date/Calendar/Profiles.pod blib\lib\Date\Calendar\Profiles.pod
cp lib/Date/Calendar/Year.pod blib\lib\Date\Calendar\Year.pod
  STBEY/Date-Calc-6.3.tar.gz
  C:\myperl\c\bin\dmake.exe -- OK
CPAN: YAML::XS loaded ok (v0.39)
Running make test
C:\myperl\perl\bin\perl.exe "-MExtUtils::Command::MM" "-e" "test_harness(0, 'bli
b\lib', 'blib\arch')" t/*.t
t/f000.t .. ok
t/f001.t .. ok
t/f002.t .. ok
t/f003.t .. ok
t/f004.t .. ok
t/f005.t .. ok
t/f006.t .. ok
t/f007.t .. ok
t/f008.t .. ok
t/f009.t .. ok
t/f010.t .. ok
t/f011.t .. ok
t/f012.t .. ok
t/f013.t .. ok
t/f014.t .. ok
t/f015.t .. ok
t/f016.t .. Failed 16/25 subtests
t/f017.t .. ok
t/f018.t .. ok
t/f019.t .. ok
t/f020.t .. ok
t/f021.t .. ok
t/f022.t .. ok
t/f023.t .. ok
t/f024.t .. ok
t/f025.t .. ok
t/f026.t .. ok
t/f027.t .. Failed 22/46 subtests
t/f028.t .. Failed 22/46 subtests
t/f029.t .. ok
t/f030.t .. ok
t/f031.t .. ok
t/f032.t .. ok
t/f033.t .. ok
t/f034.t .. ok
t/f035.t .. ok
t/f036.t .. ok
t/f037.t .. ok
t/m001.t .. ok
t/m002.t .. ok
t/m003.t .. ok
t/m004.t .. ok
t/m005.t .. ok
t/m006.t .. ok
t/m007.t .. ok
t/m008.t .. ok
t/m009.t .. ok
t/m010.t .. ok
t/m011.t .. ok
t/m012.t .. ok
t/m013.t .. ok

Test Summary Report
-------------------
t/f016.t (Wstat: 0 Tests: 25 Failed: 16)
  Failed tests:  1-4, 6-7, 9-12, 15-17, 21-23
t/f027.t (Wstat: 0 Tests: 46 Failed: 22)
  Failed tests:  7-15, 22, 24-27, 30-35, 44-45
t/f028.t (Wstat: 0 Tests: 46 Failed: 22)
  Failed tests:  7-15, 22, 24-27, 30, 32, 34-37, 44-45
Files=51, Tests=3381,  4 wallclock secs ( 0.33 usr +  0.14 sys =  0.47 CPU)
Result: FAIL
Failed 3/51 test programs. 60/3381 subtests failed.
dmake.exe:  Error code 255, while making 'test_dynamic'
  STBEY/Date-Calc-6.3.tar.gz
  C:\myperl\c\bin\dmake.exe test -- NOT OK
//hint// to see the cpan-testers results for installing this module, try:
  reports STBEY/Date-Calc-6.3.tar.gz
Running make install
  make test had returned bad status, won't install without force
Stopping: 'install' failed for 'Date::Calc'.

I'm running Windows 8.1 x64 with "strawberry-perl-5.16.3.1-64bit-portable.zip" extracted to c:\myperl\

MrC · January 2015

@nicky‌,

I'm sorry for your troubles. This issue is a simple bug in the automatic tests that are performed when installing modules. This bug was triggered when the date crossed into 2015, but can be safely ignored. In step 2d, instead of running cpan Date::Calc, run the command cpan -fT Date::Calc. This will bypass the automatic testing, but still build and install the module.

I will update the README. I was hoping the Date::Calc module maintainer might have been able to update the module soon to allow the tests to pass, and then everything would be back to normal.

Edit 2015-03-12: The issue with the Date::Calc module has been resolved and the installation as described in the README file now works correctly again.

nicky · January 2015

@MrC,

Thanks, this did the trick and password import is now complete, this saves me a great amount of time.

mikebeaumont · January 2015

Hi I am trying to convert my secure v3.5.4 data to 1 password 5 format following the instructions in your readme file. When i export from msecure i am unable to save a txt file and it only lets me save the file as pm_export.txt.csv. When i paste this into terminal as per your guidelines i get the following

Last login: Tue Jan 13 17:28:51 on ttys000
Michaels-MacBook-Air-2:~ mikejbeaumont$ cd Desktop/convert_to_1p4
Michaels-MacBook-Air-2:convert_to_1p4 mikejbeaumont$ perl5.16 convert_to_1p4.pl msecure -v ../../pm_export.txt.csv
Unable to open CSV file: ../../pm_export.txt.csv
No such file or directory

If i remove the .csv extension i get the same message"
Michaels-MacBook-Air-2:convert_to_1p4 mikejbeaumont$ perl5.16 convert_to_1p4.pl msecure -v ../../pm_export.txt
Unable to open CSV file: ../../pm_export.txt
No such file or directory

I have both the 1p4 converter and the pm_export file on the desktop but I'm not at all familiar with terminal so not really sure what is going on.

Appreciate any help you might be able to give me

Thanks

MrC · January 2015

Hi @mikebeaumont,

The filename pm_export.txt.csv is fine. Just use that name when you run the commands instead of pm_export.txt. The name of the file does not matter - I just selected one so that it would be consistent and familiar throughout the README and its commands.

Do you have "the 1p4 converter" (i.e. convert_to_1p4.pl) on the Desktop, or the folder named convert_to_1p4? You want the folder named convert_to_1p4, with all of its contents intact, on the Desktop.

I see in the README file, I left an extra folder level in the conversion command. I've corrected that in the _testing version. The command you want to run is:

perl5.16 convert_to_1p4.pl msecure -v ../pm_export.txt.csv

If you want Folder support, use the 1.03 version in _testing, and add the --folders option anywhere after the converter name on the command line. For example:

perl5.16 convert_to_1p4.pl msecure -v --folders ../pm_export.txt.csv

Edit: the 1.03 version has been promoted as stable and is posted.

MrC · January 2015

@mikebeaumont and others,

I've created a video showing the entire conversion process, from download of the utilities, to the final import and deletion of the export and conversion files. The example password manager selected for the video was mSecure, but the procedure is basically the same for the other password managers.

In my AgileBits Utiities folder, there is an Extra Help folder. Download and watch the video named Running convert_to_1p4 on OS X.

DarwinsTale · January 2015

This SUCKS! I want to push a button and import all 473 passwords in keychain... NOT read 473 posts on how I need to program a damn command prompt.

I bought 1PW so that I could USE your program... not write programs.

There must be an easier way.

MrC · January 2015

Hi @DarwinsTale,

I don't work for AgileBits, and they are not responsible for the tool I created to help users convert.

I don't know that Apple provides a way for a program such as 1Password to import the entire Keychain database.

Megan · January 2015

Hi @DarwinsTale,

I sincerely apologize for the frustration here. Unfortunately, each password program handles the export of data slightly differently, so it's no simple feat to create an import tool that covers all the various password programs. We're really grateful that @MrC has taken the time to write these converters to make it more easy for our users to import their data into 1Password.

kf0s · January 2015

Hello. First of all, Great tool! I tried to convert my SplashID file in the past and wasn't totally happy with the results. Using this converter I am very happy with the results.

I will ask the same question in the #23-#24 posts. Would it be difficult to put the Date Modified from SplashID (which you are already putting into the Notes field) into the Last Modified field in 1Password. If it's difficult, don't worry about it, but I do (since I have things across multiple devices) sort on the modified field. It would be create if I would see the same order in 1Password.

I did try and look at some of the code to see if I could see what I could maybe change, but a change didn't jump right out.

Thanks again for your tool! It is much appreciated!

Eric

MrC · January 2015

Hi @kf0s‌,

Great to hear - this type of feedback and positive results always makes for a nice way to end a day!

Regarding the Modified Date, it can be placed into the 1PIF export. I don't know how that might affect WatchTower, however, but I'm willing to parse the date field and make it an option for you.

Which platform did you run the conversion on? Windows or Mac?