MrC's Convert to 1Password Utility (mrc-converter-suite)

1235752

Comments

  • drtome
    drtome
    Community Member

    Hi MrC,
    Thank you for the fast support. That worked great.
    Cheers,
    Thomas

  • MrC
    MrC
    Volunteer Moderator

    Hi @drtome,

    That's great news. You're very welcome, and Happy New Year.

  • falconeye
    falconeye
    Community Member

    After upgrading to Yosemite, I decided to dump eWallet and give 1Password a try.

    This has been made possible by the great work of MrC and his script. No way I could have reentered all my data manually...

    Therefore, big KUDOS to MrC's outstanding work! Also, I like 1Password a lot so far.

    However, while I appreciate to have found the link to MrC script on the entry HowTo section of Agilebits website, I do NOT appreciate to see no sign whatsoever about any compensation received by MrC for his work.

    I herewith STRONGLY encourage to donate part of the revenue of my purchase to MrC. He more than deserves it.

  • MrC
    MrC
    Volunteer Moderator
    edited December 2014

    Hi @falconeye,

    Thank you so much for you kind remarks and feedback. It's really great to hear that you've converted and imported successfully.

    No compensation is necessary; I'm happy to help users like yourself enjoy 1Password as much as I have. AgileBits' Chief Eliminator of Obstacles Jeff Shiner and I had a nice conversation last July, so all is good.

    Happy New Year!

  • falconeye
    falconeye
    Community Member

    Ok, after the above general comment, let me share my experience with the conversion script and a few workarounds ...

    1. The documentation mentions three different toplevel folders under Desktop (Mac). It is no big issue as the precise folder only matters to those not understanding a file path. But it is inconsistent nonetheless.
    2. I edited the ewallet export file to only include a single ewallet folder. Thereby, I noticed that the script would import zero cards if file isn't ending in two empty lines. Took me a while to debug ...
    3. The script writes tag entries for each leaf-level folder name in ewallet. 1Password imports the tags but no folders. But within 1Password, all entries for a given tag can easily be moved into a manually created folder hierarchy. And tags can then be deleted within 1Password if desired. Using these steps, an existing ewallet fokder structure can be migrated to 1Password.
    4. The HowTo though lacks a description of the workflow described in #3 and made me edit the export file into partial exports files per folder and debug them. A step new 1Password users could avoid with a better HowTo section.
    5. Attachments are not migrated (eWallet ignores them when generating the txt file). Nothing the script could do about, but should be mentioned somewhere ...

    Overall though, a great migration tool which made the purchase of 1Password possible for me.

  • falconeye
    falconeye
    Community Member
    edited December 2014

    One last comment ...

    With upgrading to Yosemite, eWallet started to crash for me (and the "fix" described by iliumsoft is only temporary). Moreover, eWallet seems like stalled in comparison to 1Password, lacking iCloud support on Mac or Safari or TouchID plugins.

    After migration, I can recommend an upgrade from eWallet to 1Password to most eWallet users. Thanks for such a great product.

  • MrC
    MrC
    Volunteer Moderator

    Comments are welcome.

    1. The documentation mentions three different toplevel folders under Desktop (Mac)...

    Which folders are you referring to?

    1. I edited the ewallet export file to only include a single ewallet folder...

    I suppose you're in somewhat of a no-man's land when manually editing an export. This seems rare enough that I'll leave this one as documented here in the thread.

    1. The script writes tag entries for each leaf-level folder name in ewallet. 1Password imports the tags but no folders...

    The creation of Folders is on my To Do list. Its a larger project that requires more attention than I've been able to give during the Holidays, and with some other large projects on my plate. Folder import will only support (if possible at all) vaults with no existing folders.

    1. The HowTo though lacks a description of the workflow described in #3...

    True. I described what is done with folders, tags, etc. in the README:

    Groups, Tags, and Other Indicators

    Some password management programs allow placement of items into groups or folders (flat or hierarchical). When available in the export, these will be recreated as colon-separated lists of 1Password Tags (note: these are not currently shown in the main UI of the Windows version of 1Password, however, the Tags do exist within the database file, and can be seen in the Tags section when editing an entry).

    Likewise, any tags, favorite or star markers, when available for an entry, will become 1Password tags.

    Finally, other specific indicators may be placed into the 1Password entries notes section (e.g. Color: Red).

    but don't really describe any workflow for a user to migrate tagged items into folder items. If I get folder creation to work, this will be moot. Either way, I can mention how to migrate tags --> folders.

    1. Attachments are not migrated ... but should be mentioned somewhere ...

    Agreed. I'll add it to the README. There's only one indirect indication that attachments are not supported (in the SplashID section).

    Thanks again!

  • falconeye
    falconeye
    Community Member

    @MrC, thanks for the comment that no compensation is necessary.
    I see. But I would still like to see Agilebits donate back to you. After all, your wirk is generating a lot of revenue for them. Because many users out there already have a password manager locking them in w/o your enthusiasm.

  • falconeye
    falconeye
    Community Member
    edited December 2014

    Which folders are you referring to?

    I am away from my Mac. From my memory:

    • onepassword-utilities
    • onepassword-utilities-master
    • convert_to_1_p4 folder must be placed on your Desktop (from the website's HowTo)
  • MrC
    MrC
    Volunteer Moderator
    edited December 2014

    Text Added:

    3. Export Your Data

    See your password manager's specific section under Exporting from your Password Manager below, and proceed to Step 4 when the data is exported. Note that some password managers support attachments and may or may not support exporting these attachments. Regardless, convert_to_1p4 does not support attachment conversion.

    Text Added:

    Groups, Tags, and Other Indicators

    ...

    >
    >

    _To place a set of tagged 1Password entries into a folder, select a given tag and create an appropriately named folder for that tag. With the Folder list expanded, and the tagged items selected, drag the items onto the folder name.

    _

    From your earlier post:

    With upgrading to Yosemite, eWallet started to crash for me (and the "fix" described by iliumsoft is only temporary). Moreover, eWallet seems like stalled in comparison to 1Password, lacking iCloud support on Mac or Safari or TouchID plugins.

    Yup, eWallet crashes in Yosemite. The desktop versions are old programs, from back in the days of Palm Pilot. I can't imagine it would be cost-effective to rewrite them today, so they are stuck in the past and linger.

  • jonathanmorgan
    jonathanmorgan
    Community Member

    Hello MrC!

    The script is working great. Turning folders into tags rather than having them in notes works like a charm (well, except for one thing).

    I am not sure if you remember, but we worked through some issues with login items that did not have a username, and I tested out the 1pif work you were doing a while ago. Just migrated my wife from keepass to 1Password using the latest script (well, imported keepassx 0.4.3 *.kdb file into latest keepass for Windows - 2.28 - then exported as keepass XML v.2.0, then ran your script on that with type of "keepass2"), and it worked like a charm except for one thing: Ampersands in folder names in the keepass XML cause a name parsing problem, such that the first time you encounter an ampersand in a name, that folder name is converted to a tag like this: <part_before_ampersand> ::&:: <part_after_ampersand>. Then, each subsequent folder, even if it is not a child folder, is appended to the end of the "first part plus ampersand" part of that tag name value, so something like: <part_before_ampersand> ::&:: <folder_name_including_children_,_properly_delimited>. This also compounds each time you get an additional ampersand. So, say you have the following folders:

    • Design & Work
    • Important Stuff
    • More Stuff
    • Pinterest
    • Banking & Finance
    • Work
      • work 1
      • work 2

    The tags I got were:

    • Design ::&:: Work
    • Design ::&::Important Stuff
    • Design ::&::More Stuff
    • Design ::&::Pinterest
    • Design ::&::Banking ::&:: Finance
    • Design ::&::Banking ::&::Work
    • Design ::&::Banking ::&::Work::work 1
    • Design ::&::Banking ::&::Work::work 2

    So, the tags are preserved, but you have to understand what is going on to find and remove the stuff from the beginning and clean up the names of those folders/tags that have ampersands in them.

    I did a little digging into the XML, and it looks like the problem might be related to how the text inside each <Name> element, nested inside each <Group> element, is escaped, and so related to how you or your XML parser are parsing the <Group><Name>. In the XML, ampersands are entities - &amp; (since they don't put them in a CDATA wrapper). For example:

    <Name>Design &amp; Work</Name>
    

    So, you might check to see what a group name with an ampersand entity resolves to in your script, and that might tell you what is going on with the rest of this.

    Let me know if there is anything I can do to help troubleshoot, or if you have trouble reproducing. If it would help, I could send you an anonymized sample keypass XML file, so you can play with it yourself.

    Thanks,

    Jonathan Morgan

  • jonathanmorgan
    jonathanmorgan
    Community Member

    Also, you might consider adding the actual values for the different types of converters to the README, either in a list in step 4 where you mention that you have to have the right converter, or in the list at the top that you refer to in step 4 that includes all the converters, but not the strings you'd use in the command line. It is good that running perl convert_to_1p4.pl --help does list out the actual string converter values that are valid, but it would be nice if they were also in the README since they are actually different values from the values that are in the list at the top to which you refer (in case and in spaces being excluded, at the least).

    Again, this is a great script, the documentation is good, and this was the only suggestion I have for the doc.

  • MrC
    MrC
    Volunteer Moderator

    @jonathanmorgan‌,

    Good to talk with you again. I recall our past email conversation well. Thanks so much for bringing the ampersand bug to my attention. It probably would have been triggered by a few more reserved XML characters too.

    I've updated the converter to correct this bug; version 1.02 is in the _testing directory of the AgileBits Utilities folder (link in post #1). In addition, I've rewritten Step 4 in the README to accommodate your feedback above.

    Perhaps you can give the new converter a test, and see if the README adequately addresses your suggestion.

  • MrC
    MrC
    Volunteer Moderator

    @falconeye,

    1. The documentation mentions three different toplevel folders under Desktop (Mac). It is no big issue as the precise folder only matters to those not understanding a file path. But it is inconsistent nonetheless.

    ...

    I am away from my Mac. From my memory:

    >

    onepassword-utilities
    onepassword-utilities-master
    convert_to_1_p4 folder must be placed on your Desktop (from the website's HowTo)
    

    >

    I see what you mean now. The main landing page on AgileBit's site is both inconsistent, and has changed with respect to the name of the onepassword-utilities folder and the zip archive name. I'll notify someone that the page needs to be updated. Thanks!

  • MrC
    MrC
    Volunteer Moderator
    edited February 2015

    Some good news. A new feature is coming. Folders.

    The creation of Folders is on my To Do list. Its a larger project that requires more attention than I've been able to give during the Holidays, and with some other large projects on my plate. Folder import will only support (if possible at all) vaults with no existing folders.

    I've written the code to convert a password manager's Folders / Groups to 1Password folders. It will be optional, off by default. This is because there is a limitation, as suggested above - the converter is unaware of existing folders in your vault. If you use this option, any existing Folder names are ignored, and the converter will create new Folders, possibly with names identical to those already in your vault. In addition, re-running the converter and re-importing will duplicate the Folder names as well, since new unique identifiers are created each time the converter is run. Obviously the best results will be on a fresh vault.

    I'll post an update in the _testing area soon, and will update this post with an edit that the code has been posted.

    Edit: Jan 4: I've posted the 1.03 version which includes folder support to the _testing area. See the Changes and README files, and the --folders option. I've tested this for the supported converters on OS X, but not Windows yet.

    Edit: Feb 18: the 1.03 version has been promoted as stable and is posted.

  • rkempton
    rkempton
    Community Member

    THANK YOU for the help, MrC.....I am only slightly technical - here is what my terminal session says...can you tell me how to respond? (thank you)

    • Last login: Sun Jan 4 13:07:12 on ttys000
      Defaults-iMac-2:~ User$ cd Desktop/convert_to_1p4
      Defaults-iMac-2:convert_to_1p4 User$ cpan Text::CSV

    CPAN.pm requires configuration, but most of it can be done automatically.
    If you answer 'no' below, you will enter an interactive dialog for each
    configuration option instead.

    Would you like to configure as much as possible automatically? [yes]

  • MrC
    MrC
    Volunteer Moderator

    Hi @rkempton‌,

    On OS X, you don't need CPAN - that's only for Windows. OS X comes with all the necessary software. Just be sure to follow the OS X portion of the README, and ignore the Windows instructions.

  • Anthony Main
    Anthony Main
    Community Member
    edited January 2015

    FYI Everyone - the folders feature is looking good on Windows from v1.03, but it only supports a single level of folders, not subfolders

  • MrC
    MrC
    Volunteer Moderator
    edited January 2015

    Thanks for the Folder feedback.

    Apparently sub-folder importing is not yet supported in 1Password for Windows. The folder data is present in the 1PIF; however, the full folder hierarchy is not presented. Only the final folder is shown.

    I have pinged the Windows folks and am awaiting a reply...

    Edit: the bug has been fixed in the upcoming version 4.1.0.BETA-535.

  • nicky
    nicky
    Community Member

    I somehow get an error "installing" with the following command cpan Date::Calc, the other commands get a --OK but this one doesn't.
    When I run the command I get this result, can someone please help me with this, because I don't understand what I'm doing wrong:

    c:\myperl>cpan Date::Calc
    CPAN: CPAN::SQLite loaded ok (v0.202)
    Database was generated on Mon, 12 Jan 2015 14:24:37 GMT
    Running install for module 'Date::Calc'
    Running make for S/ST/STBEY/Date-Calc-6.3.tar.gz
    CPAN: Digest::SHA loaded ok (v5.84)
    CPAN: Compress::Zlib loaded ok (v2.06)
    Checksum for C:\myperl\cpan\sources\authors\id\S\ST\STBEY\Date-Calc-6.3.tar.gz o
    k
    CPAN: Archive::Tar loaded ok (v1.90)
    CPAN: File::Temp loaded ok (v0.22)
    CPAN: CPAN::Meta loaded ok (v2.120921)
    
      CPAN.pm: Building S/ST/STBEY/Date-Calc-6.3.tar.gz
    
    
    *************************************************************
    ****** BEWARE: Use "make install UNINST=1" to install! ******
    *************************************************************
    
    Checking if your kit is complete...
    Looks good
    Writing Makefile for Date::Calc
    Writing MYMETA.yml and MYMETA.json
    CPAN: Module::CoreList loaded ok (v2.83)
    cp lib/Date/Calendar.pm blib\lib\Date\Calendar.pm
    cp lib/Date/Calc/Object.pod blib\lib\Date\Calc\Object.pod
    cp lib/Date/Calendar/Year.pm blib\lib\Date\Calendar\Year.pm
    cp lib/Date/Calc/PP.pod blib\lib\Date\Calc\PP.pod
    cp lib/Date/Calendar/Profiles.pm blib\lib\Date\Calendar\Profiles.pm
    cp lib/Date/Calendar.pod blib\lib\Date\Calendar.pod
    cp lib/Date/Calc/PP.pm blib\lib\Date\Calc\PP.pm
    cp lib/Date/Calc.pod blib\lib\Date\Calc.pod
    cp lib/Date/Calc.pm blib\lib\Date\Calc.pm
    cp lib/Date/Calc/Object.pm blib\lib\Date\Calc\Object.pm
    cp lib/Date/Calendar/Profiles.pod blib\lib\Date\Calendar\Profiles.pod
    cp lib/Date/Calendar/Year.pod blib\lib\Date\Calendar\Year.pod
      STBEY/Date-Calc-6.3.tar.gz
      C:\myperl\c\bin\dmake.exe -- OK
    CPAN: YAML::XS loaded ok (v0.39)
    Running make test
    C:\myperl\perl\bin\perl.exe "-MExtUtils::Command::MM" "-e" "test_harness(0, 'bli
    b\lib', 'blib\arch')" t/*.t
    t/f000.t .. ok
    t/f001.t .. ok
    t/f002.t .. ok
    t/f003.t .. ok
    t/f004.t .. ok
    t/f005.t .. ok
    t/f006.t .. ok
    t/f007.t .. ok
    t/f008.t .. ok
    t/f009.t .. ok
    t/f010.t .. ok
    t/f011.t .. ok
    t/f012.t .. ok
    t/f013.t .. ok
    t/f014.t .. ok
    t/f015.t .. ok
    t/f016.t .. Failed 16/25 subtests
    t/f017.t .. ok
    t/f018.t .. ok
    t/f019.t .. ok
    t/f020.t .. ok
    t/f021.t .. ok
    t/f022.t .. ok
    t/f023.t .. ok
    t/f024.t .. ok
    t/f025.t .. ok
    t/f026.t .. ok
    t/f027.t .. Failed 22/46 subtests
    t/f028.t .. Failed 22/46 subtests
    t/f029.t .. ok
    t/f030.t .. ok
    t/f031.t .. ok
    t/f032.t .. ok
    t/f033.t .. ok
    t/f034.t .. ok
    t/f035.t .. ok
    t/f036.t .. ok
    t/f037.t .. ok
    t/m001.t .. ok
    t/m002.t .. ok
    t/m003.t .. ok
    t/m004.t .. ok
    t/m005.t .. ok
    t/m006.t .. ok
    t/m007.t .. ok
    t/m008.t .. ok
    t/m009.t .. ok
    t/m010.t .. ok
    t/m011.t .. ok
    t/m012.t .. ok
    t/m013.t .. ok
    
    Test Summary Report
    -------------------
    t/f016.t (Wstat: 0 Tests: 25 Failed: 16)
      Failed tests:  1-4, 6-7, 9-12, 15-17, 21-23
    t/f027.t (Wstat: 0 Tests: 46 Failed: 22)
      Failed tests:  7-15, 22, 24-27, 30-35, 44-45
    t/f028.t (Wstat: 0 Tests: 46 Failed: 22)
      Failed tests:  7-15, 22, 24-27, 30, 32, 34-37, 44-45
    Files=51, Tests=3381,  4 wallclock secs ( 0.33 usr +  0.14 sys =  0.47 CPU)
    Result: FAIL
    Failed 3/51 test programs. 60/3381 subtests failed.
    dmake.exe:  Error code 255, while making 'test_dynamic'
      STBEY/Date-Calc-6.3.tar.gz
      C:\myperl\c\bin\dmake.exe test -- NOT OK
    //hint// to see the cpan-testers results for installing this module, try:
      reports STBEY/Date-Calc-6.3.tar.gz
    Running make install
      make test had returned bad status, won't install without force
    Stopping: 'install' failed for 'Date::Calc'.
    

    I'm running Windows 8.1 x64 with "strawberry-perl-5.16.3.1-64bit-portable.zip" extracted to c:\myperl\

  • MrC
    MrC
    Volunteer Moderator
    edited March 2015

    @nicky‌,

    I'm sorry for your troubles. This issue is a simple bug in the automatic tests that are performed when installing modules. This bug was triggered when the date crossed into 2015, but can be safely ignored. In step 2d, instead of running cpan Date::Calc, run the command cpan -fT Date::Calc. This will bypass the automatic testing, but still build and install the module.

    I will update the README. I was hoping the Date::Calc module maintainer might have been able to update the module soon to allow the tests to pass, and then everything would be back to normal.

    Edit 2015-03-12: The issue with the Date::Calc module has been resolved and the installation as described in the README file now works correctly again.

  • nicky
    nicky
    Community Member

    @MrC,

    Thanks, this did the trick and password import is now complete, this saves me a great amount of time.

  • mikebeaumont
    mikebeaumont
    Community Member

    Hi I am trying to convert my secure v3.5.4 data to 1 password 5 format following the instructions in your readme file. When i export from msecure i am unable to save a txt file and it only lets me save the file as pm_export.txt.csv. When i paste this into terminal as per your guidelines i get the following

    Last login: Tue Jan 13 17:28:51 on ttys000
    Michaels-MacBook-Air-2:~ mikejbeaumont$ cd Desktop/convert_to_1p4
    Michaels-MacBook-Air-2:convert_to_1p4 mikejbeaumont$ perl5.16 convert_to_1p4.pl msecure -v ../../pm_export.txt.csv
    Unable to open CSV file: ../../pm_export.txt.csv
    No such file or directory

    If i remove the .csv extension i get the same message"
    Michaels-MacBook-Air-2:convert_to_1p4 mikejbeaumont$ perl5.16 convert_to_1p4.pl msecure -v ../../pm_export.txt
    Unable to open CSV file: ../../pm_export.txt
    No such file or directory

    I have both the 1p4 converter and the pm_export file on the desktop but I'm not at all familiar with terminal so not really sure what is going on.

    Appreciate any help you might be able to give me

    Thanks

  • MrC
    MrC
    Volunteer Moderator
    edited February 2015

    Hi @mikebeaumont,

    The filename pm_export.txt.csv is fine. Just use that name when you run the commands instead of pm_export.txt. The name of the file does not matter - I just selected one so that it would be consistent and familiar throughout the README and its commands.

    Do you have "the 1p4 converter" (i.e. convert_to_1p4.pl) on the Desktop, or the folder named convert_to_1p4? You want the folder named convert_to_1p4, with all of its contents intact, on the Desktop.

    I see in the README file, I left an extra folder level in the conversion command. I've corrected that in the _testing version. The command you want to run is:

    perl5.16 convert_to_1p4.pl msecure -v ../pm_export.txt.csv
    

    If you want Folder support, use the 1.03 version in _testing, and add the --folders option anywhere after the converter name on the command line. For example:

    perl5.16 convert_to_1p4.pl msecure -v --folders ../pm_export.txt.csv
    

    Edit: the 1.03 version has been promoted as stable and is posted.

  • MrC
    MrC
    Volunteer Moderator
    edited January 2015

    @mikebeaumont and others,

    I've created a video showing the entire conversion process, from download of the utilities, to the final import and deletion of the export and conversion files. The example password manager selected for the video was mSecure, but the procedure is basically the same for the other password managers.

    In my AgileBits Utiities folder, there is an Extra Help folder. Download and watch the video named Running convert_to_1p4 on OS X.

  • DarwinsTale
    DarwinsTale
    Community Member

    This SUCKS! I want to push a button and import all 473 passwords in keychain... NOT read 473 posts on how I need to program a damn command prompt.

    I bought 1PW so that I could USE your program... not write programs.

    There must be an easier way.

  • MrC
    MrC
    Volunteer Moderator

    Hi @DarwinsTale,

    I don't work for AgileBits, and they are not responsible for the tool I created to help users convert.

    I don't know that Apple provides a way for a program such as 1Password to import the entire Keychain database.

  • Megan
    Megan
    1Password Alumni

    Hi @DarwinsTale,

    I sincerely apologize for the frustration here. Unfortunately, each password program handles the export of data slightly differently, so it's no simple feat to create an import tool that covers all the various password programs. We're really grateful that @MrC has taken the time to write these converters to make it more easy for our users to import their data into 1Password.

  • kf0s
    kf0s
    Community Member

    Hello. First of all, Great tool! I tried to convert my SplashID file in the past and wasn't totally happy with the results. Using this converter I am very happy with the results.

    I will ask the same question in the #23-#24 posts. Would it be difficult to put the Date Modified from SplashID (which you are already putting into the Notes field) into the Last Modified field in 1Password. If it's difficult, don't worry about it, but I do (since I have things across multiple devices) sort on the modified field. It would be create if I would see the same order in 1Password.

    I did try and look at some of the code to see if I could see what I could maybe change, but a change didn't jump right out.

    Thanks again for your tool! It is much appreciated!

    Eric

  • MrC
    MrC
    Volunteer Moderator

    Hi @kf0s‌,

    Great to hear - this type of feedback and positive results always makes for a nice way to end a day!

    Regarding the Modified Date, it can be placed into the 1PIF export. I don't know how that might affect WatchTower, however, but I'm willing to parse the date field and make it an option for you.

    Which platform did you run the conversion on? Windows or Mac?

This discussion has been closed.