MrC's Convert to 1Password Utility (mrc-converter-suite)

1101113151652

Comments

  • dszp
    dszp
    Community Member

    @MrC Thanks for the pointer. I'm definitely not trying to say it wasn't there in the instructions, just that I didn't see it--I have a history with Perl scripting so I'm sure I skimmed the directions more lightly than I should have :-) But hey, I can apparently clean up my own messes as well at least...thanks again! Instead of diving in on the converter for my own data, I used the Mac 1Password to import from LastPass for myself back in November, which worked mostly fine except I had to do two separate types (two imports) and, now that I think about it, also ran into issues with Mac Chrome's export and had to use another browser to get an importable file. Is your memory supposed to start slipping in your early 30s? :-)

  • MrC
    MrC
    Volunteer Moderator
    edited January 2016

    @dszp - gottcha.

    I really wish I had a more dynamic wizard-like documentation language / flow, where users could actively engage with the documentation, as they follow the steps, and the documentation would tailor itself to their situation.

    Does your memory start to fade in your 30s? I forget.

  • dszp
    dszp
    Community Member

    @MrC Yeah, it seems like the standardized argument list would work well with a basic GUI wrapper/wizard, but then there's the whole GUI wrapper to write and maintain...Robocopy has the RobocopyGUI is kind of what I'm thinking, but that's kind of hypocritical of me since I prefer the Robocopy CLI to the GUI wrapper I guess :-)

  • stace3
    stace3
    Community Member

    @MrC I appreciate your speedy response. I tried your method but received this error:

    C:\Users\Stacey\Desktop\convert_to_1p4>perl convert_to_1p4.pl spbwallet -v ..\My_Wallet.swl
    Enter your SPB Wallet password:
    DBD::SQLite::db prepare failed: no such table: spbwlt_Template at Converters/Spbwallet.pm line 232, line 1.
    Can't call method "execute" on an undefined value at Converters/Spbwallet.pm
    line 233, line 1 (#1)
    (F) You used the syntax of a method call, but the slot filled by the
    object reference or package name contains an undefined value. Something
    like this will reproduce the error:

        $BADREF = undef;
        process $BADREF 1,2,3;
        $BADREF->process(1,2,3);
    

    Uncaught exception from user code:
    Can't call method "execute" on an undefined value at Converters/Spbwallet.pm line 233, line 1.
    Converters::Spbwallet::do_import('..\My_Wallet.swl', undef) called at convert_to_1p4.pl line 116

    C:\Users\Stacey\Desktop\convert_to_1p4>

    Any possible next steps? Would love to get this data into 1Password without manually entering everything.

  • MrC
    MrC
    Volunteer Moderator
    edited January 2016

    Hi @stace3,

    Sorry for your troubles. Let's see if we can figure out what's going wrong.

    Run the same command, but add the additional command line voodoo:

     -d | grep print_fileinfo
    

    so your full command will be:

    perl convert_to_1p4.pl spbwallet -v ..\My_Wallet.swl  -d | grep print_fileinfo
    

    The command will hang waiting for your password to be entered - enter it and the command will complete (but fail of course). Copy and paste the print_fileinfo lines here in a reply.

  • stace3
    stace3
    Community Member

    Thank you. I think I did this correctly but receive the following error message:

    C:\Users\Stacey\Desktop\convert_to_1p4>perl convert_to_1p4.pl spbwallet -v ..\My_Wallet.swl -d | grep print_fileinfo
    'grep' is not recognized as an internal or external command,
    operable program or batch file.

    I copied and pasted from your message above.

  • MrC
    MrC
    Volunteer Moderator

    Ah, darn, I wasn't paying attention. This is Windows. Just rerun your command adding the -d option, and copy and paste only the three print_fileinfo lines. Those lines won't contain any confidential information. It will look like this:

    print_fileinfo : Export file info
    print_fileinfo : size: 223232
    print_fileinfo : kind: SQLite 3.x database
    print_fileinfo : mime: application/octet-stream; charset=binary

  • stace3
    stace3
    Community Member

    I'm sorry, you've lost me. And I realize I have made a mistake in my initial run. The file name is not My_Wallet.swl it is My Wallet.swl This is the error I receive:

    C:\Users\Stacey\Desktop\convert_to_1p4>perl convert_to_1p4.pl spbwallet -v ..\My Wallet.swl -d
    main : Command Line: spbwallet -v ..\My Wallet.swl -d
    Missing export_text_file name - please specify the file to convert
    Usage: convert_to_1p4.pl

    converters:
    clipperz csv dataguardian datavault essentialpim ewallet handysafe ironkeyim keepass2
    keepassx keychain lastpass licensekeeper msecure nortonis onepif2html passpack
    passwordagent passworddepot passwordwallet safeincloud safewallet spbwallet splashid
    vcard

    options:
    -d or --debug # enable debug output
    -D or --dump # dump category / field table
    -e or --exptypes # comma separated list of one or more export types from list below
    -f or --folders # create and assign items to folders
    -h or --help # output help and usage text
    -i or --imptypes # comma separated list of one or more import types from list below
    -o or --outfile # use file named ofile.1pif as the output file
    -v or --verbose # output operations more verbosely
    --nowatchtower # do not set creation date for logins to trigger Watchtower checks

    supported import types:
    bankacct contact creditcard driverslicense emailacct frequentflyer hosting idcard
    librarycard membership note onlineshoppingacct passport personalinsurance server
    socialsecurity software website
    supported export types:
    bankacct creditcard driverslicense email identity login membership note passport rewards
    server socialsecurity software

    It seems it will not recognize the file with the space in the name? Is that correct? I apologize, this is quite beyond anything I have done before.

  • MrC
    MrC
    Volunteer Moderator
    edited January 2016

    @stace3,

    Ok, add double quotes around your file name. That might just be the issue. Try this first:

     perl convert_to_1p4.pl spbwallet -v "..\My Wallet.swl"
    

    If that fails, then add the -d and report the print_fileinfo lines:

     perl convert_to_1p4.pl spbwallet -v "..\My Wallet.swl"  -d
    

    The double quotes are required because your filename has a space in it, and we have to tell the windows command shell that it is part of the parameter being passed to the perl command (and not the two separate words "..\My" and "Wallet.swl").

    You're doing great, by the way!

  • stace3
    stace3
    Community Member

    It worked! Thank you so much! You have saved me so much work.

  • MrC
    MrC
    Volunteer Moderator

    @stace3 - You're awesome! Nice work. Enjoy 1Password.

  • jrsforums
    jrsforums
    Community Member

    @MRC. I am using Win 10 and Roboform 7.9.17.5. Installed latest 'stable bits' and output html file for logins. On running, got following error message.

    Unexpected duplicate key 'passed-username' in card_field_space{login}

    I can send html file, but it is ~2.4 mb

  • MrC
    MrC
    Volunteer Moderator
    edited January 2016

    Hi @jrsforums ,

    The converter checks to be sure there are not multiple identical field names in a record. I commented about this in another thread. Are you able to change one of the field names in the record in question and rerun the converter?

    For this converter, I may have to handle duplicate field names specially, but need to think about it. I could use help in knowing which website(s) trigger this.

    Please never send your exported data! I won't read it, and more importantly, you don't want it traversing the Internet unencrypted.

  • MrC
    MrC
    Volunteer Moderator

    Hi @jrsforums,

    I have more info to add. I'd forgotten about how the RoboForm converter matches username and password field names. It uses patterns, and so what I think is happening is that your 'passed-username' pattern is matching one of the username patterns. If you can tell me the two field names in that record that appear to look like or "username" fields, I can tighten up the pattern that matches these. This is something I was going to do before the 1.08 release but forgot about that. There isn't a set list of patterns - rather, it will vary based on each user's web site logins, since these field labels vary across the web in no formal manner.

  • onemico
    onemico
    Community Member

    Hi,

    I have attempted this several times over the last day or so, to convert on OS X El Capitan, using Keychain, and I keep getting the following:

    XXXXXXXX-MacBook-Pro:convert_to_1p4 XXXXX$ perl5.16 convert_to_1p4.pl keychain -v ../../../Desktop/pm_export-icloud.txt
    Odd number of elements in hash assignment at Converters/Keychain.pm line 146 (#1)
    (W misc) You specified an odd number of elements to initialize a hash,
    which is odd, because hashes come in key/value pairs.

    Use of uninitialized value in list assignment at Converters/Keychain.pm line
    146 (#2)
    (W uninitialized) An undefined value was used as if it were already
    defined. It was interpreted as a "" or a 0, but maybe it was a mistake.
    To suppress this warning assign a defined value to your variables.

    To help you figure out what was undefined, perl will try to tell you
    the name of the variable (if any) that was undefined.  In some cases
    it cannot do this, so it also tells you what operation you used the
    undefined value in.  Note, however, that perl optimizes your program
    anid the operation displayed in the warning may not necessarily appear
    literally in your program.  For example, "that $foo" is usually
    optimized into "that " . $foo, and the warning will refer to the
    concatenation (.) operator, even though there is no . in
    your program.
    

    Examined 378 items
    Skipped 378 non-login items
    Skipped 0 duplicate items
    Imported 0 items
    Exported 0 total items

    Can anyone confirm what I am doing wrong, followed both the PDF guide, and the explicit OS X guide on the "Extra Help" page on the first post?

    Any help greatly appreciated, pulling my hair out here :)

  • MrC
    MrC
    Volunteer Moderator
    edited January 2016

    @onemico,

    What version of the converters are you running? Your command seems to follow some instructions from older versions.

    If you are not using the 1.08 version of the converters, please update. If you are, try the following command and report the output:

     perl convert_to_1p4.pl keychain -v ../../../Desktop/pm_export-icloud.txt  -d  | grep print_fileinfo
    

    I'll take a look at the code in the morning to see under what circumstances you might get this error.

  • onemico
    onemico
    Community Member
    edited January 2016

    @MrC

    Hi, I am using 1.08, attached is the output you requested, hopefully this will help, thanks very much.

    Also to add, it does create the 1P_import.1pif file, but it is a blank 0 bytes file

    xxxxxx-MacBook-Pro:convert_to_1p4 xxxxx$ perl convert_to_1p4.pl keychain -v ../../../Desktop/pm_export-icloud.txt -d | grep print_fileinfo
    print_fileinfo : Export file info
    print_fileinfo : size: 316432
    print_fileinfo : kind: ASCII text, with very long lines
    Odd number of elements in hash assignment at Converters/Keychain.pm line 146 (#1)
    (W misc) You specified an odd number of elements to initialize a hash,
    which is odd, because hashes come in key/value pairs.

    Use of uninitialized value in list assignment at Converters/Keychain.pm line
    146 (#2)
    (W uninitialized) An undefined value was used as if it were already
    defined. It was interpreted as a "" or a 0, but maybe it was a mistake.
    To suppress this warning assign a defined value to your variables.

    To help you figure out what was undefined, perl will try to tell you
    the name of the variable (if any) that was undefined.  In some cases
    it cannot do this, so it also tells you what operation you used the
    undefined value in.  Note, however, that perl optimizes your program
    anid the operation displayed in the warning may not necessarily appear
    literally in your program.  For example, "that $foo" is usually
    optimized into "that " . $foo, and the warning will refer to the
    concatenation (.) operator, even though there is no . in
    your program.
    

    print_fileinfo : mime: text/plain; charset=us-ascii

  • MrC
    MrC
    Volunteer Moderator
    edited January 2016

    Hi @onemico,

    Thanks for the info. Now we can move onto the next steps. I'll need you to download the 1.09 version in Testing Bits. I've added some debug code to help isolate the issue.

    Run your command with the -d debug option included;

    perl convert_to_1p4.pl keychain -v ~/Desktop/pm_export-icloud.txt -d
    

    There will be a lot of output, but the converter should abort when it encounters your error. It will output the name of the attribute which is causing the problem, as well as the entire list of attributes for the record. It will look like this:

    Unexpected undefined value in class=inet attribute '0x00000008 <blob>'
        0x00000007 <blob>="YouTube"
        0x00000008 <blob>=
        "acct"<blob>="gomer@example.com"
        "atyp"<blob>=<NULL>
        "cdat"<timedate>=0x32303135303132373035343731335A00  "20150127054713Z\000"
        "crtr"<uint32>=<NULL>
        "cusi"<sint32>=<NULL>
        "desc"<blob>=<NULL>
        "icmt"<blob>=<NULL>
        "invi"<sint32>=<NULL>
        "mdat"<timedate>=0x32303135303132373035343731335A00  "20150127054713Z\000"
        "nega"<sint32>=<NULL>
        "path"<blob>=<NULL>
        "port"<uint32>=0x00000000
        "prot"<blob>=<NULL>
        "ptcl"<uint32>=0x00000000
        "scrp"<sint32>=<NULL>
        "sdmn"<blob>=<NULL>
        "srvr"<blob>="YouTube"
        "type"<uint32>=<NULL>
    

    What I need to understand is which attribute name is causing the issue. In this case, it is the attribute named 0x00000008 <blob>, and as you can see above, there is nothing after the = character for it in the attribute list. Don't report any values in your reply, but reply with the error line and the srvr or svce attribute, as in:

    Unexpected undefined value in class=inet attribute '0x00000008 <blob>'
        "srvr"<blob>="YouTube"
    

    Let's see if this gets us closer to the error.

  • onemico
    onemico
    Community Member

    Thanks @MrC but I am not getting that output with 1.09 version, below is my output"

    xxxxxx-MacBook-Pro:convert_to_1p4 xxxxx$ perl convert_to_1p4.pl keychain -v ~/Desktop/pm_export-icloud.txt -d
    main : Command Line: keychain -v /Users/xxxxx/Desktop/pm_export-icloud.txt -d
    main : Output file: /Users/xxxxx/Desktop/1P_import.1pif
    print_fileinfo : Export file info
    print_fileinfo : size: 316432
    print_fileinfo : kind: ASCII text, with very long lines
    print_fileinfo : mime: text/plain; charset=us-ascii
    do_import : Entry 1
    do_import : raw attibute list:
    /Users/xxxxx/Library/Keychains/local-icloud.keychain
    Unexpected undefined value in class=/Users/xxxxx/Library/Keychains/local-icloud.keychain attribute '/Users/xxxxx/Library/Keychains/local-icloud.keychain'
    /Users/xxxxx/Library/Keychains/local-icloud.keychain

  • MrC
    MrC
    Volunteer Moderator
    edited January 2016

    @onemico,

    Ok, that is progress then! So it appears the block that represents an entry is not in a format I'm expecting, and it is the first entry you're having problems with. A block should look like this, in this format:

    keychain: "/Users/fredsmith/Library/Keychains/login.keychain"
    class: "inet"
    attributes:
        0x00000007 <blob>="YouTube"
        0x00000008 <blob>=<NULL>
        "acct"<blob>="gomer@example.com"
        "atyp"<blob>=<NULL>
        "cdat"<timedate>=0x32303135303132373035343731335A00  "20150127054713Z\000"
        "crtr"<uint32>=<NULL>
        "cusi"<sint32>=<NULL>
        "desc"<blob>=<NULL>
        "icmt"<blob>=<NULL>
        "invi"<sint32>=<NULL>
        "mdat"<timedate>=0x32303135303132373035343731335A00  "20150127054713Z\000"
        "nega"<sint32>=<NULL>
        "path"<blob>=<NULL>
        "port"<uint32>=0x00000000
        "prot"<blob>=<NULL>
        "ptcl"<uint32>=0x00000000
        "scrp"<sint32>=<NULL>
        "sdmn"<blob>=<NULL>
        "srvr"<blob>="YouTube"
        "type"<uint32>=<NULL>
    data:
    "My Secret Password"
    

    It should start with the keychain name, followed by the class type, followed by the (indented) attributes name=value pairs, and then followed by a data block, where the value is on the next line.

    Can you comment on what you're seeing?

  • onemico
    onemico
    Community Member
    edited January 2016

    Thanks, this is the first block is pasted below, with the obvious changes made to username and password
    keychain: "/Users/xxxxx/Library/Keychains/local-icloud.keychain" version: 256 class: "genp" attributes: 0x00000007 <blob>="AirPort" 0x00000008 <blob>=<NULL> "acct"<blob>="TP-LINK_55FF68" "cdat"<timedate>=0x32303136303132323038323434375A00 "20160122082447Z\000" "crtr"<uint32>=<NULL> "cusi"<sint32>=<NULL> "desc"<blob>=<NULL> "gena"<blob>=<NULL> "icmt"<blob>=<NULL> "invi"<sint32>=<NULL> "mdat"<timedate>=0x32303136303132323038323434375A00 "20160122082447Z\000" "nega"<sint32>=<NULL> "prot"<blob>=<NULL> "scrp"<sint32>=<NULL> "svce"<blob>="AirPort" "type"<uint32>=<NULL> data: "Password"

  • MrC
    MrC
    Volunteer Moderator

    @onemico,

    You're an awesome debug assistant! I've spotted the issue and have updated the 1.09 converter in Testing Bits. Give the new one a try!

  • onemico
    onemico
    Community Member

    @MrC

    You are a legend, working now perfectly. Thank you very much. And thanks, not a coder, but do work in IT, reading debugs all day, so always try to format OK and reply ASAP :)

    Thanks again for the great tool and for helping troubleshoot individual issues like this

  • MrC
    MrC
    Volunteer Moderator

    @onemico,

    Outstanding! Enjoy 1Password.

  • jrsforums
    jrsforums
    Community Member

    @MrC Not exactly sure what to look at in the data. Am i looking for a duplicate or problem tag or data in a tag? Any indication of what tag? Would it be possible for you error routine to indicate which record the problem is in or to bypass the record. Obviously, I have a lot of records in the 2.4 mb of data.

  • MrC
    MrC
    Volunteer Moderator
    edited January 2016

    Hi @jrsforums,

    It is a little hard to describe. Roboform, unlike other password managers doesn't have fixed field names for a record, such as Username, Password, URL, Notes, etc., or fixed positions in an export. Instead, it uses the actual names within the HTML web page as the field names. So the converter uses patterns to try to find the name of these fields so that it can assign them to 1Password's fields.

    You've discovered one field name - it is passed-username. But this is probably matching some other field name in that record, perhaps it is simply username or user, and then the converter complains about duplicate fields.

    The problem is that if i make the field detection too restrictive, many of your username and password fields won't get mapped correctly. If I make the patterns too relaxed, it will incorrectly detect those extra fields that RF grabbed from the web page.

    I'm going to change the code today to allow you to more easily enter patterns that can be used to detect these fields, as well as prevent certain patterns from being used. That way, you can just add your field such as passed-username into a table, and it will no longer be considered as a candidate for Username, and therefore won't trigger the duplicate field detector.

    I'll have something for you this evening (PST).

  • jrsforums
    jrsforums
    Community Member

    @MrC Searched html file with notepad and crimson editor. Could not find string passed-username

  • MrC
    MrC
    Volunteer Moderator

    Hi @jrsforums,

    Since you reported that the duplicate field was named passed-username, that was the term I used. I don't know what you actually have, so you'll have to double check your results.

    I just updated the 1.09 version in Testing Bits. Can you try that out?

  • jrsforums
    jrsforums
    Community Member

    I copied, exactly, the error message. Yes, I will try 1.09

  • jrsforums
    jrsforums
    Community Member

    @MrC. To go to 1.09 does one need to do a full install or just bring in new modules? It may be in doc, but I do not remember seeing.

This discussion has been closed.