MrC's Convert to 1Password Utility (mrc-converter-suite)

cs88rf

^ Answering my own question, they are moved to notes section and most non-English characters retained.

I have two issues though:

1) Many of username and password fields are now in Notes section. I have about 750 logins and the successful conversion rate is about 1/4 :angry: Will adding -a command solve this issue to a degree?

2) A smaller issue is the automatic tag addition as "imported [date]". I want to get rid of this tag but I probably will not go through 700 items manually! This may not have anything to do with the conversion though I assume?

MrC

@cs88rf,

Roboform is a curious password manager, at least in its export format. It exports only sufficiently enough to create an HTML page. It doesn't concern itself in the least with defined fields or any structure to help interpret record category or field semantics.

So the converter has to match against the strings in what appears like an "entry" in the HTML table. And of the string values present, it then has to match those against a table of known category / field mappings. If certain key fields match, then the category can be determined. When the strings don't match, the best that can be done is to provide either a Key: Value pair in the entry, or worst case, place the record as a Secure Note.

So, to localize the converter, I need the localize a single string that helps identify the record category, but that have to also add various possible strings that are present on web pages (Roboform works by using the web forum strings), so identifying what is a "password" isn't always trivial.

Does that make sense?

Edit: I was posting this during your post above. I think I've somewhat addressed what you are asking. I'll need to know what strings are used in your web forms.

The -a option won't help, since adding fields will just place the custom fields in a custom section, not in the correct "username" and "password' fields, to allow open-fill and auto-fill to work.

For your (2), we can customize your converter to omit the items you don't want.

cs88rf

It makes perfect sense but I'm just not technically capable enough to fully understand the details haha

By Key:Value I assume you meant thing like this(?):

_fieldformat$1#username##username: xxxx
_fieldformat$1#password##password: xxxx

I have many many logins right now that have no username or password but fall under notes like this.
Another example is this:

logininfo.logintype: 网站用户 (meaning website user)
logininfo.identifytype: 服务密码 (meaning service password)
webusername: xxxx
webpassword: xxxx

And not just for some weird websites! This following instance is for citi online login:

_fieldformat$1#user id##cA-cardsUseridMasked: xxxx
_fieldformat$1#password##PASSWORD: xxxx
_fieldformat$1#remember##cA-DD-remember-Check-Box: *

For automatically generated tag of imported date. How can I customize the converter? I am not sure but it may be generated by 1Password program during the import process? Edit: just did a quick search and looks like batch tag edit is just being developed and will come out in future updates, so I'm not too worried about it now. It will just be serving as a temporary visual nuisance for an OCD user =D

MrC

@cs88rf ,

Yes, you've got it. You're seeing the converter taking the raw _fieldformat$1#username##username form field, and not detecting that it is a username value of the Login. The reason for this is simple - I made the "patterns" that match usernames very strict, so as not to mismatch (false positive) too many items.

So all you or I have to do is add those patterns to the converter. For example, this is the pattern matcher "regular expression" that matches usernames:

my $username_re = qr/^(?:(?:user|login)(?:[\s_]*(?:name|id))?|email|log|uid|value\.login)$/i;

That looks like a lot of gibberish, but it basically defines what to look for. All we have to do is modify it a bit to match your username string, such as:

my $username_re = qr/^(?:(?:user|login)(?:[\s_]*(?:name|id))?|email|log|uid|value\.login|.*username)$/i;

and the same with password, which becomes:

my $password_re = qr/^(?:pass(?:word)?|pwd|loginpw|pword|pswd|.*password)$/i;

And for the Citi bank fields:

my $username_re = qr/^(?:(?:user|login)(?:[\s_]*(?:name|id))?|email|log|uid|value\.login|.*username|.*user id.*)$/i;

I've updated the 1.10 version of the converter to match the patterns above. Go ahead and update and see how the new one works.

I think I'm going to make adding new patterns much simpler for users. I'll take care of that tomorrow (Sunday).

Sorry, about the imported date, I misspoke - that's being automatically generated when you import into 1Password, and there's nothing I can do about it.

cs88rf

@MrC They look gibberish to me at first, but comparing the two I see you threw in the wildcard *username and *password there, then it became clearer. I skimmed through all the logins with mismatched fields, and looks like a wildcard solves the issue almost perfectly. About 90% of these logins partially contains "username" and "password" in their fields. I've seen one quirky out of ten logins so far, that uses something like "passwd", but this is very rare.

Allowing user to edit the matcher would greatly improve the versatility of this tool I imagine. Less people will report mismatched fields here and ask for your aid here like I am doing, as they become able to tune the converter on their own. That would hopefully reduce your workload.

And again, thanks! You are amazing! Troubleshooting for me on Saturday night :chuffed:

MrC

@cs88rf ,

That's great! Glad I could help.

I'll have an update tomorrow, which will simplify adding new strings and patterns. It will also include passwd, which probobly is common enough.

cs88rf

Thank you! I'm giving it a try right now. By the way, what is that separate keychain.pm file in the Testing Bits folder on dropbox? Is it something to do only with Apple's keychain?

cs88rf

@MrC Just pulled off another round of conversion. It treated all previously mentioned instances =D

Just reporting another mismatch, actually this one is for this forum:

Password$: xxxxx

Username was picked up, but somehow password fell into notes section, which is interesting, because it contains password but got ignored by the wildcard. Is it because of the capital p in the beginning?

MrC

@cs88rf ,

Let me make the changes today and I'll address all of your posts, here and in the other thread.

cs88rf

Thanks a lot! I'm actually quite happy with it right now. With about 80% of the logins converted correctly, I can tackle the rest manually quite easily. And once the fields are tidied up, OP fills like a champ!

I now become more concerned about the Android app performance heh. It feels even more of a work in process compared to Windows app!

AGAlumB

@cs88rf: I'm glad to hear that MrC was able to help out! I did want to jump in to mention that if you're having issues with 1Password for Android (or any other version), be sure to start a new discussion in the appropriate category with the details so we can help too. Cheers! :)

cs88rf

Thanks I will!

MrC

@cs88rf,

Here are my replies to your questions above:

By the way, what is that separate keychain.pm file in the Testing Bits folder on dropbox? Is it something to do only with Apple's keychain?

This was a test version of the OS X keychain converter I'd made available to a user to resolve a bizarre issue with his keychain export. It has since been included in the 1.10 release, so I've removed the Keychain.pm file from Testing Bits.

Re: the unmatched "Password$" string mentioned above, the update now works for this. It wasn't matched due to the trailing $ character. The new way I'm handling creation of the pattern match-ers should work better now.

You can now add new strings or patterns to some tables at the top of the converter module Converters\Roboform.pm. It is pretty heavily commented, so should be mostly clear for addition of simple strings. Ask if you need more assistance.

cs88rf

@MrC :

Reporting back with the following error:

C:\Users\SCVM\Desktop\convert_to_1p4>perl convert_to_1p4.pl roboform -v ..\passcards.htm
Unexpected duplicate key '_fieldformat$1#password##fm-login-password' in card_field_specs{login}

MrC

@cs88rf

Two questions....

Do you have the most recent version of the 1.10 converters?

Did you modify the Password or Username tables in the converter?

DKudler

I really appreciate the help you've provide here. I was able to convert my macOS Sierra (10.12.2) Keychain (once I figured out that I needed to use the newest version of the script — 1.10).

HOWEVER, the only login/password entries that converted were OS logins — admin logins for the family server, that kind of thing. None of the web browser passwords converted (Safari, Chrome, Firefox, Opera, etc.). Out of a bit over 300 total entries on the keychain, only 31 actually made it into 1Password.

I'm running 1Password 6. Is there any way of importing all of those passwords? I'd prefer not to spend the rest of my trial teaching the app passwords/logins that I've already saved!

MrC

Hi @DKudler,

Firefox passwords are not stored in Keychain.

Safari passwords are stored in Keychain. Do you see those in your Keychain if you use Keychain Access? If not, perhaps they are stored in the iCloud Keychain (which you would need to copy, export, and convert separately).

Many of the items inside the local Keychain are not useful, and are skipped (they are various security tokens, etc.). But only 31 of 300 seems low. Can you identify one that you expect to see imported that failed to import?

cs88rf

@MrC I downloaded latest 1.10 from the testing bits link in the first page after reading your last post yesterday. And I did not "modify" anything in the process. How do I do that?

MrC

@cs88rf ,

I've updated the code a couple of times today; since I'm not sure when you downloaded it, it would be best to grab a new copy.

Read the comments near the top of the code module Converters/Roboform.pm, which start as:

# Roboform does not use standardized Username and Password fields - rather, it uses the names of the
# fields in the HTML form data for the web page login, and those strings get stored in the exported

It will explain how to add new strings. Patterns you'll have to be careful with. If you need something more than just a simple text string, I'll help you define the pattern.

cs88rf

@MrC How do I read the comments and edit .pm file? I only knew to

perl convert_to_1p4.pl roboform --help

Just downloaded the newest (modified ~1 hour ago), and I do see the new options --windows added.

MrC

@cs88rf,

You can just open the file with any text editor, such as TextEdit (just don't modify -- save -- the file).

cs88rf

Ahh I see. I just opened with notepad and it's not easy to read. Can I assume #is used to break lines, which Notepad does not support?

BTW, technically what is the new --windows option for?

MrC

@cs88rf,

Don't use Notepad, its terrible for this since it doesn't handle the line endings correctly. Use Wordpad instead.

Let's take this offline - can you communicate via email instead? mike ( at ) cappella ( dot ) us

Te new option is for me to test conversion on a Mac of Windows export files.

cs88rf

Just did! Thanks :blush:

AGAlumB

Sounds like MrC should be able to help with the conversion, but be sure to let us know if you have any other questions about 1Password. :)

amfca

MrC,

Not able to get the AppleScript utility to work. When dragging exported roboform html file onto the script, terminal opens and has open perl script problem. macOS Sierra 10.12.3.

cd '/private/var/folders/28/hhnzfsvx0vs7p21vp7p11bnm0000gn/T/AppTranslocation/3FEA0EBD-7EC5-427C-9BDB-380D254DB880/d/' && /usr/bin/perl5.16 convert_to_1p4.pl roboform '/Users/alanfink/Desktop/pm_export_identities.html' -v
Can't open perl script "convert_to_1p4.pl": No such file or directory

Have attached gif showing issue. Any thoughts?? Thanks. Alan

MrC

@amfca ,

That has to be THE very best pictorial representation of a problem that has ever been reported to me - awesome!

Which version of the converter are you using? Check the Changes.txt file for the version number. You'll want the 1.09 version at least. But I'd also suggest the 1.10 version in Testing Bits mentioned in the first post of the converter suite , since I've been doing some heavy work on it to better match the username and password patterns seen on various websites, and allow simpler additions of new patterns.

amfca

Wow that was fast and appreciate the reply!
I always believe in using .mov or .gif captures of steps for trouble shooting bugs. I downloaded from agile bits site, it's 1.08. I will follow your link to converter suite and give 1.10 a look, then report back.

amfca

Sorry to report that 1.10 yields the same perl script error

....
Can't open perl script "convert_to_1p4.pl": No such file or directory

MrC

@amfca ,

The changes to Sierra have made this maddening. The sandboxing has made unsigned AppleScripts potentially fail, since it moves scripts to a private area before running them.

Just use the command line for now. It will look like:

cd ~/Desktop/convert_to_1p4
perl  convert_to_1p4.pl   roboform -v  ../pm_export.html

assuming your Roboform export is on the Desktop and is named named pm_export.html .