MrC's Convert to 1Password Utility (mrc-converter-suite)

MrC

I've updated the script to resolve the empty website issue.

I'd still like to know what the multiple websites entries look like, so that I can ensure the CSV is encoded correctly, and perhaps even place the extra websites into another column. Are the comma or semicolon separated, or space, or are they separated by newlines?

Thanks for all your help @sebapostol @JonathanFalcon

This week I'll work on combining the two scripts into a single script and having it called directly by the AppleScript_Conversion_Helper script, so that it can do the resulting CSV conversion automatically.

JonathanFalcon

It turns out my blank entry came back on that second try, and that's where it stopped. I'll try your updated script, since the blank website appears to be an iCloud Keychain bug I forgot about. I'll later attach an image showing how the CSV appears. For now, here's how the multiple website entries appear in the Password window:

JonathanFalcon

I had to turn off my Wi-Fi to make sure iCloud stopped adding it back, but it definitely worked when I had no blank website entries. Unfortunately, it seems like it still doesn't work with blank entries. Either that or it's just my darn iCloud Keychain bug. Since the blank websites aren't of any importance and are causing some trouble, you might just add a warning in the post to check for such.

Also, I looked at the CSV, and it looks like the script prefers the first entry and ignores the second. In my prior screenshot, it chose "turnitin.com" and ignored the second entry "www.turnitin.com". The CSV just doesn't even show the second website. And from what I can tell, it did that with all of the accounts. That's fine with me, since they were essentially just duplicates from Safari's failure to understand that adding "www." doesn't make it a new website. Thank you for bearing through the entire troubleshooting process with us!!

MrC

@JonathanFalcon ,

After much trial and error, I've finally been able to create my own sample Safari password entries that replicate what you have on your system. I have entries with empty URL fields and with multiple URLs.

I now have code working that handles the empty cases, and retrieves all the coalesced URLs.

Since I can't know which URL is the correct one when there are multiple URLs, I can do one of two things: 1) grab and use the first one, and store the others in the Notes area of the entry, or 2) create separate 1Password entries, one for each URL. I think option (1) is much cleaner, as you can later simply edit the URL or have 1Password save it for you when you log into the site, and you won't have a potentially confusing number of essentially duplicate Login items, differing only by URL. So I'm leaning towards (1). Your thoughts?

I'm tired tonight - I'll update the final code in the morning.

Extra info below:

Since I had no samples to test with, I had to assume how empty URL values were being handled. My assumptions were incorrect, so my most recent update didn't fix the issue for you.

And I had no idea how the multiple URLs were being stored, so didn't know what to do with the URL value(s). Your screenshot above had the key clue which helped my figure out how to create these. Multiple URLs are "created" when you have 2 or more saved Safari password entries which point to the same base domain. These are fictions in Safari's passwords area - in actuality, Safari is coalescing these multiple entries into a single virtual line item. The text in your screenshot ... "and 1 more" ... told me there was some way to enter multiple URLs and that once I'd figured out how to do it, I'd see that same text. And I now have such an entry.

FYI (technical alert ahead)...
"adding "www." doesn't make it a new website". Actually, it can. This is entirely a DNS and/or web server configuration choice. There may be two different address records for "example.com" and "www.example.com", and the web server itself may dispatch these to entirely different web sites. It is common for the domain's DNS provider to allow configuring the root domain to resolve to an address host record. The "www" host part is just a de facto standard, eventually realized to be generally superfluous, so admins configured their systems (DNS and web server) to respond identically. And the world got tired of pronouncing and hearing "W W W dot blah dot com" on TV, radio, etc. (6 syllables for three letters - shesh).

MrC

@JonathanFalcon @sebapostol

OK, let's see if the update I just posted resolves the issues for you.,

Extra URLs will go into the Notes area. The URL order is somewhat arbitrary when Safari presents coalesced URLs. I just use the first one in the list.

JonathanFalcon

Apologies for the late response; I have classes in the morning. I about 80% understand what you said about "www.", but I get the main idea.

And with the extra URL's, what you did was a good idea, since, in my short time with it, 1Password doesn't appear to be confused about "www." and subdomains.

Lastly, iCloud put the empty URL back in there, so I can test it. Just finished testing, and it still presents the same error. I believe I might have discovered what's going wrong though. To view an entry fully, you can either double-click the entry or highlight it and click the "Details..." button. It would seem that the entries with blank URL's don't follow that rule. You must either quadruple-click or highlight and click the "Details..." button twice. I'm sure the blank URL bug in iCloud is responsible for the strange requirement. If only I had realized this sooner!

MrC

@JonathanFalcon ,

Oof, this will drive me batty yet!

I have an empty URL entry:

and it works just fine, so I'm burning my brain trying to think what might be different in your case. To open any of the row entries, you just select the row, and hit Enter. That's what the code is doing - selecting the row, and simulating the Enter key press. The Details button is the default button, so that's why Enter works. No need to quadruple click - double click also works for me, on even the blank URL entries. For the empty URL entries, does the "sheet" open for you when you select the row and hit Enter?

The empty URLs can be created in older Safaris. I've had to use old Safari's, old macOS' and Keychain Access to manipulate these. Newer versions of these things prevent these kind of useless entries.

JonathanFalcon

Even with selecting it and hitting enter, it does nothing. I have to hit enter again to do anything. Using enter once works on everything but the blank URL entry.

Okay, so you saying you had to use Keychain Access gave me the idea to see how that entry looked in Keychain Access. I finally found it and here’s what it looks like:
It appears blank in Safari and on my iPhone, but Keychain Access shows the URL as a lone "https://" without the rest of a usable URL.

And again, I really think it’s just a bug with iCloud Keychain. I never actually created the entry with a blank URL, and every time I’ve deleted it, it came right back.

MrC

@JonathanFalcon

Can you give me the error line that is returned in Script Editor when it fails?

We can ignore the "https://" that Keychain Access is showing you - it presents that leader text, but it doesn't mean it is actually stored in the record. Keychain Access and Safari both present values different from how they are actually stored, and it is hard to know without examining them in a debugger.

I'm happy to skip over the blank entry, but I don't know yet how to identify it uniquely.

JonathanFalcon

Before I start that whole process again, here's a video showing what's happening. I think what’s happening is the script presumes that the “sheet” will pop up, but it doesn’t on that one entry thus presenting the error message.

MrC

@JonathanFalcon ,

This may very well be a bug in Safari, that prevents the sheet from opening. I'll see if I can write up a workaround (try to open the sheet, if it fails to open, try again once more).

MrC

@JonathanFalcon

I've posted an update which tries to skip those rows that don't open normally. Since I have no way yet to test this, let's see if this helps.

JonathanFalcon

It worked! A CSV was saved to my desktop, and the blank entry had no effect on the file. I then converted that CSV and imported it into 1Password to make sure it was good, and 1Password displayed the extra URL’s in the notes, per your design. Hopefully, nobody will have any problems now! If there’s anything else you need from me, I’m happy to help, but I think you solved just about everything.

MrC

Outstanding!

Thanks for hanging in there, and for all the help.

@JonathanFalcon @sebapostol

scheb

Worked as designed - Thank you @MrC !

kaym00

Hello, I have spent days trying to make this work using various methods, and came across this thread.

I am trying to move form keychain in OSX, however the output isn't helping me. (running 10.14.1 OSX)

perl convert_to_1p4.pl keychain --verbose output
Enter the keychain's password:
Invalid Keychain Format

Any tips/pointers?

Thanks.

MrC

Hi @kaym00 ,

Are you really running 10.4.1 ?

kaym00

Oops! I hope not (edited). 10.14.1 ;)

Thanks.

MrC

@kaym00 ,

The command you would want is:

perl convert_to_1p4.pl keychain --verbose

And not

perl convert_to_1p4.pl keychain --verbose output 

kaym00

Hm, sorry to be a little thick here.

I did try that and it wanted more, as part of the command to execute - essentially I was trying to see if I could see some helpful output to help resolve the issue.

$ perl convert_to_1p4.pl keychain --verbose
Missing export_text_file name - please specify the file to convert
Usage: convert_to_1p4.pl

I then tried with debug which gave me this:

$ perl convert_to_1p4.pl keychain -d output
main : Runninng script from '/Users/XXX/Downloads/onepassword-utilities/convert_to_1p4'
main : Command Line: keychain -d output
main : Output file: /Users/XXX/Desktop/1P_import.1pif
print_fileinfo : Export file info: "output"
print_fileinfo : size: 0
print_fileinfo : kind: empty
print_fileinfo : mime: inode/x-empty; charset=binary
Enter the keychain's password:
Invalid Keychain Format

Any tips/pointers would be fantastic.

Thank you.

MrC

@kaym00

Use the AppleScript_Conversion_Helper - it will help you select the keychain database file. You can supply -v (and -d if you want) when the dialog asks for additional conversion options.

The --debug output you have above tells us that the file you are suppling as the keychain file is 0 bytes long (its empty). That makes sense, since you like have no file named "output".

If you don't want to use the AppleScript_Conversion_Helper, you'll have to supply the correct path to the Keychain DB file.

MrC

@kaym00

You can list the keychains, should you want to build the command line yourself, with the following command:

security list-keychains

and the output will be something like:

 "/Users/MrC/Library/Keychains/login.keychain-db"
 "/Library/Keychains/System.keychain"

MrC

@kaym00

Did you get your keychain exported?

I see from your output above a line:

'/Users/XXX/Downloads/onepassword-utilities/convert_to_1p4'

and this leads me to believe you have version 1.10 of the converter suite. You want version 1.11 in Testing Bits, mentioned in the first post of this thread.

kaym00

Perfect, thank you - I can confirm 1.11 worked flawlessly. However one more headache to get around, so I extracted the "login" keychain which I assume is the default keychain, most of my entries seem to be under the "icloud" keychain, which I cannot ascertain where it actually is located, I assumed it was actually held on the magical icloud.

I have attempted to google, but seems to be an unknown topic. I did attempt to copy entries out in to a local keychain, but I see the error "One or more parameters passed to a function were not valid.

Have you come across this? again thank you. :)

MrC

That’s great news @kaym00 !

ICloud password items are no longer copyable in the latest macOS versions. Most likely the items you want are your Safari passwords.

You can extract your Safari passwords using the Get_Safari12_Passwords script available in Testing Bits if you are using Safari 12. Use the 11 version for Safari 11.

Use Get_Safari12_Passwords by opening it with Script Editor, and run it. It will guide you through the process.

kaym00

@MrC All working and imported now. again thank you so much for your help! :). the last script worked perfectly, it just took a while to extract, but once that was completed it was a slick process.

MrC

😁

Enjoy 1Password, @kaym00 !

creed_erickson

A pair of issues with Mac OS 10.14.2 with Safari 12.0.2 using your "Testing Bits" download ...

Issue 1) Using the Get_Safari12_Passwords applescript I got the following error:

"System Events got an error: Can’t get window \"General\" of application process \"Safari\"." number -1728 from window "General" of application process "Safari"

This was resolved by adding a line "set prefsWin to window 1" after line 69. Because the window title changes when the tab bar button is pressed the script is looking for the table in a non-existant "General" window. Doing this reassignment seems to clear it up.

Issue 2) Attempts to unzip the convert_to_1p4_1.11.zip file using the archive utility results in an output file of convert_to_1p4_1.11.zip.cpgz. Using the command line "unzip" utility results in an error about not finding a directory marker. This has me stymied.

MrC

@creed_erickson ,

Thank for the assist.

Re: (1): I've posted an update to the script. I have no idea why it works for me as is, but not you. Applescript UI scripting is so fickle.

Re: (2): This confuses me. Both double-clicking and using unzip(1) work fine for me on the convert_to_1p4_1.11.zip archive in Testing Bits. Check your results against mine below:

$ /sbin/md5 convert_to_1p4_1.11.zip
MD5 (convert_to_1p4_1.11.zip) = 978c259a06ba0773afb2e3f10f551b45

$ /usr/bin/zip -sf convert_to_1p4_1.11.zip
Archive contains:
  AppleScript_Conversion_Helper.app/
  AppleScript_Conversion_Helper.app/Contents/
  AppleScript_Conversion_Helper.app/Contents/Info.plist
  ...
  Utils/
  Utils/Normalize.pm
  Utils/PIF.pm
  Utils/PwCheck.pm
  Utils/Utils.pm
  convert_to_1p4.pl
  install_modules.bat
Total 94 entries (1476443 bytes)

creed_erickson

@MrC

I have no idea why The AppleScript did not work here as is, but the change seemed sensible and made it work. Maybe my corner of the universe has its polarity mixed up. ¯_(ツ)_/¯

Re: unzipping problem - Corrupted download. Sorry for the false alarm and thanks for helping.