Bug: 3rd Party Integration gone in 1Password 4.4

I have found the cause of the bug! Scroll down to the Edit below:

I've now confirmed this on a 2nd computer which was running beta 4.3-something (the last beta before 4.4 came out), which means that the bug was introduced even further back.

Here's something fun:

• Disabling "3rd party integration" causes bookmarks-default.json to be deleted
• Enabling "3rd party integration" does NOT cause bookmarks-default.json to be created again.
• BUT: BEFORE I did the disable/enable-dance that caused the file to be deleted, I actually HAD the file thanks to an earlier version having created it, AND it was being kept UPDATED by the latest 1P4.
• This means: 1P4 is now unable to CREATE a NEW bookmarks-default.json, but it WILL maintain any existing file that was in the folder. This shows that the bug could have existed for a long time with nobody noticing.

Edit: Hmm, I just did some more testing:

• If I use Time Machine to RESTORE a valid bookmarks-default.json file, and then MODIFY any item inside of 1P4, it will cause the whole bookmarks-default.json file to be refreshed/rewritten. This was on 1P4 4.4! It gives me an idea...
• Next, I tried disabling/re-enabling "3rd party integration" to cause 1P4 to delete the file again, then deleted the whole "3rd Party Integration" folder to start fresh, re-started 1Password 4, made sure that "3rd party integration" was still enabled (with no file existing), and then I tried MODIFYING an item in 1P4... and: Nope, it did NOT cause bookmarks-default.json to be created.

There you have it! I have isolated the bug to this:

• If there is no bookmarks-default.json file already, then none will be created (this is the bug). If one exists, it will be maintained. If you disable/re-enable "3rd party integration" then the file will be deleted and then not re-created, due to this bug.

Hello @‌MikeT

Your request to have 1Password submit sites saved in its database might happen as long as the user permits it first and as long as we have an encrypted but more importantly, anonymous method of transmitting the list of sites.

Okay. The unfortunate thing is that Watchtower isn't very effective against Heartbleed without an auto-submission feature. It's very misleading to have a "Watchtower" section that supposedly gives users vulnerability-info for their sites, without telling them that most of their sites aren't even tested (unless they manually type them in on the watchtower website) and therefore won't be listed. The current system also suffers from stale results, due to someone perhaps testing a site 14 days ago and then never again, which means that everybody with that particular site in their DB might see it as "vulnerable" when it no longer is, or whatever.

Not sure you guys have time for programming an auto-submission system, but just throwing these ideas out there for how it can be structured server-wise:

HTTPS with the valid certificate signature built into the app to prevent impersonation. Make a server that sits behind Cloudflare (which won't log the customer's HTTPS POST submissions) and have a back-end script that likewise doesn't log the request or the originating IP, just the list of domains. This will be anonymous, since your server will only see the Cloudflare IPs in your access logs (since the actual, real customer IP is in an extra "proxy" header by Cloudflare, See: https://support.cloudflare.com/hc/en-us/articles/200170786-Why-do-my-server-logs-show-CloudFlare-s-IPs-using-CloudFlare-).

That makes it anonymous enough.

Make sure that the 1P4 client only uploads a list of domains; not the full URLs. I.e. www.sensitivesite.com rather than www.sensitivesite.com/secreturl. And make it submit them all at once in one big, efficient blob for delayed-processing by your server.

Then make your back-end script merge the incoming blob of the customer's domains into a global queue of "incoming domains," which is de-duplicated-on-merge (since lots of customers have the same sites and we only want to add one entry for each).

Then periodically run a cron job on this "incoming" list, which scans through the entire list of domains that people have auto-submitted and throws away all entries that it has already RECENTLY tested (keep a separate DB of the last test-time of all domains), and then pushes all remaining, new sites / sites not tested in a while into a 2nd queue for further processing. When this step is done, you can clear the current "incoming" queue.

Then have a 2nd cron job which runs through the 2nd queue of sites-to-actually-test and performs the testing, and then pushes those results into a 3rd queue of finished test-results.

Lastly, have a 3rd cron job which reads the queue of finished test-results once per hour or so, and merges the new results into the vuln.sqlite database on the server, that customers finally download.

Also remember to use a read/write lockfile for each individual queue so that the 3 separate cron jobs don't attempt to modify the same files at the exact same time due to their intervals periodically overlapping. That would be bad.

That's how I would design an anonymous, load-balanced system that can handle the volume of sites that would be constantly flooding in via customers. Queue, de-duplicate and rate-limit as much as possible. Consider using a Node.JS server and keeping all of the 3 queues in-memory and it'll be ultra-efficient (no PHP/Apache overhead). To improve things further, it would be good if the 1P4 clients ••intelligently•• used their vuln.sqlite on their local system, with a new "last-tested-at" timestamp AND a "vulnerability certainty", to determine which sites their clients no longer need to submit since they've already been definitively tested. This would further lessen the load on the server.

I was both lazy and a programmer so I wrote the PHP script above to do all of the test-submissions for me... Other people are not so lucky, unfortunately. ;)

Did you do this while you were in a secondary vault? We're tracking down a bug where if you're in a secondary vault, it will not create the file.
If you were in the primary vault, it does work for me. Even on a fresh install that never had this feature enabled before and thus the file was never created before.

Nope.

My vault was created in 1P 4.2-BETAsomething as a blank, new vault. I then imported my 1PIF file from 1P3 into that vault. Not sure if this info helps.

I have no extra vaults. Just one main vault.

The latest time that my "3rd party integration" file was successfully created from scratch on this system was back after I had freshly installed Mavericks and 1P4 on this laptop about 2 months ago. You were in the 4.2-BETA series at the time. I enabled "3rd party integration" after installing and the file was created. So it worked back then.

So yeah... the bug can obviously affect main vaults too and occurred somewhere between 4.2 and 4.4... not sure how helpful this info is.

It makes no sense to me that the file cannot be created, but that it will be maintained as long as it already exists, because that reveals that 1P4 knows the exact filesystem-path to use but yet it can't create the file itself... Sandboxing issue? File-creation permissions issue?

@MikeT Tagging since my report about May 4th may be useful to you.

@MikeT‌

The only good news is that I am able to reproduce this bug with 100% success.

• Opened Console and cleared the log.
• Launched 1Password and unlocked it.
• Cleared the log.
• Opened 1Password Preferences and unchecked "3rd party integration" and the bookmarks-default.json file vanished (but not the 3rd party integration folder).
• Nothing showed up in Console.
• Enabled "3rd party integration" and of course bookmarks-default.json did not appear.
• Nothing showed up in Console.
• I checked permissions on all folders under ~/Library/Application Support/1Password 4 and see that they are all owned by MyUsername:staff and all have read/write/execute access (as expected for folders).
• If I wait a few days bookmarks-default.json will probably show up again, like last time.

I am on Mavericks 10.9.2, fully up-to-date, and see this exact failed behavior on both a MacBook Pro 2010 and a Mac Pro 2012 (running the same OS version). Both are running the absolutely latest 1Password 4.4.1.BETA-1, but this bug appeared a few versions ago and went unnoticed by me since I wasn't fiddling with the "enable/disable 3rd party integration" checkbox back then.

Are you guys not seeing this at all? I see it across multiple machines. :/

Did uncover one small bug for you (attempting to draw something on a non-initialized context), as I saw this message immediately after unlocking:

5/10/14 3:36:56.236 PM 1Password 4[18372]: It does not make sense to draw an image when [NSGraphicsContext currentContext] is nil. This is a programming error. Break on void _NSWarnForDrawingImageWithNoCurrentContext() to debug. This will be logged only once. This may break in the future.

@Megan That's the best news I've heard since inventing a time machine, traveling back in time and telling myself some good news! In other words, that's some good news you're bringing there!