The state of sync on 1Password 5.1 (OS X) and 5.2 (iOS) (caught on video!!)

ravigupta

I switched to 1Password a few days ago. Super impressed by the fit and finish, I put in a few hours to input 190 credentials and 33 Software Licenses into it, and firmly said goodbye to Dashlane, my previous solution of choice.
All of this work was done on a MacBook Air. I then installed the iOS version and purchased the Pro pack to unlock full functionality. After a few hiccups, all items from my Mac started showing up in the iOS app.
Satisfied, I began using it in earnest, i.e. adding and removing information on-the-go, between 2 Macs (running OS X 10.10.2) and an iPhone (running iOS 8.1.3). I chose Dropbox as the syncing solution since it seemed a more robust choice than iCloud drive and I did not want to deal with any of the syncing problems developers usually claim to be having with Apple's solution.
To say that my experience has been abysmal would be an understatement. Sync seems to be fundamentally flawed. I don't know if this is limited to Dropbox, or does it affect iCloud Drive too. I am afraid to find out because I don't know how to safely move my data from Dropbox to iCloud Drive. But that is besides the point of this post. If AgileBits is selling an app with Dropbox as one of the syncing mechanisms, the current state of sync is unacceptable.
Since the preferred solution proposed by every member on the forum in such cases is to create a secure note on each of the devices and see the results, I decided to do exactly that. But this time I also create a video. As you can see, nothing gets synced automatically unless I "kickstart" it by modifying something on the device on which I want the latest results. This is an as-is recording with no post-processing.
You know what's worse? I have a MacBook Pro from work that was sitting right next to me all the time I was recording this video on my MacBook Air, with 1Password open in the foreground no less, screen set not to sleep, and it did not see one single change from the activity on the other 2 devices that appear in the video. In fact when I just checked, the Sync tab under preferences for this machine says "Last Sync: 45 minutes ago".
I mean come on guys, this is 2015, even Apple's own iCloud keychain sync works better than this.
To those who missed the link to the video, here it is again - https://www.dropbox.com/s/w7nakpta0efwrgm/2015-02-05_12-26-52.mp4?dl=0.

Ben

Hi @ravigupta,

Thanks for taking the time to write this out, and especially for going as far as to share a video. If a picture is worth a thousand words, a video is worth a million. :)

This appears to be functioning as intended. 1Password for iOS intentionally doesn't sync all the time to conserve both battery life and bandwidth. It only syncs when a change is made, or when the app is closed and re-launched. It will also sync if you manually go into the Settings > Sync menu and force a sync. This should not be required normal usage, but is a way to force changes that have made it to Dropbox to appear on the iOS device without relaunching the app or making a change to an item.

It sounds to me, on initial view, that the problem is more one of expectations not matching intentions, but if I've missed something here please let us know. :)

Thanks!

Ben

ravigupta

Thanks for the explanation. You completely missed the part where changes made on iOS did not reflect on OS X. I can understand the iOS app does not pull changes all the time, but what about push. And surely the OS X app has more than enough resources at hand to do a great job with syncing. That does not seem to be the case either. You also completely missed the part about the MacBook Pro not seeing any changes at all during this whole exercise. So yes, the question is definitely of expectations too. Expectation that the app that I rely on day-to-day work should sync reliably in a well defined fashion.
Also, if this is the intended behavior, why is sync touted as a tent-pole feature all over the site without an explanation as to how it actually works. I challenge you to post the video I made on your website among promotional material and see how many people actually buy this software after that.

Ben

You completely missed the part where changes made on iOS did not reflect on OS X

They did, though. When you moved the 1st test item to the Trash, the 2nd test item (that you created on iOS) showed up on the Mac. Because you made a change, a refresh was forced.

And surely the OS X app has more than enough resources at hand to do a great job with syncing.

Yes, because the Dropbox app runs in the background all of the time on the Mac syncing (to and from Dropbox) does happen nearly instantly. That still doesn't mean the changes have made it to/from iOS to/from Dropbox yet. I outlined the actions that cause a sync between iOS and Dropbox above.

If you were running two Macs side-by-side I think you'd see more of the behavior you were looking for here.

Also, if this is the intended behavior, why is sync touted as a tent-pole feature all over the site without an explanation as to how it actually works.

I'm not sure I follow. What sort of an explanation are you looking for?

I challenge you to post the video I made on your website among promotional material and see how many people actually buy this software after that.

Running both devices side by side and making changes on both nearly simultaneously wouldn't be common usage... I can't say as I've ever run into a scenario where I had the inclination to do that. If I'm in front of my Mac I try to do as much as possible on that... I even text from the Messages app instead of pulling the phone out. That isn't to say that iOS isn't very capable if I'm not in front of my Mac.

ravigupta

Well that is awesome. So let's take the iOS app completely out of the picture. Let's just focus on the OS X apps here. Here is another one for your viewing pleasure. As you can see, Dropbox is indeed updating files nearly instantaneously. The OS X client is not picking up those changes unless you make some random change to the client you expect to refresh. How is this good UX?
And please, before you compare the current state of syncing on 1Password to Messages app, try sending a picture to one of your contacts on the Mac while keeping the iOS client open on your phone. It's not even close. Which is to say you might not be using it that way, but the Messages app does do right by syncing.

Ben

And please, before you compare the current state of syncing on 1Password to Messages app

Sorry, I wasn't trying to make a comparison between them at all. Simply pointing out my usage patterns, and what I assume are fairly common usage patterns.

The OS X client is not picking up those changes unless you make some random change to the client you expect to refresh.

This is indeed working differently than I expected. I'll check with our developers on this and see what might be the cause/reasoning here.

Thanks!

ravigupta

And here is one more piece. A screenshot from the OS X client on MacBook Air which has not synced changes from the OS X client on MacBook Pro. Both machines are connected to Wi-Fi, set not to sleep, Dropbox running and up-to-date in the background. I don't know what is this thing waiting for to start syncing. Even quitting the app completely and opening it again does nothing. The only way to get to see the changes is to create something or delete something to trigger a refresh.

In my book, this is a bug, or super lame UX at best, unless this is another one of your this is how it is expected to work scenarios. If that is indeed the case, I would like to know what time should I expect between these syncs so I can validate that claim.

rickfillion

Hi @ravigupta,

Thanks for these videos that clearly demonstrate the issues you're experiencing. I think you're hitting an OS X bug that I myself hit just last week while working on sync. Changes from OS X to iOS can take time to get there, the mechanism for getting notified about those changes isn't super great. But OS X to OS X via dropbox should be nearly instant. Certainly as fast as Dropbox itself moves the data around + a couple seconds for us to process the changes.

The reason the changes show up as soon as you do an edit is that as soon as an edit takes place, it triggers a sync. The first part of sync is to first import remote changes, then export our local changes. The fact that you're seeing the changes means that the remote changes have made it to your local dropbox.

So what's broken is the mechanism that alerts the app that there are changes in the agilekeychain in the dropbox. This is actually really simple code (i.e. not a lot of room for bugs to hide in there). In general it works GREAT... but like I said, just last week I hit an issue where the symptoms were exactly what you're seeing here. I spent 5 hours chasing this, got most of the way towards totally recoding that notification mechanism. Eventually I got convinced that it was an OS X bug, and eventually found a workaround that got it back up and running. I'd love for you to do exactly what I did and see if this gets OS X behaving properly.

• Shut down 1Password completely on all devices that are syncing via via this agilekeychain file in dropbox.
• Open a Finder window, and navigate into ~/Dropbox/Apps/1Password/
• Rename 1Password.agilekeychain to 1Password.new.agilekeychain
• Rename 1Password.new.agilekeychain back to 1Password.agilekeychain
• Relaunch 1Password on the machines.

Let's see if that knocks some sense into it. It may feel like a bit of magic, and I'd totally agree if it wasn't for the fact that these steps got it up and running for me again. It wasn't just within 1Password that I couldn't notifications for changes in that directory, it was even in a sample app that I coded to help me better understand the problem. The system just wasn't giving me the notifications it's supposed to via the FSEvents system.

I really hope this helps. If it doesn't we'll dive into different things. We'll figure this out.

Rick

ravigupta

Thanks for the lucid explanation, acknowledgement and directions @rickfillion. Here is how I followed them:
0. Quit all browsers and 1Password mini and 1Password from MacBook Pro and Air.
1. On MacBook Pro, renamed 1Password.agilekeychain to 1Password.new.agilekeychain.
2. Waited for changes to sync on both OS X Dropbox clients.
3. On MacBook Pro, renamed 1Password.new.agilekeychain to 1Password.agilekeychain.
4. Waited for changes to sync on both OS X Dropbox clients.
5. On MacBook Air, renamed 1Password.agilekeychain to 1Password.new.agilekeychain.
6. Waited for changes to sync on both OS X Dropbox clients.
7. On MacBook Air, renamed 1Password.new.agilekeychain to 1Password.agilekeychain.
8. Waited for changes to sync on both OS X Dropbox clients.
9. Launched 1Password on both Pro and Air, keeping them in the foreground. Checked that both were in absolute sync, including the changes I saw missing before following these steps.
10. Observed that there are 3 notes in the Secure Notes category as expected.
11. Deleted a secure note (Note1) from MacBook Air and waited for Dropbox on MacBook Pro to notify. 5 mins after, 1Password on Pro still showed 3 notes.
12. Deleted a secure note (Note 3) from MacBook Pro. This also deleted the previously present Note 1 from MacBook Pro (since deletion of Note 3 triggered a sync.)
13. Waited for Dropbox client on Air to update. 15 mins in, the Air still has Note3 and Note2 while Pro has only Note2.

This is affecting iOS - Mac syncing too, since changes made on iOS apps are being pushed to Dropbox but the OS X client is not picking them up, making the iOS app look bad.

rickfillion

Hi @ravigupta,

Thanks for the update. I'm still suspecting the OS X bug. It's easier for us to get visibility into OS X so I propose that we first work on getting OS X sorted out then we'll figure out what's going on with iOS. Let's see if we can get confirmation of the bug, and if not maybe we'll get a better idea of what's going on. I just whipped up a little app that has the same code we use to get notifications from the dropbox modifications. It should give us a little more visibility into the problem.

The sync notification system when used via AgileKeychain/Dropbox works in a couple steps:

• Remote device makes a change, and writes it to disk within its local AgileKeychain file.
• Change gets uploaded to Dropbox
• Dropbox syncs this to another device
• Dropbox writes the change into the local AgileKeychain file
• OS X broadcasts that a write has happened. Anyone who's watching for FSEvents for a path that includes the written to path should get notified.
• 1Password locally gets notified.
• 1Password locally runs sync, which picks up changes to the local AgileKeychain.

It was the OS X broadcast that wasn't functioning right on my system. This app will test this out for us. Please download this sample app. It's not pretty, 20 minutes will only get you so much. :smile: Run it, and click the 'Choose Path' button. Select the AgileKeychain file.

Now I'd like you to do local changes within 1Password. This should trigger writes within the AgileKeychain file. If we're getting the notifications we should be seeing:

<2015-02-06 14:54:27 +0000> Notification for : ( "/Users/rfillion/Dropbox/1Password/demo/demo.agilekeychain/" )

If that works, then let's move to doing remote changes. On the other Mac do changes. The chain of events described above should happen. The changes should make their way to your local Mac eventually. Does the little app log a notification for it? Note that the little app may take up to 2 seconds to log the notifications it gets from the system.

If it does log a notification, but you don't see the changes within 1Password, then we should see if 1Password is logging anything in the console. To do that:

• Launch Console.app from /Application/Utilities
• Make sure 'All Messages' is selected on the left side bar
• Search for 'onepassword' in the top right search field which will filter out messages to be only ones from 1Password.
• Notice that the left side of each message has a timestamp. Do any of those timestamps roughly correspond to ones from the little app showing a notification? Can you copy/paste those here?

I'm really sorry that sync isn't working as well as it should for you. When sync doesn't work right, it really ruins the perception of 'magic'. I would love for us to get to the bottom of this.

Rick

ravigupta

Thank you @rickfillion for the hard work on getting this resolved. You are absolutely right, sync not working really takes away from the experience. Here is the video for local changes. Seems like the local notifications are not being generated. Please let me know if you still need the video for remote changes.

rickfillion

Hi @ravigupta,

It certainly sounds like you're running into the same issue I had. The renaming files step wasn't necessarily what fixed it here, it may have just been what I was doing around the time the system started to behave again, coincidentally. I've filed a bug with Apple about this (rdar://19765000). It'd be nice if they could provide us with a list of things we could provide them to help them chase this down. My experience filing bugs with them tells me that this is unlikely, unfortunately. So let's forge on ahead with the assumption that we're on our own here.

Let's start with the obvious : a reboot. You said that this was affecting both Macs though? Seems so weird/unlikely that this bug would be happening on two systems at the same time. Possible, I suppose, but just seems unlikely. Could you run the PathWatcher app, and point it to your desktop as opposed to the keychain directory? Then save a file onto your desktop. Let's see if we get that notification.

Could you create a new Folder in Finder next to your agilekeychain? Point the PathWatcher at that folder, then write a file into it?

Are you a fairly technical user? Have you used the Terminal before?

What kind of Mac are you using? Do you have more than one harddisk attached? Is Dropbox's contents on the main disk?

Looking forward to hearing from you.

Rick

ravigupta

Hi @rickfillion. I was getting tired of the sync issues between the two MacBooks and decided to see if iCloud Drive would prove better than Dropbox. It really did. I could see changes being made in one reflecting on the other one within a few seconds, without any intervention. However, this did not solve the OS X to iOS and vice versa sync issues.
I can revert to Dropbox sync if you are interested in following up on this bug. If you would like to switch focus to OS X - iOS syncing I am up for that too. To answer some of your questions yes I am a fairly technical user and have used the Terminal before. I have a MacBook Air 2012 Core i5 with 4 GB RAM and 256 GB SSD. There is an external storage of 4TB that is sometimes connected to the MacBook. Dropbox's contents are indeed on the main disk.

littlebobbytables

Hi @ravigupta,

We can't blame you for switching sync methods if iCloud is going to work for you. We may have found Rick another guinea pig to perform his tests but thank you for your offer, we may very well take you up on it but Rick will let you know.

If not, happy iCloud syncing!

ravigupta

Does that mean no investigation on iOS <-> OS X syncing issues with iCloud Drive?

Ben

iCloud Drive does not have an API for iOS, so we wouldn't be able to sync on iOS with iCloud Drive at this point. We do support CloudKit, which is the iCloud API Apple offers for apps such as ours. It just doesn't use iCloud Drive.

ravigupta

Thanks. I really don't care that much about what the underlying technology is as long as the iOS and OS X apps are able to sync in a well defined and reliable fashion. How do we go about investigating the anomalies in that.

rickfillion

Hi @ravigupta,

iCloud/CloudKit sync it is! I quite like our CloudKit sync. It's simpler than the others, the only unfortunate part is that it relies on the black box that is iCloud which we have little visibility into.

So in the case of CloudKit sync... the underlying mechanism that's going to be telling your iOS device about updates are Push Notifications. If you have anything on your phone that could be blocking those, make sure it's disabled.

There's an issue with CloudKit sync where in some cases I've seen it fail to register for these push notifications. If you create a Diagnostics Report on iOS, I should be able to see if that's the case. If you email that to support+forums@agilebits.com and include a link to this thread so that we can connect the dots, I should get notified when that comes in and can take a look.

The workaround for this bug (which I've got here locally as OPM-2827) is to disable sync, and to reconnect. Unfortunately there's no UI to tell the user that we encountered this issue. I suspect that this is what you're encountering, but I could be wrong. I'm happy to work through this with you.

I'm really sorry that you've so far had such a terrible time with sync. Thanks for your patience, we really appreciate it.

Rick

ref: OPM-2827