Folder sync completely broke recently

JC_Denton

Here's my sync setup:
A MacBook Pro ("Mac 1") that serves as my master instance of 1Password. This Mac runs the WiFi sync server and also provides a folder sync out to ~/blah/path. There are two iOS devices, an iPhone and an iPad. Both use WiFi sync. There's another Mac, a MacBook ("Mac 2"), that gets its vault sync via "rsync -avE --delete mac1:/Users/me/blah/path/ ~/blah/path/". That machine's ~/blah/path is also a Folder Sync location (on that Mac). There's also a Windows computer with 1Password for that I'll occasionally manually copy vaults to, but never the other way around.

What happened today:
The vaults were slightly out of sync with each other. I had a burst of activity on Mac 2 and did some work today on Mac 1. Since Mac 1 is the master, I rsync'd Mac 2's folder sync location to Mac 1. Mac 1 got the updated and added passwords from Mac 2 while still retaining its new and updated passwords. So far, so good! I now proceeded to sync Mac 1 to Mac 2 so it could get the new accounts I had recently added. Despite syncing several times, the new items never appeared. I eventually ended up turning off folder sync on both Macs and starting things anew. That seemed to work, except on one of the iOS devices. It's WiFi sync steadfastly refused to update. I had to disable and re-enable the sync to get the new data loaded.

This sync strategy has served me well for a couple years now, so I was very surprised to see it break today. I guess my biggest question is, is there any way to avoid this problem in the future? For now, I think I can write it off as one-off. Maybe there's something I should be doing better though. Sadly, I am not interested in using Dropbox or iCloud sync. Despite being encrypted, I prefer to keep my vault data as "local" as possible.

Any insights are greatly appreciated :)

---

1Password Version: 6.6.1
Extension Version: 4.6.3.90
OS Version: macOS 10.12.3
Sync Type: Folder Sync

firebeyer

Hey @JC_Denton,

Interesting sync system you have. I don't really want to truly speculate why this happened without seeing any rsync logs. However, in response to "is there any way to avoid this problem in the future?" Have you ever tried unison? I believe it might be more useful than rsync for your situation as it can handle updates to both replicas instead of being unidirectional. I've used unison to handle changes across multiple unix systems with great success, but if you do want to give it a try get a solid backup of your data while getting it configured.

I would be interested to know if anyone else has a better solution than unison but it's the best I've used to date. If you want some assistance with unison I would be glad to help.

Cheers,
Andrew

JC_Denton

Hey Andrew,

My rsync logs are fairly unremarkable compared to previous successful transfers. So I don't think the fault lies with rsync.

Do you know how unison handles binary files? I'd be worried about it clobbering the .attachment files in the Folder Sync directories.

Cheers!

firebeyer

@JC_Denton,

It's an interesting question.

Normally the way it works is unison keeps track of any files that have changed in both replicas. As long as the same file hasn't changed on both ends then there will be no conflict as it will just replace the file on either side. Running unison as frequently as possible, preferably using cron or it's watch system usually keeps things very clean.

However, since it doesn't look like you can modify attachments within 1Password, if you were to change the attachment and copy it back into 1Password it results in a different file name.

A bigger concern while I was testing a few things and reading here is that with only 16 band files if you update a different password on both sides that fall within the same band it will cause a conflict.

So I think my suggestion might have been a bad one sadly. :blush: I'll play with it some more but you should probably forget everything I said, might not be any better than what you have going now.

Cheers,
Andrew

EDIT: This method probably would have worked when "The Agile Keychain format used a separate file for each item." However, that not how it is done anymore.

firebeyer

@JC_Denton,

I messed up and somehow deleted my long winded comment about how I don't actually believe unison would be a good method for you.

Since the OPVault design uses band files now instead of individual files for each item, if you updated a item in the same band on both replicas before unison synced it will cause a conflict. So my advice upon further research isn't so helpful.

Sorry about that,
Andrew

rickfillion

Hi @JC_Denton,

You're right to worry about clobbering files with a 'smarter' sync system. The band file conflict scenario @firebeyer mentioned is a very real one. The conflict file handling is one of the major reasons we recommend Dropbox and iCloud. Both systems provide ways to have us resolve conflicts.

What you described seemed like a reasonable sync setup (though definitely a more complex one than most users :) ). I can't think of anything that I would change to achieve what you're looking for.

Rick