Entire Team Locked Out: "The server's time may be out of sync"
Hi Agilebits,
Preface: We've looked for a phone number and can't find it. For a mission-critical tool, phone support is extremely important.
We attempted to add 2FA via Duo, following the instructions in your guide. We entered the hostnames and integration/secret keys exactly as they were supplied by Duo, and have correctly added the 1Password application and users in Duo. After doing this, attempts to log in failed with "The server's time may be out of sync". Removing the application from Duo yields "Invalid integration".
All of these errors are blocking our entire team from logging in. We're at a dead stop and spending thousands per hour in lost productivity. What's going on here?
1Password Version: Web
Extension Version: Not Provided
OS Version: All
Sync Type: Not Provided
Comments
-
Hi browndav - I'm sorry for the troubles with Duo. I noticed you emailed us as well so let's continue the conversation via email. I'm going to ask one of my team members to look into the issue for you and we'll get right back to you via email as soon as possible. I apologize for any inconvenience.
ref: ZUK-99964-497
0 -
Hi,
Our original issue was absolutely related to time sync. We have screenshots of the message, multiple witnesses, and can absolutely prove it.
What complicated the issue was our attempt to disable Duo integration in response to the time sync issue – the error message changed to "invalid integration" once we did that. Further, the integration removal process seems to run on some sort of time delay or batch processing mode, so it took a couple hours to resolve. We now have access again, but the resolution was to simply wait.
Do you have a postmortem or any additional information available for the original time sync problem? We have many users who depend on the web interface and want to make sure we don't encounter it again.
Finally, do you plan to add support for any simpler TOTP clients like Google Authenticator?
0 -
Our original issue was absolutely related to time sync. We have screenshots of the message, multiple witnesses, and can absolutely prove it.
@browndav: Indeed, I don't have any problem believing that, as it's happened to me. wkleem 's issue, on the other hand, resulted from not being able to authenticate with Duo after wiping the phone being used for authentication, so I do think it's important to keep these discussions separate to avoid confusion.
What complicated the issue was our attempt to disable Duo integration in response to the time sync issue – the error message changed to "invalid integration" once we did that. Further, the integration removal process seems to run on some sort of time delay or batch processing mode, so it took a couple hours to resolve. We now have access again, but the resolution was to simply wait.
I wouldn't want that to be anyone's takeaway from this, as you totally did the right thing by getting in touch. It really depends on the situation, so it's always best to reach out immediately for assistance. That's what we're here for, to help avoid further issues when anyone runs into trouble, and get you back up and running.
Do you have a postmortem or any additional information available for the original time sync problem? We have many users who depend on the web interface and want to make sure we don't encounter it again.
I just responded to you via email not realizing you'd followed up here. But you raise a good point: others may have similar questions, so I'll share some of my reply here as it applies generally to one-time password (though of course with nothing specific to your situation, as this is a public forum):
[I've personally had issues] with one-time passwords in the past, and we do hear similar reports with regard to other sites at times.
There are effectively two factors (no pun intended) involved with one-time password authentication: the device generating the code, and the server validating it (or not). The one-time passwords system has some leeway built in to compensate for time drift, as the chances of the times being synchronized exactly on both ends and transit time for communication between them are pretty slim.
From here it gets into things specific to Duo's implementation, so you'd have to check with them for specifics, but since Duo is doing the authentication, 1Password isn't actually involved at that point, so even in the case of a small variance with our server, when you're prompted to authenticate, you're needing to do so directly with Duo.
Generally one-time password setups allow codes to "live" for 30-90 seconds, so even if your device is "slow" compared to the server time, as long as you're not more than a minute or so behind, the code will be excepted before it fully expires. Most devices just don't synchronize the time that frequently, and there may be a delay introduced during synchronization, causing the time to be slightly off. Finally, between checks, the time needs to be tracked accurately, and nothing is perfect in this regard, so it can get further out of sync over time, just like a watch.
[...] As far as mitigation, in the past I and others have manually checked and/or configured date/time settings on the affected device(s) to get it working. It was never clear if this was a hardware or software problem, though I've noticed that more reports come from folks trying to authenticate with Wi-Fi-only iPads. I suspect that the consistent cell network connection helps iPhones and cellular iPads maintain a more consistent time sync, but I don't have any hard evidence to support that, just my own experience and helping others troubleshoot.
Account recovery performed by a team (or family) member is generally a good option if a single user gets locked out.
Finally, do you plan to add support for any simpler TOTP clients like Google Authenticator?
It's something we can consider for the future, but most companies asking for multi factor authentication really have one particular flavour in mind, and Duo offers a wide variety of options. And of course TOTP (as Google Authenticator uses) also depends on the client (your browser) and server (Duo authentication) being in sync. So while it's simpler, it isn't able to solve the problem. We're always exploring different possibilities though, so it helps to know you'd prefer that!
0 -
Two more questions that might help our understanding of this:
(i) When that message appeared, which clocks were out of sync? Is it the client browser's clock vs. Duo's clock? The browser's clock vs. 1Password's clock? Some other combination? The message appeared before we're ever prompted for any time-based login information from the Duo app. Is there some other clock sync process I'm not understanding? We're running NTP here on all devices and definitely had correct clocks. We observed the problem in the PDT time zone (-0700 UTC), if that helps (probably not relevant).
(ii) Does that time sync error message originate from a 1Password server, or from the Duo integration (e.g. in a post-1Password-auth iframe served by Duo, or something like that?)
TIA
ref: ZUK-99964-497
0 -
@browndav: Thanks for following up! Good questions:
(i) When that message appeared, which clocks were out of sync? Is it the client browser's clock vs. Duo's clock? The browser's clock vs. 1Password's clock? Some other combination? The message appeared before we're ever prompted for any time-based login information from the Duo app. Is there some other clock sync process I'm not understanding? We're running NTP here on all devices and definitely had correct clocks. We observed the problem in the PDT time zone (-0700 UTC), if that helps (probably not relevant).
The time zone shouldn't matter, but it seems that sometimes it can if it's been set incorrectly. For example, the system time may be set correctly but if the date and/or zone are off what appears correct to the user won't be accurate with regard to UTC, which is what's being used under the hood. For example, the clock on my Mac here shows only the local time, but it's being calculated as UTC+/-zoneoffset.
Going back to the first part of the question, it's the client (your device) and server (Duo authentication) which must match. Your browser (which is transmitting the authentication attempt) and authenticator (whatever is generating the code for you) are probably just using the system time. I suppose it's possible that the Duo app may be performing its own NTP check, though you'd have to check with Duo on that.
(ii) Does that time sync error message originate from a 1Password server, or from the Duo integration (e.g. in a post-1Password-auth iframe served by Duo, or something like that?)
Duo, since that's the server performing authentication. 1Password.com doesn't perform this function, which is why we've added (beta) support for Duo authentication. I've generally had good experiences with Duo myself, but it's always frustrating when something outside of our control causes issues for a 1Password customer. :(
ref: b5-2672
0 -
An update for anyone reading this in the future:
(copied from my message on another thread)
In our last deploy we released a feature where disabling the Duo-1Password integration by deleting or changing API keys from Duo's administration panel will automatically disable the Duo integration on your 1Password account to prevent you from getting locked out. This can help restore access to your 1Password account due to any Duo-related problems, like local clocks being out of sync, or whatever the issue might be: you delete the integration from within Duo's admin panel and we will detect that and disable Duo from your account. Note that Duo is not disabled if we cannot reach Duo due to network errors, clock sync, or other problems, only in the cases of deleted integration, or resetting the Duo-1Password integration API keys.This can be a great way to disable Duo account protection in a lock-out situation.
Steven
0
