Renewing Let's Encrypt certificate in AKS

flammable
flammable
Community Member
edited May 2022 in SCIM Bridge

Hello 1Password Support -

I'm taking over management of our 1Password SCIM bridge from a previous employee who left our organization. I'm kinda new to Kubernetes, but I'm fairly comfortable with Docker and Docker Compose, so this has been an interesting learning experience!

Using these instructions, the previous employee deployed the SCIM bridge using Kubernetes in Azure. So far, everything has worked great, except the Let's Encrypt certificate just doesn't seem to be renewing itself.

A couple of months ago, we updated the SCIM bridge from 2.2.1 to 2.3.0 - we had noticed the certificate expired, and since the SCIM bridge was out of date, it seemed like a good time to take care of both issues. It looked like the lack of certificate renewal might have been a bug that was resolved with the SCIM bridge. However, we found the certificate still didn't auto-renew (and was due to expire this week), so we updated to 2.3.1 this afternoon. This generated a new certificate, which gives us some time to determine what's not working properly.

I noticed in the installation instructions, this note is included:

If you use Azure Firewall, open ports 80 and 443 for your Azure Kubernetes cluster. The Let’s Encrypt service uses port 80 to renew the SSL certificate every 60 days. All other SCIM bridge traffic uses port 443.

We don't use Azure Firewall, but I noticed that the yaml files for the SCIM bridge container only open port 443, so I was wondering if that could be related. The file 'op-scim-service.yaml' includes a commented section for opening port 80 to the container, but I wanted to ask if that would be the right approach or not - we're not using a reverse proxy, so I don't want to open our SCIM bridge to anything that could be dangerous. Plus, I don't know if that'd actually solve the Let's Encrypt issue.

Please let me know if I forgot any details that'd be useful in troubleshooting this. Thanks for the help!

Mike

EDIT: I forgot - I wanted to post this, which shows port 80 is not currently open:

% kubectl get services
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)         AGE
kubernetes       ClusterIP      <redacted>     <none>          443/TCP         343d
op-scim-bridge   LoadBalancer   <redacted>     <redacted>      443:31887/TCP   343d
op-scim-redis    ClusterIP      <redacted>     <none>          6379/TCP        343d

1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided

Comments

  • graham_1P
    graham_1P

    Hi @flammable,

    Thanks for the detailed and well outlined message!

    Excitingly for your firewall, that installation instruction is out of date. As of version 2.2.0 of the SCIM bridge we moved to the TLS-ALPN-01 challenge for Let's Encrypt, meaning the challenge can happen with TLS enabled, rather than just on an unencrypted connection. Therefore it is fine to just have 8443/443 open on a v2.2.0+ bridge. I will file an issue to update the documentation.

    Regarding the failure to update, it is plausible you may be impacted by the mass LetsEncrypt TLS-ALPN-01 revocation from late January 2022. Specifically, a LetsEncrypt renewal dependency used in v2.3.0 would fail in some scenarios to auto-renew for TLS-ALPN-01 challenges. We updated that dependency for v2.3.1 (issue #1947) to fix that issue. Please try updating to v2.3.1 to see if it fixes your issue. I apologise the changelog is not more detailed on that point.

    If that does not resolve your renewal issue, I would recommend reaching out to our support team for more personalised support to dig into the details. Then we can leverage non-public details to help resolve your issue.

    Let me know what further questions you have!

    Graham

  • flammable
    flammable
    Community Member

    Hi @graham_1P,

    Thanks! That's excellent news. I'm really glad we don't need to open port 80. I checked back on the timelines:

    • On January 24th, we started receiving emails that our SCIM bridge (running version 2.2.1) wasn't working. The certificate had expired, so communication was failing.
    • On January 28th, my colleague updated the SCIM bridge to version 2.3.0, which generated a new certificate.
    • On April 22nd, I noticed the certificate was due to expire on May 5th, which was less than 30 days away - per the documentation, the SCIM bridge attempts to renew Let's Encrypt certificates after 60 days, so I assumed the renewal process hadn't been working.
    • On May 4th, I updated the SCIM bridge to 2.3.1, which generated a new certificate. This certificate doesn't expire until August 2nd.

    If I'm reading the Bugzilla thread correctly, we somehow might have dodged the entire issue. Our old certificate expired before the bug was found on January 25th, and our new certificate was generated before the certificate revocation on January 30th. That said, we were running 2.3.0 until this week, so it's possible that's why the certificate didn't renew last month.

    Since our current certificate doesn't expire until August, I'll keep an eye on things and reach out to support in mid-July if it doesn't auto-renew itself before then. I really appreciate the help with this!

    Also, thanks for the fantastic documentation! I absolutely wouldn't have been able to update our SCIM bridge without the detailed instructions. :)

    Mike

    P.S. - Is there a good way to keep up with SCIM bridge updates? The 1Password admin console will show if there's a pending update on the "Integrations" page, but the bridge itself doesn't notify us. Would it be possible to add an RSS feed to the release notes page, or set up a mailing list?

  • Hass.Khalid_1P
    Hass.Khalid_1P

    Hi @flammable

    Glad to hear that upgrading the SCIM bridge 2.3.1 has resolved your initial issue.

    Regarding the SCIM bridge updates, I will create an issue to add release notification subscription. As you suggested adding a RSS feed to the release notes page would be a good place to start.

    Let me know if you have any further questions.

    Hass

  • DougWalker
    DougWalker
    Community Member

    Just to add to this and save starting a new thread

    We also saw this issue, we noticed that new users were not being provisioned in 1Password and the SSL certificate had expired when visiting our SCIM bridge webpage. The bearer token continued to work but testing the connection on Azure Provisioning flagged an error.

    We noticed the update available for the SCIM bridge and have updated following the instructions (which was flawless) and everything is working again.

    I assume we got caught in the 2.3.0 issue and from here on it should auto renew, we are setting up further monitoring on the SSL cert anyway.

    Also second a notification for when a SCIM bridge update is available, we wouldn't have seen this update had it not been for the failed user syncs.

This discussion has been closed.