Users in Okta to 1password groups not syncing

We have a similar issue. We deployed a SCIM test bridge in Azure Kubernetes and receive below error. Public IP allocated, DNS zone available, port 80 opened. Any idea or solution identified?

ERR failed to GetTLSConfig, retrying after backoff delay error="Network: (could not obtain Let's Encrypt certificate), tst1pscim-dns-b06af904.hcp.germanywestcentral.azmk8s.io: obtaining certificate: [tst1pscim-dns-b06af904.hcp.germanywestcentral.azmk8s.io] Obtain: [tst1pscim-dns-b06af904.hcp.germanywestcentral.azmk8s.io] solving challenges: tst1pscim-dns-b06af904.hcp.germanywestcentral.azmk8s.io: no solvers available for remaining challenges (configured=[tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01]remaining=[http-01 dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/268659220/37271842900) (ca=https://acme-v02.api.letsencrypt.org/directory)" application=op-scim build=202011 version=2.2.1

Hi @varun118 ,

We haven't been able to reproduce the Let's Encrypt issues you are seeing, even on v2.2.0.

Just to clarify:

On 2.1.0, you noticed Okta issues, so you attempted to upgrade to 2.2.x. But on 2.2.x, Let's Encrypt is now failing, correct?

Are you using any sort of HTTPS rewrite functionality in your AWS DNS? We've seen Cloudfare DNS cause some problems before, but your set up looks ok from what you listed. My other thought is that perhaps you have run into a rate limit with attempting to acquire a certificate for your domain.

We will continue to investigate and get back to you as quickly as we can.

Hi
these are errors which we notice when we upgrade to 2.2.0

6:33AM ERR failed to GetTLSConfig, retrying after backoff delay error="Network: (could not obtain Let's Encrypt certificate), 1password-scim.internal.icims.io: obtaining certificate: [1password-scim.internal.icims.io] Obtain: [1password-scim.internal.icims.io] solving challenges: 1password-scim.internal.icims.io: no solvers available for remaining challenges (configured=[tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[http-01 dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/264427320/36502081140) (ca=https://acme-v02.api.letsencrypt.org/directory)" application=op-scim build=202001 version=2.2.0
6:33AM INF TLS attempting to acquire certificate application=op-scim build=202001 domain=1password-scim.internal.icims.io version=2.2.0
6:33AM ERR failed to GetTLSConfig, retrying after backoff delay error="Network: (could not obtain Let's Encrypt certificate), 1password-scim.internal.icims.io: obtaining certificate: [1password-scim.internal.icims.io] Obtain: [1password-scim.internal.icims.io] solving challenges: 1password-scim.internal.icims.io: no solvers available for remaining challenges (configured=[tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[http-01 dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/264427320/36502102450) (ca=https://acme-v02.api.letsencrypt.org/directory)" application=op-scim build=202001 version=2.2.0
6:33AM INF TLS attempting to acquire certificate application=op-scim build=202001 domain=1password-scim.internal.icims.io version=2.2.0
6:33AM ERR failed to GetTLSConfig, retrying after backoff delay error="Network: (could not obtain Let's Encrypt certificate), 1password-scim.internal.icims.io: obtaining certificate: [1password-scim.internal.icims.io] Obtain: [1password-scim.internal.icims.io] solving challenges: 1password-scim.internal.icims.io: no solvers available for remaining challenges (configured=[tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[http-01 dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/264427320/36502169590) (ca=https://acme-v02.api.letsencrypt.org/directory)" application=op-scim build=202001 version=2.2.0
6:35AM INF TLS attempting to acquire certificate application=op-scim build=202001 domain=1password-scim.internal.icims.io version=2.2.0
6:35AM ??? [ERROR] TLS-ALPN challenge server: handshake: no certificate available for '172.18.0.3' application=op-scim build=202001 version=2.2.0
6:35AM ERR failed to GetTLSConfig, retrying after backoff delay error="Network: (could not obtain Let's Encrypt certificate), 1password-scim.internal.icims.io: obtaining certificate: [1password-scim.internal.icims.io] Obtain: [1password-scim.internal.icims.io] solving challenges: 1password-scim.internal.icims.io: no solvers available for remaining challenges (configured=[tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[http-01 dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/264427320/36502548970) (ca=https://acme-v02.api.letsencrypt.org/directory)" application=op-scim build=202001 version=2.2.0

Port 80 is open and is listening

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1029/sshd
tcp6 0 0 :::80 :::* LISTEN 6727/docker-proxy-c
tcp6 0 0 :::22 :::* LISTEN 1029/sshd
tcp6 0 0 :::3002 :::* LISTEN 6740/docker-proxy-c
tcp6 0 0 :::443 :::* LISTEN 6708/docker-proxy-c
udp 0 0 0.0.0.0:68 0.0.0.0:* 821/dhclient
udp 0 0 127.0.0.1:323 0.0.0.0:* 546/chronyd
udp6 0 0 ::1:323 :::* 546/chronyd

Hi @varun118 ,

I'm sorry you're experiencing these issues. I'm looking into this with the team.

In the meantime I'm hoping you could answer a couple questions. Are you saying you have only started seeing these issues after trying to upgrade the SCIM bridge? The changelog you mentioned is referencing a feature introduced in 2.2.0 but that log line showing the error is running 2.1.0. What steps did you take prior to encountering the error?

What errors are you seeing that make you think the Let's Encrypt changes are related?

Thanks for posting, hoping to get all the issues resolved quickly.
Chas