Users in Okta to 1password groups not syncing

varun118 · November 2021

{"level":"info","version":"2.1.0","build":"201001","application":"op-scim","component":"SCIMServer","request_id":"c5sli21dq3sf0bdhs7v0","group":"i7xsp2dz3y4utwllvufz7hx5kq","time":"2021-10-27T13:50:00Z","message":"group found"}
{"level":"info","version":"2.1.0","build":"201001","application":"op-scim","component":"SCIMServer","request_id":"c5sli21dq3sf0bdhs7v0","group":"i7xsp2dz3y4utwllvufz7hx5kq","group":"i7xsp2dz3y4utwllvufz7hx5kq","user":"NF2HGT7Y5FBUZEGH53II5KM47Q","time":"2021-10-27T13:50:00Z","message":"user not found"}

This does not pickup actual user id
It picks up the group id instead of user id

We are using 2.1.0 and tried to upgrade the scimbridge to 2.2.0 and 2.2.1 but we have seen errors related to this new feature
Moved to TLS-ALPN-01 challenge for Let's Encrypt, and improved Let's Encrypt reliability. {858}

We have built a SCIMBRIDGE container on top of EC2 instance.

Need help on this

Thanks
Varun

1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided

Chas_1P · November 2021

Hi @varun118 ,

I'm sorry you're experiencing these issues. I'm looking into this with the team.

In the meantime I'm hoping you could answer a couple questions. Are you saying you have only started seeing these issues after trying to upgrade the SCIM bridge? The changelog you mentioned is referencing a feature introduced in 2.2.0 but that log line showing the error is running 2.1.0. What steps did you take prior to encountering the error?

What errors are you seeing that make you think the Let's Encrypt changes are related?

Thanks for posting, hoping to get all the issues resolved quickly.
Chas

varun118 · November 2021

Hi
these are errors which we notice when we upgrade to 2.2.0

6:33AM ERR failed to GetTLSConfig, retrying after backoff delay error="Network: (could not obtain Let's Encrypt certificate), 1password-scim.internal.icims.io: obtaining certificate: [1password-scim.internal.icims.io] Obtain: [1password-scim.internal.icims.io] solving challenges: 1password-scim.internal.icims.io: no solvers available for remaining challenges (configured=[tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[http-01 dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/264427320/36502081140) (ca=https://acme-v02.api.letsencrypt.org/directory)" application=op-scim build=202001 version=2.2.0
6:33AM INF TLS attempting to acquire certificate application=op-scim build=202001 domain=1password-scim.internal.icims.io version=2.2.0
6:33AM ERR failed to GetTLSConfig, retrying after backoff delay error="Network: (could not obtain Let's Encrypt certificate), 1password-scim.internal.icims.io: obtaining certificate: [1password-scim.internal.icims.io] Obtain: [1password-scim.internal.icims.io] solving challenges: 1password-scim.internal.icims.io: no solvers available for remaining challenges (configured=[tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[http-01 dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/264427320/36502102450) (ca=https://acme-v02.api.letsencrypt.org/directory)" application=op-scim build=202001 version=2.2.0
6:33AM INF TLS attempting to acquire certificate application=op-scim build=202001 domain=1password-scim.internal.icims.io version=2.2.0
6:33AM ERR failed to GetTLSConfig, retrying after backoff delay error="Network: (could not obtain Let's Encrypt certificate), 1password-scim.internal.icims.io: obtaining certificate: [1password-scim.internal.icims.io] Obtain: [1password-scim.internal.icims.io] solving challenges: 1password-scim.internal.icims.io: no solvers available for remaining challenges (configured=[tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[http-01 dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/264427320/36502169590) (ca=https://acme-v02.api.letsencrypt.org/directory)" application=op-scim build=202001 version=2.2.0
6:35AM INF TLS attempting to acquire certificate application=op-scim build=202001 domain=1password-scim.internal.icims.io version=2.2.0
6:35AM ??? [ERROR] TLS-ALPN challenge server: handshake: no certificate available for '172.18.0.3' application=op-scim build=202001 version=2.2.0
6:35AM ERR failed to GetTLSConfig, retrying after backoff delay error="Network: (could not obtain Let's Encrypt certificate), 1password-scim.internal.icims.io: obtaining certificate: [1password-scim.internal.icims.io] Obtain: [1password-scim.internal.icims.io] solving challenges: 1password-scim.internal.icims.io: no solvers available for remaining challenges (configured=[tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[http-01 dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/264427320/36502548970) (ca=https://acme-v02.api.letsencrypt.org/directory)" application=op-scim build=202001 version=2.2.0

Port 80 is open and is listening

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1029/sshd
tcp6 0 0 :::80 :::* LISTEN 6727/docker-proxy-c
tcp6 0 0 :::22 :::* LISTEN 1029/sshd
tcp6 0 0 :::3002 :::* LISTEN 6740/docker-proxy-c
tcp6 0 0 :::443 :::* LISTEN 6708/docker-proxy-c
udp 0 0 0.0.0.0:68 0.0.0.0:* 821/dhclient
udp 0 0 127.0.0.1:323 0.0.0.0:* 546/chronyd
udp6 0 0 ::1:323 :::* 546/chronyd

Chas_1P · November 2021

Hi @varun118 ,

We haven't been able to reproduce the Let's Encrypt issues you are seeing, even on v2.2.0.

Just to clarify:

On 2.1.0, you noticed Okta issues, so you attempted to upgrade to 2.2.x. But on 2.2.x, Let's Encrypt is now failing, correct?

Are you using any sort of HTTPS rewrite functionality in your AWS DNS? We've seen Cloudfare DNS cause some problems before, but your set up looks ok from what you listed. My other thought is that perhaps you have run into a rate limit with attempting to acquire a certificate for your domain.

We will continue to investigate and get back to you as quickly as we can.

fdietrich · November 2021

We have a similar issue. We deployed a SCIM test bridge in Azure Kubernetes and receive below error. Public IP allocated, DNS zone available, port 80 opened. Any idea or solution identified?

ERR failed to GetTLSConfig, retrying after backoff delay error="Network: (could not obtain Let's Encrypt certificate), tst1pscim-dns-b06af904.hcp.germanywestcentral.azmk8s.io: obtaining certificate: [tst1pscim-dns-b06af904.hcp.germanywestcentral.azmk8s.io] Obtain: [tst1pscim-dns-b06af904.hcp.germanywestcentral.azmk8s.io] solving challenges: tst1pscim-dns-b06af904.hcp.germanywestcentral.azmk8s.io: no solvers available for remaining challenges (configured=[tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01]remaining=[http-01 dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/268659220/37271842900) (ca=https://acme-v02.api.letsencrypt.org/directory)" application=op-scim build=202011 version=2.2.1

varun118 · November 2021

we have spunned up a docker container with scimbridge on ec2 instance which has a ELB with aws certs on it
but there are no specific redirection rules present on it

varun118 · November 2021

Hi

Do you have any update on this

Thanks

DeVille_1P · November 2021

Hi @varun118.

Thank you for providing the additional information. You mentioned that you have port 80 open but I wanted to ask if you have port 443 open?

With the release of version 2.2.0 (and later) we have moved to a TLS-ALPN-01 challenge for Let's Encrypt. This means that a direct connection using port 443 is possible, which is one of the main advantages. This means that port 80 is no longer required for obtaining a certificate from Let's Encrypt.

DeVille_1P · November 2021

Hi @fdietrich.

Thanks for sharing the error log.

Could you also check your configuration to ensure that port 443 is open?

varun118 · November 2021

Hi @DeVille_1P both port 80 and 443 were open
but still having this issue.

fdietrich · November 2021

Hi @DeVille_1P the ports were opened. In our case the problem was with the DNS. We we're using as fqdn and DNS record the kubernetes API generated from Azure, but it required an additional custom domain to be registered.

DeVille_1P · November 2021

Hi @varun118.

Thank you for confirming the open ports.

Based on your earlier message it sounds like you are running the SCIM bridge behind a load balancer (ELB) that is already doing TLS termination (using an AWS certificate). If this is the case then you will not need the SCIM bridge to obtain its own certificate.

You can override the default behaviour of the SCIM bridge by setting the following environment variables for the SCIM bridge:

OP_LETSENCRYPT_DOMAIN to an empty string (""). This will prevent the SCIM bridge from trying to obtain a certificate from Let's Encrypt.
OP_PORT to the port you have configured the load balancer to use to forward traffic to the SCIM bridge, such as 80 or 8080 for example. This changes the listening port of the SCIM bridge when not using LetsEncrypt TLS, and it defaults to 3002.

Note that both of these variables can also be passed to the SCIM bridge as command line arguments, --letsencrypt-domain and --port respectively.

DeVille_1P · November 2021

Hi @fdietrich. I'm happy that you managed to solve the issue and thank you for sharing your solution.

varun118 · November 2021

Hi @DeVille_1P we have finally upgraded it to 2.2.1 by adding empty --letsencrypt-domain and --port values respectively.
Scim-bridge is up and running and on latest version.
Health also looks good.

But we were unable to solve the main issue "Okta groups not syncing with 1password user groups"
We tried to make changes to the group by adding new users and pushed
in the UI it says "Automated User Provisioning updated the group okta-1password-cs 9:52 pm"
but no changes are reflecting in the 1password user group

Group id "i7xsp2dz3y4utwllvufz7hx5kq"

and these are the logs

Need some help on this
let me know if you need any further details

Hass.Khalid_1P · November 2021

Hi @varun118. It's good to hear you were able to upgrade the SCIM bridge to v2.2.1. In order to look into this issue further, we need some more details about your account and a complete log. Can I please ask you to contact us at our support email so you can provide these to us? You can find the contact form here, https://support.1password.com/contact. Please let me know once you have done that and we can expedite the support process from our end. Thank you!

Users in Okta to 1password groups not syncing

Comments