Kubernetes Proxy Support / 1Password Connect Error 500

zhorvath_globys
zhorvath_globys
Community Member
in Secrets Automation

Hi Folks,

I'm evaluating 1Password Connect against Hashicorp Vault for a Kubernetes product and am running into a bit of a hiccup in our environment. I'm running Kubernetes v 1.21 on-prem with Rancher v2.5, and our network model requires a http proxy for outbound communication to the internet. Typically how I do this is I use a PodPreset to create "http_proxy" and "no_proxy" (and the capitalized versions thereof) environment variables in every pod that's created in a namespace. I've confirmed that my proxy variables exist on the 1Password Connect and Operator pods because I can see the output of env in the Connect pod, and Rancher tells me it exists in the Operator pod. I checked the proxy server and confirmed there was no incoming requests for 1password.com or onepassword.com.

What leads me to believe it's a networking issue is because when I create a OnePasswordItem using the vault and item ID, I get the following log outputs:

connect-api

{"log_message":"(I) GET /v1/vaults/xxxxxxxx/items/abcdefg123456 completed (500: Internal Server Error)","timestamp":"2021-04-19T23:56:25.850031751Z","level":3,"scope":{"request_id":"ed1025a1-f5c5-4f5e-a87b-3f251a865e71","jti":"mvz2vukz7fn4co6tfvnetdc6gi"}}

connect-sync

{"log_message":"(I) token recieved, getting credentials and initializing API","timestamp":"2021-04-19T23:56:25.702484532Z","level":3}
{"log_message":"(I) unable to get credentials and initialize API","timestamp":"2021-04-19T23:56:25.848687723Z","level":3}
{"log_message":"(I) ### syncer credentials bootstrap ### ","timestamp":"2021-04-19T23:56:25.849152954Z","level":3}

operator

{"level":"error","ts":1618876603.117083,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"onepassworditem-controller","request":"default/test-secret","error":"Failed to retrieve item: Unable to retrieve item. Receieved \"500 Internal Server Error\" for \"/v1/vaults/pwdyfwdtzwr62xohxihb7ks2pa/items/xgnylw6m6etr5dm7izd2xieg54\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/workspace/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}

For deployment I used the Helm charts available here: https://github.com/1Password/connect-helm-charts. My understanding is that without proxy support I can't even use the Vault plugin since the dependency is still on the Connect server being able to talk out.

Thanks a lot for your time!
Zack

1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

Comments

  • James_1P
    James_1P

    Hi,

    Thanks for the feedback! You are correct that it is a networking issue. The API is reporting a 500 because it was unable to perform an initial sync so there is no data available to retrieve.

    We discovered that the connect-sync container is unfortunately squashing the specific networking error that is likely causing your outbound proxy headache. It is being hidden in this log line:

    {"log_message":"(I) unable to get credentials and initialize API","timestamp":"2021-04-19T23:56:25.848687723Z","level":3}

    We will be doing a patch release for the containers soon with better logging for this error.

    The containers will both attempt to make an outbound https connection to your accounts sign-in address (i.e. .1password.com) for communicating with 1password.com.

    • connect-api will communicate to create/edit items and to report Item use.
    • connect-sync is responsible to syncing your selected vaults into your Kubernetes cluster.
  • zhorvath_globys
    zhorvath_globys
    Community Member

    Thanks for the response!

    It sounds like the containers do support proxy configurations then, is that correct? While I wait for the patch release should I try any other method of passing http variables to the containers?

  • James_1P
    James_1P

    I just did a quick test and confirmed that the containers do support proxy configurations. All communications with 1password.com by the containers are over https so it will require setting the https_proxy environment variable. With that set you should see the containers properly make requests to <something>.1password.com in your proxy server logs.

  • zhorvath_globys
    zhorvath_globys
    Community Member

    Hi James,

    Sorry, I took a few days off :) I was able to get this working!

    After doing some fudging around I managed to snag these from my proxy server once I got the environment variables set up correctly:

    TCP_MISS/503 5065 GET http://onepassword-connect:8080/v1/vaults? - HIER_NONE/- text/html

    I had to add "onepassword-connect" to my NO_PROXY envvar.

  • James_1P
    James_1P

    Thanks for the followup! I totally neglected to mention that the operator would communicate with the connect pod inside the cluster because I misread the original ask and assumed your proxy was only intercepting traffic outside of the cluster.

    I am glad that you got it working!

This discussion has been closed.