Impossible to use via GitHub Actions Jobs Services, what solutions are there?

Hello!

I was excited to see that the new 1Password Connect Automation had been not only released but was available to individual account users like myself (I've been using 1Password since 2011, before the 1password.com login), and I would rather not have to rely on weird hacks regarding parsing out the JSON from the CLI. e.g., (I've never gotten the op get item examples to work ever)

I was also happy to see the example docker-compose.yml file was effectively the same as using github actions jobs.<job-id>.services

jobs:
  job-name:
    runs-on: ubuntu-latest
    name: ignore-this-name
    services:
      op-connect-sync:
        image: 1password/connect-api:latest
        ports:
          - 8080:8080
        volumes:
          - ${{github.workspace}}/data:/home/opuser/.op/data
      op-connect-api:
        image: 1password/connect-sync:latest
        ports:
          - 8081:8080
        volumes:
          - ${{github.workspace}}/data:/home/opuser/.op/data

(The above has some personal information removed. I've not bothered to add a volume mount for a non-existant file, and mounting ${{github.workspace}} as /home/opuser/.op just doesn't work. The above is effectively the result of fruitlessly trying to get the sync and api services to not immediately error because of directory permissions)

There are quite a few issues I've discovered while poking around and I'm hoping they could be resolved, or someone could find a better so I can access the connect API during github jobs.

  1. The 1password-credentials.json file must exist on disk and these containers are started before I can write the file out from ${{secrets.OP_CRED_JSON}}. If I could pass it in as an environment variable, or even just break up the entire credentials file as arguments to them, that would improve things marginally.
  2. Even if I put a sleep 30s later on during the build (as that's the maximum "check for file" time I discovered while trying to get this to work), they will still complain that their mounted directories have incorrect permissions. This persist even if chown -hR root is executed on the github workspace and --user root is passed to each docker image (as well as changing the home/opuser/.op to /root/.op.
  3. --entrypoint cannot be used to override the entrypoint of connect-api and connect-sync, preventing the chance of making sure file permissions are correct with a bash while-loop

For the record, what I am trying to do is use 1password connect to extract certain keys for deployments that I can easily rotate as needed without having to use the github approach to key rotation (as well as store a few important bits of credential data that might be generated during an workflow)

Any suggestions (or any ability to improve on what I've mentioned above) would be greatly appreciated. None of the behavior I discovered above is currently documented anywhere that I could 😅

Thanks!


1Password Version: Not Provided
Extension Version: Not Provided
OS Version: Not Provided
Sync Type: Not Provided

Comments

  • James_1PJames_1P

    Team Member

    Hi,

    Thanks for your detailed feedback on your attempts to use the Connect server in GitHub Actions like this. I'll start with the quick answer which is that we will have more details on using 1Password Secrets Automation in GitHub Actions soon.

    The more detailed response here is that you are going to run into several issues attempting to run the containers as services.

    1. Providing the credentials file to the server is technically supported when base64 encoding the json and setting the value as OP_SESSION. This is the same feature that is used in our 1Password SCIM Bridge but is deprecated so we did not include it in the official documentation.
    2. The Connect API/Sync images are built to run as a non root user and for security purposes will not write data to a folder that is owned by a root user.
    3. Both containers do support using --entrypoint overrides but I am not familiar enough with GitHub Actions to know if their entry point support matches docker
    $ docker run --rm -it --entrypoint bash 1password/connect-api
    [email protected]:/$
    
  • I've been able to run the Connect server in GitHub Actions here: https://github.com/RohanNagar/op-connect-sdk-java/blob/master/.github/workflows/ci.yml

    I'm not using the services feature, I'm just running sudo docker-compose up in a step before I need to access the API. I have the credentials.json file contents and the API access token in secrets and use them like below:

    # Create the credentials file that is mounted in docker-compose.yaml
    - name: Set up 1Password Credentials
      run: echo $OP_CREDENTIALS > local-connect/1password-credentials.json
      env:
        OP_CREDENTIALS: ${{ secrets.OP_CREDENTIALS }}
    
    # Start the local server
    - name: Start OPConnect Server
      run: sudo docker-compose -f local-connect/docker-compose.yaml up -d
    
    # Call the API (My Java unit tests are calling the API and need the access token)
    - name: Build with Maven
      run: mvn package jacoco:report
      env:
        OP_ACCESS_TOKEN: ${{ secrets.OP_ACCESS_TOKEN }}
    

    There's maybe a cleaner way to do this with the services feature, but this works for now.

  • we will have more details on using 1Password Secrets Automation in GitHub Actions soon.

    This is good to hear!

    Providing the credentials file to the server is technically supported when ... but is deprecated

    Ahh, the best kind of support :tongue: (I come from the C++ world so "technically supported because of deprecation" is... well another discussion for another time :smile:)

    The Connect API/Sync images are built to run as a non root user and for security purposes will not write data to a folder that is owned by a root user.

    Yes I expected this to happen, however even when the folder was not owned by root (but instead created by an account with a different UID/GID than that of the docker user opuser, it would instead give the same error as root. I did a chown as I was at the "grasping at straw" stage of bit bashing

    Both containers do support using --entrypoint overrides

    It only seemed to work if the interactive flag was passed. If I tried to do /bin/bash -c 'some-script' it would then not run my script but just execute the connect-* command anyhow and I would get error logs.


    I've been able to run the Connect server in GitHub Actions here:

    I was trying to avoid using a docker-compose file and keep everything within the .github workflows as I can enforce these at an organization level as a template if I ever want to take something like this to work, and it keep access policies regarding files changing to a minimum. But that said, it's good to know that you can launch something like docker-compose in the background on GHA workers. (Though I suppose my next line of thought is if docker secrets works and if that could be used to keep the 1password-credentials.json off disk, and then OP_SESSION could just be /run/secrets/1password-credentials.json :smile: )

  • James_1PJames_1P

    Team Member

    the best kind of support :tongue: (I come from the C++ world so "technically supported because of deprecation" is... well another discussion for another time :smile:)

    I felt it best to be honest and admit that it would technically work in the off chance that you had come across some old support document for the SCIM Bridge. Also I am genuinely curious how far you can get running this in GitHub Actions this way.

    I did a chown as I was at the "grasping at straw" stage of bit bashing

    That should work okay. In fact that is how the helm chart deals with persistentVolume default permissions for clusters pre 1.20.

    If I tried to do /bin/bash -c 'some-script' it would then not run my script

    Entrypoint overriding in a non interactive case needs to have the entry point pieces provided as an array. For example, in the compose file you can add

        entrypoint:
          - "/bin/bash"
          - "-c"
          - "echo Hello World"
    

    and then running docker compose up will result in

    $ docker compose up
    Attaching to connect-api_1, connect-sync_1
    connect-api_1   | Hello World
    
  • Also I am genuinely curious how far you can get running this in GitHub Actions this way.

    Most likely the whole way, if that's the case. The credentials file is under 64kb in size, and it would be masked in the logs as an actions secret even if set to an environment variable. You'd need to base64 encode it prior to setting the secret, but it should work.

    That should work okay.

    Looking around online, other folks have run into this permissions issue with GHA, even when using a self-hosted docker in docker runner, where the actions user is set. (You'll notice that we need to pass sudo like everyone else)

    Entrypoint overriding in a non interactive case needs to have the entry point pieces provided as an array.

    I was aware that was the case for docker-compose, but the github actions services syntax differs as you can only pass it via --entrypoint and I don't believe that takes a json (or json-like in the case of the docker-compose yaml file) array

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file