Export specific data from vault

jwilson5607

Hello, any help would be greatly appreciated on this. I have to export a list of logins from a specific Vault as part of a divestiture with the company I work for. I have been attempting to pull the data in a way that I get the Title and the Username for now (I will need to export this to a csv but need to get the data returned for now). However, I cannot get the data I am wanting.

function Get-1passworditems {
    [CmdletBinding()]
    param ([Parameter(Mandatory=$true)][string]$vaultuuid)

    #Set Console Encoding
    [Console]::OutputEncoding = [System.Text.Encoding]::UTF8

    #Ensure variables are empty and ready to go
    $results = @()
    $id =@()
    $data =@()
    $item = @()

    #Get Initial list of item uuids
    $results = op item list --vault $vaultuuid --format=JSON

    #Convert from JSON to powershell objects

    #$data = $results | ConvertFrom-Json

    $id = $results.id

    Write-Host "Starting foreach" -ForegroundColor Green
    foreach($item in $id){
        $item
        & $op item get $item --vault $vaultuuid --field username
    }
}

If I attempt to convert the data using ConvertFrom-JSON (as per another discussion I saw here), I get all kinds of errors. I know there is something I am missing here, but am not sure what. Attempts to use the desktop app to export that specific vault is not working in a way that I can safely say its specific to that vault and not a full export of the 40+ vaults we have.

---

1Password Version: Desktop 8.7.3, CLI 2.5.1
Extension Version: Not Provided
OS Version: Win 10
Browser:_ Not Provided

jwilson5607

I started to get somewhere with these commands but its still being a bit of a pain and randomly giving me these errors in the screenshot. There are about 4200 entries in this Vault and I KNOW API limits are likely an issue, so I added the start-sleep after each one, to avoid it. It might be too much of a timer, but I am trying to get all the results back.

function Get-1passworditems {
    [CmdletBinding()]
    param ([Parameter(Mandatory=$true)][string]$vaultuuid)

    #Set Console Encoding

    #Ensure variables are empty and ready to go
    $results = @()
    $data =@()
    $item = @()
    $export = @()
    $vaultname = @()

    #Get Initial list of item uuids
    $results = op item list --vault $vaultuuid --format=json
    $vaultname = op vault get $vaultuuid
    #Convert from JSON to powershell objects

    $data = $results | ConvertFrom-Json

    foreach($item in $data){
        $obj = @()
        $name = op item get $item.id --vault $vaultuuid --field username
        $obj = New-Object psobject -Property @{
            Username = $name
            UUID = $data.id
            Title = $data.title
        }
        $export += $obj
        Start-Sleep -Seconds 10
    }

    Export-Excel -Path C:\Scripts\Results\1password_logins.xlsx -WorksheetName $vaultname.Name -Append -AutoSize -AutoFilter -FreezeTopRow
}

Justin.Yoon_1P

Hey there @jwilson5607

I've gotten your script to work locally with some modifications, but Instead of exporting to Excel, I just printed the fields of interest, as I don't have that specific Export-Excel module on my Windows test device.

function Get-1passworditems {
    [CmdletBinding()]
    param ([Parameter(Mandatory=$true)][string]$vaultuuid)
    #Ensure variables are empty and ready to go
    $results = @()
    $data = @()
    $item = @()
    $vault = @()
    #Get Initial list of item uuids
    $results = op item list --vault $vaultuuid --format=json
    Write-Host ($results)
    $vault = op vault get $vaultuuid --format json
    Write-Host ($vault)
        $vaultObj = $vault | ConvertFrom-Json
    Write-Host ("vaultName: " + $vaultObj.name)
    #Convert from JSON to powershell objects
    $data = $results | ConvertFrom-Json
    foreach($item in $data){
        Start-Sleep -Seconds 1
        $name = op item get $item.id --vault $vaultuuid --field username
        Write-Host ("name: " + $name + " uuid: " + $data.id + " title: " + $data.title)
    }
}

Some things I modified are:

• the $vault = op vault get $vaultuuid should specify the --format json flag
• the $vault output should be converted from JSON to access its name in subsequent commands
• With these changes I am able to output the vault name, item username, item id, and item title.

As for the error you encountered, I have a feeling that could be caused by continuous execution of op commands after one of them has returned an error, please do let me know if you keep seeing it.

jwilson5607

I will give yours a go here in a bit, I DID manage to get things SOMEWHAT working. I have removed the sleep statement in the foreach loop as I was testing it without it this time. I have approximately 4200 entries in this vault in particular I am needing to export from. Supports suggestion was to setup a 1password connect server, but I am not sure that it would work the way I want it to, and the cost for setting it up seems a bit much for the use it would get.

function Get-1passworditems {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory=$true)][string]$vault
    )

    #Create initial set of empty variables/arrays

        #Array for exporting the data to excel
        $results = @()

        #Vault details for worksheet name/id info
        $vaultdetails = @()
        $vaultdetailsconverted = @()

        #Summary items in vault - keys such as ID, type, name, etc
        $vaultitems = @()

        #Array for the $vaultitem records converted from JSON format
        $vaultitemsconverted = @()

    #Start process of getting data
        #Get Vault Details
        Write-Host "Getting Vault info and item list" -ForegroundColor Green
        $vaultdetails = op vault get $vault --format=json
        $vaultdetailsconverted = $vaultdetails | ConvertFrom-Json
        $vaultdetailsconverted
        $vaultitems = op item list --vault $vault --categories Login --format=json
        $vaultitemsconverted = $vaultitems | ConvertFrom-Json
        Write-Host -ForegroundColor Yellow "Items in the data set to parse: $($vaultitemsconverted.count)"
    #Start looping through the item details to extract username and password
        Write-Host -ForegroundColor Green "Starting to get item details"
        foreach($id in $vaultitemsconverted){
            $c++
            $obj = @()
            $item =@()
            $obj1 =@()
            $username = @()
            $password = @()
            $item = op item get $id.id --vault $vaultdetailsconverted.id --format=json
            $obj = $item | ConvertFrom-Json
            foreach($field in $obj.fields){
                if($field.id -eq "username"){
                    $username = $field.value
                }elseif ($field.id -eq "password") {
                    $password = $field.value
                }else {

                }
            }
            $obj1 = New-Object psobject -Property @{
                Vault_ID = $obj.vault.id
                Vault_Name = $obj.vault.name
                User_Name = $username
                Password = $password
                Tags = $id.Tags
                Item_ID = $id.ID
                Title = $id.Title
            }
            $results += $obj1
            Write-host -ForegroundColor Green "$c of $(($vaultitemsconverted.id).count)"
        }
        $results | Export-Excel -Path C:\Scripts\Results\1password_vault_info.xlsx -AutoSize -AutoFilter -FreezeTopRow
}

You could remove the export-excel part to see if you are getting what I am - I know I tested this on my personal 1password account and was able to retrieve all 150 results in there.

jwilson5607

@Justin.Yoon_1P -

Quick question for you: Why am I getting random errors like those pictured (there are others where it says something to the effect of errors with tcp to xxx.xxx.xxx.xxx [assuming 1password server], and others about secure key error or the software terminated [i will try to capture some of them but haven't yet])? I cannot make sense of it because I have gotten them even at the beginning of the script where its getting the $vaultdetails or $vaultitems.

Plus, is there a difference with --format=json and > --format json?

Justin.Yoon_1P

Hey again @jwilson5607 ,

Why am I getting random errors like those pictured

I've taken a look at the stack, and it looks like it's erroring out in the biometric auth process, in which case, concurrent attempts to send info through a named pipe may have caused the error. Do you have biometric auth turned on, and if so, has op been authorized before running the script?

Also, this is not an ideal error message. It is the result of an unhandled error causing a panic and is not very readable or understandable. I will be treating it as a bug report once we figure out why it's being printed.

Plus, is there a difference with --format=json and > --format json?

No, there is not! I did notice, however, that the flag was missing in the $vaultname = op vault get $vaultuuid command, which is why I suggested it in my last post.

Edit: I just took a look at your updated script, and it looks good to me, I will test it on my end tomorrow and get back to you.

Are you still running into issues without the sleep statement? How about with it on, but around 1 second?
Is it the same error as earlier, or a general 429 rate limited one?

Thanks for your info so far!!

jwilson5607

Yes, I have biometrics turned on and allowed the auth. I do NOT get prompted again while its running the pipeline though.

I noticed that you had both instances set with and without the = sign and wanted to make sure there was no difference.

Justin.Yoon_1P

Hey @jwilson5607

Yes, I have biometrics turned on and allowed the auth. I do NOT get prompted again while its running the pipeline though.

This should not cause any error states in that case - this has to be a bug on our end, are you running into it consistently?

Also I'm not sure if you caught the added questions in the edit of my last message:

Are you still running into issues without the sleep statement? How about with it on, but around 1 second?
Is it the same error as earlier, or a general 429 rate limited one?

Lastly, I've just tested your new script and can confirm that there are no issues with it (I didn't test the Excel export part, but printing the $results variable looked good to me).

jwilson5607

Running it with a second delay on each loop gives me the same errors as before. I did manage to capture a few others as well, as shown here.

jwilson5607

There is no discernable pattern to it either. Sometimes its every other one, sometimes it will go quite a few and then hit one or more of these errors. If I need to attach the errors in another way because they are too hard to read - let me know.

jwilson5607

Attached a few as files here along with a new one @Justin.Yoon_1P

Justin.Yoon_1P

Thanks @jwilson5607

We're going to start looking into this by trying to reproduce it on a Windows binary on a vault with many items. After some digging around, it turns out we've had the same error reported on another Windows system user with lots of items in their vault - so you are not alone in seeing this error.

I am not quite sure if you have a timeline on fixing this error due to the divesture from your current company, but because the fix for this error is quite undefined at this point, I would advise trying to circumvent the error by adding a longer sleep, or trying to use op on a different platform, like mac or linux OSs for example.

jwilson5607

The code I executed on my personal MacBook ran flawlessly. It seems that the 1password CLI in Windows is flawed to the point I have issue using it - which is disappointing because my team is PC only. I look forward to the fix to the Windows libraries so that i can utilize this on my work PC moving forward.

Horia.Culea_1P

Thank you for your detailed diagnostics and your time!
We have an internal ticket tracking all the reports that we receive, related to this.
We'll make sure to post an update as soon as we'll have any developments on this.

Best,
Horia

Joris_1P

Just wanted to drop by to let you know that we have just released v2.11.0 of 1Password CLI, which should address this issue. See Update to the latest version of 1Password CLI for instructions on how to update.

Let us know if you are still having issues with v2.11.0 or later.