How to manage Pulumi Secrets with 1Password
This post is part of my series on migrating my homelab IaC from Terraform to Pulumi. In this article, I explain how I manage secrets in my Pulumi setup using 1Password and what I learned along the way.
Why 1Password
One of my goals for the Pulumi migration was to automate the setup of SSH connections to my containers. I use 1Password to handle all my passwords, and SSH felt like a convenient way to avoid manually copying the private SSH keys that Pulumi creates into my .ssh folder.
I also had plenty of secrets defined in a separate secrets file that I wanted to move into 1Password, so that there would be a single source of truth for all API keys, usernames, passwords, and tokens in both my homelab and my browser sessions.
However, I found the Pulumi 1Password provider quite limiting for this setup:
- It only supports reading items, and I want to create new items when I add new containers to my homelab.
- Access is rate limited by the service account restrictions. There is a limit of 1000 read operations per hour and 100 write operations per hour for 1Password and 1Password Families accounts per service account token. Those limits also apply on a daily basis to the 1Password account. My homelab vault already has around 50 secrets, which effectively limits me to about 20 executions per day. That is not enough on days when I’m actively experimenting.
So I decided to build my own 1Password client. There are some important details to take into account if you follow this approach. I ran into a number of issues and eventually managed to sort them out.
Avoiding rate limits
Service account rate limits are too restrictive for my use case, so I am effectively forced to use the desktop app authentication mechanism to avoid them.
It is possible to set up a Connect Server that does not face the same rate limits, but it requires spinning up two Docker containers, and I found that too overcomplicated for my simple needs.
When I started my Pulumi migration, the latest version of the 1Password JS SDK available was v0.3.1, which only supported service accounts as an authentication mechanism. I decided to wrap the 1Password CLI in a TypeScript class so I could use the desktop app as the authentication mechanism and avoid the service account rate limits.
Later on, version v0.4.0 of the 1Password JS SDK added support for using the desktop app as an authentication mechanism, but by then I already had all those utilities in place.
If I were to start from scratch again, I would stick to the JS SDK using the desktop app as the authentication mechanism instead.
CLI client issues
Using the CLI directly introduced a few practical issues.
First, dollar sign ($) characters were incorrectly double-escaped. This led to my client trying to overwrite, on each execution, existing passwords that contained dollar signs.
The plans still worked correctly, but this had a noticeable performance impact. I could not figure out how to properly fix this, so I decided to remove the dollar sign from the set of special characters allowed to be used by the Pulumi random.randomPassword provider.
Second, sequential access was too slow, but simply making access parallel caused 1Password to ask multiple times, concurrently, for permissions to access the vault.
To fix this, I forced the first access to a secret to be sequential, with the rest of the accesses running in parallel. This way, after I allow access once, the remaining accesses do not need to ask for permissions again.
SSH Keys
One of the things I wanted to achieve was a better integration with my SSH client, so that whenever I apply a Pulumi plan I get the SSH keys to connect to the container in 1Password, and I can SSH into it directly from my terminal without manually copying private keys all over the place.
However, there is no way to import SSH keys to 1Password with the CLI, so I had to delegate the actual SSH key generation to 1Password and then copy it over to the remote container.
Localized labels
My development machine is set up in Spanish, so when I create a login item in 1Password the username field is labeled nombre de usuario and the password field is labeled contraseña. This also affects the credential field on API key items, which is labeled credencial.
I normally don’t pay much attention to those labels, but they are important when requesting items from 1Password, because those field labels are what the CLI client will return. To support multiple languages I had to define a list of alternative labels for each kind of field I wanted to request, and then make my client return the first matching field.
Caching items
My custom 1Password wrapper was the slowest part of my Pulumi codebase, taking about 80% of the runtime (100 seconds out of 120-second executions). I managed to reduce this to a negligible duration with caching and parallelization:
- Keep a cache for all items that you read. You probably don’t need to worry about stale items because this will run in less than a minute, and if you update any item in this period you can just re-run the plan later. Don’t forget to invalidate the cache entry if you update the item as part of the plan.
- Parallelize as much work as possible. Describe a strong dependency tree for your resources so that you only try to read 1Password items that have already been created by a parent resource. That way you will be able to run most 1Password interactions in parallel.
- Minimize cold starts and permission requests by gating all concurrent requests behind a sequential gate. This way you will get asked for permission only once, and after this initial request everything will run in parallel. Otherwise, every single request will be asking for permissions and hitting cold internal caches as they all run in parallel.
No replies on “How to manage Pulumi Secrets with 1Password”