Boostrap fails with encrypted GPG signing subkey

[TristanCottam]

Describe the bug

Bootstrapping fails when specifying an encrypted GPG master key for commit signing which delegates signing to a subkey. Specifying an encrypted GPG master key which is capable of signing directly fixes the issue.

Bootstrap log:

► cloning branch "main" from Git repository "[REPOSITORY URL]"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✗ failed to commit component manifests: openpgp: invalid argument: signing key is encrypted

Steps to reproduce

1. Create an encrypted GPG master key with the Certify capability only.
2. Create a GPG subkey of it with the Signing capability.
3. Run flux boostrap [PROVIDER] with the relevant --gpg-key-ring, --gpg-key-id, and --gpg-key-passphrase arguments.

Expected behavior

Flux successfully bootstraps the cluster.

OS / Distro

Debian 12

Flux version

v2.2.2

[kingdonb]

I have just tested flux bootstrap github with a GPG key, and it worked, although this seems quite under-documented.

(For reference, a simple signing workflow on MacOS...)

# MacOS won't keep the keys accessible in a regular disk file in ~/.gnupg/ by default, so export them first:
gpg --keyring secring.gpg --export-secret-keys > ~/.gnupg/secring.gpg
# Get the key id of the key you want to use to sign, in 0xLONG format (16-digit hex):
gpg --keyid-format 0xLONG --list-keys
# Use the public key id, marked "pub   rsa4096/0x..."
flux bootstrap github --owner=$GITHUB_USER --repository=$GITHUB_REPO \
  --branch=main --path=clusters/mycluster \
  --gpg-key-id 0x7FE74CDE6BFFD00E --gpg-key-ring ~/.gnupg/secring.gpg \
  --gpg-passphrase [...]

So if I understand correctly, this report is about using a special type of GPG key and you probably already know that bootstrapping GPG with a "gpg master key capable of signing directly" will work, and you're making a specific report.

I was merely trying to confirm for a reporter in #flux on Slack that GPG keys signing is available at bootstrap time.

I apologize if this is adding noise, but this feature is really under-documented and it took me a while just to confirm this basic feature really works at all, because I couldn't find any tests for it. (I didn't find even a basic example showing the above.)

Have you found any discussions elsewhere about this topic? I think I noticed something related to your report in the discussion around when this feature was first merged:

#1854

It looks like the discussion about future expansion was centered around GPG Agent and didn't go there at all. I don't know if anyone has ever looked into Certify/Signing key splits before, (or even if I've really understood what you are asking, frankly!)

[rwunderer]

Just jumping in as I've tried to sign bootstrapping commits a while ago, but gave up.

For me a huge part of the problem is this:

MacOS won't keep the keys accessible in a regular disk file in ~/.gnupg/ by default, so export them first:

AFAIK that's not MacOS specific, but "modern" gpg does not keep a secret keyring anymore, but stores secret keys in a different format.

The workaround of "exporting" to create the expected secring.gpg does not work if you keep your secret key on a hardware device (e.g. Yubikey) - not having access to the key outside the device is part of the feature set.

IIRC the root of all these issues is the library used for gpg signing. Fixing these would require switching to a more modern library (maybe on the at was talking to GPG agent and handing all this off to the agent).

Disclaimer: all of the above is from memory from when I was troubleshooting this a few months back.

[kingdonb]

Thanks for that - this helps clarify the expectations, and it makes sense that you won't be able to export your private key from the Yubikey.

So there are two different sets of expectations, broadly, to be reconciled when it comes to commit signing support in Flux.

(I'm just mentioning this here for disambiguation, since Yubikey only makes sense for one of them - and it's the less common one)

If you are generating signed commits as an ImageUpdateAutomation, you need to have a key file, exported into a secret in the cluster. This is pretty much non-negotiable, because Flux isn't running interactively and you need to have access to a key in order to sign with it.

When we are talking about bootstrap, this is an interactive process. The flux bootstrap git CLI produces signed commits, if you invoke it a certain way. You do currently need to have a keyring, which is not optimal for Yubikey users who can't do that.

Are you able to sign commits with your Yubikey interactively? (How does that work? Can you point to some docs?)

If there are already existing go libraries that can handle commit signing from a Yubikey, that will be pretty surprising, I guess.

One workaround would be to push the commits to a non-protected branch, amend them with the correct branch name and a signature, then merge them to the protected branch, so bootstrap can proceed without generating a new commit. (Bootstrap is idempotent, so if the content it would have generated is already there, in the repo, in a commit signed properly, then it won't have anything to commit, and the bootstrap would succeed, without access to any keys.)

This is a kludge, but this isn't really an issue I see a lot of time being committed to in the near future, so I am really trying to figure out what use cases can and can't be covered, (+ how deep into kludges we should go for the sake of documentation.)

[rwunderer]

Are you able to sign commits with your Yubikey interactively? (How does that work? Can you point to some docs?)

Actually, the hard part is getting gpg to work with the Yubikey at all. Once that's done, signing commits works just like with regular gpg. All you have to do is add your key id to gitconfig (the ! needs to be set when using a subkey):

git config --global user.signingkey AABBCCDDAABBCCDD!

As for setting up gpg with Yubikey I found this guide to be great:
https://github.com/drduh/YubiKey-Guide

It looks like a lot, but it's really worth going through all the steps.

If however the Yubikey is already completely configured for GPG, then all that needs to be done to make it accessible on your current workstations is (taken from the above guide):

To use YubiKey on multiple computers, import the corresponding public keys, then confirm YubiKey is visible with gpg --card-status. Trust the imported public keys ultimately with trust and 5, then gpg --list-secret-keys will show the correct and trusted key.