Security: Image Provenance

Next up in our blog series about Flux Security is how and why we use signatures for the Flux CLI and all its controller images and what you can do to verify image provenance in your workflow.

Since Flux 0.26 our Security Docs had this addition:

The Flux CLI and the controllers' images are signed using Sigstore Cosign and GitHub OIDC. The container images along with their signatures are published on GitHub Container Registry and Docker Hub.

To verify the authenticity of Flux’s container images, install cosign and run cosign verify.

We are very pleased to bring this to you and encourage you to make use of it in your workflows to make your clusters more secure. But let’s unpack everything the above section says.

Why sign release artifacts in the first place?

Essentially we want you to be able to verify Flux’s image provenance, which boils down to making sure that

1. The release you just downloaded actually comes from us, the Flux team
2. It hasn’t been tampered with

Cryptographic signatures are the go-to choice for this and have been used for decades, but are not without challenges.

The (excellent) sigstore docs say this:

Software supply chains are exposed to multiple risks. Users are susceptible to various targeted attacks, along with account and cryptographic key compromise. Keys in particular are a challenge for software maintainers to manage. Projects often have to maintain a list of current keys in use, and manage the keys of individuals who no longer contribute to a project. Projects all too often store public keys and digests on git repo readme files or websites, two forms of storage susceptible to tampering and less than ideal means of securely communicating trust.

The tool sets we’ve historically relied on were not built for the present circumstance of remote teams either. This can be seen by the need to create a web of trust, with teams having to meet in person and sign each others’ keys. The current tooling (outside of controlled environments) all too often feel inappropriate to even technical users.

We are very happy that sigstore exists. It is a Linux Foundation project backed by Google, Red Hat and Purdue University striving to establish a new standard for signing, verifying and provenance checks for the open source community.

The way it works is that our cosign workflow uses

• cosign to sign our release artifacts and store the signatures in an OCI registry (GHCR and Docker Hub in our case)
• OpenID Connect (OIDC) beforehand to be identified via our email address
• Fulcio, a root certification authority (CA), which issues a time-stamped certificate for the identified user that has been authenticated
• Rekor, acts as the transparency log storage, where the certificate and signed metadata is stored in a searchable ledger, that can’t be tampered with

That’s a lot of terminology and project names, but what’s beautiful about cosign is that you can relatively easily integrate this using GitHub Actions (see how it was done in source-controller).

How to verify signatures

If you want to do this manually as an one-off, install the cosign tool and essentially just run cosign verify for the image you want to verify:

cosign verify ghcr.io/fluxcd/source-controller:v1.0.0-rc.5 \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  --certificate-identity-regexp=^https://github.com/fluxcd/.*$

With --certificate-oidc-issuer we verify that the Flux image was signed by GitHub Actions OIDC issuer. With --certificate-identity-regexp we verify that the image originates from the FluxCD GitHub organization.

The output should be:

Verification for ghcr.io/fluxcd/source-controller:v1.0.0-rc.5 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

Now let’s take a look at how to automate this further.

Enforcing verified signatures in a cluster

Luckily cosign is compatible with and supported by policy engines such as Connaisseur, Kyverno and OPA Gatekeeper. Let’s go with Kyverno for now. To make sure that Flux image signatures are verified, all you need to do is add the following manifest:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-flux-images
spec:
  validationFailureAction: enforce
  background: false
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
  rules:
    - name: verify-cosign-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/fluxcd/source-controller:*"
            - "ghcr.io/fluxcd/kustomize-controller:*"
            - "ghcr.io/fluxcd/helm-controller:*"
            - "ghcr.io/fluxcd/notification-controller:*"
            - "ghcr.io/fluxcd/image-reflector-controller:*"
            - "ghcr.io/fluxcd/image-automation-controller:*"
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/fluxcd/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

Have a look at fluxcd/flux2-multi-tenancy to see a more elaborate example and how the Kyverno policies are wired up there.

By verifying all our artifacts, you ensure their provenance and guarantee that they haven’t been modified from the moment we signed and shipped them. This is just one more measure we are taking to keep you more secure.

Talk to us

We love feedback, questions and ideas, so please let us know your personal use-cases today. Ask us if you have any questions and please

• join our upcoming dev meetings
• find us in the #flux channel on CNCF Slack
• add yourself as an adopter if you haven’t already

See you around!

• ←Previous
• Next→