Using Flux on Azure

AKS Cluster Options

It’s important to follow some guidelines when installing Flux on AKS.

CNI and Network Policy

Previously, there has been an issue with Flux and Network Policy on AKS. ( Upstream Azure Issue) ( Flux Issue) If you ensure your AKS cluster is upgraded, and your Nodes have been restarted with the most recent Node images, this could resolve flux reconciliation failures where source-controller is unreachable. Using --network-plugin=azure --network-policy=calico has been tested to work properly. This issue only affects you if you are using --network-policy on AKS, which is not a default option.

AAD Workload Identity

AAD Workload Identities can be used to enable access to AAD resources from workloads in your Kubernetes cluster.

In order to leverage AAD Workload Identities, you’ll need to have --enable-oidc-issuer and --enable-workload-identity configured in your AKS cluster.

You will also need to establish an identity that has access to ACR.

You can then establish a federated identity credential between the identity and the Flux source-controller ServiceAccount.

Please follow guides for OCIRepositories and AAD Workload Identities and HelmRepositories and AAD Workload Identities.

AAD Pod Identity

Depending on the features you are interested in using with Flux, you may want to install AAD Pod Identity. With AAD Pod-Identity, we can create Pods that have their own cloud credentials for accessing Azure services like Azure Container Registry(ACR) and Azure Key Vault(AKV).

If you do not use AAD Pod-Identity, you’ll need to manage and store Service Principal credentials in K8s Secrets, to integrate Flux with other Azure Services.

As a pre-requisite, your cluster must have --enable-managed-identity configured.

This software can be installed via Helm (unmanaged by Azure). Use Flux’s HelmRepository and HelmRelease object to manage the aad-pod-identity installation from a bootstrap repository and keep it up to date.

Cluster Creation

The following creates an AKS cluster with some minimal configuration that will work well with Flux:

az aks create \
 --network-plugin="azure" \
 --network-policy="calico" \
 --enable-managed-identity \
 --enable-oidc-issuer \
 --enable-workload-identity \
 --name="my-cluster" 

Flux Installation for Azure DevOps

You can install Flux using a Azure Devops repository using the flux bootsrap git command. Ensure you can login to dev.azure.com for your proper organization, and create a new repository to hold your Flux install and other Kubernetes resources.

To bootstrap using HTTPS only, run the following command:

flux bootstrap git \
  --url=https://dev.azure.com/<org>/<project>/_git/<repository> \
  --branch=main \
  --password=${AZ_PAT_TOKEN} \
  --token-auth=true

Please consult the Azure DevOps documentation on how to generate personal access tokens for Git repositories. Azure DevOps PAT’s always have an expiration date, so be sure to have some process for renewing or updating these tokens. Similar to the lack of repo-specific deploy keys, a user needs to generate a user-specific PAT. If you are using a machine-user, you can generate a PAT or simply use the machine-user’s password which does not expire.

To bootstrap using HTTPS but drive the reconciliation in your cluster by cloning the repository using SSH, run:

flux bootstrap git \
  --url=https://dev.azure.com/<org>/<project>/_git/<repository> \
  --branch=main \
  --password=${AZ_PAT_TOKEN}} \
  --ssh-hostname=ssh.dev.azure.com

To bootstrap using SSH, run:

flux bootstrap git \
  --url=ssh://git@ssh.dev.azure.com/v3/<org>/<project>/<repository>
  --branch=main \
  --ssh-key-algorithm=rsa \
  --ssh-rsa-bits=4096 \
  --interval=1m

The above two commands will prompt you to add a deploy key to your repository, but Azure DevOps does not support repository or org-specific deploy keys. You may add the deploy key to a user’s personal SSH keys, but take note that revoking the user’s access to the repository will also revoke Flux’s access. The better alternative is to create a machine-user whose sole purpose is to store credentials for automation. Using a machine-user also has the benefit of being able to be read-only or restricted to specific repositories if this is needed.

Helm Repositories on Azure Container Registry

The Flux HelmRepository object currently supports Chart Repositories as well as fetching HelmCharts from paths in GitRepository sources.

Azure Container Registry has a sub-command ( az acr helm) for working with ACR-Hosted Chart Repositories, but it is deprecated. If you are using these deprecated Azure Chart Repositories, you can use Flux HelmRepository objects with them.

Using Helm OCI with Azure Container Registry

You can use Helm OCI Charts in Azure Container Registry with Flux.

You have to declare a HelmRepository object on your cluster:

flux create source helm podinfo \
  --url=oci://my-user.azurecr.io/charts/my-chart
  --username=username \
  --password=password

or if you are using a private registry:

flux create source helm my-helm-repo \
  --url=oci://my-user.azurecr.io/charts/my-chart
  --secret-ref=regcred

You can create the secret by running:

kubectl create secret docker-registry regcred \
 --docker-server=my-user.azurecr.io \
 --docker-username=az-user \
 --docker-password=az-token

Then, you can use the HelmRepository object in your HelmRelease:

flux create hr my-helm-release \
  --interval=10m \
  --source=HelmRepository/my-helm-repo \
  --chart=my-chart \
  --chart-version=">6.0.0"

Secrets Management with SOPS and Azure Key Vault

You will need to create an Azure Key Vault and bind a credential such as a Service Principal or Managed Identity to it. If you want to use Managed Identities, install or enable AAD Pod Identity.

Patch kustomize-controller with the proper Azure credentials, so that it may access your Azure Key Vault, and then begin committing SOPS encrypted files to the Git repository with the proper Azure Key Vault configuration.

See the Mozilla SOPS Azure Guide for further detail.

Image Updates with Azure Container Registry

You will need to create an ACR registry and bind a credential such as a Service Principal or Managed Identity to it. If you want to use Managed Identities, install or enable AAD Pod Identity.

You may need to update your Flux install to include additional components:

flux install \
  --components-extra="image-reflector-controller,image-automation-controller" \
  --export > ./clusters/my-cluster/flux-system/gotk-components.yaml

Follow the Image Update Automation Guide and see the ACR specific section for more details.

Your AKS cluster’s configuration can also be updated to allow the kubelets to pull images from ACR without ImagePullSecrets as an optional, complimentary step.

Azure Event Hub with Notification controller

The Notification Controller supports both JWT and SAS based tokens but it also assumes that you will provide the notification-controller with a fresh token when needed.

For JWT token based auth we have created a small example on how to automatically generate a new token that the notification-controller can use.

First you will need to create a Azure Event Hub and bind a credential such as a Service Principal or Managed Identity to it. If you want to use Managed Identities, install or enable AAD Pod Identity.

We have two ways to automatically generate new JWT tokens. Ether running as a deployment or a cronjob.

If you are using Azure Event Hub in Azure we recommend that you use aadpodidentity. If you do you will need to update the AzureIdentity config example.

If you are in none Azure environment like on-prem or another cloud then you can utilize client secret which you will find in the example generic folder. Just like aadpodidentity you can use deployment based or a cronjob.

For more info on how to use Azure Event Hub with the notification controller.