Struggling with setting up Workload Identity Federation for my Azure DevOps Pipeline

[mardonner]

Hi, as the title says, I'm currently trying to setup Workload Identity Federation (WIF) in my Azure DevOps Pipeline in order to move away from long lived service account keys that I currently supply with the STACKIT_SERVICE_ACCOUNT_KEY environment variable.

I followed the guide here and opted for the method with the STACKIT_SERVICE_ACCOUNT_EMAIL and STACKIT_USE_OIDC environment variables but after the setup, I'm getting this error:

╷
│ Error: Error configuring provider
│ 
│   with provider["registry.opentofu.org/stackitcloud/stackit"],
│   on providers.tf line 1, in provider "stackit":
│    1: provider "stackit" {
│ 
│ Setting up authentication: configuring no auth client: error initializing
│ client: error reading federated token file - open
│ /var/run/secrets/stackit.cloud/serviceaccount/token: no such file or
│ directory
╵

Side note: The error messages are kind of cryptic. For example, in my first pipeline run, I forgot to set the mail properly and got this error.

╷
│ Error: Error configuring provider
│ 
│   with provider["registry.opentofu.org/stackitcloud/stackit"],
│   on providers.tf line 1, in provider "stackit":
│    1: provider "stackit" {
│ 
│ Setting up authentication: configuring no auth client: error initializing
│ client: client ID cannot be empty
╵

Side note no.2: Something I noticed is that the provider has no authentication fall-through mechanic when supplying STACKIT_USE_OIDC. If this fails, there is no attempt to check the service account key methods.

---

Back to the first error. I'm not sure if I correctly set up Federated Identity Providers in the stackit portal. When adding a new provider, the form has one entry by default:
aud → equals → sts.accounts.stackit.cloud

It was not clear to me whether I should keep it or not and just added the other entry:
aud->equals->api://AzureADTokenExchange

Resulting in having both entries. For testing, I removed the stackit one but that resulted in the same error.

The second part that I might have misconfigured is the sub entry. In the docs it says:

Using a hypothetical pipeline named terraform-ado-oidc inside the project '[https://myorg.azure.com/project-abc`](https://myorg.azure.com/project-abc%60) as example ....
....
sub->equals->p://myorg/project-abc/terraform-ado-oidc # This is the pipeline where the process is running

I assume, myorg is the part after https://dev.azure.com/ and before the next '/' from $(System.CollectionUri) or $(System.TeamFoundationCollectionUri).
I assume, project-abc is $(System.TeamProject). In practice, when using a normal git repo url as example https://dev.azure.com/<myorg>/<project-abc>/_git/<repo-name>
In theory, there is also $(System.TeamProjectId).

But for the pipeline name, terraform-ado-oidc in the example, this is not obvious. I can have two pipelines with the same name under different folders in Azure DevOps. They are under different paths but have the same name.

Both of the return cluster, when printing $(Build.DefinitionName) in them.

That's why I think it is imprecise to tell the users to use the pipeline name. I can differentiate them via their ID $(System.DefinitionId) when printing it in the pipeline or in the url (definitionId=012345), when on the pipeline runs page. Further details can be queried under https://dev.azure.com/<orgname>/<projectname>/_apis/pipelines/<DefinitionId>?api-version=7.1. In that response, there is also the folder attribute. In my case above, \\k8s and for the other one \\test.

I tried changing the pipeline name in the sub assertion to the $(System.DefinitionId) value but I still get the same error.

I also tried:

sub->equals->p://<myorg>/<myproject>/\\test\cluster 

but that also didn't work.

Can you help me?

[JorTurFer]

Hello

│ Setting up authentication: configuring no auth client: error initializing
│ client: error reading federated token file - open
│ /var/run/secrets/stackit.cloud/serviceaccount/token: no such file or
│ directory

This error usually means that the federation source hasn't been found. This probably happens because the terraform provider version doesn't support AzDO natively (this was added quite recently as part of v0.84.0 and not as part of v0.82.0).
Could you check if your provider version is at least v0.84.0? If yes, could you share your pipeline yaml file? Ofc, removing any sensitive data

[mardonner]

I'm on:

• OpenTofu v1.11.5
• stackit provider v0.89.0

Before pasting my pipeline in here, I compared it to the example in the guide again and noticed one difference.
I wasn't explicitly mapping SYSTEM_ACCESSTOKEN in the task environment. After I saw this, I'm certain that this the issue.

I'm now getting this error:

╷
│ Error: Reading network
│ 
│   with module.cluster.data.stackit_network.sna_ske,
│   on .terraform/modules/cluster/data.tf line 5, in data "stackit_network" "sna_ske":
│    5: data "stackit_network" "sna_ske" {
│ 
│ Network with ID "xxx" does not exist in
│ project "xxx".
╵

This error usually means that there is a permission error. The last few times I had this error (while using service account keys), it was due to not having set the "editor" permission in the portal.

In this case right here with WIF, I suspect it's because I have the wrong settings in the Federated Identity Providers for the service account. I'll try the variations from my inital post again and report back.

[mardonner]

Ok, it works, nice!
The remaining issue was the sub → equals → p://<myorg>/<myproject>/<pipelineName> assertion.

I noticed that it indeed needs to be the pipeline name, so $(Build.DefinitionName). $(System.DefinitionId) does not work. So technically this is not optimal, since this is not unique. Meaning that two pipelines with the same name under different folders could be authenticated even if this is not intended.

The guide says Note: This is just an example, you can use more or less fine-grained assertions. Do you have a link to some docs at hand where i can see the other available assertions I can make to make this more fine grained? (probably a microsoft docs page)

edit: I was trying to inspect to token by just trying cat /var/run/secrets/stackit.cloud/serviceaccount/token after the plan finished. Did not work. It says that the file doesn't exits.

[JorTurFer]

You shouldn't use the build ID as assertion IMHO as it means that you will have to update the assertion for each build run as assertions are evaluated as "equal", if you include the unique execution ID as assertion, it'll be valid for a single execution and not for the pipeline itself (just one valid execution forever).

We are thinking about also support Service Connections for this flow. Would it be useful for your case?

[JorTurFer]

edit: I was trying to inspect to token by just trying cat /var/run/secrets/stackit.cloud/serviceaccount/token after the plan finished. Did not work. It says that the file doesn't exits.

That's a default value in the SDK itself (for kubernetes), not related with Azure DevOps. If you want to introspect the content of the pipeline OIDC token, you can request one using this API

[mardonner]

You shouldn't use the build ID as assertion IMHO as it means that you will have to update the assertion for each build run as assertions are evaluated as "equal", if you include the unique execution ID as assertion, it'll be valid for a single execution and not for the pipeline itself (just one valid execution forever).

You're referring to $(Build.BuildId), which is indeed unique for each run and also not predictable. $(System.DefinitionId) is the static unique identifier of the whole pipeline that does not change between runs.

We are thinking about also support Service Connections for this flow. Would it be useful for your case?

This would be nice, since it'd allow me to allow only specific pipelines to use this service connection via AzD UI.
If I can achieve the same effect with just assertions, I'd also be fine with that.

That's a default value in the SDK itself (for kubernetes), not related with Azure DevOps.

Ok, I thought the provider writes to final short lived token into this file after the authentication flow finished.

If you want to introspect the content of the pipeline OIDC token, you can request one using this API

Thank you, I'll have to check, which additional assertions would make sense based on that.

[mardonner]

Also, in the case service connection support is implemented, I'd appreciate a guide on how to create and configure them. For example, configuring WIF for AzD Pipelines to GCP requires a specifc way to setup the service connection.

[mardonner]

Ok, so I took the time to analyze the oidc token. To get one, the fastest way for me was to use a task:

    - script: |
        testToken=$(curl -sSL -X POST \
          -H "Content-Type: application/json" \
          -H "Authorization: Bearer $(System.AccessToken)" \
          -d "" \
          "$(System.OidcRequestUri)?api-version=7.1-preview.1")
        # can't just print it. the pipeline logs will be masked
        mkdir tokenTest
        echo $testToken > tokenTest/token.json
    - publish: $(System.DefaultWorkingDirectory)/tokenTest
      artifact: tokenTest

After decoding the token, I mapped its contents to the predefined variables and concluded this mapping:

{
  "jti": "xxx",
  "sub": "p://<myOrgName (no variable for that)>/$(System.TeamProject)/$(Build.DefinitionName)",
  "aud": "api://AzureADTokenExchange",
  "org_id": "$(System.CollectionId)",
  "prj_id": "$(System.TeamProjectId)",
  "def_id": "$(System.DefinitionId)",
  "rpo_id": "$(Build.Repository.ID)",
  "rpo_uri": "$(Build.Repository.Uri)",
  "rpo_ver": "$(Build.SourceVersion)",
  "rpo_ref": "$(Build.SourceBranch)",
  "run_id": "$(Build.BuildId)",
  "iss": "https://vstoken.dev.azure.com/$(System.TeamProjectId)",
  "nbf": 0123456789,
  "exp": 0123456789,
  "iat": 0123456789
}

<myOrgName> can only be programatically determined from a variable like $(System.CollectionUri) with for example:
echo "$(System.CollectionUri)" | xargs basename
or any other awk, sed or rev + cut command

Depending on how restrictive the assertions in the Federated Identity Providers setting of the service account should be, I guess it'd make sense to use:

aud → equals → api://AzureADTokenExchange
iss  → equals → https://vstoken.dev.azure.com/$(System.TeamProjectId)
sub → equals → p://<myOrgName>/$(System.TeamProject)/$(Build.DefinitionName)
rpo_ref → equals → $(Build.SourceBranch) ( "refs/heads/main" in most cases, or maybe a specific tag)
rpo_id → equals → $(Build.Repository.ID)  (only allow the pipeline from a specific repo, AzD allows repo renaming, so use id instead of url)
def_id → equals → $(System.DefinitionId) (only allow a specific pipeline)

One limitation that makes this slightly inferior to the service connection feature is that currently, the only operator in the assertion is 'equals'. This means, with the settings above, only one single pipeline is able to authenticate. For multiple I'd either need the 'contains' operator, so I can supply multiple values for the token attributes, or I'd need to add an additional Federated Identity Provider entry....or use one and be less restrictive. I assume the assertions are logically ANDed.

Another thing might be that Azure DevOps organization administrators might want to provide teams some default service connection for cloud projects ordered through their internal developer platform. In that case, I guess the setup of service connections via AzD API would be easier.

[JorTurFer]

{
"jti": "xxx",
"sub": "p://<myOrgName (no variable for that)>/$(System.TeamProject)/$(Build.DefinitionName)",
"aud": "api://AzureADTokenExchange",
"org_id": "$(System.CollectionId)",
"prj_id": "$(System.TeamProjectId)",
"def_id": "$(System.DefinitionId)",
"rpo_id": "$(Build.Repository.ID)",
"rpo_uri": "$(Build.Repository.Uri)",
"rpo_ver": "$(Build.SourceVersion)",
"rpo_ref": "$(Build.SourceBranch)",
"run_id": "$(Build.BuildId)",
"iss": "https://vstoken.dev.azure.com/$(System.TeamProjectId)",
"nbf": 0123456789,
"exp": 0123456789,
"iat": 0123456789
}

I think that this info could be quite interesting to be added to the current guide, great job with the mapping! Could you open a PR adding this info the the current guide?

Also, in the case service connection support is implemented, I'd appreciate a guide on how to create and configure them. For example, configuring WIF for AzD Pipelines to GCP requires a specifc way to setup the service connection.

That's correct, the main blocker there is that we would need to create a custom extension for Azure DevOps and we have to prepare a published for that