Missing permissions after upgrading to trident 26.02 and kubernetes 1.34

[jdm85nl]

Describe the bug

This issue is created because we are running in to a new issue after upgrading to kubernetes 1.34 and trident 26.02

After adding the new v1beta2 CRDs the trident csi-snapshotter ( external-snapshotter:8.5.0 ) is running fine.
But after this i've noticed that the csi-resizer is missing some permissions for volumeattributesclasses.storage.k8s
After looking into the release notes of csi-resizer:v2.0.0 i've notice the following:

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)
This resizer version needs Kubernetes 1.34.0+ to support volume modification via the VolumeAttributeClass APIs, as the storage/v1 VolumeAttributesClass object is available starting in v1.34. If the emulation version is below 1.34 and the v1beta1 VolumeAttributeClass API is disabled, the volume modification feature will be disabled.
Once the emulated version is updated to 1.34, a resizer restart is required.

Source: https://github.com/kubernetes-csi/external-resizer/blob/release-2.0/CHANGELOG/CHANGELOG-2.0.md

Since you need to be on kubernetes 1.34 for the csi-snapshotter but also the csi-resizer this could be an issue.

Looking in the code of the trident-controller, by default the feature "feature-gates=CSIVolumeGroupSnapshot=true" is enabled.

    - name: ADDRESS
      value: /var/lib/csi/sockets/pluginproxy/csi.sock
    image: registry.k8s.io/sig-storage/csi-resizer:v2.0.0
    imagePullPolicy: IfNotPresent
    name: csi-resizer
    resources:
      requests:
        cpu: 3m
        memory: 20Mi
    securityContext:
      capabilities:
        drop:
        - all
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/csi/sockets/pluginproxy/
      name: socket-dir
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-dm7t8
      readOnly: true
  - args:
    - --v=2
    - --csi-address=$(ADDRESS)
    - --worker-threads=25
    - --timeout=600s
    - --retry-interval-start=10s
    - --retry-interval-max=300s
    - --feature-gates=CSIVolumeGroupSnapshot=true

When the csi-resizer starts is will look for the API "v1.VolumeAttributesClass".
If it doesn't find the v1.VolumeAttributesClass is will be disabled.

kubernetes 1.33 csi-resizer logs:

I0227 09:03:43.672996       1 main.go:117] "Version" version="v2.0.0"
I0227 09:03:43.673151       1 feature_gate.go:385] feature gates: {map[]}
I0227 09:03:43.673549       1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0227 09:03:43.673608       1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0227 09:03:43.673621       1 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I0227 09:03:43.673628       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0227 09:03:43.673638       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0227 09:03:43.682156       1 main.go:162] "Disabling VolumeAttributesClass feature gate because the VolumeAttributesClass v1 API is not available"
W0227 09:03:43.682544       1 feature_gate.go:506] Overriding default of GA feature gate VolumeAttributesClass=false. It will be removed in a future release.
I0227 09:03:43.684531       1 common.go:143] "Probing CSI driver for readiness"
I0227 09:03:43.686755       1 main.go:187] "CSI driver name" driverName="csi.trident.netapp.io"
I0227 09:03:43.688559       1 controller.go:135] "Register Pod informer for resizer" controller="csi.trident.netapp.io"
I0227 09:03:43.688605       1 controller.go:277] "Starting external resizer" controller="csi.trident.netapp.io"
I0227 09:03:43.693426       1 reflector.go:436] "Caches populated" type="*v1.PersistentVolumeClaim" reflector="k8s.io/client-go/informers/factory.go:160"
I0227 09:03:43.693440       1 reflector.go:436] "Caches populated" type="*v1.PersistentVolume" reflector="k8s.io/client-go/informers/factory.go:160"
I0227 09:03:43.775947       1 reflector.go:436] "Caches populated" type="*v1.Pod" reflector="k8s.io/client-go/informers/factory.go:160"

But when is finds the v1.VolumeAttributesClass it will be enabled and the missing permission error rises.

kubernetes 1.34 csi-resizer logs:

I0306 08:48:46.892800       1 main.go:117] "Version" version="v2.0.0"
I0306 08:48:46.892939       1 feature_gate.go:385] feature gates: {map[]}
I0306 08:48:46.893235       1 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I0306 08:48:46.893273       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0306 08:48:46.893281       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0306 08:48:46.893288       1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0306 08:48:46.893294       1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0306 08:48:46.898605       1 main.go:160] "VolumeAttributesClass v1 API is available"
W0306 08:48:46.900751       1 logging.go:55] [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/csi/sockets/pluginproxy/csi.sock", ServerName: "localhost", Attributes: {"<%!p(networktype.keyType=grpc.internal.transport.network
I0306 08:48:47.902718       1 common.go:143] "Probing CSI driver for readiness"
I0306 08:48:47.904659       1 main.go:187] "CSI driver name" driverName="csi.trident.netapp.io"
I0306 08:48:47.907243       1 controller.go:135] "Register Pod informer for resizer" controller="csi.trident.netapp.io"
I0306 08:48:47.907576       1 controller.go:215] "Starting external resizer for modify volume" controller="csi.trident.netapp.io"
I0306 08:48:47.907692       1 controller.go:277] "Starting external resizer" controller="csi.trident.netapp.io"
I0306 08:48:47.911302       1 reflector.go:436] "Caches populated" type="*v1.PersistentVolume" reflector="k8s.io/client-go/informers/factory.go:160"
I0306 08:48:47.911308       1 reflector.go:436] "Caches populated" type="*v1.PersistentVolumeClaim" reflector="k8s.io/client-go/informers/factory.go:160"
E0306 08:48:47.911291       1 reflector.go:205] "Failed to watch" err="failed to list *v1.VolumeAttributesClass: volumeattributesclasses.storage.k8s.io is forbidden: User \"system:serviceaccount:trident:trident-controller\" cannot list resource \"volumeattributesclasses\" in A
I0306 08:48:47.931657       1 reflector.go:436] "Caches populated" type="*v1.Pod" reflector="k8s.io/client-go/informers/factory.go:160"
E0306 08:48:49.264075       1 reflector.go:205] "Failed to watch" err="failed to list *v1.VolumeAttributesClass: volumeattributesclasses.storage.k8s.io is forbidden: User \"system:serviceaccount:trident:trident-controller\" cannot list resource \"volumeattributesclasses\" in A

Looking into the clusterrole, you will need to add the following:

- apiGroups:
  - storage.k8s.io
  resources:
  - volumeattributesclasses
  verbs:
  - list
  - watch

We can patch it via kustomize, but it would be nice to get it fixed via de chart.
I think it would be usefull to add minimal kubernetes versions for all dependancy images.

Environment
Provide accurate information about the environment to help us reproduce the issue.

• Trident version: [e.g. 26.02]
• Container runtime: [e.g. ContainerD]
• Kubernetes version: [1.34.2]
• Kubernetes orchestrator: [Rancher v2.13.1]
• Kubernetes enabled feature gates: [e.g. CSINodeInfo]
• OS: [Ubuntu 22.04]
• NetApp backend types: [ONTAP AFF 9.16.1P10]
• Other:

To Reproduce
Steps to reproduce the behavior:

Install v1beta2 CRDs
Install/upgrade trident to 26.02
Observe csi-resizer container logs

Expected behavior
A clear and concise description of what you expected to happen.

E0306 08:48:47.911291       1 reflector.go:205] "Failed to watch" err="failed to list *v1.VolumeAttributesClass: volumeattributesclasses.storage.k8s.io is forbidden: User \"system:serviceaccount:trident:trident-controller\" cannot list resource \"volumeattributesclasses\" in A
I0306 08:48:47.931657       1 reflector.go:436] "Caches populated" type="*v1.Pod" reflector="k8s.io/client-go/informers/factory.go:160"
E0306 08:48:49.264075       1 reflector.go:205] "Failed to watch" err="failed to list *v1.VolumeAttributesClass: volumeattributesclasses.storage.k8s.io is forbidden: User \"system:serviceaccount:trident:trident-controller\" cannot list resource \"volumeattributesclasses\" in A

[khassel]

Same problem with v26.02.

This was already mentioned in #1114 but this issue was closed without fixing the missing rbac.