I1201 19:31:34.303493 1 controller.go:215] Need to replace PV 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8'
I1201 19:31:34.846833 1 controller.go:137] failed to reconcile resource: failed to replace PV: failed to update property value: Message: 'Successfully set property key(s): Aux/affinity-updater-saved-pv' next error: Message: 'Resource definition 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' modified.'; Details: 'Resource definition 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' UUID is: 89b79f20-d6a0-4fa9-98ce-0a2f93650323' next error: Message: '(okd-node-05) Resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' [DRBD] deleted.' next error: Message: '(okd-node-10) Resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' [DRBD] deleted.' next error: Message: '(Node: 'okd-node-18') Generated resource file for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' is invalid.'; Cause: 'Verification of resource file failed'; Details: 'The error reported by the runtime environment or operating system is:
Resource definition: pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8'; Reports: '[6374ECDC-00000-020347,6374F2B0-62487-000017]'
$ oc describe pv pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
Name: pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: linstor.csi.linbit.com
volume.kubernetes.io/provisioner-deletion-secret-name:
volume.kubernetes.io/provisioner-deletion-secret-namespace:
Finalizers: [kubernetes.io/pv-protection external-attacher/linstor-csi-linbit-com]
StorageClass: ssd
Status: Bound
Claim: NS/PVC
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 5Gi
Node Affinity:
Required Terms:
Term 0: linbit.com/hostname in [okd-node-10]
Term 1: linbit.com/hostname in [okd-node-18]
Term 2: linbit.com/hostname in [okd-node-19]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: linstor.csi.linbit.com
FSType: xfs
VolumeHandle: pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
ReadOnly: false
VolumeAttributes: linstor.csi.linbit.com/mount-options=noatime
linstor.csi.linbit.com/post-mount-xfs-opts=
linstor.csi.linbit.com/remote-access-policy=false
linstor.csi.linbit.com/uses-volume-context=true
storage.kubernetes.io/csiProvisionerIdentity=1668600378626-8081-linstor.csi.linbit.com
Events: <none>
$ oc describe pvc -n NS PVC
...
Status: Bound
Volume: pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
resize.topolvm.io/increase: 25%
resize.topolvm.io/storage_limit: 100Gi
resize.topolvm.io/threshold: 20%
volume.beta.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
volume.kubernetes.io/selected-node: okd-node-18
volume.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 5Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: POD
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal VolumeAffinityUpdated 130m LinstorAffinityController Affinity was out of sync with LINSTOR resource state
Warning ClaimLost 130m (x3 over 15h) persistentvolume-controller Bound claim has lost its PersistentVolume. Data on the volume is lost!
ERROR REPORT 6374ECDC-00000-020347
============================================================
Application: LINBIT�� LINSTOR
Module: Controller
Version: 1.20.0
Build ID: 9c6f7fad48521899f7a99c564b1d33aeacfdbfa8
Build time: 2022-10-18T07:19:30+00:00
Error time: 2022-12-01 19:31:34
Node: linstor-piraeus-cs-controller-6b47864974-rwts8
Peer: RestClient(ip; 'Go-http-client/1.1')
============================================================
Reported error:
===============
Category: RuntimeException
Class name: ApiRcException
Class canonical name: com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at: Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #337
Error message: (Node: 'okd-node-18') Generated resource file for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' is invalid.
Error context:
(Node: 'okd-node-18') Generated resource file for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' is invalid.
ApiRcException entries:
Nr: 1
Message: (Node: 'okd-node-18') Generated resource file for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' is invalid.
Details: The error reported by the runtime environment or operating system is:
The external command 'drbdadm' exited with error code 10
Cause: Verification of resource file failed
Asynchronous stage backtrace:
Error has been observed at the following site(s):
|_ checkpoint ? Modify resource-definition
Stack trace:
Call backtrace:
Method Native Class:Line number
handleAnswer N com.linbit.linstor.proto.CommonMessageProcessor:337
Suppressed exception 1 of 1:
===============
Category: RuntimeException
Class name: OnAssemblyException
Class canonical name: reactor.core.publisher.FluxOnAssembly.OnAssemblyException
Generated at: Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #337
Error message:
Error has been observed at the following site(s):
|_ checkpoint ��� Modify resource-definition
Stack trace:
Error context:
(Node: 'okd-node-18') Generated resource file for resource 'pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8' is invalid.
Call backtrace:
Method Native Class:Line number
handleAnswer N com.linbit.linstor.proto.CommonMessageProcessor:337
handleDataMessage N com.linbit.linstor.proto.CommonMessageProcessor:284
doProcessInOrderMessage N com.linbit.linstor.proto.CommonMessageProcessor:235
lambda$doProcessMessage$3 N com.linbit.linstor.proto.CommonMessageProcessor:220
subscribe N reactor.core.publisher.FluxDefer:46
subscribe N reactor.core.publisher.Flux:8357
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:418
drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:414
drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:679
onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:243
drainFused N reactor.core.publisher.UnicastProcessor:286
drain N reactor.core.publisher.UnicastProcessor:329
onNext N reactor.core.publisher.UnicastProcessor:408
next N reactor.core.publisher.FluxCreate$IgnoreSink:618
next N reactor.core.publisher.FluxCreate$SerializedSink:153
processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:383
doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:218
lambda$processMessage$2 N com.linbit.linstor.proto.CommonMessageProcessor:164
onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:177
runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:439
run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:526
call N reactor.core.scheduler.WorkerTask:84
call N reactor.core.scheduler.WorkerTask:37
run N java.util.concurrent.FutureTask:264
run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker N java.util.concurrent.ThreadPoolExecutor:1128
run N java.util.concurrent.ThreadPoolExecutor$Worker:628
run N java.lang.Thread:829
END OF ERROR REPORT.
root@okd-node-18:/# linstor error-reports show 6374F2B0-62487-000017
root@okd-node-18:/#
root@okd-node-18:/# linstor r l -r pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
╭────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞════════════════════════════════════════════════════════════════╡
╰────────────────────────────────────────────────────────────────╯
root@okd-node-18:/#
root@okd-node-18:/# linstor rd l -r pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
╭───────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Port ┊ ResourceGroup ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8 ┊ 7043 ┊ sc-8c6a2bce-347e-5b46-92df-0ac4616314d8 ┊ ok ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
root@okd-node-18:/#
root@okd-node-18:/# drbdadm status pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8
pvc-26525a5a-72dd-4bb5-90d9-e4287ad7cce8 role:Secondary
okd-node-05 connection:Connecting
okd-node-10 connection:Connecting
okd-node-19 role:Secondary
