Overview
Same title as #4319 but this is on the pgdata volume.
The auto grow volume monitor bash script seems to have a bug.
|
TOKEN=$(cat "${SERVICEACCOUNT}/token") |
The TOKEN is only read once, outside the loop. This means that autogrow will function correctly immediately after the container is started (or restarted). However if the storage threshold is reached after the token has expired the bash script tries to use the expired token, and fails quietly.
Checking the logs confirms this, showing a constant stream of 401 unauthorised errors.
kubectl logs hippo-main-hg94-0 -c replication-cert-copy --tail 10
Dload Upload Total Spent Left Speed
100 251 100 157 100 94 22428 13428 --:--:-- --:--:-- --:--:-- 35857
}{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
This aligns with the experience we've had with CD/Snowflake support where restarting the pod "fixes it" but noone knows why.
Environment
Please provide the following details:
- Platform: EKS
- Platform Version: 1.34
- PGO Image Tag: 5.8.5
- Postgres Version: 14
- Storage: gp3
Steps to Reproduce
- Turn on autogrow
- Wait for token expiry
- Fill up disk
- See no resize
- Restart container
- See resize
Proposed Fix
Move the TOKEN read inside the loop, possibly only in the case that it's actually going to be used.
Overview
Same title as #4319 but this is on the pgdata volume.
The auto grow volume monitor bash script seems to have a bug.
postgres-operator/internal/postgres/config.go
Line 327 in 668bcae
The
TOKENis only read once, outside the loop. This means that autogrow will function correctly immediately after the container is started (or restarted). However if the storage threshold is reached after the token has expired the bash script tries to use the expired token, and fails quietly.Checking the logs confirms this, showing a constant stream of 401 unauthorised errors.
This aligns with the experience we've had with CD/Snowflake support where restarting the pod "fixes it" but noone knows why.
Environment
Please provide the following details:
Steps to Reproduce
Proposed Fix
Move the TOKEN read inside the loop, possibly only in the case that it's actually going to be used.