Barman cloud WAL archive check exception: 'utf-8' codec can't encode characters in position 4-6: surrogates not allowed

[sdaguet]

Hello,

I'm using cnpg v1.28.0 and barman plugin v0.10.0 on private on premises k8s vanilla cluster.
I try to configure WAL archiving on a private S3 server.

Configuration and access sounds ok, if I try with the same parameters to run barman-cloud-check-wal-archive on a separate server after configuring the .aws/credendials file, all sounds ok ($? = 0)

barman-cloud-check-wal-archive --endpoint-url https://s3.xxx/ --cloud-provider aws-s3 "s3://fra-bkp/pg-cluster-backups/" pg-cluster

However, can't manage to get it working, get utf-8 errors ...

{"level":"info","ts":"2026-01-15T16:19:01.661515283Z","msg":"barman-cloud-check-wal-archive checking the first wal","logging_pod":"pg-cluster-1"} {"level":"info","ts":"2026-01-15T16:19:02.831635279Z","logger":"barman-cloud-check-wal-archive","msg":"2026-01-15 16:19:02,831 [94] ERROR: Barman cloud WAL archive check exception: 'utf-8' codec can't encode characters in position 4-6: surrogates not allowed","pipe":"stderr","logging_pod":"pg-cluster-1"} {"level":"error","ts":"2026-01-15T16:19:02.896804809Z","msg":"Error invoking barman-cloud-check-wal-archive","logging_pod":"pg-cluster-1","options":["--endpoint-url","https://s3.xxx/","--cloud-provider","aws-s3","s3://fra-bkp/pg-cluster/","pg-cluster"],"exitCode":-1,"error":"exit status 4","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\t/go/pkg/mod/github.com/cloudnative-pg/machinery@v0.3.3/pkg/log/log.go:125\ngithub.com/cloudnative-pg/barman-cloud/pkg/walarchive.(*BarmanArchiver).CheckWalArchiveDestination\n\t/go/pkg/mod/github.com/cloudnative-pg/barman-cloud@v0.4.0/pkg/walarchive/cmd.go:184\ngithub.com/cloudnative-pg/barman-cloud/pkg/archiver.(*WALArchiver).CheckWalArchiveDestination\n\t/go/pkg/mod/github.com/cloudnative-pg/barman-cloud@v0.4.0/pkg/archiver/archiver.go:131\ngithub.com/cloudnative-pg/plugin-barman-cloud/internal/cnpgi/common.CheckBackupDestination\n\t/workspace/internal/cnpgi/common/check.go:52\ngithub.com/cloudnative-pg/plugin-barman-cloud/internal/cnpgi/common.WALServiceImplementation.Archive\n\t/workspace/internal/cnpgi/common/wal.go:165\ngithub.com/cloudnative-pg/cnpg-i/pkg/wal._WAL_Archive_Handler.func1\n\t/go/pkg/mod/github.com/cloudnative-pg/cnpg-i@v0.3.1/pkg/wal/wal_grpc.pb.go:194\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.3.3/interceptors/recovery/interceptors.go:34\ngoogle.golang.org/grpc.getChainUnaryHandler.func1.getChainUnaryHandler.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.78.0/server.go:1241\ngithub.com/cloudnative-pg/cnpg-i-machinery/pkg/pluginhelper/http.(*Server).Start.loggingUnaryServerInterceptor.func3\n\t/go/pkg/mod/github.com/cloudnative-pg/cnpg-i-machinery@v0.4.2/pkg/pluginhelper/http/grpc.go:39\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.78.0/server.go:1241\ngithub.com/cloudnative-pg/cnpg-i-machinery/pkg/pluginhelper/http.(*Server).Start.logFailedRequestsUnaryServerInterceptor.func2\n\t/go/pkg/mod/github.com/cloudnative-pg/cnpg-i-machinery@v0.4.2/pkg/pluginhelper/http/grpc.go:51\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.78.0/server.go:1232\ngithub.com/cloudnative-pg/cnpg-i/pkg/wal._WAL_Archive_Handler\n\t/go/pkg/mod/github.com/cloudnative-pg/cnpg-i@v0.3.1/pkg/wal/wal_grpc.pb.go:196\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.78.0/server.go:1428\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.78.0/server.go:1832\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.78.0/server.go:1063"}

Objectstore is

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: pg-cluster-store
  namespace: pg-cluster
spec:
  configuration:
    destinationPath: "s3://fra-bkp/pg-cluster/"
    endpointURL: "https://s3.xxx/"
    endpointCA:
      name: root-acme
      key: root_ca.crt
    s3Credentials:
      accessKeyId:
        name: s3-creds
        key: ACCESS_KEY_ID
      secretAccessKey:
        name: s3-creds
        key: ACCESS_SECRET_KEY

Postgres logs includes that
{"level":"error","ts":"2026-01-15T16:44:58.517145Z","logger":"wal-archive","msg":"failed to run wal-archive command","logging_pod":"pg-cluster-1","error":"rpc error: code = Unknown desc = unexpected failure invoking barman-cloud-wal-archive: exit status 4","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.1/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.5/x64/src/runtime/proc.go:285"}

Any idea to debug that ?

[dosubot]

Hi @sdaguet! I'm Dosu and I’m helping the plugin-barman-cloud team.

This UTF-8 error is likely caused by a value (such as a WAL file name, S3 path, or credential) containing invalid or non-UTF-8 characters, or by a locale/environment mismatch inside the CNPG pod. Since the same command works outside the pod, the issue is probably environment-specific.

Here’s how to debug and resolve it:

• Add -vv to the archiveAdditionalCommandArgs in your ObjectStore spec to increase verbosity. This will help pinpoint which value is causing the encoding error.
• Exec into the CNPG pod and run env and locale to compare environment and locale settings with your standalone environment. Look for differences, especially in LANG, LC_ALL, or Python-related variables.
• Check all secrets and configmaps used for S3 credentials, endpoint URLs, and paths for any non-UTF-8 or invisible/surrogate characters. Even a stray whitespace or hidden character can cause this.
• If possible, print/log the actual WAL file names and S3 paths being processed at the time of the error.
• If you find a problematic value, sanitize or correct it in the relevant secret, configmap, or CRD.

If the issue persists after these checks, try creating a minimal test pod with only barman-cloud and the same environment to further isolate the source.

Increasing verbosity and carefully checking all environment/config values has helped others resolve similar subtle issues in the past, especially when the root cause was a hidden character or locale mismatch. See this example for how increased log detail revealed a whitespace in a secret value. The plugin passes values directly to the CLI without extra validation, so any encoding issue in the environment or config will surface as a Python UTF-8 error in the CLI process.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[sdaguet]

Hi @dosu,

Here are the elements:

• -vv was already tested, with no additional useful information found - does it apply to check-wal-archive or only to uploads ?
• no specific LANG/... have been set
• it is initial setup, the first wal is not able to upload

[sdaguet]

Hello,

I did some tests with no success.

1. Double check urls, credentials, ... no non utf-8 character or trailing space found
2. Update to latest development snapshot, not better
3. Activate trace logs, no more usable information found
4. Use other postgresl versions (ie 16)

Still the same kind of error in the logs.

{"level":"trace","ts":"2026-01-16T14:43:07.722709699Z","msg":"Executing barman-cloud-check-wal-archive","caller":"/go/pkg/mod/github.com/cloudnative-pg/barman-cloud@v0.4.1-0.20260108104508-ced266c145f5/pkg/walarchive/cmd.go:172","logging_pod":"sf-pg-cluster-1","options":["--endpoint-url","https://s3-xxx","--cloud-provider","aws-s3","s3://fra-bkp","pg-cluster"]}
{"level":"info","ts":"2026-01-16T14:43:08.65495736Z","logger":"barman-cloud-check-wal-archive","msg":"2026-01-16 14:43:08,654 [108] ERROR: Barman cloud WAL archive check exception: 'utf-8' codec can't encode characters in position 4-6: surrogates not allowed","pipe":"stderr","logging_pod":"pg-cluster-1"}

Then, I connected to postgres pod, adding credentials and certificate by env_vars, all sounds good :

• check is ok
• wal upload to s3 is ok

barman-cloud-check-wal-archive  --cloud-provider aws-s3 --endpoint-url https://s3-xxx s3://fra-bkp pg-cluster
echo $? -> 0
barman-cloud-wal-archive  --cloud-provider aws-s3 --endpoint-url https://s3-xxx s3://fra-bkp pg-cluster /var/lib/postgresql/wal/pg_wal/000000010000000000000001
echo $? -> 0

As plugin sidecar is distroless, didn't manage to execute the commands from it. Maybe I missed a trick.