From af67e0cbe82a62d2a759dd63b900461eeab97603 Mon Sep 17 00:00:00 2001 From: "mintlify[bot]" <109931778+mintlify[bot]@users.noreply.github.com> Date: Fri, 19 Jun 2026 14:45:23 +0000 Subject: [PATCH] docs: sync upstream clickhouse-operator docs updates --- .../guides/configuration.mdx | 517 +++++++++++++++++- .../guides/introduction.mdx | 4 +- .../kubernetes-operator/guides/monitoring.mdx | 378 +++++++++++++ .../kubernetes-operator/guides/scaling.mdx | 99 ++++ products/kubernetes-operator/guides/tls.mdx | 318 +++++++++++ products/kubernetes-operator/install/helm.mdx | 2 +- .../kubernetes-operator/install/kubectl.mdx | 10 +- products/kubernetes-operator/install/olm.mdx | 1 + products/kubernetes-operator/navigation.json | 5 +- products/kubernetes-operator/overview.mdx | 4 +- .../reference/api-reference.mdx | 69 ++- 11 files changed, 1362 insertions(+), 45 deletions(-) create mode 100644 products/kubernetes-operator/guides/monitoring.mdx create mode 100644 products/kubernetes-operator/guides/scaling.mdx create mode 100644 products/kubernetes-operator/guides/tls.mdx diff --git a/products/kubernetes-operator/guides/configuration.mdx b/products/kubernetes-operator/guides/configuration.mdx index 9f2cb468c..b9b60657f 100644 --- a/products/kubernetes-operator/guides/configuration.mdx +++ b/products/kubernetes-operator/guides/configuration.mdx @@ -87,6 +87,78 @@ spec: Operator can modify existing PVC only if the underlying storage class supports volume expansion. +### Multi-disk (JBOD) storage {#multi-disk-jbod-storage} + +`additionalVolumeClaimTemplates` attaches extra disks to each ClickHouse replica, on top of the primary `dataVolumeClaimSpec`, which is required to use them. +Each entry is a PVC template — a `metadata.name` plus a PVC `spec`. +The disks are reconciled exactly like the primary data disk — as StatefulSet `volumeClaimTemplates` — so the StatefulSet controller creates and retains one PVC per replica, named `--0`. + +```yaml +spec: + dataVolumeClaimSpec: + storageClassName: fast-ssd + resources: + requests: + storage: 100Gi + additionalVolumeClaimTemplates: + - metadata: + name: disk1 + spec: + storageClassName: fast-ssd + resources: + requests: + storage: 100Gi + - metadata: + name: disk2 + spec: + storageClassName: fast-ssd + resources: + requests: + storage: 100Gi +``` + +The operator mounts each additional volume at `/var/lib/clickhouse/disks/` and adds it to a generated ClickHouse storage configuration. +Hyphens in a name become underscores in the ClickHouse disk identifier; the mount path keeps the original name. + +The primary data disk and every additional disk are placed in a single volume of the `default` storage policy, so ClickHouse spreads new data parts across all of them in round-robin fashion. +Usable capacity is the sum of all disks, and every table that does not set its own `storage_policy` (including `system.*` tables) uses the combined set. + + +PVC names must match `^[a-z]([-a-z0-9]*[a-z0-9])?$` and must not collide with the primary data volume name. +Like the primary data disk, the set of additional disks is fixed at creation: adding, removing, or renaming entries after creation is rejected. +Additional PVCs are retained when the cluster is deleted, like the primary data disk. +Storage size on an existing entry can be expanded if the StorageClass supports expansion. + + +## Cluster domain {#cluster-domain} + +`spec.clusterDomain` sets the Kubernetes DNS suffix the operator uses when it builds +the fully-qualified pod host names it writes into the ClickHouse server +configuration. It defaults to `cluster.local` and exists on both +`ClickHouseCluster` and `KeeperCluster`. + +```yaml +spec: + clusterDomain: cluster.local # default; override only for a custom domain +``` + +The operator addresses every pod through the headless Service as +`...svc.`. That suffix flows into +two parts of the generated configuration: + +- On a `ClickHouseCluster`, its value is used for the replica host names in + `remote_servers` (cross-replica and `Distributed` queries). +- On a `KeeperCluster`, its value builds the Keeper node host names that + ClickHouse uses for coordination. + + +Only override this when your cluster's `kubelet` runs with a `--cluster-domain` +other than `cluster.local`. If the value does not match the real cluster domain, +ClickHouse cannot resolve the Keeper and replica host names — coordination and +`Distributed` queries fail with DNS resolution errors. Set the **same** value on the +`ClickHouseCluster` and the `KeeperCluster` it references. + + ## Pod configuration {#pod-configuration} ### Automatic topology spread and affinity {#automatic-topology-spread-and-affinity} @@ -129,12 +201,12 @@ For ClickHouse clusters with more than one shard, **one PDB is created per shard The operator picks safe defaults based on the cluster size so that a fresh `apply` already protects against accidental quorum loss. -| Resource | Topology | Default PDB | -|---|---|---| -| `ClickHouseCluster` | `replicas: 1` (single-replica shard) | `maxUnavailable: 1` — disruption is allowed for a single-node cluster so that node drains are not blocked | -| `ClickHouseCluster` | `replicas: 2+` (multi-replica shard) | `minAvailable: 1` — at least one replica per shard must stay up | -| `KeeperCluster` | `replicas: 1` | `maxUnavailable: 1` — disruption is allowed for a single-node cluster so that node drains are not blocked | -| `KeeperCluster` | `replicas: 3+` | `maxUnavailable: replicas/2` — preserves the RAFT quorum for a `2F+1` cluster (3 replicas tolerate 1 down, 5 replicas tolerate 2 down) | +| Resource | Topology | Default PDB | +|---------------------|--------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------| +| `ClickHouseCluster` | `replicas: 1` (single-replica shard) | `maxUnavailable: 1` — disruption is allowed for a single-node cluster so that node drains are not blocked | +| `ClickHouseCluster` | `replicas: 2+` (multi-replica shard) | `minAvailable: 1` — at least one replica per shard must stay up | +| `KeeperCluster` | `replicas: 1` | `maxUnavailable: 1` — disruption is allowed for a single-node cluster so that node drains are not blocked | +| `KeeperCluster` | `replicas: 3+` | `maxUnavailable: replicas/2` — preserves the RAFT quorum for a `2F+1` cluster (3 replicas tolerate 1 down, 5 replicas tolerate 2 down) | For a 3-shard ClickHouseCluster with `replicas: 3`, the operator creates three PDBs, one per shard, each with `minAvailable: 1`. @@ -176,11 +248,11 @@ spec: `spec.podDisruptionBudget.policy` lets you choose **how aggressively** the operator manages PDBs: -| Policy | Behavior | -|---|---| -| `Enabled` (default) | The operator creates and updates the PDB on every reconcile. This is the safe production default. | -| `Disabled` | The operator does **not** create PDBs and **deletes** any existing ones with matching labels. Useful for development clusters where every voluntary disruption should be allowed. | -| `Ignored` | The operator neither creates nor deletes PDBs. Existing PDBs are left alone. Use this when another system (e.g. policy admission, GitOps tool) owns PDB management for you. | +| Policy | Behavior | +|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `Enabled` (default) | The operator creates and updates the PDB on every reconcile. This is the safe production default. | +| `Disabled` | The operator does **not** create PDBs and **deletes** any existing ones with matching labels. Useful for development clusters where every voluntary disruption should be allowed. | +| `Ignored` | The operator neither creates nor deletes PDBs. Existing PDBs are left alone. Use this when another system (e.g. policy admission, GitOps tool) owns PDB management for you. | Example — disable PDB management completely on a development cluster: @@ -237,10 +309,10 @@ spec: resources: requests: cpu: "250m" - memory: "256Mi" + memory: "512Mi" limits: cpu: "1" - memory: "1Gi" + memory: "512Mi" ``` ### Environment variables {#environment-variables} @@ -293,10 +365,9 @@ spec: ### SSL certificate secret format {#ssl-certificate-secret-format} -It is expected that the Secret contains the following keys: +It is expected that the Secret contains the server keypair: - `tls.crt` - PEM encoded server certificate - `tls.key` - PEM encoded private key -- `ca.crt` - PEM encoded CA certificate chain This format is compatible with cert-manager generated certificates. @@ -306,10 +377,9 @@ This format is compatible with cert-manager generated certificates. If KeeperCluster has TLS enabled, ClickHouseCluster would use secure connection to Keeper nodes automatically. -ClickHouseCluster should be able to verify Keeper nodes certificates. -If ClickHouseCluster has TLS enabled, is uses `ca.crt` bundle for verification. Otherwise, default CA bundle is used. +ClickHouseCluster verifies Keeper node certificates against the system trust store, plus any `caBundle` you configure. -User may provide a custom CA bundle reference: +To trust a private CA (for example, a self-signed or internal CA), provide a custom CA bundle reference: ```yaml spec: @@ -320,35 +390,380 @@ spec: key: ``` +## External Secret {#external-secret} + +By default the operator creates and owns a Secret containing the cluster's internal credentials (interserver password, management password, keeper identity, cluster secret, named-collections key). The Secret is named after the cluster and lives in the cluster's namespace. + +If you want to manage these credentials yourself — for example, sourcing them from HashiCorp Vault, AWS Secrets Manager, or [External Secrets Operator](https://external-secrets.io/) — point the operator at a pre-existing Secret using `spec.externalSecret`: + +```yaml +apiVersion: clickhouse.com/v1alpha1 +kind: ClickHouseCluster +metadata: + name: sample +spec: + replicas: 2 + keeperClusterRef: + name: sample + dataVolumeClaimSpec: + resources: + requests: + storage: 10Gi + externalSecret: + name: my-clickhouse-credentials + policy: Observe +``` + + +The referenced Secret must reside in the **same namespace** as the ClickHouseCluster. The operator never deletes a Secret it did not create. + + +### Required keys {#external-secret-required-keys} + +The Secret must contain the following keys: + +| Key | Format | When required | +|-------------------------|--------------------------------------------|----------------------------| +| `interserver-password` | plaintext password | Always | +| `management-password` | plaintext password | Always | +| `keeper-identity` | `clickhouse:` | Always | +| `cluster-secret` | plaintext password | Always | +| `named-collections-key` | hex-encoded 16-byte AES key (32 hex chars) | ClickHouse `>= 25.12` only | + +A complete Secret looks like this: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: my-clickhouse-credentials + namespace: sample +type: Opaque +stringData: + interserver-password: "a-strong-random-password" + management-password: "another-strong-password" + keeper-identity: "clickhouse:keeper-auth-password" + cluster-secret: "cluster-internal-secret" + named-collections-key: "0123456789abcdef0123456789abcdef" # 32 hex chars = 16 bytes +``` + +### Policy: Observe vs Manage {#external-secret-policy} + +`spec.externalSecret.policy` controls how the operator handles missing required keys: + +| Policy | Behavior on missing keys | +|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `Observe` (default) | Reconciliation is **blocked** until every required key is present. The operator reports each missing key — and the format hint for it — via the `ExternalSecretValid` condition (with reason `ExternalSecretInvalid`) and a `Warning` event. | +| `Manage` | The operator **generates** any missing required keys and writes them back to the same Secret. Useful for bootstrapping: create an empty Secret, let the operator fill it, then optionally tighten access. The operator still never deletes the Secret. | + + +Even with `policy: Manage` the Secret must already exist in the namespace — the operator never creates the Secret itself, it only writes generated keys into an existing one. If the referenced Secret is missing, reconciliation is blocked with the `ExternalSecretNotFound` reason regardless of policy. + + +Pick `Observe` when an external system (Vault, ESO, sealed-secrets, GitOps) is the source of truth and you want the operator to fail loudly on misconfiguration. Pick `Manage` when you want self-sufficient bootstrapping but still want to retain ownership of the Secret object itself (for example, to back it up). + +### Status condition and troubleshooting {#external-secret-status} + +The operator exposes a `ExternalSecretValid` condition on `ClickHouseCluster.status.conditions`. Inspect it when reconciliation looks stuck: + +```bash +# Plain kubectl — works out of the box +kubectl describe clickhousecluster sample | sed -n '/Conditions:/,$p' + +# Same data as YAML +kubectl get clickhousecluster sample -o yaml | sed -n '/conditions:/,/^[^ ]/p' + +# Pretty-printed JSON (requires jq) +kubectl get clickhousecluster sample -o jsonpath='{.status.conditions}' | jq +``` + +Possible reasons: + +| `reason` | Meaning | Fix | +|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------| +| `ExternalSecretNotFound` | The referenced Secret does not exist in the namespace. | Create the Secret, or fix `spec.externalSecret.name`. | +| `ExternalSecretInvalid` | The Secret exists but lacks required keys (only with `Observe`). The message lists each missing key together with its expected format. | Add the missing keys, or switch to `policy: Manage`. | +| `ExternalSecretValid` | All required keys are present and the operator is using the Secret. | — | + +The operator requeues reconciliation while the Secret is invalid, so once you add the missing keys the next reconcile picks them up automatically — no need to bounce pods. + + +The set of required keys depends on the running ClickHouse version. `named-collections-key` is only validated once the operator's version probe has detected ClickHouse `25.12` or newer. On older versions the key may be absent from the Secret. + + +## Additional ports {#additional-ports} + +The operator exposes a fixed set of ports on every ClickHouse Pod and its headless Service: `8123` HTTP, `9000` native, `9009` interserver, `9001` management, `9363` Prometheus metrics, and the TLS variants `8443`/`9440` when TLS is enabled. To make ClickHouse listen on additional protocols — MySQL, PostgreSQL, gRPC, or any custom port — declare them in `spec.additionalPorts`: + +```yaml +spec: + additionalPorts: + - name: mysql + port: 9004 + - name: postgres + port: 9005 + - name: grpc + port: 9100 +``` + +The operator adds those ports to the Pod's `containerPorts` and to the headless Service. The complete example lives at [`examples/custom_protocols.yaml`](https://github.com/ClickHouse/clickhouse-operator/blob/main/examples/custom_protocols.yaml). + + +`additionalPorts` only opens the ports on the Kubernetes side. It does **not** configure the ClickHouse server to listen on them. You also have to enable the matching protocol in `spec.settings.extraConfig.protocols`. Without that, the port is open on the Service but nothing inside the pod is answering. + + +### End-to-end example: MySQL wire protocol {#additional-ports-mysql-example} + +To expose ClickHouse over the MySQL wire protocol on port `9004`: + +```yaml +apiVersion: clickhouse.com/v1alpha1 +kind: ClickHouseCluster +metadata: + name: sample +spec: + replicas: 1 + keeperClusterRef: + name: sample + dataVolumeClaimSpec: + resources: + requests: + storage: 2Gi + + # 1) Open the port on the Pod and the headless Service. + additionalPorts: + - name: mysql + port: 9004 + + # 2) Tell ClickHouse server to actually listen on it. + settings: + extraConfig: + protocols: + mysql: + type: mysql + port: 9004 + description: "MySQL wire protocol" +``` + +After applying, verify from inside the cluster: + +```bash +kubectl exec sample-clickhouse-0-0-0 -- \ + clickhouse-client --port 9004 --query "SELECT 1" +``` + +### Field constraints {#additional-ports-constraints} + +| Field | Rule | +|--------|------------------------------------------------------------------------------------------------------------------------------------------| +| `name` | Must match the DNS_LABEL pattern `^[a-z]([-a-z0-9]*[a-z0-9])?$`, max 63 characters. Uniqueness is enforced by the CRD as a list-map key. | +| `port` | Integer in `[1, 65535]`. The webhook rejects duplicate port numbers within the list. | + +### Reserved ports and names {#additional-ports-reserved} + +The validating webhook rejects `additionalPorts` entries that would collide with ports the operator binds itself. All TLS-related ports are reserved **unconditionally** so that flipping `spec.settings.tls.enabled` later cannot break a previously valid cluster. + +| Port | Reserved for | +|--------|--------------------| +| `8123` | HTTP | +| `8443` | HTTPS | +| `9000` | native TCP | +| `9440` | native TLS | +| `9009` | interserver | +| `9001` | management | +| `9363` | Prometheus metrics | + +The following names are also rejected — they are the operator's internal protocol-type identifiers (not the human-readable aliases): + +| Name | +|---------------| +| `http` | +| `http-secure` | +| `tcp` | +| `tcp-secure` | +| `interserver` | +| `management` | +| `prometheus` | + +A rejected request produces an error such as: + +``` +spec.additionalPorts[0].port: 8123 is reserved for the operator-managed HTTP port +spec.additionalPorts[0].name: "http" is reserved by the operator +``` + +## Version probe and upgrade channel {#version-probe-and-upgrade-channel} + +The operator does two independent things with cluster versions: + +1. **Version reporting** — for `ClickHouseCluster`, a Kubernetes `Job` runs the container image once to detect the running ClickHouse version; for `KeeperCluster`, the operator reads the server-reported version from running replicas. The detected version is recorded in `.status.version` and used by other reconciliation steps (e.g. the `External Secret` named-collections key is only required from ClickHouse `25.12`). +2. **Upgrade channel** — a periodic check against the public ClickHouse release feed (`https://clickhouse.com/data/version_date.tsv`). The operator reports whether a newer version is available via the `VersionUpgraded` status condition. It never upgrades the cluster on its own — the user is in control of the image tag. + +### Choosing a release channel {#upgrade-channel-choosing} + +`spec.upgradeChannel` selects which set of upstream releases the operator compares against. Same field exists on both `ClickHouseCluster` and `KeeperCluster`. + +```yaml +spec: + upgradeChannel: lts # or "stable", or "25.8", or omitted +``` + +Allowed values (validated by the CRD with the pattern `^(lts|stable|\d+\.\d+)?$`): + +| Value | Behavior | +|-----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| _empty_ (default) | The operator proposes only **minor** updates within the currently-running major.minor line. A cluster on `25.8.3.1` will be told about `25.8.4.x` but not `25.9.x`. | +| `stable` | Tracks the upstream `stable` channel — the latest release that ClickHouse Inc. flags as stable on the main release line. Receives major upgrades sooner than the `lts` channel. | +| `lts` | Tracks the upstream `lts` channel — long-term support releases. Receives major upgrades less frequently, with longer support windows. | +| `25.8` (or any `.`) | Pins the channel to a specific major.minor line. Major upgrades beyond it are not proposed even if a newer version exists upstream. | + +For production, pinning the channel to an explicit `.` (e.g. `25.8`) is generally preferred. It locks the cluster to the intended major release line and lets the operator surface a `WrongReleaseChannel` warning if any replica somehow drifts onto a different major — which matters especially when the image is referenced by a digest (`@sha256:...`) rather than by a human-readable tag. The empty default is fine for development clusters where major-version jumps are not a concern. + +### Status conditions {#version-status-conditions} + +Two conditions surface the result of the probe and the upgrade check: + +| Condition | Reason | Meaning | +|-------------------|------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `VersionInSync` | `VersionMatch` | All replicas report the same version | +| `VersionInSync` | `VersionMismatch` | Replicas are running different versions. This reason is suppressed during a planned rolling upgrade. It typically surfaces when a mutable image tag has been pinned (for example `latest` or a bare major like `26.3`) and the underlying registry has shifted between pulls, so different replicas ended up on different patches of the same tag. | +| `VersionInSync` | `VersionPending` | Version probe Job has not finished yet, or no Keeper replica version has been observed yet | +| `VersionInSync` | `VersionProbeFailed` | ClickHouse probe Job failed; the operator cannot determine the running version | +| `VersionUpgraded` | `UpToDate` | The cluster is on the latest version available in the selected channel | +| `VersionUpgraded` | `MinorUpdateAvailable` | A newer patch is available in the same `major.minor` line | +| `VersionUpgraded` | `MajorUpdateAvailable` | A newer `major.minor` is available within the chosen channel | +| `VersionUpgraded` | `VersionOutdated` | The running version is out of date and will no longer receive fixes from the selected channel — typically because the major line has been dropped from `lts` or `stable` upstream | +| `VersionUpgraded` | `WrongReleaseChannel` | The running image does not belong to the selected `upgradeChannel`. Example: a cluster running `26.5` with `upgradeChannel: lts`, since `26.5` is not part of the upstream `lts` line. | +| `VersionUpgraded` | `UpgradeCheckFailed` | The operator could not reach the upstream release feed | + +Inspect them with: + +```bash +kubectl get clickhousecluster sample -o yaml | sed -n '/conditions:/,/^[^ ]/p' +``` + +### Overriding the version probe Job {#version-probe-template} + +This applies to `ClickHouseCluster` only. `KeeperCluster` no longer runs a version-probe Job — its version is read directly from the running Keeper replicas — so `spec.versionProbeTemplate` is deprecated and has no effect there. + +The probe is implemented as a regular Kubernetes `Job`. If your cluster has admission policies that require specific Tolerations, node selectors, security contexts, or you want to limit how long completed probe Jobs linger, override the template via `spec.versionProbeTemplate`: + +```yaml +spec: + versionProbeTemplate: + spec: + ttlSecondsAfterFinished: 600 # delete completed probe Jobs 10 minutes after completion + template: + spec: + nodeSelector: + kubernetes.io/arch: amd64 + tolerations: + - key: dedicated + operator: Equal + value: clickhouse + effect: NoSchedule + containers: + - name: version-probe + resources: + requests: + cpu: 50m + memory: 64Mi +``` + +The container name `version-probe` is the operator's default — the entry under `containers:` matches it by name, so the operator deep-merges the user-provided fields on top of the defaults. + +### Operator-wide controls {#version-operator-flags} + +Two flags on the operator manager control the upgrade-check loop globally: + +| Flag | Default | Effect | +|-----------------------------------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------| +| `--version-update-interval` | `24h` | How often the operator re-fetches the upstream version list | +| `--disable-version-update-checks` | `false` | Disables the upgrade checker entirely. The `VersionUpgraded` condition is not set, and no outbound HTTP traffic to `clickhouse.com` is generated | + +Set `--disable-version-update-checks=true` in air-gapped environments or when egress to `clickhouse.com` is not allowed. + ## ClickHouse settings {#clickhouse-settings} ### Default user password {#default-user-password} -Set the default user password: +`spec.settings.defaultUserPassword` sets the password for the built-in `default` +user. Provide the value from a key in a Secret (recommended) or a ConfigMap that +you create, rather than inline in the CR: ```yaml spec: settings: defaultUserPassword: - passwordType: # Default: password - : - name: - key: + passwordType: password # default; see "Password types" below + secret: # exactly one of secret or configMap + name: clickhouse-password # name of the Secret/ConfigMap + key: password # the key inside it, not the password value +``` + +Provide exactly one of `secret` or `configMap`, each with both `name` (the object) +and `key` (the entry that holds the password). + +#### Password types {#password-types} + +`passwordType` tells ClickHouse how to interpret the value. It defaults to +`password` (plaintext); the alternatives are hashed forms such as +`password_sha256_hex` and `password_double_sha1_hex`. Prefer a hashed type so the +plaintext is never stored. See the +[ClickHouse user settings](https://clickhouse.com/docs/operations/settings/settings-users#user-namepassword) +for the full list. + +#### Full example with a Secret {#default-password-secret-example} + +Create the Secret, then reference its key: + +```bash +kubectl create secret generic clickhouse-password \ + --from-literal=password='your-secure-password' +``` + +```yaml +apiVersion: clickhouse.com/v1alpha1 +kind: ClickHouseCluster +metadata: + name: my-cluster +spec: + settings: + defaultUserPassword: + passwordType: password + secret: + name: clickhouse-password + key: password ``` -It isn't recommended to use ConfigMap to store plain text passwords. +With `passwordType: password`, the in-pod `clickhouse-client` is configured with +this password, which is handy for debugging. -Create the secret: +For a hashed password, store the hash instead of the plaintext: ```bash -kubectl create secret generic clickhouse-password --from-literal=password='your-secure-password' +echo -n 'your-secure-password' | sha256sum # use the hex digest as the value +kubectl create secret generic clickhouse-password \ + --from-literal=password='' ``` -#### Using ConfigMap for user passwords {#using-configmap-for-user-passwords} +```yaml +spec: + settings: + defaultUserPassword: + passwordType: password_sha256_hex + secret: + name: clickhouse-password + key: password +``` + +#### Using a ConfigMap {#using-configmap-for-user-passwords} -You can also use ConfigMap for non-sensitive default passwords: +A ConfigMap works the same way, but its contents are not protected like a Secret. +Use it only for non-sensitive or already-hashed values, such as a +`password_sha256_hex` digest: ```yaml spec: @@ -360,6 +775,11 @@ spec: key: default_password ``` + +Do not put a plaintext password in a ConfigMap. Use a Secret for any plaintext +(`passwordType: password`) value. + + ### Custom users in configuration {#custom-users-in-configuration} Configure additional users in configuration files. @@ -424,6 +844,47 @@ spec: When enabled, the operator synchronizes Replicated and integration tables to new replicas. +### Server logging {#server-logging} + +Configure the ClickHouse server log through `spec.settings.logger`. Every field is optional with a safe default, so a cluster you never touch already logs at `trace` to both the container console and a rotated file on disk. + +```yaml +spec: + settings: + logger: + logToFile: true # Default: true. Set false to log only to the console + jsonLogs: false # Default: false. Set true for structured JSON log lines + level: trace # Default: trace + size: 1000M # Default: 1000M. Rotate a log file once it reaches this size + count: 50 # Default: 50. Number of rotated files to keep +``` + +| Field | Default | Description | +|-------------|---------|-------------------------------------------------------------------------------------------------------------------| +| `logToFile` | `true` | When `false`, the operator drops the file targets and the server logs only to the container console. | +| `jsonLogs` | `false` | When `true`, the operator adds `formatting.type: json` so each line is a JSON object. | +| `level` | `trace` | Log verbosity. One of `test`, `trace`, `debug`, `information`, `notice`, `warning`, `error`, `critical`, `fatal`. | +| `size` | `1000M` | Maximum size of a single log file before rotation. | +| `count` | `50` | Number of rotated log files the server retains. | + +The operator always keeps console logging on so that `kubectl logs` works, and layers file logging on top when `logToFile` is `true`. A cluster with the defaults renders this `logger` block: + +```yaml +logger: + console: true + level: trace + log: /var/log/clickhouse-server/clickhouse-server.log + errorlog: /var/log/clickhouse-server/clickhouse-server.err.log + size: 1000M + count: 50 +``` + +The same `spec.settings.logger` block applies to a `KeeperCluster`; the operator writes its files under `/var/log/clickhouse-keeper/` instead. + + +Console logging stays on regardless of `logToFile`, so `kubectl logs` keeps working even when you disable file logging. Set `jsonLogs: true` when you ship logs to a structured log store that parses JSON. + + ## Custom configuration {#custom-configuration} ### Embedded extra configuration {#embedded-extra-configuration} diff --git a/products/kubernetes-operator/guides/introduction.mdx b/products/kubernetes-operator/guides/introduction.mdx index b2a26d628..547e73eea 100644 --- a/products/kubernetes-operator/guides/introduction.mdx +++ b/products/kubernetes-operator/guides/introduction.mdx @@ -169,10 +169,10 @@ To completely remove storage: kubectl delete clickhousecluster my-cluster # Wait for pods to terminate -kubectl wait --for=delete pod -l app.kubernetes.io/instance=my-cluster +kubectl wait --for=delete pod -l app.kubernetes.io/instance=my-cluster-clickhouse # Delete PVCs -kubectl delete pvc -l app.kubernetes.io/instance=sample-cluster +kubectl delete pvc -l app.kubernetes.io/instance=my-cluster-clickhouse ``` ## Default configuration highlights {#default-configuration-highlights} diff --git a/products/kubernetes-operator/guides/monitoring.mdx b/products/kubernetes-operator/guides/monitoring.mdx new file mode 100644 index 000000000..15416801a --- /dev/null +++ b/products/kubernetes-operator/guides/monitoring.mdx @@ -0,0 +1,378 @@ +--- +position: 3 +slug: /clickhouse-operator/guides/monitoring +title: Monitoring the ClickHouse Operator +keywords: ['kubernetes', 'prometheus', 'monitoring', 'metrics'] +description: 'How to scrape, secure, and use the operator metrics and health endpoints.' +doc_type: 'guide' +--- + +The operator exposes Prometheus-compatible metrics and Kubernetes health probes so that you can observe its reconciliation activity, detect stalled controllers, and alert on failures. + +This guide covers what the operator exposes, how to scrape it, and which queries are useful day to day. + + +This guide is about the **operator process itself** (the controller manager). For ClickHouse server metrics (queries, parts, replication lag), use the [Prometheus endpoint in ClickHouse](/reference/settings/server-settings/settings#prometheus) to scrape it separately. + + +## Endpoints {#endpoints} + +The operator process exposes two HTTP endpoints inside the manager pod: + +| Endpoint | Default port | Path | Purpose | +|---|---|---|---| +| Metrics | `8080` (Helm) / `0` disabled (binary default) | `/metrics` | Prometheus exposition format | +| Health probe | `8081` | `/healthz`, `/readyz` | Kubernetes liveness and readiness | + +The metrics endpoint is **off by default** when running the operator binary directly (`--metrics-bind-address=0`). The Helm chart turns it on with `metrics.enable: true` and `metrics.port: 8080`. + +The health probe endpoint is always on; the deployment template wires `/healthz` and `/readyz` to the pod's liveness and readiness probes on port `8081`. + +## Operator binary flags {#operator-binary-flags} + +The relevant `manager` flags (defined in [`cmd/main.go`](https://github.com/ClickHouse/clickhouse-operator/blob/main/cmd/main.go)): + +| Flag | Default | Description | +|---|---|---| +| `--metrics-bind-address` | `0` (disabled) | Bind address for the metrics endpoint. Set to `:8443` for HTTPS or `:8080` for HTTP. Leave as `0` to disable the metrics server. | +| `--metrics-secure` | `true` | Serve metrics over HTTPS with authn/authz. Set to `false` for plain HTTP. | +| `--metrics-cert-path` | empty | Directory containing TLS cert files (`tls.crt`, `tls.key`) for the metrics server. | +| `--metrics-cert-name` | `tls.crt` | Cert file name inside `--metrics-cert-path`. | +| `--metrics-cert-key` | `tls.key` | Key file name inside `--metrics-cert-path`. | +| `--enable-http2` | `false` | Enable HTTP/2 for the metrics **and webhook** servers. Off by default to mitigate CVE-2023-44487 / CVE-2023-39325. | +| `--leader-elect` | `false` (binary) / `true` (Helm chart) | Enable leader election so only one replica reconciles at a time. The Helm chart sets this flag in `manager.args` by default. | +| `--health-probe-bind-address` | `:8081` | Bind address for `/healthz` and `/readyz`. | + + +The `8443` (HTTPS) / `8080` (HTTP) convention in the flag's help text is only a hint. The Helm chart serves HTTPS on `8080` because it sets both `metrics.port: 8080` and `metrics.secure: true`. There is no port-based mode detection — `--metrics-secure` is what selects HTTPS or HTTP. + + +## Enable metrics via Helm {#enable-metrics-via-helm} + +The chart already creates a `Service` for the metrics port and, optionally, a `ServiceMonitor` for prometheus-operator. + +The metrics endpoint itself is on by default (`metrics.enable: true`, port `8080`, served over HTTPS via `metrics.secure: true`). The only setting you typically need to flip is `prometheus.enable` to have the chart create a `ServiceMonitor` for you: + +```yaml +# values.yaml — minimal override +prometheus: + enable: true +``` + +If you do not use cert-manager, additionally set `certManager.enable: false` and the ServiceMonitor will scrape with `insecureSkipVerify: true`, relying on bearer-token authentication only. + +The full set of metrics-related defaults is: + +```yaml +metrics: + enable: true + port: 8080 + secure: true # HTTPS with authn/authz enforced on every scrape + +certManager: + enable: true # Issues the metrics server certificate + +prometheus: + enable: false # Set to true to render the ServiceMonitor + scraping_annotations: false # Alternative: prometheus.io/scrape pod annotations +``` + +Apply: + +```bash +helm upgrade --install clickhouse-operator \ + oci://ghcr.io/clickhouse/clickhouse-operator-helm \ + -n clickhouse-operator-system --create-namespace \ + -f values.yaml +``` + +After install the chart creates: + +- `Service/-metrics-service` — exposes port `8080` (HTTPS when `metrics.secure: true`). +- `ServiceMonitor/-controller-manager-metrics-monitor` — when `prometheus.enable: true`. +- `ClusterRole/-metrics-reader` — non-resource URL `/metrics` with `get` verb. + +## Securing the metrics endpoint {#securing-the-metrics-endpoint} + +When `metrics.secure: true` the metrics server enforces TLS **and** Kubernetes authentication/authorization on every scrape. Scrapers must: + +1. Present a valid Kubernetes bearer token. +2. Belong to a ServiceAccount bound to a ClusterRole granting `get` on the non-resource URL `/metrics`. + +The chart ships such a ClusterRole: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: clickhouse-operator-metrics-reader +rules: + - nonResourceURLs: + - /metrics + verbs: + - get +``` + +Bind it to the ServiceAccount used by your scraper (typically Prometheus): + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: prometheus-clickhouse-operator-metrics-reader +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: clickhouse-operator-metrics-reader +subjects: + - kind: ServiceAccount + name: + namespace: +``` + + +If you see `401 Unauthorized` or `403 Forbidden` from the metrics endpoint, the scraper is using HTTPS but is missing/unauthorized for a Kubernetes bearer token, or its ServiceAccount lacks the binding above. Disabling security by setting `metrics.secure: false` is **not recommended** in shared clusters because anyone with network reachability to the pod could scrape the endpoint. + + +## ServiceMonitor reference {#servicemonitor-reference} + +The chart renders a ServiceMonitor of this shape when `prometheus.enable: true`: + +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: -controller-manager-metrics-monitor + namespace: + labels: + control-plane: controller-manager +spec: + selector: + matchLabels: + control-plane: controller-manager + endpoints: + - path: /metrics + port: https # "http" when metrics.secure: false + scheme: https + bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + tlsConfig: + serverName: -metrics-service..svc + ca: + secret: + name: metrics-server-cert + key: ca.crt + cert: + secret: + name: metrics-server-cert + key: tls.crt + keySecret: + name: metrics-server-cert + key: tls.key +``` + +If your Prometheus instance does not run cert-manager, set `tlsConfig.insecureSkipVerify: true` and rely on bearer-token authentication only — the chart already does this when `certManager.enable: false`. + +## Standalone Prometheus example {#standalone-prometheus-example} + +If you do not use kube-prometheus-stack, the repository ships a self-contained example at [`examples/prometheus_secure_metrics_scraper.yaml`](https://github.com/ClickHouse/clickhouse-operator/blob/main/examples/prometheus_secure_metrics_scraper.yaml). It creates a ServiceAccount, the necessary RBAC, and a `Prometheus` CR that selects the operator's ServiceMonitor. + +## Health probe endpoints {#health-probe-endpoints} + +| Path | Used by | Returns | +|---|---|---| +| `/healthz` | Kubernetes liveness probe | `200 OK` as long as the probe server is listening. | +| `/readyz` | Kubernetes readiness probe | `200 OK` as long as the probe server is listening. | + +Both endpoints are registered with the same trivial ping check (`healthz.Ping` from `sigs.k8s.io/controller-runtime`). A failing probe therefore means "the manager process is not serving HTTP on `:8081`" — not "controllers are unhealthy". To detect controller-level problems, use the [reconciliation metrics](#reconciliation-activity) instead. + +Both endpoints are served on port `8081` by default. They are wired to the deployment as: + +```yaml +livenessProbe: + httpGet: + path: /healthz + port: 8081 + initialDelaySeconds: 15 + periodSeconds: 20 +readinessProbe: + httpGet: + path: /readyz + port: 8081 + initialDelaySeconds: 5 + periodSeconds: 10 +``` + +A repeatedly failing probe usually means the probe server itself never came up — for example, the manager exited early during startup. Check the manager logs for `unable to start manager`, RBAC failures, or `cache did not sync` errors. + +## Metrics catalog {#metrics-catalog} + +The operator does not register custom Prometheus collectors. Everything below is exposed by the underlying `controller-runtime` and `client-go` libraries. The most useful series, grouped by purpose: + +### Reconciliation activity {#reconciliation-activity} + +| Metric | Type | Labels | +|---|---|---| +| `controller_runtime_reconcile_total` | counter | `controller`, `result` (`success` / `error` / `requeue` / `requeue_after`) | +| `controller_runtime_reconcile_errors_total` | counter | `controller` | +| `controller_runtime_reconcile_time_seconds_bucket` | histogram | `controller` | +| `controller_runtime_active_workers` | gauge | `controller` | +| `controller_runtime_max_concurrent_reconciles` | gauge | `controller` | + +The `controller` label is derived by `controller-runtime` from the resource type registered with `For(...)`. With the current code in `internal/controller/clickhouse` and `internal/controller/keeper` this resolves to `clickhousecluster` and `keepercluster` respectively. If you have customized the operator, verify with a one-time scrape of `/metrics`. + +### Work queue {#work-queue} + +| Metric | Type | Labels | +|---|---|---| +| `workqueue_depth` | gauge | `name`, `controller`, `priority` | +| `workqueue_adds_total` | counter | `name`, `controller` | +| `workqueue_retries_total` | counter | `name`, `controller` | +| `workqueue_unfinished_work_seconds` | gauge | `name`, `controller` | +| `workqueue_longest_running_processor_seconds` | gauge | `name`, `controller` | +| `workqueue_queue_duration_seconds_bucket` | histogram | `name`, `controller` | +| `workqueue_work_duration_seconds_bucket` | histogram | `name`, `controller` | + +The `name` and `controller` labels carry the same value (the controller name). + +### API server traffic {#api-server-traffic} + +| Metric | Type | Labels | +|---|---|---| +| `rest_client_requests_total` | counter | `code`, `method`, `host` | + +### Leader election {#leader-election} + +| Metric | Type | Labels | +|---|---|---| +| `leader_election_master_status` | gauge | `name` (= `d4ceba06.clickhouse.com`) | + +The Helm chart enables `--leader-elect` by default, so this metric is present in standard Helm installs. When running the binary directly without the flag, the metric is absent. + +### Runtime {#runtime} + +Standard Go process and runtime collectors — `go_goroutines`, `go_memstats_*`, `process_cpu_seconds_total`, `process_resident_memory_bytes`, etc. + +## Useful PromQL queries {#useful-promql-queries} + +### Health overview + +```promql +# Reconciliation rate per controller +sum by (controller) (rate(controller_runtime_reconcile_total[5m])) + +# Error rate per controller (alert if > 0 sustained) +sum by (controller) (rate(controller_runtime_reconcile_errors_total[5m])) + +# p99 reconcile latency +histogram_quantile( + 0.99, + sum by (le, controller) (rate(controller_runtime_reconcile_time_seconds_bucket[5m])) +) +``` + +### Backlog detection + +```promql +# Pending items in the work queue — a sustained value > 0 indicates a backlog, +# but short spikes during large reconciles are normal. +avg_over_time(workqueue_depth[10m]) + +# Reconciles that have been running for a long time +workqueue_longest_running_processor_seconds > 60 +``` + +### Throttling and API pressure + +```promql +# Throttled requests to the API server +sum by (code, host) (rate(rest_client_requests_total{code=~"4..|5.."}[5m])) +``` + +### Leader status (HA deployment) + +```promql +# Should be exactly 1 across the replica set (Helm install enables --leader-elect by default) +sum(leader_election_master_status{name="d4ceba06.clickhouse.com"}) +``` + +## Suggested alerts {#suggested-alerts} + +Starting point for a PrometheusRule (tune thresholds for your environment): + +```yaml +groups: + - name: clickhouse-operator + rules: + - alert: ClickHouseOperatorReconcileErrors + # > 0.1 errors/s sustained = > ~6 errors/min, filters transient conflicts. + expr: sum by (controller) (rate(controller_runtime_reconcile_errors_total[5m])) > 0.1 + for: 15m + labels: + severity: warning + annotations: + summary: 'ClickHouse operator is failing to reconcile {{ $labels.controller }}' + + - alert: ClickHouseOperatorWorkqueueBacklog + # avg_over_time avoids alerting on transient bursts during large reconciles. + expr: avg_over_time(workqueue_depth[10m]) > 5 + for: 30m + labels: + severity: warning + annotations: + summary: 'Operator work queue backlog sustained for 30m' + + - alert: ClickHouseOperatorReconcileSlow + expr: | + histogram_quantile( + 0.99, + sum by (le, controller) (rate(controller_runtime_reconcile_time_seconds_bucket[10m])) + ) > 30 + for: 15m + labels: + severity: warning + annotations: + summary: 'p99 reconcile latency for {{ $labels.controller }} > 30s' + + - alert: ClickHouseOperatorNoLeader + expr: absent(leader_election_master_status{name="d4ceba06.clickhouse.com"}) == 1 + for: 5m + labels: + severity: critical + annotations: + summary: 'No leader for the ClickHouse operator (HA deployment)' +``` + +The last rule is only meaningful when leader election is enabled. + +## Verifying the setup {#verifying-the-setup} + +A quick end-to-end check, assuming the chart was installed in `clickhouse-operator-system`: + +```bash +NS=clickhouse-operator-system + +# The metrics Service exists and selects the manager pod +kubectl -n $NS get svc -l control-plane=controller-manager + +# The ServiceMonitor exists (only with prometheus.enable=true) +kubectl -n $NS get servicemonitor -l control-plane=controller-manager + +# Manager pod is Ready (readiness probe answers) +kubectl -n $NS get pod -l control-plane=controller-manager + +# Direct scrape from inside the cluster (with the metrics-reader binding) +kubectl -n $NS run curl-metrics --rm -it --restart=Never \ + --image=curlimages/curl:8.10.1 -- sh -c ' + TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) + curl -sk -H "Authorization: Bearer $TOKEN" \ + https://-metrics-service.'$NS'.svc:8080/metrics \ + | head -20 + ' +``` + +If the scrape returns metrics in the Prometheus exposition format, the endpoint and RBAC are correctly wired. + +## Related guides {#related-guides} + +- [Installation](/products/kubernetes-operator/install/helm) — Helm values relevant to monitoring. +- [Configuration](/products/kubernetes-operator/guides/configuration) — TLS configuration shared with the metrics server. diff --git a/products/kubernetes-operator/guides/scaling.mdx b/products/kubernetes-operator/guides/scaling.mdx new file mode 100644 index 000000000..a5a5c6ca1 --- /dev/null +++ b/products/kubernetes-operator/guides/scaling.mdx @@ -0,0 +1,99 @@ +--- +position: 4 +slug: /clickhouse-operator/guides/scaling +title: Scaling clusters +keywords: ['kubernetes', 'scaling', 'replicas', 'shards', 'keeper', 'quorum'] +description: 'How to scale ClickHouse replicas and shards and Keeper quorum members, and what the operator does automatically.' +doc_type: 'guide' +--- + +You scale a cluster by editing the replica and shard counts on the Custom Resource. The operator reconciles the running cluster toward the new topology: it creates or removes the per-replica StatefulSets, keeps the schema in sync, and surfaces progress through status conditions. + +This guide covers how to scale `ClickHouseCluster` replicas and shards, how to scale a `KeeperCluster` quorum safely, and which conditions to watch while a scale operation is in flight. + + +A `ClickHouseCluster` always needs a Keeper, referenced through the required `spec.keeperClusterRef` field — the operator coordinates the cluster through it regardless of size. To run more than one replica per shard, the data must also live in `ReplicatedMergeTree` tables, since replication is what lets a second replica serve the same rows. + + +## Scaling replicas {#scaling-replicas} + +`spec.replicas` sets the number of replicas in every shard. Each replica runs in its own StatefulSet named `-clickhouse--`, so a cluster with `shards: 2` and `replicas: 3` runs six StatefulSets. + +Raise or lower the count in place: + +```yaml +spec: + replicas: 3 # was 1 + keeperClusterRef: + name: my-keeper +``` + +On scale up the operator creates the new per-replica StatefulSets, waits for each pod to become ready, and then synchronizes the schema to the new replicas (see [Automatic schema sync](#automatic-schema-sync)). On scale down it removes the surplus StatefulSets and cleans up the stale replicated-database replica registrations the removed replicas left behind. + +## Scaling shards {#scaling-shards} + +`spec.shards` sets the number of shards. Each new shard adds a full set of per-replica StatefulSets, and the operator creates one [PodDisruptionBudget per shard](/products/kubernetes-operator/guides/configuration#pod-disruption-budgets) so a disruption in one shard cannot count against another. + +```yaml +spec: + shards: 3 # was 1 + replicas: 2 +``` + +Each shard holds a distinct slice of the data, and the operator does not copy or move rows between shards. A `Distributed` table or an explicit routing scheme decides which shard a row lands on, so adding a shard gives new writes somewhere to land without touching the rows already stored in the existing shards. + +## Automatic schema sync {#automatic-schema-sync} + +When `spec.settings.enableDatabaseSync` is `true` (the default), the operator keeps the schema aligned as the topology changes: + +- **On scale up** — once at least two replicas are ready, the operator replicates the database definitions to the newly created replicas, so a fresh replica joins with the same `Replicated` and integration databases as the rest of the cluster. +- **On scale down** — before a replica disappears, the operator drops that replica's registration from each `Replicated` database with `SYSTEM DROP DATABASE REPLICA`, so the shrunk cluster does not wait on a `Replicated` database replica that no longer exists. + +This covers `Replicated` databases and integration database engines. It does not move table data — row data lives in `ReplicatedMergeTree` tables and replicates through Keeper independently of this schema sync. With a single ready replica there is nothing to replicate to, so the operator skips the step and logs that it has no target. + +Set `enableDatabaseSync: false` to turn the behavior off, for example when an external tool owns schema propagation. The operator then reports the `SchemaSyncDisabled` reason on the `SchemaInSync` condition. + +## Conditions to watch {#scaling-conditions} + +Inspect progress on the Custom Resource while a scale operation runs: + +```bash +kubectl get clickhousecluster sample -o yaml | sed -n '/conditions:/,/^[^ ]/p' +``` + +| Condition | Reason | Meaning | +|---|---|---| +| `ClusterSizeAligned` | `UpToDate` | Running replica count matches the requested topology | +| `ClusterSizeAligned` | `ScalingUp` | The operator is adding replicas | +| `ClusterSizeAligned` | `ScalingDown` | The operator is removing replicas | +| `SchemaInSync` | `ReplicasInSync` | Databases exist on all replicas and stale metadata is cleaned up | +| `SchemaInSync` | `DatabasesNotCreated` | The operator has not finished creating databases on the new replicas | +| `SchemaInSync` | `ReplicasNotCleanedUp` | Stale replica metadata from a scale down is not yet removed | +| `SchemaInSync` | `SchemaSyncDisabled` | `enableDatabaseSync` is `false` | +| `Ready` | `AllShardsReady` | Every shard has a ready replica | +| `Ready` | `SomeShardsNotReady` | At least one shard has no ready replica | + +A scale operation is complete when `ClusterSizeAligned` reports `UpToDate`, `SchemaInSync` reports `ReplicasInSync`, and `Ready` reports `AllShardsReady`. + +## Scaling Keeper {#scaling-keeper} + +A `KeeperCluster` runs a RAFT quorum, so the operator changes its membership **one replica at a time** and only while the cluster is in a stable state. This protects the quorum: a `2F+1` cluster tolerates `F` members down, so a 3-node cluster keeps working with one member missing and a 5-node cluster with two. + +```yaml +spec: + replicas: 5 # was 3 +``` + +On scale up the operator adds the lowest free replica ID to the quorum; on scale down it removes the highest ID. Each step waits for the quorum to settle before the next one starts. The [Keeper PodDisruptionBudget](/products/kubernetes-operator/guides/configuration#pod-disruption-budgets) defaults to `maxUnavailable: replicas/2` to preserve the quorum during voluntary disruptions. + +The `ScaleAllowed` condition reports whether the quorum can change membership right now: + +| Reason | Meaning | +|---|---| +| `ReadyToScale` | The quorum is stable and the operator can add or remove a member | +| `ReplicaHasPendingChanges` | A replica still has a pending configuration change | +| `ReplicaNotReady` | A replica is not ready, so membership changes wait | +| `NoQuorum` | The cluster has no quorum and cannot change membership safely | +| `WaitingFollowers` | The operator is waiting for followers to catch up | + +Scale Keeper in single steps and let `ScaleAllowed` return to `ReadyToScale` between changes. Jumping several members at once does not bypass the one-at-a-time reconcile — the operator still walks the quorum one member per step. diff --git a/products/kubernetes-operator/guides/tls.mdx b/products/kubernetes-operator/guides/tls.mdx new file mode 100644 index 000000000..d87a23d37 --- /dev/null +++ b/products/kubernetes-operator/guides/tls.mdx @@ -0,0 +1,318 @@ +--- +position: 5 +slug: /clickhouse-operator/guides/tls +title: Securing a cluster with TLS +keywords: ['kubernetes', 'tls', 'ssl', 'cert-manager', 'security', 'certificates'] +description: 'How to secure a ClickHouse cluster with TLS using cert-manager, including client connections and Keeper encryption.' +doc_type: 'guide' +--- + +This guide walks through encrypting a ClickHouse cluster end to end: issuing a +certificate with [cert-manager](https://cert-manager.io/), enabling TLS on the +cluster, connecting a client over the secure ports, and extending encryption to +Keeper coordination traffic. + +It is task oriented. For the field-by-field reference of `spec.settings.tls`, see +[Configuration → TLS/SSL configuration](/products/kubernetes-operator/guides/configuration#tls-ssl-configuration) +and the [API Reference](/products/kubernetes-operator/reference/api-reference#clustertlsspec). + +## Prerequisites {#prerequisites} + +- A running ClickHouse cluster managed by the operator (see [Introduction](/products/kubernetes-operator/guides/introduction)). +- [cert-manager](https://cert-manager.io/docs/installation/) installed in the cluster. +- `kubectl` access to the cluster's namespace. + +The operator does not generate certificates itself — it consumes a Kubernetes +`Secret` that you provide. cert-manager is the recommended way to produce and +rotate that Secret, but any tool that writes a Secret in the expected format works. + +## How the operator expects certificates {#secret-format} + +TLS is enabled by pointing `spec.settings.tls.serverCertSecret` at a Secret that +contains the server keypair: + +| Secret key | Contents | Required | +|------------|--------------------------------|----------| +| `tls.crt` | PEM-encoded server certificate | Yes | +| `tls.key` | PEM-encoded private key | Yes | + +This is exactly the layout cert-manager writes for a `Certificate` resource, so no +conversion is needed. The operator mounts the keypair into each pod at +`/etc/clickhouse-server/tls/` and wires it into ClickHouse's `openSSL` configuration. + + +`serverCertSecret` is **mandatory** when `tls.enabled: true`. The validating +webhook rejects a cluster that enables TLS without it, and rejects `required: true` +unless `enabled: true`. + + +## Step 1 — Bootstrap a CA with cert-manager {#step-1-ca} + +The most reproducible setup is a self-signed CA that then signs the server +certificate. This gives you a stable `ca.crt` that clients can trust. + +```yaml +# A self-signed issuer used only to mint the CA certificate +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: selfsigned-bootstrap + namespace: +spec: + selfSigned: {} +--- +# The CA certificate itself +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: clickhouse-ca + namespace: +spec: + isCA: true + commonName: clickhouse-ca + secretName: clickhouse-ca + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-bootstrap + kind: Issuer +--- +# A CA issuer that signs leaf certificates from the CA above +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: clickhouse-ca-issuer + namespace: +spec: + ca: + secretName: clickhouse-ca +``` + +In production, replace the self-signed bootstrap with your real issuer (a +corporate CA, Vault, ACME, etc.). Only Step 2 changes — the cluster wiring is +identical. + +## Step 2 — Issue the server certificate {#step-2-cert} + +Request a leaf certificate from the CA issuer. The `dnsNames` must cover how +clients address the pods. The operator creates a single **headless** Service named +`-clickhouse-headless`, and each replica pod is addressable at +`-clickhouse---0.-clickhouse-headless..svc.cluster.local`. +A wildcard over the headless service domain covers every replica: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: clickhouse-server + namespace: +spec: + secretName: clickhouse-cert # <-- the Secret the operator will read + duration: 8760h # 1 year + renewBefore: 720h # rotate 30 days early + issuerRef: + name: clickhouse-ca-issuer + kind: Issuer + dnsNames: + - "*.-clickhouse-headless..svc" + - "*.-clickhouse-headless..svc.cluster.local" + - "localhost" +``` + + +The operator does **not** create a cluster-wide (load-balanced) Service. If you +want a single stable endpoint to connect to, create your own `ClusterIP` Service +selecting the cluster's pods and add its DNS name to `dnsNames` above. + + +cert-manager creates the `clickhouse-cert` Secret with `tls.crt`, `tls.key`, and +`ca.crt`, and refreshes it before expiry. Verify it exists: + +```bash +kubectl -n get secret clickhouse-cert -o jsonpath='{.data}' | jq 'keys' +# ["ca.crt","tls.crt","tls.key"] +``` + +## Step 3 — Enable TLS on the cluster {#step-3-enable} + +Point the cluster at the Secret: + +```yaml +apiVersion: clickhouse.com/v1alpha1 +kind: ClickHouseCluster +metadata: + name: + namespace: +spec: + settings: + tls: + enabled: true + required: true # disable the insecure ports entirely + serverCertSecret: + name: clickhouse-cert +``` + +### What the operator does {#what-the-operator-does} + +When `tls.enabled: true`, the operator: + +- **Opens the secure ports** on every pod and the headless Service: `9440` + (native TLS) and `8443` (HTTPS). These are added alongside the existing ports. +- **Mounts the Secret** at `/etc/clickhouse-server/tls/` and generates the + ClickHouse `openSSL` block with `verificationMode: relaxed`, + `disableProtocols: sslv2,sslv3`, and `preferServerCiphers: true`. These are + defaults — see [Customizing the TLS settings](#custom-tls-settings) to override them. + +When you also set `required: true`, the operator additionally: + +- **Removes the insecure ports** `9000` (native) and `8123` (HTTP) — only the TLS + variants remain, so plaintext clients can no longer connect. +- **Switches the pod liveness probe** to the secure native port `9440`, so health + checking continues to work without a plaintext listener. + + +The TLS ports `8443` and `9440` are reserved by the webhook **unconditionally**, +even when TLS is off, so toggling `tls.enabled` later never collides with a +`spec.additionalPorts` entry. See +[Configuration → `additionalPorts`](/products/kubernetes-operator/guides/configuration#additional-ports). + + +## Step 4 — Connect over TLS {#step-4-connect} + +With `required: true`, clients must use the secure ports and trust the CA. Address +a specific replica pod through the headless Service (or your own `ClusterIP` +Service if you created one). + +**Native protocol** (`clickhouse-client`, port `9440`): + +```bash +clickhouse-client --secure \ + --host -clickhouse-0-0-0.-clickhouse-headless..svc.cluster.local \ + --port 9440 \ + --ca-certificate /path/to/ca.crt \ + --query "SELECT 1" +``` + +**HTTPS** (port `8443`): + +```bash +curl --cacert /path/to/ca.crt \ + "https://-clickhouse-0-0-0.-clickhouse-headless..svc.cluster.local:8443/?query=SELECT%201" +``` + +Pull `ca.crt` straight from the Secret for local testing: + +```bash +kubectl -n get secret clickhouse-cert \ + -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt +``` + +## Encrypting Keeper traffic {#keeper-tls} + +Enabling TLS on the ClickHouse cluster does **not** encrypt the link to Keeper. +Enable it on the `KeeperCluster` independently — issue a certificate for the Keeper +service (Steps 1–2 with the Keeper service `dnsNames`) and reference it: + +```yaml +apiVersion: clickhouse.com/v1alpha1 +kind: KeeperCluster +metadata: + name: + namespace: +spec: + settings: + tls: + enabled: true + required: true + serverCertSecret: + name: keeper-cert +``` + +Keeper exposes its secure client port on `2281`. Once Keeper has TLS enabled, **the +ClickHouse cluster connects to it over TLS automatically** — no extra setting on the +ClickHouseCluster side. ClickHouse verifies the Keeper certificate against the system +trust store, plus any [`caBundle`](#custom-ca) you configure. + +## Custom CA bundle {#custom-ca} + +By default ClickHouse verifies the peers it connects to (other replicas, Keeper, HTTPS +dictionary sources, S3, …) against the **system trust store**. To **additionally** trust +a private CA — a self-signed or internal CA whose root is not in the system store — +supply a `caBundle`: + +```yaml +spec: + settings: + tls: + enabled: true + serverCertSecret: + name: clickhouse-cert + caBundle: + name: + key: ca.crt +``` + +The operator mounts this bundle and adds it to the `openSSL` client trust store +(`caConfig`). The system trust store stays in effect — your private CA is trusted **in +addition to** the public roots, so connections to public endpoints keep working. For a +self-signed setup, point `caBundle` at the `ca.crt` key of the same Secret cert-manager +wrote (as in the `cluster_with_ssl` example). + +## Customizing the TLS settings {#custom-tls-settings} + +The `openSSL` block the operator generates is a default, not a ceiling. It is written +into the main server configuration; anything under `spec.settings.extraConfig` is rendered to +`config.d/99-extra-config.yaml`, which ClickHouse merges **last** — so it overrides the +generated values. + +To harden the defaults — for example, require strict peer verification and raise the +minimum protocol to TLS 1.2 — set the `openSSL.server` keys you want to change: + +```yaml +spec: + settings: + extraConfig: + openSSL: + server: + verificationMode: strict + disableProtocols: "sslv2,sslv3,tlsv1,tlsv1_1" +``` + +The merge is per-key: only the values you set are replaced, and the generated keys you +omit (certificate paths, CA configuration) are preserved. See the +[`openSSL` server settings](/reference/settings/server-settings/settings#openssl) +for the available options, and +[Configuration → Embedded extra configuration](/products/kubernetes-operator/guides/configuration#embedded-extra-configuration) +for how `extraConfig` is merged. + +## Verify and troubleshoot {#troubleshoot} + +**Confirm the secure ports are live on the headless Service:** + +```bash +kubectl -n get svc -clickhouse-headless \ + -o jsonpath='{.spec.ports[*].name}' +# expect: ... tcp-secure http-secure (and NO tcp/http when required: true) +``` + +**Confirm the cert is mounted in the pod:** + +```bash +kubectl -n exec -- ls /etc/clickhouse-server/tls/ +# clickhouse-server.crt clickhouse-server.key (plus custom-ca.crt when caBundle is set) +``` + +| Symptom | Likely cause | +|------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Pods fail to start / volume mount error after enabling TLS | The referenced Secret is missing or lacks `tls.crt`/`tls.key` (or, when `caBundle` is set, the Secret/key it references). The operator does not validate the Secret's contents — missing keys surface as a pod volume-mount failure, not a dedicated status condition. Inspect the pod with `kubectl describe pod`. | +| Webhook rejects the cluster | `required: true` set without `enabled: true`, or `enabled: true` without `serverCertSecret`. | +| Client `certificate verify failed` | Client is not trusting the CA. Pass the `ca.crt` from the Secret, or check the `dnsNames` on the certificate cover the host you connect to. | +| A plaintext client suddenly can't connect | `required: true` removed ports `9000`/`8123`. Switch the client to `9440`/`8443`, or set `required: false` to keep insecure ports open during migration. | + +## See also {#see-also} + +- [Configuration → TLS/SSL configuration](/products/kubernetes-operator/guides/configuration#tls-ssl-configuration) — field reference +- [Configuration → `additionalPorts`](/products/kubernetes-operator/guides/configuration#additional-ports) — reserved ports +- [API Reference → ClusterTLSSpec](/products/kubernetes-operator/reference/api-reference#clustertlsspec) +- [`openSSL` server settings](/reference/settings/server-settings/settings#openssl) — TLS options you can override via `extraConfig` diff --git a/products/kubernetes-operator/install/helm.mdx b/products/kubernetes-operator/install/helm.mdx index 9ce318546..eefd34253 100644 --- a/products/kubernetes-operator/install/helm.mdx +++ b/products/kubernetes-operator/install/helm.mdx @@ -53,7 +53,7 @@ Install a specific operator version helm install clickhouse-operator oci://ghcr.io/clickhouse/clickhouse-operator-helm \ --create-namespace \ -n clickhouse-operator-system \ - --set-json="manager.container.tag= + --set manager.image.tag= ``` ### From Local Chart {#from-local-chart} diff --git a/products/kubernetes-operator/install/kubectl.mdx b/products/kubernetes-operator/install/kubectl.mdx index 2919e25fc..afa3e1d3f 100644 --- a/products/kubernetes-operator/install/kubectl.mdx +++ b/products/kubernetes-operator/install/kubectl.mdx @@ -24,7 +24,13 @@ Requires cert-manager to issue webhook certificates. Install the operator and CRDs from the latest release: ```bash -kubectl apply -f https://github.com/ClickHouse/clickhouse-operator/releases/latest/download/clickhouse-operator.yaml +kubectl apply --server-side --force-conflicts -f https://github.com/ClickHouse/clickhouse-operator/releases/latest/download/clickhouse-operator.yaml +``` + +Server-side apply is required because the combined CRDs exceed the client-side apply size limit. For environments that only support client-side apply, use the description-stripped CRD variant: + +```bash +kubectl apply -f https://github.com/ClickHouse/clickhouse-operator/releases/latest/download/clickhouse-operator-stripped-crds.yaml ``` This will: @@ -94,7 +100,7 @@ Build the operator manifests and apply them: ```bash make build-installer VERSION= [IMG=] -kubectl apply -k dist/install.yaml +kubectl apply --server-side --force-conflicts -f dist/install.yaml ``` diff --git a/products/kubernetes-operator/install/olm.mdx b/products/kubernetes-operator/install/olm.mdx index 25051f352..8ff3cb140 100644 --- a/products/kubernetes-operator/install/olm.mdx +++ b/products/kubernetes-operator/install/olm.mdx @@ -97,3 +97,4 @@ More info about uninstalling can be found in the [OLM documentation](https://olm ## Additional Resources {#additional-resources} - [Operator Lifecycle Manager Documentation](https://olm.operatorframework.io/docs) + diff --git a/products/kubernetes-operator/navigation.json b/products/kubernetes-operator/navigation.json index de8697923..24942f13b 100644 --- a/products/kubernetes-operator/navigation.json +++ b/products/kubernetes-operator/navigation.json @@ -17,7 +17,10 @@ "expanded": true, "pages": [ "products/kubernetes-operator/guides/introduction", - "products/kubernetes-operator/guides/configuration" + "products/kubernetes-operator/guides/configuration", + "products/kubernetes-operator/guides/monitoring", + "products/kubernetes-operator/guides/scaling", + "products/kubernetes-operator/guides/tls" ] }, { diff --git a/products/kubernetes-operator/overview.mdx b/products/kubernetes-operator/overview.mdx index ba1c47d29..b10069a11 100644 --- a/products/kubernetes-operator/overview.mdx +++ b/products/kubernetes-operator/overview.mdx @@ -34,7 +34,9 @@ Choose your preferred installation method: - **[Introduction](/products/kubernetes-operator/guides/introduction)** - General overview of ClickHouse Operator concepts - **[Configuration Guide](/products/kubernetes-operator/guides/configuration)** - Configure ClickHouse and Keeper clusters -- **[Monitoring](/products/kubernetes-operator/guides/introduction)** - Monitor clickhouse-operator using Prometheus metrics +- **[Monitoring](/products/kubernetes-operator/guides/monitoring)** - Prometheus metrics and health probes for the operator +- **[Scaling](/products/kubernetes-operator/guides/scaling)** - Scale replicas, shards, and Keeper quorum +- **[Securing with TLS](/products/kubernetes-operator/guides/tls)** - Encrypt clusters end-to-end with cert-manager ## Reference {#reference} diff --git a/products/kubernetes-operator/reference/api-reference.mdx b/products/kubernetes-operator/reference/api-reference.mdx index 43e59de85..11cb49a00 100644 --- a/products/kubernetes-operator/reference/api-reference.mdx +++ b/products/kubernetes-operator/reference/api-reference.mdx @@ -8,9 +8,32 @@ doc_type: 'reference' sidebarTitle: 'API reference' --- - This document provides detailed API reference for the ClickHouse Operator custom resources. +## AdditionalPort {#additionalport} + +AdditionalPort declares one extra TCP port to expose on the ClickHouse Pod and the operator-managed headless Service. + +| Field | Type | Description | Required | Default | +|-------|------|-------------|----------|---------| +| `name` | string | Name uniquely identifies the port within the list. Used as both the container port name and the Service port name.
This must be a DNS_LABEL. | true | | +| `port` | integer | Port is the TCP port number to expose. | true | | + +Appears in: +- [ClickHouseClusterSpec](#clickhouseclusterspec) + +## CABundleSelector {#cabundleselector} + +CABundleSelector selects a key holding a CA bundle from a Secret in the cluster's namespace. + +| Field | Type | Description | Required | Default | +|-------|------|-------------|----------|---------| +| `name` | string | The name of the secret in the cluster's namespace to select from. | true | | +| `key` | string | The key of the secret to select from. Must be a valid secret key. | false | ca.crt | + +Appears in: +- [ClusterTLSSpec](#clustertlsspec) + ## ClickHouseCluster {#clickhousecluster} ClickHouseCluster is the Schema for the `clickhouseclusters` API. @@ -57,6 +80,7 @@ ClickHouseClusterSpec defines the desired state of ClickHouseCluster. | `podTemplate` | [PodTemplateSpec](#podtemplatespec) | Parameters passed to the ClickHouse pod spec. | false | | | `containerTemplate` | [ContainerTemplateSpec](#containertemplatespec) | Parameters passed to the ClickHouse container spec. | false | | | `dataVolumeClaimSpec` | [PersistentVolumeClaimSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#persistentvolumeclaimspec-v1-core) | Specification of persistent storage for ClickHouse data. | false | | +| `additionalVolumeClaimTemplates` | [PersistentVolumeClaimTemplate](#persistentvolumeclaimtemplate) array | Additional per-pod PVC templates for JBOD / multi-disk storage.
Each entry is propagated in StatefulSet volumeClaimTemplate, mounted at /var/lib/clickhouse/disks/ and
added to the generated JBOD storage policy.
The set of disks is fixed at creation. | false | | | `labels` | object (keys:string, values:string) | Additional labels that are added to resources. | false | | | `annotations` | object (keys:string, values:string) | Additional annotations that are added to resources. | false | | | `podDisruptionBudget` | [PodDisruptionBudgetSpec](#poddisruptionbudgetspec) | PodDisruptionBudget configures the PDB created for each shard.
When unset, the operator defaults to maxUnavailable=1 for single-replica
shards and minAvailable=1 for multi-replica shards. | false | | @@ -65,6 +89,7 @@ ClickHouseClusterSpec defines the desired state of ClickHouseCluster. | `upgradeChannel` | string | UpgradeChannel specifies the release channel for major version upgrade checks.
When empty, only minor updates will be proposed. Allowed values are: stable, lts or specific major.minor version (e.g. 25.8). | false | | | `versionProbeTemplate` | [VersionProbeTemplate](#versionprobetemplate) | VersionProbeTemplate overrides for the version detection Job. | false | | | `externalSecret` | [ExternalSecret](#externalsecret) | ExternalSecret is an optional reference to an externally-managed Secret containing cluster secrets.
The secret must reside in the same namespace as the cluster. | false | | +| `additionalPorts` | [AdditionalPort](#additionalport) array | AdditionalPorts declares extra TCP ports to expose on the ClickHouse Pod and the operator-managed headless Service.
The operator only adds the ports to the Kubernetes resources, it does not configure the ClickHouse server to listen on them. | false | | Appears in: - [ClickHouseCluster](#clickhousecluster) @@ -112,8 +137,8 @@ ClusterTLSSpec defines cluster TLS configuration. |-------|------|-------------|----------|---------| | `enabled` | boolean | Enabled indicates whether TLS is enabled, determining if secure ports should be opened. | false | false | | `required` | boolean | Required specifies whether TLS must be enforced for all connections. Disables not secure ports. | false | false | -| `serverCertSecret` | [LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#localobjectreference-v1-core) | ServerCertSecretRef is a reference to a TLS Secret containing the server certificate.
It is expected that the Secret has the same structure as certificates generated by cert-manager,
with the certificate and private key stored under "tls.crt" and "tls.key" keys respectively. | false | | -| `caBundle` | [SecretKeySelector](#secretkeyselector) | CABundle is a reference to a TLS Secret containing the CA bundle.
If empty and ServerCertSecret is specified, the CA bundle from certificate will be used.
Otherwise, system trusted CA bundle will be used.
Key is defaulted to "ca.crt" if not specified. | false | | +| `serverCertSecret` | [LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#localobjectreference-v1-core) | ServerCertSecret is a reference to a TLS Secret containing the server certificate.
It is expected that the Secret has the same structure as certificates generated by cert-manager,
with the certificate and private key stored under "tls.crt" and "tls.key" keys respectively. | false | | +| `caBundle` | [CABundleSelector](#cabundleselector) | CABundle is a reference to a Secret key holding a CA bundle used to verify peer certificates.
If empty, the system trusted CA bundle is used.
Key is defaulted to "ca.crt" if not specified. | false | | Appears in: - [ClickHouseSettings](#clickhousesettings) @@ -152,7 +177,7 @@ ContainerTemplateSpec describes the container configuration overrides for the cl |-------|------|-------------|----------|---------| | `image` | [ContainerImage](#containerimage) | Image is the container image to be deployed. | true | | | `imagePullPolicy` | [PullPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#pullpolicy-v1-core) | ImagePullPolicy for the image, which defaults to IfNotPresent. | false | | -| `resources` | [ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#resourcerequirements-v1-core) | Resources is the resource requirements for the server container.
Deep-merged with operator defaults via SMP. Individual limits and requests override only matching
keys; unset fields preserve operator defaults. | false | | +| `resources` | [ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#resourcerequirements-v1-core) | Resources is the resource requirements for the server container.
Applied as a whole: operator defaults are used only when all resource fields are empty. | false | | | `volumeMounts` | [VolumeMount](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#volumemount-v1-core) array | VolumeMounts is the list of volume mounts for the container.
Concatenated with operator-generated mounts. Entries sharing a `mountPath` with an operator
mount are merged into a projected volume. | false | | | `env` | [EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#envvar-v1-core) array | Env is the list of environment variables to set in the container.
Merged with operator defaults by name. | false | | | `securityContext` | [SecurityContext](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#securitycontext-v1-core) | SecurityContext defines the security options the container should be run with.
A non-nil SecurityContext fully replaces operator defaults; the user owns the
entire struct. When nil, operator defaults are preserved.
More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ | false | | @@ -258,11 +283,11 @@ KeeperClusterSpec defines the desired state of KeeperCluster. | `dataVolumeClaimSpec` | [PersistentVolumeClaimSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#persistentvolumeclaimspec-v1-core) | Specification of persistent storage for ClickHouse Keeper data. | false | | | `labels` | object (keys:string, values:string) | Additional labels that are added to resources. | false | | | `annotations` | object (keys:string, values:string) | Additional annotations that are added to resources. | false | | -| `podDisruptionBudget` | [PodDisruptionBudgetSpec](#poddisruptionbudgetspec) | PodDisruptionBudget configures the PDB created for the Keeper cluster.
When unset, the operator defaults to maxUnavailable=replicas/2
(preserving quorum for a 2F+1 cluster). | false | | +| `podDisruptionBudget` | [PodDisruptionBudgetSpec](#poddisruptionbudgetspec) | PodDisruptionBudget configures the PDB created for the Keeper cluster.
When unset, the operator defaults to maxUnavailable=replicas/2
(preserving quorum for a 2F+1 cluster); single-replica clusters use maxUnavailable=1. | false | | | `settings` | [KeeperSettings](#keepersettings) | Configuration parameters for ClickHouse Keeper server. | false | | | `clusterDomain` | string | ClusterDomain is the Kubernetes cluster domain suffix used for DNS resolution. | false | cluster.local | | `upgradeChannel` | string | UpgradeChannel specifies the release channel for major version upgrade checks.
When empty, only minor updates will be proposed. Allowed values are: stable, lts or specific major.minor version (e.g. 25.8). | false | | -| `versionProbeTemplate` | [VersionProbeTemplate](#versionprobetemplate) | VersionProbeTemplate overrides for the version detection Job. | false | | +| `versionProbeTemplate` | [VersionProbeTemplate](#versionprobetemplate) | VersionProbeTemplate overrides for the version detection Job.
Deprecated: Keeper version probe Jobs are not used; this field is retained for backward compatibility. | false | | Appears in: - [KeeperCluster](#keepercluster) @@ -278,10 +303,10 @@ KeeperClusterStatus defines the observed state of KeeperCluster. | `configurationRevision` | string | ConfigurationRevision indicates target configuration revision for every replica. | true | | | `statefulSetRevision` | string | StatefulSetRevision indicates target StatefulSet revision for every replica. | true | | | `currentRevision` | string | CurrentRevision indicates latest applied KeeperCluster spec revision. | true | | -| `updateRevision` | string | CurrentRevision indicates latest requested KeeperCluster spec revision. | true | | +| `updateRevision` | string | UpdateRevision indicates latest requested KeeperCluster spec revision. | true | | | `observedGeneration` | integer | ObservedGeneration indicates latest generation observed by controller. | true | | -| `version` | string | Version indicates the version reported by the container image. | false | | -| `versionProbeRevision` | string | VersionProbeRevision is the image hash of the last successful version probe.
When this matches the current image hash, the cached Version is used directly. | false | | +| `version` | string | Version indicates the version reported by the Keeper server. | false | | +| `versionProbeRevision` | string | VersionProbeRevision is the image hash of the last successful version probe.
Deprecated: Keeper version probe Jobs are not used; this field is retained for backward compatibility. | false | | Appears in: - [KeeperCluster](#keepercluster) @@ -315,6 +340,19 @@ Appears in: - [ClickHouseSettings](#clickhousesettings) - [KeeperSettings](#keepersettings) +## NamedTemplateMeta {#namedtemplatemeta} + +NamedTemplateMeta defines supported metadata settings for template objects that require a name. + +| Field | Type | Description | Required | Default | +|-------|------|-------------|----------|---------| +| `name` | string | Name is the resource identifier. | true | | +| `labels` | object (keys:string, values:string) | Labels are labels applied to the template objects. | false | | +| `annotations` | object (keys:string, values:string) | Annotations are annotations applied to the template objects. | false | | + +Appears in: +- [PersistentVolumeClaimTemplate](#persistentvolumeclaimtemplate) + ## PDBPolicy {#pdbpolicy} PDBPolicy controls whether PodDisruptionBudgets are created. @@ -328,6 +366,18 @@ PDBPolicy controls whether PodDisruptionBudgets are created. Appears in: - [PodDisruptionBudgetSpec](#poddisruptionbudgetspec) +## PersistentVolumeClaimTemplate {#persistentvolumeclaimtemplate} + +PersistentVolumeClaimTemplate is a named template for a per-replica PersistentVolumeClaim. + +| Field | Type | Description | Required | Default | +|-------|------|-------------|----------|---------| +| `metadata` | [NamedTemplateMeta](#namedtemplatemeta) | Refer to Kubernetes API documentation for fields of `metadata`. | true | | +| `spec` | [PersistentVolumeClaimSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#persistentvolumeclaimspec-v1-core) | Spec defines the desired characteristics of a volume requested by a pod author.
More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims | true | | + +Appears in: +- [ClickHouseClusterSpec](#clickhouseclusterspec) + ## PodDisruptionBudgetSpec {#poddisruptionbudgetspec} PodDisruptionBudgetSpec configures the PDB created for the cluster. @@ -381,7 +431,6 @@ SecretKeySelector selects a key of a Secret. | `key` | string | The key of the secret to select from. Must be a valid secret key. | true | | Appears in: -- [ClusterTLSSpec](#clustertlsspec) - [DefaultPasswordSelector](#defaultpasswordselector) ## TemplateMeta {#templatemeta}