From 45e520d20f5c67c7b7224b66b60bb6cac3ccf3ef Mon Sep 17 00:00:00 2001 From: Ian Maddaus Date: Thu, 14 May 2026 15:10:55 -0400 Subject: [PATCH 1/3] Update knife plugins Signed-off-by: Ian Maddaus --- content/tools/knife/plugin_knife_opc.md | 391 ------------------ .../tools/knife/plugins/knife_ec_backup.md | 326 +++++++++++++++ content/tools/knife/plugins/knife_tidy.md | 312 ++++++++++++++ tools/vale/chef/foreign.yml | 1 - 4 files changed, 638 insertions(+), 392 deletions(-) delete mode 100644 content/tools/knife/plugin_knife_opc.md create mode 100644 content/tools/knife/plugins/knife_ec_backup.md create mode 100644 content/tools/knife/plugins/knife_tidy.md diff --git a/content/tools/knife/plugin_knife_opc.md b/content/tools/knife/plugin_knife_opc.md deleted file mode 100644 index 1711dab..0000000 --- a/content/tools/knife/plugin_knife_opc.md +++ /dev/null @@ -1,391 +0,0 @@ -+++ -title = "knife opc" -draft = false - -[menu] - [menu.tools] - title = "knife opc" - identifier = "tools/knife/reference/plugin_knife_opc.md knife opc" - parent = "tools/knife/reference" -+++ - - -The `knife opc` subcommand is used to manage organizations and users in Chef Infra Server. - -{{< note >}} - -Administrator permissions are required to add, remove, or edit users. To manage organizations, or change a user's assignment to an organization, the pivotal key is required. Rather than using the knife-opc plugin commands below, which are an implementation detail, use the equivalent "user-" and "org-" subcommands directly on Chef Infra Server. Those wrapped subcommands already have the needed permissions applied and access to sensitive commands will then be centralized. [See chef-server-ctl for details](https://docs.chef.io/server/ctl_chef_server//). - -{{< /note >}} - -{{< note >}} - -Review the list of [common options](/tools/knife/common_options/) available to this (and all) knife subcommands and plugins. - -{{< /note >}} - -## config.rb Configuration [plugin_knife_opc-knife-rb-configuration] - -Unlike other knife subcommands the subcommands in the knife-opc plugin make API calls against the root of your Chef Infra Server installation's API endpoint. - -Typically the `chef_server_url` for your Chef Infra Server installation may look like this: - -```ruby -chef_server_url 'https://chef.example.com/organizations/ORG_NAME' -``` - -To configure knife-opc, set the `chef_server_root` option to the root of your Chef Infra Server installation: - -```ruby -chef_server_root 'https://chef.example.com/' -``` - -If your `chef_server_url` configuration ends with `/organizations/ORG_NAME` (as shown above), this setting will default to `https://chef.example.com/`. - -{{< note >}} - -On Chef Infra Server, the majority of the commands provided by this plugin can be accessed with `chef-server-ctl` wrapper commands. [See chef-server-ctl for details](https://docs.chef.io/server/ctl_chef_server//). - -{{< /note >}} - -## opc user list [plugin_knife_opc-opc-user-list] - -Show a list of all users in your Chef Infra Server installation. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc user list (options) -``` - -### Options - -This argument has the following options: - -`-w`, `--with-uri` - -: Show corresponding URIs. - -### Example - -```bash -knife opc user list -alice -pivotal -knife opc user list -w -alice: https://chef-server.fqdn/users/alice -pivotal: https://chef-server.fqdn/users/pivotal -``` - -## opc user show - -Shows the details of a user in your Chef Infra Server installation. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc user show USER_NAME (options) -``` - -### Options - -This argument has the following options: - -`-l`, `--with-orgs` - -: Show the organizations of which the user is a member. - -### Example - -```bash -knife opc user show alice -l -display_name: Alice Schmidt -email: alice@chef.io -first_name: Alice -last_name: Schmidt -middle_name: -organizations: acme -public_key: -----BEGIN PUBLIC KEY----- -[...] ------END PUBLIC KEY----- - - -username: alice -``` - -## opc user create [plugin_knife_opc-opc-user-create] - -Creates a new user in your Chef Infra Server installation. The user's private key will be returned in response. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc user create USER_NAME FIRST_NAME [MIDDLE_NAME] LAST_NAME EMAIL PASSWORD (options) -``` - -### Options - -This argument has the following options: - -`-f FILENAME`, `--filename FILENAME` - -: Write private key to `FILENAME` rather than `STDOUT`. - -### Example - -```bash -knife opc user create arno arno schmidt arno@chef.io password ------BEGIN RSA PRIVATE KEY----- -[...] ------END RSA PRIVATE KEY----- -``` - -## opc user delete - -Deletes the given OPC user. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc user delete USER_NAME [-d] [-R] -``` - -### Options - -This argument has the following options: - -`-d`, `--no-disassociate-user` - -: Don't disassociate the user first. - -`-R`, `--remove-from-admin-groups` - -: If the user is a member of any org admin groups, attempt to remove from those groups. Ignored if `--no-disassociate-user` is set. - -### Example - -```bash -knife opc user delete arno -Do you want to delete the user arno? (Y/N) Y -Checking organization memberships... -Deleting user arno. -``` - -## opc user edit - -Will open `$EDITOR` to edit a user. When finished editing, knife will update the given Chef Infra Server user. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc user edit USER_NAME -``` - -#### Example [plugin_knife_opc-opc-user-password] - -```bash -EDITOR=ed knife opc user edit arno -639 -1,%p -{ - "username": "arno", - "email": "arno@chef.io", - "display_name": "arno schmidt", - "first_name": "arno", - "last_name": "schmidt", - "middle_name": "", - "public_key": "-----BEGIN PUBLIC KEY-----\n[...]\n-----END PUBLIC KEY-----\n\n" -} -/email/s/chef.io/opscode.com/p -"email": "arno@opscode.com", -wq -643 -Saved arno. -knife opc user show arno -display_name: arno schmidt -email: arno@opscode.io -first_name: arno -last_name: schmidt -middle_name: -public_key: -----BEGIN PUBLIC KEY----- -[...] ------END PUBLIC KEY----- - - -username: arno -``` - -## opc user password - -Command for managing password and authentication for a user. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc user password USER_NAME [PASSWORD | --enable_external_auth] -``` - -The last argument should either be a string to use as password or `--enable_external_auth` instead of a password to enable external authentication for this user. - -### Example - -```bash -knife opc user password arno newpassword -{"username"=>"arno", "email"=>"arno@opscode.com", "display_name"=>"arno schmidt", "first_name"=>"arno", "last_name"=>"schmidt", "middle_name"=>"", "public_key"=>"-----BEGIN PUBLIC KEY-----\n[...]\n-----END PUBLIC KEY-----\n\n", "password"=>"newpassword", "recovery_authentication_enabled"=>true} -Authentication info updated for arno. -``` - -## opc org list [plugin_knife_opc-opc-org-list] - -Show a list of all organizations in your Chef Infra Server installation. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc org list (options) -``` - -### Options - -This argument has the following options: - -`-w`, `--with-uri` - -: Show corresponding URIs. - -`-a`, `--all-orgs` - -: Display auto generated hidden orgs. - -### Example - -```bash -knife opc org list -w -a -acme: https://chef-server.fqdn/organizations/acme -``` - -## opc org show [plugin_knife_opc-opc-org-show] - -Shows the details of an organization in your Chef Infra Server installation. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc org show ORG_NAME -``` - -### Example - -```bash -knife opc org show acme -full_name: Acme -guid: cc9f9d0d4f6e7e35272e327e22e7affc -name: acme -``` - -## opc org create [plugin_knife_opc-opc-org-create] - -Creates a new Chef Infra Server organization. The private key for the organization's validator client is returned. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc org create ORG_NAME ORG_FULL_NAME (options) -``` - -### Options - -This argument has the following options: - -`-f FILENAME`, `--filename FILENAME` - -: Write private key to `FILENAME` rather than `STDOUT`. - -`-a USER_NAME`, `--association_user USER_NAME` - -: Associate `USER_NAME` with the organization after creation. - -### Example - -```bash -knife opc org create acme2 "The Other Acme" -a arno ------BEGIN RSA PRIVATE KEY----- -[...] ------BEGIN RSA PRIVATE KEY----- -``` - -## opc org delete [plugin_knife_opc-opc-org-delete] - -Deletes the given Chef Infra Server organization. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc org delete ORG_NAME -``` - -### Example - -```bash -knife opc org delete acme2 -Do you want to delete the organization acme2? (Y/N) Y -full_name: The Other Acme -guid: 2adec1140cf777a15d82d9099304da71 -name: acme2 -``` - -## opc org user add - -Adds a user to an organization. Requires that the named organization and user both exist. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc org user add ORG_NAME USER_NAME -``` - -### Example - -```bash -knife opc org user add acme2 alice -``` - -## opc org user remove - -Removes a user from an organization. Requires that the named organization and user both exist, and that the user is currently associated with the organization. - -### Syntax - -This argument has the following syntax: - -```bash -knife opc org user remove ORG_NAME USER_NAME -``` - -### Example - -```bash -knife opc org user remove acme2 alice -``` diff --git a/content/tools/knife/plugins/knife_ec_backup.md b/content/tools/knife/plugins/knife_ec_backup.md new file mode 100644 index 0000000..4c4700f --- /dev/null +++ b/content/tools/knife/plugins/knife_ec_backup.md @@ -0,0 +1,326 @@ ++++ +title = "knife-ec-backup plugin" + + +[menu] + [menu.tools] + title = "knife-ec-backup" + identifier = "tools/knife/plugins/knife-ec-backup" + parent = "tools/knife/plugins" ++++ + +The `knife-ec-backup` plugin backs up and restores the data in a Chef Infra Server installation, including all organizations, users, nodes, cookbooks, roles, and associated ACLs. +Use it before server migrations, version upgrades, or to create point-in-time snapshots of your Chef Infra Server. + +## Install knife-ec-backup + +```shell +gem install knife-ec-backup +``` + +## Common options + +The following options are shared across all `knife ec` subcommands: + +`--only-org ` +: Limit the operation to a single named organization. + By default, all organizations are included. + +`--concurrency ` +: Maximum number of simultaneous API requests (default: 10). + +`--skip-useracl` +: Skip downloading or restoring user ACLs. + Required for Enterprise Chef 11.0.2 and earlier. + +`--skip-version-check` +: Skip the Chef Infra Server version check and all autoconfigured options. + +`--webui-key ` +: Path to the WebUI private key. + Defaults to reading from the secrets store, `/etc/opscode/webui_priv.pem`, or the Automate path. + +`--secrets-file ` +: Path to `private-chef-secrets.json` (default: `/etc/opscode/private-chef-secrets.json`). + +`--purge` +: Sync deletions from the backup source to the restore destination. + +`--dry-run` +: Report what actions would be taken without performing any. + +`--error-log-dir ` +: Directory where errors are logged during the operation. + +## knife ec backup + +```shell +knife ec backup +``` + +Downloads all users, user ACLs, organizations, cookbooks, nodes, roles, environments, data bags, groups, and ACLs from a Chef Infra Server to a local directory. + +### Common invocations + +```shell +# Back up all organizations to a local directory +knife ec backup /var/backups/chef-server +``` + +```shell +# Back up a single organization +knife ec backup /var/backups/chef-server --only-org org-name +``` + +```shell +# Back up including user and key table data from PostgreSQL +knife ec backup /var/backups/chef-server --with-user-sql --with-key-sql \ + --sql-user opscode_chef --sql-password +``` + +```shell +# Back up and remove local users that no longer exist on the server +knife ec backup /var/backups/chef-server --purge +``` + +### Key flags + +`--with-user-sql` +: Export user data directly from the PostgreSQL database. + Required to correctly preserve user passwords and keys. + +`--with-key-sql` +: Export rotated key data from the PostgreSQL `keys` table. + Required for Chef Infra Server 12 and later when keys have been rotated. + +`--skip-frozen-cookbook-status` +: Skip writing a `status.json` file for each cookbook. + Use this when you don't want to persist a cookbook's frozen status across restores. + +`--sql-host ` +: PostgreSQL hostname (default: `localhost`). + +`--sql-port ` +: PostgreSQL port (default: `5432`). + +`--sql-user ` +: PostgreSQL user for the `opscode_chef` database. + +`--sql-password ` +: PostgreSQL password. + +### How knife ec backup works + +`knife ec backup` downloads all users and user ACLs first, then iterates through every organization and downloads org-level objects using the Chef Infra Server REST API. +On Enterprise Chef 11 and earlier, pre-created unassigned organizations are skipped automatically. +The backup writes everything to a directory tree under `/users/` and `/organizations//`. +If `--with-user-sql` or `--with-key-sql` is set, the command connects directly to PostgreSQL and writes `key_dump.json` and `key_table_dump.json` alongside the backup. + +## knife ec restore + +```shell +knife ec restore +``` + +Restores all users, organizations, cookbooks, and ACLs from a local backup directory to a Chef Infra Server. +Run this after `knife ec backup` to migrate data to a new server or recover from failure. + +### Common invocations + +```shell +# Restore all data from a backup directory +knife ec restore /var/backups/chef-server +``` + +```shell +# Restore a single organization only +knife ec restore /var/backups/chef-server --only-org org-name +``` + +```shell +# Restore without restoring users (for example, when users are managed by LDAP) +knife ec restore /var/backups/chef-server --skip-users +``` + +```shell +# Restore and overwrite pivotal's key (use with caution) +knife ec restore /var/backups/chef-server --overwrite-pivotal +``` + +### Key flags + +`--skip-users` +: Skip restoring global users. + Use this when users are managed externally, for example through LDAP or SAML. + +`--overwrite-pivotal` +: Overwrite the `pivotal` superuser key during restore. + After using this flag, all future API requests will fail until you update the private key on the server. + By default, `pivotal` is skipped. + +`--[no-]skip-user-ids` +: When enabled (default), reuses user IDs from the restore destination to avoid database conflicts. + Pass `--no-skip-user-ids` to use IDs from the backup. + +### How knife ec restore works + +`knife ec restore` reads the directory structure written by `knife ec backup`. +It restores users first (skipping `pivotal` unless `--overwrite-pivotal` is set), then creates or updates each organization, restores open invitations, adds users to orgs, and uploads all org-level data. +If an organization already exists on the target server, the command updates it rather than failing. +User ACLs are restored last; pass `--skip-useracl` to skip this step if the target server runs Enterprise Chef 11.0.2 or earlier. + +## knife ec import + +```shell +knife ec import +``` + +Imports objects from a backup directory into a Chef Infra Server. +`knife ec import` is similar to `knife ec restore` but designed for Automate-managed servers and environments where a tenant ID header is required. + +### Common invocations + +```shell +# Import all organizations from a backup directory +knife ec import /var/backups/chef-server +``` + +```shell +# Import with a tenant identifier header (required for some Automate deployments) +knife ec import /var/backups/chef-server --tenant-id tenant-uuid +``` + +### Key flags + +`--tenant-id ` +: Adds an `X-Tenant-Id` header to all import requests. + Required for Chef Automate deployments that use tenant-based routing. + +## knife ec key export + +```shell +knife ec key export [] [] +``` + +Exports user and key table data directly from the Chef Infra Server PostgreSQL database to JSON files. +This command is invoked automatically by `knife ec backup --with-user-sql` and `--with-key-sql`, but you can also run it independently. + +### Common invocations + +```shell +# Export to default output files (key_dump.json and key_table_dump.json) +knife ec key export \ + --sql-user opscode_chef --sql-password +``` + +```shell +# Export to custom output file paths +knife ec key export user_table.json key_table.json \ + --sql-user opscode_chef --sql-password +``` + +```shell +# Export only the keys table, skipping the users table +knife ec key export --skip-users-table \ + --sql-user opscode_chef --sql-password +``` + +### Expected output + +```console +knife ec key export --sql-user opscode_chef --sql-password secret +``` + +On success, the command writes `key_dump.json` and `key_table_dump.json` in the current directory (or the paths you provided) and exits silently. + +### Key flags + +`--skip-users-table` +: Skip exporting the `users` table. + Use this when you only need rotated key data. + +`--skip-keys-table` +: Skip exporting the `keys_by_name` table. + Use this flag with Chef Infra Server 11, which doesn't have a keys table. + +`--sql-host ` +: PostgreSQL hostname (default: `localhost`). + +`--sql-port ` +: PostgreSQL port (default: `5432`). + +`--sql-user ` +: PostgreSQL user for the `opscode_chef` database. + +`--sql-password ` +: PostgreSQL password. + +### How knife ec key export works + +`knife ec key export` connects directly to the Chef Infra Server PostgreSQL database using the Sequel gem. +It queries the `users` table (written to `key_dump.json` by default) and the `keys_by_name` table joined with `orgs` (written to `key_table_dump.json` by default). +If you pass custom file name arguments, those paths are used instead. +On Chef Infra Server 11, use `--skip-keys-table` to avoid a `PG::UndefinedTable` error. + +## knife ec key import + +```shell +knife ec key import [] [] +``` + +Imports user and key table data from JSON files directly into the Chef Infra Server PostgreSQL database. +Run this after `knife ec restore` when you used `--with-user-sql` or `--with-key-sql` during backup. + +### Common invocations + +```shell +# Import from default file paths (key_dump.json and key_table_dump.json) +knife ec key import \ + --sql-user opscode_chef --sql-password +``` + +```shell +# Import from custom file paths +knife ec key import user_table.json key_table.json \ + --sql-user opscode_chef --sql-password +``` + +```shell +# Import only user data, skipping client key data +knife ec key import --users-only \ + --sql-user opscode_chef --sql-password +``` + +```shell +# Import only client key data, skipping user table +knife ec key import --clients-only \ + --sql-user opscode_chef --sql-password +``` + +### Key flags + +`--[no-]skip-pivotal` +: Skip importing the `pivotal` user key (default: enabled). + Pass `--no-skip-pivotal` to upload the pivotal key from the backup. + +`--[no-]skip-user-ids` +: Reuse user IDs from the restore destination to avoid database conflicts (default: enabled). + Pass `--no-skip-user-ids` to use IDs from the backup files. + +`--users-only` +: Import only user data from the users table; skip client key data. + +`--clients-only` +: Import only client key data; skip the users table. + +`--skip-users-table` +: Skip importing the users table entirely. + +`--skip-keys-table` +: Skip importing the `keys_by_name` table. + +### How knife ec key import works + +`knife ec key import` reads the JSON files written by `knife ec key export` and writes directly to the Chef Infra Server PostgreSQL database. +For clients, the organization ID is looked up dynamically by org name (with caching) because org IDs differ between source and destination servers. +Organizations must exist on the destination server before you run `knife ec key import`---run `knife ec restore` first. diff --git a/content/tools/knife/plugins/knife_tidy.md b/content/tools/knife/plugins/knife_tidy.md new file mode 100644 index 0000000..45c20de --- /dev/null +++ b/content/tools/knife/plugins/knife_tidy.md @@ -0,0 +1,312 @@ ++++ +title = "knife-tidy plugin" + +[menu] + [menu.tools] + title = "knife-tidy" + identifier = "tools/knife/plugins/knife-tidy" + parent = "tools/knife/plugins" ++++ + +The `knife-tidy` plugin reports on and removes stale nodes, associated clients, ACLs, and unused cookbook versions from a Chef Infra Server. +It also cleans data-integrity issues from [`knife-ec-backup`](/tools/knife/plugins/knife-ec-backup/) directories before you restore them to a server. + +## Install knife-tidy + +`knife tidy` is available as part of the `knife-ec-backup` Habitat package or as a standalone Ruby gem. + +- Install the Habitat package (recommended): + + ```bash + hab pkg install chef/knife-ec-backup -bf + ``` + + After installation, prefix all `knife tidy` commands with `hab pkg exec chef/knife-ec-backup`. + +- Install the Ruby gem: + + ```bash + gem install knife-tidy + ``` + + After installation, run `knife tidy` commands directly without the `hab pkg exec` prefix. + +## Common options + +The following option is shared across all `knife tidy` subcommands that connect to a Chef Infra Server: + +`--orgs ` +: Limit the operation to the named organizations. + By default, all organizations are included. + +## knife tidy server report + +```shell +knife tidy server report (options) +``` + +Scans a Chef Infra Server and writes JSON report files to the `reports/` directory in the current working directory. +The reports identify stale nodes and unused cookbook versions for each organization. + +### Common invocations + +```shell +# Report on all organizations using default thresholds +knife tidy server report [FLAGS] +``` + +```shell +# Report on specific organizations +knife tidy server report --orgs brewinc,acmeinc +``` + +```shell +# Treat nodes as stale after 50 days of inactivity, keep at least 2 cookbook versions +knife tidy server report --node-threshold 50 --keep-versions 2 +``` + +### Output files + +The command writes the following files to the `reports/` directory for each organization: + +| File | Contents | +|---|---| +| `_stale_nodes.json` | Nodes that haven't checked in within the threshold period | +| `_cookbook_count.json` | Version count for each cookbook in the organization | +| `_unused_cookbooks.json` | Cookbooks and versions not referenced in any node's run list | + +### Key flags + +`--node-threshold ` +: Number of days since last check-in before a node is considered stale (default: 30). + +`--keep-versions ` +: Minimum number of cookbook versions to keep per cookbook, even if they aren't in use (default: 1). + +### How knife tidy server report works + +`knife tidy server report` queries the Chef Infra Server search index and REST API to build a picture of which nodes are active and which cookbook versions are in use. +It cross-references node run lists, environment cookbook pins, and the full cookbook list on the server to identify unused versions. +If the search index is out of sync with the database, the command logs a warning and skips that organization---run `chef-server-ctl reindex` to resync. +Nodes running Chef Infra Client older than 12.3 generate an additional warning because their cookbook version data isn't factored into the unused-cookbooks calculation. +Run the report iteratively: after deleting stale nodes, more unused cookbooks may surface on the next run. + +## knife tidy server clean + +```shell +knife tidy server clean (options) +``` + +Reads the JSON report files produced by `knife tidy server report` and deletes stale nodes, associated clients, ACLs, and unused cookbooks from the Chef Infra Server. +Always take a current backup using [`knife ec backup`](/tools/knife/plugins/knife-ec-backup/) before running this command. + +### Common invocations + +```shell +# Clean all stale objects identified in reports, for all organizations +knife tidy server clean --backup-path /var/backups/chef-server +``` + +```shell +# Clean only stale nodes (skip unused cookbooks) +knife tidy server clean --backup-path /var/backups/chef-server --only-nodes +``` + +```shell +# Clean only unused cookbooks (skip stale nodes) +knife tidy server clean --backup-path /var/backups/chef-server --only-cookbooks +``` + +```shell +# Preview what would be deleted without making any changes +knife tidy server clean --backup-path /var/backups/chef-server --dry-run +``` + +```shell +# Clean specific organizations with increased concurrency +knife tidy server clean --backup-path /var/backups/chef-server \ + --orgs brewinc,acmeinc --concurrency 4 +``` + +### Key flags + +`--backup-path ` +: Path to the most recent `knife-ec-backup` backup directory. + The command prompts interactively if this flag is omitted. + +`--only-cookbooks` +: Delete only unused cookbooks. + Can't be used together with `--only-nodes`. + +`--only-nodes` +: Delete only stale nodes and their associated clients and ACLs. + Can't be used together with `--only-cookbooks`. + +`--dry-run` +: Print the deletions that would occur without performing any. + +`--concurrency ` +: Maximum number of simultaneous deletion requests (default: 1). + +### How knife tidy server clean works + +`knife tidy server clean` reads the `reports/` directory in the current working directory and determines the list of organizations from the report filenames. +It then iterates through each organization's `_unused_cookbooks.json` and `_stale_nodes.json` files and issues REST API delete calls for each item. +The command asks for confirmation before deleting unless `--unattended` is set. +If a `knife-tidy-server-warnings.txt` file was written by the report step, any warnings it contains are printed again before you confirm. +Specifying both `--only-cookbooks` and `--only-nodes` at the same time is an error and the command exits immediately. + +## knife tidy backup clean + +```shell +knife tidy backup clean (options) +``` + +Fixes data-integrity issues in a `knife-ec-backup` directory so the backup can be successfully restored with `knife ec restore`. +Run this command against a backup directory before attempting a restore operation. + +### Common invocations + +```shell +# Generate a boilerplate substitutions file +knife tidy backup clean --gen-gsub +``` + +```shell +# Clean a backup directory using the generated substitutions file +knife tidy backup clean --backup-path backups/ --gsub-file substitutions.json +``` + +```shell +# Clean a backup directory without any global substitutions +knife tidy backup clean --backup-path backups/ +``` + +### Expected output + +Running `--gen-gsub` creates a file called `substitutions.json` in the current directory: + +```console +INFO: Creating boiler plate gsub file: 'substitutions.json' +``` + +When issues are repaired during a clean run, the command prints lines like: + +```console +INFO: Validating org object for myorg +REPAIRING: org object for myorg contains extra/missing fields. Fixing that for you +REPAIRING: Correcting `name` in organizations/myorg/cookbooks/my-cookbook-1.0.0/metadata.rb +INFO: Generating new metadata.json for organizations/myorg/cookbooks/my-cookbook-1.0.0 +** Finished ** +``` + +If any issues can't be repaired automatically, the command writes them to `knife-tidy-actions-needed.txt` and prints a warning at the end: + +```console +WARNING: ** Unrepairable Items ** +Please see knife-tidy-actions-needed.txt +``` + +### Key flags + +`--backup-path ` +: Path to the `knife-ec-backup` backup directory to clean. + Required unless `--gen-gsub` is set. + +`--gsub-file ` +: Path to a JSON substitutions file. + If the file doesn't exist yet, generate one first with `--gen-gsub`. + +`--gen-gsub` +: Generate a new boilerplate `substitutions.json` file in the current directory and exit. + Edit this file before passing it to `--gsub-file`. + +### Substitutions file format + +The substitutions file maps glob patterns to search-and-replace rules applied to files in the backup. +To generate a starting point, run `knife tidy backup clean --gen-gsub`. +The following example shows the file structure: + +```json +{ + "your-problem-descriptor": { + "organizations/*/cookbooks/*/metadata.rb": [ + { + "pattern": "^version .*GO_PIPELINE_LABEL", + "replace": "version !COOKBOOK_VERSION!" + } + ] + } +} +``` + +### How knife tidy backup clean works + +`knife tidy backup clean` validates and repairs common issues that prevent `knife ec restore` from completing successfully. +Repairs include: fixing or regenerating corrupt org objects, correcting missing or mismatched cookbook `name` fields in `metadata.rb` and `metadata.json`, regenerating `metadata.json` from `metadata.rb`, removing self-dependencies in cookbook metadata, fixing null metadata values, fixing invalid platform constraint arrays, repairing invalid run list items in roles, ensuring clients are present in the clients group, and removing corrupt org invitations. +If a cookbook can't be loaded after repairs, it's moved to a `cookbooks.broken/` directory within the org. +Global text substitutions from `--gsub-file` are applied after the structural repairs. + +## knife tidy notify + +```shell +knife tidy notify (options) +``` + +Reads the JSON report files produced by `knife tidy server report` and emails a summary to each organization's admin users through an SMTP server. +Run this command from the directory containing your `reports/` folder. + +### Common invocations + +```shell +# Send reports using a local SMTP relay +knife tidy notify --smtp_server smtp.example.com --smtp_from admin@example.com +``` + +```shell +# Send reports with SMTP authentication and TLS +knife tidy notify \ + --smtp_server smtp.example.com \ + --smtp_port 587 \ + --smtp_from admin@example.com \ + --smtp_username myuser \ + --smtp_password \ + --smtp_use_tls +``` + +```shell +# Send reports for specific organizations only +knife tidy notify --orgs brewinc,acmeinc \ + --smtp_server smtp.example.com --smtp_from admin@example.com +``` + +### Key flags + +`--smtp_server ` / `-s` +: SMTP server hostname (default: `localhost`). + +`--smtp_port ` / `-p` +: SMTP port (default: `25`). + +`--smtp_from
` +: From address for outgoing email reports. + +`--smtp_username ` / `-u` +: SMTP authentication username. + +`--smtp_password ` +: SMTP authentication password. + +`--smtp_use_tls` / `-t` +: Enable STARTTLS for the SMTP connection (default: `false`). + +`--smtp_helo ` / `-h` +: SMTP HELO domain (default: `localhost`). + +### How knife tidy notify works + +`knife tidy notify` reads the `reports/` directory, identifies organizations by parsing report filenames, and queries the Chef Infra Server for the admin users of each organization. +It generates an HTML email with three tables---total cookbook versions, unused cookbooks, and stale nodes---and attaches the raw JSON report files. +The email is sent to all admin users of each organization. +The `pivotal` superuser is excluded from the recipient list. +If report files for an organization can't be parsed, the command prints an error and exits without sending. diff --git a/tools/vale/chef/foreign.yml b/tools/vale/chef/foreign.yml index 2072b8c..fda2faa 100644 --- a/tools/vale/chef/foreign.yml +++ b/tools/vale/chef/foreign.yml @@ -10,7 +10,6 @@ swap: '\b(?:eg|e\.g\.)[\s,]': for example '\b(?:ie|i\.e\.)[\s,]': that is \bvia\b: through, with - '\b(?:per)[\s,]': for each colour: colour neighbour: neighbor apologise: apologize From c06c8b75becc83bd88f37642b4b050248a1cff5f Mon Sep 17 00:00:00 2001 From: Ian Maddaus Date: Fri, 15 May 2026 13:12:44 -0400 Subject: [PATCH 2/3] More updates to knife plugin docs Signed-off-by: Ian Maddaus --- .../tools/knife/plugins/knife_ec_backup.md | 322 +++++------------- content/tools/knife/plugins/knife_tidy.md | 42 ++- 2 files changed, 114 insertions(+), 250 deletions(-) diff --git a/content/tools/knife/plugins/knife_ec_backup.md b/content/tools/knife/plugins/knife_ec_backup.md index 4c4700f..f33c13d 100644 --- a/content/tools/knife/plugins/knife_ec_backup.md +++ b/content/tools/knife/plugins/knife_ec_backup.md @@ -12,315 +12,157 @@ title = "knife-ec-backup plugin" The `knife-ec-backup` plugin backs up and restores the data in a Chef Infra Server installation, including all organizations, users, nodes, cookbooks, roles, and associated ACLs. Use it before server migrations, version upgrades, or to create point-in-time snapshots of your Chef Infra Server. -## Install knife-ec-backup +## How knife ec backup works -```shell -gem install knife-ec-backup -``` +`knife ec backup` downloads all users and user ACLs first, then iterates through every organization and downloads org-level objects using the Chef Infra Server REST API. -## Common options +The backup writes everything to a directory tree under `/users/` and `/organizations//`. +If `--with-user-sql` or `--with-key-sql` is set, the command connects directly to PostgreSQL and writes `key_dump.json` and `key_table_dump.json` alongside the backup. -The following options are shared across all `knife ec` subcommands: +## Install knife-ec-backup -`--only-org ` -: Limit the operation to a single named organization. - By default, all organizations are included. +You can install knife-ec-backup as a Ruby gem or as a Habitat package. -`--concurrency ` -: Maximum number of simultaneous API requests (default: 10). +Install knife-ec-backup as a Ruby gem: -`--skip-useracl` -: Skip downloading or restoring user ACLs. - Required for Enterprise Chef 11.0.2 and earlier. +```shell +gem install knife-ec-backup +``` -`--skip-version-check` -: Skip the Chef Infra Server version check and all autoconfigured options. +Install knife-ec-backup as a Habitat package: -`--webui-key ` -: Path to the WebUI private key. - Defaults to reading from the secrets store, `/etc/opscode/webui_priv.pem`, or the Automate path. +```bash +hab pkg install chef/knife-ec-backup -bf +``` -`--secrets-file ` -: Path to `private-chef-secrets.json` (default: `/etc/opscode/private-chef-secrets.json`). +{{< note >}} -`--purge` -: Sync deletions from the backup source to the restore destination. +The `-bf` options binlink the `knife-ec-backup` binary into a system-wide directory in your `PATH` (for example, `/bin` or `/usr/local/bin`). +You can then run `knife-ec-backup` directly from your shell. -`--dry-run` -: Report what actions would be taken without performing any. +{{< /note >}} -`--error-log-dir ` -: Directory where errors are logged during the operation. +## Syntax -## knife ec backup +If you've installed `knife-ec-backup` as a Ruby gem or binlinked the Habitat package: ```shell knife ec backup ``` -Downloads all users, user ACLs, organizations, cookbooks, nodes, roles, environments, data bags, groups, and ACLs from a Chef Infra Server to a local directory. - -### Common invocations +If you've installed `knife-ec-backup` as a Habitat package and didn't binlink it, execute it using the Habitat CLI and the full package name: ```shell -# Back up all organizations to a local directory -knife ec backup /var/backups/chef-server +hab pkg exec chef/knife-ec-backup knife ec backup ``` -```shell -# Back up a single organization -knife ec backup /var/backups/chef-server --only-org org-name -``` +### Options and flags -```shell -# Back up including user and key table data from PostgreSQL -knife ec backup /var/backups/chef-server --with-user-sql --with-key-sql \ - --sql-user opscode_chef --sql-password -``` +`--concurrency ` +: Maximum number of simultaneous API requests (default: 10). -```shell -# Back up and remove local users that no longer exist on the server -knife ec backup /var/backups/chef-server --purge -``` +`--dry-run` +: Report what actions would be taken without performing any. -### Key flags +`--error-log-dir ` +: Directory where errors are logged during the operation. -`--with-user-sql` -: Export user data directly from the PostgreSQL database. - Required to correctly preserve user passwords and keys. +`--only-org ` +: Limit the operation to a single named organization. + By default, all organizations are included. -`--with-key-sql` -: Export rotated key data from the PostgreSQL `keys` table. - Required for Chef Infra Server 12 and later when keys have been rotated. +`--purge` +: Sync deletions from the backup source to the restore destination. + +`--secrets-file ` +: Path to `private-chef-secrets.json` (default: `/etc/opscode/private-chef-secrets.json`). `--skip-frozen-cookbook-status` : Skip writing a `status.json` file for each cookbook. Use this when you don't want to persist a cookbook's frozen status across restores. +`--skip-useracl` +: Skip downloading or restoring user ACLs. + Required for Enterprise Chef 11.0.2 and earlier. + +`--skip-version-check` +: Skip the Chef Infra Server version check and all autoconfigured options. + `--sql-host ` : PostgreSQL hostname (default: `localhost`). +`--sql-password ` +: PostgreSQL password. + `--sql-port ` : PostgreSQL port (default: `5432`). `--sql-user ` : PostgreSQL user for the `opscode_chef` database. -`--sql-password ` -: PostgreSQL password. - -### How knife ec backup works - -`knife ec backup` downloads all users and user ACLs first, then iterates through every organization and downloads org-level objects using the Chef Infra Server REST API. -On Enterprise Chef 11 and earlier, pre-created unassigned organizations are skipped automatically. -The backup writes everything to a directory tree under `/users/` and `/organizations//`. -If `--with-user-sql` or `--with-key-sql` is set, the command connects directly to PostgreSQL and writes `key_dump.json` and `key_table_dump.json` alongside the backup. - -## knife ec restore - -```shell -knife ec restore -``` - -Restores all users, organizations, cookbooks, and ACLs from a local backup directory to a Chef Infra Server. -Run this after `knife ec backup` to migrate data to a new server or recover from failure. - -### Common invocations - -```shell -# Restore all data from a backup directory -knife ec restore /var/backups/chef-server -``` - -```shell -# Restore a single organization only -knife ec restore /var/backups/chef-server --only-org org-name -``` - -```shell -# Restore without restoring users (for example, when users are managed by LDAP) -knife ec restore /var/backups/chef-server --skip-users -``` - -```shell -# Restore and overwrite pivotal's key (use with caution) -knife ec restore /var/backups/chef-server --overwrite-pivotal -``` - -### Key flags - -`--skip-users` -: Skip restoring global users. - Use this when users are managed externally, for example through LDAP or SAML. - -`--overwrite-pivotal` -: Overwrite the `pivotal` superuser key during restore. - After using this flag, all future API requests will fail until you update the private key on the server. - By default, `pivotal` is skipped. - -`--[no-]skip-user-ids` -: When enabled (default), reuses user IDs from the restore destination to avoid database conflicts. - Pass `--no-skip-user-ids` to use IDs from the backup. - -### How knife ec restore works +`--webui-key ` +: Path to the WebUI private key. + Defaults to reading from the secrets store, `/etc/opscode/webui_priv.pem`, or the Automate path. -`knife ec restore` reads the directory structure written by `knife ec backup`. -It restores users first (skipping `pivotal` unless `--overwrite-pivotal` is set), then creates or updates each organization, restores open invitations, adds users to orgs, and uploads all org-level data. -If an organization already exists on the target server, the command updates it rather than failing. -User ACLs are restored last; pass `--skip-useracl` to skip this step if the target server runs Enterprise Chef 11.0.2 or earlier. +`--with-key-sql` +: Export rotated key data from the PostgreSQL `keys` table. + Required for Chef Infra Server 12 and later when keys have been rotated. -## knife ec import +`--with-user-sql` +: Export user data directly from the PostgreSQL database. + Required to correctly preserve user passwords and keys. -```shell -knife ec import -``` +## Examples -Imports objects from a backup directory into a Chef Infra Server. -`knife ec import` is similar to `knife ec restore` but designed for Automate-managed servers and environments where a tenant ID header is required. +Use the following examples to back up Chef Infra Server data for common scenarios. -### Common invocations +### Back up all organizations -```shell -# Import all organizations from a backup directory -knife ec import /var/backups/chef-server -``` +To create a full backup of all organizations and global objects in a local directory, run: ```shell -# Import with a tenant identifier header (required for some Automate deployments) -knife ec import /var/backups/chef-server --tenant-id tenant-uuid +knife ec backup /var/backups/chef-server ``` -### Key flags +### Back up a single organization -`--tenant-id ` -: Adds an `X-Tenant-Id` header to all import requests. - Required for Chef Automate deployments that use tenant-based routing. - -## knife ec key export +To back up only one organization, pass the organization name with `--only-org`: ```shell -knife ec key export [] [] +knife ec backup /var/backups/chef-server --only-org org-name ``` -Exports user and key table data directly from the Chef Infra Server PostgreSQL database to JSON files. -This command is invoked automatically by `knife ec backup --with-user-sql` and `--with-key-sql`, but you can also run it independently. - -### Common invocations - -```shell -# Export to default output files (key_dump.json and key_table_dump.json) -knife ec key export \ - --sql-user opscode_chef --sql-password -``` +### Back up with PostgreSQL user and key data -```shell -# Export to custom output file paths -knife ec key export user_table.json key_table.json \ - --sql-user opscode_chef --sql-password -``` +To include user and key-table exports from PostgreSQL, add `--with-user-sql` and `--with-key-sql`: ```shell -# Export only the keys table, skipping the users table -knife ec key export --skip-users-table \ +knife ec backup /var/backups/chef-server --with-user-sql --with-key-sql \ --sql-user opscode_chef --sql-password ``` -### Expected output - -```console -knife ec key export --sql-user opscode_chef --sql-password secret -``` - -On success, the command writes `key_dump.json` and `key_table_dump.json` in the current directory (or the paths you provided) and exits silently. - -### Key flags - -`--skip-users-table` -: Skip exporting the `users` table. - Use this when you only need rotated key data. - -`--skip-keys-table` -: Skip exporting the `keys_by_name` table. - Use this flag with Chef Infra Server 11, which doesn't have a keys table. - -`--sql-host ` -: PostgreSQL hostname (default: `localhost`). - -`--sql-port ` -: PostgreSQL port (default: `5432`). +### Back up and purge deleted objects -`--sql-user ` -: PostgreSQL user for the `opscode_chef` database. - -`--sql-password ` -: PostgreSQL password. - -### How knife ec key export works - -`knife ec key export` connects directly to the Chef Infra Server PostgreSQL database using the Sequel gem. -It queries the `users` table (written to `key_dump.json` by default) and the `keys_by_name` table joined with `orgs` (written to `key_table_dump.json` by default). -If you pass custom file name arguments, those paths are used instead. -On Chef Infra Server 11, use `--skip-keys-table` to avoid a `PG::UndefinedTable` error. - -## knife ec key import +To remove objects from the destination that no longer exist on the source server, use `--purge`: ```shell -knife ec key import [] [] -``` - -Imports user and key table data from JSON files directly into the Chef Infra Server PostgreSQL database. -Run this after `knife ec restore` when you used `--with-user-sql` or `--with-key-sql` during backup. - -### Common invocations - -```shell -# Import from default file paths (key_dump.json and key_table_dump.json) -knife ec key import \ - --sql-user opscode_chef --sql-password +knife ec backup /var/backups/chef-server --purge ``` -```shell -# Import from custom file paths -knife ec key import user_table.json key_table.json \ - --sql-user opscode_chef --sql-password -``` +### Create a full backup -```shell -# Import only user data, skipping client key data -knife ec key import --users-only \ - --sql-user opscode_chef --sql-password -``` +Create a full backup using required authentication and key flags: -```shell -# Import only client key data, skipping user table -knife ec key import --clients-only \ - --sql-user opscode_chef --sql-password +```bash +knife ec backup backup_$(date '+%Y%m%d%H%M%s') \ + --webui-key /etc/opscode/webui_priv.pem \ + --with-key-sql \ + -s https://chef.example.com \ + -u pivotal \ + -k /etc/opscode/pivotal.pem ``` -### Key flags - -`--[no-]skip-pivotal` -: Skip importing the `pivotal` user key (default: enabled). - Pass `--no-skip-pivotal` to upload the pivotal key from the backup. - -`--[no-]skip-user-ids` -: Reuse user IDs from the restore destination to avoid database conflicts (default: enabled). - Pass `--no-skip-user-ids` to use IDs from the backup files. - -`--users-only` -: Import only user data from the users table; skip client key data. - -`--clients-only` -: Import only client key data; skip the users table. - -`--skip-users-table` -: Skip importing the users table entirely. - -`--skip-keys-table` -: Skip importing the `keys_by_name` table. - -### How knife ec key import works +## More information -`knife ec key import` reads the JSON files written by `knife ec key export` and writes directly to the Chef Infra Server PostgreSQL database. -For clients, the organization ID is looked up dynamically by org name (with caching) because org IDs differ between source and destination servers. -Organizations must exist on the destination server before you run `knife ec key import`---run `knife ec restore` first. +- [knife-ec-backup GitHub repository](https://github.com/chef/knife-ec-backup/) +- [knife tidy](/tools/knife/plugins/knife_tidy/) diff --git a/content/tools/knife/plugins/knife_tidy.md b/content/tools/knife/plugins/knife_tidy.md index 45c20de..4519304 100644 --- a/content/tools/knife/plugins/knife_tidy.md +++ b/content/tools/knife/plugins/knife_tidy.md @@ -9,27 +9,44 @@ title = "knife-tidy plugin" +++ The `knife-tidy` plugin reports on and removes stale nodes, associated clients, ACLs, and unused cookbook versions from a Chef Infra Server. -It also cleans data-integrity issues from [`knife-ec-backup`](/tools/knife/plugins/knife-ec-backup/) directories before you restore them to a server. +It also cleans data-integrity issues from [`knife-ec-backup`](/tools/knife/plugins/knife_ec_backup/) directories before you restore them to a server. ## Install knife-tidy -`knife tidy` is available as part of the `knife-ec-backup` Habitat package or as a standalone Ruby gem. +You can install `knife-tidy` as a Ruby gem or as a Habitat package. -- Install the Habitat package (recommended): +- Install `knife-tidy` as a Ruby gem: - ```bash - hab pkg install chef/knife-ec-backup -bf + ```shell + gem install knife-tidy ``` - After installation, prefix all `knife tidy` commands with `hab pkg exec chef/knife-ec-backup`. - -- Install the Ruby gem: +- Install `knife-tidy` as a Habitat package: ```bash - gem install knife-tidy + hab pkg install chef/knife-tidy -bf ``` - After installation, run `knife tidy` commands directly without the `hab pkg exec` prefix. + {{< note >}} + + The `-bf` options binlink the `knife-tidy` binary into a system-wide directory in your `PATH` (for example, `/bin` or `/usr/local/bin`). + You can then run `knife-tidy` directly from your shell. + + {{< /note >}} + +## Syntax + +If `knife-tidy` is installed as a Ruby gem or as a binlinked Chef Habitat package: + +```shell +knife tidy (options) +``` + +If you've installed `knife-tidy` as a Habitat package and didn't binlink it, execute it using the Habitat CLI and the full package name: + +```shell +hab pkg exec chef/knife-ec-backup knife tidy (options) +``` ## Common options @@ -310,3 +327,8 @@ It generates an HTML email with three tables---total cookbook versions, unused c The email is sent to all admin users of each organization. The `pivotal` superuser is excluded from the recipient list. If report files for an organization can't be parsed, the command prints an error and exits without sending. + +## More information + +- [knife-tidy GitHub repository](https://github.com/chef/knife-tidy/) +- [knife ec backup](/tools/knife/plugins/knife_ec_backup/) From c5e98d5b2a0a968a2cce1f6750b831c3dbc83524 Mon Sep 17 00:00:00 2001 From: Ian Maddaus Date: Fri, 15 May 2026 14:53:14 -0400 Subject: [PATCH 3/3] Cspell lint fixes Signed-off-by: Ian Maddaus --- content/tools/knife/plugins/knife_ec_backup.md | 2 ++ content/tools/knife/plugins/knife_tidy.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/content/tools/knife/plugins/knife_ec_backup.md b/content/tools/knife/plugins/knife_ec_backup.md index f33c13d..c1e0035 100644 --- a/content/tools/knife/plugins/knife_ec_backup.md +++ b/content/tools/knife/plugins/knife_ec_backup.md @@ -9,6 +9,8 @@ title = "knife-ec-backup plugin" parent = "tools/knife/plugins" +++ + + The `knife-ec-backup` plugin backs up and restores the data in a Chef Infra Server installation, including all organizations, users, nodes, cookbooks, roles, and associated ACLs. Use it before server migrations, version upgrades, or to create point-in-time snapshots of your Chef Infra Server. diff --git a/content/tools/knife/plugins/knife_tidy.md b/content/tools/knife/plugins/knife_tidy.md index 4519304..9ca9b2c 100644 --- a/content/tools/knife/plugins/knife_tidy.md +++ b/content/tools/knife/plugins/knife_tidy.md @@ -8,6 +8,8 @@ title = "knife-tidy plugin" parent = "tools/knife/plugins" +++ + + The `knife-tidy` plugin reports on and removes stale nodes, associated clients, ACLs, and unused cookbook versions from a Chef Infra Server. It also cleans data-integrity issues from [`knife-ec-backup`](/tools/knife/plugins/knife_ec_backup/) directories before you restore them to a server.