Skip to content

[v2] Add more granular metrics to benchmarking framework: plugin-load time, static imports time, client creation time.#10292

Open
aemous wants to merge 9 commits into
aws:v2from
aemous:plugin-load-time-perf
Open

[v2] Add more granular metrics to benchmarking framework: plugin-load time, static imports time, client creation time.#10292
aemous wants to merge 9 commits into
aws:v2from
aemous:plugin-load-time-perf

Conversation

@aemous
Copy link
Copy Markdown
Contributor

@aemous aemous commented May 8, 2026

Description of changes:

  • The benchmarking framework (callable from ./scripts/performance/run-benchmarks) now emits plugin-load time, static imports time, and client creation time.
  • Fixed bug with performance framework on IMDS-compatible instances where the CLI would attempt to reach the IMDS endpoints to retrieve credentials but the stubbed responses did not support this. Now, we supply mock credentials in the stubbed config files in each of our stubbed JSON benchmarks.
  • In vendored botocore, emit before-create-ciient and after-create-client before and after client creation, respectively.
  • In create_clidriver, add a new optional event_hooks argument to allow callers to provide their own event emitter. This change was needed so the benchmark framework can register against the load_plugins event, which is emitted during create_clidriver.
  • Modified --debug-dir parameter in run-benchmarks script so that it supports relative paths (previously it only supported absolute paths).

Description of tests:

  • Manually ran the benchmarking framework locally and on an IMDS-compatible EC2 instance, and verified the numbers are successfully being emitted, and are in an expected range aligned with previous benchmarking efforts.
  • Manually tested the --debug-dir parameter with relative and absolute paths, and verified expected behavior in both cases.
  • Successfully ran an internal pre-prod build workflow.

Example plugin-load time emission:

{
      "name": "cloudwatch.getmetricdata.plugins.import.time",
      "description": "Total time spent loading all built-in plugins.",
      "unit": "Seconds",
      "dimensions": [],
      "measurements": [
        0.08212709426879883
      ]
}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@aemous aemous requested a review from a team May 8, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant