feat: add token usage metrics with OpenTelemetry integration#563
Open
ajbozarth wants to merge 8 commits intogenerative-computing:mainfrom
Open
feat: add token usage metrics with OpenTelemetry integration#563ajbozarth wants to merge 8 commits intogenerative-computing:mainfrom
ajbozarth wants to merge 8 commits intogenerative-computing:mainfrom
Conversation
Add mellea.llm.tokens.input/output counters following Gen-AI semantic conventions with zero overhead when disabled Signed-off-by: Alex Bozarth <ajbozart@us.ibm.com>
…LM backends Add record_token_usage_metrics() calls to all backend post_processing methods to track input/output tokens. Add get_value() helper in backends/utils.py to handle dict/object attribute extraction. Signed-off-by: Alex Bozarth <ajbozart@us.ibm.com>
Calculate token counts from input_ids and output sequences. Records to both tracing spans and metrics using helper function. Signed-off-by: Alex Bozarth <ajbozart@us.ibm.com>
- Add integration tests for Ollama, OpenAI, LiteLLM, HuggingFace, WatsonX - Tests revealed metrics were coupled with tracing (architectural issue) - Fixed: Metrics now record independently of tracing spans - WatsonX: Store full response to preserve usage information - HuggingFace: Add zero-overhead guard, optimize test model Signed-off-by: Alex Bozarth <ajbozart@us.ibm.com>
…ation Use MonkeyPatch for cleanup and update Watsonx to granite-4-h-small. Signed-off-by: Alex Bozarth <ajbozart@us.ibm.com>
- Add Token Usage Metrics section to docs/dev/telemetry.md with metric definitions, backend support table, and configuration examples - Create metrics_example.py demonstrating token tracking with tested console output - Update telemetry_example.py to reference new metrics example - Update examples/telemetry/README.md with metrics quick start guide Signed-off-by: Alex Bozarth <ajbozart@us.ibm.com>
Contributor
|
The PR description has been updated. Please fill out the template for your PR to be reviewed. |
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🟢 Enforce conventional commitWonderful, this rule succeeded.Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
Signed-off-by: Alex Bozarth <ajbozart@us.ibm.com>
Contributor
Author
|
After opening this I had Bob and Claude to in depth reviews and they came back with a handful of things I want to address. I will work on fixing those tomorrow |
Signed-off-by: Alex Bozarth <ajbozart@us.ibm.com>
Contributor
Author
|
I've push a small update to test and doc streaming support, as suggested by AI review. As of now this is ready for full review and merge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Misc PR
Type of PR
Description
Summary
Adds token usage metrics tracking across all Mellea backends using OpenTelemetry metrics counters, following Gen-AI Semantic Conventions for standardized observability.
Changes
Core Implementation
record_token_usage_metrics()function inmellea/telemetry/metrics.pymellea.llm.tokens.input,mellea.llm.tokens.output)MELLEA_METRICS_CONSOLE=true)Configuration
MELLEA_METRICS_ENABLED(default:false)MELLEA_METRICS_CONSOLE(default:false)OTEL_EXPORTER_OTLP_ENDPOINTMetrics Attributes
All token metrics include Gen-AI semantic convention attributes:
gen_ai.system- Backend system name (e.g.,openai,ollama)gen_ai.request.model- Model identifiermellea.backend- Backend class nameTesting
Documentation
docs/dev/telemetry.mdwith complete metrics documentationBackend Support
usage.prompt_tokens,usage.completion_tokensprompt_eval_count,eval_countinput_token_count,generated_token_countusage.prompt_tokens,usage.completion_tokensBreaking Changes
None - metrics are disabled by default and require explicit opt-in via
MELLEA_METRICS_ENABLED=true.Testing