Skip to content

Add Arize and Phoenix LLM observability skills#1204

Open
jimbobbennett wants to merge 6 commits intogithub:stagedfrom
jimbobbennett:add-arize-skills
Open

Add Arize and Phoenix LLM observability skills#1204
jimbobbennett wants to merge 6 commits intogithub:stagedfrom
jimbobbennett:add-arize-skills

Conversation

@jimbobbennett
Copy link
Copy Markdown

@jimbobbennett jimbobbennett commented Mar 27, 2026

Pull Request Checklist

  • I have read and followed the CONTRIBUTING.md guidelines.
  • I have read and followed the Guidance for submissions involving paid services.
  • My contribution adds a new instruction, prompt, agent, skill, or workflow file in the correct directory.
  • The file follows the required naming convention.
  • The content is clearly structured and follows the example format.
  • I have tested my instructions, prompt, agent, skill, or workflow with GitHub Copilot.
  • I have run npm start and verified that README.md is up to date.
  • I am targeting the staged branch for this pull request.

Description

Add skills for Arize AI platform covering trace export, instrumentation, datasets, experiments, evaluators, AI provider integrations, annotations, prompt optimization, and deep linking to the Arize UI.

Arize Skills source - https://github.com/Arize-ai/arize-skills
Phoenix Skills source - https://github.com/Arize-ai/phoenix/tree/main/skills

We will be adding an action to auto publish to this repo when we publish changes, as discussed in #1169


Type of Contribution

  • New instruction file.
  • New prompt file.
  • New agent file.
  • New plugin.
  • New skill file.
  • New agentic workflow.
  • Update to existing instruction, prompt, agent, plugin, skill, or workflow.
  • Other (please specify):

Additional Notes


By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.

Add skills for Arize AI platform covering trace export, instrumentation,
datasets, experiments, evaluators, AI provider integrations, annotations,
prompt optimization, and deep linking to the Arize UI.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jimbobbennett jimbobbennett changed the base branch from main to staged March 27, 2026 23:10
Add skills for Phoenix (Arize open-source) covering CLI debugging,
LLM evaluation workflows, and OpenInference tracing/instrumentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jimbobbennett jimbobbennett changed the title Add 9 Arize LLM observability skills Add Arize and Phoenix LLM observability skills Mar 27, 2026
jimbobbennett and others added 3 commits March 27, 2026 16:20
…ll ignore

Remove .DS_Store artifact from winmd-api-search asset listing in generated
README.skills.md so it matches the CI Linux build output. Add queston to
codespell ignore list (intentional misspelling example in arize-dataset skill).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bundle the 9 Arize skills into an arize-ax plugin and the 3 Phoenix
skills into a phoenix plugin for easier installation as single packages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jimbobbennett jimbobbennett marked this pull request as ready for review March 27, 2026 23:47
Copilot AI review requested due to automatic review settings March 27, 2026 23:47
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Arize/Phoenix observability content to the repo as installable Copilot CLI plugins, including new skill documentation for Phoenix tracing/evals and Arize AX workflows.

Changes:

  • Introduces new Phoenix plugin with tracing/evals/CLI skills and extensive reference docs.
  • Introduces new Arize AX plugin bundling multiple AX-focused skills (trace export, datasets/experiments/evaluators, annotations, prompt optimization, deep links).
  • Registers both plugins in the docs index and marketplace metadata; updates codespell ignore list for an intentional example typo.

Reviewed changes

Copilot reviewed 101 out of 101 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
skills/phoenix-tracing/SKILL.md Index for Phoenix tracing skill (setup/instrumentation/span kinds/production/feedback).
skills/phoenix-tracing/references/README.md Reference README describing the tracing docs organization.
skills/phoenix-tracing/references/fundamentals-overview.md Introduces trace/span concepts and hierarchy.
skills/phoenix-tracing/references/fundamentals-required-attributes.md Defines required/recommended attributes and valid OpenInference span kinds.
skills/phoenix-tracing/references/fundamentals-universal-attributes.md Documents cross-cutting attributes like I/O, session/user, metadata.
skills/phoenix-tracing/references/fundamentals-flattening.md Explains dot-notation flattening rules for OpenInference attributes.
skills/phoenix-tracing/references/span-llm.md LLM span attribute guide and examples.
skills/phoenix-tracing/references/span-chain.md CHAIN span attribute guide and examples.
skills/phoenix-tracing/references/span-retriever.md RETRIEVER span schema (flattened documents) and examples.
skills/phoenix-tracing/references/span-reranker.md RERANKER span schema and examples.
skills/phoenix-tracing/references/span-tool.md TOOL span schema and examples.
skills/phoenix-tracing/references/span-agent.md AGENT span overview and example.
skills/phoenix-tracing/references/span-embedding.md EMBEDDING span schema and examples.
skills/phoenix-tracing/references/span-guardrail.md GUARDRAIL span schema and examples.
skills/phoenix-tracing/references/span-evaluator.md EVALUATOR span schema and examples.
skills/phoenix-tracing/references/setup-python.md Python install/config for Phoenix OTEL tracing.
skills/phoenix-tracing/references/setup-typescript.md TypeScript/JS install/config for Phoenix OTEL tracing.
skills/phoenix-tracing/references/instrumentation-auto-python.md Python auto-instrumentation guidance for supported libraries.
skills/phoenix-tracing/references/instrumentation-auto-typescript.md TypeScript auto-instrumentation guidance and limitations.
skills/phoenix-tracing/references/instrumentation-manual-python.md Python manual instrumentation patterns (decorators/context managers).
skills/phoenix-tracing/references/instrumentation-manual-typescript.md TypeScript manual instrumentation patterns (wrappers/withSpan).
skills/phoenix-tracing/references/projects-python.md Project naming/grouping patterns for Phoenix (Python).
skills/phoenix-tracing/references/projects-typescript.md Project naming/grouping patterns for Phoenix (TypeScript).
skills/phoenix-tracing/references/sessions-python.md Session tracking via context propagation (Python).
skills/phoenix-tracing/references/sessions-typescript.md Session tracking patterns using withSpan and/or Context API (TypeScript).
skills/phoenix-tracing/references/metadata-python.md Adding session/user/metadata/tags via openinference instrumentation helpers (Python).
skills/phoenix-tracing/references/metadata-typescript.md Adding custom metadata using context/span APIs (TypeScript).
skills/phoenix-tracing/references/production-python.md Production masking/filtering examples (Python).
skills/phoenix-tracing/references/production-typescript.md Production batching/masking/filtering/shutdown patterns (TypeScript).
skills/phoenix-tracing/references/annotations-overview.md Conceptual overview of annotations and fields.
skills/phoenix-tracing/references/annotations-python.md Python client patterns for span/trace/session/document annotations.
skills/phoenix-tracing/references/annotations-typescript.md TypeScript client patterns for span/trace/session/document annotations.
skills/phoenix-evals/SKILL.md Index for Phoenix evals skill (setup, evaluators, datasets/experiments, production).
skills/phoenix-evals/references/setup-python.md Phoenix evals/experiments package setup (Python) with “Evals 2.0” import guidance.
skills/phoenix-evals/references/setup-typescript.md Phoenix evals/experiments package setup (TypeScript).
skills/phoenix-evals/references/fundamentals.md High-level eval concepts and score structure/patterns.
skills/phoenix-evals/references/fundamentals-model-selection.md Guidance for judge model selection and iteration.
skills/phoenix-evals/references/fundamentals-anti-patterns.md Common mistakes and corrective patterns.
skills/phoenix-evals/references/evaluators-overview.md Framework for when/how to build evaluators.
skills/phoenix-evals/references/evaluators-pre-built.md Notes on pre-built evaluators and validation requirement.
skills/phoenix-evals/references/evaluators-code-python.md Patterns for deterministic evaluators (Python).
skills/phoenix-evals/references/evaluators-code-typescript.md Patterns for deterministic evaluators (TypeScript).
skills/phoenix-evals/references/evaluators-llm-python.md Patterns for LLM-based evaluators (Python).
skills/phoenix-evals/references/evaluators-llm-typescript.md Patterns for LLM-based evaluators (TypeScript).
skills/phoenix-evals/references/evaluators-custom-templates.md Template structure guidance for LLM judges.
skills/phoenix-evals/references/evaluators-rag.md RAG-specific eval approach (retrieval vs generation).
skills/phoenix-evals/references/experiments-overview.md Core structure of dataset/task/evaluators/experiment.
skills/phoenix-evals/references/experiments-datasets-python.md Dataset creation and usage patterns (Python).
skills/phoenix-evals/references/experiments-datasets-typescript.md Dataset creation and usage patterns (TypeScript).
skills/phoenix-evals/references/experiments-running-python.md Running experiments and interpreting results (Python).
skills/phoenix-evals/references/experiments-running-typescript.md Running experiments and interpreting results (TypeScript).
skills/phoenix-evals/references/experiments-synthetic-python.md Synthetic dataset generation patterns (Python).
skills/phoenix-evals/references/experiments-synthetic-typescript.md Synthetic dataset generation patterns (TypeScript).
skills/phoenix-evals/references/evaluate-dataframe-python.md Batch evaluation API details and result extraction (Python).
skills/phoenix-evals/references/validation.md Evaluator validation metrics and dataset practices.
skills/phoenix-evals/references/validation-evaluators-python.md Evaluator validation computations/patterns (Python).
skills/phoenix-evals/references/validation-evaluators-typescript.md Evaluator validation experiment pattern (TypeScript).
skills/phoenix-evals/references/observe-tracing-setup.md Capturing the right tracing attributes to enable eval workflows.
skills/phoenix-evals/references/observe-sampling-python.md Sampling strategies for reviewing production traces (Python).
skills/phoenix-evals/references/observe-sampling-typescript.md Sampling strategies for reviewing production traces (TypeScript).
skills/phoenix-evals/references/error-analysis.md Workflow for discovering failure modes from traces.
skills/phoenix-evals/references/error-analysis-multi-turn.md Guidance for analyzing multi-turn conversation failures.
skills/phoenix-evals/references/axial-coding.md Method to group notes into failure taxonomies and annotate spans.
skills/phoenix-evals/references/production-overview.md CI evals vs production monitoring overview (Python/TS examples).
skills/phoenix-evals/references/production-guardrails.md Guardrails vs evaluators distinction.
skills/phoenix-evals/references/production-continuous.md Continuous evaluation loop and alerting guidance.
skills/phoenix-cli/SKILL.md Phoenix CLI skill describing trace/span/dataset/experiment/query workflows.
skills/arize-trace/references/ax-setup.md AX CLI troubleshooting reference.
skills/arize-trace/references/ax-profiles.md AX profile/auth setup reference and security guidance for keys.
skills/arize-prompt-optimization/references/ax-setup.md AX CLI troubleshooting reference (prompt optimization skill).
skills/arize-prompt-optimization/references/ax-profiles.md AX profile/auth setup reference (prompt optimization skill).
skills/arize-link/references/EXAMPLES.md Concrete Arize UI deep-link URL templates.
skills/arize-link/SKILL.md Deep-link generation skill instructions and templates.
skills/arize-instrumentation/references/ax-profiles.md AX profile/auth setup reference (instrumentation skill).
skills/arize-experiment/references/ax-setup.md AX CLI troubleshooting reference (experiment skill).
skills/arize-experiment/references/ax-profiles.md AX profile/auth setup reference (experiment skill).
skills/arize-evaluator/references/ax-setup.md AX CLI troubleshooting reference (evaluator skill).
skills/arize-evaluator/references/ax-profiles.md AX profile/auth setup reference (evaluator skill).
skills/arize-dataset/references/ax-setup.md AX CLI troubleshooting reference (dataset skill).
skills/arize-dataset/references/ax-profiles.md AX profile/auth setup reference (dataset skill).
skills/arize-annotation/references/ax-setup.md AX CLI troubleshooting reference (annotation skill).
skills/arize-annotation/references/ax-profiles.md AX profile/auth setup reference (annotation skill).
skills/arize-ai-provider-integration/references/ax-setup.md AX CLI troubleshooting reference (AI provider integration skill).
skills/arize-ai-provider-integration/references/ax-profiles.md AX profile/auth setup reference (AI provider integration skill).
plugins/phoenix/README.md Plugin README describing included Phoenix skills.
plugins/phoenix/.github/plugin/plugin.json Plugin manifest for the Phoenix plugin.
plugins/arize-ax/README.md Plugin README describing included Arize AX skills.
plugins/arize-ax/.github/plugin/plugin.json Plugin manifest for the Arize AX plugin.
docs/README.plugins.md Adds both plugins to the plugins documentation table.
.github/plugin/marketplace.json Registers both plugins in the marketplace metadata.
.codespellrc Adds an ignore word for an intentional misspelling example.

Comment on lines +21 to +23
"./skills/phoenix-cli/",
"./skills/phoenix-evals/",
"./skills/phoenix-tracing/"
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plugin.json skills entries use paths like ./skills/phoenix-cli/, but there is no skills/ directory under plugins/phoenix/.github/plugin/ (or under plugins/phoenix/). These paths will resolve to non-existent locations when loading the plugin. Update the paths to correctly reference the intended skill directories (likely under the repo root).

Suggested change
"./skills/phoenix-cli/",
"./skills/phoenix-evals/",
"./skills/phoenix-tracing/"
"../../../skills/phoenix-cli/",
"../../../skills/phoenix-evals/",
"../../../skills/phoenix-tracing/"

Copilot uses AI. Check for mistakes.
Comment on lines +22 to +30
"./skills/arize-ai-provider-integration/",
"./skills/arize-annotation/",
"./skills/arize-dataset/",
"./skills/arize-evaluator/",
"./skills/arize-experiment/",
"./skills/arize-instrumentation/",
"./skills/arize-link/",
"./skills/arize-prompt-optimization/",
"./skills/arize-trace/"
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plugin.json skills entries use paths like ./skills/arize-trace/. Unless this plugin includes its own skills/ folder colocated with the plugin.json, these relative paths will point to non-existent locations. Please verify the expected resolution base for plugin item paths and adjust these skill paths accordingly so the plugin loads its skills.

Suggested change
"./skills/arize-ai-provider-integration/",
"./skills/arize-annotation/",
"./skills/arize-dataset/",
"./skills/arize-evaluator/",
"./skills/arize-experiment/",
"./skills/arize-instrumentation/",
"./skills/arize-link/",
"./skills/arize-prompt-optimization/",
"./skills/arize-trace/"
"../../skills/arize-ai-provider-integration/",
"../../skills/arize-annotation/",
"../../skills/arize-dataset/",
"../../skills/arize-evaluator/",
"../../skills/arize-experiment/",
"../../skills/arize-instrumentation/",
"../../skills/arize-link/",
"../../skills/arize-prompt-optimization/",
"../../skills/arize-trace/"

Copilot uses AI. Check for mistakes.
Comment on lines +103 to +116
```bash
# By category prefix
rules/setup-* # Installation and configuration
rules/instrumentation-* # Auto and manual tracing
rules/span-* # Span type specifications
rules/sessions-* # Session tracking
rules/production-* # Production deployment
rules/fundamentals-* # Core concepts
rules/attributes-* # Attribute specifications

# By language
rules/*-python.md # Python implementations
rules/*-typescript.md # TypeScript implementations
```
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The skill docs refer to a rules/ directory (including the navigation examples), but the repository structure for this skill uses references/. This makes the paths in the instructions incorrect and will confuse users following the guidance. Update the references from rules/ to the actual directory name used by this skill (or rename the folder to match).

Copilot uses AI. Check for mistakes.
Comment on lines +81 to +90
### Reference Files

- `fundamentals-overview` - Traces, spans, attributes basics
- `fundamentals-required-attributes` - Required fields per span type
- `fundamentals-universal-attributes` - Common attributes (user.id, session.id)
- `fundamentals-flattening` - JSON flattening rules
- `attributes-messages` - Chat message format
- `attributes-metadata` - Custom metadata schema
- `attributes-graph` - Agent workflow attributes
- `attributes-exceptions` - Error tracking
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This SKILL index lists several attributes-* reference files (messages/metadata/graph/exceptions), but those files do not exist under skills/phoenix-tracing/references/. Either add the missing reference docs or remove/replace these entries so the index doesn’t point to dead links.

Copilot uses AI. Check for mistakes.
Comment on lines +9 to +16
## File Organization

All files in flat `rules/` directory with semantic prefixes:

- `span-*` - Span kinds (LLM, CHAIN, TOOL, etc.)
- `setup-*`, `instrumentation-*` - Getting started guides
- `fundamentals-*`, `attributes-*` - Reference docs
- `annotations-*`, `export-*` - Advanced features
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This README states that all files live in a rules/ directory, but the skill actually stores them under references/. Please align the directory name in this description with the real folder structure so users can find the files.

Copilot uses AI. Check for mistakes.
Comment on lines +96 to +101
npx @arizeai/phoenix-cli traces \
--endpoint http://localhost:6006 \
--project your-app \
--format raw \
--no-progress | \
jq '.[] | select(.spans[0].attributes["session.id"] == "YOUR-SESSION-ID")'
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shell example has an extra pipe before the line continuation (--no-progress | \), which will break the command. Remove the stray | so the pipeline to jq works as intended.

Copilot uses AI. Check for mistakes.
Move arize supporting files from references/ to root level and rename
phoenix references/ to rules/ to exactly match the original source
repository folder structures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 101 out of 101 changed files in this pull request and generated 14 comments.

Comment on lines +8 to +9
import { createClient } from "phoenix-client";
const client = createClient(); // Default: http://localhost:6006
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TypeScript examples import from the phoenix-client npm package, which is unrelated to Arize Phoenix and will cause these examples to fail for users following this guide. Use @arizeai/phoenix-client (and the appropriate subpath exports such as @arizeai/phoenix-client/spans) consistently instead.

Copilot uses AI. Check for mistakes.
Comment on lines +15 to +19
// TypeScript
import { registerPhoenix } from "@arizeai/phoenix-otel";

registerPhoenix({ projectName: "my-app", autoInstrument: true });
```
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

registerPhoenix isn’t an export of @arizeai/phoenix-otel (the init API is register). This snippet will fail to compile as-is; update the import and call to the supported API.

Copilot uses AI. Check for mistakes.
Comment on lines +100 to +101
```python theme={null}
@tracer.chain
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code fence includes theme={null} after the language tag, which isn’t standard GitHub-flavored markdown and can break rendering/copy-paste. Remove the extra fence attributes so it’s just a normal python code block.

Copilot uses AI. Check for mistakes.
Comment on lines +154 to +155
```python theme={null}
def process_query(query: str):
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code fence includes theme={null} after the language tag; GitHub markdown doesn’t support that attribute. Use a plain ```python fence so the snippet renders and copies correctly.

Copilot uses AI. Check for mistakes.
Comment on lines +115 to +116
**Note:** `@opentelemetry/api` is NOT needed - it's only for manual span management.

Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This note says @opentelemetry/api is not needed, but the later “Context API Pattern” section imports it. Clarify that it’s not needed for the basic withSpan approach, but is required if readers choose the Context API approach.

Copilot uses AI. Check for mistakes.

If an `ax` command fails, troubleshoot based on the error:
- `command not found` or version error → see ax-setup.md
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This troubleshooting step says to “ask the user for their Arize API key”, which risks prompting users to paste secrets into chat/logs. Please align with ax-profiles.md: instruct users to set ARIZE_API_KEY in their own shell (or .env) and never paste the raw key into the conversation.

Copilot uses AI. Check for mistakes.

If an `ax` command fails, troubleshoot based on the error:
- `command not found` or version error → see ax-setup.md
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This troubleshooting step says to “ask the user for their Arize API key”, which risks encouraging secret sharing in chat/logs. Please instead direct users to set ARIZE_API_KEY in their own environment (per ax-profiles.md) and explicitly state not to paste the key into the conversation.

Copilot uses AI. Check for mistakes.

If an `ax` command fails, troubleshoot based on the error:
- `command not found` or version error → see ax-setup.md
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This suggests asking the user for their Arize API key, which risks prompting users to paste secrets into the conversation. Please update to instruct users to set ARIZE_API_KEY locally and explicitly say not to share the key in chat (see ax-profiles.md).

Suggested change
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via ax-profiles.md. If `.env` has no key either, instruct the user to create or locate an Arize API key in https://app.arize.com/admin and set it locally as the `ARIZE_API_KEY` environment variable (for example in their `.env` file). Explicitly tell the user not to paste or share their API key in this chat or in any logs.

Copilot uses AI. Check for mistakes.

If an `ax` command fails, troubleshoot based on the error:
- `command not found` or version error → see ax-setup.md
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This suggests asking the user for their Arize API key, which can lead to secret leakage in chat logs. Please update to instruct users to set ARIZE_API_KEY locally (env var / .env) and avoid requesting the raw key in the conversation (consistent with ax-profiles.md).

Suggested change
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via ax-profiles.md. If `.env` has no key either, instruct the user (without asking them to paste the key here) to generate an API key in https://app.arize.com/admin > API Keys and set it locally as `ARIZE_API_KEY` (env var or `.env`), then rerun the profile setup

Copilot uses AI. Check for mistakes.

If an `ax` command fails, troubleshoot based on the error:
- `command not found` or version error → see ax-setup.md
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This suggests asking the user for their Arize API key; please avoid prompting users to paste secrets into chat. Align with ax-profiles.md by instructing users to set ARIZE_API_KEY locally and explicitly stating not to share the raw key in the conversation.

Suggested change
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it locally to create/update the profile via ax-profiles.md. If `.env` has no key either, instruct the user to generate an API key in the Arize UI (https://app.arize.com/admin > API Keys) and set it as `ARIZE_API_KEY` in their local environment or `.env` file. Do **not** ask the user to paste or share the raw API key in this chat.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor

@aaronpowell aaronpowell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the skills, can you ensure that they are following the structure from the specification, so you probably want to leverage the references folder for things like ax-profiles.md and ax-setup.md.

Also, these files don't appear to be referenced from their companion SKILL.md file, so the agent using the skill isn't likely to pick them up/know when to use them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants