test: add category selection flags

[rosetta-livekit-bot]

TL;DR: Add explicit source of truth for tests categorization and unified selection

Select and run tests as

uv run pytest --unit 
uv run pytest --realtime
uv run pytest --plugin openai

Categorize new test modules via

import pytest
...
pytestmark = pytest.mark.unit  # or pytest.mark.plugin("provider-name")

Motivation

Currently there is no clear indication of which tests to run where and how. ./makefile has a manually maintained list of unit-tests modules to run: all new modules must be added in order to be tested in CI, but nothing keeps this in check. This leads to stale/ignored tests

Plain pytest ./tests doesn't work on most environment, even with -k selection, as the collection will fail due to missing dependencies.

This PR fixes both problems by defining single source of truth and mechanism for tests isolation.

Changes

• Add pytestmark = pytest.mark.<category> to existing modules
• Add pytest --<category> flags that runs tests of that category (with dependencies isolation)
• Add pytest --list-categories arg that lists all modules and their categories
• Require categorizing future tests to prevent orphaned tests (can be temporarily disabled via --allow-uncategorized)
• Fix broken tests in test_audio_recognition_aclose.py and test_speaker_id_grouping.py
• Fix deadlock caused by test_cli_log_level.py

Categories and dependencies:

Category	Needs
--unit	—
--plugin [name]	that provider's dependencies/keys
--stt / --tts	cross-provider STT/TTS suite + keys
--realtime	live realtime model + keys
--evals	LiveKit inference gateway
--docs	the docs dependency group

Notes

Stale tests ignored in CI before this PR

#	Module	Tests	Note
1	test_audio_emitter.py	10
2	test_audio_recognition_aclose.py	2	fixed 1 broken test
3	test_cli_log_level.py	25	fixed deadlock when it's enabled
4	test_drain_timeout.py	6
5	test_interruption/test_interruption_failover.py	6
6	test_nested_agent_task.py	1
7	test_speaker_id_grouping.py	7	fixed 2 broken tests
8	test_speech_start_time_persistence.py	3
9	test_stt_base.py	4
10	test_user_turn_exceeded.py	9

pytest --list-categories output

Test categories (select with --<category>):

  unit     45 modules
             - tests/test_agent_session.py
             - tests/test_aio.py
             - tests/test_aio_itertools.py
             - tests/test_amd_classifier.py
             - tests/test_audio_decoder.py
             - tests/test_audio_emitter.py
             - tests/test_audio_recognition_aclose.py
             - tests/test_audio_recognition_handoff.py
             - tests/test_audio_recognition_push_audio.py
             - tests/test_chat_ctx.py
             - tests/test_cli_log_level.py
             - tests/test_config.py
             - tests/test_connection_pool.py
             - tests/test_debounce.py
             - tests/test_drain_timeout.py
             - tests/test_endpointing.py
             - tests/test_google_thought_signatures.py
             - tests/test_http_context_helper.py
             - tests/test_inference_stt_fallback.py
             - tests/test_inference_tts_fallback.py
             - tests/test_interruption/test_interruption_failover.py
             - tests/test_interruption/test_overlapping_speech_event.py
             - tests/test_ipc.py
             - tests/test_ivr_activity.py
             - tests/test_langgraph.py
             - tests/test_language.py
             - tests/test_nested_agent_task.py
             - tests/test_recording.py
             - tests/test_room.py
             - tests/test_room_io.py
             - tests/test_schema_gemini.py
             - tests/test_session_host.py
             - tests/test_speaker_id_grouping.py
             - tests/test_speech_start_time_persistence.py
             - tests/test_stt_base.py
             - tests/test_stt_fallback.py
             - tests/test_tokenizer.py
             - tests/test_tool_proxy.py
             - tests/test_tool_search.py
             - tests/test_tools.py
             - tests/test_transcription_filter.py
             - tests/test_tts_fallback.py
             - tests/test_user_turn_exceeded.py
             - tests/test_utils/test_audio_array_buffer.py
             - tests/test_utils/test_bounded_dict.py
  plugin   15 modules
             - tests/test_personaplex_realtime_model.py
             - tests/test_plugin_anthropic.py
             - tests/test_plugin_assemblyai_stt.py
             - tests/test_plugin_cerebras.py
             - tests/test_plugin_elevenlabs_tts.py
             - tests/test_plugin_gnani_stt.py
             - tests/test_plugin_gnani_tts.py
             - tests/test_plugin_google_llm.py
             - tests/test_plugin_google_stt.py
             - tests/test_plugin_gradium_stt.py
             - tests/test_plugin_inworld_tts.py
             - tests/test_plugin_perplexity.py
             - tests/test_plugin_perplexity_responses.py
             - tests/test_plugin_soniox_stt.py
             - tests/test_vad.py
  realtime 1 module
             - tests/test_realtime/test_realtime.py
  stt      1 module
             - tests/test_stt.py
  tts      1 module
             - tests/test_tts.py
  evals    2 modules
             - tests/test_evals.py
             - tests/test_workflows.py
  docs     1 module
             - tests/test_convert_html_docs.py

New ### Testing section in AGENTS.md (authored by Claude)

Testing

uv run pytest tests/test_tools.py                  # Run a single test file
uv run pytest tests/test_tools.py -k "test_name"   # Run specific test
uv run pytest --unit                               # Run unit tests that don't require cloud accounts

Test categories

Every test module declares exactly one category via a module-level marker, and
each category has a matching --<category> selection flag. Selection happens
before import, so a category run never imports (or fails on) modules outside
it.

Marker	Flag	Meaning
pytest.mark.unit	--unit	fast, hermetic, no external providers/credentials/network
pytest.mark.plugin("name")	--plugin [name]	provider integration test (needs that provider's deps/keys)
pytest.mark.stt	--stt	cross-provider speech-to-text suite (tests/test_stt.py)
pytest.mark.tts	--tts	cross-provider text-to-speech suite (tests/test_tts.py)
pytest.mark.realtime("name")	--realtime [name]	realtime-model test
pytest.mark.evals	--evals	behavioral evals against the LiveKit inference gateway
pytest.mark.docs	--docs	tests for the docs-build tooling under .github/

uv run pytest --unit                    # the CI unit gate (no cloud accounts)
uv run pytest --plugin openai           # only the openai provider tests
uv run pytest --list-categories         # list every module grouped by category, then exit

Adding a test: give the new module a category marker (pytestmark = pytest.mark.unit, etc.) — collection fails with a hint if it lacks one. Run
pytest with the --allow-uncategorized option to temporarily disable this rule
(CI keeps it on by default).

2 STT failures seem to be unrelated to the changes.

[changeset-bot]

⚠️ No Changeset found

Latest commit: c29c402

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[devin-ai-integration]

Devin Review found 1 potential issue.

View 4 additional findings in Devin Review.

[devin-ai-integration]

🟡 TTS glob pattern tts*.test.ts misses gemini_tts.test.ts

The glob pattern plugins/${provider}/src/**/tts*.test.ts at vitest.config.ts:32 requires filenames to start with tts, but plugins/google/src/beta/gemini_tts.test.ts starts with gemini_. Running pnpm test -- --tts or pnpm test -- --tts google silently skips Google's Gemini TTS test. The same issue exists in scripts/vitest.js:73 where categoriesFor checks normalized.includes('/tts') — this matches paths containing /tts as a path segment (like /tts.test.ts) but misses _tts in filenames like gemini_tts.test.ts, so --list-categories also won't show this file under the tts category.

Prompt for agents

The glob pattern `tts*.test.ts` in vitest.config.ts:32 (and similarly `stt*.test.ts` in line 27 which could face the same issue in the future) uses a prefix match, but some TTS test files like `plugins/google/src/beta/gemini_tts.test.ts` use a different naming convention with a prefix before `tts`. Two places need fixing:

1. In vitest.config.ts, the `tts` case (line 32): change `tts*.test.ts` to `*tts*.test.ts` (or `*tts{,_*}.test.ts` to be more targeted). Similarly consider changing `stt*.test.ts` to `*stt*.test.ts` in line 27 for forward compatibility.

2. In scripts/vitest.js, the `categoriesFor` function (line 73): the check `normalized.includes('/tts')` misses files like `gemini_tts.test.ts` because the `/tts` substring isn't in the path (it appears as `_tts`). Consider changing to check for `tts` in the filename itself, e.g. using a regex that matches the basename containing `tts`.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[toubatbrian]

This seems like python-specific one