Skip to content

fix(ipc): retry proc acquisition when spawns fail#1669

Open
rosetta-livekit-bot[bot] wants to merge 1 commit into
mainfrom
ancients-dullard-precise
Open

fix(ipc): retry proc acquisition when spawns fail#1669
rosetta-livekit-bot[bot] wants to merge 1 commit into
mainfrom
ancients-dullard-precise

Conversation

@rosetta-livekit-bot
Copy link
Copy Markdown
Contributor

@rosetta-livekit-bot rosetta-livekit-bot Bot commented Jun 1, 2026

Summary

Fixes #5868. When every worker process fails to initialize, launch_job previously hung forever on _warmed_proc_queue.get() because nothing was ever put on the queue, and the 3-attempt retry loop was unreachable.

Split the responsibility: _acquire_proc now owns the wait-for-a-warmed-process loop. It races queue.get() against every in-flight spawn task and only retries once all in-flight spawns settle without producing a proc — so peer spawns still in flight don't burn a retry attempt. After MAX_ACQUIRE_ATTEMPTS such cycles, it raises a RuntimeError. launch_job keeps its own 3-attempt budget for post-acquire launch failures (the original retry semantics, untouched).

Alternative considered

#5871 uses None sentinels on _warmed_proc_queue to unblock waiters when a spawn fails. Two issues with that approach:

  1. It wakes too eagerly — a sentinel is pushed on each individual spawn failure, even though peer spawns may still succeed, burning a retry attempt.
  2. The retry doesn't trigger a fresh spawn — on loop-back the queue isn't empty (it has more None sentinels), so launch_job consumes another sentinel without ever requesting a new process. MAX_ATTEMPTS=3 becomes "fail 3 sentinels and give up".

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Jun 1, 2026

🦋 Changeset detected

Latest commit: 93866f2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 33 packages
Name Type
@livekit/agents Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-assemblyai Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-cerebras Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents-plugin-fishaudio Patch
@livekit/agents-plugin-google Patch
@livekit/agents-plugin-hedra Patch
@livekit/agents-plugin-hume Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-liveavatar Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-minimax Patch
@livekit/agents-plugin-mistral Patch
@livekit/agents-plugin-mistralai Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-perplexity Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-runway Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugin-tavus Patch
@livekit/agents-plugins-test Patch
@livekit/agents-plugin-trugen Patch
@livekit/agents-plugin-xai Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants