Run async UDFs on a dedicated event-loop thread#124
Open
nkitsaini wants to merge 19 commits into
Open
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
nkitsaini
force-pushed
the
users/asaini/event-loop-worker
branch
from
2f24109 to
53e6c9a
Compare
nkitsaini
force-pushed
the
users/asaini/event-loop-worker
branch
from
53e6c9a to
78a96f8
Compare
kesmit13
approved these changes
Jun 3, 2026
kkampli-singlestore
had a problem deploying
to
publish
GitHub Actions
Failure
nkitsaini
force-pushed
the
users/asaini/event-loop-worker
branch
from
4c8958b to
f27488e
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 018db7a. Configure here.
Async UDFs were running directly in uvicorn's event loop via asyncio.create_task, competing with connection handling under heavy concurrent load. This caused unresponsiveness when running from Jupyter notebooks where the event loop is shared. The fix introduces a dedicated event loop in a background thread for async UDF execution. Coroutines are submitted via run_coroutine_threadsafe() and awaited from the server loop, isolating UDF work from HTTP I/O while preserving cooperative async scheduling between UDFs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cancel the concurrent.futures.Future in the UDF loop on disconnect/timeout so the coroutine is interrupted promptly, not just at the next cancel_on_event row check. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ture asyncio.create_task() requires a coroutine but asyncio.wrap_future() returns a Future. Use asyncio.ensure_future() which accepts both. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevents UDF event loop thread leaks when run_udf_app() is called repeatedly in Jupyter notebooks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Cancel udf_future when func_task is in pending set after asyncio.wait - Cancel udf_future in finally block to ensure cleanup on any exit path - Wrap post-construction code in try/except to call app.shutdown() if validation, config, or registration fails after Application is created Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move udf_future initialization before input_handler['load']() to prevent NameError in finally block if parsing raises - Lazily create UDF event loop on first async UDF invocation instead of unconditionally in __init__, avoiding wasted resources for sync-only or metadata-only usage - Guard shutdown() against None loop/thread Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After stopping the event loop and joining the thread, set both _udf_loop and _udf_thread back to None so that _get_udf_loop() can safely recreate them if called after shutdown. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dedicated shared event loop still caused starvation under concurrent async UDF calls. Switch to the same model used by sync UDFs: each request gets its own thread with asyncio.run(), eliminating loop contention. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wraps the inner coroutine in _cancellable_run which polls cancel_event and raises CancelledError at the next await (~100ms), ensuring vector UDFs respect disconnect/timeout signals without waiting for completion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace asyncio.run() with _run_with_graceful_shutdown() that drains pending callbacks before closing the loop, preventing RuntimeError from httpx/anyio TLS cleanup in async UDFs calling OpenAI/LangChain APIs. Add 17 unit tests covering graceful shutdown, cancellation timing, exception propagation, context variable isolation, and concurrent safety. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
nkitsaini
force-pushed
the
users/asaini/event-loop-worker
branch
from
018db7a to
1669fb2
Compare
nkitsaini
force-pushed
the
users/asaini/event-loop-worker
branch
from
1669fb2 to
0a7a0a9
Compare
cursor
Bot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Medium Risk
Changes the core UDF invoke path (threading, shared event loop, and cancellation), which can affect concurrency and teardown behavior under load or client disconnects.
Overview
Async UDF execution is moved onto a process-wide dedicated event-loop thread so loop-bound clients (e.g. httpx pools) can be reused across requests and concurrent async invocations are not serialized on per-request worker threads. Sync UDFs stay on worker threads with
asyncio.runso they cannot block that shared loop.Cancellation for work off the request loop uses
_cancellable_run(pollingthreading.Event) plus settingcancel_eventwhen the function task is still pending after timeout/disconnect races, and always infinally.Managed Python UDF servers get a higher default uvicorn keep-alive (
SINGLESTOREDB_UDF_KEEPALIVE_TIMEOUT, default 120s) to reduce idle connection churn and TIME_WAIT under load.New tests cover dispatch concurrency, cancellation, routing in
Application, and keep-alive wiring where applicable.Reviewed by Cursor Bugbot for commit 05077b2. Bugbot is set up for automated code reviews on this repo. Configure here.
We have seen that UDFs can block the event loop and stop uvicorn from accepting new connections. In this PR I have moved the execution of async UDFs to a dedicated daemon thread.
I have tested this in preview by rolling out the Nova Image.