Skip to content

feat: Segment Timout Per Datasource#19221

Open
mshahid6 wants to merge 2 commits intoapache:masterfrom
mshahid6:seg-timeout-by-datasource
Open

feat: Segment Timout Per Datasource#19221
mshahid6 wants to merge 2 commits intoapache:masterfrom
mshahid6:seg-timeout-by-datasource

Conversation

@mshahid6
Copy link
Copy Markdown
Contributor

Description

Adds the ability to configure perSegmentTimeout per datasource via BrokerDynamicConfig, so different per-segment timeout thresholds for can be set for different datasources e.g. larger datasources may need different thresholds than small ones.

Broker adds per-datasource perSegmentTimeout to query context via BrokerDynamicConfig before forwarding to historicals. Historicals enforce it using the existing perSegmentTimeout mechanism — no historical-side changes needed.

There is also a monitorOnly mode that logs which queries would be affected without enforcing, for safe rollout.

Precedence:

  1. User query context ("perSegmentTimeout": 2000)
  2. Per-datasource config (BrokerDynamicConfig.perSegmentTimeoutConfig)
  3. System default (druid.query.default.context.perSegmentTimeout)
  4. Built-in default (0 = disabled)

Example API/config

GET /druid/coordinator/v1/config/broker

POST /druid/coordinator/v1/config/broker
  {
    "perSegmentTimeoutConfig": {
      "my_large_datasource": { "perSegmentTimeoutMs": 5000, "monitorOnly": false },
      "my_new_datasource": { "perSegmentTimeoutMs": 3000, "monitorOnly": true }
    }
  }

Release note

A per-segment timeout per datasource that is set via BrokerDynamicConfig, enforced by historicals through query context and includes a monitorOnly mode.

Key changed/added classes in this PR
  • PerSegmentTimeoutConfig
  • BrokerDynamicConfig
  • QueryLifecycle.initialize()
  • QueryLifecycleFactory
  • Web console / UI for BrokerDynamicConfig

[I can add to docs when the design is reviewed]

This PR has:

  • been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@mshahid6 mshahid6 changed the title Segment Timout Per Datasource feat: Segment Timout Per Datasource Mar 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant