feat(admin-api): support inter-table dependencies in derived dataset validation

[mitchhs12]

Summary

Adds full inter-table dependency support for derived datasets — both validation (admin API) and runtime (dump engine). Tables within a derived dataset can now reference sibling tables using self.<table_name> syntax (e.g., SELECT * FROM self.blocks_base), consistent with the existing self. UDF convention.

Part 1: Validation (admin API)

• Add SelfSchemaProvider.add_table() for progressive schema registration during topological processing
• Add cycle detection via topological_sort(), returning CYCLIC_DEPENDENCY 400 error
• Add explicit SELF_REF_TABLE_NOT_FOUND error when self. references target non-existent sibling tables
• Process tables in dependency order in /schema and /manifests endpoints
• 5 integration tests: basic self-ref, 3-table chain, cycle rejection, self-referencing table, mixed deps

Part 2: Runtime (dump engine)

• Split physical_for_dump::create() into resolve_external_deps() + build_catalog() so callers can inject self-ref entries alongside external deps
• Extract partition_table_refs() to separate self. refs from external deps
• Register sibling tables in both planning (SelfSchemaProvider for column types) and execution (ResolvedTableEntry for physical data) phases
• Notification-driven polling loop — self-ref tables wait for sibling data via notification pipeline
• Remove unreachable dataset_start_block fallback (validation now guarantees tables have at least one reference via NO_TABLE_REFERENCES check from fix(admin-api): reject derived tables with no source table references #1944)
• Pass sibling PhysicalTable map from orchestrator to each table task
• Tables dump in parallel (no topological ordering at runtime) — the existing streaming/notification system handles dependency ordering naturally
• Un-ignore intra_deps_test E2E test

Error codes

New error codes added by this PR:

Code	Status	Description
SELF_REFERENCING_TABLE	400	A table references itself via self.<own_name>
SELF_REF_TABLE_NOT_FOUND	400	A self.-qualified reference targets a non-existent sibling table
CYCLIC_DEPENDENCY	400	Inter-table references form a cycle
CATALOG_QUALIFIED_TABLE	400	3-part catalog-qualified table reference (renamed from TABLE_REFERENCE_RESOLUTION)
INVALID_TABLE_NAME	400	Table name does not conform to identifier rules

Breaking change: TABLE_REFERENCE_RESOLUTION error code renamed to CATALOG_QUALIFIED_TABLE for catalog-qualified table references. Acceptable given status: unstable.

Key design decisions

• self. convention: Aligns with UDF convention (self.functionName()). Parsed by DataFusion as TableReference::Partial { schema: "self", table: "..." }
• Resolve + build split: for_dump.rs has zero self-ref knowledge — it resolves external deps and builds catalogs from generic entries. Self-ref resolution lives in table.rs where it belongs
• Parallel, not sequential: Per Leo's feedback, tables dump in parallel. The streaming query notification pipeline handles ordering — same mechanism as external deps
• Notification-driven start block: Self-ref tables subscribe to sibling notifications and wait until data appears, protected by FailFastJoinSet cancellation if a sibling fails

Files changed

File	Changes
common/src/self_schema_provider.rs	add_table() for progressive schema registration
common/src/catalog/physical/for_dump.rs	Split create() → resolve_external_deps() + build_catalog(), add ResolvedTableEntry
worker-datasets-derived/src/job_impl.rs	Build siblings map, pass to each materialize_table() call; 1 unit test
worker-datasets-derived/src/job_impl/table.rs	partition_table_refs(), self-ref resolution in both phases, notification-driven polling; 5 unit tests
datasets-derived/src/sorting.rs	topological_sort() and CyclicDepError
admin-api/src/handlers/schema.rs	Topological ordering, cycle detection, self-ref validation in /schema
admin-api/src/handlers/common.rs	Topological ordering, cycle detection, self-ref validation in /manifests
tests/src/tests/it_dependencies.rs	Remove #[ignore] from intra_deps_test
docs/feat/data-inter-table-dependencies.md	Feature documentation with all error codes

Related

• Prior runtime implementation: Table self reference #1524 (closed — codebase has since been refactored)
• Feature doc: docs/feat/data-inter-table-dependencies.md

Test plan

• 5 integration tests for validation (self-ref, chain, cycle, self-cycle, mixed deps)
• 6 unit tests for runtime (partition logic, error fatality)
• E2E intra_deps_test passes (dump + query with inter-table deps)
• Manual testing: verified SELF_REFERENCING_TABLE, CYCLIC_DEPENDENCY, NO_TABLE_REFERENCES errors via curl
• Format, check, clippy — zero warnings