Skip to content

BE-456: HashQL: Entity type model enrichment and graph store query extensions#8524

Open
indietyp wants to merge 5 commits intobm/be-429-hashql-per-island-entitypath-requirements-and-fetchislandfrom
bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query
Open

BE-456: HashQL: Entity type model enrichment and graph store query extensions#8524
indietyp wants to merge 5 commits intobm/be-429-hashql-per-island-entitypath-requirements-and-fetchislandfrom
bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query

Conversation

@indietyp
Copy link
Member

@indietyp indietyp commented Mar 8, 2026

🌟 What is the purpose of this PR?

Enriches the HashQL entity type model to match the full database schema, extends the graph-store SQL query builder with new expression types needed by the postgres compiler, and adds a database migration for the continuation composite type. This is the foundation layer for the postgres translation stack (BE-457, BE-306).

🔍 What does this change?

hashql-core:

  • Adds IdSnapshotVec, a typed-ID-indexed vector with transactional snapshot/rollback support (used by the postgres compiler for scoped state)
  • Expands the entity type definition with TemporalInterval, EntityTemporalMetadata, Confidence, provenance types, EntityMetadata, and LinkData fields to match the full database schema. Previously only EntityId, EntityRecordId, and partial LinkData were modeled.
  • Adds edition_id and ontology_type_ids symbols

graph-postgres-store query builder:

  • New expression types: FieldAccess (composite field extraction), JsonAgg, Constant::Null, Constant::U128, UnaryOperator::IsNotFalse
  • New cast targets: JsonB, Continuation, Numeric, Int, BigInt
  • Unnest now accepts multiple expressions (UNNEST(a, b, c))
  • SelectStatement gains OFFSET support
  • Convenience methods: .cast(), .not() and .grouped() now take self
  • Alias type changed from &'static str to Identifier for consistency
  • Visibility: mod postgres and mod table made pub for use by the HashQL postgres compiler
  • Unit tests for all new expression types

Database migration (v010__query):

  • Creates the continuation composite type: (filter boolean, block int, locals int[], values jsonb[]). This is the row type returned by compiled filter subqueries; the interpreter reads .filter to decide row inclusion and .block/.locals/.values to resume multi-block CFGs.

hashql-eval:

  • Reworks graph/read/path.rs to handle the expanded entity field structure
  • Scaffolding in lib.rs and error.rs for the postgres module (filled in by BE-306)

Test output updates:

  • ~55 files of updated .jsonc/.stdout/.stderr fixtures across eval (graph/read), HIR (lower/checking, specialization, graph-hoisting), and MIR (reify, inline) test suites, all cascading from the entity type changes

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

The entity type model still does not cover every field in the database schema (e.g., property_metadata inner structure is opaque). Fields are added as needed by the postgres compiler; remaining gaps will be filled incrementally.

🛡 What tests cover this?

  • IdSnapshotVec has doc tests and inline unit tests covering push/set/rollback/nested snapshots
  • New graph-store query builder tests for FieldAccess, JsonAgg, Null, U128, IsNotFalse, UNNEST with multiple args, cast types, and OFFSET
  • Existing compiletest suites (eval graph/read, HIR lowering, MIR reify/inline) with blessed output updates

❓ How to test this?

cargo nextest run -p hashql-core
cargo nextest run -p hashql-eval
cargo nextest run -p hash-graph-postgres-store --lib
cargo run --package hashql-compiletest -- run

indietyp added 3 commits March 8, 2026 16:17
feat: checkpoint (II)

feat: checkpoint (III)

feat: snapshot vec

feat: add dedicated filter

feat: checkpoint

feat: filter implementation

feat: filter implementation (mostly) done

chore: environment capture note

chore: always postgres bigint

feat: target clone

feat: simplify lookup

feat: move storage up

feat: eval entity path

chore: checkpoint

chore: checkpoint

chore: find entrypoint

feat: eval context

feat: eval cleanup

chore: cleanup

feat: track index

feat: wire up filter

feat: add error reporting

chore: checkpoint

feat: add traverse, and first postgres compiler outline

feat: traverse bitmap

feat: move traversal out

feat: projections

feat: projections

fix: clippy

feat: subquery projection for lateral

feat: checkpoint

feat: test plan

feat: checkpoint

feat: checkpoint – failing tests ;-;

feat: checkpoint – failing tests ;-;

feat: checkpoint — passing tests

fix: import

fix: entity type

feat: checkpoint

feat: attribute a cost to terminator placement switches

fix: import

feat: checkpoint

feat: checkpoint

chore: lint
@indietyp indietyp requested a review from a team as a code owner March 8, 2026 15:46
@vercel
Copy link

vercel bot commented Mar 8, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview, Comment Mar 20, 2026 8:19am
petrinaut Ready Ready Preview Mar 20, 2026 8:19am
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
hashdotdesign Ignored Ignored Preview Mar 20, 2026 8:19am
hashdotdesign-tokens Ignored Ignored Preview Mar 20, 2026 8:19am

@cursor
Copy link

cursor bot commented Mar 8, 2026

PR Summary

Medium Risk
Medium risk because it changes core SQL query AST/transpilation (new expression forms, casting, UNNEST semantics, OFFSET) and introduces a new Postgres composite type migration, which can impact query correctness and runtime DB compatibility.

Overview
Adds a new DB migration to create/drop the continuation composite type and mirrors it in the postgres-store migration set.

Extends the postgres-store SQL builder with new expression/transpile capabilities (e.g., composite FieldAccess, jsonb_agg, NULL/u128 constants, IS NOT FALSE, richer cast targets, multi-arg UNNEST, and SELECT ... OFFSET), switches SELECT aliases to a quoted Identifier, and makes postgres/table modules publicly accessible for downstream compiler use.

Enriches HashQL’s graph entity type model to include metadata/temporal/provenance/confidence/encodings fields and updates graph read path resolution plus a large set of eval/HIR test fixtures to reflect the new vertex.metadata.record_id... access patterns.

Written by Cursor Bugbot for commit 3ba579f. This will update automatically on new commits. Configure here.

@indietyp indietyp changed the title HashQL: Entity type model enrichment and graph store query extensions BE-456: HashQL: Entity type model enrichment and graph store query extensions Mar 8, 2026
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

  • ✅ Fixed: Misleading enum variant name for jsonb_agg function
    • Renamed Function::JsonAgg to Function::JsonbAgg and updated all internal usages/tests to match the emitted jsonb_agg(...) SQL.
  • ✅ Fixed: Multiple new symbols defined but never referenced
    • Removed the eight unreferenced symbol definitions from sym.rs to eliminate dead entries and duplicate provenance naming.

Create PR

Or push these changes by commenting:

@cursor push 928f4a2c0b
Preview (928f4a2c0b)
diff --git a/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs b/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
--- a/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
+++ b/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
@@ -17,7 +17,7 @@
 pub enum Function {
     Min(Box<Expression>),
     Max(Box<Expression>),
-    JsonAgg(Box<Expression>),
+    JsonbAgg(Box<Expression>),
     JsonExtractText(Box<Expression>),
     JsonExtractAsText(Box<Expression>, PathToken<'static>),
     JsonExtractPath(Vec<Expression>),
@@ -55,7 +55,7 @@
                 expression.transpile(fmt)?;
                 fmt.write_char(')')
             }
-            Self::JsonAgg(expression) => {
+            Self::JsonbAgg(expression) => {
                 fmt.write_str("jsonb_agg(")?;
                 expression.transpile(fmt)?;
                 fmt.write_char(')')
@@ -821,9 +821,9 @@
     }
 
     #[test]
-    fn transpile_json_agg() {
+    fn transpile_jsonb_agg() {
         assert_eq!(
-            Expression::Function(Function::JsonAgg(Box::new(Expression::Parameter(1))))
+            Expression::Function(Function::JsonbAgg(Box::new(Expression::Parameter(1))))
                 .transpile_to_string(),
             "jsonb_agg($1)"
         );

diff --git a/libs/@local/hashql/core/src/symbol/sym.rs b/libs/@local/hashql/core/src/symbol/sym.rs
--- a/libs/@local/hashql/core/src/symbol/sym.rs
+++ b/libs/@local/hashql/core/src/symbol/sym.rs
@@ -36,7 +36,6 @@
     entity,
     entity_edition_id,
     entity_id,
-    entity_type_ids,
     entity_uuid,
     eq,
     Err,
@@ -55,8 +54,6 @@
     left_entity_confidence,
     left_entity_id,
     left_entity_provenance,
-    left_entity_uuid,
-    left_entity_web_id,
     link_data,
     List,
     lt,
@@ -76,10 +73,7 @@
     or,
     pow,
     properties,
-    property_metadata,
     provenance,
-    provenance_edition,
-    provenance_inferred,
     provided,
     r#as: "as",
     r#as_force: "as!",
@@ -101,8 +95,6 @@
     right_entity_confidence,
     right_entity_id,
     right_entity_provenance,
-    right_entity_uuid,
-    right_entity_web_id,
     Some,
     special_form,
     String,
This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.

@codecov
Copy link

codecov bot commented Mar 8, 2026

Codecov Report

❌ Patch coverage is 66.17954% with 162 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.58%. Comparing base (74ce977) to head (3ba579f).

Files with missing lines Patch % Lines
libs/@local/hashql/eval/src/graph/read/path.rs 13.54% 134 Missing ⚠️
libs/@local/hashql/core/src/id/snapshot_vec.rs 79.83% 25 Missing ⚠️
...store/src/store/postgres/query/statement/insert.rs 0.00% 2 Missing ⚠️
...src/store/postgres/query/expression/conditional.rs 98.85% 1 Missing ⚠️
Additional details and impacted files
@@                                           Coverage Diff                                           @@
##           bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland    #8524      +/-   ##
=======================================================================================================
- Coverage                                                                62.86%   61.58%   -1.28%     
=======================================================================================================
  Files                                                                     1325      915     -410     
  Lines                                                                   135282    98837   -36445     
  Branches                                                                  5523     4244    -1279     
=======================================================================================================
- Hits                                                                     85048    60873   -24175     
+ Misses                                                                   49320    37347   -11973     
+ Partials                                                                   914      617     -297     
Flag Coverage Δ
apps.hash-ai-worker-py ?
apps.hash-ai-worker-ts 1.40% <ø> (ø)
apps.hash-api 0.00% <ø> (ø)
backend-integration-tests ?
blockprotocol.type-system ?
deer ?
error-stack ?
local.claude-hooks ?
local.harpc-client ?
local.hash-backend-utils ?
local.hash-graph-sdk 9.63% <ø> (ø)
local.hash-isomorphic-utils 0.00% <ø> (ø)
local.hash-subgraph ?
rust.antsi ?
rust.deer ?
rust.error-stack ?
rust.harpc-codec ?
rust.harpc-net ?
rust.harpc-tower ?
rust.harpc-types ?
rust.harpc-wire-protocol ?
rust.hash-codec ?
rust.hash-graph-api 2.52% <ø> (ø)
rust.hash-graph-authorization ?
rust.hash-graph-postgres-store 26.81% <97.63%> (+0.08%) ⬆️
rust.hash-graph-store ?
rust.hash-graph-temporal-versioning ?
rust.hash-graph-types ?
rust.hash-graph-validation ?
rust.hashql-ast 87.23% <ø> (ø)
rust.hashql-compiletest 29.69% <ø> (ø)
rust.hashql-core 82.44% <87.30%> (+0.05%) ⬆️
rust.hashql-diagnostics ?
rust.hashql-eval 62.99% <13.54%> (-6.15%) ⬇️
rust.hashql-hir 89.06% <ø> (ø)
rust.hashql-mir 92.45% <ø> (ø)
rust.hashql-syntax-jexpr 94.05% <ø> (ø)
rust.sarif ?
sarif ?
tests.hash-backend-integration ?
unit-tests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@augmentcode
Copy link

augmentcode bot commented Mar 8, 2026

🤖 Augment PR Summary

Summary: This PR enriches the HashQL entity type model to better reflect the full graph database schema, and extends the Postgres query-builder/AST to support additional SQL constructs needed by the HashQL→Postgres compiler.

Changes:

  • Adds a new DB migration creating a continuation composite type used as the return row type for compiled filter subqueries.
  • Introduces IdSnapshotVec, a typed-ID-indexed vector with snapshot/rollback support for scoped compiler state.
  • Expands ::graph::types::knowledge::entity with temporal metadata, confidence/provenance, enriched link data, entity metadata, and encodings.
  • Extends the graph Postgres query builder with composite field access, jsonb_agg, additional constants/casts, multi-arg UNNEST, and OFFSET.
  • Updates HashQL eval graph path resolution to traverse the new entity field structure.
  • Blesses updated UI/compiletest fixtures across eval/HIR/MIR to reflect the new entity model.

Technical Notes: Query builder aliasing is standardized on an Identifier type; new expression/unit tests cover the added SQL features.

🤖 Was this summary useful? React with 👍 or 👎

Copy link

@augmentcode augmentcode bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. 3 suggestions posted.

Fix All in Augment

Comment augment review to trigger a new review at any time.

@codspeed-hq
Copy link

codspeed-hq bot commented Mar 8, 2026

Merging this PR will not alter performance

✅ 80 untouched benchmarks


Comparing bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query (3ba579f) with bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland (572197c)1

Open in CodSpeed

Footnotes

  1. No successful run was found on bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland (74ce977) during the generation of this report, so 35d2856 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from ef9b858 to a91293c Compare March 8, 2026 16:56
@indietyp indietyp force-pushed the bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland branch from 922e42b to 572197c Compare March 8, 2026 17:03
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from a91293c to 43775ba Compare March 8, 2026 17:03
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 43775ba to 3ba579f Compare March 20, 2026 08:11
@indietyp indietyp force-pushed the bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland branch from 572197c to 74ce977 Compare March 20, 2026 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

2 participants