Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
28c47c4
feat: revised MVP scope
juanmardefago Mar 16, 2026
bbd855c
docs: align agent workflow with MVP planning
juanmardefago Mar 17, 2026
550432d
docs: record MVP-033 network contract
juanmardefago Mar 18, 2026
5d18b9f
devenv: make demo-ready state the default
juanmardefago Mar 23, 2026
799f001
docs: realign MVP planning docs
juanmardefago Mar 25, 2026
73fab75
Align runtime payment contract and fix validation stability
juanmardefago Mar 27, 2026
2e73c3e
docs: define provider persistence boundary for MVP-003
juanmardefago Mar 27, 2026
bd37adf
feat: updated implementation sequencing doc
juanmardefago Mar 27, 2026
5ce40d9
provider: add MVP-010 low-funds session stop handling
juanmardefago Mar 27, 2026
017a445
provider: add deterministic RAV request thresholds
juanmardefago Mar 27, 2026
9dbb322
Define MVP provider operator auth contract
juanmardefago Mar 27, 2026
f9bcdbf
Tighten firecore integration and isolate SDS runtime drift
juanmardefago Mar 28, 2026
8ff7feb
Validate MVP-014 with local firecore runtime workflow
juanmardefago Mar 28, 2026
ad3420a
Wire plugin metering into gateway payment state
juanmardefago Mar 28, 2026
1171ed0
Harden firecore payment-state assertion
juanmardefago Mar 28, 2026
6a79b07
Mark MVP-015 complete in planning docs
juanmardefago Mar 28, 2026
964554b
Enforce low-funds stops in live firecore streams
juanmardefago Mar 28, 2026
f9bf963
Add standalone oracle discovery service
juanmardefago Mar 30, 2026
30c581c
Integrate oracle discovery into the consumer sidecar
juanmardefago Mar 30, 2026
46c54c5
feat: update docs
juanmardefago Mar 30, 2026
3aa93c0
Align MVP docs with provider-originated runtime control
juanmardefago Mar 31, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .reflex
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
DLOG=".*=debug" ./devel/sds consumer sidecar \
--grpc-listen-addr=:9002 \
--plaintext \
--signer-private-key=0xe4c2694501255921b6588519cfd36d4e86ddc4ce19ab1bc91d9c58057c040304 \
--signer-private-key=0x0bba7d355d1750fce9756af7887e826e8071a56d9d8e327f546b1f34c78f9281 \
--collector-address=0x1d01649b4f94722b55b5c3b3e10fe26cd90c1ba9'

-r "\.(go|sql)$" -s -R "devel/.*" -- sh -c \
Expand Down
39 changes: 0 additions & 39 deletions .reflex.stack

This file was deleted.

20 changes: 20 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,26 @@ grt, err := sds.NewGRT(<accepts all types>)
sds.MustNewGRT(<accepts all types>)
```

## MVP Planning References

For MVP-scoped work:

- Use `docs/mvp-scope.md` as the target-state definition.
- Use `plans/mvp-gap-analysis.md` for current-state assessment.
- Use `plans/mvp-implementation-backlog.md` as the active execution backlog.
- Treat `plans/implementation-backlog.md` as historical context unless explicitly requested.

## Commit Messages

- When asked to create a commit, first inspect recent commits with `git log --format='%s%n%b' -n <N>` and follow the prevailing repo style instead of inventing a new format.
- In this repo, the expected format is:
- one short imperative subject line
- a blank line
- a flat bullet list in the commit body, with each bullet starting with `- `
- The commit body must contain real newlines. Never pass a single shell-escaped string containing literal `\n` sequences as the body.
- Prefer either multiple `-m` flags or a temporary commit message file so Git receives the intended paragraph and bullet formatting verbatim.
- Do not create a commit until `go vet ./...` and `go test ./...` pass unless the user explicitly asks otherwise.

## Notes

- All builds must pass before committing
Expand Down
53 changes: 32 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,17 +33,7 @@ Now `sds` invokes `devel/sds` directly. Use [reflex](https://github.com/cespare/
reflex -c .reflex
```

Both reflex configs pass `--plaintext` explicitly for the local/demo sidecar↔gateway path. Outside local/demo usage, configure TLS certificate/key files instead of relying on plaintext defaults.

To keep on-chain state stable while restarting the rest of the stack, run `devenv` separately and use the stack-only reflex config:

```bash
./devel/sds devenv
./devel/sds demo setup # writes devel/.demo.env required by `.reflex.stack`
reflex -c .reflex.stack
```

`.reflex.stack` now fails fast if `devel/.demo.env` is missing or does not contain the required demo variables.
The default `.reflex` flow uses the deterministic demo signer that `sds devenv` authorizes automatically. It also passes `--plaintext` explicitly for the local/demo sidecar↔gateway path. Outside local/demo usage, configure TLS certificate/key files instead of relying on plaintext defaults.

We have `devel/sds_sink` helper that can be used to sink in data service mode (invokes `sds sink ...` configured for development environment):

Expand All @@ -58,6 +48,13 @@ sds_sink run common@v0.1.0 map_clocks -s -1

The `sds devenv` command starts an Anvil node and deploys Graph Protocol contracts (requires Docker). It deploys the original `PaymentsEscrow`, `GraphPayments`, and `GraphTallyCollector` contracts, plus `SubstreamsDataService` and various mock contracts (GRTToken, Controller, Staking, etc.) for testing. Integration tests use the same devenv via testcontainers.

After deployment, `devenv` also prepares the default local demo state:

- payer escrow funded for the default service provider
- data service provision minimum set to `0`
- default service provider provisioned and registered
- deterministic demo signer authorized for the default payer

```bash
sds devenv # Prints contract addresses and test accounts
```
Expand Down Expand Up @@ -115,6 +112,9 @@ Test accounts (10 ETH + 10,000 GRT each):
| User1 | `0x90353af8461a969e755ef1e1dbadb9415ae5cb6e` | `0xdd02564c0e9836fb570322be23f8355761d4d04ebccdc53f4f53325227680a9f` |
| User2 | `0x9585430b90248cd82cb71d5098ac3f747f89793b` | `0xbc3def46fab7929038dfb0df7e0168cba60d3384aceabf85e23e5e0ff90c8fe3` |
| User3 | `0x37305c711d52007a2bcfb33b37015f1d0e9ab339` | `0x7acd0f26d5be968f73ca8f2198fa52cc595650f8d5819ee9122fe90329847c48` |
| Demo Signer | `0x82b6f0bbbab50f0ddc249e5ff60c6dc64d55340e` | `0x0bba7d355d1750fce9756af7887e826e8071a56d9d8e327f546b1f34c78f9281` |

`sds demo setup` no longer mutates the chain. It verifies the default demo-ready state from `sds devenv` and writes `devel/.demo.env` for any manual env-driven workflows, but it is no longer required for the default `.reflex` flow.

### Running Tests

Expand All @@ -125,20 +125,30 @@ go test ./test/integration/... -v # Integration tests (requires Docker)

### Running Full System with Firecore

To run the full Substreams Data Service stack with a Firehose provider, you need `firecore` and `dummy-blockchain` binaries (see [Prerequisites](#prerequisites)). Clone the repositories and build from source:
`MVP-014` is currently validated through a local-first runtime workflow. The published `ghcr.io/streamingfast/dummy-blockchain:v1.7.7` image is still stale for the current SDS provider/plugin contract, so the supported path is to rebuild `firehose-core` and `dummy-blockchain` locally and point `TestFirecore` at that local image.

Build the local runtime images from sibling checkouts:

```bash
# Build firecore
#
# IMPORTANT: `firecore` must include SDS plugin registration support, otherwise the `sds://...` plugins
# configured in `devel/firecore.config.yaml` won't load. Use at least this commit:
# 536bcd99495f42a27b67b340ccf8416f0fc967bf
go install github.com/streamingfast/firehose-core/cmd/firecore@536bcd99495f42a27b67b340ccf8416f0fc967bf

# Build dummy-blockchain
go install github.com/streamingfast/dummy-blockchain@latest
# Build a local firecore image. If SDS plugin/runtime contracts changed after the
# currently pinned SDS dependency in firehose-core, update that dependency first.
cd ../firehose-core
docker build -t ghcr.io/streamingfast/firehose-core:sds-local .

# Build a local dummy-blockchain image on top of the local firecore tag.
cd ../dummy-blockchain
docker build \
--build-arg FIRECORE_VERSION=sds-local \
-t ghcr.io/streamingfast/dummy-blockchain:sds-local .

# Run the SDS firecore integration test against the local runtime image.
cd ../data-service
SDS_TEST_DUMMY_BLOCKCHAIN_IMAGE=ghcr.io/streamingfast/dummy-blockchain:sds-local \
go test ./test/integration -run TestFirecore -v
```

`TestFirecore` defaults to `ghcr.io/streamingfast/dummy-blockchain:v1.7.7`. Override it with `SDS_TEST_DUMMY_BLOCKCHAIN_IMAGE` when validating locally rebuilt runtimes. Refreshing the upstream published images so the default path works again is tracked in `MVP-036`.

A sample firecore configuration is provided in `devel/firecore.config.yaml` that uses dummy-blockchain as the reader node and configures the SDS plugins (auth, session, metering) to connect to the provider gateway on `:9001`.

Sanity check (what to look for in logs):
Expand All @@ -149,6 +159,7 @@ Sanity check (what to look for in logs):
- Bad:
- `executable file not found in $PATH` for `dummy-blockchain` → ensure `$(go env GOPATH)/bin` is on `PATH`
- errors about unknown `sds` plugin kind/scheme → your `firecore` binary is too old
- auth/session/usage contract mismatch against the current SDS provider/plugin gateway → rebuild the local `firehose-core` and `dummy-blockchain` images and rerun `TestFirecore`

## Architecture

Expand Down
9 changes: 9 additions & 0 deletions buf.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Generated by buf. DO NOT EDIT.
version: v2
deps:
- name: buf.build/streamingfast/firehose
commit: 9d6efa32daca4ea69d36b6a0bd1f2a72
digest: b5:3086eb5291e95e37c48e90385521218d172225c3b73794145d0e1bc48a6851177c1323c98a519991e52107ab8b752ee51401cd76ccc7305a74edd915497618c6
- name: buf.build/streamingfast/substreams
commit: 6e9da29004384d689725be6625a08df7
digest: b5:0acef6f414da08f2bff3db8ac8997928a2e1fcf26d2292e6e974b749e833bf57652bf1748a5d79ee6a577afbf3fa542a9218affc2233e74611abe90fea091149
2 changes: 2 additions & 0 deletions buf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ version: v2
modules:
- path: proto
name: buf.build/graphprotocol/substreams-data-service
deps:
- buf.build/streamingfast/substreams
lint:
use:
- STANDARD
Expand Down
3 changes: 3 additions & 0 deletions cmd/sds/consumer_sidecar.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ var consumerSidecarCmd = Command(
flags.String("signer-private-key", "", "Private key for signing RAVs (hex, required)")
flags.Uint64("chain-id", 1337, "Chain ID for EIP-712 domain")
flags.String("collector-address", "", "Collector contract address for EIP-712 domain (required)")
flags.String("oracle-endpoint", "", "Oracle endpoint used for provider discovery when no direct provider override is supplied")
flags.Bool("plaintext", false, "Serve plaintext h2c instead of TLS (local/demo only)")
flags.String("tls-cert-file", "", "Path to the TLS certificate PEM file")
flags.String("tls-key-file", "", "Path to the TLS private key PEM file")
Expand All @@ -45,6 +46,7 @@ func runConsumerSidecar(cmd *cobra.Command, args []string) error {
signerKeyHex := sflags.MustGetString(cmd, "signer-private-key")
chainID := sflags.MustGetUint64(cmd, "chain-id")
collectorHex := sflags.MustGetString(cmd, "collector-address")
oracleEndpoint := sflags.MustGetString(cmd, "oracle-endpoint")
plaintext := sflags.MustGetBool(cmd, "plaintext")
tlsCertFile := sflags.MustGetString(cmd, "tls-cert-file")
tlsKeyFile := sflags.MustGetString(cmd, "tls-key-file")
Expand All @@ -69,6 +71,7 @@ func runConsumerSidecar(cmd *cobra.Command, args []string) error {
ListenAddr: listenAddr,
SignerKey: signerKey,
Domain: horizon.NewDomain(chainID, collectorAddr),
OracleEndpoint: oracleEndpoint,
PaymentSessionRoundtripTimeout: paymentSessionRoundtripTimeout,
TransportConfig: transportConfig,
}
Expand Down
9 changes: 5 additions & 4 deletions cmd/sds/demo_flow.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ var demoFlowCmd = Command(
Flags(func(flags *pflag.FlagSet) {
flags.String("consumer-sidecar-addr", "http://localhost:9002", "Consumer sidecar address")
flags.String("provider-sidecar-addr", "http://localhost:9001", "Provider gateway address (used for status checks)")
flags.String("provider-endpoint", "http://localhost:9001", "Provider gateway endpoint to pass to consumer Init (PaymentGatewayService)")
flags.String("provider-control-plane-endpoint", "http://localhost:9001", "Provider control-plane endpoint to pass to consumer Init (PaymentGatewayService)")

flags.String("payer-address", "", "Payer address (required)")
flags.String("receiver-address", "", "Receiver/service provider address (required)")
Expand All @@ -55,7 +55,7 @@ func runDemoFlow(cmd *cobra.Command, args []string) error {

consumerSidecarAddr := strings.TrimSpace(sflags.MustGetString(cmd, "consumer-sidecar-addr"))
providerSidecarAddr := strings.TrimSpace(sflags.MustGetString(cmd, "provider-sidecar-addr"))
providerEndpoint := strings.TrimSpace(sflags.MustGetString(cmd, "provider-endpoint"))
providerControlPlaneEndpoint := strings.TrimSpace(sflags.MustGetString(cmd, "provider-control-plane-endpoint"))

payerHex := sflags.MustGetString(cmd, "payer-address")
receiverHex := sflags.MustGetString(cmd, "receiver-address")
Expand All @@ -68,7 +68,7 @@ func runDemoFlow(cmd *cobra.Command, args []string) error {

cli.Ensure(consumerSidecarAddr != "", "<consumer-sidecar-addr> is required")
cli.Ensure(providerSidecarAddr != "", "<provider-sidecar-addr> is required")
cli.Ensure(providerEndpoint != "", "<provider-endpoint> is required")
cli.Ensure(providerControlPlaneEndpoint != "", "<provider-control-plane-endpoint> is required")

cli.Ensure(payerHex != "", "<payer-address> is required")
payer, err := eth.NewAddress(payerHex)
Expand All @@ -94,14 +94,15 @@ func runDemoFlow(cmd *cobra.Command, args []string) error {
Receiver: commonv1.AddressFromEth(receiver),
DataService: commonv1.AddressFromEth(dataService),
},
GatewayEndpoint: providerEndpoint,
ProviderControlPlaneEndpoint: providerControlPlaneEndpoint,
}))
cli.NoError(err, "consumer Init failed")

sessionID := strings.TrimSpace(initResp.Msg.GetSession().GetSessionId())
cli.Ensure(sessionID != "", "consumer Init returned an empty session_id")

fmt.Printf(" session_id: %s\n", sessionID)
fmt.Printf(" data_plane_endpoint: %s\n", initResp.Msg.GetDataPlaneEndpoint())

fmt.Printf("\nStep 2: ReportUsage loop\n")
var totalBlocksSent uint64
Expand Down
Loading