dstore is a simple abstraction on top of Local storage and Cloud
storage. It handles commonly used functions to store things (locally,
or on cloud storage providers), list files, delete, etc..
It is used by StreamingFast.
It currently supports:
- AWS S3 (
s3://[bucket]/path?region=us-east-1, with AWS-specific env vars)- Minio (through the S3 interface)
- Google Storage (
gs://[bucket]/path, withGOOGLE_APPLICATION_CREDENTIALSenv var set) - Azure Blob Storage (
az://[account].[container]/path, withAZURE_STORAGE_KEYenv var set) - Local file systems (including virtual of fused-based) (
file:///prefix)
The storetests package contains all our integration tests we perform on our store implementation.
Some of the store implementations can be tested directly while few others, from Cloud Providers
essentially, requires some extra environment variables to run. They are skipped if the correct
environment variables for the provider are not set.
A docker-compose.yml is provided at the root of the repository. It starts:
- MinIO on port
9000— S3-compatible object storage - Ceph RGW on port
8080— built fromdocker/ceph-localusingquay.io/ceph/ceph:v19(native arm64 + amd64); bootstraps a single-node cluster on first start (~30–60 s) - fake-gcs-server on port
4443— GCS-compatible object storage (native arm64 + amd64); data is in-memory (lost on restart)
docker compose up -dOnce the containers are healthy, run the local tests:
STORETESTS_S3_MINIO_STORE_URL="s3://localhost:9000/store-tests?region=none&insecure=true&access_key_id=minioadmin&secret_access_key=minioadmin" \
STORETESTS_S3_CEPH_STORE_URL="s3://localhost:8080/store-tests?region=none&insecure=true&access_key_id=cephaccesskey&secret_access_key=cephsecretkey" \
STORETESTS_GS_EMULATOR_STORE_URL="gs://store-tests" \
STORAGE_EMULATOR_HOST="localhost:4443" \
go test ./...To also run against real cloud providers, supply the relevant environment variables:
STORETESTS_GS_STORE_URL="gs://streamingfast-developement-random/store-tests" \
STORETESTS_S3_STORE_URL="s3://streamingfast-customer-outbox/store-tests?region=us-east-2" \
go test ./...Note
The bucket names above are placeholders — replace them with real buckets you have access to.
Any variable that is not set will cause the corresponding tests to be skipped automatically.
Issues and PR in this repo related strictly to the dstore library.
Report any protocol-specific issues in their respective repositories
Please first refer to the general StreamingFast contribution guide, if you wish to contribute to this code base.