Skip to content

feat(spanner): Improve and extend support for Spanner Client#247

Open
aseering wants to merge 3 commits intoGoogleCloudPlatform:mainfrom
aseering:feat-spanner-native-support
Open

feat(spanner): Improve and extend support for Spanner Client#247
aseering wants to merge 3 commits intoGoogleCloudPlatform:mainfrom
aseering:feat-spanner-native-support

Conversation

@aseering
Copy link

Extend the Spanner Client to be more robust and more featureful.

  • Support both Spanner PG and Spanner GSQL databases
  • Support connecting to the Spanner Emulator, and for launching a managed instance of the Spanner Emulator (important for large test runs where it becomes necessary to stop/start the Emulator process regularly to deal with deliberate memory leaks)
  • Clean up some PG-isms
  • Switch to using the native Spanner Python Client. I didn't end up needing this -- could try to put it back if you prefer -- but it let me play around with using mutations for data ingest

@google-cla
Copy link

google-cla bot commented Feb 21, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@aseering aseering force-pushed the feat-spanner-native-support branch from 474ad8d to 2c8df8a Compare February 21, 2026 03:55
Copy link
Collaborator

@IsmailMehdi IsmailMehdi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be good to have a run config that showcases the emulator.

Copy link
Collaborator

@IsmailMehdi IsmailMehdi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please take a look at the style issues.

aseering added 2 commits March 9, 2026 08:23
Replaces the SQLAlchemy-based implementation with a native Google Cloud Spanner client to improve stability and support for PostgreSQL dialects.

- Implements native  and  using  queries.
- Adds robust  with multi-pass retry logic for Foreign Key cycles and proper quoting for case-sensitive Postgres tables.
- Implements high-performance  using the Mutation API with type-aware processing.
- Adds support for .
- Updates SQLGenBasePromptGenerator to detect Spanner dialect (GoogleSQL vs PostgreSQL) from db config.
- Adds specific prompt templates for Spanner GoogleSQL (using backticks) and Spanner PostgreSQL (using quotes).
@aseering aseering force-pushed the feat-spanner-native-support branch from 2c8df8a to 5302dc2 Compare March 9, 2026 08:24
@aseering
Copy link
Author

aseering commented Mar 9, 2026

Resolved the style issues. What do you think? Ready to submit?

If it's ok with you, I'd like to do a follow-up with Emulator run-config examples. I've been testing and playing with it locally some more, and am thinking to get this core-Spanner work in now and Emulator clean-ups as a follow-on CL. (One key challenge w.r.t. the Emulator is that Spanner's Emulator is purely in-memory -- no disk persistence -- and it never frees dropped or deleted data. So you effectively have to run it as "create a database, load the schema, load the data, run some test queries; then close down the whole Emulator process and start a whole new one for the next database", lest you run out of RAM. This requires wiring up the logic for database setup / tear-down, which we have for DDL, for all databases -- but right now it's only in place (as far as I can tell?) for datasets that contain DDL statements, so some of my dataset updates are also required to make the Emulator work.

Copy link
Collaborator

@IsmailMehdi IsmailMehdi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove the added import evalbench.xxx

import asyncio
from aiologger import Logger
import grpc
from evalproto import eval_request_pb2, eval_connect_pb2, eval_config_pb2
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not needed

@aseering
Copy link
Author

aseering commented Mar 9, 2026

The evalbench.* changes were requested by the linter. Should I look into a linter config change, if that's not as desired?

@IsmailMehdi
Copy link
Collaborator

IsmailMehdi commented Mar 9, 2026 via email

@aseering
Copy link
Author

aseering commented Mar 9, 2026

ah, I see: It's not which linter, it's running the linter without setting PYTHONPATH the same way that many of the scripts in this repo do. (I guess(?) the default assumption is that things in a Git repo should be relative to the root of that repo.)

- Update .pycodestyle to ignore W504
- Format files with autopep8 to resolve spacing, trailing whitespace, and E704 errors
- Update `evalbench/test/mongodb_test.py` to match expected data format
- Update `evalbench/test/spanner_test.py` to use `batch_execute` for DDL statements
@aseering aseering force-pushed the feat-spanner-native-support branch from 907b0b6 to cc119f3 Compare March 9, 2026 18:10
@IsmailMehdi
Copy link
Collaborator

/gcbrun

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants