V1.0.0.1 release#663
Conversation
Bumps [pillow](https://github.com/python-pillow/Pillow) from 12.1.1 to 12.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@12.1.1...12.2.0) --- updated-dependencies: - dependency-name: pillow dependency-version: 12.2.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…, and field event participants
…ategy for field event participants
feat: enhance water well details with site name, historic depth notes, and field event participants
… used it in well_inventory
Bumps [python-multipart](https://github.com/Kludex/python-multipart) from 0.0.22 to 0.0.26. - [Release notes](https://github.com/Kludex/python-multipart/releases) - [Changelog](https://github.com/Kludex/python-multipart/blob/master/CHANGELOG.md) - [Commits](Kludex/python-multipart@0.0.22...0.0.26) --- updated-dependencies: - dependency-name: python-multipart dependency-version: 0.0.26 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…e handlers to sync and enhancing error handling
feat: implement API concurrency fix strategy by converting async rout…
…and sampler validation - create or reuse field event participants for imported staff columns - link imported samples to the measuring participant - fail a row when measuring_person does not resolve to exactly one participant
- Keep `water_level_notes` as freeform notes - Stop duplicating field staff and sampler into field event/activity notes - Treat structured participants as the authoritative staff source
- Verify the well details payload exposes the measuring person on latest_field_event_sample.contact - Verify the latest field event participants list includes imported staff
…tion `Contact` rows are unique on `name + organization`, so the water-level importer now uses that same key when resolving field staff contacts. This avoids missing an existing contact with a different contact_type and attempting a duplicate insert. Also adds regression coverage for reusing an existing same-name/same-organization contact during import.
… tests Water-level importer tests were leaving participant contacts behind in the shared test database, which caused test_get_contacts to fail during full-suite runs. Update fixture and CLI test cleanup so importer-created staff contacts are removed after tests finish, while keeping the contact API assertions strict.
…ipant creation Normalize the fixed field_staff CSV columns into a single iterable shape after validation, so participant creation no longer hardcodes each slot in the importer. This keeps the input CSV/schema unchanged while making the participant logic easier to read and easier to extend later if the staff-field shape changes.
Bumps [mako](https://github.com/sqlalchemy/mako) from 1.3.10 to 1.3.11. - [Release notes](https://github.com/sqlalchemy/mako/releases) - [Changelog](https://github.com/sqlalchemy/mako/blob/main/CHANGES) - [Commits](https://github.com/sqlalchemy/mako/commits) --- updated-dependencies: - dependency-name: mako dependency-version: 1.3.11 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [authlib](https://github.com/authlib/authlib) from 1.6.9 to 1.6.11. - [Release notes](https://github.com/authlib/authlib/releases) - [Changelog](https://github.com/authlib/authlib/blob/v1.6.11/docs/changelog.rst) - [Commits](authlib/authlib@v1.6.9...v1.6.11) --- updated-dependencies: - dependency-name: authlib dependency-version: 1.6.11 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…53-refactor-water-level-importer kas-bdms-749-thru-753-refactor-water-level-importer
[BDMS-123]: Add UTC normalization for date_time and measurement_date_time in well inventory importer
… views and documentation
…#661) Bumps the gha-minor-and-patch group with 1 update: [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv). Updates `astral-sh/setup-uv` from 8.0.0 to 8.1.0 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v8.0.0...v8.1.0) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: 8.1.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: gha-minor-and-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…i-fix [codex] Reshape well details response around field events
The field_events query returns the 25 most recent events sorted newest-first. For wells with many visits the oldest event falls outside that window, making it impossible to retrieve first-visit participants from the paged list. Adds a separate targeted query that always fetches the single oldest field event (with participants) and returns it as first_field_event alongside the existing paged field_events list.
Add first_field_event to well details response
There was a problem hiding this comment.
Pull request overview
This PR prepares the API for a V1 release by normalizing datetime handling to UTC, improving well details payload structure (field events/participants), and updating dependency locks and workflows.
Changes:
- Normalize CSV/import datetimes to UTC (timezone-naive treated as America/Denver; timezone-aware converted via provided offset).
- Store water-level field staff as structured
FieldEventParticipantrecords and expose them in well details payloads. - Convert many DB-backed FastAPI routes from
async deftodefto avoid blocking the event loop with synchronous SQLAlchemy sessions; update dependencies/workflows accordingly.
Reviewed changes
Copilot reviewed 65 out of 68 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Bumps pinned dependency artifacts (authlib, mako, pillow, python-multipart). |
| requirements.txt | Updates pinned dependency versions/hashes to match lock updates. |
| pyproject.toml | Updates dependency pins to match lock updates. |
| services/util.py | Adds UTC-normalization helper; adjusts quad name helper return type. |
| schemas/well_inventory.py | Normalizes key datetime fields to UTC using shared utility. |
| schemas/water_level_csv.py | Normalizes water level CSV datetime fields to UTC; adjusts validators. |
| services/water_level_csv.py | Creates/reuses field event participants/contacts; links samples to measuring participant; stops embedding staff in notes. |
| services/well_details_helper.py | Reworks details payload to return field events/participants/activities/samples rather than “latest sample/recent obs” fields. |
| schemas/well_details.py | Introduces new well-details response shapes for field events/participants/activities/samples/observations. |
| schemas/field.py | Adds FieldEventParticipantResponse for structured participant output. |
| schemas/thing.py | Adds site_name, historic/access note list fields, changes open_status to boolean; refactors well screen response base type. |
| schemas/location.py | Adds release_status to geojson location response. |
| db/thing.py | Adds site_name, historic_depth_to_water, well_location_note properties; converts open_status to boolean derivation. |
| db/notes.py | Orders notes by created_at and sorts results in _get_notes(). |
| api/thing.py | Converts many routes to sync def; adds field_event_limit param for details; expands DB error handling. |
| api/sample.py | Converts routes to sync def; expands DB error handling. |
| api/contact.py | Cleans imports, converts routes to sync def; expands DB error handling. |
| api/sensor.py | Converts routes to sync def. |
| api/search.py | Converts route to sync def. |
| api/publication.py | Converts route to sync def. |
| api/observation.py | Converts many routes to sync def. |
| api/ngwmn.py | Converts routes to sync def. |
| api/location.py | Converts routes to sync def. |
| api/lexicon.py | Converts routes to sync def. |
| api/group.py | Converts routes to sync def. |
| api/geospatial.py | Converts routes to sync def. |
| api/geochronology.py | Converts routes to sync def. |
| api/author.py | Converts route to sync def. |
| tests/test_well_inventory.py | Updates assertions for ordered notes and boolean open status; updates UTC expectations. |
| tests/test_water_level_csv_service.py | Adds coverage for participant creation, idempotency, ambiguity errors, and contact reuse. |
| tests/test_cli_commands.py | Updates expectations around notes and cleans up participant-created contacts in teardown. |
| tests/test_thing.py | Expands details payload assertions to include field events/participants and new well fields; adds tests for site name/open status. |
| tests/conftest.py | Enhances fixture teardown to remove importer-created participant contacts safely. |
| tests/test_transfer_legacy_dates.py | Updates migration wording and import ordering. |
| tests/test_nma_chemistry_lineage.py | Makes driver-agnostic note about NOT NULL violations. |
| tests/integration/test_nma_legacy_relationships.py | Makes driver-agnostic note and adjusts orphan id string. |
| tests/features/well-inventory-csv.feature | Updates BDD spec for UTC normalization semantics and supports timezone-aware inputs. |
| tests/features/steps/well-inventory-csv.py | Updates step implementations to validate/normalize timezone handling consistent with new semantics. |
| features/admin/README.md | Updates project directory name references. |
| admin/views/init.py | Reorders imports and updates package docstring naming. |
| admin/config.py | Updates docstring naming; import ordering cleanup. |
| admin/auth.py | Import ordering cleanup; updates docstring naming. |
| admin/init.py | Updates package docstring naming. |
| admin/views/thing.py | Updates docstring naming. |
| admin/views/surface_water.py | Updates docstring naming. |
| admin/views/sensor.py | Updates docstring naming. |
| admin/views/sample.py | Updates docstring naming. |
| admin/views/parameter.py | Updates docstring naming. |
| admin/views/observation.py | Updates docstring naming. |
| admin/views/notes.py | Updates docstring naming. |
| admin/views/location.py | Updates docstring naming. |
| admin/views/lexicon.py | Updates docstring naming. |
| admin/views/group.py | Updates docstring naming. |
| admin/views/geologic_formation.py | Updates docstring naming. |
| admin/views/field.py | Updates docstring naming. |
| admin/views/deployment.py | Updates docstring naming. |
| admin/views/data_provenance.py | Updates docstring naming. |
| admin/views/contact.py | Updates docstring naming. |
| admin/views/asset.py | Updates docstring naming. |
| admin/views/aquifer_type.py | Updates docstring naming. |
| admin/views/aquifer_system.py | Updates docstring naming. |
| ADR2.md | Adds ADR documenting sync DB + async route concurrency issue and remediation plan. |
| .gitignore | Ignores .codex directory. |
| .github/workflows/tests.yml | Updates setup-uv action version. |
| .github/workflows/jira_codex_pr.yml | Updates pinned setup-uv action ref. |
| .github/workflows/CD_testing.yml | Adds new testing deploy workflow (App Engine deploy + tagging). |
| .github/workflows/CD_staging.yml | Updates setup-uv version and increases gunicorn workers. |
| .github/workflows/CD_production.yml | Updates setup-uv version and increases gunicorn workers. |
| assert ( | ||
| thing.open_status == "Open" | ||
| thing.open_status is True | ||
| if file_content["is_open"].lower() == "true" | ||
| else "Closed" | ||
| else thing.open_status is False | ||
| ) |
There was a problem hiding this comment.
This assertion is hard to read and uses a conditional expression with is True / is False. Since open_status is now boolean, consider asserting directly against the expected boolean (e.g., compare to (file_content['is_open'].lower() == 'true')) to make the intent clearer.
| def _get_notes(self, note_type: str) -> list[Notes]: | ||
| return [n for n in self.notes if n.note_type == note_type] | ||
| notes = [n for n in self.notes if n.note_type == note_type] | ||
| return sorted(notes, key=lambda n: n.created_at) |
There was a problem hiding this comment.
The relationship is already ordered by Notes.created_at, but _get_notes() sorts again. This duplicates work and can still be unstable when multiple notes share the same created_at. Consider relying on the relationship ordering (or sorting by (created_at, id) once) and avoid the extra sorted(...) here.
| - name: Check out source repository | ||
| uses: actions/checkout@v6.0.2 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Install uv in container | ||
| uses: astral-sh/setup-uv@v8.1.0 | ||
| with: | ||
| version: "latest" | ||
|
|
||
| - name: Generate requirements.txt | ||
| run: | | ||
| uv export \ | ||
| --format requirements-txt \ | ||
| --no-emit-project \ | ||
| --no-dev \ | ||
| --output-file requirements.txt | ||
|
|
||
| - name: Authenticate to Google Cloud | ||
| uses: 'google-github-actions/auth@v3' | ||
| with: | ||
| credentials_json: ${{ secrets.CLOUD_DEPLOY_SERVICE_ACCOUNT_KEY }} | ||
|
|
||
| - name: Run Alembic migrations on staging database | ||
| env: | ||
| DB_DRIVER: "cloudsql" | ||
| CLOUD_SQL_INSTANCE_NAME: "${{ secrets.CLOUD_SQL_INSTANCE_NAME }}" | ||
| CLOUD_SQL_DATABASE: "${{ vars.CLOUD_SQL_DATABASE }}" | ||
| CLOUD_SQL_USER: "${{ secrets.CLOUD_SQL_USER }}" | ||
| CLOUD_SQL_IAM_AUTH: true | ||
| run: | | ||
| uv run alembic upgrade head | ||
|
|
||
| - name: Refresh materialized views on staging database | ||
| env: | ||
| DB_DRIVER: "cloudsql" | ||
| CLOUD_SQL_INSTANCE_NAME: "${{ secrets.CLOUD_SQL_INSTANCE_NAME }}" | ||
| CLOUD_SQL_DATABASE: "${{ vars.CLOUD_SQL_DATABASE }}" | ||
| CLOUD_SQL_USER: "${{ secrets.CLOUD_SQL_USER }}" | ||
| CLOUD_SQL_IAM_AUTH: true | ||
| run: | | ||
| uv run python -m cli.cli refresh-pygeoapi-materialized-views | ||
|
|
||
| - name: Ensure envsubst is available | ||
| run: | | ||
| if ! command -v envsubst >/dev/null 2>&1; then | ||
| sudo apt-get update | ||
| sudo apt-get install -y gettext-base | ||
| fi | ||
|
|
||
| - name: Render App Engine configs |
There was a problem hiding this comment.
The steps: list is indented at the same level as its - name entries, which makes this workflow invalid YAML (list items must be indented under steps:). Adjust indentation so each step item is nested under steps: (matching the style used in other workflows).
| - name: Check out source repository | |
| uses: actions/checkout@v6.0.2 | |
| with: | |
| fetch-depth: 0 | |
| - name: Install uv in container | |
| uses: astral-sh/setup-uv@v8.1.0 | |
| with: | |
| version: "latest" | |
| - name: Generate requirements.txt | |
| run: | | |
| uv export \ | |
| --format requirements-txt \ | |
| --no-emit-project \ | |
| --no-dev \ | |
| --output-file requirements.txt | |
| - name: Authenticate to Google Cloud | |
| uses: 'google-github-actions/auth@v3' | |
| with: | |
| credentials_json: ${{ secrets.CLOUD_DEPLOY_SERVICE_ACCOUNT_KEY }} | |
| - name: Run Alembic migrations on staging database | |
| env: | |
| DB_DRIVER: "cloudsql" | |
| CLOUD_SQL_INSTANCE_NAME: "${{ secrets.CLOUD_SQL_INSTANCE_NAME }}" | |
| CLOUD_SQL_DATABASE: "${{ vars.CLOUD_SQL_DATABASE }}" | |
| CLOUD_SQL_USER: "${{ secrets.CLOUD_SQL_USER }}" | |
| CLOUD_SQL_IAM_AUTH: true | |
| run: | | |
| uv run alembic upgrade head | |
| - name: Refresh materialized views on staging database | |
| env: | |
| DB_DRIVER: "cloudsql" | |
| CLOUD_SQL_INSTANCE_NAME: "${{ secrets.CLOUD_SQL_INSTANCE_NAME }}" | |
| CLOUD_SQL_DATABASE: "${{ vars.CLOUD_SQL_DATABASE }}" | |
| CLOUD_SQL_USER: "${{ secrets.CLOUD_SQL_USER }}" | |
| CLOUD_SQL_IAM_AUTH: true | |
| run: | | |
| uv run python -m cli.cli refresh-pygeoapi-materialized-views | |
| - name: Ensure envsubst is available | |
| run: | | |
| if ! command -v envsubst >/dev/null 2>&1; then | |
| sudo apt-get update | |
| sudo apt-get install -y gettext-base | |
| fi | |
| - name: Render App Engine configs | |
| - name: Check out source repository | |
| uses: actions/checkout@v6.0.2 | |
| with: | |
| fetch-depth: 0 | |
| - name: Install uv in container | |
| uses: astral-sh/setup-uv@v8.1.0 | |
| with: | |
| version: "latest" | |
| - name: Generate requirements.txt | |
| run: | | |
| uv export \ | |
| --format requirements-txt \ | |
| --no-emit-project \ | |
| --no-dev \ | |
| --output-file requirements.txt | |
| - name: Authenticate to Google Cloud | |
| uses: 'google-github-actions/auth@v3' | |
| with: | |
| credentials_json: ${{ secrets.CLOUD_DEPLOY_SERVICE_ACCOUNT_KEY }} | |
| - name: Run Alembic migrations on staging database | |
| env: | |
| DB_DRIVER: "cloudsql" | |
| CLOUD_SQL_INSTANCE_NAME: "${{ secrets.CLOUD_SQL_INSTANCE_NAME }}" | |
| CLOUD_SQL_DATABASE: "${{ vars.CLOUD_SQL_DATABASE }}" | |
| CLOUD_SQL_USER: "${{ secrets.CLOUD_SQL_USER }}" | |
| CLOUD_SQL_IAM_AUTH: true | |
| run: | | |
| uv run alembic upgrade head | |
| - name: Refresh materialized views on staging database | |
| env: | |
| DB_DRIVER: "cloudsql" | |
| CLOUD_SQL_INSTANCE_NAME: "${{ secrets.CLOUD_SQL_INSTANCE_NAME }}" | |
| CLOUD_SQL_DATABASE: "${{ vars.CLOUD_SQL_DATABASE }}" | |
| CLOUD_SQL_USER: "${{ secrets.CLOUD_SQL_USER }}" | |
| CLOUD_SQL_IAM_AUTH: true | |
| run: | | |
| uv run python -m cli.cli refresh-pygeoapi-materialized-views | |
| - name: Ensure envsubst is available | |
| run: | | |
| if ! command -v envsubst >/dev/null 2>&1; then | |
| sudo apt-get update | |
| sudo apt-get install -y gettext-base | |
| fi | |
| - name: Render App Engine configs |
| # ":" are not alloed in git tags, so replace with "-" | ||
| - name: Tag commit | ||
| run: | | ||
| git tag -a "testing-deploy-$(date -u +%Y-%m-%d)T$(date -u +%H-%M-%S%z)" -m "testing gcloud deployment: $ | ||
| (date | ||
| -u +%Y-%m-%d)T$(date -u +%H:%M:%S%z)" |
There was a problem hiding this comment.
The tag creation command is split across multiple lines inside a single quoted -m argument, which will break the shell command and likely fail the workflow. Consider building the tag message in a variable (or using a single line) and fix the typo in the comment (“allowed”).
| # ":" are not alloed in git tags, so replace with "-" | |
| - name: Tag commit | |
| run: | | |
| git tag -a "testing-deploy-$(date -u +%Y-%m-%d)T$(date -u +%H-%M-%S%z)" -m "testing gcloud deployment: $ | |
| (date | |
| -u +%Y-%m-%d)T$(date -u +%H:%M:%S%z)" | |
| # ":" are not allowed in git tags, so replace with "-" | |
| - name: Tag commit | |
| run: | | |
| TAG_TIMESTAMP="$(date -u +%Y-%m-%d)T$(date -u +%H-%M-%S%z)" | |
| MESSAGE_TIMESTAMP="$(date -u +%Y-%m-%d)T$(date -u +%H:%M:%S%z)" | |
| TAG_MESSAGE="testing gcloud deployment: $MESSAGE_TIMESTAMP" | |
| git tag -a "testing-deploy-$TAG_TIMESTAMP" -m "$TAG_MESSAGE" |
| else: | ||
| detail = { | ||
| "loc": ["body"], | ||
| "msg": error_message, | ||
| "type": "value_error", | ||
| "input": {}, | ||
| } |
There was a problem hiding this comment.
The fallback branch returns the raw database error message to clients. This can leak internal schema/constraint details and produces inconsistent API errors. Consider returning a generic message (and logging the full DB error server-side), or map additional known constraints explicitly.
| else: | ||
| detail = { | ||
| "loc": ["body"], | ||
| "msg": error_message, | ||
| "type": "value_error", | ||
| "input": {}, | ||
| } |
There was a problem hiding this comment.
Same as in api/thing.py: the fallback handler returns raw DB error strings to clients, which can leak internal details. Prefer a generic client message and log the original error, or explicitly handle additional expected constraints.
| else: | ||
| detail = { | ||
| "loc": ["body"], | ||
| "msg": error_message, | ||
| "type": "value_error", | ||
| "input": {}, | ||
| } |
There was a problem hiding this comment.
Same as in api/thing.py: the fallback branch returns the raw DB error string to clients. Consider returning a generic validation message (and logging the DB error) to avoid leaking constraint/table details.
| @@ -263,7 +265,7 @@ class WellResponse(BaseThingResponse): | |||
| well_pump_depth: float | None | |||
| well_pump_depth_unit: str = "ft" | |||
| well_status: str | None | |||
| open_status: str | None | |||
| open_status: bool | None | |||
| datalogger_suitability_status: str | None | |||
| measuring_point_height: float | None | |||
| measuring_point_height_unit: str = "ft" | |||
| @@ -275,6 +277,7 @@ class WellResponse(BaseThingResponse): | |||
| permissions: list[PermissionHistoryResponse] | |||
| formation_completion_code: FormationCode | None | |||
| nma_formation_zone: str | None | |||
| well_location_note: list[str] = [] | |||
|
|
|||
There was a problem hiding this comment.
historic_depth_to_water and well_location_note use mutable list literals as defaults. This can lead to shared state across model instances; use Field(default_factory=list) instead (consistent with other response schemas in this repo).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c66d10d27e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def _get_notes(self, note_type: str) -> list[Notes]: | ||
| return [n for n in self.notes if n.note_type == note_type] | ||
| notes = [n for n in self.notes if n.note_type == note_type] |
There was a problem hiding this comment.
Guard note sorting against unpersisted timestamps
_get_notes now sorts by created_at, but created_at is DB-populated and is None on newly added Notes objects until flush/commit. Python sorting with None keys raises TypeError, so calling note-backed properties (for example Thing.historic_depth_to_water/well_location_note) on an object with in-memory notes can fail before persistence. This is a runtime regression introduced by the new unconditional sort.
Useful? React with 👍 / 👎.
order_sort_filter now walks every filter JSON in the request (optional filters list plus legacy single filter_). _apply_json_filter_clause supports contains, ncontains, startswith, endswith, eq, ne, comparison operators, null checks, and in, in addition to the old contains-only behavior. Any list or report that sends filter= more than once or uses operators other than contains will now get real SQL instead of silent no-ops.
get_db_things accepts filters as a list of JSON strings (merged with legacy single filter_) and passes them to order_sort_filter. Adds name_contains to narrow rows with Thing.name ILIKE for substring search without changing the full-text query path.
GET /thing/water-well, GET /thing/spring, and GET /thing now bind filter as a list so multiple Refine filters are forwarded. Same routes accept optional name_contains for substring matching on thing name alongside existing query.
…st-filtering BDMS 774 Fix WELL List filters
V1 release of the api