Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
ea5a07b
feat: add LU tables to transfers for mapping to lexicon
jacob-a-brown Sep 10, 2025
84654ae
Merge branch 'jab-lexicon-updates' into jab-lexicon-lookup-tables
jacob-a-brown Sep 10, 2025
25627e3
Merge branch 'pre-production' into jab-lexicon-lookup-tables
jacob-a-brown Sep 10, 2025
cfd88ff
feat: make LU to lexicon mapper for data transfer
jacob-a-brown Sep 10, 2025
34705aa
Merge branch 'pre-production' into jab-lexicon-lookup-tables
jacob-a-brown Sep 11, 2025
dd970d3
refactor: move utils to services for use throughout API
jacob-a-brown Sep 11, 2025
dc89117
feat: set location state/county/quad from point
jacob-a-brown Sep 11, 2025
5dec7f8
feat: simplify deployment configurations
jirhiker Sep 12, 2025
f3758d7
feat: enhance logging and commit frequency during data transfer proce…
jirhiker Sep 12, 2025
6bdf14e
Formatting changes
jirhiker Sep 12, 2025
f753800
feat: implement replace_nans utility function and refactor data handl…
jirhiker Sep 12, 2025
54772a3
Formatting changes
jirhiker Sep 12, 2025
6c0e314
feat: set elevation from national map if none in nma
jacob-a-brown Sep 12, 2025
afd6261
Merge branch 'staging' into jab-lexicon-lookup-tables
jacob-a-brown Sep 12, 2025
948b17b
feat: map lu tables to lexicon for transfer
jacob-a-brown Sep 12, 2025
db3a857
fix: delete LU tables locally - get from cloud
jacob-a-brown Sep 12, 2025
8bedd14
refactor: use variable for WGS84 SRID throughout
jacob-a-brown Sep 12, 2025
3556584
Merge branch 'jab-lexicon-lookup-tables' into jab-location-transfer-u…
jacob-a-brown Sep 12, 2025
9251bb8
WIP: apply LU-lexicon mapper
jacob-a-brown Sep 12, 2025
965f563
feat: prepend mapper key with LU table name for uniqueness
jacob-a-brown Sep 12, 2025
237e52a
Merge branch 'jab-lexicon-lookup-tables' into jab-location-transfer-u…
jacob-a-brown Sep 12, 2025
b3cc66f
refactor: remove name from location until future decision is made
jacob-a-brown Sep 12, 2025
97ea48f
feat: skip record if error, not stop process
jacob-a-brown Sep 12, 2025
22394a0
refactor: increase timeout for location geographic requests
jacob-a-brown Sep 12, 2025
8fccdf4
refactor: decrease limit for development
jacob-a-brown Sep 12, 2025
16ab98a
fix: skip and log records with duplicates
jacob-a-brown Sep 12, 2025
bf58bb6
refactor: use pointid not index in log statement
jacob-a-brown Sep 12, 2025
4040a54
WIP: work on location and dt transfers
jacob-a-brown Sep 12, 2025
6fef249
feat: convert times from mst/mdt to utc
jacob-a-brown Sep 15, 2025
c885ef5
note: remove duplicative note
jacob-a-brown Sep 15, 2025
ff98139
refactor: wait for AMP feedback before transfering coord accuracy values
jacob-a-brown Sep 15, 2025
1b8af65
refactor: made elevation separate from point field
jacob-a-brown Sep 15, 2025
dbfef6c
fix: run tests for update branch names
jacob-a-brown Sep 15, 2025
96dbff8
fix: run tests on new branch names
jacob-a-brown Sep 15, 2025
98ac4ac
Merge pull request #137 from DataIntegrationGroup/jab-location-transf…
jirhiker Sep 15, 2025
4ecb5f1
feat: add documentation summarizing the purpose and contents of the d…
ksmuczynski Sep 16, 2025
b08461a
feat: add polymorphic helper mixin HasStatusHistory
ksmuczynski Sep 16, 2025
87bd7ca
refactor: update mixin name from `HasStatusHistory` to `StatusHistory…
ksmuczynski Sep 17, 2025
343af2b
Merge pull request #138 from DataIntegrationGroup/statushistory_model
jirhiker Sep 17, 2025
d6466f9
update: create new `Permission` model
ksmuczynski Sep 18, 2025
3eb3e41
update: add `PermissionMixin` to `db/base.py` to automatically create…
ksmuczynski Sep 18, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions .github/workflows/CD_production.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,6 @@ jobs:
echo "runtime: python313" >> app.yaml
echo "entrypoint: gunicorn -w 4 -k uvicorn.workers.UvicornWorker main:app" >> app.yaml
echo "instance_class: F4" >> app.yaml
echo "inbound_services:" >> app.yaml
echo " - warmup" >> app.yaml
echo "automatic_scaling:" >> app.yaml
echo " min_instances: 0" >> app.yaml
echo " max_instances: 10" >> app.yaml
echo "" >> app.yaml
echo "env_variables:" >> app.yaml
echo " MODE: \"production\"" >> app.yaml
Expand Down
5 changes: 0 additions & 5 deletions .github/workflows/CD_staging.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,6 @@ jobs:
echo "runtime: python313" >> app.yaml
echo "entrypoint: gunicorn -w 4 -k uvicorn.workers.UvicornWorker main:app" >> app.yaml
echo "instance_class: F4" >> app.yaml
echo "inbound_services:" >> app.yaml
echo " - warmup" >> app.yaml
echo "automatic_scaling:" >> app.yaml
echo " min_instances: 0" >> app.yaml
echo " max_instances: 10" >> app.yaml
echo "" >> app.yaml
echo "env_variables:" >> app.yaml
echo " MODE: \"production\"" >> app.yaml
Expand Down
83 changes: 0 additions & 83 deletions .github/workflows/dev_deploy.yml

This file was deleted.

2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ name: Tests

on:
pull_request:
branches: [ "main",'pre-production', 'transfer']
branches: ['production', 'staging', 'transfer']

permissions:
contents: read
Expand Down
8 changes: 5 additions & 3 deletions api/geospatial.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# limitations under the License.
# ===============================================================================
import json
from typing import Annotated, List, Union
from typing import Annotated, List

from fastapi import APIRouter, Query, HTTPException
from fastapi.responses import FileResponse
Expand Down Expand Up @@ -100,7 +100,9 @@ def get_feature_collection(

things = get_thing_features(session, thing_type, group)

def make_feature_dict(thing, geometry, *other):
def make_feature_dict(thing, geometry, elevation, *other):
geometry = json.loads(geometry)
geometry["coordinates"].append(elevation)
return {
"type": "Feature",
"properties": {
Expand All @@ -109,7 +111,7 @@ def make_feature_dict(thing, geometry, *other):
"name": thing.name,
"group": group,
},
"geometry": json.loads(geometry),
"geometry": geometry,
}

features = [make_feature_dict(*item) for item in things]
Expand Down
9 changes: 7 additions & 2 deletions api/location.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
from services.geospatial_helper import make_within_wkt
from services.query_helper import make_query, order_sort_filter, simple_get_by_id
from services.crud_helper import model_patcher, model_deleter, model_adder
from services.location_helper import set_geographic_attributes

from fastapi import APIRouter

Expand All @@ -48,7 +49,9 @@ async def create_location(
"""
Create a new sample location in the database.
"""
return model_adder(session, Location, location_data, user=user)
location = model_adder(session, Location, location_data, user=user)
set_geographic_attributes(session, location_data, location)
return location


@router.patch(
Expand All @@ -64,7 +67,9 @@ async def update_location(
"""
Update a sample location in the database.
"""
return model_patcher(session, Location, location_id, location_data, user=user)
location = model_patcher(session, Location, location_id, location_data, user=user)
set_geographic_attributes(session, location_data, location)
return location


# @router.get("/shapefile", summary="Get location as shapefile")
Expand Down
1 change: 1 addition & 0 deletions constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,5 @@
# ===============================================================================

SRID_WGS84 = 4326
SRID_UTM_ZONE_13N = 26913
# ============= EOF =============================================
89 changes: 86 additions & 3 deletions db/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,45 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ===============================================================================
"""
db/base.py

This file defines the foundational components for the SQLAlchemy models.
It includes:
1. The declarative base class (`Base`) that all models will inherit from.
2. A helper function (`lexicon_term`) to create standardized foreign key columns
referencing the `lexicon_term` table.
3. A helper function (`pascal_to_snake`) to convert class names from PascalCase to snake_case
for automatic table naming.
4. Mixins for common functionality:
- `AutoBaseMixin`: Adds automatic table naming and an auto-incrementing primary key.
- `PropertiesMixin`: Adds a JSONB properties column for storing additional attributes.
- `ReleaseMixin`: Adds a release status column referencing the `lexicon_term` table.
- `AuditMixin`: Adds standard audit columns (created_at, created_by, updated_at, updated_by).
5. A simple `User` model for tracking user information in audit columns.
6. Polymorphic helper mixins (`StatusHistoryMixin`, `NotesMixin`, `AttributionMixin`, `PermissionMixin`.)
which provide a clean, reusable way to add relationships to the polymorphic
metadata tables. Any model that can have a status history (like Thing or Location)
can simply inherit from the `StatusHistoryMixin` mixin.
7. An `AuditMixin` to add standard audit columns to tables.
"""

from sqlalchemy import (
Column,
DateTime,
func,
Integer,
JSON,
String,
Boolean,
Text,
ForeignKey,
)
from sqlalchemy.orm import DeclarativeBase, declared_attr, Mapped, mapped_column
from sqlalchemy.orm import (
DeclarativeBase,
declared_attr,
Mapped,
mapped_column,
relationship,
)
from sqlalchemy_searchable import make_searchable
from sqlalchemy_continuum import make_versioned
import re
Expand All @@ -41,6 +68,12 @@ class Base(DeclarativeBase):


def lexicon_term(foreignkeykw=None, **kw):
"""Create a SQLAlchemy mapped column for a self-referencing lexicon term.

This helper function simplifies the creation of a string column that also
acts as a foreign key to the 'term' column of the 'lexicon_term' table.
It standardizes the column type to String(100) and sets the onupdate
behavior to "CASCADE"."""

fkw = foreignkeykw if foreignkeykw else {}

Expand All @@ -55,13 +88,18 @@ def pascal_to_snake(name):
return re.sub(r"(?<!^)(?=[A-Z])", "_", name).lower()


# ============= Common Mixins =============================================
class ReleaseMixin:
"""Mixin to add release status to a model."""

@declared_attr
def release_status(self):
return lexicon_term(default="draft")


class AuditMixin:
"""Mixin to add standard audit columns to a model."""

@declared_attr
def created_at(self):
return Column(
Expand Down Expand Up @@ -109,6 +147,8 @@ def updated_by_id(self):


class AutoBaseMixin(AuditMixin):
"""Mixin to add automatic table naming and an auto-incrementing primary key."""

@declared_attr
def __tablename__(self):
return pascal_to_snake(self.__name__)
Expand All @@ -119,6 +159,8 @@ def id(self):


class PropertiesMixin:
"""Mixin to add a JSONB properties column for storing additional attributes."""

@declared_attr
def properties(self):
return Column(
Expand All @@ -129,7 +171,48 @@ def properties(self):
)


# ============= Polymorphic Helper Mixins =============================================
class StatusHistoryMixin:
"""
Mixin for models that can have a status history (e.g., Thing, Location).
It automatically creates a polymorphic One-to-Many relationship to the
StatusHistory table.
"""

@declared_attr
def status_history(self):
# One-to-Many polymorphic relationship
return relationship(
"StatusHistory",
primaryjoin=f"and_({self.__name__}.{self.__name__.lower()}_id==StatusHistory.statusable_id, "
f"StatusHistory.statusable_type=='{self.__name__}')",
cascade="all, delete-orphan",
lazy="selectin",
)


class PermissionMixin:
"""
Mixin for models that can have permissions (e.g., Thing, Location).
It automatically creates a polymorphic One-to-Many relationship to the
Permission table.
"""

@declared_attr
def permissions(self):
# One-to-Many polymorphic relationship
return relationship(
"Permission",
primaryjoin=f"and_({self.__name__}.{self.__name__.lower()}_id==Permission.permissible_id, "
f"Permission.permissible_type=='{self.__name__}')",
lazy="selectin",
viewonly=True,
)


class User(Base):
"""Represents a user in the system."""

__tablename__ = "user"

id: Mapped[int] = mapped_column(Integer, primary_key=True, nullable=False)
Expand Down
3 changes: 2 additions & 1 deletion db/group.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
from sqlalchemy.orm import relationship, Mapped
from sqlalchemy.testing.schema import mapped_column

from constants import SRID_WGS84
from db.base import Base, AutoBaseMixin, ReleaseMixin


Expand All @@ -28,7 +29,7 @@ class Group(Base, AutoBaseMixin, ReleaseMixin):
description: Mapped[str] = mapped_column(String(255), nullable=True)
name: Mapped[str] = mapped_column(String(100), nullable=False, unique=True)
project_area: Mapped[Optional[WKBElement]] = mapped_column(
Geometry(geometry_type="MULTIPOLYGON", srid=4326, spatial_index=True)
Geometry(geometry_type="MULTIPOLYGON", srid=SRID_WGS84, spatial_index=True)
)

# Foreign Keys
Expand Down
16 changes: 10 additions & 6 deletions db/location.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@
from uuid import UUID

from sqlalchemy import (
Column,
Integer,
String,
ForeignKey,
DateTime,
Expand All @@ -32,6 +30,7 @@
from sqlalchemy.orm import relationship, Mapped, mapped_column
from sqlalchemy.ext.associationproxy import association_proxy, AssociationProxy

from constants import SRID_WGS84
from db.base import Base, AutoBaseMixin, ReleaseMixin
from db.lexicon import lexicon_term

Expand All @@ -43,9 +42,12 @@ class Location(Base, AutoBaseMixin, ReleaseMixin):
String(36), nullable=True, unique=True
)
description: Mapped[str] = mapped_column
name: Mapped[str] = mapped_column(String(255), nullable=True)
# name: Mapped[str] = mapped_column(String(255), nullable=True)
point: Mapped[WKBElement] = mapped_column(
Geometry(geometry_type="POINTZ", srid=4326, spatial_index=True)
Geometry(geometry_type="POINT", srid=SRID_WGS84, spatial_index=True)
)
elevation: Mapped[float] = mapped_column(
nullable=False, comment="in meters with vertical datum of NAVD88"
)

state: Mapped[str] = lexicon_term(nullable=True, default="New Mexico")
Expand All @@ -65,7 +67,7 @@ class Location(Base, AutoBaseMixin, ReleaseMixin):
)

# --- Proxy Definitions ---
things: AssociationProxy[list["Thing"]] = association_proxy(
things: AssociationProxy[list["Thing"]] = association_proxy( # noqa: F821
"thing_associations", "thing"
)

Expand Down Expand Up @@ -95,7 +97,9 @@ class LocationThingAssociation(Base, AutoBaseMixin):

# --- Relationship Definitions ---
location: Mapped["Location"] = relationship(back_populates="thing_associations")
thing: Mapped["Thing"] = relationship(back_populates="location_associations")
thing: Mapped["Thing"] = relationship( # noqa: F821
back_populates="location_associations"
)


# ============= EOF =============================================
Loading
Loading