Skip to content

NMBGMR Timeout fix#67

Open
jacob-a-brown wants to merge 10 commits intomainfrom
jab-nmbgmr-timeout-fix
Open

NMBGMR Timeout fix#67
jacob-a-brown wants to merge 10 commits intomainfrom
jab-nmbgmr-timeout-fix

Conversation

@jacob-a-brown
Copy link
Copy Markdown
Contributor

@jacob-a-brown jacob-a-brown commented Apr 29, 2026

Why

This PR addresses the following problem / context:

  • Some sites have many records, causing the server to take too long to respond and therefore raise httpx read timeout errors

How

Implementation summary - the following was changed / added / removed:

  • Increase the timeout parameter to 15 minutes
  • If the request fails for any reason, timeout or otherwise, retry the request up to 7 times

Notes

Any special considerations, workarounds, or follow-up work to note?

  • The logic for retries was moved to BaseSource so it applies everywhere where the _execute_json_request and _execute_text_request methods are invoked.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the NMBGMR connector to better tolerate slow upstream responses by increasing request timeouts and adding retry loops around key API calls to avoid httpx read timeouts when sites have many records.

Changes:

  • Introduces a 30-minute timeout constant and applies it to NMBGMR API requests.
  • Adds retry loops around site, analyte, and water level (including pagination) requests when requests fail or return None.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread backend/connectors/nmbgmr/source.py Outdated
Comment thread backend/connectors/nmbgmr/source.py Outdated
Comment thread backend/connectors/nmbgmr/source.py Outdated
Comment thread backend/connectors/nmbgmr/source.py Outdated
Comment thread backend/connectors/nmbgmr/source.py Outdated
this prevents code duplication and allows all sources to benefit from retry
logic when making requests to external APIs.
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to reduce httpx read timeouts and improve resilience for slow upstream APIs (notably NMBGMR) by increasing request timeouts and centralizing bounded retry logic in BaseSource so all connectors benefit.

Changes:

  • Added retry + backoff behavior to BaseSource._execute_json_request and _execute_text_request (default 7 attempts).
  • Introduced a 15-minute timeout constant for NMBGMR connector requests and applied it to key endpoints.
  • Updated request call sites in the NMBGMR connector to use the new timeout.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File Description
backend/source.py Centralizes bounded retries/backoff for text + JSON HTTP requests in BaseSource.
backend/connectors/nmbgmr/source.py Applies a longer timeout constant to NMBGMR requests to reduce read timeouts for large/slow responses.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread backend/source.py Outdated
Comment thread backend/source.py
Comment thread backend/source.py Outdated
Comment thread backend/source.py Outdated
Comment thread backend/source.py
Comment on lines 224 to 226
if "timeout" not in kw:
kw["timeout"] = 10

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intent is for this to apply to NMBGMR, at least so far. @jirhiker and @likithabommasani21 should the 15 minute timeout be the default for all sources?

Comment thread backend/source.py
Comment thread backend/connectors/nmbgmr/source.py Outdated
@jacob-a-brown jacob-a-brown requested review from jirhiker and likithabommasani21 and removed request for jirhiker and likithabommasani21 May 1, 2026 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants