NMBGMR Timeout fix#67
Conversation
There was a problem hiding this comment.
Pull request overview
Updates the NMBGMR connector to better tolerate slow upstream responses by increasing request timeouts and adding retry loops around key API calls to avoid httpx read timeouts when sites have many records.
Changes:
- Introduces a 30-minute timeout constant and applies it to NMBGMR API requests.
- Adds retry loops around site, analyte, and water level (including pagination) requests when requests fail or return
None.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
this prevents code duplication and allows all sources to benefit from retry logic when making requests to external APIs.
There was a problem hiding this comment.
Pull request overview
This PR aims to reduce httpx read timeouts and improve resilience for slow upstream APIs (notably NMBGMR) by increasing request timeouts and centralizing bounded retry logic in BaseSource so all connectors benefit.
Changes:
- Added retry + backoff behavior to
BaseSource._execute_json_requestand_execute_text_request(default 7 attempts). - Introduced a 15-minute timeout constant for NMBGMR connector requests and applied it to key endpoints.
- Updated request call sites in the NMBGMR connector to use the new timeout.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
backend/source.py |
Centralizes bounded retries/backoff for text + JSON HTTP requests in BaseSource. |
backend/connectors/nmbgmr/source.py |
Applies a longer timeout constant to NMBGMR requests to reduce read timeouts for large/slow responses. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if "timeout" not in kw: | ||
| kw["timeout"] = 10 | ||
|
|
There was a problem hiding this comment.
The intent is for this to apply to NMBGMR, at least so far. @jirhiker and @likithabommasani21 should the 15 minute timeout be the default for all sources?
Why
This PR addresses the following problem / context:
How
Implementation summary - the following was changed / added / removed:
Notes
Any special considerations, workarounds, or follow-up work to note?
BaseSourceso it applies everywhere where the_execute_json_requestand_execute_text_requestmethods are invoked.