feat(providers): support large agent-block attachments via Files APIs and remote URLs#5092
feat(providers): support large agent-block attachments via Files APIs and remote URLs#5092waleedlatif1 wants to merge 16 commits into
Conversation
… and remote URLs Agent-block file uploads were inlined as base64 with a hard 10MB cap. Files above the threshold now use each provider's native large-file path: - OpenAI / Gemini: upload to the provider Files API, reference by file_id/uri - Anthropic: GA url content-block source (no Files API beta, no upload) - OpenRouter/Groq/Together/Baseten/xAI/vLLM: remote signed URL in image_url/file - Limits live per-provider in models.ts; the agent block + /models page reflect them Files <=10MB keep the identical base64 path (zero regression). Server-only file handles are stripped from untrusted input to prevent SSRF.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
|
@greptile |
|
@cursor review |
PR SummaryHigh Risk Overview Provider pipeline: Product surface: Per-provider Tests cover strategy/ceiling behavior, handle-based builder output, and stripping forged handles from client file input. Reviewed by Cursor Bugbot for commit 68849dd. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
attachLargeFileRemoteUrls early-returned for inline-strategy providers before clearing server-only handle fields, so a forged remoteUrl on an inline-provider file could still reach a builder (e.g. buildOpenAICompatibleChatContent for mistral/ollama). Clear the handles for every provider before the strategy check.
Greptile SummaryThis PR upgrades agent-block file attachments from a flat base64 model (hard-capped at 10 MB) to a tiered system: files ≤10 MB continue to inline as base64, while larger files are routed through each provider's native large-file path — OpenAI/Gemini via Files API upload, Anthropic via URL content-block sources, and the remaining supported providers via presigned HTTPS URLs in
Confidence Score: 3/5The core tiered-attachment pipeline is architecturally sound and the SSRF defense is thorough, but the new Gemini upload path contains an unsafe type assertion that will produce a confusing, hard-to-diagnose failure if the file API response is incomplete. The overall design is well-structured — presigned URL generation, handle-based deduplication, and per-provider ceiling validation all work together correctly. The SSRF strip in file-utils and the handle-clearing in attachLargeFileRemoteUrls are solid defense layers. The one clear defect is in uploadGeminiFile: the uploaded.name as string assertion silences TypeScript but lets undefined flow into ai.files.get(), which would surface as a cryptic Gemini API error in production rather than pointing at the missing name. This would silently block all large-file uploads to Gemini until diagnosed. The OpenAI expires_after bracket-notation encoding is an uncertainty about file lifecycle management rather than functional breakage, but worth verifying against the live API. apps/sim/providers/file-attachments.server.ts — specifically the Gemini polling loop and the OpenAI expires_after field encoding. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant AH as AgentHandler
participant FAS as file-attachments.server
participant SS as StorageService
participant PI as providers/index
participant OAI as OpenAI Files API
participant GM as Gemini Files API
participant PR as Provider (execute)
AH->>AH: "hydrateUserFilesWithBase64(maxBytes=10MB)"
AH->>FAS: attachLargeFileRemoteUrls(files, providerId)
FAS->>FAS: Clear providerFileId/Uri/remoteUrl (SSRF defense)
loop each file over 10MB on capable provider
FAS->>FAS: verifyFileAccess(key, userId)
FAS->>SS: generatePresignedDownloadUrl(key, 1h)
SS-->>FAS: presignedUrl
FAS->>FAS: "file.remoteUrl = presignedUrl"
end
AH->>AH: missingFile check (base64 OR remoteUrl required)
AH->>PI: executeProviderRequest(request)
PI->>FAS: uploadLargeFilesToProvider(request, providerId)
alt files-api strategy (openai/google)
loop each uploadable file group
FAS->>FAS: fetchRemoteFileBlob(file.remoteUrl)
alt OpenAI
FAS->>OAI: POST /v1/files (purpose, expires_after, blob)
OAI-->>FAS: file id
FAS->>FAS: "file.providerFileId = id"
else Gemini
FAS->>GM: ai.files.upload(blob, mimeType)
loop poll until ACTIVE
FAS->>GM: ai.files.get(name)
GM-->>FAS: state + uri
end
FAS->>FAS: "file.providerFileUri = uri"
end
end
end
PI->>PR: provider.executeRequest(sanitizedRequest)
note over PR: Builders use file_id / fileData / remoteUrl / base64
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant AH as AgentHandler
participant FAS as file-attachments.server
participant SS as StorageService
participant PI as providers/index
participant OAI as OpenAI Files API
participant GM as Gemini Files API
participant PR as Provider (execute)
AH->>AH: "hydrateUserFilesWithBase64(maxBytes=10MB)"
AH->>FAS: attachLargeFileRemoteUrls(files, providerId)
FAS->>FAS: Clear providerFileId/Uri/remoteUrl (SSRF defense)
loop each file over 10MB on capable provider
FAS->>FAS: verifyFileAccess(key, userId)
FAS->>SS: generatePresignedDownloadUrl(key, 1h)
SS-->>FAS: presignedUrl
FAS->>FAS: "file.remoteUrl = presignedUrl"
end
AH->>AH: missingFile check (base64 OR remoteUrl required)
AH->>PI: executeProviderRequest(request)
PI->>FAS: uploadLargeFilesToProvider(request, providerId)
alt files-api strategy (openai/google)
loop each uploadable file group
FAS->>FAS: fetchRemoteFileBlob(file.remoteUrl)
alt OpenAI
FAS->>OAI: POST /v1/files (purpose, expires_after, blob)
OAI-->>FAS: file id
FAS->>FAS: "file.providerFileId = id"
else Gemini
FAS->>GM: ai.files.upload(blob, mimeType)
loop poll until ACTIVE
FAS->>GM: ai.files.get(name)
GM-->>FAS: state + uri
end
FAS->>FAS: "file.providerFileUri = uri"
end
end
end
PI->>PR: provider.executeRequest(sanitizedRequest)
note over PR: Builders use file_id / fileData / remoteUrl / base64
Reviews (2): Last reviewed commit: "fix(providers): clear forged file handle..." | Re-trigger Greptile |
…ge-text-doc handling - OpenAI upload now uses the SDK (client.files.create) so expires_after is serialized as a real nested object; the prior expires_after[anchor] bracket FormData keys were ignored by OpenAI's server, leaving files un-expiring. - Anthropic url document source only supports PDFs/images; large non-PDF text docs now throw a clear error instead of emitting an unsupported url source. - Warn when an oversized file can't be sent because cloud storage is unavailable.
|
@greptile |
|
@cursor review |
…-file UI limit) - Download files for OpenAI/Gemini uploads via validateUrlWithDNS + IP-pinned fetch so a forged URL can't reach internal addresses (covers all callers). - Reject files above the provider ceiling before downloading/uploading. - UI now validates each file against the provider's per-file ceiling instead of summing all files against it, matching server-side per-file validation. - Lower Anthropic ceiling to 50MB (documented 32MB request cap / page limits).
|
@greptile |
|
@cursor review |
Read OpenAI/Gemini upload bytes through downloadFileFromStorage instead of HTTP-fetching the presigned URL. Removes any server-side URL fetch (no SSRF vector) and works with internal object storage (e.g. self-hosted MinIO), which an IP-pinned URL fetch would have blocked.
|
@greptile review |
…oad path uploadLargeFilesToProvider runs on raw request messages for every caller (incl. the internal providers passthrough), so harden it independently of the agent path: - verifyFileAccess on each file's storage key before reading its bytes, so a forged key can't exfiltrate another user's file. - clear any inbound providerFileId/providerFileUri up front (legit ids are only set by the upload itself), so a forged id can't reference a file in a hosted account.
|
@greptile review |
…ider helper as execution The file-upload control imported getProviderFromModel from @/providers/models, but the execution path and every other consumer use the one in @/providers/utils (runtime registry + reseller patterns). Align the UI so its size cap can't disagree with server-side validation for reseller or dynamically-listed models.
|
@greptile review |
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 9c4ad83. Configure here.
attachments.ts now reads getProviderFileAttachment / INLINE_ATTACHMENT_MAX_BYTES from @/providers/models; the provider unit tests that fully mock that module need both exports or attachments.ts fails to load.
ai.files.upload returns name as string | undefined; guard it (instead of an as-string cast) so a missing name surfaces a clear error at the upload site rather than an opaque files.get failure on the first poll.
…e fields
The 'as const' readonly tuple widened omit's K to all keys, collapsing
Omit<UserFile, K> to {} and failing the production build's type check. Declare
the array as Array<keyof handle fields> so K is the precise literal union.
…quest for all callers Move attachLargeFileRemoteUrls out of the agent handler and into executeProviderRequest (right before uploadLargeFilesToProvider), so every entry point — including the internal providers passthrough — clears forged handles and mints/access-checks large-file URLs uniformly. The agent handler now only hydrates base64; its missing-file guard exempts large files (resolved downstream).
…part PreparedProviderAttachment.dataUrl is now optional (large files carry a handle instead); azure-openai builds chat content inline and assigned it directly to a required url field, failing the production build's type check.
… part The installed openai SDK (4.104) does not type expires_after on files.create, so upload via POST /v1/files directly with the documented expires_after[...] form fields (gives the file an auto-expiry). Also wrap the storage Buffer in a Uint8Array for the Blob, which the production build's stricter lib types require. These two type errors were masked locally because tsc was OOMing silently without the type-check script's --max-old-space-size flag.
|
@greptile review |
|
@cursor review |
…derRequest Large-attachment prep now needs request.userId for presigned URLs and access checks; the authenticated providers proxy has auth.userId but wasn't passing it, so oversized attachments failed for logged-in callers. Forwarding it makes large files work there and keeps the access check (verifyFileAccess) intact.
|
@greptile review |
|
@cursor review |
…rage The doc claimed a base64 fallback that doesn't exist — above the inline cap there is no base64, so without cloud storage the file previously reached the builder and died with a generic read error. Throw a clear 'requires cloud file storage' error at the point of detection and correct the doc.
|
@greptile review |
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 68849dd. Configure here.
1 participant
Summary
file_id/fileUri(with processing-state polling for Gemini,expires_afterauto-expiry for OpenAI).urlcontent-block source (URLImageSource/URLPDFSource) — no Files API beta header, no upload, no lifecycle.image_url(and OpenRouterfile.file_datafor PDFs) instead of base64.models.tsat the provider level; the agent block's file input and the/modelsprovider pages now reflect the real ceiling.Architecture
agent-handlermints a presigned download URL onto large files (has execution context + access checks); base64 hydration stays capped at the 10MB inline threshold so a 512MB file is never base64-encoded.providers/index.ts(after the API key is resolved) uploadsfiles-apiproviders' large files from that URL — so hosted and BYOK keys both work.providerFileId/providerFileUri/remoteUrl), else fall back to base64.Type of Change
Testing
providers/attachments.test.ts(capability table, strategy/ceiling per provider, handle-based builder output, oversize rejection) and added an SSRF-strip test infile-utils.test.ts— 34 passing.bunx tsc --noEmitclean;bun run check:api-validationpasses.Checklist