Skip to content

feat: switch to s3 storage#926

Merged
coodos merged 7 commits intomainfrom
feat/object-store-for-files
Mar 16, 2026
Merged

feat: switch to s3 storage#926
coodos merged 7 commits intomainfrom
feat/object-store-for-files

Conversation

@coodos
Copy link
Contributor

@coodos coodos commented Mar 16, 2026

Description of change

Change from db to object store

Issue Number

Type of change

  • Update (a change which updates existing functionality)

How the change has been tested

Change checklist

  • I have ensured that the CI Checks pass locally
  • I have removed any unnecessary logic
  • My code is well documented
  • I have signed my commits
  • My code follows the pattern of the application
  • I have self reviewed my code

Summary by CodeRabbit

  • New Features
    • Integrated S3-compatible cloud storage for file uploads
    • Implemented presigned upload flow for direct client-to-cloud transfers
    • Files now accessible via direct URLs for improved performance on downloads and previews

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 16, 2026

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 752c3fcb-d800-43ee-8d82-ef7eadc43d95

📥 Commits

Reviewing files that changed from the base of the PR and between 7ce20f5 and 5eb8c47.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (24)
  • platforms/esigner/api/package.json
  • platforms/esigner/api/src/controllers/FileController.ts
  • platforms/esigner/api/src/controllers/WebhookController.ts
  • platforms/esigner/api/src/database/entities/File.ts
  • platforms/esigner/api/src/database/migrations/1773657072411-addurl.ts
  • platforms/esigner/api/src/index.ts
  • platforms/esigner/api/src/services/FileService.ts
  • platforms/esigner/api/src/services/S3Service.ts
  • platforms/esigner/api/src/web3adapter/mappings/file.mapping.json
  • platforms/esigner/client/src/lib/stores/files.ts
  • platforms/esigner/client/src/routes/(protected)/files/[id]/+page.svelte
  • platforms/file-manager/api/package.json
  • platforms/file-manager/api/src/controllers/FileController.ts
  • platforms/file-manager/api/src/controllers/WebhookController.ts
  • platforms/file-manager/api/src/database/entities/File.ts
  • platforms/file-manager/api/src/database/migrations/1773657041144-addurl.ts
  • platforms/file-manager/api/src/index.ts
  • platforms/file-manager/api/src/services/FileService.ts
  • platforms/file-manager/api/src/services/S3Service.ts
  • platforms/file-manager/api/src/web3adapter/mappings/file.mapping.json
  • platforms/file-manager/client/src/lib/stores/files.ts
  • platforms/file-manager/client/src/routes/(protected)/files/+page.svelte
  • platforms/file-manager/client/src/routes/(protected)/files/[id]/+page.svelte
  • services/ontology/schemas/file.json

📝 Walkthrough

Walkthrough

This PR implements S3-compatible object storage integration for file uploads across both esigner and file-manager platforms. It introduces presigned upload endpoints, a new S3Service class for AWS SDK operations, modifies the File entity to add a url field and make data nullable, updates database schemas via migrations, and refactors client-side upload flows from server-side multipart to direct S3 uploads with presigned URLs.

Changes

Cohort / File(s) Summary
Package Dependencies
platforms/esigner/api/package.json, platforms/file-manager/api/package.json
Added AWS SDK dependencies (@aws-sdk/client-s3, @aws-sdk/s3-request-presigner) and repositioned signature-validator within the dependencies block.
S3 Service Integration
platforms/esigner/api/src/services/S3Service.ts, platforms/file-manager/api/src/services/S3Service.ts
New S3Service class encapsulating AWS SDK v3 operations with lazy initialization, presigned URL generation, bucket/region derivation, and methods for upload URLs, public URLs, object metadata, and streaming (identical implementations across both platforms).
File Controller Upload Endpoints
platforms/esigner/api/src/controllers/FileController.ts, platforms/file-manager/api/src/controllers/FileController.ts
Added presignUpload and confirmUpload methods to handle S3 presigned URL generation and upload confirmation; presignUpload generates uploadUrl and fileId, while confirmUpload validates inputs, fetches S3 object metadata (MD5 hash), and creates file records.
File Service S3 Integration
platforms/esigner/api/src/services/FileService.ts, platforms/file-manager/api/src/services/FileService.ts
Added s3Service instance, createFileWithUrl method to create file records with S3 URLs, and updated getDocumentsWithStatus/getFileMetadataById to include url field in response payloads.
Webhook File Handling
platforms/esigner/api/src/controllers/WebhookController.ts, platforms/file-manager/api/src/controllers/WebhookController.ts
Updated file creation and update paths to handle URL assignment from local.data.url and initialize file data directly with base64 decoding (when present) instead of temporary buffer construction.
File Entity & Schema
platforms/esigner/api/src/database/entities/File.ts, platforms/file-manager/api/src/database/entities/File.ts
Made data field nullable (Buffer | null) and added new optional url field (string | null) as text column with nullable: true.
Database Migrations
platforms/esigner/api/src/database/migrations/1773657072411-addurl.ts, platforms/file-manager/api/src/database/migrations/1773657041144-addurl.ts
Added TypeORM migrations that add url column to files table and make data column nullable, with reversible up/down flows for both platforms.
API Routes
platforms/esigner/api/src/index.ts, platforms/file-manager/api/src/index.ts
Added POST /api/files/presign and POST /api/files/confirm routes guarded by authGuard, enabling presigned upload and upload confirmation workflows.
Web3 Adapter Mappings
platforms/esigner/api/src/web3adapter/mappings/file.mapping.json, platforms/file-manager/api/src/web3adapter/mappings/file.mapping.json
Added "url": "url" field mapping in localToUniversalMap for File entities.
Client File Store
platforms/esigner/client/src/lib/stores/files.ts, platforms/file-manager/client/src/lib/stores/files.ts
Added optional url field to Document/File interface; refactored upload flow from multipart FormData to three-step presigned S3 workflow (presign → upload → confirm) with error handling for non-OK S3 responses.
Client Upload UI
platforms/esigner/client/src/routes/(protected)/files/[id]/+page.svelte, platforms/file-manager/client/src/routes/(protected)/files/+page.svelte
Updated file preview and download logic to prefer file.url when available, falling back to API-based blob fetching; refactored upload handler to use presigned URL workflow with XMLHttpRequest PUT and progress tracking.
Ontology Schema
services/ontology/schemas/file.json
Added url property (type: ["string", "null"], format: "uri") and updated data description to mark it as legacy.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Controller
    participant S3Service
    participant S3 as S3 Storage
    participant FileService
    participant Database

    rect rgba(100, 150, 255, 0.5)
    Note over Client,Database: Presign Upload Flow
    Client->>Controller: POST /api/files/presign<br/>(filename, mimeType, size)
    Controller->>S3Service: generateKey(userId, fileId, filename)
    S3Service-->>Controller: key
    Controller->>S3Service: generateUploadUrl(key, contentType)
    S3Service-->>Controller: presigned uploadUrl
    Controller-->>Client: { uploadUrl, key, fileId }
    end

    rect rgba(150, 200, 100, 0.5)
    Note over Client,S3: Direct S3 Upload
    Client->>S3: PUT uploadUrl<br/>(file data, Content-Type, x-amz-acl)
    S3-->>Client: 200 OK
    end

    rect rgba(200, 150, 100, 0.5)
    Note over Client,Database: Confirm Upload Flow
    Client->>Controller: POST /api/files/confirm<br/>(key, fileId, filename, mimeType, size)
    Controller->>S3Service: headObject(key)
    S3Service->>S3: HEAD object
    S3-->>S3Service: { contentLength, etag }
    S3Service-->>Controller: metadata with md5Hash
    Controller->>S3Service: getPublicUrl(key)
    S3Service-->>Controller: public url
    Controller->>FileService: createFileWithUrl(..., url)
    FileService->>Database: save File record
    Database-->>FileService: File entity
    FileService-->>Controller: File
    Controller-->>Client: { file metadata }
    end

    rect rgba(100, 200, 200, 0.5)
    Note over Client,S3: Download/Preview Flow
    alt file.url exists
    Client->>Client: use file.url<br/>(direct redirect or open)
    else file.url is null
    Client->>Controller: GET /api/files/[id]/download
    Controller->>FileService: getFileDataStream(fileId)
    FileService->>Database: fetch File
    Database-->>FileService: File record
    FileService-->>Controller: Readable stream
    Controller-->>Client: file data
    end
    end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • feat: per user limit #922: Modifies file upload and quota logic in FileController and FileService, touching the same upload-path handlers affected by presign/confirm endpoints.
  • fix: file manager limits #663: Updates FileController in file-manager API with upload-related handler and flow changes, directly overlapping with this PR's controller modifications.
  • feat:(file-manager) fix download speeds #756: Adds server-side file streaming and ZIP export features to file-manager backend (FileController and FileService), operating on the same file storage and retrieval domain.

Suggested reviewers

  • sosweetham
  • xPathin

Poem

🐰 A presigned dance on S3's stage,
No more uploads tied to server's cage,
Direct to storage, swift and clean,
With URLs stored—the finest scene,
File flows that flow, now cloud-aligned! 🌧️✨

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/object-store-for-files
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coodos coodos marked this pull request as ready for review March 16, 2026 11:45
@coodos coodos merged commit d2efd8b into main Mar 16, 2026
3 of 4 checks passed
@coodos coodos deleted the feat/object-store-for-files branch March 16, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant