Skip to content

Commit 3bcdf49

Browse files
authored
add ftp support & upload for url/ftp/s3. Reorg tests (#1260)
* add ftp support & upload for url/ftp/s3. Reorg tests * c2d output upload (#1263) * c2d output upload
1 parent ba510a3 commit 3bcdf49

32 files changed

Lines changed: 2182 additions & 1215 deletions

docs/Storage.md

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Storage Types
22

3-
Ocean Node supports four storage backends for assets (e.g. algorithm or data files). Each type is identified by a `type` field on the file object and has its own shape and validation rules.
3+
Ocean Node supports five storage backends for assets (e.g. algorithm or data files). Each type is identified by a `type` field on the file object and has its own shape and validation rules.
44

55
## Supported types
66

@@ -10,6 +10,7 @@ Ocean Node supports four storage backends for assets (e.g. algorithm or data fil
1010
| **IPFS** | `ipfs` | File identified by IPFS CID |
1111
| **Arweave**| `arweave` | File identified by Arweave transaction ID |
1212
| **S3** | `s3` | File in S3-compatible storage (AWS, Ceph, MinIO, etc.) |
13+
| **FTP** | `ftp` | File served via FTP or FTPS |
1314

1415
All file objects can optionally include encryption metadata: `encryptedBy` and `encryptMethod` (e.g. `AES`, `ECIES`).
1516

@@ -163,11 +164,55 @@ Files are stored in S3-compatible object storage. The node uses the AWS SDK and
163164

164165
---
165166

167+
## FTP storage
168+
169+
Files are fetched or uploaded via FTP or FTPS. The node uses a single `url` field containing the full FTP(S) URL (including optional credentials). Functionality mirrors URL storage: stream download, file metadata (size; content-type is `application/octet-stream`), and upload via STOR.
170+
171+
### File object shape
172+
173+
```json
174+
{
175+
"type": "ftp",
176+
"url": "ftp://user:password@ftp.example.com:21/path/to/file.zip"
177+
}
178+
```
179+
180+
For FTPS (TLS):
181+
182+
```json
183+
{
184+
"type": "ftp",
185+
"url": "ftps://user:password@secure.example.com:990/pub/data.csv"
186+
}
187+
```
188+
189+
| Field | Required | Description |
190+
| ------ | -------- | ----------- |
191+
| `type` | Yes | Must be `"ftp"` |
192+
| `url` | Yes | Full FTP or FTPS URL. Supports `ftp://` and `ftps://`. May include credentials as `ftp://user:password@host:port/path`. Default port is 21 for FTP and 990 for FTPS. |
193+
194+
### Validation
195+
196+
- `url` must be present.
197+
- URL must use protocol `ftp://` or `ftps://`.
198+
- If the node config defines `unsafeURLs` (list of regex patterns), any URL matching a pattern is rejected.
199+
200+
### Node configuration
201+
202+
- Optional: `unsafeURLs` – array of regex strings; URLs matching any of them are considered unsafe and rejected (same as URL storage).
203+
204+
### Upload
205+
206+
FTPStorage supports `upload(filename, stream)`. If the file object’s `url` ends with `/`, the filename is appended to form the remote path; otherwise the URL is used as the full target path. Uses FTP STOR command.
207+
208+
---
209+
166210
## Summary
167211

168212
- **URL**: flexible HTTP(S) endpoints; optional custom headers and `unsafeURLs` filtering.
169213
- **IPFS**: CID-based; requires `ipfsGateway` in config.
170214
- **Arweave**: transaction-ID-based; requires `arweaveGateway` in config.
171215
- **S3**: S3-compatible object storage (AWS, Ceph, MinIO, etc.); credentials and endpoint in the file object; `region` optional (defaults to `us-east-1`).
216+
- **FTP**: FTP/FTPS URLs; stream download, metadata (size), and upload via STOR; optional `unsafeURLs` filtering.
172217

173218
The storage implementation lives under `src/components/storage/`. The node selects the backend from the file object’s `type` (case-insensitive) and validates the shape and config before fetching or streaming the file.

0 commit comments

Comments
 (0)