Skip to content

feat: add endpoint_url support to s3_sync for MinIO compatibility#589

Open
koriyoshi2041 wants to merge 1 commit intoOpenPipe:mainfrom
koriyoshi2041:feat/s3-endpoint-url-param
Open

feat: add endpoint_url support to s3_sync for MinIO compatibility#589
koriyoshi2041 wants to merge 1 commit intoOpenPipe:mainfrom
koriyoshi2041:feat/s3-endpoint-url-param

Conversation

@koriyoshi2041
Copy link

Summary

Addresses #435.

Add an optional endpoint_url parameter to s3_sync() and ensure_bucket_exists() in art.utils.s3. When not provided, both functions fall back to the AWS_ENDPOINT_URL environment variable. This enables S3-compatible services like MinIO without requiring callers to manage the env var themselves.

Changes

  • s3_sync(): new endpoint_url keyword argument, passed as --endpoint-url to the AWS CLI
  • ensure_bucket_exists(): same parameter, applied to both the head-bucket check and create-bucket commands
  • Both functions fall back to os.environ.get("AWS_ENDPOINT_URL") when the parameter is None
  • Also documented the previously undocumented delete parameter in s3_sync()

Usage

# Option 1: Set the env var (works for all code paths)
os.environ["AWS_ENDPOINT_URL"] = "http://minio:9000"

# Option 2: Pass explicitly
await s3_sync(source, dest, endpoint_url="http://minio:9000")

Fully backward-compatible — the parameter defaults to None and all existing callers are unaffected.

Test plan

  • Verify s3_sync() includes --endpoint-url when parameter is set
  • Verify s3_sync() includes --endpoint-url when AWS_ENDPOINT_URL env var is set
  • Verify s3_sync() omits --endpoint-url when neither is set
  • Verify ensure_bucket_exists() passes endpoint to all three CLI commands
  • uv run prek run --all-files passes

…enPipe#435)

Add an optional endpoint_url parameter to s3_sync and
ensure_bucket_exists. When not provided, falls back to the
AWS_ENDPOINT_URL environment variable. This enables S3-compatible
services like MinIO without requiring callers to manage the env var
themselves.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant