Skip to content

Use aws s3 cp instead of wget for benchmark index downloads #927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ofiryanai wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Use aws s3 cp instead of wget for benchmark index downloads #927

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dc8b37a
Use aws s3 cp instead of wget for benchmark index downloads
ofiryanai Apr 5, 2026
0e57c58
Fix: install AWS CLI v2 standalone to avoid Python conflicts
ofiryanai Apr 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion .github/workflows/benchmark-runner.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,21 @@ jobs:
- name: Install benchmark dependencies
run: |
sudo .install/install_script.sh
sudo apt install python3-pip -y
sudo apt install python3-pip unzip -y
pip3 install --upgrade pip PyYAML setuptools redisbench-admin
pip3 install -r requirements.txt
- name: Install AWS CLI v2
run: |
curl "https://awscli.amazonaws.com/awscli-exe-linux-$(uname -m).zip" -o /tmp/awscliv2.zip
unzip -q /tmp/awscliv2.zip -d /tmp
sudo /tmp/aws/install
aws --version
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION_BENCHMARK }}
- name: Download pre-generated indices
timeout-minutes: 20
run: ./tests/benchmark/bm_files.sh ${{ inputs.setup }}
Expand Down
23 changes: 20 additions & 3 deletions tests/benchmark/bm_files.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,25 @@
BM_TYPE=$1
set -euo pipefail

BM_TYPE=${1:-}
alg="hnsw"

S3_BUCKET="dev.cto.redis"
S3_PREFIX="VectorSimilarity"
DEST_DIR="tests/benchmark/data"

# Download a file from S3 given its full HTTPS URL.
# Extracts the object key from the URL and uses aws s3 cp.
download_s3() {
local url="$1"
local filename
filename=$(basename "$url")
aws s3 cp "s3://${S3_BUCKET}/${S3_PREFIX}/${filename}" "${DEST_DIR}/${filename}"
}
export -f download_s3
export S3_BUCKET S3_PREFIX DEST_DIR

if [ -z "$BM_TYPE" ] || [ "$BM_TYPE" = "benchmarks-all" ]; then
cat tests/benchmark/data/hnsw_indices/*.txt tests/benchmark/data/svs_indices/*.txt | xargs -n 1 -P 0 wget --no-check-certificate -P tests/benchmark/data
cat tests/benchmark/data/hnsw_indices/*.txt tests/benchmark/data/svs_indices/*.txt | grep -v '^$' | sort -u | xargs -n 1 -P 0 -I {} bash -c 'download_s3 "$@"' _ {}
exit 0
elif [ "$BM_TYPE" = "benchmarks-default" ] \
|| [ "$BM_TYPE" = "bm-basics-fp32-single" ] \
Expand Down Expand Up @@ -57,4 +74,4 @@ else
exit 0
fi

cat tests/benchmark/data/${alg}_indices/${alg}_indices_$file_name.txt | xargs -n 1 -P 0 wget --no-check-certificate -P tests/benchmark/data
cat tests/benchmark/data/${alg}_indices/${alg}_indices_$file_name.txt | grep -v '^$' | xargs -n 1 -P 0 -I {} bash -c 'download_s3 "$@"' _ {}
Loading