s3stor_cpp: C++ Port of S3 Backup Utility

This is a C++ implementation/port of s3stor (my other project which was intially coded in golang) - a command-line tool for backing up files to S3-compatible storage (e.g., AWS S3, Wasabi) with block-based deduplication, point-in-time snapshots, and efficient file management. This is a POC/tester - you may want to use restic in production.

Features

• Block-Based Deduplication: Splits files into blocks, stores unique blocks by SHA-256 hash
• Point-in-Time Snapshots: Creates consistent backups with independent file maps
• Multi-Writer Safety: Uses S3-based locking to prevent conflicts
• S3 Compatibility: Works with AWS S3, Wasabi, and other S3-compatible providers
• Full Feature Parity: Implements all functions from the Go version

Building

Prerequisites

• Compiler: C++17 compatible compiler
  • Windows: Visual Studio 2017 or later (tested with VS 2022)
  • Linux: GCC 7+ or Clang 5+
  • macOS: Clang 5+ (Xcode 9.3+)
• Build System:
  • Windows: MSBuild (included with Visual Studio)
  • Linux/macOS: CMake 3.15 or higher
• Package Manager: vcpkg for dependency management

Dependencies (managed by vcpkg)

• AWS SDK for C++ (S3 and Transfer Manager components)
• nlohmann-json (JSON library)
• OpenSSL (for SHA-256 hashing)

Build Instructions

Windows (Visual Studio)

The project includes Visual Studio solution files for easy building on Windows.

Option 1: Using Visual Studio IDE (Recommended)

1. Install vcpkg (if not already installed):

   git clone https://github.com/Microsoft/vcpkg.git C:\vcpkg
   cd C:\vcpkg
   .\bootstrap-vcpkg.bat
   .\vcpkg integrate install

2. Install dependencies via vcpkg:

   .\vcpkg install aws-sdk-cpp[s3,transfer]:x64-windows
   .\vcpkg install nlohmann-json:x64-windows
   .\vcpkg install openssl:x64-windows

3. Open the solution:

   • Open s3stor_cpp.sln in Visual Studio
   • The solution contains two projects:
     • s3stor_lib - Static library
     • s3stor_test - Test executable

4. Configure project settings (if needed):

   • Right-click s3stor_test project → Properties
   • Ensure C++ Language Standard is set to ISO C++17 (/std:c++17)
   • Verify vcpkg integration is enabled in project properties

5. Build the solution:

   • Select configuration: Debug or Release
   • Select platform: x64 (recommended) or x86
   • Build → Build Solution (or press Ctrl+Shift+B)

6. Output files will be in:

   • x64\Debug\s3stor_lib.lib - Static library
   • x64\Debug\s3stor_test.exe - Test executable

Option 2: Using MSBuild Command Line

# From the project directory
MSBuild s3stor_cpp.sln /p:Configuration=Release /p:Platform=x64

Linux/macOS (CMake)

# Install vcpkg if not already installed
git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install

# Install dependencies
./vcpkg install aws-sdk-cpp[s3,transfer]
./vcpkg install nlohmann-json
./vcpkg install openssl

# Build the project
cd /path/to/s3stor_cpp
mkdir build && cd build
cmake .. -DCMAKE_TOOLCHAIN_FILE=/path/to/vcpkg/scripts/buildsystems/vcpkg.cmake
cmake --build .

# Output files in build/
# - libs3stor_lib.a
# - s3stor_test

Project Structure (Windows)

s3stor_cpp/
├── s3stor_cpp.sln              # Visual Studio solution
├── s3stor_lib.vcxproj          # Library project
├── s3stor_test.vcxproj         # Test executable project
├── vcpkg.json                  # vcpkg manifest (dependencies)
├── vcpkg-configuration.json    # vcpkg configuration
├── s3stor_lib.h                # Library header
├── s3stor_lib.cpp              # Library implementation
├── s3stor_test.cpp             # Test executable source
├── CMakeLists.txt              # CMake build file (Linux/macOS)
├── x64/                        # Build output (Windows)
│   ├── Debug/
│   └── Release/
└── vcpkg_installed/            # Installed packages (auto-generated)

Configuration

Configure using environment variables (same as Go version):

export S3_PROVIDER=wasabi
export S3_BUCKET=your-bucket-name
export S3_REGION=us-east-1
export S3_ENDPOINT=https://s3.us-east-1.wasabisys.com
export AWS_ACCESS_KEY_ID=your-wasabi-access-key
export AWS_SECRET_ACCESS_KEY=your-wasabi-secret-key

On Windows run below comman in Powershell :

# Set S3 provider
[Environment]::SetEnvironmentVariable("S3_PROVIDER", "<S3_PROVIDER_VALUE>", "User")

# Set S3 bucket
[Environment]::SetEnvironmentVariable("S3_BUCKET", "<S3_BUCKET_NAME>", "User")

# Set S3 region - if not set then you'll performance issues when initializing AWS/S3 SDK client
# it will be slow / will timeout before finishing initialization 
[Environment]::SetEnvironmentVariable("S3_REGION", "<S3_REGION>", "User")

# Set S3 endpoint
[Environment]::SetEnvironmentVariable("S3_ENDPOINT", "<S3_ENDPOINT_URL>", "User")

# Set AWS secret access key
[Environment]::SetEnvironmentVariable("AWS_SECRET_ACCESS_KEY", "<AWS_SECRET_ACCESS_KEY>", "User")

# Set AWS access key ID
[Environment]::SetEnvironmentVariable("AWS_ACCESS_KEY_ID", "<AWS_ACCESS_KEY_ID>", "User")

# Set AWS default region - if not set then you'll performance issues when initializing AWS/S3 SDK client
# it will be slow / will timeout before finishing initialization  
[Environment]::SetEnvironmentVariable("AWS_DEFAULT_REGION", "<AWS_DEFAULT_REGION>", "User")

Usage

The C++ version provides the same command-line interface as the Go version.

Windows

Run from the build output directory:

# Navigate to output directory
cd x64\Debug  # or x64\Release

# Sync a file to S3
.\s3stor_test.exe sync ..\..\test_out\file1.txt

# Create a snapshot
.\s3stor_test.exe snapshot ..\..\test_out sn001 file1.txt

# List files in global catalog
.\s3stor_test.exe ls

# List files in snapshot
.\s3stor_test.exe ls sn001

# Restore a file from snapshot
.\s3stor_test.exe get sn001 file1.txt .\restore

# Display file block map
.\s3stor_test.exe map file1.txt

# Delete a file from global catalog
.\s3stor_test.exe delete file1.txt

# Delete a snapshot
.\s3stor_test.exe delete-snapshot sn001

# Cleanup unreferenced blocks
.\s3stor_test.exe cleanup-blocks

Linux/macOS

Run from the build directory:

# Sync a file to S3
./s3stor_test sync test_out/file1.txt

# Create a snapshot
./s3stor_test snapshot test_out sn001 file1.txt

# List files in global catalog
./s3stor_test ls

# List files in snapshot
./s3stor_test ls sn001

# Restore a file from snapshot
./s3stor_test get sn001 file1.txt ./restore

# Display file block map
./s3stor_test map file1.txt

# Delete a file from global catalog
./s3stor_test delete file1.txt

# Delete a snapshot
./s3stor_test delete-snapshot sn001

# Cleanup unreferenced blocks
./s3stor_test cleanup-blocks

Library Usage

You can also use s3stor as a library in your C++ projects:

#include "s3stor_lib.h"

int main() {
    // Setup configuration
    s3stor::S3ProviderConfig config;
    config.bucket_name = "my-bucket";
    config.region = "us-east-1";
    config.access_key_id = "...";
    config.secret_access_key = "...";
    
    // Initialize AWS SDK
    Aws::SDKOptions options;
    Aws::InitAPI(options);
    
    {
        // Create S3Stor instance
        s3stor::S3Stor s3stor(config);
        
        // Sync a file
        s3stor.sync_file("/path/to/file.txt", "/path/to");
        
        // Create snapshot
        std::vector<std::string> files = {"file.txt"};
        s3stor.create_snapshot("/path/to", "snapshot-001", "my-host", files);
        
        // List files
        s3stor.list_files();
    }
    
    Aws::ShutdownAPI(options);
    return 0;
}

Project Structure

Windows Visual Studio Project

• s3stor_cpp.sln - Visual Studio solution file
• s3stor_lib.vcxproj - Static library project
• s3stor_test.vcxproj - Test executable project
• vcpkg.json - vcpkg dependency manifest
• vcpkg-configuration.json - vcpkg configuration

Source Files

• s3stor_lib.h - Library header with class and structure definitions
• s3stor_lib.cpp - Library implementation (~31KB, all Go functions ported)
• s3stor_test.cpp - Test executable with command-line interface

Build System

• CMakeLists.txt - CMake build configuration (Linux/macOS)
• *.vcxproj - Visual Studio project files (Windows)

Documentation

• README.md - This file
• BUILD_FIX_INSTRUCTIONS.md - Windows build troubleshooting
• CROSS_PLATFORM_CHANGES.md - Cross-platform compatibility notes
• CPP_PORT_SUMMARY.md - Detailed porting summary
• README_CPP.md - Additional C++ documentation

Output Directories

• Windows: x64/Debug/ or x64/Release/
• Linux/macOS: build/

Implemented Functions

All functions from the Go version are implemented:

Core Operations

• sync_file() - Upload a file with deduplication
• sync_dir() - Upload a directory recursively
• list_files() - List files in global catalog
• list_snapshot_files() - List files in a snapshot
• get_file() - Restore a file from catalog or snapshot
• get_file_map() - Display block mappings for a file
• create_snapshot() - Create a point-in-time snapshot
• delete_snapshot() - Delete a snapshot
• cleanup_blocks() - Remove unreferenced blocks
• delete_file() - Remove file from global catalog

Helper Functions

• hash_block() - Compute SHA-256 hash of a block
• upload_block() - Upload a block to S3
• put_object() - Upload an object to S3
• get_object() - Download an object from S3
• acquire_lock() - Acquire distributed lock
• release_lock() - Release distributed lock
• update_catalog() - Update global catalog
• And more...

Platform-Specific Notes

Windows

Cross-Platform Compatibility:

• The code uses #ifdef _WIN32 to handle Windows-specific code
• Winsock2 is automatically initialized for hostname retrieval
• Thread-safe time functions (gmtime_s on Windows, gmtime_r on Unix)
• Windows macros (GetObject) are undefined to prevent conflicts with AWS SDK

Build Requirements:

• Visual Studio 2017 or later (tested with VS 2022)
• Platform Toolset: v143 or v142
• C++ Language Standard: ISO C++17 (/std:c++17)
• Windows SDK 10.0 or later

Project Settings:

• Character Set: Unicode
• Configuration Type: Application (s3stor_test), Static Library (s3stor_lib)
• vcpkg Integration: Enabled via VcpkgEnableManifest
• Preprocessor: _CRT_SECURE_NO_WARNINGS to suppress safe CRT warnings

Known Issues:

• If you see error C1083: Cannot open include file: 'nlohmann/json.hpp', ensure vcpkg is properly integrated
• If you see STL4038: <filesystem> requires C++17, update project to use C++17 standard
• See BUILD_FIX_INSTRUCTIONS.md for detailed troubleshooting

Linux

Build Requirements:

• GCC 7+ or Clang 5+
• CMake 3.15+
• Standard POSIX headers (unistd.h, etc.)

Platform Features:

• Direct gethostname() call without Winsock initialization
• Uses gmtime_r() for thread-safe time conversion
• Standard filesystem paths with forward slashes

macOS

Build Requirements:

• Xcode 9.3+ (includes Clang 5+)
• CMake 3.15+
• Similar to Linux build process

Platform Features:

• Same POSIX-compliant code as Linux
• Homebrew can be used instead of vcpkg if preferred

Differences from Go Version

The C++ version maintains full functional parity with minor implementation differences:

1. Uses AWS SDK for C++ instead of Go's AWS SDK v2
2. Uses nlohmann-json for JSON handling instead of Go's encoding/json
3. Uses OpenSSL for SHA-256 instead of Go's crypto/sha256
4. Uses C++17 filesystem library instead of Go's filepath
5. Error handling uses exceptions and return values instead of Go's error values

License

MIT License - Same as the original Go version

© 2025 Alexey Shvechkov. All rights reserved.