AwsChunkedStream: unbounded std::stringstream memory growth causes OOM

[Shekaboy]

Describe the bug

AwsChunkedStream::BufferedRead() uses a std::stringstream (m_chunkingStream) as an intermediate buffer. On every call, it reads a 64KB chunk from the source stream and appends the chunked-encoded data via writeChunk(). The std::stringstream's underlying std::string never reclaims memory behind the read pointer — clear() only resets error/EOF flags, not the buffer.

For a single PutObject of size N, the stringstream's backing string grows monotonically to ~N bytes. At ~4GB, std::string::reserve() attempts to double to 8GB+1 (0x200000001), which exceeds the maximum supported allocation size (0x200000000), causing SIGABRT / std::bad_alloc.

Root cause location: AwsChunkedStream.h, BufferedRead() and writeChunk()

The clear() calls on lines 82, 93 only reset stream state flags (eofbit/failbit). They do not call str("") to release the underlying buffer. The pptr (write pointer) advances on every writeChunk() but consumed data is never freed.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Memory usage during chunked upload should be O(1) — bounded by the chunk size (~65KB), not the total upload size.

Current Behavior

Memory grows linearly with bytes uploaded. A 5GB single PutObject consumes ~10GB of RAM for the stringstream buffer alone. At ~4GB the allocation fails:

==12345==ERROR: AddressSanitizer: requested allocation size 0x200000001 exceeds maximum supported size of 0x200000000
#0 in operator new(unsigned long)
#1 in std::__cxx11::basic_string::reserve(unsigned long)
#2 in std::__cxx11::basic_string::_M_replace_aux(unsigned long, unsigned long, unsigned long, char)
#3 in Aws::Utils::Stream::AwsChunkedStream<65536ul>::writeChunk(unsigned long)
#4 in Aws::Utils::Stream::AwsChunkedStream<65536ul>::BufferedRead(char*, unsigned long)
#5 in Aws::Http::CurlHttpClient::MakeRequest() [ReadBody]

Reproduction Steps

Use (CURL-based, not CRT) with SDK >= v1.11.486
Issue a single PutObject for an object > ~2GB
Set a checksum algorithm (or rely on the new default CRC trailing checksum from v1.11.486+)
Observe monotonic memory growth proportional to bytes uploaded

Aws::S3::Model::PutObjectRequest request;
request.SetBucket("my-bucket");
request.SetKey("large-object");
request.SetBody(largeStream); // > 2GB, non-seekable
request.SetChecksumAlgorithm(Aws::S3::Model::ChecksumAlgorithm::CRC32C);
// OR just rely on the SDK default (crc64nvme since v1.11.486)

auto outcome = s3Client.PutObject(request);
// → OOM crash during upload

Possible Solution

Before writing a new chunk when the stream is fully drained, reset the stringstream buffer:

size_t BufferedRead(char *dst, size_t amountToRead) {
assert(dst != nullptr);

bool chunkingStreamEmpty =
    (m_chunkingStream->peek() == EOF || m_chunkingStream->eof()) &&
    !m_chunkingStream->bad();

if (chunkingStreamEmpty && m_stream->good()) {
  // CRITICAL: Reset the stringstream buffer to free consumed memory.
  // clear() only resets flags; str("") actually releases the string.
  m_chunkingStream->str("");
  m_chunkingStream->clear();

  m_stream->read(m_data.GetUnderlyingData(), DataBufferSize);
  size_t bytesRead = static_cast<size_t>(m_stream->gcount());
  writeChunk(bytesRead);

  if ((m_stream->peek() == EOF || m_stream->eof()) && !m_stream->bad()) {
    writeTrailerToUnderlyingStream();
  }
}
// ... rest unchanged

This bounds memory at ~130KB (one 65KB read buffer + one 65KB+overhead stringstream buffer).

Additional Information/Context

Pre-v1.11.486: This code path was unreachable because MD5 checksums were computed in the header by default. The aws-chunked encoding was never activated for PutObject.

v1.11.486+ ([#3253]): Default S3 integrity changed to CRC trailing checksums, activating AwsChunkedStream for all PutObject calls.

PR [#3635]
This issue specifically affects users doing single large PutObject (not multipart) via the CURL-based S3Client. S3CrtClient and TransferManager users are unaffected because they split into small parts.

AWS CPP SDK version used

1.11.638

Compiler and Version used

GCC 11

Operating System and version

Linux

[sbiscigl]

Hey @Shekaboy first and foremost thanks for the issue, this is something that is really important to us that we paid attention to while making this change, if there is a issue we absolutely want to treat it with the highest priority. When we try to replicate this issue however we cannot. given the replication application:

#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/PutObjectRequest.h>

#include <fstream>

using namespace Aws;
using namespace Aws::Client;
using namespace Aws::S3;
using namespace Aws::S3::Model;

namespace {
const char* TEST_LOG_TAG = "test-application";

class sdk_context {
 public:
  sdk_context(SDKOptions&& options) : options_{options} { InitAPI(options_); }

  ~sdk_context() { ShutdownAPI(options_); }

 private:
  SDKOptions options_;
};
}  // namespace

int main() {
  SDKOptions options;
  options.loggingOptions.logLevel = Utils::Logging::LogLevel::Debug;
  sdk_context context{std::move(options)};

  S3Client client{};
  auto request =
      PutObjectRequest{}.WithBucket("your-bucket-name").WithKey("test-key");

  request.SetBody(Aws::MakeShared<FStream>(
      TEST_LOG_TAG, "/path/to/2gb//ile",
      std::ios_base::in | std::ios_base::binary));

  const auto response = client.PutObject(request);
  if (!response.IsSuccess()) {
    std::cout << response.GetError().GetMessage() << std::endl;
  }

  return 0;
}

when run with the memory montioring script

#!/bin/bash
"$@" &
PID=$!
echo "timestamp,rss_mb,vsz_mb"
while kill -0 $PID 2>/dev/null; do
  ps -o rss=,vsz= -p $PID | awk -v ts="$(date +%s)" '{printf "%s,%.2f,%.2f\n", ts, $1/1024, $2/1024}'
  sleep 0.5
done

where
rss_mb - The amount of physical RAM (in memory) currently being used by the process. This is the actual memory occupied in RAM, excluding swapped-out memory.
vsz_mb - The total amount of virtual memory allocated to the process. This includes all memory the process can access: physical RAM, swapped memory, shared libraries, and memory-mapped files.
It's typically larger than RSS.

i see the output of

$ ./monitor_memory.sh ~/path/to/example/app
timestamp,rss_mb,vsz_mb
1771950498,1.23,400481.28
1771950499,14.95,401439.84
1771950499,14.95,401439.84
1771950500,15.02,401439.84
1771950500,15.31,401455.84
1771950501,15.33,401455.84
1771950501,15.47,401471.84
1771950502,15.53,401471.84
1771950502,15.53,401471.84
1771950503,15.53,401471.84
1771950504,15.53,401471.84
1771950504,15.52,401471.30
1771950505,15.52,401471.30
1771950505,15.52,401471.30
1771950506,15.52,401471.30
1771950506,15.52,401471.30
1771950507,15.52,401471.30
1771950507,15.52,401471.30
1771950508,15.52,401471.30
1771950509,15.52,401471.30
1771950509,15.52,401471.30
1771950510,15.52,401471.30
1771950510,15.52,401471.30
1771950511,15.55,401471.30
1771950511,15.55,401471.30
1771950512,15.55,401471.30
1771950512,15.55,401471.30
1771950513,15.55,401471.30
1771950513,15.55,401471.30
1771950514,15.55,401471.30
1771950515,15.55,401471.30
...
1771950571,15.55,401471.30
1771950571,15.55,401471.30
1771950572,15.55,401471.30
1771950573,15.55,401471.30
1771950573,15.55,401471.30
1771950574,15.55,401471.30
1771950574,15.55,401471.30
1771950575,15.55,401471.30
1771950575,15.55,401471.30
1771950576,15.55,401471.30
1771950577,15.55,401471.30
1771950577,15.55,401471.30
1771950578,15.55,401471.30
1771950578,15.55,401471.30
1771950579,15.55,401471.30
1771950579,15.55,401471.30
1771950580,15.55,401471.30
1771950580,15.55,401471.30
1771950581,15.55,401471.30
1771950581,15.55,401471.30
1771950582,15.55,401471.30
1771950583,15.55,401471.30
1771950583,15.55,401471.30
1771950584,15.55,401471.30
1771950584,15.55,401471.30

which doesnt show unbounded growth at runtime. and for a ~2GB file i never see memory usage go above 15.55 MiB. using this script and this application do you see something different on your end? what environment are you running in?

[Shekaboy]

Hi @sbiscigl

Thank you for the quick response. The reason your reproduction doesn't show the growth is that FStream is seekable, which allows the SDK to bypass AwsChunkedStream in many configurations. To reproduce this, you must use a non-seekable (read-once) stream and a trailing checksum, which forces the SDK to use the chunked encoding wrapper.

I have used a non seekable stream( with a trailing checksum (Aws::S3::Model::ChecksumAlgorithm::CRC32C), using Aws::Client::RequestChecksumCalculation::WHEN_REQUIRED, along with using a curl client

Combination of these two forces the sdk to calculate a trailing checksum, which triggers buffered read and hence the OOM.

This is the entire stack trace of the memory leak being detected - This clearly shows the code path where the leak is happening

2026-02-23T06:15:54.590874517Z stderr F ==16==ERROR: LeakSanitizer: requested allocation size 0x200000001 exceeds maximum supported size of 0x200000000 2026-02-23T06:15:54.590891098Z stderr F ==16==Using libbacktrace symbolizer. 2026-02-23T06:16:01.802738945Z stderr F #0 0x7fd1d004cf05 in operator new(unsigned long) ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:248 2026-02-23T06:16:01.802762779Z stderr F #1 0x7fd1c6d8fd64 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) (/lib/x86_64-linux-gnu/libstdc++.so.6+0x151d64) (BuildId: 133b71e0013695cc7832680a74edb51008c4fc4c) 2026-02-23T06:16:01.802766626Z stderr F #2 0x7fd1c6d84674 in std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::overflow(int) (/lib/x86_64-linux-gnu/libstdc++.so.6+0x146674) (BuildId: 133b71e0013695cc7832680a74edb51008c4fc4c) 2026-02-23T06:16:01.802769542Z stderr F #3 0x7fd1c6d8e099 in std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) (/lib/x86_64-linux-gnu/libstdc++.so.6+0x150099) (BuildId: 133b71e0013695cc7832680a74edb51008c4fc4c) 2026-02-23T06:16:01.802772577Z stderr F #4 0x7fd1c6d7e3f5 in std::ostream::write(char const*, long) (/lib/x86_64-linux-gnu/libstdc++.so.6+0x1403f5) (BuildId: 133b71e0013695cc7832680a74edb51008c4fc4c) 2026-02-23T06:16:01.802775212Z stderr F #5 0x7fd1c73113f7 in Aws::Utils::Stream::AwsChunkedStream<65536ul>::writeChunk(unsigned long) (/opt/aws-sdk-cpp/lib/libaws-cpp-sdk-core.so+0x3a23f7) (BuildId: 1bf31dfcecd89182421f3094132cd1be4e330565) 2026-02-23T06:16:01.802778047Z stderr F #6 0x7fd1c7310db2 in Aws::Utils::Stream::AwsChunkedStream<65536ul>::BufferedRead(char*, unsigned long) /root/rbriglio/various-repos/hexos-aws-sdk-cpp/build/aws-sdk-cpp/src/aws-cpp-sdk-core/include/aws/core/utils/stream/AwsChunkedStream.h:44 2026-02-23T06:16:01.802783528Z stderr F #7 0x7fd1c730c9fe in ReadBody /root/rbriglio/various-repos/hexos-aws-sdk-cpp/build/aws-sdk-cpp/src/aws-cpp-sdk-core/source/http/curl/CurlHttpClient.cpp:338 2026-02-23T06:16:01.802786333Z stderr F #8 0x7fd1c730cb26 in ReadBodyFunc /root/rbriglio/various-repos/hexos-aws-sdk-cpp/build/aws-sdk-cpp/src/aws-cpp-sdk-core/source/http/curl/CurlHttpClient.cpp:366 2026-02-23T06:16:01.802789388Z stderr F #9 0x7fd1c6734ec8 (/lib/x86_64-linux-gnu/libcurl.so.4+0x71ec8) (BuildId: 12a2b20fb5a1c5b152670354e16dcc056969bf73) 2026-02-23T06:16:01.802792043Z stderr F #10 0x7fd1c6718944 (/lib/x86_64-linux-gnu/libcurl.so.4+0x55944) (BuildId: 12a2b20fb5a1c5b152670354e16dcc056969bf73) 2026-02-23T06:16:01.802794628Z stderr F #11 0x7fd1c6718e11 (/lib/x86_64-linux-gnu/libcurl.so.4+0x55e11) (BuildId: 12a2b20fb5a1c5b152670354e16dcc056969bf73) 2026-02-23T06:16:01.802797323Z stderr F #12 0x7fd1c67368d0 (/lib/x86_64-linux-gnu/libcurl.so.4+0x738d0) (BuildId: 12a2b20fb5a1c5b152670354e16dcc056969bf73) 2026-02-23T06:16:01.802799908Z stderr F #13 0x7fd1c673255c (/lib/x86_64-linux-gnu/libcurl.so.4+0x6f55c) (BuildId: 12a2b20fb5a1c5b152670354e16dcc056969bf73) 2026-02-23T06:16:01.802802473Z stderr F #14 0x7fd1c66daeec (/lib/x86_64-linux-gnu/libcurl.so.4+0x17eec) (BuildId: 12a2b20fb5a1c5b152670354e16dcc056969bf73) 2026-02-23T06:16:01.802819474Z stderr F #15 0x7fd1c67329cb (/lib/x86_64-linux-gnu/libcurl.so.4+0x6f9cb) (BuildId: 12a2b20fb5a1c5b152670354e16dcc056969bf73) 2026-02-23T06:16:01.80282247Z stderr F #16 0x7fd1c6747013 (/lib/x86_64-linux-gnu/libcurl.so.4+0x84013) (BuildId: 12a2b20fb5a1c5b152670354e16dcc056969bf73) 2026-02-23T06:16:01.802825486Z stderr F #17 0x7fd1c6729cb0 (/lib/x86_64-linux-gnu/libcurl.so.4+0x66cb0) (BuildId: 12a2b20fb5a1c5b152670354e16dcc056969bf73) 2026-02-23T06:16:01.802828361Z stderr F #18 0x7fd1c672b619 in curl_multi_perform (/lib/x86_64-linux-gnu/libcurl.so.4+0x68619) (BuildId: 12a2b20fb5a1c5b152670354e16dcc056969bf73) 2026-02-23T06:16:01.802830916Z stderr F #19 0x7fd1c66f797a (/lib/x86_64-linux-gnu/libcurl.so.4+0x3497a) (BuildId: 12a2b20fb5a1c5b152670354e16dcc056969bf73) 2026-02-23T06:16:01.802833831Z stderr F #20 0x7fd1c730f4a8 in Aws::Http::CurlHttpClient::MakeRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const /root/rbriglio/various-repos/hexos-aws-sdk-cpp/build/aws-sdk-cpp/src/aws-cpp-sdk-core/source/http/curl/CurlHttpClient.cpp:868 2026-02-23T06:16:01.802836596Z stderr F #21 0x7fd1c72642fc in operator() /root/rbriglio/various-repos/hexos-aws-sdk-cpp/build/aws-sdk-cpp/src/aws-cpp-sdk-core/source/client/AWSClient.cpp:589 2026-02-23T06:16:01.802839441Z stderr F #22 0x7fd1c7278f13 in __invoke_impl<std::shared_ptr<Aws::Http::HttpResponse>, Aws::Client::AWSClient::AttemptOneRequest(const std::shared_ptr<Aws::Http::HttpRequest>&, const Aws::AmazonWebServiceRequest&, char const*, char const*, char const*) const::<lambda()>&> /usr/include/c++/14/bits/invoke.h:61 2026-02-23T06:16:01.802873785Z stderr F #23 0x7fd1c7275a80 in __invoke_r<std::shared_ptr<Aws::Http::HttpResponse>, Aws::Client::AWSClient::AttemptOneRequest(const std::shared_ptr<Aws::Http::HttpRequest>&, const Aws::AmazonWebServiceRequest&, char const*, char const*, char const*) const::<lambda()>&> /usr/include/c++/14/bits/invoke.h:139 2026-02-23T06:16:01.802884876Z stderr F #24 0x7fd1c7271d03 in _M_invoke /usr/include/c++/14/bits/std_function.h:291 2026-02-23T06:16:01.802889304Z stderr F #25 0x7fd1c72b6228 in std::function<std::shared_ptr<Aws::Http::HttpResponse> ()>::operator()() const (/opt/aws-sdk-cpp/lib/libaws-cpp-sdk-core.so+0x347228) (BuildId: 1bf31dfcecd89182421f3094132cd1be4e330565) 2026-02-23T06:16:01.802893762Z stderr F #26 0x7fd1c72a001d in std::shared_ptr<Aws::Http::HttpResponse> smithy::components::tracing::TracingUtils::MakeCallWithTiming<std::shared_ptr<Aws::Http::HttpResponse> >(std::function<std::shared_ptr<Aws::Http::HttpResponse> ()>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, smithy::components::tracing::Meter const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/opt/aws-sdk-cpp/lib/libaws-cpp-sdk-core.so+0x33101d) (BuildId: 1bf31dfcecd89182421f3094132cd1be4e330565) 2026-02-23T06:16:01.802897068Z stderr F #27 0x7fd1c726508e in Aws::Client::AWSClient::AttemptOneRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const /root/rbriglio/various-repos/hexos-aws-sdk-cpp/build/aws-sdk-cpp/src/aws-cpp-sdk-core/source/client/AWSClient.cpp:593 2026-02-23T06:16:01.802899563Z stderr F 2026-02-23T06:16:01.802920302Z stderr F ==16==HINT: if you don't care about these errors you may set allocator_may_return_null=1 2026-02-23T06:16:01.802934929Z stderr F SUMMARY: LeakSanitizer: allocation-size-too-big ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:248 in operator new(unsigned long) 2026-02-23T06:16:01.802938075Z stderr F ==16==ERROR: LeakSanitizer: requested allocation size 0x200000001 exceeds maximum supported size of 0x200000000

[sbiscigl]

The reason your reproduction doesn't show the growth is that FStream is seekable, which allows the SDK to bypass AwsChunkedStream in many configurations.

Which configuration does the reproduction satisfy to bypass AwsChunkedStream? The only configuration I know that allows you to bypass AwsChunkedStream is if you provide a pre-calculated checksum in the request. The above reproduction does use AwsChunkedStream if you look at the logs from the reproduction you see:

[DEBUG] 2026-02-24 16:52:03.159 CURL [0x20ab4a240] (HeaderOut) PUT /test-key-big HTTP/1.1
Host: ...
Accept: */*
amz-sdk-invocation-id: ...
amz-sdk-request: attempt=1
authorization: ...
content-encoding: aws-chunked
content-type: binary/octet-stream
transfer-encoding: chunked
user-agent: ...
x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date: ...
x-amz-decoded-content-length: 5368709120
x-amz-security-token: ...
x-amz-trailer: x-amz-checksum-crc64nvme
Expect: 100-continue

To reproduce this, you must use a non-seekable (read-once) stream and a trailing checksum, which forces the SDK to use the chunked encoding wrapper

Seekable streams do use the chunked wrapper unless a pre-calculated checksum is provided. If you can update the example i provided to replicate it, would be more than happy to dig in further.

[Shekaboy]

Hi @sbiscigl,

Thank you for the logs. They confirm we are looking at the same code path (content-encoding: aws-chunked), but there is a fundamental reason your memory monitor shows only 15.55 MiB while my ASAN trace shows an 8GB allocation failure.

The std::stringstream (or the buffer in the newer ChunkingInterceptor) grows by reserving contiguous blocks of Virtual Memory. On many Linux systems, RSS (Resident Set) will stay low until those pages are actually written to/zeroed by the OS, but the Virtual Size (VSZ) will climb linearly. My ASAN trace proves that the SDK is requesting an 8GB allocation, which triggers a crash once the address space or std::string max_size is exceeded.

How to Reproduce
To see the leak, you must use a non-seekable stream to prevent the SDK from using file-size optimizations, and you should monitor VSZ or the internal buffer size.

2. Watch the Buffer Size:
   If you add this log line inside AwsChunkedStream::writeChunk (or the equivalent in the new Interceptor), you will see the leak regardless of what the OS memory monitor says:

3. Regarding PR Refactor AWS Chunked logic from HTTP clients #3635:
   I see the refactor in PR Refactor AWS Chunked logic from HTTP clients #3635 moves this logic to a ChunkingInterceptor. Does that implementation specifically address the fact that m_chunkingStream.clear() does not reclaim memory? If the new interceptor still appends to a buffer without a str("") or pointer-reset equivalent, the OOM will persist for all non-seekable large uploads.

[sbiscigl]

How to Reproduce
To see the leak, you must use a non-seekable stream to prevent the SDK from using file-size optimizations, and you should monitor VSZ or the internal buffer size
My ASAN trace proves that the SDK is requesting an 8GB allocation, which triggers a crash once the address space or std::string max_size is exceeded

Can you update the example i provided to show this? if you can i'd be more than happy to look into it. when i run with ASAN (we do this in CI) i dont see any memory leaks.

I am very much not against making this code change. But we need a reproduction before making a code change.

[Shekaboy]

Hi @sbiscigl,

To ensure your reproduction matches my environment exactly, here are the specific configurations and the request logic I am using. The key difference is the use of a non-seekable stream combined with the Curl client and trailing checksums.

1. Client Configuration
   I am explicitly forcing the Curl Client and using the WHEN_SUPPORTED setting which triggers the new default integrity behavior.

Aws::Client::ClientConfiguration client_config; client_config.httpLibOverride = Aws::Http::TransferLibType::CURL_CLIENT; client_config.requestChecksumCalculation = Aws::Client::RequestChecksumCalculation::WHEN_SUPPORTED;

2. Request Implementation
   I am wrapping a read-once replication pipe in a custom ReplicationStreamReader. Because this stream is non-seekable, the SDK must use aws-chunked encoding to calculate the CRC32C on the fly.

std::shared_ptr<ReplicationStreamReader> aws_sdk_stream = std::make_shared<ReplicationStreamReader>(stream); aws_sdk_request.SetBody(aws_sdk_stream); aws_sdk_request.SetChecksumAlgorithm(Aws::S3::Model::ChecksumAlgorithm::CRC32C); aws_sdk_request.SetContentLength(content_length);

3. Why the "15MB RSS" in your test is misleading
   In your reproduction with a 2GB FStream, your monitor likely shows low RSS (Physical Memory) because the OS hasn't committed the physical pages for the stringstream's buffer yet. However, the VSZ (Virtual Memory) is growing linearly.

My ASAN trace proves this growth leads to a crash:
#1 std::__cxx11::basic_string::reserve(0x200000001) // 8GB allocation request ---> This is just insane, maximum size supported is 5gb, I cant think of any other logic as to why the SDK woudld request 8gb here. Even loading the entire object into memory should be 5gb at max

#5 Aws::Utils::Stream::AwsChunkedStream::writeChunk

Simple way to check if the code is getting exercised:
Please run your 2GB test again and print m_chunkingStream.str().size() inside the writeChunk loop. (please share those logs here)

If you use a truly non-seekable stream (returning -1 for seekoff) and a file large enough to trigger a reallocation exceeding max_size (usually >4GB), you will see the same std::bad_alloc or ASAN failure I am reporting.

[sbiscigl]

To ensure your reproduction matches my environment exactly, here are the specific configurations and the request logic I am using. The key difference is the use of a non-seekable stream combined with the Curl client and trailing checksums.

Please update the example I provided so that we can get to the bottom of this.

Update the example provided to match your use case. From everything you have said so far, the reproduction matches that besides the unseekable stream, which should not change the underlying behavior. however if you can update the example to match that, we can take a look into it.

[Shekaboy]

Hi @sbiscigl,

Please use something like this

`
// Non-seekable stream wrapper - key to triggering the code path
class NonSeekableStream : public Aws::IOStream {
public:
explicit NonSeekableStream(std::shared_ptrstd::istream src)
: Aws::IOStream(src->rdbuf()), m_src(std::move(src)) {}

std::streampos seekoff(std::streamoff, std::ios_base::seekdir,
                       std::ios_base::openmode) override { return -1; }
std::streampos seekpos(std::streampos,
                       std::ios_base::openmode) override { return -1; }

private:
std::shared_ptrstd::istream m_src;
};

int main() {
// ...SDK init as before...
// ... Additional client configs

Aws::Client::ClientConfiguration config;
config.httpLibOverride = Aws::Http::TransferLibType::CURL_CLIENT;
config.requestChecksumCalculation =
    Aws::Client::RequestChecksumCalculation::WHEN_SUPPORTED;

S3Client client{config};

auto fileStream = Aws::MakeShared<std::fstream>(
    "tag", "/path/to/file/over/4gb", std::ios_base::in | std::ios_base::binary);
auto nonSeekable = Aws::MakeShared<NonSeekableStream>("tag", fileStream);

auto request = PutObjectRequest{}
    .WithBucket("your-bucket")
    .WithKey("test-key")
    .WithChecksumAlgorithm(Aws::S3::Model::ChecksumAlgorithm::CRC32C)
    .WithContentLength(ContentLength); // usually optional - but necessary in case of non seekable streams
request.SetBody(nonSeekable);

client.PutObjectAsync(request);

}

`

[sbiscigl]

Please use something like this

can you update the complete code example I provide here. The snippets you give me arent complete. Please provide a updated version of the code example I gave earlier, that compiles, and i'd be happy to look into it.

[Shekaboy]

Hi @sbiscigl,

To give you complete visibility, here is the implementation of the stream I am using. It is a custom std::streambuf that dequeues and decompresses data on-the-fly from an asynchronous pipeline.

`
// Key implementation detail of my Stream
intStream::underflow() {
// ... async dequeue and decompression logic ...
_current_parsed_buffer = get_buffer_op->take_buffer();

char* const data_start = const_cast<char*>(data_pointer);

char* const data_end = const_cast<char*>(data_pointer + data_length);

setg(data_start, data_start, data_end);

return static_cast<unsigned char>(*gptr());

}
`

I have further "wire-level" logs from my environment that confirm the SDK is indeed using aws-chunked encoding and sending data in 64KB increments, yet failing to purge the internal buffer. (without ASAN and insanely high memory limits)

DEBUG: [AWS-CURL] (DataOut) 65532 bytes ... (repeats thousands of times) ... DEBUG: [AWS-CURL] (Text) upload completely sent off: 5370102018 bytes (5.37 GB)

2. The Contradiction
   The Log shows: Data is sent in 64KB chunks.

The ASAN trace shows: A crash at std::string::reserve(8GB).

The Logic: If the SDK were correctly "streaming," the internal stringstream should be drained after every 64KB DataOut event. The fact that the SDK eventually tries to allocate 8GB (the next power-of-two jump for a 5.3GB string) proves that every chunk ever sent is still being held in the internal stringstream buffer.

Code:
This is what I could come up with, since requiring the actual code file would not be realistically possible over a github comment

`
#include <aws/core/Aws.h>
#include <aws/core/utils/memory/stl/AWSStreamFwd.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/PutObjectRequest.h>

#include
#include

using namespace Aws;
using namespace Aws::Client;
using namespace Aws::S3;
using namespace Aws::S3::Model;

namespace {

const char* TEST_LOG_TAG = "test-application";

class sdk_context {
public:
sdk_context(SDKOptions&& options) : options_{options} { InitAPI(options_); }
~sdk_context() { ShutdownAPI(options_); }

private:
SDKOptions options_;
};

class NonSeekableStream : public Aws::IOStream {
public:
explicit NonSeekableStream(std::shared_ptrstd::istream src)
: Aws::IOStream(src->rdbuf()), m_src(std::move(src)) {}

private:
std::shared_ptrstd::istream m_src;
};

}

int main() {
SDKOptions options;
options.loggingOptions.logLevel = Utils::Logging::LogLevel::Debug;
sdk_context context{std::move(options)};

ClientConfiguration config;
config.endpointOverride = "endpoint-for-your-s3-compatible-service.com";
config.httpLibOverride = Aws::Http::TransferLibType::CURL_CLIENT;
config.checksumConfig.requestChecksumCalculation =
Aws::Client::RequestChecksumCalculation::WHEN_SUPPORTED;

S3Client client{config};

const std::string filePath = "/path/to/file/over/4gb";
auto fileStream = std::make_sharedstd::fstream(
filePath, std::ios_base::in | std::ios_base::binary);

if (!fileStream->is_open()) {
std::cerr << "Failed to open file: " << filePath << std::endl;
return 1;
}

fileStream->seekg(0, std::ios_base::end);
const auto contentLength = static_cast(fileStream->tellg());
fileStream->seekg(0, std::ios_base::beg);

auto nonSeekableBody =
Aws::MakeShared(TEST_LOG_TAG, fileStream);

auto request = PutObjectRequest{}
.WithBucket("your-bucket-name")
.WithKey("test-key")
.WithChecksumAlgorithm(ChecksumAlgorithm::CRC32C)
.WithContentLength(contentLength);
request.SetBody(nonSeekableBody);

const auto response = client.PutObject(request);
if (!response.IsSuccess()) {
std::cerr << response.GetError().GetMessage() << std::endl;
return 1;
}

std::cout << "Upload succeeded." << std::endl;
return 0;
}
`

[Shekaboy]

copy pasting the code again - apologies for the mixup in earlier comment - sorry i cant get the 2 includes to work, they are present when i edit but absent after i post

#include <aws/core/Aws.h>
#include <aws/core/utils/memory/stl/AWSStreamFwd.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/PutObjectRequest.h>

#include
#include

using namespace Aws;
using namespace Aws::Client;
using namespace Aws::S3;
using namespace Aws::S3::Model;

namespace {

const char* TEST_LOG_TAG = "test-application";

class sdk_context {
public:
sdk_context(SDKOptions&& options) : options_{options} { InitAPI(options_); }
~sdk_context() { ShutdownAPI(options_); }

private:
SDKOptions options_;
};

class NonSeekableStreamBuf : public std::streambuf {
public:
explicit NonSeekableStreamBuf(std::streambuf* src) : m_src(src) {}

protected:
// Forward reads to the underlying buffer
int_type underflow() override {
return m_src->sgetc();
}
int_type uflow() override {
return m_src->sbumpc();
}
std::streamsize xsgetn(char* s, std::streamsize n) override {
return m_src->sgetn(s, n);
}

// Block all seeks
std::streampos seekoff(std::streamoff, std::ios_base::seekdir,
                       std::ios_base::openmode) override {
    return std::streampos(-1);
}
std::streampos seekpos(std::streampos,
                       std::ios_base::openmode) override {
    return std::streampos(-1);
}

private:
std::streambuf* m_src;
};

class NonSeekableStream : public Aws::IOStream {
public:
explicit NonSeekableStream(std::shared_ptrstd::istream src)
: Aws::IOStream(&m_buf),
m_src(std::move(src)),
m_buf(m_src->rdbuf())
{}

private:
std::shared_ptrstd::istream m_src;
NonSeekableStreamBuf m_buf;
};

}

int main() {
SDKOptions options;
options.loggingOptions.logLevel = Utils::Logging::LogLevel::Debug;
sdk_context context{std::move(options)};

ClientConfiguration config;
config.endpointOverride = "endpoint-for-your-s3-compatible-service.com";
config.httpLibOverride = Aws::Http::TransferLibType::CURL_CLIENT;
config.checksumConfig.requestChecksumCalculation =
Aws::Client::RequestChecksumCalculation::WHEN_SUPPORTED;

S3Client client{config};

const std::string filePath = "/path/to/file/over/4gb";
auto fileStream = std::make_sharedstd::fstream(
filePath, std::ios_base::in | std::ios_base::binary);

if (!fileStream->is_open()) {
std::cerr << "Failed to open file: " << filePath << std::endl;
return 1;
}

fileStream->seekg(0, std::ios_base::end);
const auto contentLength = static_cast(fileStream->tellg());
fileStream->seekg(0, std::ios_base::beg);

auto nonSeekableBody =
Aws::MakeShared(TEST_LOG_TAG, fileStream);

auto request = PutObjectRequest{}
.WithBucket("your-bucket-name")
.WithKey("test-key")
.WithChecksumAlgorithm(ChecksumAlgorithm::CRC32C)
.WithContentLength(contentLength);
request.SetBody(nonSeekableBody);

const auto response = client.PutObject(request);
if (!response.IsSuccess()) {
std::cerr << response.GetError().GetMessage() << std::endl;
return 1;
}

std::cout << "Upload succeeded." << std::endl;
return 0;
}