feat(buffer): Disable flush-to-disk for ephemeral storage

[jjbayer]

Follow-up to #5746: With our current setup using local SSDs we should not flush in-flight data to disk on shutdown, and rather attempt to process all data like we do for the in-memory buffer.

ref: INGEST-510

[linear-code]

INGEST-510 dropped envelope: internal error

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: New is_ephemeral method is never called anywhere
  • shutdown() now uses self.is_ephemeral() for its flush guard, making the helper method actively used while preserving behavior.

Or push these changes by commenting:

@cursor push f4454c5117

Preview (f4454c5117)

diff --git a/relay-server/src/services/buffer/envelope_buffer/mod.rs b/relay-server/src/services/buffer/envelope_buffer/mod.rs
--- a/relay-server/src/services/buffer/envelope_buffer/mod.rs
+++ b/relay-server/src/services/buffer/envelope_buffer/mod.rs
@@ -191,8 +191,9 @@
         // Currently, we want to flush the buffer only for disk, since the in memory implementation
         // tries to not do anything and pop as many elements as possible within the shutdown
         // timeout.
+        let is_ephemeral = self.is_ephemeral();
         match self {
-            Self::Sqlite(buffer) if !buffer.stack_provider.ephemeral() => {
+            Self::Sqlite(buffer) if !is_ephemeral => {
                 buffer.flush().await;
                 true
             }

_{This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[cursor]

Ephemeral SQLite blocked by memory backpressure during shutdown

Medium Severity

When an ephemeral SQLite buffer returns false from shutdown(), the service loop continues to drain envelopes — matching the in-memory behavior. However, system_ready still gates unspooling on buffer.is_memory() || self.memory_ready(), and is_memory() returns false for all SQLite variants including ephemeral. Under high memory pressure (>80%), the ephemeral buffer gets stuck waiting indefinitely in system_ready, unlike the in-memory buffer which always bypasses that check. Data on ephemeral storage would then be lost when the process is eventually killed.

 

[jjbayer]

This is intended behavior. The memory implementation bypasses the memory check because it would otherwise get stuck -- it cannot lower memory pressure without unspooling & processing envelopes. The disk spooler, even the ephemeral one, can lower memory pressure by pausing the dequeue.