Make message and forward async to avoid contention and losing logs

[ncode]

This pull request introduces a robust asynchronous side-effect processing system to the audit server, focusing on improving request-path latency and providing tunable reliability for notifications and forwarding. It adds async configuration options, implements an async worker and queue system, introduces optional durable (persistent) retry support, and updates documentation and configuration accordingly. Performance benchmarking and tuning guidance are also included.

Key changes:

Asynchronous Side-Effect Processing:

• Added a new async worker and queue system (sideTask, enqueueSide, startSideWorkers, etc.) to decouple notifications/forwarding from the request path, significantly reducing request latency under load. (pkg/auditserver/async.go [1] pkg/auditserver/server.go [2] [3] [4] [5]
• Introduced configuration options for async queue size, worker count, enqueue mode, and timeouts, all with sensible defaults and validation. (pkg/auditserver/server.go [1] configs/development/config/config.yaml [2] README.md [3] [4]

Durable Side-Effect Delivery (Optional):

• Added a file-based persistent side-effect store with dead-letter support for durable retry of failed notifications/forwards, and replay of pending tasks on startup. (pkg/auditserver/durable.go [1] pkg/auditserver/server.go [2] [3] [4]

Timeouts and Reliability Improvements:

• Added per-operation timeouts for Slack/webhook messaging and UDP forwarding. (pkg/auditserver/server.go [1] [2] pkg/forwarder/forwarder.go [3] [4] [5]
• Updated the forwarding and messaging code to accept and use these timeouts. (pkg/auditserver/server.go [1] [2]

Documentation and Benchmarking:

• Expanded documentation with async configuration, tuning profiles, performance findings, and operational guidance for side-effect reliability vs. latency trade-offs. (README.md [1] [2]
• Added benchmark tests for core logging and rule evaluation paths. (pkg/auditserver/bench_test.go pkg/auditserver/bench_test.goR1-R43)

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.39%. Comparing base (d5e6e1a) to head (0f1a0ba).

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #22      +/-   ##
==========================================
+ Coverage   88.88%   95.39%   +6.50%     
==========================================
  Files           8       10       +2     
  Lines         279      543     +264     
==========================================
+ Hits          248      518     +270     
+ Misses         24       20       -4     
+ Partials        7        5       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.