expire: skip per-article SMretrieve via cancel tombstone log

[kev009]

Adds an opt-in (expiretombstone in inn.conf, default false) side-channel that records every cancellation alongside the existing storage and overview deletions, so a subsequent expire run can drop matching history entries without doing per-article SMretrieve(RETR_STAT) syscalls. At million-article scale on tradspool/timehash this turns a million access(2) calls into a few thousand hash lookups.

Two log files cooperate:

${pathdb}/expireover.tombstone: written by expireover after each
successful inline SMcancel; in delayrm mode written up front and
promoted by expirerm after fastrm succeeds. Atomic .NEW -> final
rename under an exclusive non-blocking fcntl POSIX lock. A
crashed-or-failed previous run's leftover .NEW is verified
per-token via SMretrieve and merged into the next run, so partial
rmfile failures cannot orphan articles on disk.

${pathdb}/cancels.tombstone: appended by innd's ARTcancel and by
sm -r through a new public SMcanceltombstone() helper. Shared
fcntl POSIX lock for appenders, exclusive for expire's snapshot
rename to .processing; one retry on EAGAIN closes the appender
vs consumer race window. POSIX O_APPEND atomicity for sub-
PIPE_BUF writes keeps single-line tokens from interleaving.

expire reads both files into one hashset, treats every article in either log as already gone, and falls back to per-article SMretrieve only for storage methods where SMprobe(SELFEXPIRE) is true (CNFS wrap-around). An empty-but-present tombstone correctly says "no cancels this cycle" and skips SMretrieve for everything; truly absent files fall back to the slow path.

The format is line-per-token; readers ignore blank and #-prefixed lines, and expireover writes "# inn-tombstone v1" as a forward- compatibility header. Disk and memory footprint scale linearly: ~38 bytes per entry on disk, ~50 bytes in expire's hash table.

Tests cover the public library (tombstone_hash_create, _read, _rename_for_processing, _present), SMcanceltombstone branches including expiretombstone=false bypass and unwritable-pathdb non-crash, two-file merge with dedup, comment-line skip, HISexpire integration with a callback that mirrors EXPdoline's full decision tree (kept / tombstoned / selfexpire-gone / selfexpire-alive), and an end-to-end shell test that exercises sm + cancels.tombstone + expiretombstone toggling.

Default is false; sites should opt in after validating the option's behavior against their workload. Revisit default after wider testing.

[Julien-Elie]

A great idea! Thanks for it, Kevin.
I need a bit of time to catch up on all your recent improvements. Will be done before the end of the month.

[kev009]

This one is definitely more invasive than the expireover change (which is basically a free lunch), and it is less critical (i.e. on my spool expire takes about 27h before). But they are designed to work together, and once effective (after first run) this will make expire a seconds to minutes runtime even for a very busy site. And with that kind of speedup, the option also opens up to run these more frequently and disable nnrpdcheckart.

[Julien-Elie]

Isn't there any need from sometimes doing a full legacy run? I am wondering whether we won't have any orphan never expired even after a few runs either with the bloom filter or this new feature.
Manually removing a file from tradspool or other edge-cases.

[kev009]

@Julien-Elie yes, the gap is manually 'rm' something or filesystem corruption, in which case a full expire run would be necessary. We need to make sure the documentation about that is clear, these are intended for healthy spools where inn tools are used to manage the spool. If that is invalidated, the admin needs to do a patrol expire.

[kev009]

Actual runtime on my system. It takes a little longer than I hypothesized because I forgot the actual data copy cost of writing out a new history file. But plenty fast:

expire begin Mon May 18 04:40:40 UTC 2026: (-v1)
    Loaded 10 + 2 + 0 tombstone entries (10 unique)
    Article lines processed 68900687
    Articles retained       68900677
    Entries expired               10
    Tombstone hits                10
expire end Mon May 18 04:44:07 UTC 2026

[Julien-Elie]

A very clean and thoughtful patch. Thanks again Kevin for your work!

I'm wondering whether the 2 expiretombstone-without-groupbaseexpiry checks wouldn't be better in the innconf_validate() function in lib/innconf.c so that it is directly noticed by the news admin. I can move it there if you also think the consistency should be done there.
Anyway, thanks for having taking care of bypassing the SMcanceltombstone function if not correctly configured.

As for the use of sm -r, if I understand well the new usage to have a consistent ownership for cancels.tombstone, shouldn't we say in the sm documentation that when expiretombstone is enabled, the program needs being run as the news user and group?

[Julien-Elie]

Like for the improvements to expireover, I'll be glad to hear from other news admins who run large news spools about how expiretombstone performs.
I do not have enough articles to obtain a benefit for my usage (only a couple of seconds), but hopefully it helps others!
As soon as the issue about tagged hash history is fixed, we'll have a snapshot generated for CURRENT that I'll advertise for wider testings before merging them into STABLE for the INN 2.7.4 release.

Without expiretombstone:

% time expire -N -v1
Article lines processed  4108518
Articles retained        4108518
Entries expired                0
expire -N -v1  7,16s user 2,94s system 99% cpu 10,145 total

With expiretombstone:

% time expire -N -v1
Loaded 0 + 0 + 0 tombstone entries (0 unique)
Article lines processed  4108519
Articles retained        4108519
Entries expired                0
Tombstone hits                 0
expire -N -v1  7,10s user 1,51s system 99% cpu 8,656 total