Skip to content

Conversation

@cesco69
Copy link
Contributor

@cesco69 cesco69 commented Oct 9, 2025

TLTR: the current benchmark is not reliable
see #801 (comment)

In this PR have migrated the current benchmark based on benchmark.js (abbandoned, last update 7 years ago) to tinybench.
I have also preserved the same output

PR (tinybench)

short string............................................. x 11,148,323 ops/sec ±96.84% (449942 runs sampled)
unsafe short string...................................... x 17,983,965 ops/sec ±0.41% (1559790 runs sampled)
short string with double quote........................... x 8,865,937 ops/sec ±7.80% (668640 runs sampled)
long string without double quotes........................ x 9,749 ops/sec ±35.67% (471 runs sampled)

MASTER (benchmark.js)

short string............................................. x 25,309,923 ops/sec ±2.39% (180 runs sampled)
unsafe short string...................................... x 95,582,481 ops/sec ±6.63% (157 runs sampled)
short string with double quote........................... x 16,337,301 ops/sec ±2.08% (182 runs sampled)
long string without double quotes........................ x 52,829 ops/sec ±2.25% (182 runs sampled)

Benchmark compare: PR vs PR

image

Benchmark compare: MASTER VS MASTER

image

Raw result patch-10 VS patch-10

Checking out "patch-10"
Execute "npm run bench"

> fast-json-stringify@6.1.1 bench
> node --expose-gc ./benchmark/bench.js

short string............................................. x 11,706,183 ops/sec ±0.51% (1490282 runs sampled)
unsafe short string...................................... x 13,448,873 ops/sec ±0.39% (1751476 runs sampled)
short string with double quote........................... x 9,955,676 ops/sec ±1.22% (1042841 runs sampled)
long string without double quotes........................ x 58,338 ops/sec ±2.10% (5234 runs sampled)
unsafe long string without double quotes................. x 14,098,104 ops/sec ±0.70% (1838793 runs sampled)
long string.............................................. x 32,270 ops/sec ±2.72% (2562 runs sampled)
unsafe long string....................................... x 15,952,407 ops/sec ±0.35% (2090631 runs sampled)
number................................................... x 16,597,008 ops/sec ±0.33% (2172467 runs sampled)
integer.................................................. x 12,844,829 ops/sec ±0.45% (1659085 runs sampled)
formatted date-time...................................... x 1,296,819 ops/sec ±0.63% (125221 runs sampled)
formatted date........................................... x 1,114,027 ops/sec ±1.02% (108293 runs sampled)
formatted time........................................... x 1,106,145 ops/sec ±0.56% (107010 runs sampled)
short array of numbers................................... x 78,312 ops/sec ±1.59% (7337 runs sampled)
short array of integers.................................. x 78,772 ops/sec ±1.55% (7361 runs sampled)
short array of short strings............................. x 21,574 ops/sec ±3.07% (1811 runs sampled)
short array of long strings.............................. x 22,968 ops/sec ±2.94% (1982 runs sampled)
short array of objects with properties of different types x 11,311 ops/sec ±2.55% (1070 runs sampled)
object with number property.............................. x 15,830,834 ops/sec ±0.38% (2075975 runs sampled)
object with integer property............................. x 14,275,757 ops/sec ±0.37% (1872645 runs sampled)
object with short string property........................ x 10,257,803 ops/sec ±0.70% (1198907 runs sampled)
object with long string property......................... x 42,880 ops/sec ±2.65% (3489 runs sampled)
object with properties of different types................ x 1,302,833 ops/sec ±2.31% (104535 runs sampled)
simple object............................................ x 8,235,082 ops/sec ±1.18% (675282 runs sampled)
simple object with required fields....................... x 7,885,421 ops/sec ±1.86% (608887 runs sampled)
object with const string property........................ x 17,662,142 ops/sec ±0.35% (2313928 runs sampled)
object with const number property........................ x 14,926,448 ops/sec ±0.37% (1960630 runs sampled)
object with const bool property.......................... x 14,163,588 ops/sec ±0.34% (1852894 runs sampled)
object with const object property........................ x 14,341,321 ops/sec ±0.35% (1875053 runs sampled)
object with const null property.......................... x 13,666,385 ops/sec ±0.30% (1785938 runs sampled)

Checking out "patch-10"
Execute "npm run bench"

> fast-json-stringify@6.1.1 bench
> node --expose-gc ./benchmark/bench.js

short string............................................. x 11,570,845 ops/sec ±0.46% (1469542 runs sampled)
unsafe short string...................................... x 11,351,172 ops/sec ±0.57% (1430163 runs sampled)
short string with double quote........................... x 9,530,411 ops/sec ±1.35% (941579 runs sampled)
long string without double quotes........................ x 53,824 ops/sec ±2.38% (4621 runs sampled)
unsafe long string without double quotes................. x 14,104,935 ops/sec ±0.52% (1842289 runs sampled)
long string.............................................. x 48,667 ops/sec ±1.26% (4648 runs sampled)
unsafe long string....................................... x 12,879,691 ops/sec ±0.85% (1668303 runs sampled)
number................................................... x 13,393,812 ops/sec ±0.79% (1732412 runs sampled)
integer.................................................. x 14,141,119 ops/sec ±0.54% (1845326 runs sampled)
formatted date-time...................................... x 1,280,996 ops/sec ±0.71% (124255 runs sampled)
formatted date........................................... x 1,090,780 ops/sec ±0.95% (106092 runs sampled)
formatted time........................................... x 1,094,414 ops/sec ±0.62% (105745 runs sampled)
short array of numbers................................... x 76,887 ops/sec ±1.51% (7195 runs sampled)
short array of integers.................................. x 78,839 ops/sec ±1.65% (7345 runs sampled)
short array of short strings............................. x 24,590 ops/sec ±1.24% (2378 runs sampled)
short array of long strings.............................. x 21,519 ops/sec ±2.97% (1806 runs sampled)
short array of objects with properties of different types x 11,028 ops/sec ±2.55% (1045 runs sampled)
object with number property.............................. x 15,872,653 ops/sec ±0.32% (2084372 runs sampled)
object with integer property............................. x 17,123,302 ops/sec ±0.32% (2249903 runs sampled)
object with short string property........................ x 11,338,634 ops/sec ±0.34% (1426428 runs sampled)
object with long string property......................... x 41,316 ops/sec ±3.13% (3156 runs sampled)
object with properties of different types................ x 1,840,636 ops/sec ±1.55% (152921 runs sampled)
simple object............................................ x 8,278,110 ops/sec ±0.95% (683770 runs sampled)
simple object with required fields....................... x 7,021,421 ops/sec ±1.75% (507834 runs sampled)
object with const string property........................ x 16,731,578 ops/sec ±0.43% (2188026 runs sampled)
object with const number property........................ x 14,244,972 ops/sec ±0.64% (1864377 runs sampled)
object with const bool property.......................... x 16,654,446 ops/sec ±0.32% (2179446 runs sampled)
object with const object property........................ x 16,938,277 ops/sec ±0.42% (2227930 runs sampled)
object with const null property.......................... x 16,442,077 ops/sec ±0.34% (2154620 runs sampled)

short string..............................................+1.17%
unsafe short string......................................+18.48%
short string with double quote............................+4.46%
long string without double quotes.........................+8.39%
unsafe long string without double quotes..................-0.05%
long string..............................................-33.69%
unsafe long string.......................................+23.86%
number...................................................+23.92%
integer...................................................-9.17%
formatted date-time.......................................+1.24%
formatted date............................................+2.13%
formatted time............................................+1.07%
short array of numbers....................................+1.85%
short array of integers...................................-0.08%
short array of short strings.............................-12.27%
short array of long strings...............................+6.73%
short array of objects with properties of different types.+2.57%
object with number property...............................-0.26%
object with integer property.............................-16.63%
object with short string property.........................-9.53%
object with long string property..........................+3.79%
object with properties of different types................-29.22%
simple object.............................................-0.52%
simple object with required fields.......................+12.31%
object with const string property.........................+5.56%
object with const number property.........................+4.78%
object with const bool property..........................-14.96%
object with const object property........................-15.33%
object with const null property..........................-16.88%
Back to bench f2c2c6f

close #798

Signed-off-by: francesco <francesco.bagnoli.69@gmail.com>
@cesco69 cesco69 changed the title feat: new benchmar with tinybench feat: new benchmark with tinybench Oct 9, 2025
@cesco69 cesco69 marked this pull request as ready for review October 9, 2025 10:38
Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@mcollina
Copy link
Member

mcollina commented Oct 9, 2025

@Uzlopak why do you think there is a difference?

@cesco69
Copy link
Contributor Author

cesco69 commented Jan 9, 2026

tinybench also for library comparison benchmarks

npm run benchmark

> fast-json-stringify@6.1.1 benchmark
> node --expose-gc ./benchmark/bench-cmp-lib.js

FJS creation x 8,155 ops/sec ±2.71% (779 runs sampled)
CJS creation x 334,242 ops/sec ±1.91% (30572 runs sampled)
AJV Serialize creation x 15,209,946 ops/sec ±0.79% (1985087 runs sampled)
json-accelerator creation x 862,370 ops/sec ±6.56% (79686 runs sampled)
JSON.stringify array x 9,599 ops/sec ±3.80% (937 runs sampled)
fast-json-stringify array default x 9,328 ops/sec ±1.93% (911 runs sampled)
json-accelerator array x 6,612 ops/sec ±1.87% (648 runs sampled)
fast-json-stringify array json-stringify x 9,282 ops/sec ±1.37% (913 runs sampled)
compile-json-stringify array x 7,906 ops/sec ±1.64% (776 runs sampled)
AJV Serialize array x 7,952 ops/sec ±1.59% (782 runs sampled)
JSON.stringify large array x 335 ops/sec ±4.34% (64 runs sampled)
fast-json-stringify large array default x 395 ops/sec ±15.21% (64 runs sampled)
fast-json-stringify large array json-stringify x 331 ops/sec ±5.06% (64 runs sampled)
compile-json-stringify large array x 329 ops/sec ±7.93% (64 runs sampled)
AJV Serialize large array x 331 ops/sec ±6.85% (64 runs sampled)
JSON.stringify long string x 49,708 ops/sec ±1.49% (4865 runs sampled)
fast-json-stringify long string x 50,632 ops/sec ±1.41% (4957 runs sampled)
json-accelerator long string x 50,959 ops/sec ±1.43% (4993 runs sampled)
compile-json-stringify long string x 49,675 ops/sec ±1.91% (4756 runs sampled)
AJV Serialize long string x 19,487 ops/sec ±1.24% (1904 runs sampled)
JSON.stringify short string x 9,465,923 ops/sec ±0.46% (883063 runs sampled)
fast-json-stringify short string x 12,688,760 ops/sec ±0.38% (1651925 runs sampled)
json-accelerator short string x 13,488,073 ops/sec ±0.40% (1769180 runs sampled)
compile-json-stringify short string x 11,334,480 ops/sec ±0.59% (1413838 runs sampled)
AJV Serialize short string x 11,771,091 ops/sec ±0.34% (1497447 runs sampled)
JSON.stringify obj x 4,429,689 ops/sec ±1.26% (414605 runs sampled)
fast-json-stringify obj x 7,467,161 ops/sec ±0.36% (615026 runs sampled)
json-accelerator obj x 6,193,328 ops/sec ±1.22% (544495 runs sampled)
compile-json-stringify obj x 10,122,151 ops/sec ±0.31% (1118763 runs sampled)
AJV Serialize obj x 9,640,357 ops/sec ±0.44% (917048 runs sampled)
JSON stringify date x 707,997 ops/sec ±0.29% (69538 runs sampled)
fast-json-stringify date format x 1,126,868 ops/sec ±3.33% (108114 runs sampled)
json-accelerate date format x 1,124,867 ops/sec ±0.31% (110023 runs sampled)
compile-json-stringify date format x 707,242 ops/sec ±0.28% (69549 runs sampled)

@mcollina mcollina merged commit dbcaa31 into fastify:main Jan 9, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve benchmarks

2 participants