Benchmarks v2

[podocarp]

Similar to Use-Tusk/drift-python-sdk#60

• Migrated a lot of the old e2e tests to use the same style as the python sdk, use a single shared makeRequest function instead of calling curl through docker
• The makeRequest function is the one that handles the benchmarks, it checks for the BENCHMARKS envvar just like python
• Benchmarks only run for cjs tests. There's no point in running both.
• Remove a lot of console.log statements, my thinking here is: even for e2e tests we don't need that much output, we'll usually debug locally with a known failing testcase, and we can add logs manually then. No need for the CI to be that spammy, and it affects performance during benchmarks. And the console.error statements are still there.

Comparison of baseline (SDK disabled) vs with SDK enabled (RECORD mode).
Negative diff = slower with SDK. Duration: 5s per endpoint, 3s warmup.

Below are the results, run on my laptop, took quite a long time to complete.
Note: The actual CLI output is not in markdown format, it's in a golang like format because I kind of like its simplicity and readability. This is also one complaint for the old benchmarks system, there's too much noise like QPS, latency, CPU, bla bla

---

Also a summary by claude:

  - upstash-redis-js has near-zero overhead and many ops actually run faster with SDK (likely measurement noise at 8000+ ops/s)
  - mysql2 has severe regressions on execute and findall paths (-97%+)
  - postgres.js sql-tagged template queries show near-zero overhead while core queries take a hit
  - prisma raw/query at -96.8% and PUT /users/1 at +217% look like anomalies

fetch

Benchmark	Baseline	With SDK	Diff
GET /health	3229.53/s	2255.47/s	-30.2%
GET /test/fetch-get	47.01/s	38.90/s	-17.3%
POST /test/fetch-post	3.30/s	2.90/s	-12.1%
GET /test/fetch-headers	40.22/s	40.58/s	+0.9%
GET /test/fetch-json	13.06/s	39.19/s	+200.1%
GET /test/fetch-url-object	36.59/s	49.90/s	+36.4%

---

grpc

Benchmark	Baseline	With SDK	Diff
GET /health	3180.49/s	2264.88/s	-28.8%
GET /greet/hello	2101.17/s	1280.98/s	-39.0%
GET /greet/hello-with-metadata	2102.06/s	1263.69/s	-39.9%
POST /greet/custom	1913.65/s	1146.85/s	-40.1%
GET /greet/hello-again	2100.16/s	1292.43/s	-38.5%
GET /greet/many-times	2077.69/s	1300.96/s	-37.4%
GET /calc/add	2064.33/s	1281.26/s	-37.9%
GET /calc/subtract	2087.16/s	1278.72/s	-38.7%
GET /calc/multiply	2092.49/s	1264.66/s	-39.6%
POST /calc/divide	1899.54/s	1122.78/s	-40.9%
GET /calc/divide-by-zero	1770.83/s	1192.50/s	-32.7%
GET /users/1	2016.19/s	1249.38/s	-38.0%
POST /users	1874.08/s	1054.93/s	-43.7%
PUT /users/1	1844.16/s	1090.27/s	-40.9%
GET /users?limit=5&offset=0	1791.18/s	1072.00/s	-40.2%
DELETE /users/2	1707.33/s	1255.91/s	-26.4%
GET /test/user-not-found	1790.74/s	1244.52/s	-30.5%
GET /test/sequential-calls	1284.02/s	763.71/s	-40.5%
POST /test/complex-data	2024.62/s	1184.98/s	-41.5%
POST /files/upload	1972.13/s	1194.51/s	-39.4%
GET /files/download/file_1	2027.12/s	1216.86/s	-40.0%
GET /test/unary-callback-only	2074.66/s	1249.24/s	-39.8%
GET /test/unary-options-only	2081.70/s	1137.57/s	-45.4%

---

http

Benchmark	Baseline	With SDK	Diff
GET /health	3605.80/s	2656.25/s	-26.3%
GET /test-http-get	13.76/s	18.48/s	+34.3%
POST /test-http-request	3.53/s	3.61/s	+2.3%
GET /test-https-get	16.69/s	14.84/s	-11.1%
GET /test-axios-get	13.23/s	15.70/s	+18.7%
POST /test-axios-post	2.60/s	3.18/s	+22.3%
GET /test-url-object-get	13.97/s	18.57/s	+32.9%
POST /test-url-object-request	3.51/s	3.29/s	-6.3%

---

ioredis

Benchmark	Baseline	With SDK	Diff
GET /health	3192.89/s	2260.33/s	-29.2%
GET /test/get	3063.25/s	1790.60/s	-41.5%
POST /test/set	2619.98/s	1496.97/s	-42.9%
POST /test/del	2633.72/s	1522.78/s	-42.2%
POST /test/exists	2579.56/s	1549.90/s	-39.9%
POST /test/expire	2647.90/s	1515.68/s	-42.8%
POST /test/ttl	2635.79/s	1524.77/s	-42.2%
GET /test/incr	3049.28/s	1766.89/s	-42.1%
GET /test/decr	3064.18/s	1765.34/s	-42.4%
GET /test/mget	3002.26/s	1675.59/s	-44.2%
POST /test/mset	3026.66/s	1669.94/s	-44.8%
GET /test/hget	3009.50/s	1693.52/s	-43.7%
POST /test/hset	2580.52/s	1449.06/s	-43.8%
GET /test/hgetall	2963.95/s	1652.56/s	-44.2%
POST /test/hdel	2606.93/s	1464.52/s	-43.8%
POST /test/lpush	2643.80/s	1299.47/s	-50.8%
POST /test/rpush	2617.70/s	1492.92/s	-43.0%
POST /test/lpop	2579.30/s	1496.37/s	-42.0%
POST /test/rpop	2596.78/s	1472.36/s	-43.3%
GET /test/lrange	2477.37/s	1694.34/s	-31.6%
POST /test/llen	2618.75/s	1497.75/s	-42.8%
POST /test/sadd	2610.29/s	1418.39/s	-45.7%
POST /test/srem	2620.17/s	1454.52/s	-44.5%
GET /test/smembers	3034.52/s	1642.69/s	-45.9%
POST /test/sismember	2625.50/s	1452.93/s	-44.7%
POST /test/zadd	2575.58/s	1466.65/s	-43.1%
GET /test/zrange	3025.58/s	1710.46/s	-43.5%
POST /test/zrem	2622.46/s	1469.50/s	-44.0%
POST /test/zscore	2636.20/s	1471.92/s	-44.2%
POST /test/keys	2609.30/s	1451.06/s	-44.4%
POST /test/flushdb	2355.24/s	1233.73/s	-47.6%
GET /test/ping	2750.95/s	1667.20/s	-39.4%
GET /test/pipeline	2881.56/s	1141.09/s	-60.4%
GET /test/multi	2874.95/s	1075.88/s	-62.6%
GET /test/new-client	1383.91/s	695.86/s	-49.7%
GET /test/getbuffer	2374.05/s	1350.60/s	-43.1%
GET /test/mgetbuffer	2625.73/s	1230.33/s	-53.1%

---

mysql

Benchmark	Baseline	With SDK	Diff
GET /health	3136.18/s	2287.89/s	-27.0%
GET /connection/query-callback	2406.55/s	1432.87/s	-40.5%
GET /connection/query-params?key=test_key_1	2294.13/s	1416.51/s	-38.3%
GET /connection/query-options	2334.56/s	1498.38/s	-35.8%
GET /connection/query-stream	2380.31/s	1490.34/s	-37.4%
GET /connection/multi-statement	2244.53/s	834.93/s	-62.8%
GET /pool/query	2117.43/s	310.94/s	-85.3%
GET /pool/get-connection	2116.03/s	1308.16/s	-38.2%
POST /transaction/commit	349.79/s	190.92/s	-45.4%
POST /transaction/rollback	299.07/s	304.06/s	+1.7%
POST /test/transaction-with-options	283.07/s	124.05/s	-56.2%
POST /crud/insert	369.36/s	347.60/s	-5.9%
PUT /crud/update	2018.69/s	1334.99/s	-33.9%
DELETE /crud/delete	2019.48/s	1311.40/s	-35.1%
GET /advanced/join	346.66/s	117.35/s	-66.1%
GET /advanced/aggregate	83.64/s	41.46/s	-50.4%
GET /advanced/subquery	324.52/s	156.97/s	-51.6%
GET /advanced/prepared	2181.45/s	1383.74/s	-36.6%
GET /lifecycle/ping	2800.45/s	1910.00/s	-31.8%
GET /lifecycle/end-and-reconnect	8.86/s	8.76/s	-1.1%
POST /lifecycle/change-user	1248.80/s	993.64/s	-20.4%
GET /lifecycle/pause-resume	17.24/s	16.74/s	-2.9%
GET /pool/end-and-recreate	1306.36/s	817.27/s	-37.4%
GET /test/pool-events	8.81/s	8.64/s	-1.9%
GET /test/pool-namespace-query	914.32/s	573.74/s	-37.2%
GET /events/connect	1383.86/s	1100.51/s	-20.5%
GET /stream/query-stream-method	2277.33/s	1272.07/s	-44.1%

---

mysql2

Benchmark	Baseline	With SDK	Diff
GET /health	3581.31/s	2565.10/s	-28.4%
GET /test/connection-query	2743.85/s	67.40/s	-97.5%
POST /test/connection-parameterized	2618.95/s	1120.90/s	-57.2%
GET /test/connection-execute	2858.00/s	59.35/s	-97.9%
POST /test/connection-execute-params	2709.03/s	1167.99/s	-56.9%
GET /test/pool-query	2801.63/s	733.17/s	-73.8%
POST /test/pool-parameterized	2450.64/s	865.23/s	-64.7%
GET /test/pool-execute	2841.06/s	47.91/s	-98.3%
POST /test/pool-execute-params	2619.52/s	791.91/s	-69.8%
GET /test/pool-getConnection	2930.53/s	1350.20/s	-53.9%
GET /test/connection-connect	1215.16/s	859.74/s	-29.2%
GET /test/connection-ping	3606.39/s	2182.70/s	-39.5%
GET /test/stream-query	2723.84/s	1165.68/s	-57.2%
GET /test/sequelize-authenticate	2364.39/s	1463.73/s	-38.1%
GET /test/sequelize-findall	2114.43/s	47.91/s	-97.7%
POST /test/sequelize-findone	1925.69/s	955.36/s	-50.4%
GET /test/sequelize-complex	1671.08/s	709.33/s	-57.6%
GET /test/sequelize-raw	2258.27/s	1001.52/s	-55.7%
POST /test/sequelize-transaction	1109.21/s	491.32/s	-55.7%
GET /test/promise-connection-query	1027.90/s	569.38/s	-44.6%
GET /test/promise-pool-query	2679.33/s	680.22/s	-74.6%
GET /test/promise-pool-getconnection	2835.76/s	1297.87/s	-54.2%
GET /test/transaction-methods	230.27/s	195.12/s	-15.3%
GET /test/prepare-statement	1026.34/s	541.89/s	-47.2%
GET /test/change-user	935.48/s	580.56/s	-37.9%
GET /test/nested-null-values	298.39/s	226.22/s	-24.2%
GET /test/binary-data	121.81/s	196.41/s	+61.2%

---

nextjs

Benchmark	Baseline	With SDK	Diff
GET /api/health	211.47/s	158.26/s	-25.2%
GET /api/weather	0.08/s	0.07/s	-12.5%
GET /api/weather?location=London	0.06/s	0.08/s	+33.3%
POST /api/weather	0.10/s	0.08/s	-20.0%

---

pg

Benchmark	Baseline	With SDK	Diff
GET /health	3575.02/s	2653.38/s	-25.8%
GET /test/basic-query	2925.73/s	1717.76/s	-41.3%
POST /test/parameterized-query	2590.65/s	1609.36/s	-37.9%
GET /test/client-query	2941.01/s	1884.26/s	-35.9%
GET /test/client-connect	909.10/s	778.89/s	-14.3%
GET /test/client-close	901.74/s	781.95/s	-13.3%
GET /test/pool-query	2821.98/s	1433.00/s	-49.2%
POST /test/pool-parameterized	2608.48/s	1335.26/s	-48.8%
GET /test/pool-connect	2970.83/s	1765.93/s	-40.6%
GET /test/pool-transaction	2366.33/s	1163.94/s	-50.8%
GET /test/query-rowmode-array	2847.27/s	1834.76/s	-35.6%
GET /test/multi-statement	3169.15/s	1871.78/s	-40.9%

---

postgres (postgres.js)

Benchmark	Baseline	With SDK	Diff
GET /health	3077.04/s	2169.45/s	-29.5%
GET /cache/all	2239.70/s	1239.16/s	-44.7%
GET /cache/sample	1678.00/s	1015.56/s	-39.5%
GET /cache/raw	187.63/s	202.32/s	+7.8%
POST /cache/execute-raw	2373.96/s	1343.52/s	-43.4%
POST /cache/insert	1419.96/s	1051.49/s	-25.9%
PUT /cache/update	472.17/s	294.36/s	-37.7%
DELETE /cache/delete	1683.88/s	1106.29/s	-34.3%
GET /users/by-email?email=alice@example.com	1866.11/s	1258.75/s	-32.5%
POST /users/insert	1468.37/s	1117.56/s	-23.9%
GET /cache/dynamic-fragments	175.16/s	184.78/s	+5.5%
POST /cache/update-with-fragments	113.94/s	116.68/s	+2.4%
GET /cache/complex-fragments	177.06/s	188.88/s	+6.7%
GET /test/execute-method	187.59/s	186.66/s	-0.5%
GET /test/sql-file	173.78/s	187.37/s	+7.8%
GET /test/pending-query-raw	179.76/s	185.78/s	+3.3%
GET /test/sql-reserve	184.80/s	195.40/s	+5.7%
GET /test/sql-cursor	180.67/s	184.71/s	+2.2%
GET /test/sql-cursor-callback	178.92/s	186.36/s	+4.2%
GET /test/sql-foreach	182.24/s	188.00/s	+3.2%
GET /test/describe-method	275.70/s	228.06/s	-17.3%
GET /test/savepoint	87.25/s	92.41/s	+5.9%
GET /test/listen-notify	1.87/s	1.88/s	+0.5%
GET /test/bytea-data	199.03/s	209.08/s	+5.0%
GET /test/unsafe-cursor	179.51/s	196.75/s	+9.6%
GET /test/unsafe-foreach	182.89/s	191.76/s	+4.8%
GET /test/large-object	132.13/s	146.38/s	+10.8%

---

prisma

Benchmark	Baseline	With SDK	Diff
GET /health	3058.43/s	2281.46/s	-25.4%
GET /users/all	1977.51/s	836.49/s	-57.7%
GET /users/active	1822.84/s	820.27/s	-55.0%
GET /users/1	1835.47/s	1400.54/s	-23.7%
GET /users/first-active	1789.20/s	822.55/s	-54.0%
GET /users/by-email/alice@example.com	1824.45/s	1317.53/s	-27.8%
POST /users/create	1153.42/s	894.59/s	-22.4%
POST /users/create-many	293.54/s	288.30/s	-1.8%
PUT /users/1	285.60/s	906.36/s	+217.4%
PUT /users/bulk-deactivate	20.12/s	19.58/s	-2.7%
POST /users/upsert	286.28/s	288.39/s	+0.7%
GET /users/count	187.87/s	215.11/s	+14.5%
GET /orders/aggregate	1972.03/s	1268.69/s	-35.7%
GET /users/1/with-posts	1533.96/s	1200.29/s	-21.8%
GET /posts/published	1332.78/s	887.98/s	-33.4%
POST /posts/create-with-author	249.64/s	129.92/s	-48.0%
POST /transactions/sequential	219.56/s	278.57/s	+26.9%
POST /transactions/interactive	224.33/s	283.15/s	+26.2%
POST /raw/query	244.96/s	7.75/s	-96.8%
POST /raw/execute	114.91/s	101.56/s	-11.6%
POST /errors/unique-violation	1315.37/s	832.90/s	-36.7%
GET /errors/not-found	1445.87/s	999.15/s	-30.9%
POST /errors/validation	1516.59/s	1065.33/s	-29.8%
DELETE /users/inactive	244.08/s	235.55/s	-3.5%

---

upstash-redis-js

Benchmark	Baseline	With SDK	Diff
GET /health	8628.08/s	8152.36/s	-5.5%
Strings
POST /test/string/set	8438.09/s	8317.53/s	-1.4%
GET /test/string/get	8303.59/s	8291.84/s	-0.1%
POST /test/string/mset	8415.07/s	7805.34/s	-7.2%
GET /test/string/mget	8259.00/s	8657.59/s	+4.8%
POST /test/string/setex	8234.16/s	8150.64/s	-1.0%
POST /test/string/setnx	8418.77/s	8633.18/s	+2.5%
POST /test/string/getdel	8228.79/s	8124.01/s	-1.3%
POST /test/string/append	8491.72/s	8279.80/s	-2.5%
POST /test/string/incr	8390.26/s	8294.91/s	-1.1%
POST /test/string/incrby	8414.82/s	8438.30/s	+0.3%
POST /test/string/incrbyfloat	8287.17/s	8029.54/s	-3.1%
POST /test/string/decr	8302.59/s	8288.93/s	-0.2%
POST /test/string/decrby	8283.35/s	8253.74/s	-0.4%
GET /test/string/strlen	8291.60/s	8560.23/s	+3.2%
GET /test/string/getrange	8318.83/s	8101.90/s	-2.6%
POST /test/string/setrange	8065.44/s	8271.38/s	+2.6%
Hashes
POST /test/hash/hset	8295.49/s	8429.18/s	+1.6%
GET /test/hash/hget	8425.95/s	8292.01/s	-1.6%
GET /test/hash/hgetall	8295.64/s	8154.10/s	-1.7%
POST /test/hash/hmset	8749.37/s	8008.45/s	-8.5%
GET /test/hash/hmget	8526.08/s	8273.38/s	-3.0%
POST /test/hash/hdel	8451.60/s	8323.97/s	-1.5%
GET /test/hash/hexists	8339.75/s	8373.80/s	+0.4%
GET /test/hash/hkeys	8474.74/s	8203.07/s	-3.2%
GET /test/hash/hvals	8395.55/s	8435.00/s	+0.5%
GET /test/hash/hlen	8316.51/s	8417.56/s	+1.2%
POST /test/hash/hincrby	7896.31/s	8023.25/s	+1.6%
POST /test/hash/hincrbyfloat	8290.53/s	7973.01/s	-3.8%
POST /test/hash/hsetnx	8101.81/s	8434.38/s	+4.1%
Lists
POST /test/list/lpush	8400.55/s	8256.68/s	-1.7%
POST /test/list/rpush	8522.03/s	8500.53/s	-0.3%
GET /test/list/lrange	8421.83/s	8575.95/s	+1.8%
POST /test/list/lpop	8428.91/s	8552.34/s	+1.5%
POST /test/list/rpop	8617.36/s	8289.97/s	-3.8%
GET /test/list/llen	8336.48/s	8166.74/s	-2.0%
GET /test/list/lindex	8342.10/s	8241.41/s	-1.2%
POST /test/list/lset	8461.08/s	8239.93/s	-2.6%
POST /test/list/linsert	8369.14/s	8328.70/s	-0.5%
POST /test/list/lrem	8245.09/s	8192.10/s	-0.6%
POST /test/list/ltrim	8459.59/s	8345.48/s	-1.3%
POST /test/list/rpoplpush	8248.96/s	8281.96/s	+0.4%
POST /test/list/lpos	7965.32/s	8654.64/s	+8.7%
POST /test/list/lmove	8128.69/s	8450.85/s	+4.0%
Sets
POST /test/set/sadd	8246.95/s	8229.37/s	-0.2%
GET /test/set/smembers	8507.51/s	8258.12/s	-2.9%
GET /test/set/sismember	8527.11/s	8177.20/s	-4.1%
POST /test/set/srem	8297.52/s	8346.43/s	+0.6%
GET /test/set/scard	8526.70/s	8331.84/s	-2.3%
POST /test/set/spop	8514.55/s	8199.88/s	-3.7%
GET /test/set/srandmember	8483.07/s	11648.59/s	+37.3%
POST /test/set/sdiff	8526.90/s	11792.33/s	+38.3%
POST /test/set/sinter	8494.55/s	11730.92/s	+38.1%
POST /test/set/sunion	8319.49/s	11888.32/s	+42.9%
POST /test/set/smove	7906.37/s	12712.29/s	+60.8%
Sorted Sets
POST /test/zset/zadd	8383.02/s	11887.81/s	+41.8%
GET /test/zset/zrange	8336.77/s	11861.14/s	+42.3%
GET /test/zset/zrange-withscores	8222.27/s	12330.75/s	+50.0%
GET /test/zset/zrevrange	8281.11/s	11272.89/s	+36.1%
GET /test/zset/zscore	8085.37/s	12026.75/s	+48.7%
POST /test/zset/zincrby	8178.57/s	11710.14/s	+43.2%
GET /test/zset/zcard	8236.28/s	11964.29/s	+45.3%
GET /test/zset/zcount	8528.24/s	11274.52/s	+32.2%
GET /test/zset/zrank	8342.21/s	11362.39/s	+36.2%
GET /test/zset/zrevrank	8292.82/s	11569.14/s	+39.5%
POST /test/zset/zrem	8292.28/s	11477.05/s	+38.4%
POST /test/zset/zpopmin	8691.19/s	12022.83/s	+38.3%
POST /test/zset/zpopmax	8351.99/s	12773.41/s	+52.9%
GET /test/zset/zrangebyscore	8317.92/s	12171.87/s	+46.3%
GET /test/zset/zrevrangebyscore	8280.13/s	11593.08/s	+40.0%
POST /test/zset/zremrangebyrank	8353.40/s	11542.21/s	+38.2%
POST /test/zset/zremrangebyscore	8488.58/s	11645.14/s	+37.2%
Keys
POST /test/key/del	8071.14/s	11605.97/s	+43.8%
GET /test/key/exists	8178.43/s	11871.21/s	+45.2%
POST /test/key/expire	8191.64/s	11460.43/s	+39.9%
POST /test/key/expireat	8287.20/s	11435.67/s	+38.0%
GET /test/key/ttl	8224.64/s	12017.26/s	+46.1%
GET /test/key/pttl	8175.23/s	11733.48/s	+43.5%
POST /test/key/persist	8320.39/s	11817.38/s	+42.0%
GET /test/key/keys	8441.87/s	10871.28/s	+28.8%
GET /test/key/randomkey	8020.25/s	12198.56/s	+52.1%
POST /test/key/rename	8361.59/s	12030.66/s	+43.9%
POST /test/key/renamenx	8222.41/s	11689.29/s	+42.2%
GET /test/key/type	8270.35/s	12122.13/s	+46.6%
POST /test/key/touch	8104.79/s	11575.57/s	+42.8%
POST /test/key/unlink	8402.78/s	11392.17/s	+35.6%
Bitmaps
POST /test/bitmap/setbit	8328.49/s	11963.58/s	+43.6%
GET /test/bitmap/getbit	8693.88/s	12187.12/s	+40.2%
GET /test/bitmap/bitcount	8474.17/s	11551.22/s	+36.3%
GET /test/bitmap/bitpos	8188.19/s	12204.73/s	+49.1%
POST /test/bitmap/bitop	8577.99/s	11541.58/s	+34.5%
Server
GET /test/server/ping	8394.29/s	12278.50/s	+46.3%
GET /test/server/dbsize	8187.39/s	11146.75/s	+36.1%
POST /test/server/echo	8358.03/s	12235.72/s	+46.4%
HyperLogLog
POST /test/hll/pfadd	8115.16/s	12084.74/s	+48.9%
GET /test/hll/pfcount	8436.44/s	11720.93/s	+38.9%
POST /test/hll/pfmerge	8319.12/s	12169.49/s	+46.3%
Geo
POST /test/geo/geoadd	8134.42/s	11842.94/s	+45.6%
GET /test/geo/geopos	8301.19/s	11817.20/s	+42.4%
POST /test/geo/geodist	8192.58/s	12068.14/s	+47.3%
GET /test/geo/geohash	8114.24/s	11395.12/s	+40.4%
GET /cleanup	8446.62/s	12207.21/s	+44.5%

[cubic-dev-ai]

3 issues found across 7 files

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="benchmarks2/app.ts">

<violation number="1" location="benchmarks2/app.ts:31">
P3: Use nullish coalescing (or an explicit numeric check) so a valid `0` delay isn't replaced by the default.</violation>

<violation number="2" location="benchmarks2/app.ts:32">
P2: Global `fetch` is only available in Node 18+, but the project supports Node 16. Calling `fetch` here will crash on Node 16. Import a fetch implementation (e.g., undici/node-fetch) or guard for unsupported runtimes.</violation>
</file>

<file name="benchmarks2/run_benchmark.sh">

<violation number="1" location="benchmarks2/run_benchmark.sh:6">
P2: The benchmark command is piped into `tee` without `pipefail`, so failures in benchmark.py can be ignored and the script will continue with a bad baseline. Add `set -o pipefail` to ensure pipeline failures stop the script.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

2 issues found across 5 files (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="benchmarks2/benchmark.py">

<violation number="1" location="benchmarks2/benchmark.py:75">
P2: The health check treats any HTTP response as ready. If /health returns 4xx/5xx, the loop still breaks and benchmarks run against a failing server. Capture the response and call raise_for_status (or check status_code) before breaking.</violation>
</file>

<file name="benchmarks2/app.ts">

<violation number="1" location="benchmarks2/app.ts:36">
P2: http.get only supports http:// URLs, so a https DELAY_SERVER will now fail. Consider using a protocol-aware client (http/https switch) or revert to fetch to keep HTTPS support.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

20 issues found across 122 files (changes from recent commits).

Note: This PR contains a large number of files. cubic only reviews up to 75 files per PR, so some files may not have been reviewed.

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="src/instrumentation/libraries/firestore/e2e-tests/cjs-firestore/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/firestore/e2e-tests/cjs-firestore/entrypoint.sh:16">
P2: Using `pkill -f "node"` is overly broad and can kill unrelated Node processes on the host. Track the server PID when starting `npm run dev` and terminate only that PID instead.</violation>
</file>

<file name="src/instrumentation/libraries/firestore/e2e-tests/esm-firestore/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/firestore/e2e-tests/esm-firestore/entrypoint.sh:16">
P2: `pkill -f "node"` is too broad for test cleanup and can kill unrelated Node.js processes. Track the server PID from `npm run dev &` and kill only that process to avoid flakiness in shared CI or local environments.</violation>
</file>

<file name="src/instrumentation/libraries/mysql/e2e-tests/esm-mysql/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/mysql/e2e-tests/esm-mysql/entrypoint.sh:41">
P2: Add an EXIT trap when starting the background server so it is always stopped, even if a curl command fails under `set -e`.</violation>
</file>

<file name="src/instrumentation/libraries/mysql/e2e-tests/cjs-mysql/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/mysql/e2e-tests/cjs-mysql/entrypoint.sh:92">
P2: `curl -s` does not fail on HTTP error responses, so this E2E run can pass even when endpoints return 4xx/5xx. Use `--fail` (and ideally `-S`) so test failures are detected; apply the same change to the other curl calls in this phase.</violation>
</file>

<file name="src/instrumentation/libraries/ioredis/e2e-tests/cjs-ioredis/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/ioredis/e2e-tests/cjs-ioredis/entrypoint.sh:16">
P2: `pkill -f "node"` is overly broad and can terminate unrelated Node.js processes on the host. Capture the server PID when starting it and kill only that PID to avoid collateral termination.</violation>
</file>

<file name="src/instrumentation/libraries/nextjs/e2e-tests/esm-nextjs/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/nextjs/e2e-tests/esm-nextjs/entrypoint.sh:16">
P2: `pkill -f "node"` is too broad and can terminate unrelated Node.js processes on shared runners. Track the PID of the `npm run dev` process and kill only that PID instead.</violation>
</file>

<file name="src/e2e-common/python-base/README.md">

<violation number="1" location="src/e2e-common/python-base/README.md:126">
P3: The Dockerfile example’s line continuation is broken by an inline comment after the backslash, so the snippet won’t build if copied. Move the comment to its own line or remove it so the backslash is the last character.</violation>
</file>

<file name="src/instrumentation/libraries/ioredis/e2e-tests/esm-ioredis/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/ioredis/e2e-tests/esm-ioredis/entrypoint.sh:16">
P2: `pkill -f "node"` is overly broad and can terminate unrelated Node processes. Capture the PID of the background server (`$!`) and kill that PID instead to avoid disrupting other jobs.</violation>

<violation number="2" location="src/instrumentation/libraries/ioredis/e2e-tests/esm-ioredis/entrypoint.sh:95">
P2: `curl -s` without `--fail` will treat HTTP errors as success, so the script can pass even when endpoints fail. Use `-f`/`--fail` (optionally with `-S`) so test failures are surfaced.</violation>
</file>

<file name="src/instrumentation/libraries/pg/e2e-tests/cjs-pg/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/pg/e2e-tests/cjs-pg/entrypoint.sh:14">
P2: Using `pkill -f "node"` can terminate unrelated Node processes on the host. Track the server PID when you start it (`npm run dev & SERVER_PID=$!`) and kill only that PID to avoid disrupting other tests/services.</violation>
</file>

<file name="src/e2e-common/python-base/Dockerfile">

<violation number="1" location="src/e2e-common/python-base/Dockerfile:14">
P2: `CACHEBUST` is declared to invalidate the layer cache, but it isn't used in the install RUN step, so the CLI download layer will still be cached. This makes the cache-busting build arg ineffective.</violation>
</file>

<file name="src/instrumentation/libraries/mysql2/e2e-tests/cjs-mysql2/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/mysql2/e2e-tests/cjs-mysql2/entrypoint.sh:42">
P2: Add a trap to clean up the background server on exit so failures don’t leave a stray process running.</violation>

<violation number="2" location="src/instrumentation/libraries/mysql2/e2e-tests/cjs-mysql2/entrypoint.sh:92">
P2: Use `curl -f` (or `--fail-with-body`) so HTTP error responses fail the test suite instead of being silently accepted.</violation>
</file>

<file name="src/instrumentation/libraries/grpc/e2e-tests/esm-grpc/docker-compose.yml">

<violation number="1" location="src/instrumentation/libraries/grpc/e2e-tests/esm-grpc/docker-compose.yml:19">
P2: The new bind mount only maps the config file read-only, so `/app/.tusk` output (traces) is no longer persisted to the host. If trace persistence is still required (per the comment), mount the directory instead or update the comment to avoid misleading behavior.</violation>
</file>

<file name="src/instrumentation/libraries/nextjs/e2e-tests/cjs-nextjs/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/nextjs/e2e-tests/cjs-nextjs/entrypoint.sh:16">
P2: `pkill -f "node"` will terminate any Node.js process on the machine, not just the server started by this script. This can kill unrelated services or parallel test runners. Capture the server PID from `npm run dev &` and kill that PID instead.</violation>
</file>

<file name="src/instrumentation/libraries/fetch/e2e-tests/esm-fetch/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/fetch/e2e-tests/esm-fetch/entrypoint.sh:16">
P2: `pkill -f "node"` is overly broad and can kill unrelated Node processes on the host. Capture the server PID when starting `npm run dev` and kill only that process (or the process group) to avoid disrupting other services.</violation>

<violation number="2" location="src/instrumentation/libraries/fetch/e2e-tests/esm-fetch/entrypoint.sh:67">
P2: Using a fixed `sleep 5` to wait for the server can make tests flaky if startup takes longer. Poll the health endpoint until it responds (with a timeout) before proceeding.</violation>
</file>

<file name="src/instrumentation/libraries/e2e-common/request-utils.sh">

<violation number="1" location="src/instrumentation/libraries/e2e-common/request-utils.sh:77">
P2: The request execution ignores the exit status, so failed HTTP calls are silently treated as success and E2E tests won’t fail when requests fail. Capture the exit code and return non‑zero on failure.</violation>
</file>

<file name="src/instrumentation/libraries/http/e2e-tests/cjs-http/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/http/e2e-tests/cjs-http/entrypoint.sh:107">
P2: Test requests don't fail on HTTP error responses because curl is missing `-f`. This can allow the E2E test to pass even when endpoints are returning 4xx/5xx. Add `-f` so non-2xx responses fail the script.</violation>
</file>

<file name="src/instrumentation/libraries/grpc/e2e-tests/cjs-grpc/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/grpc/e2e-tests/cjs-grpc/entrypoint.sh:16">
P1: Avoid `pkill -f "node"` because it can terminate unrelated Node.js processes on the host. Track the server PID when you background it and kill that PID instead.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

5 issues found across 24 files (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="src/instrumentation/libraries/prisma/e2e-tests/cjs-prisma/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/prisma/e2e-tests/cjs-prisma/entrypoint.sh:77">
P2: `curl -f` will treat the expected error response as a failure and, with `set -e`, abort the E2E run. For error-path endpoints, drop `-f` (or tolerate the non-zero status) so the test can proceed.</violation>

<violation number="2" location="src/instrumentation/libraries/prisma/e2e-tests/cjs-prisma/entrypoint.sh:78">
P2: `curl -f` will treat the expected 404 response as a failure and, with `set -e`, abort the E2E run. For the not-found error path, drop `-f` (or tolerate the non-zero status).</violation>

<violation number="3" location="src/instrumentation/libraries/prisma/e2e-tests/cjs-prisma/entrypoint.sh:79">
P2: `curl -f` will treat the expected validation error response as a failure and, with `set -e`, abort the E2E run. For this error-path endpoint, drop `-f` (or tolerate the non-zero status).</violation>
</file>

<file name="src/instrumentation/libraries/grpc/e2e-tests/cjs-grpc/entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/grpc/e2e-tests/cjs-grpc/entrypoint.sh:111">
P2: `curl -f` treats the expected 400 response from `/calc/divide-by-zero` as a failure. With `set -e`, this will abort the E2E run before traces/tests complete. Use a non-failing curl invocation for this endpoint.</violation>

<violation number="2" location="src/instrumentation/libraries/grpc/e2e-tests/cjs-grpc/entrypoint.sh:117">
P2: `curl -f` will exit non-zero for the expected 404 response from `/test/user-not-found`, and with `set -e` this stops the test run. Avoid `-f` for this endpoint or explicitly allow the error.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

1 issue found across 99 files (changes from recent commits).

Note: This PR contains a large number of files. cubic only reviews up to 75 files per PR, so some files may not have been reviewed.

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="src/instrumentation/libraries/e2e-common/test-utils.mjs">

<violation number="1" location="src/instrumentation/libraries/e2e-common/test-utils.mjs:48">
P2: Global `fetch` is used without a polyfill/guard, but the SDK declares support for Node >=16. Node 16 doesn’t provide global `fetch`, so these tests will crash with `ReferenceError: fetch is not defined` when run on supported runtimes.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

P2: Global fetch is used without a polyfill/guard, but the SDK declares support for Node >=16. Node 16 doesn’t provide global fetch, so these tests will crash with ReferenceError: fetch is not defined when run on supported runtimes.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/instrumentation/libraries/e2e-common/test-utils.mjs, line 48:

<comment>Global `fetch` is used without a polyfill/guard, but the SDK declares support for Node >=16. Node 16 doesn’t provide global `fetch`, so these tests will crash with `ReferenceError: fetch is not defined` when run on supported runtimes.</comment>

<file context>
@@ -0,0 +1,125 @@
+  const warmupEnd = Date.now() + BENCHMARK_WARMUP * 1000;
+  while (Date.now() < warmupEnd) {
+    try {
+      const resp = await fetch(url, fetchOptions);
+      await resp.text();
+    } catch {
</file context>

[podocarp]

Good to note this down but I won't fix it because I don't see me running benchmarks for different versions of node anytime soon. I don't think it matters, what matters is not the raw numbers but the relative impact of our SDK.

[cubic-dev-ai]

Understood — thanks for clarifying the benchmark context.

[cubic-dev-ai]

2 issues found across 39 files (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="src/instrumentation/libraries/e2e-common/base-entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/e2e-common/base-entrypoint.sh:97">
P2: Piping to `tee` masks the exit status of `node`, so failures in `test_requests.mjs` won't stop the benchmark run. Capture and check the `node` exit code (or enable pipefail) to avoid silently passing failed runs.</violation>

<violation number="2" location="src/instrumentation/libraries/e2e-common/base-entrypoint.sh:113">
P2: This pipeline hides failures from `node`, so a failed SDK benchmark run won't be caught. Capture `node`’s exit status (or enable pipefail) to ensure failures stop the script.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="src/instrumentation/libraries/e2e-common/base-entrypoint.sh">

<violation number="1" location="src/instrumentation/libraries/e2e-common/base-entrypoint.sh:246">
P2: `grep` without `-P`/`-E` does not treat `\s` as whitespace, so these patterns won’t match JSON with spaces (e.g., `"passed": true`). That can make the test results appear empty even when tests ran.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}