release httpx v1.9.0 by dogancanbakir · Pull Request #2435 · projectdiscovery/httpx

dogancanbakir · 2026-03-05T11:11:28Z

- Introduce `MarkDownOutput` flag and CLI option (`-markdown` / `-md`). - Implement `Result.MarkdownOutput` to generate a formatted Markdown table with optional title and CDN columns, and a response body preview. - Add markdown escaping helper. - Update runner to create `.md` output file, handle markdown generation per result, and write to file or stdout. - Adjust option parsing and related logic to include markdown handling.

- Split Markdown generation into `MarkdownHeader` and `MarkdownRow` for clearer separation. - Remove body‑preview block from Markdown rows. - Add guard in runner to write the header only once per run.

…d` flag * Consolidate output‑type checks into a single `jsonOrCsvOrMD` variable. * Adjust file‑opening logic to add the “.md” suffix only when needed. * Update related conditionals to respect the new combined flag handling. * Ensure markdown output is correctly written without creating “.md.md” files.

Add support for storing httpx scan results directly to databases with both CLI flags and YAML config file options. Supported databases: - MongoDB - PostgreSQL - MySQL Features: - Batched writes for performance (configurable batch size) - Auto-flush with configurable interval - Individual columns for each Result field (not JSON blob) - Support for environment variable HTTPX_DB_CONNECTION_STRING - Option to omit raw request/response data (-rdbor) New CLI flags under OUTPUT group: - -rdb, -result-db: enable database storage - -rdbc, -result-db-config: path to YAML config file - -rdbt, -result-db-type: database type (mongodb, postgres, mysql) - -rdbcs, -result-db-conn: connection string - -rdbn, -result-db-name: database name (default: httpx) - -rdbtb, -result-db-table: table/collection name (default: results) - -rdbbs, -result-db-batch-size: batch size (default: 100) - -rdbor, -result-db-omit-raw: omit raw data Closes #1973 Closes #2360 Closes #2361 Closes #2362

- Fix SQL injection in postgres.go using pq.QuoteIdentifier for table/index names - Fix SQL injection in mysql.go using custom quoteIdentifier function - Fix race condition in writer.go by checking closed state before channel send - Add missing idx_url index in mysql.go for parity with PostgreSQL schema - Include RawHeaders in omitRaw check for consistency

Bumps the modules group with 5 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.19` | `1.2.20` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.33` | `0.1.34` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.4` | `1.3.5` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.106` | `0.0.107` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.64` | `0.2.65` | Updates `github.com/projectdiscovery/cdncheck` from 1.2.19 to 1.2.20 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](projectdiscovery/cdncheck@v1.2.19...v1.2.20) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.33 to 0.1.34 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](projectdiscovery/networkpolicy@v0.1.33...v0.1.34) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.4 to 1.3.5 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](projectdiscovery/retryablehttp-go@v1.3.4...v1.3.5) Updates `github.com/projectdiscovery/useragent` from 0.0.106 to 0.0.107 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](projectdiscovery/useragent@v0.0.106...v0.0.107) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.64 to 0.2.65 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](projectdiscovery/wappalyzergo@v0.2.64...v0.2.65) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.20 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.34 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.3.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.107 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.65 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps the modules group with 7 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.20` | `1.2.21` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.12` | `0.8.13` | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.5.3` | `0.5.4` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.67` | `1.1.68` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.99` | `0.0.100` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.5` | `1.3.6` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.65` | `0.2.66` | Updates `github.com/projectdiscovery/cdncheck` from 1.2.20 to 1.2.21 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](projectdiscovery/cdncheck@v1.2.20...v1.2.21) Updates `github.com/projectdiscovery/dsl` from 0.8.12 to 0.8.13 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](projectdiscovery/dsl@v0.8.12...v0.8.13) Updates `github.com/projectdiscovery/fastdialer` from 0.5.3 to 0.5.4 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](projectdiscovery/fastdialer@v0.5.3...v0.5.4) Updates `github.com/projectdiscovery/gologger` from 1.1.67 to 1.1.68 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](projectdiscovery/gologger@v1.1.67...v1.1.68) Updates `github.com/projectdiscovery/hmap` from 0.0.99 to 0.0.100 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](projectdiscovery/hmap@v0.0.99...v0.0.100) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.5 to 1.3.6 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](projectdiscovery/retryablehttp-go@v1.3.5...v1.3.6) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.65 to 0.2.66 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](projectdiscovery/wappalyzergo@v0.2.65...v0.2.66) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.21 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.13 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.5.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.68 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.100 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.3.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.66 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com>

…dev/modules-60805faa73 chore(deps): bump the modules group with 7 updates

fix: prevent data loss on interrupt by implementing graceful shutdown

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.49.0 to 0.50.0. - [Commits](golang/net@v0.49.0...v0.50.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.50.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [github.com/refraction-networking/utls](https://github.com/refraction-networking/utls) from 1.7.1 to 1.8.2. - [Release notes](https://github.com/refraction-networking/utls/releases) - [Commits](refraction-networking/utls@v1.7.1...v1.8.2) --- updated-dependencies: - dependency-name: github.com/refraction-networking/utls dependency-version: 1.8.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>

…dev/golang.org/x/net-0.50.0 chore(deps): bump golang.org/x/net from 0.49.0 to 0.50.0

…github.com/refraction-networking/utls-1.8.2 chore(deps): bump github.com/refraction-networking/utls from 1.7.1 to 1.8.2

Bumps the modules group with 2 updates: [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) and [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo). Updates `github.com/projectdiscovery/cdncheck` from 1.2.21 to 1.2.22 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](projectdiscovery/cdncheck@v1.2.21...v1.2.22) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.66 to 0.2.67 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](projectdiscovery/wappalyzergo@v0.2.66...v0.2.67) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.22 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.67 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com>

feat: add database output support for storing scan results

Closes #2403

Replace page classifier with dit, add -fpt flag

Bumps [filippo.io/edwards25519](https://github.com/FiloSottile/edwards25519) from 1.1.0 to 1.1.1. - [Commits](FiloSottile/edwards25519@v1.1.0...v1.1.1) --- updated-dependencies: - dependency-name: filippo.io/edwards25519 dependency-version: 1.1.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.50.0 to 0.51.0. - [Commits](golang/net@v0.50.0...v0.51.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.51.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

…filippo.io/edwards25519-1.1.1 chore(deps): bump filippo.io/edwards25519 from 1.1.0 to 1.1.1

…dev/golang.org/x/net-0.51.0 chore(deps): bump golang.org/x/net from 0.50.0 to 0.51.0

…dev/modules-6b7541c889 chore(deps): bump the modules group with 2 updates

fix resolver parsing

neo-by-projectdiscovery-dev · 2026-03-05T11:11:41Z

Neo - PR Security Review

No security issues found

Highlights

Updates to ProjectDiscovery library dependencies (dsl v0.8.13, goflags v0.1.74, gologger v1.1.68, utils v0.9.0)
No code changes in application logic
Security scanners report clean (0 findings from TruffleHog, Semgrep, and ast-grep)

_{Comment @pdneo help for available commands. · Open in Neo}

coderabbitai · 2026-03-05T11:11:43Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8a15bc15-0c86-42d2-9537-e2036b5d1f79

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch dev

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Add Markdown output support

Bumps the modules group with 2 updates: [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) and [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo). Updates `github.com/projectdiscovery/cdncheck` from 1.2.23 to 1.2.26 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](projectdiscovery/cdncheck@v1.2.23...v1.2.26) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.68 to 0.2.71 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](projectdiscovery/wappalyzergo@v0.2.68...v0.2.71) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.26 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.71 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [github.com/lib/pq](https://github.com/lib/pq) from 1.10.9 to 1.11.2. - [Release notes](https://github.com/lib/pq/releases) - [Changelog](https://github.com/lib/pq/blob/master/CHANGELOG.md) - [Commits](lib/pq@v1.10.9...v1.11.2) --- updated-dependencies: - dependency-name: github.com/lib/pq dependency-version: 1.11.2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [go.mongodb.org/mongo-driver](https://github.com/mongodb/mongo-go-driver) from 1.17.6 to 1.17.9. - [Release notes](https://github.com/mongodb/mongo-go-driver/releases) - [Commits](mongodb/mongo-go-driver@v1.17.6...v1.17.9) --- updated-dependencies: - dependency-name: go.mongodb.org/mongo-driver dependency-version: 1.17.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

…dev/github.com/lib/pq-1.11.2 chore(deps): bump github.com/lib/pq from 1.10.9 to 1.11.2

…dev/go.mongodb.org/mongo-driver-1.17.9 chore(deps): bump go.mongodb.org/mongo-driver from 1.17.6 to 1.17.9

Bumps [github.com/weppos/publicsuffix-go](https://github.com/weppos/publicsuffix-go) from 0.50.3-0.20260104170930-90713dec78f2 to 0.50.3. - [Changelog](https://github.com/weppos/publicsuffix-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/weppos/publicsuffix-go/commits/v0.50.3) --- updated-dependencies: - dependency-name: github.com/weppos/publicsuffix-go dependency-version: 0.50.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [github.com/happyhackingspace/dit](https://github.com/happyhackingspace/dit) from 0.0.9 to 0.0.14. - [Release notes](https://github.com/happyhackingspace/dit/releases) - [Commits](HappyHackingSpace/dit@v0.0.9...v0.0.14) --- updated-dependencies: - dependency-name: github.com/happyhackingspace/dit dependency-version: 0.0.14 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

…dev/github.com/happyhackingspace/dit-0.0.14 chore(deps): bump github.com/happyhackingspace/dit from 0.0.9 to 0.0.14

…dev/modules-314bd53680 chore(deps): bump the modules group with 2 updates

…dev/github.com/weppos/publicsuffix-go-0.50.3 chore(deps): bump github.com/weppos/publicsuffix-go from 0.50.3-0.20260104170930-90713dec78f2 to 0.50.3

ayuxsec and others added 30 commits December 26, 2025 17:37

fix(md output): emit markdown table header only once per run

1517710

- Split Markdown generation into `MarkdownHeader` and `MarkdownRow` for clearer separation. - Remove body‑preview block from Markdown rows. - Add guard in runner to write the header only once per run.

fix(md output): always include Title & CDN columns and escape URLs.

cabc1ae

fix: handle errcheck lint errors

3bcbe1b

refactor: make MarkdownHeader a method on Result

bad939e

refactor: dynamic markdown header/row generation from struct tags

6399f78

fix resolver parsing

8222571

refactor: use batcher utility and StringVarEnv

c1c1fd0

Merge branch 'dev' into feature/database-output

96ac054

chore: add pull request template

374fcd1

warn when resolver looks like file path but doesn't exist

8080434

fix: prevent data loss on interrupt by implementing graceful shutdown

fa95023

Merge pull request #2386 from projectdiscovery/dependabot/go_modules/…

d54cb54

…dev/modules-60805faa73 chore(deps): bump the modules group with 7 updates

merge: reconcile Result struct md tags with dev fields

299cb60

Merge branch 'dev' into feat-add-md-output-support

c52b709

Merge pull request #2393 from assakafpix/fix/resume-lost-targets

1a11e4c

fix: prevent data loss on interrupt by implementing graceful shutdown

Merge pull request #2406 from projectdiscovery/dependabot/go_modules/…

f469225

…dev/golang.org/x/net-0.50.0 chore(deps): bump golang.org/x/net from 0.49.0 to 0.50.0

Merge pull request #2411 from projectdiscovery/dependabot/go_modules/…

b23103e

…github.com/refraction-networking/utls-1.8.2 chore(deps): bump github.com/refraction-networking/utls from 1.7.1 to 1.8.2

style: compact struct and const block

e3965ce

Merge pull request #2363 from projectdiscovery/feature/database-output

4311f8b

feat: add database output support for storing scan results

feat: replace page classifier with dit, add -fpt flag

1c4561b

Closes #2403

fix: bump golangci-lint-action to v7 for Go 1.25 support

0affea0

dogancanbakir and others added 11 commits March 3, 2026 21:03

fix: use fmt.Fprintf instead of WriteString(fmt.Sprintf) in healthcheck

0e3ceee

fix: suppress errcheck for deferred stmt.Close

30032e6

using explicit syntax

579fa05

Merge pull request #2404 from projectdiscovery/feat/dit-page-classifier

3390c41

Replace page classifier with dit, add -fpt flag

Merge pull request #2429 from projectdiscovery/dependabot/go_modules/…

7f91032

…filippo.io/edwards25519-1.1.1 chore(deps): bump filippo.io/edwards25519 from 1.1.0 to 1.1.1

Merge pull request #2427 from projectdiscovery/dependabot/go_modules/…

8cc0f05

…dev/golang.org/x/net-0.51.0 chore(deps): bump golang.org/x/net from 0.50.0 to 0.51.0

Merge branch 'dev' into dependabot/go_modules/dev/modules-6b7541c889

7f10f21

Merge pull request #2397 from projectdiscovery/dependabot/go_modules/…

7008f3b

…dev/modules-6b7541c889 chore(deps): bump the modules group with 2 updates

Merge pull request #2351 from projectdiscovery/2350_fix_resolver_parsing

3f1541a

fix resolver parsing

dogancanbakir self-assigned this Mar 5, 2026

auto-assign bot requested a review from dwisiswant0 March 5, 2026 11:11

dogancanbakir requested a review from knakul853 March 5, 2026 11:11

dogancanbakir and others added 2 commits March 5, 2026 14:12

bump version

91c489e

Merge pull request #2344 from ayuxsec/feat-add-md-output-support

ed0f6af

Add Markdown output support

knakul853 approved these changes Mar 5, 2026

View reviewed changes

dependabot bot and others added 10 commits March 9, 2026 10:55

Merge pull request #2444 from projectdiscovery/dependabot/go_modules/…

2262c02

…dev/github.com/lib/pq-1.11.2 chore(deps): bump github.com/lib/pq from 1.10.9 to 1.11.2

Merge pull request #2445 from projectdiscovery/dependabot/go_modules/…

95a1f7a

…dev/go.mongodb.org/mongo-driver-1.17.9 chore(deps): bump go.mongodb.org/mongo-driver from 1.17.6 to 1.17.9

Merge pull request #2446 from projectdiscovery/dependabot/go_modules/…

e948c3b

…dev/github.com/happyhackingspace/dit-0.0.14 chore(deps): bump github.com/happyhackingspace/dit from 0.0.9 to 0.0.14

Merge pull request #2442 from projectdiscovery/dependabot/go_modules/…

884668c

…dev/modules-314bd53680 chore(deps): bump the modules group with 2 updates

Merge pull request #2443 from projectdiscovery/dependabot/go_modules/…

73afc60

…dev/github.com/weppos/publicsuffix-go-0.50.3 chore(deps): bump github.com/weppos/publicsuffix-go from 0.50.3-0.20260104170930-90713dec78f2 to 0.50.3

dogancanbakir merged commit f66e469 into main Mar 9, 2026
11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

release httpx v1.9.0#2435

release httpx v1.9.0#2435
dogancanbakir merged 58 commits intomainfrom
dev

dogancanbakir commented Mar 5, 2026

Uh oh!

neo-by-projectdiscovery-dev bot commented Mar 5, 2026 •

edited

Loading

Uh oh!

coderabbitai bot commented Mar 5, 2026 •

edited

Loading

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

dogancanbakir commented Mar 5, 2026

Uh oh!

neo-by-projectdiscovery-dev bot commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Neo - PR Security Review

Highlights

Uh oh!

coderabbitai bot commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

neo-by-projectdiscovery-dev bot commented Mar 5, 2026 •

edited

Loading

coderabbitai bot commented Mar 5, 2026 •

edited

Loading