Skip to content

chore: add import Lean (cold) benchmark#14121

Open
Kha wants to merge 4 commits into
masterfrom
push-ozoqnpzqptoz
Open

chore: add import Lean (cold) benchmark#14121
Kha wants to merge 4 commits into
masterfrom
push-ozoqnpzqptoz

Conversation

@Kha

@Kha Kha commented Jun 19, 2026

Copy link
Copy Markdown
Member

No description provided.

@Kha Kha requested a review from Garmelon June 19, 2026 13:14
@github-actions github-actions Bot added the toolchain-available A toolchain is available for this PR, at leanprover/lean4-pr-releases:pr-release-NNNN label Jun 19, 2026
@leanprover-bot

Copy link
Copy Markdown
Collaborator

Reference manual CI status:

  • ❗ Reference manual CI can not be attempted yet, as the nightly-testing-2026-06-19 tag does not exist there yet. We will retry when you push more commits. If you rebase your branch onto nightly-with-manual, reference manual CI should run now. You can force reference manual CI using the force-manual-ci label. (2026-06-19 13:52:52)

@github-actions github-actions Bot added the mathlib4-nightly-available A branch for this PR exists at leanprover-community/mathlib4-nightly-testing:lean-pr-testing-NNNN label Jun 19, 2026
@Kha

Kha commented Jun 19, 2026

Copy link
Copy Markdown
Member Author

!bench

@leanprover-radar

leanprover-radar commented Jun 19, 2026

Copy link
Copy Markdown

Benchmark results for f834803 against bc5f89f are in. There are significant results. @Kha

  • build//instructions: -175.1M (-0.00%)
  • 🟥 other exited with code 1

Small changes (2🟥)

  • 🟥 build//task-clock: +42s (+2.00%)
  • 🟥 build/profile/compilation (LCNF impure)//wall-clock: +1s (+3.58%)

@mathlib-lean-pr-testing mathlib-lean-pr-testing Bot added the builds-mathlib CI has verified that Mathlib builds against this PR label Jun 19, 2026
@mathlib-lean-pr-testing

mathlib-lean-pr-testing Bot commented Jun 19, 2026

Copy link
Copy Markdown

Mathlib CI status (docs):

@Garmelon

Copy link
Copy Markdown
Contributor

!bench

@leanprover-radar

leanprover-radar commented Jun 19, 2026

Copy link
Copy Markdown

Benchmark results for 9da5dad against bc5f89f are in. There are significant results. @Garmelon

  • 🟥 build//instructions: +241.2M (+0.00%)

Large changes (2✅)

  • misc/import Lean//maxrss: -13MiB (-2.50%)
  • misc/leanchecker --fresh Init//maxrss: -10MiB (-1.84%)

Medium changes (1✅)

  • misc/re-elab Init.Data.List.Basic//maxrss: -54MiB (-7.64%)

Small changes (1✅, 4🟥)

  • 🟥 build/profile/grind ring//wall-clock: +7ms (+55.33%)
  • 🟥 misc/import Lean//instructions: +24.2M (+2.10%)
  • 🟥 misc/import Lean//task-clock: +14ms (+8.85%)
  • 🟥 misc/import Lean//wall-clock: +14ms (+9.06%)
  • misc/re-elab Init.Data.List.Sublist//maxrss: -44MiB (-6.27%)

@Garmelon Garmelon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cache eviction logic is not really related to the measure logic, so there's no need for it to live in measure.py (which is manually kept in sync across different bench suites). Instead, it should just be its own script (or part of the benchmark, until we need it in more than one place).

mathlib-nightly-testing Bot pushed a commit to leanprover-community/batteries that referenced this pull request Jun 19, 2026
mathlib-nightly-testing Bot pushed a commit to leanprover-community/mathlib4-nightly-testing that referenced this pull request Jun 19, 2026
@Kha

Kha commented Jun 20, 2026

Copy link
Copy Markdown
Member Author

Yes a separate Python script would be fine. We'll need it in Mathlib as well. The bash script takes 54 SECONDS to run for me...

The bash version was pretty slow.
mathlib-nightly-testing Bot pushed a commit to leanprover-community/batteries that referenced this pull request Jun 22, 2026
mathlib-nightly-testing Bot pushed a commit to leanprover-community/mathlib4-nightly-testing that referenced this pull request Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

builds-mathlib CI has verified that Mathlib builds against this PR mathlib4-nightly-available A branch for this PR exists at leanprover-community/mathlib4-nightly-testing:lean-pr-testing-NNNN toolchain-available A toolchain is available for this PR, at leanprover/lean4-pr-releases:pr-release-NNNN

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants