Question about config used in mle-bench experiments

[kibrq]

Hi! Thank you for open-sourcing this excellent project—it’s greatly appreciated.

I wanted to ask whether the configuration provided in the repository (excluding API keys and endpoints) is the same one used to produce the results reported in the [openai/mle-bench](https://github.com/openai/mle-bench).

If not, would it be possible to share the exact configuration used for those results?

Thank you in advance!

[kibrq]

Additionally, it would be very helpful if you could share a verbose or appropriately filtered version of the logs from your runs, as this would further support reproducibility and understanding of the results.

[YoungDubbyDu]

Thanks for your question and for your interest in the reproducibility of this project!

The configuration provided in this repository is consistent with the one we used to obtain the results on MLE-bench. It should be noted that, for the open-source release, the current codebase has been refactored and cleaned up, and due to company constraints, the original run logs cannot be directly released.

However, since the core functionality, design, and workflow have been strictly preserved, the performance of the open-source version should not differ significantly from the results reported on the leaderboard, and can maintain a high level of consistency with them.

Thank you for your understanding!

[kibrq]

Thank you for your answer!

So am I right that you did not use memory?

MLEvolve/config/config.yaml

Line 69 in b7cd3ca

use_global_memory: False

[YoungDubbyDu]

Thanks for pointing this out, and sorry for the confusion!

In our MLE-bench runs, the global memory module was indeed enabled.

The use_global_memory: False in the config was an oversight in the initial open-source release, originally set to avoid runtime issues when the memory embedding path is not specified, making the setup more robust for general users. We have now corrected this in the latest config to better reflect the actual setup.

Thanks again for the careful reading!