feat: Add `auto_explain` mode by nuno-faria · Pull Request #19316 · apache/datafusion

nuno-faria · 2025-12-14T11:33:25Z

Which issue does this PR close?

Closes Add auto_explain mode #19215.

Rationale for this change

Allowing users to check the execution plans without needing to change the existing application.

The auto_explain mode can be enabled with the datafusion.explain.auto_explain config. In addition, there are two other configs:

datafusion.explain.auto_explain_output: sets the output location of the plans. Supports stdout, stderr, and a file path.
datafusion.explain.auto_explain_min_duration: only outputs plans whose duration is greater than this value (similar to Postgres' auto_explain.log_min_duration).

Example in datafusion-cli:

-- regular mode
> select 1;
+----------+
| Int64(1) |
+----------+
| 1        |
+----------+
1 row(s) fetched.

-- with auto_explain enabled (the plan is not actually part of the result, it is sent to stdout)
> select 1;
+-------------------+------------------------------------------------------------------------------------------------------------------------------+
| plan_type         | plan                                                                                                                         |
+-------------------+------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | ProjectionExec: expr=[1 as Int64(1)], metrics=[output_rows=1, elapsed_compute=21.50µs, output_bytes=8.0 B, output_batches=1] |
|                   |   PlaceholderRowExec, metrics=[]                                                                                             |
|                   |                                                                                                                              |
+-------------------+------------------------------------------------------------------------------------------------------------------------------+
+----------+
| Int64(1) |
+----------+
| 1        |
+----------+
1 row(s) fetched.

What changes are included in this PR?

Extended the existing AnalyzeExec operator to support the auto_explain mode.
Added new explain configs.
Wrap plans in a AnalyzeExec operator when auto_explain is enabled.
Added tests.

Are these changes tested?

Yes.

Are there any user-facing changes?

New feature, but it's completely optional.

nuno-faria · 2025-12-14T11:35:07Z

datafusion/physical-plan/src/analyze.rs

+        self.cache =
+            Self::compute_properties(&self.input, Arc::clone(&self.input.schema()));


Needs to be recomputed since the output changes.

nuno-faria · 2025-12-14T11:36:00Z

datafusion/physical-plan/src/analyze.rs

+            if auto_explain {
+                if duration.as_millis() >= auto_explain_min_duration as u128 {
+                    export_auto_explain(out, &auto_explain_output)?;
+                }
+                concat_batches(&inner_schema, &batches).map_err(DataFusionError::from)
+            } else {
+                Ok(out)
+            }


The auto_explain mode will return the input's batches instead of the analyze.

nuno-faria · 2025-12-14T11:42:28Z

cc: @alamb @2010YOUY01 @NGA-TRAN @carpecodeum

datafusion/physical-plan/src/analyze.rs

martin-g · 2025-12-15T12:34:57Z

datafusion/physical-plan/src/analyze.rs

+    let fd: &mut dyn Write = match output {
+        "stdout" => &mut io::stdout(),
+        "stderr" => &mut io::stderr(),
+        _ => &mut OpenOptions::new().create(true).append(true).open(output)?,


Does this need any kind of validation of the file location ?
Or it is left to the developer/admin to make sure it is a safe place ?

Does this need some kind of synchronisation when a file path is used for the output ? Two or more DF sessions using the same config may try to write to the same file simultaneously.

Does this need any kind of validation of the file location ?
Or it is left to the developer/admin to make sure it is a safe place ?

I think it's better to leave this to the user (either way, an error is returned).

Does this need some kind of synchronisation when a file path is used for the output ? Two or more DF sessions using the same config may try to write to the same file simultaneously.

I think again the responsibility of this falls on the user. Is it common to use multiple sessions over the same config?

datafusion/common/src/config.rs

datafusion/physical-plan/src/analyze.rs

martin-g · 2025-12-15T12:52:37Z

datafusion/sqllogictest/test_files/explain_analyze.slt

+# test auto_explain
+
+statement ok
+set datafusion.explain.auto_explain_output = 'test_files/scratch/auto_explain.txt';


Does something assert the contents of this output file ?
Does something remove this file at the end ?

I originally tried to load the file to a table as CSV, as I think it is the only feasible way to check the contents, but since the file cannot be removed the result would always change. I mainly added these sqllogictests just to check the "set ..." commands.

As for removing the file, I'm not sure it is possible. With that said, I don't think it is necessary since it's written to the sqllogictest temporary dir.

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

alamb

Thanks @nuno-faria and @martin-g -- sorry for the delay in reviewing this PR

Form my perspective, this is a valuable feature but having the output to stdout / stderr in the core library is not idea.

Instead, it seems to me that auto explain belongs in the client applications themselves (e.g. datafusion-cli)

I tried to explain my thinking more here

#19215 (comment)

nuno-faria · 2026-03-02T20:24:23Z

@alamb thanks for the review. I also get the concerns of polluting stdout with plans, and in my case I would be setting datafusion.explain.auto_explain_output pointed to some file to avoid this with the auto_explain enabled. It was a way to simulate the auto_explain of Postgres, but since DataFusion does not have a default log, it had to be stdout/stderr (or fail if a file is not provided). As for exposing it only in datafusion-cli, in my case it would be less useful, as I'm using the library directly.

I will have to look at the Observer suggestion once I get the time.

alamb · 2026-03-04T23:55:22Z

pointed to some file to avoid this with the auto_explain enabled. It was a way to simulate the auto_explain of Postgres,

What about using log::info! or something (aka the existing logging hooks 🤔 )

nuno-faria · 2026-03-08T17:15:52Z

What about using log::info! or something (aka the existing logging hooks 🤔 )

I added log::info (as well as log::error, log::warn, ...) instead of stdout/stderr.

I was also thinking of adding the query to the output, similar to Postgre's auto_explain. It would have to come from the unparsed logical plan since the original query is not available. However, maybe some users would not need the query, or maybe some would need other info that is not outputted. So maybe it might actually be better to leave this auto_explain feature for users to implement downstream as they wish.

alamb · 2026-03-10T13:01:27Z

What about using log::info! or something (aka the existing logging hooks 🤔 )

I added log::info (as well as log::error, log::warn, ...) instead of stdout/stderr.

I was also thinking of adding the query to the output, similar to Postgre's auto_explain. It would have to come from the unparsed logical plan since the original query is not available. However, maybe some users would not need the query, or maybe some would need other info that is not outputted. So maybe it might actually be better to leave this auto_explain feature for users to implement downstream as they wish.

I think if you wanted to make it easier to implement downstream maybe we would add some sort of API / callback thing (trait object?) that could get the info for auto_explains.

Then the default implementation could log with info and downstream users could override it if they wanted

nuno-faria · 2026-03-10T20:00:49Z

I think if you wanted to make it easier to implement downstream maybe we would add some sort of API / callback thing (trait object?) that could get the info for auto_explains.

Then the default implementation could log with info and downstream users could override it if they wanted

Sounds good. I'll try to tackle this soon.

feat: Add auto_explain mode

fecb8d5

github-actions bot added documentation Improvements or additions to documentation core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) common Related to common crate physical-plan Changes to the physical-plan crate labels Dec 14, 2025

Merge branch 'main' into auto_explain

359dec3

nuno-faria commented Dec 14, 2025

View reviewed changes

Fix warnings

1a8d6f8

martin-g reviewed Dec 15, 2025

View reviewed changes

nuno-faria and others added 7 commits December 15, 2025 19:32

Avoid storing batches in AnalyzeExec when not needed

fb295d0

Keep auto_explain properties in with_new_children

ae2d6a1

Update datafusion/common/src/config.rs

0bac7ba

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

Update datafusion/physical-plan/src/analyze.rs

27d1662

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

Refactor auto_explain_min_duration to auto_explain_min_duration_ms

f574b4f

Re-run CI

1743c00

Merge branch 'main' into auto_explain

166d585

github-actions bot removed the documentation Improvements or additions to documentation label Feb 1, 2026

Update configs.md

46feaf7

github-actions bot added the documentation Improvements or additions to documentation label Feb 1, 2026

nuno-faria mentioned this pull request Feb 27, 2026

Release DataFusion 53.0.0 (Feb 2026 / Mar 2026) #19692

Open

26 tasks

alamb mentioned this pull request Mar 2, 2026

Add auto_explain mode #19215

Open

alamb reviewed Mar 2, 2026

View reviewed changes

Use log instead of stdout/stderr

0397da3

Refactor to PlanObserver

884afe7

github-actions bot added the execution Related to the execution crate label Mar 15, 2026

nuno-faria added 3 commits March 15, 2026 12:13

Merge branch 'main' into auto_explain

32fda90

Fix merge

6ddec42

Add missing license

6eace4c

		self.cache =
		Self::compute_properties(&self.input, Arc::clone(&self.input.schema()));

Conversation

nuno-faria commented Dec 14, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nuno-faria commented Dec 14, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

nuno-faria commented Mar 2, 2026

Uh oh!

alamb commented Mar 4, 2026

Uh oh!

nuno-faria commented Mar 8, 2026

Uh oh!

alamb commented Mar 10, 2026

Uh oh!

nuno-faria commented Mar 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants