fix: Elasticsearch logging exception details are missing in task logs tab

[Subham-KRLX]

Description

Fixes #63736: Restores missing stack traces in Airflow 3 logs for Elasticsearch and OpenSearch.

Changes:

• Adds error_detail to TASK_LOG_FIELDS to ensure exception data is retrieved.
• Implements _format_error_detail to render structured exception payloads into readable tracebacks for the UI.
• Adds comprehensive unit tests for formatting logic and end-to-end flow in both providers.

closes: #63736

---

Was generative AI tooling used to co-author this PR?

• Yes — Gemini(Research for changes ,Title and pr description )

[potiuk]

@Subham-KRLX This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Merge conflicts: This PR has merge conflicts with the main branch. Your branch is 14 commits behind main. Please rebase your branch (git fetch origin && git rebase origin/main), resolve the conflicts, and push again. See contributing quick start.

Note: Your branch is 14 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

[pierrejeambrun]

Can't vouch for the provider change

[pierrejeambrun]

You can remove this file. We do not use bugfix newsfragment anymore.

[eladkal]

cc @Owen-CH-Leung can you help with review?

[Owen-CH-Leung]

Thanks @Subham-KRLX.

Could you please rebase on the latest main again? There will also be a new CI test covering the E2E Elasticsearch remote logging flow. Please also add an integration test in providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py to verify reading exception/error details from Elasticsearch, and a corresponding test under TestElasticsearchRemoteLogIO.

[Subham-KRLX]

Thanks @Subham-KRLX.

Could you please rebase on the latest main again? There will also be a new CI test covering the E2E Elasticsearch remote logging flow. Please also add an integration test in providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py to verify reading exception/error details from Elasticsearch, and a corresponding test under TestElasticsearchRemoteLogIO.

I have rebased the branch on the latest main, added the requested integration test in providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py, and included a corresponding unit test under TestElasticsearchRemoteLogIO in test_es_task_handler.py.Verified locally that the new tests pass and error_detail is correctly formatted.

[Owen-CH-Leung]

I run airflow on your branch and create a task that raise Exception. The task log is not rendered.

However, I can see the exception log lines are in the elasticsearch.

[Subham-KRLX]

@Owen-CH-Leung The rendering issue was due to missing level and timestamp
field mappings from Elasticsearch (which uses levelname and @timestamp). I've now fixed this in _build_log_fields
to ensure compatibility with the Airflow 3 UI schema.I have also included the unit and integration tests you requested. Please take another look at the latest push (8ccc4a8).

[Owen-CH-Leung]

Looks good now. Thanks!

One last request: can you also make changes on airflow-e2e-tests/tests/airflow_e2e_tests/remote_log_elasticsearch_tests/test_remote_logging_elasticsearch.py to include a test to retrieve stackTrace ?

Thanks a lot for your work!

[eladkal]

Note also the integration test fails
FAILED providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py::TestElasticsearchRemoteLogIOIntegration::test_upload_and_read - AssertionError: assert 'message' in {'event': 'start'}

[eladkal]

I think there is some kind of broken rebase or you attempt to fix issues in main branch as part of this PR?
There should be no changes to amazon and the other providers as part of this PR

[Subham-KRLX]

I performed a clean rebase onto main to remove all unrelated file changes, so the branch now only contains the 7 relevant files for this fix. I also relocated the new example DAG to the provider-specific path (providers/elasticsearch/src/airflow/providers/elasticsearch/example_dags/). Finally I updated the integration test assertions to expect the event field and included the new E2E test for stack trace retrieval in the Elasticsearch remote logging suite.

[eladkal]

OK I see the problem now.
What we need to do is to convert this file into a general example file like we have in
airflow-core/src/airflow/example_dags/example_skip_dag.py
kinda the same just for failure.
Then we can reference this example in elastic tests.

What confused me was that it was elastic example dag but we don't need elastic example any example of failure dag would fit (no need to mention elastic)

[Subham-KRLX]

I moved the example DAG back to Airflow Core (airflow-core/src/airflow/example_dags/example_failed_dag.py) and converted it into a generic failure example, removing any references to Elasticsearch. I also updated the Elasticsearch E2E test to reference this new general DAG ID.