[BugFix] Set "next" obs to current if native_autoreset

[lin-erica]

Description

This PR changes the "next" obs of the terminal state to the obs from the terminal state instead of setting it to NaN.

Motivation and Context

When using native_autoreset the next obs at a terminal state is set to NaN. However, this causes errors when calculating GAE. We could artificially replace it with a value like missing_obs_value (e.g. setting it to 0), but could lead to confusion or silent errors potentially. This PR proposes to just set it to the current obs, as that is what is assumed when calculating GAE. We could consider removing auto_reset_env in GAE, but I'm keeping it in for now to allow the flexibility.

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds core functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)
• Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3786

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures

As of commit 6c4605a with merge base 68f1ba5 ():

NEW FAILURES - The following jobs have failed:

• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  test/test_loggers.py::TestMLFlowLogger::test_log_hparams
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_loggers.py::TestMLFlowLogger::test_log_hparams
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  test/test_loggers.py::TestMLFlowLogger::test_log_hparams
• Unit-tests on Linux / tests-cpu (3.13) / linux-job (gh)
  test/test_loggers.py::TestMLFlowLogger::test_log_hparams
• Unit-tests on Linux / tests-cpu (3.14) / linux-job (gh)
  test/test_loggers.py::TestMLFlowLogger::test_log_hparams
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh)
  test/transforms/test_observation_transforms.py::TestNextObservationDelta::test_trans_parallel_env_check

This comment was automatically generated by Dr. CI and updates every 15 minutes.