[Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes by vmoens · Pull Request #3763 · pytorch/rl

vmoens · 2026-05-15T08:40:10Z

Stack from ghstack (oldest at bottom):

End-to-end recurrent PPO example targeting Isaac Lab. The script
demonstrates the collector / GAE features added below in the stack:

:class:~torchrl.collectors.MultiCollector (sync or async) with
policy_factory and a :class:~torchrl.weight_update.MultiProcessWeightSyncScheme
for parent->worker weight updates.
compact_obs=True + final_obs=True to drop ("next", obs) from
every step and carry the boundary obs under ("final", obs).
:class:~torchrl.objectives.value.GAE with shifted=True: bootstraps
from ("final", obs) at window boundaries.
:class:~torchrl.modules.LSTMModule's configurable recurrent_backend:
during collection (set_recurrent_mode=False) the LSTM auto-uses
cuDNN regardless of the backend; the configured backend applies during
training (set_recurrent_mode=True).
Optional torch.compile + :class:~tensordict.nn.CudaGraphModule
around the update step.

Isaac Lab is only imported inside the worker subprocess (via the lazy
import in make_env) — the main process never imports isaaclab,
so it stays light and owns the trainer. storing_device="cpu" keeps
the rollout off the collector GPU on the way back to the trainer.

knowledge_base/ISAACLAB.md adds the pip-based container flow we use
on the cluster (uv venv when ensurepip is missing, IsaacLab pip wheels,
the EULA / privacy-consent env vars to make Kit non-interactive) and a
recurrent-PPO-on-Isaac recipe pinning the collector-owned policy_factory,
the weight-sync scheme, and the compact_obs + shifted-GAE pairing.

[ghstack-poisoned]

pytorch-bot · 2026-05-15T08:40:14Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3763

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

Run pull request jobs on OSDC runners in shadow mode

❌ 2 New Failures, 2 Unrelated Failures

As of commit 97f297b with merge base 5d11fa3 ():

NEW FAILURES - The following jobs have failed:

Libs Tests on Linux / unittests-sklearn (3.10, 12.8) / linux-job (gh)
test/libs/test_datasets.py::TestOpenML::test_env[batch_size0-magic]
PR Label / add-label (gh)
Process completed with exit code 1.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

Build Windows Wheels / pytorch/rl / build-wheel-py3_10-cpu (gh) (trunk failure)
Process completed with exit code 1.
Build Windows Wheels / pytorch/rl / upload / upload-wheel-py3_10-cpu (gh) (trunk failure)
Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.10_cpu_x64

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-05-15T08:40:19Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

github-actions · 2026-05-15T08:40:40Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

github-actions · 2026-05-15T13:00:06Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

github-actions · 2026-05-15T13:55:58Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

End-to-end recurrent PPO example targeting Isaac Lab. The script demonstrates the collector / GAE features added below in the stack: - :class:`~torchrl.collectors.MultiCollector` (sync or async) with ``policy_factory`` and a :class:`~torchrl.weight_update.MultiProcessWeightSyncScheme` for parent->worker weight updates. - ``compact_obs=True`` + ``final_obs=True`` to drop ``("next", obs)`` from every step and carry the boundary obs under ``("final", obs)``. - :class:`~torchrl.objectives.value.GAE` with ``shifted=True``: bootstraps from ``("final", obs)`` at window boundaries. - :class:`~torchrl.modules.LSTMModule`'s configurable ``recurrent_backend``: during collection (``set_recurrent_mode=False``) the LSTM auto-uses cuDNN regardless of the backend; the configured backend applies during training (``set_recurrent_mode=True``). - Optional ``torch.compile`` + :class:`~tensordict.nn.CudaGraphModule` around the update step. Isaac Lab is only imported inside the worker subprocess (via the lazy import in ``make_env``) — the main process never imports ``isaaclab``, so it stays light and owns the trainer. ``storing_device="cpu"`` keeps the rollout off the collector GPU on the way back to the trainer. ``knowledge_base/ISAACLAB.md`` adds the pip-based container flow we use on the cluster (uv venv when ensurepip is missing, IsaacLab pip wheels, the EULA / privacy-consent env vars to make Kit non-interactive) and a recurrent-PPO-on-Isaac recipe pinning the collector-owned policy_factory, the weight-sync scheme, and the compact_obs + shifted-GAE pairing. ghstack-source-id: 464cf38 Pull-Request: #3763

github-actions · 2026-05-15T14:09:59Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

github-actions · 2026-05-15T14:10:15Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

End-to-end recurrent PPO example targeting Isaac Lab. The script demonstrates the collector / GAE features added below in the stack: - :class:`~torchrl.collectors.MultiCollector` (sync or async) with ``policy_factory`` and a :class:`~torchrl.weight_update.MultiProcessWeightSyncScheme` for parent->worker weight updates. - ``compact_obs=True`` + ``final_obs=True`` to drop ``("next", obs)`` from every step and carry the boundary obs under ``("final", obs)``. - :class:`~torchrl.objectives.value.GAE` with ``shifted=True``: bootstraps from ``("final", obs)`` at window boundaries. - :class:`~torchrl.modules.LSTMModule`'s configurable ``recurrent_backend``: during collection (``set_recurrent_mode=False``) the LSTM auto-uses cuDNN regardless of the backend; the configured backend applies during training (``set_recurrent_mode=True``). - Optional ``torch.compile`` + :class:`~tensordict.nn.CudaGraphModule` around the update step. Isaac Lab is only imported inside the worker subprocess (via the lazy import in ``make_env``) — the main process never imports ``isaaclab``, so it stays light and owns the trainer. ``storing_device="cpu"`` keeps the rollout off the collector GPU on the way back to the trainer. ``knowledge_base/ISAACLAB.md`` adds the pip-based container flow we use on the cluster (uv venv when ensurepip is missing, IsaacLab pip wheels, the EULA / privacy-consent env vars to make Kit non-interactive) and a recurrent-PPO-on-Isaac recipe pinning the collector-owned policy_factory, the weight-sync scheme, and the compact_obs + shifted-GAE pairing. ghstack-source-id: a9a0154 Pull-Request: #3763

github-actions · 2026-05-15T15:09:05Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

End-to-end recurrent PPO example targeting Isaac Lab. The script demonstrates the collector / GAE features added below in the stack: - :class:`~torchrl.collectors.MultiCollector` (sync or async) with ``policy_factory`` and a :class:`~torchrl.weight_update.MultiProcessWeightSyncScheme` for parent->worker weight updates. - ``compact_obs=True`` + ``final_obs=True`` to drop ``("next", obs)`` from every step and carry the boundary obs under ``("final", obs)``. - :class:`~torchrl.objectives.value.GAE` with ``shifted=True``: bootstraps from ``("final", obs)`` at window boundaries. - :class:`~torchrl.modules.LSTMModule`'s configurable ``recurrent_backend``: during collection (``set_recurrent_mode=False``) the LSTM auto-uses cuDNN regardless of the backend; the configured backend applies during training (``set_recurrent_mode=True``). - Optional ``torch.compile`` + :class:`~tensordict.nn.CudaGraphModule` around the update step. Isaac Lab is only imported inside the worker subprocess (via the lazy import in ``make_env``) — the main process never imports ``isaaclab``, so it stays light and owns the trainer. ``storing_device="cpu"`` keeps the rollout off the collector GPU on the way back to the trainer. ``knowledge_base/ISAACLAB.md`` adds the pip-based container flow we use on the cluster (uv venv when ensurepip is missing, IsaacLab pip wheels, the EULA / privacy-consent env vars to make Kit non-interactive) and a recurrent-PPO-on-Isaac recipe pinning the collector-owned policy_factory, the weight-sync scheme, and the compact_obs + shifted-GAE pairing. ghstack-source-id: a47ad52 Pull-Request: #3763

github-actions · 2026-05-15T15:13:36Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

github-actions · 2026-05-15T20:48:05Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

github-actions · 2026-05-15T20:48:20Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

github-actions · 2026-05-17T06:16:33Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

github-actions · 2026-05-17T16:39:35Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

github-actions · 2026-05-17T16:39:48Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

github-actions · 2026-05-18T14:58:44Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

github-actions · 2026-05-18T14:59:05Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

github-actions · 2026-05-18T18:44:07Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

github-actions · 2026-05-18T21:00:24Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[ghstack-poisoned]

github-actions · 2026-05-18T21:04:15Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

github-actions · 2026-05-18T21:09:50Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

github-actions · 2026-05-18T21:10:29Z

⚠️ PR Title Label Error

Unknown or invalid prefix [Example].

Current title: [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
`[BugFix]`	BugFix	`[BugFix] Fix memory leak in collector`
`[Feature]`	Feature	`[Feature] Add new optimizer`
`[Doc]` or `[Docs]`	Documentation	`[Doc] Update installation guide`
`[Refactor]`	Refactoring	`[Refactor] Clean up module imports`
`[CI]`	CI	`[CI] Fix workflow permissions`
`[Test]` or `[Tests]`	Tests	`[Tests] Add unit tests for buffer`
`[Environment]` or `[Environments]`	Environments	`[Environments] Add Gymnasium support`
`[Data]`	Data	`[Data] Fix replay buffer sampling`
`[Performance]` or `[Perf]`	Performance	`[Performance] Optimize tensor ops`
`[BC-Breaking]`	bc breaking	`[BC-Breaking] Remove deprecated API`
`[Deprecation]`	Deprecation	`[Deprecation] Mark old function`
`[Quality]`	Quality	`[Quality] Fix typos and add codespell`

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

End-to-end recurrent PPO example targeting Isaac Lab. The script demonstrates the collector / GAE features added below in the stack: - :class:`~torchrl.collectors.MultiCollector` (sync or async) with ``policy_factory`` and a :class:`~torchrl.weight_update.MultiProcessWeightSyncScheme` for parent->worker weight updates. - ``compact_obs=True`` + ``final_obs=True`` to drop ``("next", obs)`` from every step and carry the boundary obs under ``("final", obs)``. - :class:`~torchrl.objectives.value.GAE` with ``shifted=True``: bootstraps from ``("final", obs)`` at window boundaries. - :class:`~torchrl.modules.LSTMModule`'s configurable ``recurrent_backend``: during collection (``set_recurrent_mode=False``) the LSTM auto-uses cuDNN regardless of the backend; the configured backend applies during training (``set_recurrent_mode=True``). - Optional ``torch.compile`` + :class:`~tensordict.nn.CudaGraphModule` around the update step. Isaac Lab is only imported inside the worker subprocess (via the lazy import in ``make_env``) — the main process never imports ``isaaclab``, so it stays light and owns the trainer. ``storing_device="cpu"`` keeps the rollout off the collector GPU on the way back to the trainer. ``knowledge_base/ISAACLAB.md`` adds the pip-based container flow we use on the cluster (uv venv when ensurepip is missing, IsaacLab pip wheels, the EULA / privacy-consent env vars to make Kit non-interactive) and a recurrent-PPO-on-Isaac recipe pinning the collector-owned policy_factory, the weight-sync scheme, and the compact_obs + shifted-GAE pairing. ghstack-source-id: d512eb5 Pull-Request: #3763

Update

1bf098f

[ghstack-poisoned]

github-actions Bot added the Examples label May 15, 2026

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 15, 2026

Update

35474f1

[ghstack-poisoned]

Update

08b8158

[ghstack-poisoned]

Update

0dca5f3

[ghstack-poisoned]

vmoens mentioned this pull request May 15, 2026

[Refactor] Keep [B, T] dim in value estimators #3767

Merged

Update

4da8c2a

[ghstack-poisoned]

Update

d4e5ea8

[ghstack-poisoned]

Update

f7d79d2

[ghstack-poisoned]

This was referenced May 15, 2026

[Refactor] Simplify LSTM/GRUModule recurrent-mode shape normalization #3768

Merged

[Example] Add Isaac RNN PPO rollout mode flags #3769

Merged

Update

5c57128

[ghstack-poisoned]

Update

1197405

[ghstack-poisoned]

This was referenced May 17, 2026

[Test] Enable scan compile RNN tests on Windows #3770

Closed

[BugFix] Fix GAE compact path bias on recurrent value nets at internal truncations #3771

Merged

Update

38095e5

[ghstack-poisoned]

This was referenced May 18, 2026

[Example] Expose compact GAE cat dimension #3775

Merged

[Doc] Migrate shifted=True callers to legacy/compact + docstring polish #3776

Merged

Update

e5edf89

[ghstack-poisoned]

Update

05ceffa

[ghstack-poisoned]

Update

97f297b

[ghstack-poisoned]

vmoens merged commit 97f297b into gh/vmoens/279/base May 19, 2026
106 of 113 checks passed

vmoens deleted the gh/vmoens/279/head branch May 19, 2026 07:45

Conversation

vmoens commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3763

❗ 1 Active SEVs

❌ 2 New Failures, 2 Unrelated Failures

Uh oh!

github-actions Bot commented May 15, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 15, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 15, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 15, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 15, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 15, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 15, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 15, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 15, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 15, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 17, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 17, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 17, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 18, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 18, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 18, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 18, 2026

⚠️ PR Title Label Error

Supported Prefixes (case-sensitive)

Uh oh!

github-actions Bot commented May 18, 2026

vmoens commented May 15, 2026 •

edited

Loading

pytorch-bot Bot commented May 15, 2026 •

edited

Loading