Cosmos Transfer2.5 Auto-Regressive Inference Pipeline

[miguelmartin75]

What does this PR do?

This builds off #13066 by adding auto-regressive inference for Cosmos Transfer2.5. This pipeline does not require the controlnet or controls to be input. From the documentation:

The call function can be used in two modes: with or without controls.
When controls are not provided (controls is None), inference works in the same manner as predict2.5 (see
Cosmos2_5_PredictPipeline). This mode strictly uses the base transformer (self.transformer) to perform
inference and accepts as input an optional image or video along with a prompt / negative_prompt, and
can be used in the following ways:
- Text2World: image=None, video=None, prompt provided.
- Image2World: image provided, video=None, prompt provided.
- Video2World: video provided, image=None, prompt provided.
When controls are provided and a ControlNet is attached, controls drive the conditioning and video &
image is ignored. Controls are assumed to be pre-processed, e.g. edge maps are pre-computed.
Setting num_frames will restrict the total number of frames output, if not provided or assigned to None
(default) then the number of output frames will match the input video, image or controls respectively.
Auto-regressive inference is supported and thus a sliding window of num_frames_per_chunk frames are used per
denoising loop. In addition, when auto-regressive inference is performed, the previous
num_latent_conditional_frames or num_conditional_frames are used to condition the following denoising
inference loops.

Who can review?

• Pipelines: @yiyixuxu
• Docs: @stevhliu and @sayakpaul

[yiyixuxu]

thanks for the PR!

my main question is would it make sense to make this pipeline strictly ControlNet-focused? looking at the pipeline code, this would simplify the pipeline quite a bit

[yiyixuxu]

maybe we should make this pipeline strictly about controlnet (i.e. not make it optional) and then remove the image and video argument? this is how other controlnet behave anyways
if they want to use without controlnet, they can switich to the base pipeline

[yiyixuxu]

any reason we need two arguments here? is it possible we only keep num_conditional_frames?

[miguelmartin75]

this is done to provide the user with the option to provide either one, the official GH uses num_conditional_frames for transfer but num_latent_conditional_frames for predict so I figured it would be best to provide both