A/B testing — side-by-side model/prompt comparison in chat

[Thomas-Fernandes]

Is your feature request related to a problem? Please describe.

I need to compare responses from two models (or two prompts) on the same question, but Chainlit has no way to show them side-by-side.

Describe the solution you'd like

A/B testing mode: one user message → two responses displayed in split view, with two variants running in parallel on the same conversation. The user picks the better one (A, B, or tie, with optional comment). Preference stored in the data layer.

A variant can be:

• Two different Chat Profiles (model A vs model B, or prompt A vs B)
• Same Chat Profile with different Chat Settings (e.g., temperature 0.3 vs 0.9)
• Two independent runs of the same config (to measure variance)

Describe alternatives you've considered

• Chat Profiles + manual switching between two separate conversations — no side-by-side, no structured preference capture.
• External A/B tooling (LangSmith, PromptFoo…) — works offline but doesn't capture real in-app user preferences.

Prerequisite

This feature depends on the ability to run two Chat Profiles in parallel on the same conversation. As a first step, hot Chat Profile swapping (changing profile without creating a new chat) is needed — tracked in #2899.

Example :

[FosanzDev]

@Thomas-Fernandes Looks like a feasible idea. When you say "A/B testing", does A and B mean Chat profiles or same chat profile with maybe different settings?

[Thomas-Fernandes]

@FosanzDev Both should be possible. The most flexible approach would be to allow comparing any two "variants" of the same chat, where a variant can be:

• A different Chat Profile (model A vs model B, or prompt template A vs B)
• The same Chat Profile with different Chat Settings (e.g., temperature 0.3 vs 0.9)
• Even just two independent runs of the same config (to measure variance)

The flow could be

1. user writes a message
2. system sends it to two configurations in parallel
3. Both responses stream side-by-side
4. user picks A, B, or tie (optional comment for the feedbacks)
5. preference stored in data layer

[FosanzDev]

Based on how the data layer stores the step / message in a conversation, it might be conflicting to have two different chat profiles on the same chat history.

Nevertheless let me outline an approach for this. Definitely useful

[FosanzDev]

@Thomas-Fernandes please expand the Issue description to specify that you would like to have two different Chat Profiles on the same conversation.
I think the first step would be to have a "hot chat profile swapping", so we can change chat profiles without creating a new chat. With this feature, it would be easier to implement this A/B testing.

Please feel free to raise another Issue for this Hot Swap feature. I'll take over it