Fix UltraChat chat-template handling for Transformers v5

[Copilot]

Description

Transformers v5 changed apply_chat_template to return a BatchEncoding by default instead of legacy input_ids-like output. In AutoRound, the UltraChat calibration path was probing chat-template support in a way that no longer matched v5 behavior, and then unconditionally disabled chat templating even for instruct tokenizers.

• Compatibility fix

  • Switch the UltraChat instruct-tokenizer probe to tokenize=False, so capability detection is based on rendered text instead of tokenized return types.
  • Keep the probe resilient across tokenizer implementations that now return BatchEncoding.

• Behavior fix

  • Remove the unconditional fallback that forced apply_chat_template=False.
  • Only disable chat templating when the tokenizer probe actually shows that chat templates are unsupported.

• Regression coverage

  • Add focused unit coverage for:
    • a v5-style tokenizer returning BatchEncoding
    • a plain tokenizer that does not support chat templates

out = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hi"}],
    tokenize=False,
)

Type of Change

Bug fix

Related Issues

Checklist Before Submitting

• My code has been tested locally.
• Documentation has been updated as needed.
• New or updated tests are included where applicable.
• The CUDA CI has passed. You can trigger it by commenting /azp run Unit-Test-CUDA-AutoRound.