Skip to content

GGUF + torch.compile is not compatible with Hub Kernels #13151

@DN6

Description

@DN6

The set checks when using hub kernels with GGUF are not traceable with torch.compile.

DEQUANT_TYPES = STANDARD_QUANT_TYPES | KQUANT_TYPES | IMATRIX_QUANT_TYPES
MMVQ_QUANT_TYPES = STANDARD_QUANT_TYPES | KQUANT_TYPES | IMATRIX_QUANT_TYPES
MMQ_QUANT_TYPES = STANDARD_QUANT_TYPES | KQUANT_TYPES

The GGUF Kernels also need to be updated to support the latest version of torch.

Initial Solution has been proposed: #12556

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions