Cant run smoothqunat Q8 model!

[aayejinpeng]

version

OpenVINO version = 2025.4.0
OpenVINO.genai version = 2025.4.0.0

Help

I run the optimum-cli export openvino --model meta-llama/Llama-3.2-1B-Instruct --quant-mode int8 --dataset wikitext2 /home/perftest/perftool/openvino_2025.4/model/a8w8 get the smoothqunat q8 model at /home/perftest/perftool/openvino_2025.4/model/a8w8

and run optimum-cli export openvino --model meta-llama/Llama-3.2-1B-Instruct --weight-format int8 /home/perftest/perftool/openvino_2025.4/model/WOi8 get the weight-only q8 model at /home/perftest/perftool/openvino_2025.4/model/WOi8

i can run benchmark_genai with WOi8 model, but i cant run the a8w8 model

the error is:

python benchmark_genai.py -m /home/perftest/perftool/openvino_2025.4/model/a8w8
openvino runtime version: 2025.4.0-20398-7a975177ff4-releases/2025/4, genai version: 2025.4.0.0-2674-5041b1dc4e5
Traceback (most recent call last):
File "/home/perftest/perftool/openvino_2025.4/openvino.genai/samples/python/text_generation/benchmark_genai.py", line 85, in
main()
File "/home/perftest/perftool/openvino_2025.4/openvino.genai/samples/python/text_generation/benchmark_genai.py", line 53, in main
pipe = ov_genai.LLMPipeline(models_path, device, scheduler_config=scheduler_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Check 'unregistered_parameters.str().empty()' failed at src/core/src/model.cpp:267:
Model references undeclared parameters: opset1::Parameter beam_idx () -> (i32[?])
opset1::Parameter attention_mask () -> (i64[?,?])

[YuChern-Intel]

In this situation, disable continuous batching by running LLMPipeline without SchedulerConfig.

Edit the benchmark_genai.py in Line 53.
pipe = ov_genai.LLMPipeline(models_path, device)

[Wovchena]

SDPAToPagedAttention pass fails to handle quantized models apparently. Here's stack trace

#9  0x00007ffff57b2ca2 in ov::AssertFailure::create (file=0x7ffff69ed478 "/home/vzlobin/z/v/src/core/src/model.cpp", line=264,
    check_string=0x7ffff69ed4d8 "unregistered_parameters.str().empty()", context_info="",
    explanation="Model references undeclared parameters: opset1::Parameter beam_idx () -> (i32[?])\nopset1::Parameter attention_mask () -> (i64[?,?])\n")
    at /home/vzlobin/z/v/src/core/src/except.cpp:45
#10 0x00007ffff57e12ee in ov::Model::validate_nodes_and_infer_types (this=0x5555563090f0) at /home/vzlobin/z/v/src/core/src/model.cpp:264
#11 0x00007ffff5e09a44 in ov::pass::SDPAToPagedAttention::run_on_model (this=0x7fffffff8440, model=std::shared_ptr<ov::Model> (use count 1, weak count 1) = {...})
    at /home/vzlobin/z/v/src/core/src/pass/sdpa_to_paged_attention.cpp:276
#12 0x00007ffff73ac081 in ov::genai::ContinuousBatchingPipeline::ContinuousBatchingImpl::ContinuousBatchingImpl (this=0x5555561088f0,
    model=std::shared_ptr<ov::Model> (use count 1, weak count 1) = {...}, tokenizer=..., scheduler_config=..., device="CPU",
    properties=std::map with 2 elements = {...}, generation_config=..., is_validation_mode_enabled=false)
    at /home/vzlobin/z/g/src/cpp/src/continuous_batching/pipeline_impl.cpp:81
#13 0x00007ffff7333804 in std::_Construct<ov::genai::ContinuousBatchingPipeline::ContinuousBatchingImpl, std::shared_ptr<ov::Model>&, ov::genai::Tokenizer const&, ov::genai::SchedulerConfig const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ov::Any, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ov::Any> > >&, ov::genai::GenerationConfig&> (__p=0x5555561088f0)
    at /usr/include/c++/13/bits/stl_construct.h:119
#14 0x00007ffff7329d40 in std::allocator_traits<std::allocator<void> >::construct<ov::genai::ContinuousBatchingPipeline::ContinuousBatchingImpl, std::shared_ptr<ov::Model>&, ov::genai::Tokenizer const&, ov::genai::SchedulerConfig const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ov::Any, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::a--Type <RET> for more, q to quit, c to continue without paging--
llocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ov::Any> > >&, ov::genai::GenerationConfig&> (__p=0x5555561088f0) at /usr/include/c++/13/bits/alloc_traits.h:661
#15 std::_Sp_counted_ptr_inplace<ov::genai::ContinuousBatchingPipeline::ContinuousBatchingImpl, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<std::shared_ptr<ov::Model>&, ov::genai::Tokenizer const&, ov::genai::SchedulerConfig const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ov::Any, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ov::Any> > >&, ov::genai::GenerationConfig&> (this=0x5555561088e0, __a=...) at /usr/include/c++/13/bits/shared_ptr_base.h:604
#16 0x00007ffff7323277 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<ov::genai::ContinuousBatchingPipeline::ContinuousBatchingImpl, std::allocator<void>, std::shared_ptr<ov::Model>&, ov::genai::Tokenizer const&, ov::genai::SchedulerConfig const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ov::Any, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ov::Any> > >&, ov::genai::GenerationConfig&> (this=0x7fffffff89b8, __p=@0x7fffffff89b0: 0x0, __a=...) at /usr/include/c++/13/bits/shared_ptr_base.h:971
#17 0x00007ffff731afac in std::__shared_ptr<ov::genai::ContinuousBatchingPipeline::ContinuousBatchingImpl, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<void>, std::shared_ptr<ov::Model>&, ov::genai::Tokenizer const&, ov::genai::SchedulerConfig const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ov::Any, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ov::Any> > >&, ov::genai::GenerationConfig&> (this=0x7fffffff89b0, __tag=...) at /usr/include/c++/13/bits/shared_ptr_base.h:1712
#18 0x00007ffff73117e7 in std::shared_ptr<ov::genai::ContinuousBatchingPipeline::ContinuousBatchingImpl>::shared_ptr<std::allocator<void>, std::shared_ptr<ov::Model>&, ov::genai::Tokenizer const&, ov::genai::SchedulerConfig const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ov::Any, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ov::Any> > >&, ov::genai::GenerationConfig&> (this=0x7fffffff89b0, __tag=...) at /usr/include/c++/13/bits/shared_ptr.h:464
#19 0x00007ffff7307c3b in std::make_shared<ov::genai::ContinuousBatchingPipeline::ContinuousBatchingImpl, std::shared_ptr<ov::Model>&, ov::genai::Tokenizer const&, ov::genai::SchedulerConfig const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ov::Any, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ov::Any> > >&, ov::genai::GenerationConfig&> ()
    at /usr/include/c++/13/bits/shared_ptr.h:1010
#20 0x00007ffff72fe9d8 in ov::genai::ContinuousBatchingPipeline::ContinuousBatchingPipeline (this=0x55555568d790, models_path=filesystem::path "./i/a8w8" = {...},
--Type <RET> for more, q to quit, c to continue without paging--
    tokenizer=..., scheduler_config=..., device="CPU", properties=std::map with 1 element = {...})
    at /home/vzlobin/z/g/src/cpp/src/continuous_batching/pipeline.cpp:115
#21 0x00007ffff75ef7a6 in std::make_unique<ov::genai::ContinuousBatchingPipeline, std::filesystem::__cxx11::path const&, ov::genai::Tokenizer&, ov::genai::SchedulerConfig const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ov::Any, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ov::Any> > >&> () at /usr/include/c++/13/bits/unique_ptr.h:1070
#22 0x00007ffff75e8e8a in ov::genai::ContinuousBatchingAdapter::ContinuousBatchingAdapter (this=0x555555670530, models_path=filesystem::path "./i/a8w8" = {...},
    scheduler_config=..., device="CPU", plugin_config=std::map with 0 elements) at /home/vzlobin/z/g/src/cpp/src/llm/pipeline_continuous_batching_adapter.hpp:64
#23 0x00007ffff75f1efd in std::make_unique<ov::genai::ContinuousBatchingAdapter, std::filesystem::__cxx11::path const&, ov::genai::SchedulerConfig&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ov::Any, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ov::Any> > >&> () at /usr/include/c++/13/bits/unique_ptr.h:1070
#24 0x00007ffff75ed701 in ov::genai::LLMPipeline::LLMPipeline (this=0x55555566aec0, models_path=filesystem::path "./i/a8w8" = {...}, device="CPU",
    user_properties=std::map with 1 element = {...}) at /home/vzlobin/z/g/src/cpp/src/llm/pipeline.cpp:227
#25 0x00005555555858ed in ov::genai::LLMPipeline::LLMPipeline<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ov::Any>, true> (this=0x55555566aec0, models_path=filesystem::path "./i/a8w8" = {...}, device="CPU") at /home/vzlobin/z/g/src/cpp/include/openvino/genai/llm_pipeline.hpp:154
#26 0x000055555557bd49 in std::make_unique<ov::genai::LLMPipeline, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ov::Any> > () at /usr/include/c++/13/bits/unique_ptr.h:1070
#27 0x0000555555574cd7 in main (argc=3, argv=0x7fffffffb3f8) at /home/vzlobin/z/g/samples/cpp/text_generation/benchmark_genai.cpp:71