neuralmagic/granite-3.1-8b-instruct-FP8-dynamic · compressed-tensors MLA support requires fp8 activations and weights in group 'group

Generation isn't working and seeing this warning:

WARNING 02-03 16:24:35 config.py:991] compressed-tensors MLA support requires fp8 activations and weights in group 'group_0', but got activations type 'float' and weights type 'float'.

more logs that include full config:

INFO 02-03 16:21:10 api_server.py:838] vLLM API server version 0.7.1
INFO 02-03 16:21:10 api_server.py:839] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='neuralmagic/granite-3.1-8b-instruct-FP8-dynamic', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='fp8', max_model_len=8192, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, num_gpu_blocks_override=None, max_num_batched_tokens=8192, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['granite-3.1-8b-fp8-dynamic-l4'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, disable_log_requests=True, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 02-03 16:21:10 api_server.py:204] Started engine process with PID 50
INFO 02-03 16:21:15 __init__.py:183] Automatically detected platform cuda.
INFO 02-03 16:21:18 config.py:526] This model supports multiple tasks: {'classify', 'score', 'reward', 'generate', 'embed'}. Defaulting to 'generate'.
INFO 02-03 16:21:19 config.py:1097] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
WARNING 02-03 16:21:19 config.py:991] compressed-tensors MLA support requires fp8 activations and weights in group 'group_0', but got activations type 'float' and weights type 'float'.
WARNING 02-03 16:21:19 config.py:991]  Full config: {'config_groups': {'group_0': {'input_activations': {'actorder': None, 'block_structure': None, 'dynamic': True, 'group_size': None, 'num_bits': 8, 'observer': None, 'observer_kwargs': {}, 'strategy': 'token', 'symmetric': True, 'type': 'float'}, 'output_activations': None, 'targets': ['Linear'], 'weights': {'actorder': None, 'block_structure': None, 'dynamic': False, 'group_size': None, 'num_bits': 8, 'observer': 'mse', 'observer_kwargs': {}, 'strategy': 'channel', 'symmetric': True, 'type': 'float'}}}, 'format': 'float-quantized', 'global_compression_ratio': 1.5302256356540327, 'ignore': ['lm_head'], 'kv_cache_scheme': None, 'quant_method': 'compressed-tensors', 'quantization_status': 'compressed'}
INFO 02-03 16:21:24 config.py:526] This model supports multiple tasks: {'reward', 'generate', 'embed', 'score', 'classify'}. Defaulting to 'generate'.
INFO 02-03 16:21:25 config.py:1097] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
WARNING 02-03 16:21:25 config.py:991] compressed-tensors MLA support requires fp8 activations and weights in group 'group_0', but got activations type 'float' and weights type 'float'.
WARNING 02-03 16:21:25 config.py:991]  Full config: {'config_groups': {'group_0': {'input_activations': {'actorder': None, 'block_structure': None, 'dynamic': True, 'group_size': None, 'num_bits': 8, 'observer': None, 'observer_kwargs': {}, 'strategy': 'token', 'symmetric': True, 'type': 'float'}, 'output_activations': None, 'targets': ['Linear'], 'weights': {'actorder': None, 'block_structure': None, 'dynamic': False, 'group_size': None, 'num_bits': 8, 'observer': 'mse', 'observer_kwargs': {}, 'strategy': 'channel', 'symmetric': True, 'type': 'float'}}}, 'format': 'float-quantized', 'global_compression_ratio': 1.5302256356540327, 'ignore': ['lm_head'], 'kv_cache_scheme': None, 'quant_method': 'compressed-tensors', 'quantization_status': 'compressed'}
INFO 02-03 16:21:25 llm_engine.py:232] Initializing a V0 LLM engine (v0.7.1) with config: model='neuralmagic/granite-3.1-8b-instruct-FP8-dynamic', speculative_config=None, tokenizer='neuralmagic/granite-3.1-8b-instruct-FP8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=fp8,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=granite-3.1-8b-fp8-dynamic-l4, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
WARNING 02-03 16:21:25 config.py:991] compressed-tensors MLA support requires fp8 activations and weights in group 'group_0', but got activations type 'float' and weights type 'float'.
WARNING 02-03 16:21:25 config.py:991]  Full config: {'config_groups': {'group_0': {'input_activations': {'actorder': None, 'block_structure': None, 'dynamic': True, 'group_size': None, 'num_bits': 8, 'observer': None, 'observer_kwargs': {}, 'strategy': 'token', 'symmetric': True, 'type': 'float'}, 'output_activations': None, 'targets': ['Linear'], 'weights': {'actorder': None, 'block_structure': None, 'dynamic': False, 'group_size': None, 'num_bits': 8, 'observer': 'mse', 'observer_kwargs': {}, 'strategy': 'channel', 'symmetric': True, 'type': 'float'}}}, 'format': 'float-quantized', 'global_compression_ratio': 1.5302256356540327, 'ignore': ['lm_head'], 'kv_cache_scheme': None, 'quant_method': 'compressed-tensors', 'quantization_status': 'compressed'}
INFO 02-03 16:21:25 cuda.py:169] Using FlashInfer backend.
INFO 02-03 16:21:26 model_runner.py:1111] Starting to load model neuralmagic/granite-3.1-8b-instruct-FP8-dynamic...
INFO 02-03 16:21:26 weight_utils.py:251] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  1.06it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.25s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.21s/it]

WARNING 02-03 16:21:46 kv_cache.py:83] Using KV cache scaling factor 1.0 for fp8_e4m3. This may cause accuracy issues. Please make sure k/v_scale scaling factors are available in the fp8 checkpoint.
INFO 02-03 16:21:46 model_runner.py:1116] Loading model weights took 7.8353 GB
WARNING 02-03 16:21:46 config.py:991] compressed-tensors MLA support requires fp8 activations and weights in group 'group_0', but got activations type 'float' and weights type 'float'.
WARNING 02-03 16:21:46 config.py:991]  Full config: {'config_groups': {'group_0': {'input_activations': {'actorder': None, 'block_structure': None, 'dynamic': True, 'group_size': None, 'num_bits': 8, 'observer': None, 'observer_kwargs': {}, 'strategy': 'token', 'symmetric': True, 'type': 'float'}, 'output_activations': None, 'targets': ['Linear'], 'weights': {'actorder': None, 'block_structure': None, 'dynamic': False, 'group_size': None, 'num_bits': 8, 'observer': 'mse', 'observer_kwargs': {}, 'strategy': 'channel', 'symmetric': True, 'type': 'float'}}}, 'format': 'float-quantized', 'global_compression_ratio': 1.5302256356540327, 'ignore': ['lm_head'], 'kv_cache_scheme': None, 'quant_method': 'compressed-tensors', 'quantization_status': 'compressed'}
WARNING 02-03 16:21:46 config.py:991] compressed-tensors MLA support requires fp8 activations and weights in group 'group_0', but got activations type 'float' and weights type 'float'.
WARNING 02-03 16:21:46 config.py:991]  Full config: {'config_groups': {'group_0': {'input_activations': {'actorder': None, 'block_structure': None, 'dynamic': True, 'group_size': None, 'num_bits': 8, 'observer': None, 'observer_kwargs': {}, 'strategy': 'token', 'symmetric': True, 'type': 'float'}, 'output_activations': None, 'targets': ['Linear'], 'weights': {'actorder': None, 'block_structure': None, 'dynamic': False, 'group_size': None, 'num_bits': 8, 'observer': 'mse', 'observer_kwargs': {}, 'strategy': 'channel', 'symmetric': True, 'type': 'float'}}}, 'format': 'float-quantized', 'global_compression_ratio': 1.5302256356540327, 'ignore': ['lm_head'], 'kv_cache_scheme': None, 'quant_method': 'compressed-tensors', 'quantization_status': 'compressed'}
WARNING 02-03 16:21:50 config.py:991] compressed-tensors MLA support requires fp8 activations and weights in group 'group_0', but got activations type 'float' and weights type 'float'.
WARNING 02-03 16:21:50 config.py:991]  Full config: {'config_groups': {'group_0': {'input_activations': {'actorder': None, 'block_structure': None, 'dynamic': True, 'group_size': None, 'num_bits': 8, 'observer': None, 'observer_kwargs': {}, 'strategy': 'token', 'symmetric': True, 'type': 'float'}, 'output_activations': None, 'targets': ['Linear'], 'weights': {'actorder': None, 'block_structure': None, 'dynamic': False, 'group_size': None, 'num_bits': 8, 'observer': 'mse', 'observer_kwargs': {}, 'strategy': 'channel', 'symmetric': True, 'type': 'float'}}}, 'format': 'float-quantized', 'global_compression_ratio': 1.5302256356540327, 'ignore': ['lm_head'], 'kv_cache_scheme': None, 'quant_method': 'compressed-tensors', 'quantization_status': 'compressed'}
WARNING 02-03 16:21:50 config.py:991] compressed-tensors MLA support requires fp8 activations and weights in group 'group_0', but got activations type 'float' and weights type 'float'.
WARNING 02-03 16:21:50 config.py:991]  Full config: {'config_groups': {'group_0': {'input_activations': {'actorder': None, 'block_structure': None, 'dynamic': True, 'group_size': None, 'num_bits': 8, 'observer': None, 'observer_kwargs': {}, 's