Speculative decoding with Qwen 32B + Qwen 1.5B

Example:

llama-server \
    -m qwen2.5-coder-32b-instruct-q4_k_m.gguf \
    -md qwen2.5-coder-1.5b-instruct-q4_k_m.gguf \
    --ctx-size 65536

Downloads last month: 44

GGUF

Model size

1.78B params

Architecture

qwen2

4-bit

Inference Providers NEW

Text Generation

This model is not currently available via any of the supported third-party Inference Providers, and the HF Inference API does not support gguf models with pipeline type text-generation

Model tree for ggml-org/Qwen2.5-Coder-1.5B-32B-speculative-GGUF

Base model

Qwen/Qwen2.5-32B

Finetuned

Qwen/Qwen2.5-Coder-32B

Finetuned

Qwen/Qwen2.5-Coder-32B-Instruct

Quantized

(86)

this model