| # medmekk/Llama-3.2-1B-Instruct.GGUF | |
| GGUF quantized versions of [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) | |
| ## Available Formats: | |
| - `Q2_K`: Llama-3.2-1B-Instruct-Q2_K.gguf | |
| - `Q3_K_S`: Llama-3.2-1B-Instruct-Q3_K_S.gguf | |
| - `Q3_K_M`: Llama-3.2-1B-Instruct-Q3_K_M.gguf | |
| - `Q3_K_L`: Llama-3.2-1B-Instruct-Q3_K_L.gguf | |
| - `Q4_0`: Llama-3.2-1B-Instruct-Q4_0.gguf | |
| - `Q4_K_S`: Llama-3.2-1B-Instruct-Q4_K_S.gguf | |
| - `Q4_K_M`: Llama-3.2-1B-Instruct-Q4_K_M.gguf | |
| - `Q5_0`: Llama-3.2-1B-Instruct-Q5_0.gguf | |
| - `Q5_K_S`: Llama-3.2-1B-Instruct-Q5_K_S.gguf | |
| - `Q5_K_M`: Llama-3.2-1B-Instruct-Q5_K_M.gguf | |
| - `Q6_K`: Llama-3.2-1B-Instruct-Q6_K.gguf | |
| - `Q8_0`: Llama-3.2-1B-Instruct-Q8_0.gguf | |
| - `IQ3_M_IMAT`: Llama-3.2-1B-Instruct-IQ3_M_imat.gguf | |
| - `IQ3_XXS_IMAT`: Llama-3.2-1B-Instruct-IQ3_XXS_imat.gguf | |
| - `Q4_K_M_IMAT`: Llama-3.2-1B-Instruct-Q4_K_M_imat.gguf | |
| - `Q4_K_S_IMAT`: Llama-3.2-1B-Instruct-Q4_K_S_imat.gguf | |
| - `IQ4_NL_IMAT`: Llama-3.2-1B-Instruct-IQ4_NL_imat.gguf | |
| - `IQ4_XS_IMAT`: Llama-3.2-1B-Instruct-IQ4_XS_imat.gguf | |
| - `Q5_K_M_IMAT`: Llama-3.2-1B-Instruct-Q5_K_M_imat.gguf | |
| - `Q5_K_S_IMAT`: Llama-3.2-1B-Instruct-Q5_K_S_imat.gguf | |
| ## Usage with llama.cpp: | |
| ```bash | |
| # CLI: | |
| llama-cli --hf-repo medmekk/Llama-3.2-1B-Instruct.GGUF --hf-file MODEL_FILE -p "Your prompt" | |
| # Server: | |
| llama-server --hf-repo medmekk/Llama-3.2-1B-Instruct.GGUF --hf-file MODEL_FILE -c 2048 | |
| ``` | |