Hugging Face's logo

medmekk
/
Llama-3.2-1B-Instruct.GGUF

GGUF
Inference Endpoints
imatrix
conversational
Model card Files Community
Llama-3.2-1B-Instruct.GGUF / README.md
medmekk's picture
medmekk HF staff
Upload quantized models
52fc2aa verified
preview code
|
raw
history blame contribute delete
1.4 kB

medmekk/Llama-3.2-1B-Instruct.GGUF

GGUF quantized versions of meta-llama/Llama-3.2-1B-Instruct

Available Formats:

  • Q2_K: Llama-3.2-1B-Instruct-Q2_K.gguf
  • Q3_K_S: Llama-3.2-1B-Instruct-Q3_K_S.gguf
  • Q3_K_M: Llama-3.2-1B-Instruct-Q3_K_M.gguf
  • Q3_K_L: Llama-3.2-1B-Instruct-Q3_K_L.gguf
  • Q4_0: Llama-3.2-1B-Instruct-Q4_0.gguf
  • Q4_K_S: Llama-3.2-1B-Instruct-Q4_K_S.gguf
  • Q4_K_M: Llama-3.2-1B-Instruct-Q4_K_M.gguf
  • Q5_0: Llama-3.2-1B-Instruct-Q5_0.gguf
  • Q5_K_S: Llama-3.2-1B-Instruct-Q5_K_S.gguf
  • Q5_K_M: Llama-3.2-1B-Instruct-Q5_K_M.gguf
  • Q6_K: Llama-3.2-1B-Instruct-Q6_K.gguf
  • Q8_0: Llama-3.2-1B-Instruct-Q8_0.gguf
  • IQ3_M_IMAT: Llama-3.2-1B-Instruct-IQ3_M_imat.gguf
  • IQ3_XXS_IMAT: Llama-3.2-1B-Instruct-IQ3_XXS_imat.gguf
  • Q4_K_M_IMAT: Llama-3.2-1B-Instruct-Q4_K_M_imat.gguf
  • Q4_K_S_IMAT: Llama-3.2-1B-Instruct-Q4_K_S_imat.gguf
  • IQ4_NL_IMAT: Llama-3.2-1B-Instruct-IQ4_NL_imat.gguf
  • IQ4_XS_IMAT: Llama-3.2-1B-Instruct-IQ4_XS_imat.gguf
  • Q5_K_M_IMAT: Llama-3.2-1B-Instruct-Q5_K_M_imat.gguf
  • Q5_K_S_IMAT: Llama-3.2-1B-Instruct-Q5_K_S_imat.gguf

Usage with llama.cpp: 

# CLI:
llama-cli --hf-repo medmekk/Llama-3.2-1B-Instruct.GGUF --hf-file MODEL_FILE -p "Your prompt"

# Server:
llama-server --hf-repo medmekk/Llama-3.2-1B-Instruct.GGUF --hf-file MODEL_FILE -c 2048