OPEA
/

Llama-3.3-70B-Instruct-int4-sym-inc

4-bit precision

intel/auto-round

Model card Files Files and versions Community

cicdatopea commited on 2 days ago

Commit

5894db9

·

verified ·

1 Parent(s): 43d645d

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -24,7 +24,7 @@ tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
 model = AutoModelForCausalLM.from_pretrained(
     quantized_model_dir,
-    torch_dtype='auto',
     device_map="auto",
     ##revision="12cbcc0", ##AutoGPTQ format
 )

 model = AutoModelForCausalLM.from_pretrained(
     quantized_model_dir,
+    torch_dtype=torch.float16,
     device_map="auto",
     ##revision="12cbcc0", ##AutoGPTQ format
 )