Hugging Face's logo

allenai
/
Llama-3.1-Tulu-3-405B

Text Generation
Transformers
PyTorch
Safetensors
English
llama
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
1
vwxyzjn commited on
Commit
36f3570
·
verified ·
1 Parent(s): 19c41ca

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +218 -0
README.md ADDED
@@ -0,0 +1,218 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: llama3.1
3
+ language:
4
+ - en
5
+ pipeline_tag: text-generation
6
+ datasets:
7
+ - allenai/RLVR-GSM-MATH-IF-Mixed-Constraints
8
+ base_model:
9
+ - allenai/Llama-3.1-Tulu-3-405B-DPO
10
+ library_name: transformers
11
+ ---
12
+
13
+ <img src="https://huggingface.co/datasets/allenai/blog-images/resolve/main/tulu3/Tulu3-logo.png" alt="Tulu 3 banner" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
14
+
15
+ # Llama-3.1-Tulu-3-405B
16
+
17
+ Tülu3 is a leading instruction following model family, offering fully open-source data, code, and recipes designed to serve as a comprehensive guide for modern post-training techniques.
18
+ Tülu3 is designed for state-of-the-art performance on a diversity of tasks in addition to chat, such as MATH, GSM8K, and IFEval.
19
+
20
+ ## Model description
21
+
22
+ - **Model type:** A model trained on a mix of publicly available, synthetic and human-created datasets.
23
+ - **Language(s) (NLP):** Primarily English
24
+ - **License:** Llama 3.1 Community License Agreement
25
+ - **Finetuned from model:** allenai/Llama-3.1-Tulu-3-405B-DPO
26
+
27
+ ### Model Sources
28
+
29
+ - **Training Repository:** https://github.com/allenai/open-instruct
30
+ - **Eval Repository:** https://github.com/allenai/olmes
31
+ - **Paper:** https://arxiv.org/abs/2411.15124
32
+ - **Demo:** https://playground.allenai.org/
33
+
34
+ ### Model Family
35
+
36
+ | **Stage** | **Llama 3.1 8B** | **Llama 3.1 70B** |
37
+ |----------------------|----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|
38
+ | **Base Model** | [meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B) | [meta-llama/Llama-3.1-70B](https://huggingface.co/meta-llama/Llama-3.1-70B) |
39
+ | **SFT** | [allenai/Llama-3.1-Tulu-3-8B-SFT](https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B-SFT) | [allenai/Llama-3.1-Tulu-3-70B-SFT](https://huggingface.co/allenai/Llama-3.1-Tulu-3-70B-SFT) |
40
+ | **DPO** | [allenai/Llama-3.1-Tulu-3-8B-DPO](https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B-DPO) | [allenai/Llama-3.1-Tulu-3-70B-DPO](https://huggingface.co/allenai/Llama-3.1-Tulu-3-70B-DPO) |
41
+ | **Final Models (RLVR)** | [allenai/Llama-3.1-Tulu-3-8B](https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B) | [allenai/Llama-3.1-Tulu-3-70B](https://huggingface.co/allenai/Llama-3.1-Tulu-3-70B) |
42
+ | **Reward Model (RM)**| [allenai/Llama-3.1-Tulu-3-8B-RM](https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B-RM) | (Same as 8B) |
43
+
44
+ | **Stage** | **Llama 3.1 405B** |
45
+ |-----------|-------------------|
46
+ | **Base Model** | [meta-llama/llama-3.1-405B](https://huggingface.co/meta-llama/llama-3.1-405B) |
47
+ | **SFT** | [allenai/llama-3.1-Tulu-3-405B-SFT](https://huggingface.co/allenai/llama-3.1-Tulu-3-405B-SFT) |
48
+ | **Final Model (DPO)** | [allenai/llama-3.1-Tulu-3-405B](https://huggingface.co/allenai/llama-3.1-Tulu-3-405B) |
49
+ | **Reward Model (RM)**| (Same as 8B)
50
+
51
+
52
+ ## Using the model
53
+
54
+ ### Loading with HuggingFace
55
+
56
+ To load the model with HuggingFace, use the following snippet:
57
+ ```
58
+ from transformers import AutoModelForCausalLM
59
+
60
+ tulu_model = AutoModelForCausalLM.from_pretrained("allenai/Llama-3.1-Tulu-3-405B")
61
+ ```
62
+
63
+ ### VLLM
64
+
65
+ As a Llama base model, the model can be easily served with:
66
+ ```
67
+ vllm serve allenai/Llama-3.1-Tulu-3-405B
68
+ ```
69
+ Note that given the long chat template of Llama, you may want to use `--max_model_len=8192`.
70
+
71
+ ### Chat template
72
+
73
+ The chat template for our models is formatted as:
74
+ ```
75
+ <|user|>\nHow are you doing?\n<|assistant|>\nI'm just a computer program, so I don't have feelings, but I'm functioning as expected. How can I assist you today?<|endoftext|>
76
+ ```
77
+ Or with new lines expanded:
78
+ ```
79
+ <|user|>
80
+ How are you doing?
81
+ <|assistant|>
82
+ I'm just a computer program, so I don't have feelings, but I'm functioning as expected. How can I assist you today?<|endoftext|>
83
+ ```
84
+ It is embedded within the tokenizer as well, for `tokenizer.apply_chat_template`.
85
+
86
+ ### System prompt
87
+
88
+ In Ai2 demos, we use this system prompt by default:
89
+ ```
90
+ You are Tulu 3, a helpful and harmless AI Assistant built by the Allen Institute for AI.
91
+ ```
92
+ The model has not been trained with a specific system prompt in mind.
93
+
94
+ ### Bias, Risks, and Limitations
95
+
96
+ The Tülu3 models have limited safety training, but are not deployed automatically with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so).
97
+ It is also unknown what the size and composition of the corpus was used to train the base Llama 3.1 models, however it is likely to have included a mix of Web data and technical sources like books and code.
98
+ See the Falcon 180B model card for an example of this.
99
+
100
+
101
+ ## Performance
102
+
103
+ | Benchmark (eval) | Tülu 3 SFT 8B | Tülu 3 DPO 8B | Tülu 3 8B | Llama 3.1 8B Instruct | Qwen 2.5 7B Instruct | Magpie 8B | Gemma 2 9B Instruct | Ministral 8B Instruct |
104
+ |---------------------------------|----------------|----------------|------------|------------------------|----------------------|-----------|---------------------|-----------------------|
105
+ | **Avg.** | 60.4 | 64.4 | **64.8** | 62.2 | 57.8 | 44.7 | 55.2 | 58.3 |
106
+ | **MMLU (0 shot, CoT)** | 65.9 | 68.7 | 68.2 | 71.2 | **76.6** | 62.0 | 74.6 | 68.5 |
107
+ | **PopQA (15 shot)** | **29.3** | 29.3 | 29.1 | 20.2 | 18.1 | 22.5 | 28.3 | 20.2 |
108
+ | **TruthfulQA (6 shot)** | 46.8 | 56.1 | 55.0 | 55.1 | **63.1** | 57.0 | 61.4 | 55.5 |
109
+ | **BigBenchHard (3 shot, CoT)** | **67.9** | 65.8 | 66.0 | 62.8 | 21.7 | 0.9 | 2.5 | 56.2 |
110
+ | **DROP (3 shot)** | 61.3 | 62.5 | **62.6** | 61.5 | 54.4 | 49.4 | 58.8 | 56.2 |
111
+ | **MATH (4 shot CoT, Flex)** | 31.5 | 42.0 | **43.7** | 42.5 | 14.8 | 5.1 | 29.8 | 40.0 |
112
+ | **GSM8K (8 shot, CoT)** | 76.2 | 84.3 | **87.6** | 83.4 | 83.8 | 61.2 | 79.7 | 80.0 |
113
+ | **HumanEval (pass@10)** | 86.2 | 83.9 | 83.9 | 86.3 | **93.1** | 75.4 | 71.7 | 91.0 |
114
+ | **HumanEval+ (pass@10)** | 81.4 | 78.6 | 79.2 | 82.9 | **89.7** | 69.1 | 67.0 | 88.5 |
115
+ | **IFEval (prompt loose)** | 72.8 | 81.1 | **82.4** | 80.6 | 74.7 | 38.8 | 69.9 | 56.4 |
116
+ | **AlpacaEval 2 (LC % win)** | 12.4 | 33.5 | 34.5 | 24.2 | 29.0 | **49.0** | 43.7 | 31.4 |
117
+ | **Safety (6 task avg.)** | **93.1** | 87.2 | 85.5 | 75.2 | 75.0 | 46.4 | 75.5 | 56.2 |
118
+
119
+ | Benchmark (eval) | Tülu 3 70B SFT | Tülu 3 DPO 70B | Tülu 3 70B | Llama 3.1 70B Instruct | Qwen 2.5 72B Instruct | Hermes 3 Llama 3.1 70B | Nemotron Llama 3.1 70B |
120
+ |---------------------------------|-----------------|-----------------|-------------|-------------------------|-----------------------|------------------------|-------------------------|
121
+ | **Avg.** | 72.6 | 75.9 | **76.0** | 73.4 | 71.5 | 68.3 | 65.5 |
122
+ | **MMLU (0 shot, CoT)** | 78.9 | 83.3 | 83.1 | 85.3 | **85.5** | 80.4 | 83.8 |
123
+ | **PopQA (15 shot)** | **48.6** | 46.3 | 46.5 | 46.4 | 30.6 | 48.1 | 36.4 |
124
+ | **TruthfulQA (6 shot)** | 55.7 | 67.9 | 67.6 | 66.8 | **69.9** | 66.5 | 62.6 |
125
+ | **BigBenchHard (3 shot, CoT)** | **82.7** | 81.8 | 82.0 | 73.8 | 67.2 | 82.1 | 0.7 |
126
+ | **DROP (3 shot)** | **77.2** | 74.1 | 74.3 | 77.0 | 34.2 | 73.2 | 68.8 |
127
+ | **MATH (4 shot CoT, Flex)** | 53.7 | 62.3 | 63.0 | 56.4 | **74.3** | 41.9 | 55.0 |
128
+ | **GSM8K (8 shot, CoT)** | 91.1 | 93.5 | 93.5 | **93.7** | 89.5 | 90.0 | 84.7 |
129
+ | **HumanEval (pass@10)** | 92.9 | 92.4 | 92.4 | 93.6 | 94.0 | 89.6 | **94.1** |
130
+ | **HumanEval+ (pass@10)** | 87.3 | 88.4 | 88.0 | 89.5 | **90.8** | 85.9 | 85.5 |
131
+ | **IFEval (prompt loose)** | 82.1 | 82.6 | 83.2 | **88.0** | 87.6 | 76.0 | 79.9 |
132
+ | **AlpacaEval 2 (LC % win)** | 26.3 | 49.6 | 49.8 | 33.4 | 47.7 | 28.4 | **66.1** |
133
+ | **Safety (6 task avg.)** | **94.4** | 89.0 | 88.3 | 76.5 | 87.0 | 57.9 | 69.0 |
134
+
135
+ | Benchmark (eval) | Tülu 3 405B SFT | Tülu 3 405B DPO | Tülu 3 405B | Llama 3.1 405B Instruct | Nous Hermes 3 405B | Deepseek V3 | GPT 4o (11-24) |
136
+ |-----------------|----------------|----------------|-------------|------------------------|-------------------|-------------|----------------|
137
+ | **Avg w/o Safety** | 76.3 | 79.0 | 80.0 | 78.1 | 74.4 | 79.0 | **80.5** |
138
+ | **Avg w/ Safety** | 77.5 | 79.6 | 80.7 | 79.0 | 73.5 | 75.9 | **81.6** |
139
+ | **MMLU (5 shot, CoT)** | 84.4 | 86.6 | 87.0 | **88.0** | 84.9 | 82.1 | 87.9 |
140
+ | **PopQA (3 shot)** | **55.7** | 55.4 | 55.5 | 52.9 | 54.2 | 44.9 | 53.6 |
141
+ | **BigBenchHard (0 shot, CoT)** | 88.0 | 88.8 | 88.6 | 87.1 | 87.7 | **89.5** | 83.3 |
142
+ | **MATH (4 shot, Flex)** | 63.4 | 59.9 | 67.3 | 66.6 | 58.4 | **72.5** | 68.8 |
143
+ | **GSM8K (8 shot, CoT)** | 93.6 | 94.2 | **95.5** | 95.4 | 92.7 | 94.1 | 91.7 |
144
+ | **HumanEval (pass@10)** | 95.7 | **97.2** | 95.9 | 95.9 | 92.3 | 94.6 | 97.0 |
145
+ | **HumanEval+ (pass@10)** | 93.3 | **93.9** | 92.9 | 90.3 | 86.9 | 91.6 | 92.7 |
146
+ | **IFEval (prompt loose)** | 82.4 | 85.0 | 86.0 | **88.4** | 81.9 | 88.0 | 84.8 |
147
+ | **AlpacaEval 2 (LC % win)** | 30.4 | 49.8 | 51.4 | 38.5 | 30.2 | 53.5 | **65.0** |
148
+ | **Safety (6 task avg.)** | 87.7 | 85.5 | 86.7 | 86.8 | 65.8 | 72.2 | **90.9** |
149
+
150
+
151
+ ## Hyperparamters
152
+
153
+ PPO settings for RLVR:
154
+ - **Learning Rate**: 1 × 10⁻⁷
155
+ - **Discount Factor (gamma)**: 1.0
156
+ - **General Advantage Estimation (lambda)**: 0.95
157
+ - **Mini-batches (N_mb)**: 1
158
+ - **PPO Update Iterations (K)**: 4
159
+ - **PPO's Clipping Coefficient (epsilon)**: 0.2
160
+ - **Value Function Coefficient (c1)**: 0.1
161
+ - **Gradient Norm Threshold**: 1.0
162
+ - **Learning Rate Schedule**: Linear
163
+ - **Generation Temperature**: 1.0
164
+ - **Batch Size (effective)**: 1,856
165
+ - **Max Token Length**: 2,048
166
+ - **Max Prompt Token Length**: 2,048
167
+ - **Penalty Reward Value for Responses without an EOS Token**: -10.0
168
+ - **Response Length**: 2,048
169
+ - **Total Episodes**: 300,000
170
+ - **KL penalty coefficient (beta)**: 0.05
171
+ - **Warm up ratio (omega)**: 0.0
172
+
173
+ ## License and use
174
+
175
+ All Llama 3.1 Tülu3 models are released under Meta's [Llama 3.1 Community License Agreement](https://www.llama.com/llama3_1/license/).
176
+ Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc.
177
+ Tülu3 is intended for research and educational use.
178
+ For more information, please see our [Responsible Use Guidelines](https://allenai.org/responsible-use).
179
+
180
+ The models have been fine-tuned using a dataset mix with outputs generated from third party models and are subject to additional terms:
181
+ [Gemma Terms of Use](https://ai.google.dev/gemma/terms) and [Qwen License Agreement](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/main/LICENSE) (models were improved using Qwen 2.5).
182
+
183
+
184
+ ## Citation
185
+
186
+ If Tülu3 or any of the related materials were helpful to your work, please cite:
187
+ ```
188
+ @article{lambert2024tulu3,
189
+ title = {Tülu 3: Pushing Frontiers in Open Language Model Post-Training},
190
+ author = {
191
+ Nathan Lambert and
192
+ Jacob Morrison and
193
+ Valentina Pyatkin and
194
+ Shengyi Huang and
195
+ Hamish Ivison and
196
+ Faeze Brahman and
197
+ Lester James V. Miranda and
198
+ Alisa Liu and
199
+ Nouha Dziri and
200
+ Shane Lyu and
201
+ Yuling Gu and
202
+ Saumya Malik and
203
+ Victoria Graf and
204
+ Jena D. Hwang and
205
+ Jiangjiang Yang and
206
+ Ronan Le Bras and
207
+ Oyvind Tafjord and
208
+ Chris Wilhelm and
209
+ Luca Soldaini and
210
+ Noah A. Smith and
211
+ Yizhong Wang and
212
+ Pradeep Dasigi and
213
+ Hannaneh Hajishirzi
214
+ },
215
+ year = {2024},
216
+ email = {[email protected]}
217
+ }
218
+ ```