|
--- |
|
|
|
|
|
{} |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
This model is a GPT-J 6B fine-tuned on the TL;DR dataset using RLHF (reinforcement learning from human feedback), the same |
|
technique that powers ChatGPT. |
|
|
|
The TL;DR dataset is a summarization dataset, hence this model is fine-tuned for the summarization task as well. |
|
|
|
This is likely the first open-source LLM fine-tuned on RLHF available publicly, thanks to Carper AI. |
|
|
|
It aims to recreate the results of the [original paper by OpenAI](https://arxiv.org/abs/2009.01325). |
|
|
|
# Model Details |
|
|
|
- Base Model : GPT-J 6B |
|
- Fine-Tuning Method : PPO, RLHF |
|
- Fine-Tuning Dataset: TL;DR |
|
- Fine-Tuning Task: Summarization |
|
|
|
## Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
- **Developed by:** Duy V. Phung, Ayush Thakur, Louis Castricato, Jonathan Tow, Alex Havrilla |
|
- **Finetuned from model [optional]:** GPT-J 6B |
|
|
|
## Model Sources [optional] |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** https://github.com/CarperAI/trlx/tree/main/examples/summarize_rlhf |
|
|
|
|
|
|
|
## Results |
|
|
|
SFT vs PPO |
|
|
|
__ROUGE scores__ |
|
|
|
| Model | Rouge-1 | Rouge-2 | Rouge-L | Average | |
|
| --- | --- | --- | --- | --- | |
|
| SFT | 0.334 | 0.125 | 0.261 | 0.240 | |
|
| PPO | 0.323 | 0.109 | 0.238 | 0.223 | |
|
|
|
__Reward scores__ |
|
|
|
| Model | Average Reward | Reward $\Delta$ | |
|
| --- | --- | --- | |
|
| SFT | 2.729 | -0.181 | |
|
| PPO | 3.291 | +0.411 | |
|
|
