metadata
{}
Model Card for Model ID
This model is a GPT-J 6B fine-tuned on the TL;DR dataset using RLHF (reinforcement learning from human feedback), the same technique that powers ChatGPT.
The TL;DR dataset is a summarization dataset, hence this model is fine-tuned for the summarization task as well.
This is likely the first open-source LLM fine-tuned on RLHF available publicly, thanks to Carper AI.
It aims to recreate the results of the original paper by OpenAI.
Model Details
- Base Model : GPT-J 6B
- Fine-Tuning Method : PPO, RLHF
- Fine-Tuning Dataset: TL;DR
- Fine-Tuning Task: Summarization
Model Description
- Developed by: Duy V. Phung, Ayush Thakur, Louis Castricato, Jonathan Tow, Alex Havrilla
- Finetuned from model [optional]: GPT-J 6B
Model Sources [optional]
Results
SFT vs PPO
__ROUGE scores__
| Model | Rouge-1 | Rouge-2 | Rouge-L | Average |
| --- | --- | --- | --- | --- |
| SFT | 0.334 | 0.125 | 0.261 | 0.240 |
| PPO | 0.323 | 0.109 | 0.238 | 0.223 |
__Reward scores__
| Model | Average Reward | Reward $\Delta$ |
| --- | --- | --- |
| SFT | 2.729 | -0.181 |
| PPO | 3.291 | +0.411 |