BigCodeBench: Benchmarking Large Language Models on Solving Practical and Challenging Programming Tasks
•
43
BigCode
Enterprise
non-profit
AI & ML interests
None defined yet.
Recent Activity
View all activity
Articles
bigcode's activity
teelinsan
authored
a
paper
about 8 hours ago
Post
1919
Introducing OpenR1-Math-220k!
open-r1/OpenR1-Math-220k
The community has been busy distilling DeepSeek-R1 from inference providers, but we decided to have a go at doing it ourselves from scratch 💪
What’s new compared to existing reasoning datasets?
♾ Based on AI-MO/NuminaMath-1.5: we focus on math reasoning traces and generate answers for problems in NuminaMath 1.5, an improved version of the popular NuminaMath-CoT dataset.
🐳 800k R1 reasoning traces: We generate two answers for 400k problems using DeepSeek R1. The filtered dataset contains 220k problems with correct reasoning traces.
📀 512 H100s running locally: Instead of relying on an API, we leverage vLLM and SGLang to run generations locally on our science cluster, generating 180k reasoning traces per day.
⏳ Automated filtering: We apply Math Verify to only retain problems with at least one correct answer. We also leverage Llama3.3-70B-Instruct as a judge to retrieve more correct examples (e.g for cases with malformed answers that can’t be verified with a rules-based parser)
📊 We match the performance of DeepSeek-Distill-Qwen-7B by finetuning Qwen-7B-Math-Instruct on our dataset.
🔎 Read our blog post for all the nitty gritty details: https://huggingface.co/blog/open-r1/update-2
open-r1/OpenR1-Math-220k
The community has been busy distilling DeepSeek-R1 from inference providers, but we decided to have a go at doing it ourselves from scratch 💪
What’s new compared to existing reasoning datasets?
♾ Based on AI-MO/NuminaMath-1.5: we focus on math reasoning traces and generate answers for problems in NuminaMath 1.5, an improved version of the popular NuminaMath-CoT dataset.
🐳 800k R1 reasoning traces: We generate two answers for 400k problems using DeepSeek R1. The filtered dataset contains 220k problems with correct reasoning traces.
📀 512 H100s running locally: Instead of relying on an API, we leverage vLLM and SGLang to run generations locally on our science cluster, generating 180k reasoning traces per day.
⏳ Automated filtering: We apply Math Verify to only retain problems with at least one correct answer. We also leverage Llama3.3-70B-Instruct as a judge to retrieve more correct examples (e.g for cases with malformed answers that can’t be verified with a rules-based parser)
📊 We match the performance of DeepSeek-Distill-Qwen-7B by finetuning Qwen-7B-Math-Instruct on our dataset.
🔎 Read our blog post for all the nitty gritty details: https://huggingface.co/blog/open-r1/update-2
terryyz
updated
8
datasets
4 days ago
bigcode/bigcodebench-hard-results
Viewer
•
Updated
•
180
•
126
•
1
bigcode/bigcodebench-hard-solve-rate
Viewer
•
Updated
•
296
•
156
•
1
bigcode/bigcodebench-hard-domain
Viewer
•
Updated
•
331
•
117
•
1
bigcode/bigcodebench-hard-perf
Viewer
•
Updated
•
331
•
102
bigcode/bigcodebench-results
Viewer
•
Updated
•
180
•
147
•
2
bigcode/bigcodebench-solve-rate
Viewer
•
Updated
•
2.28k
•
135
bigcode/bigcodebench-domain
Viewer
•
Updated
•
269
•
96
bigcode/bigcodebench-perf
Viewer
•
Updated
•
269
•
83
lewtun
authored
a
paper
5 days ago
clefourrier
authored
a
paper
5 days ago
canders1
authored
a
paper
7 days ago
terryyz
updated
a
Space
7 days ago
fix: c# and f# config
1
#3 opened 7 days ago
by
kevmo314
EricWallace
authored
a
paper
8 days ago
Muennighoff
authored
a
paper
8 days ago
Post
10034
We are reproducing the full DeepSeek R1 data and training pipeline so everybody can use their recipe. Instead of doing it in secret we can do it together in the open!
🧪 Step 1: replicate the R1-Distill models by distilling a high-quality reasoning corpus from DeepSeek-R1.
🧠 Step 2: replicate the pure RL pipeline that DeepSeek used to create R1-Zero. This will involve curating new, large-scale datasets for math, reasoning, and code.
🔥 Step 3: show we can go from base model -> SFT -> RL via multi-stage training.
Follow along: https://github.com/huggingface/open-r1
🧪 Step 1: replicate the R1-Distill models by distilling a high-quality reasoning corpus from DeepSeek-R1.
🧠 Step 2: replicate the pure RL pipeline that DeepSeek used to create R1-Zero. This will involve curating new, large-scale datasets for math, reasoning, and code.
🔥 Step 3: show we can go from base model -> SFT -> RL via multi-stage training.
Follow along: https://github.com/huggingface/open-r1
- 5 replies
jending12
authored
a
paper
26 days ago
BrigitteTousi
posted
an
update
about 1 month ago
Post
1178
Community fine-tuned models are more carbon efficient than the models they are derived from! 🥳🌿
@alozowski @clefourrier @SaylorTwift @albertvillanova evaluated CO₂ emissions associated with model inference for over 3000 models on the Open LLM Leaderboard. Interesting trends and new insights emerged...👀
Blog Post: https://huggingface.co/blog/leaderboard-emissions-analysis
Leaderboard: open-llm-leaderboard/open_llm_leaderboard
@alozowski @clefourrier @SaylorTwift @albertvillanova evaluated CO₂ emissions associated with model inference for over 3000 models on the Open LLM Leaderboard. Interesting trends and new insights emerged...👀
Blog Post: https://huggingface.co/blog/leaderboard-emissions-analysis
Leaderboard: open-llm-leaderboard/open_llm_leaderboard