Hugging Face's logo

Lighthouz AI

company
https://lighthouz.ai/
Activity Feed

AI & ML interests

None defined yet.

Recent Activity

clefourrier  authored a paper 5 days ago
SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
clefourrier  authored a paper 2 months ago
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
rohankaran  updated a Space 6 months ago
lighthouzai/guardrails-arena
View all activity

lighthouzai's activity

clefourrier 
authored a paper 5 days ago
clefourrier 
authored a paper 2 months ago
rohankaran 
updated a Space 6 months ago
clefourrier 
authored 2 papers 8 months ago
clefourrier 
posted an update 10 months ago
view post
Post
5857
In a basic chatbots, errors are annoyances. In medical LLMs, errors can have life-threatening consequences 🩸

It's therefore vital to benchmark/follow advances in medical LLMs before even thinking about deployment.

This is why a small research team introduced a medical LLM leaderboard, to get reproducible and comparable results between LLMs, and allow everyone to follow advances in the field.

openlifescienceai/open_medical_llm_leaderboard

Congrats to @aaditya and @pminervini !
Learn more in the blog: https://huggingface.co/blog/leaderboard-medicalllm
clefourrier 
posted an update 10 months ago
view post
Post
4746
Contamination free code evaluations with LiveCodeBench! 🖥️

LiveCodeBench is a new leaderboard, which contains:
- complete code evaluations (on code generation, self repair, code execution, tests)
- my favorite feature: problem selection by publication date 📅

This feature means that you can get model scores averaged only on new problems out of the training data. This means... contamination free code evals! 🚀

Check it out!

Blog: https://huggingface.co/blog/leaderboard-livecodebench
Leaderboard: livecodebench/leaderboard

Congrats to @StringChaos @minimario @xu3kev @kingh0730 and @FanjiaYan for the super cool leaderboard!
clefourrier 
posted an update 10 months ago
view post
Post
2226
🆕 Evaluate your RL agents - who's best at Atari?🏆

The new RL leaderboard evaluates agents in 87 possible environments (from Atari 🎮 to motion control simulations🚶and more)!

When you submit your model, it's run and evaluated in real time - and the leaderboard displays small videos of the best model's run, which is super fun to watch! ✨

Kudos to @qgallouedec for creating and maintaining the leaderboard!
Let's find out which agent is the best at games! 🚀

open-rl-leaderboard/leaderboard
clefourrier 
posted an update 10 months ago
view post
Post
2231
Fun fact about evaluation, part 2!

How much do scores change depending on prompt format choice?

Using different prompts (all present in the literature, from Prompt question? to Question: prompt question?\nChoices: enumeration of all choices\nAnswer: ), we get a score range of...

10 points for a single model!
Keep in mind that we only changed the prompt, not the evaluation subsets, etc.
Again, this confirms that evaluation results reported without their details are basically bullshit.

Prompt format on the x axis, all these evals look at the logprob of either "choice A/choice B..." or "A/B...".

Incidentally, it also changes model rankings - so a "best" model might only be best on one type of prompt...
srijankedia 
authored 11 papers 10 months ago