Hugging Face's logo

krutrim-ai-labs
/
Krutrim-2-instruct

PyTorch
mistral
Krutrim
language-model
Model card Files Community
10
krutrim-admin commited on
Commit
b6400af
·
verified ·
1 Parent(s): d8eb19f

Updated Indic evals

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -94,10 +94,10 @@ We use the LM Evaluation Harness to evaluate our model on the En benchmarks task
94
  |--------------------------------------------|------------|--------------|----------------|--------------|--------------|----------------|--------|
95
  | IndicSentiment (0-shot) | Accuracy | 0.65 | 0.70 | 0.95 | 0.96 |0.99 | 0.98 |
96
  | IndicCOPA (0-shot) | Accuracy | 0.51 | 0.58 | 0.80 | 0.83 | 0.88 | 0.91 |
97
- | IndicXParaphrase (0-shot) | Accuracy | 0.67 | 0.74 | 0.88 | 0.87 | 0.89 | TBD |
98
- | IndicXNLI (0-shot) | Accuracy | 0.47 | 0.54 | 0.55 | TBD | TBD | 0.67 |
99
  | IndicQA (0-shot) | Bert Score | 0.90 | 0.90 | 0.91 | TBD | TBD | TBD |
100
- | CrossSumIN (1-shot) | chrF++ | 0.04 | 0.17 | 0.21 | 0.26 | 0.24 | TBD |
101
  | FloresIN Translation xx-en (1-shot) | chrF++ | 0.54 | 0.50 | 0.58 | 0.60 | 0.62 | 0.63 |
102
  | FloresIN Translation en-xx (1-shot) | chrF++ | 0.41 | 0.34 | 0.48 | 0.46 | 0.47 | 0.48 |
103
  | IN22 Translation xx-en (0-shot) | chrF++ | 0.50 | 0.48 | 0.57 | 0.58 | 0.55 | 0.55 |
 
94
  |--------------------------------------------|------------|--------------|----------------|--------------|--------------|----------------|--------|
95
  | IndicSentiment (0-shot) | Accuracy | 0.65 | 0.70 | 0.95 | 0.96 |0.99 | 0.98 |
96
  | IndicCOPA (0-shot) | Accuracy | 0.51 | 0.58 | 0.80 | 0.83 | 0.88 | 0.91 |
97
+ | IndicXParaphrase (0-shot) | Accuracy | 0.67 | 0.74 | 0.88 | 0.87 | 0.89 | 0.91 |
98
+ | IndicXNLI (0-shot) | Accuracy | 0.47 | 0.54 | 0.55 | 0.61 | TBD | 0.67 |
99
  | IndicQA (0-shot) | Bert Score | 0.90 | 0.90 | 0.91 | TBD | TBD | TBD |
100
+ | CrossSumIN (1-shot) | chrF++ | 0.04 | 0.17 | 0.21 | 0.26 | 0.24 | 0.24 |
101
  | FloresIN Translation xx-en (1-shot) | chrF++ | 0.54 | 0.50 | 0.58 | 0.60 | 0.62 | 0.63 |
102
  | FloresIN Translation en-xx (1-shot) | chrF++ | 0.41 | 0.34 | 0.48 | 0.46 | 0.47 | 0.48 |
103
  | IN22 Translation xx-en (0-shot) | chrF++ | 0.50 | 0.48 | 0.57 | 0.58 | 0.55 | 0.55 |