|
--- |
|
language: id |
|
license: mit |
|
datasets: |
|
- indonli |
|
- MoritzLaurer/multilingual-NLI-26lang-2mil7 |
|
pipeline_tag: zero-shot-classification |
|
widget: |
|
- text: Saya suka makan kentang goreng. |
|
candidate_labels: positif, netral, negatif |
|
hypothesis_template: Kalimat ini mengandung tema {}. |
|
multi_class: false |
|
example_title: Sentiment |
|
- text: Apple umumkan harga iPhone 14. |
|
candidate_labels: teknologi, olahraga, kuliner, bisnis |
|
hypothesis_template: Kalimat ini mengandung tema {}. |
|
multi_class: true |
|
example_title: News |
|
model-index: |
|
- name: ilos-vigil/bigbird-small-indonesian-nli |
|
results: |
|
- task: |
|
type: natural-language-inference |
|
name: Natural Language Inference |
|
dataset: |
|
name: indonli |
|
type: indonli |
|
config: indonli |
|
split: test_expert |
|
metrics: |
|
- type: accuracy |
|
value: 0.5385388739946381 |
|
name: Accuracy |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNWRhZDkxNmI2NzE3MzRlYmNlMWFjZDVmNWUwYmMwN2IxYzNjMWE4YzY4NWI3NDZkYTMzY2NjN2MyZGQ5YzEwZSIsInZlcnNpb24iOjF9.AgizskHeXOzs0v93DNojNoqR_-1bQsYBokL8jcfelFm-zt-r5YXt89WXBDLLg4oKv-Roj8sLhUwe7ei0Mf1-Ag |
|
- type: f1 |
|
value: 0.530444188199697 |
|
name: F1 Macro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMjk2YTFhY2E3NGIzNzgxY2M5YzUzNGUzYTAwOWZkNGU3Y2I5MDA1MTc0YzM4Yjg0MmIzY2Y5M2EzOGYxNjY4NiIsInZlcnNpb24iOjF9.YZ_fTuVftTCM6SFfkFCLPbJWYmYNMYL9PNHUwNFHQXZeknf6OCBgQtr1gF6VM9mX6WuU4OKEl12tsAytlkm7Ag |
|
- type: f1 |
|
value: 0.5385388739946381 |
|
name: F1 Micro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2MxMGUyZmJhZTYzN2M4NDlkMTZmMzllOGVhMjRiODhkMGVkMGMxMjY2NDBkZWM3ZWY2ZjhmZTNmYWU5ZjEzMyIsInZlcnNpb24iOjF9.f0HQlPRx4VFnOOHsrvMKFni8g1B1OJfheOyADsf47GnrvCcW_dakDgBy5c_yy4TehQYRa6ToYGHnuQnemvhnBg |
|
- type: f1 |
|
value: 0.5299257731385174 |
|
name: F1 Weighted |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTgzZjJkZWU0NDgyMGU5MDFmNzk2OWY1OWY4MzA2NTE3MDAxN2Y2MWExODJkYjdlN2I1YzgzYjljNjdkMTc1YiIsInZlcnNpb24iOjF9.lWB7MZlAiDjskKM-lx-XtLxTQYuWLz3QjyseDuZe_AxtyOKt2GZkP2NDOZxEWketHjRiTCQfBUvSfzFId-FCAg |
|
- type: precision |
|
value: 0.5592571894118881 |
|
name: Precision Macro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZDQxYTFlNTNjNDAwMWIxYmJlMzRkN2U5OWY1NWNjN2YyYTE2NzRjNjM3ZWNhMzM4NjFhYWM4MzJkYjY3MzU0YSIsInZlcnNpb24iOjF9.6OI4_M1wLX1Z1BztKUfZ-382F3coCeJjarsWc-J04TKpsFCddLjuF5ZDuBFmokpz4goRgx-FlH-5jCAsFkzkBg |
|
- type: precision |
|
value: 0.5385388739946381 |
|
name: Precision Micro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzRmY2I4YTAzMTRkMjFjNTE1NTEwZDlmZGQ4NDUyYTAxY2JhOTliMDRhNWY3OGY4OWRlNTlkNzcxODc0MDMwYyIsInZlcnNpb24iOjF9.X7ekS-JYOXH5eNmSfKQ_no1rNAbuQ3C0pNYvorPVfcna6RU8n6O6FNQor0AWvatAWdefJG6H3J7_GoC6M5zECw |
|
- type: precision |
|
value: 0.5586108016541553 |
|
name: Precision Weighted |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMjUwNjMxYjEwMTEzNzAwNzQwZDQwMTRmZDM2ZDk0ZDc3YTUxOTQzNDE5ZWI2NWI4MmJmODAxYTlmN2E0Nzk2MCIsInZlcnNpb24iOjF9.nAO1wRFHMtm5kem9VhuuRg54fpvA2uzwEutjzsnZoyemUHbI2U_1TK_dDmR4bmpPjVnCZt5sF-jEq4oZIaIbDQ |
|
- type: recall |
|
value: 0.5385813032215204 |
|
name: Recall Macro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzVkNjliYTM0Njc3MTUzMDBmYTE5NDRkNzFjNzg2NzA0NzEyMTg4YTlkNGFlZWMxZWUwOGQzYzY1ZGU0ZmIwNyIsInZlcnNpb24iOjF9.cnEbDBJR8m3UqiuzCq_g4RUFLE8BVzXDebKguVrwPgY-Biu4sBFXVQvFyZScsLGEnaHYsE-R8ctTEGDdQONVBw |
|
- type: recall |
|
value: 0.5385388739946381 |
|
name: Recall Micro |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiODZkMmNjZWY4ZDYyYjU3NjQ2ZGNhZjkyNTQyOTg2ZjNmNDgwNDYxYmU2ZDA5M2EwOWRlMjMyYmI4MGU3MGMxNCIsInZlcnNpb24iOjF9.BfMB4_MZ-SYj1YbTES8pqgKNQkNnevSOjAwUqdoL6wsNpsKKWxPHmq0Kt9XufxHoQoyTkGvPfxh-0jEe3B1nBg |
|
- type: recall |
|
value: 0.5385388739946381 |
|
name: Recall Weighted |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYmE3Yjg3OTVhMjdlMDk1YWFjMWIwNjMyZTA2Yzc3MjBlNjI1YWY5MzE0MjNkMDNiMmU5ZmIxYWExNmViYWE1NSIsInZlcnNpb24iOjF9.S9Bo-wq3wikFS-FqMQerxahu87PJyYx141G5PCWDtOs2wH1nf4texnJYWfHeVCJKZcKmS2RWn5XOjjJ9RoNJAA |
|
- type: loss |
|
value: 1.062397837638855 |
|
name: loss |
|
verified: true |
|
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTFmNDI0ZmQ2YmNlZjJlZTdmZTYwOGVkMjdjMjJkMDIzNzhlOWFiNWQzNjFiMmU5NTdiM2Y1YjYxMjU4ZjQ2ZSIsInZlcnNpb24iOjF9.15RsFRkFpbarlU1L8UyV0o0_5WCveO_mT9CdO0UYwvQsOVjScheJ8fOqHBAC-C-CMTlfFNsmMhNrU_np8c_ZCQ |
|
--- |
|
|
|
# Indonesian small BigBird model NLI |
|
|
|
## Source Code |
|
|
|
Source code to create this model and perform benchmark is available at [https://github.com/ilos-vigil/bigbird-small-indonesian](https://github.com/ilos-vigil/bigbird-small-indonesian). |
|
|
|
## Model Description |
|
|
|
This model is based on [bigbird-small-indonesian](https://huggingface.co/ilos-vigil/bigbird-small-indonesian) and was finetuned on 2 datasets. It is intended to be used for zero-shot text classification. |
|
|
|
## How to use |
|
|
|
> Inference for ZSC (Zero Shot Classification) task |
|
|
|
```py |
|
>>> pipe = pipeline( |
|
... task='zero-shot-classification', |
|
... model='./tmp/checkpoint-28832' |
|
... ) |
|
>>> pipe( |
|
... sequences='Fakta nomor 7 akan membuat ada terkejut', |
|
... candidate_labels=['clickbait', 'bukan clickbait'], |
|
... hypothesis_template='Judul video ini {}.', |
|
... multi_label=False |
|
... ) |
|
{ |
|
'sequence': 'Fakta nomor 7 akan membuat ada terkejut', |
|
'labels': ['clickbait', 'bukan clickbait'], |
|
'scores': [0.6102734804153442, 0.38972654938697815] |
|
} |
|
>>> pipe( |
|
... sequences='Samsung tuntut balik Apple dengan alasan hak paten teknologi.', |
|
... candidate_labels=['teknologi', 'olahraga', 'bisnis', 'politik', 'kesehatan', 'kuliner'], |
|
... hypothesis_template='Kategori berita ini adalah {}.', |
|
... multi_label=True |
|
... ) |
|
{ |
|
'sequence': 'Samsung tuntut balik Apple dengan alasan hak paten teknologi.', |
|
'labels': ['politik', 'teknologi', 'kesehatan', 'bisnis', 'olahraga', 'kuliner'], |
|
'scores': [0.7390161752700806, 0.6657379269599915, 0.4459509551525116, 0.38407933712005615, 0.3679264783859253, 0.14181996881961823] |
|
} |
|
``` |
|
|
|
> Inference for NLI (Natural Language Inference) task |
|
|
|
```py |
|
>>> pipe = pipeline( |
|
... task='text-classification', |
|
... model='./tmp/checkpoint-28832', |
|
... return_all_scores=True |
|
... ) |
|
>>> pipe({ |
|
... 'text': 'Nasi adalah makanan pokok.', # Premise |
|
... 'text_pair': 'Saya mau makan nasi goreng.' # Hypothesis |
|
... }) |
|
[ |
|
{'label': 'entailment', 'score': 0.25495028495788574}, |
|
{'label': 'neutral', 'score': 0.40920916199684143}, |
|
{'label': 'contradiction', 'score': 0.33584052324295044} |
|
] |
|
>>> pipe({ |
|
... 'text': 'Python sering digunakan untuk web development dan AI research.', |
|
... 'text_pair': 'AI research biasanya tidak menggunakan bahasa pemrograman Python.' |
|
... }) |
|
[ |
|
{'label': 'entailment', 'score': 0.12508109211921692}, |
|
{'label': 'neutral', 'score': 0.22146646678447723}, |
|
{'label': 'contradiction', 'score': 0.653452455997467} |
|
] |
|
``` |
|
|
|
## Limitation and bias |
|
|
|
This model inherit limitation/bias from it's parent model and 2 datasets used for fine-tuning. And just like most language model, this model is sensitive towards input change. Here's an example. |
|
|
|
```py |
|
>>> from transformers import pipeline |
|
>>> pipe = pipeline( |
|
... task='zero-shot-classification', |
|
... model='./tmp/checkpoint-28832' |
|
... ) |
|
>>> text = 'Resep sate ayam enak dan mudah.' |
|
>>> candidate_labels = ['kuliner', 'olahraga'] |
|
>>> pipe( |
|
... sequences=text, |
|
... candidate_labels=candidate_labels, |
|
... hypothesis_template='Kategori judul artikel ini adalah {}.', |
|
... multi_label=False |
|
... ) |
|
{ |
|
'sequence': 'Resep sate ayam enak dan mudah.', |
|
'labels': ['kuliner', 'olahraga'], |
|
'scores': [0.7711364030838013, 0.22886358201503754] |
|
} |
|
>>> pipe( |
|
... sequences=text, |
|
... candidate_labels=candidate_labels, |
|
... hypothesis_template='Kelas kalimat ini {}.', |
|
... multi_label=False |
|
... ) |
|
{ |
|
'sequence': 'Resep sate ayam enak dan mudah.', |
|
'labels': ['kuliner', 'olahraga'], |
|
'scores': [0.7043636441230774, 0.295636385679245] |
|
} |
|
>>> pipe( |
|
... sequences=text, |
|
... candidate_labels=candidate_labels, |
|
... hypothesis_template='{}.', |
|
... multi_label=False |
|
... ) |
|
{ |
|
'sequence': 'Resep sate ayam enak dan mudah.', |
|
'labels': ['kuliner', 'olahraga'], |
|
'scores': [0.5986711382865906, 0.4013288915157318] |
|
} |
|
|
|
``` |
|
|
|
## Training, evaluation and testing data |
|
|
|
This model was finetuned with [IndoNLI](https://huggingface.co/datasets/indonli) and [multilingual-NLI-26lang-2mil7](https://huggingface.co/datasets/MoritzLaurer/multilingual-NLI-26lang-2mil7). Although `multilingual-NLI-26lang-2mil7` dataset is machine-translated, this dataset slightly improve result of NLI benchmark and extensively improve result of ZSC benchmark. Both evaluation and testing data is only based on IndoNLI dataset. |
|
|
|
## Training Procedure |
|
|
|
The model was finetuned on single RTX 3060 with 16 epoch/28832 steps with accumulated batch size 64. AdamW optimizer is used with LR 1e-4, weight decay 0.05, learning rate warmup for first 6% steps (1730 steps) and linear decay of the learning rate afterwards. Take note while model weight on epoch 9 has lowest loss/highest accuracy, it has slightly lower performance on ZSC benchmark. Additional information can be seen on Tensorboard training logs. |
|
|
|
## Benchmark as NLI model |
|
|
|
Both benchmark show result of 2 different model as additional comparison. Additional benchmark using IndoNLI dataset is available on it's paper [IndoNLI: A Natural Language Inference Dataset for Indonesian](https://aclanthology.org/2021.emnlp-main.821/). |
|
|
|
| Model | bigbird-small-indonesian-nli | xlm-roberta-large-xnli | mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 | |
|
| ------------------------------------------ | ---------------------------- | ---------------------- | -------------------------------------------- | |
|
| Parameter | 30.6M | 559.9M | 278.8M | |
|
| Multilingual | | V | V | |
|
| Finetuned on IndoNLI | V | | V | |
|
| Finetuned on multilingual-NLI-26lang-2mil7 | V | | | |
|
| Test (Lay) | 0.6888 | 0.2226 | 0.8151 | |
|
| Test (Expert) | 0.5734 | 0.3505 | 0.7775 | |
|
|
|
## Benchmark as ZSC model |
|
|
|
[Indonesian-Twitter-Emotion-Dataset](https://github.com/meisaputri21/Indonesian-Twitter-Emotion-Dataset/) is used to perform ZSC benchmark. This benchmark include 4 different parameter which affect performance of each model differently. Hypothesis template for this benchmark is `Kalimat ini mengekspresikan perasaan {}.` and `{}.`. Take note F1 score measurement only calculate label with highest probability. |
|
|
|
| Model | Multi-label | Use template | F1 Score | |
|
| -------------------------------------------- | ----------- | ------------ | ------------ | |
|
| bigbird-small-indonesian-nli | V | V | 0.3574 | |
|
| | V | | 0.3654 | |
|
| | | V | 0.3985 | |
|
| | | | _0.4160_ | |
|
| xlm-roberta-large-xnli | V | V | _**0.6292**_ | |
|
| | V | | 0.5596 | |
|
| | | V | 0.5737 | |
|
| | | | 0.5433 | |
|
| mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 | V | V | 0.5324 | |
|
| | V | | _0.5499_ | |
|
| | | V | 0.5269 | |
|
| | | | 0.5228 | |
|
|
