metadata

license: mit
language:
  - fr
pipeline_tag: fill-mask
tags:
  - medical
  - clinical
datasets:
  - rntc/biomed-fr

CamemBERT-bio : a Tasty French Language Model Better for your Health

CamemBERT-bio is a state-of-the-art french biomedical language model built using continual-pretraining from camembert-base. It was trained on a french public biomedical corpus of 413M words containing scientific documments, drug leaflets and clinical cases extrated from theses and articles. It shows 2.54 points of F1 score improvement on average on 5 different biomedical named entity recognition tasks compared to camembert-base.

Absract

Clinical data in hospitals are increasingly accessible for research through clinical data warehouses, however these documents are unstructured. It is therefore necessary to extract information from medical reports to conduct clinical studies. Transfer learning with BERT-like models such as CamemBERT has allowed major advances, especially for named entity recognition. However, these models are trained for plain language and are less efficient on biomedical data. This is why we propose a new french public biomedical dataset on which we have continued the pre-training of CamemBERT. Thus, we introduce a first version of CamemBERT-bio, a specialized public model for the french biomedical domain that shows 2.54 points of F1 score improvement on average on different biomedical named entity recognition tasks.

Developed by: Rian Touchent, Eric Villemonte de La Clergerie
License: MIT

!### Model Sources [optional]

Training Details

Training Data

Corpus	Details	Size
ISTEX	Divers documents de la littérature scientifique indexés sur ISTEX	276 M
CLEAR	Notices de médicaments	73 M
E3C	Divers documents issus de journaux, de notices et de cas cliniques	64 M
Total		413 M

Training Procedure

We used continual-pretraining from camembert-base. We trained the model using the Masked Language Modeling (MLM) objective with Whole Word Masking for 50k steps during 39 hours with 2 Tesla V100.

Evaluation

Fine-tuning

For fine-tuning, we utilized Optuna to select the hyperparameters. The learning rate was set to 5e-5, with a warmup ratio of 0.224 and a batch size of 16. The fine-tuning process was carried out for 2000 steps. For prediction, a simple linear layer was added on top of the model. Notably, none of the CamemBERT layers were frozen during the fine-tuning process.

Scoring

To evaluate the performance of the model, we used the seqeval tool in strict mode with the IOB2 scheme. For each evaluation, the best fine-tuned model on the validation set was selected to calculate the final score on the test set. To ensure reliability, we averaged over 10 evaluations with different seeds.

Results

Style	Dataset	Score	CamemBERT	CamemBERT
Clinique	CAS1	F1	70.50 ± 1.75	73.03 ± 1.29
		P	70.12 ± 1.93	71.71 ± 1.61
		R	70.89 ± 1.78	74.42 ± 1.49
	CAS2	F1	79.02 ± 0.92	81.66 ± 0.59
		P	77.3 ± 1.36	80.96 ± 0.91
		R	80.83 ± 0.96	82.37 ± 0.69
	E3C	F1	67.63 ± 1.45	69.85 ± 1.58
		P	78.19 ± 0.72	79.11 ± 0.42
		R	59.61 ± 2.25	62.56 ± 2.50
Notices	EMEA	F1	74.14 ± 1.95	76.71 ± 1.50
		P	74.62 ± 1.97	76.92 ± 1.96
		R	73.68 ± 2.22	76.52 ± 1.62
Scientifique	MEDLINE	F1	65.73 ± 0.40	68.47 ± 0.54
		P	64.94 ± 0.82	67.77 ± 0.88
		R	66.56 ± 0.56	69.21 ± 1.32

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: 2 x Tesla V100
Hours used: 39 hours
Provider: INRIA clusters
Compute Region: Paris, France
Carbon Emitted: 0.84 kg CO2 eq.