---
library_name: transformers
base_model: BioMistral/BioMistral-7B
language:
- en
metrics:
- perplexity
pipeline_tag: text2text-generation
tags:
  - text-to-text
  - natural-language-processing
license: apache-2.0
---
<p align="center">
  <img src="https://huggingface.co/Indah1/BioChat10/resolve/main/BioChat.png?download=true" alt="drawing" width="450"/>
</p>

# BioChat Model
- **Source Paper:** [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://arxiv.org/abs/2402.10373)

- **BioChat** is a language model fine-tuned using the ChatDoctor dataset from [ChatDoctor-5k](https://huggingface.co/datasets/LinhDuong/chatdoctor-5k). Specifically designed for medical conversations, BioChat enables users to engage in interactive discussions with a virtual doctor. Whether you are seeking advice about symptoms you are experiencing, exploring possible health conditions, or looking for general medical insights, BioChat is built to assist in a reliable and informative manner.
- **NOTE**: We are still in the early stages of exploring the generation capabilities and limitations of this model. It is important to emphasize that its text generation features are intended solely for research purposes and are not yet suitable for production use.

- **Finetuned from model:** [BioMistral-7B](https://huggingface.co/BioMistral/BioMistral-7B).

# Using BioChat

You can use BioMistral with [Hugging Face's Transformers library](https://github.com/huggingface/transformers) as follow.

Loading the model and tokenizer :

```python
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("BioMistral/BioMistral-7B")
model = AutoModelForCausalLM.from_pretrained(
    "BioMistral/BioMistral-7B",
    load_in_8bit=True,
    device_map="auto",
    output_hidden_states=True  # Ensure hidden states are available
)
model = PeftModel.from_pretrained(model, "Indah1/BioChat10")
```

# Fine-Tuning Data

The fine-tuning data used for BioChat is derived from the [ChatDoctor-5k](https://huggingface.co/datasets/LinhDuong/chatdoctor-5k) dataset. This dataset contains a collection of medical conversations tailored to simulate doctor-patient interactions, making it an ideal source for training a medical conversational model. The dataset was carefully curated to ensure relevance and diversity in medical topics.

#### Training Hyperparameters

|     Hyperparameter    |             Value            |
|:-------------------:|:----------------------------------:|
|    Weigh Decay   | 0.01 | 
|  Learning Rate | 2e-05 |
|  Training Batch Size | 8  |
| Batch Size | 8 |
| Number of GPU | 1 |
| Optimizer | AdamW_8Bit |
| Warm Up Ratio | 0.03 |
| Scheduler | Cosine |
| Number of Epoch | 5, 10, 15 |

## Evaluation

To determine the best model for fine-tuning, I used ***perplexity*** as a metric to evaluate performance and select the most optimal version. By leveraging the model's capabilities, I aim to evaluate its behavior and responses using tools like the ***Word Embedding Association Test (WEAT)***. Below are the WEAT scores and perplexity values for the model at epochs 5, 10, and 15, which helped in determining the best-performing version. It is important to emphasize that its text generation features are intended solely for research purposes and are not yet suitable for production use. By releasing this model, we aim to drive advancements in biomedical NLP applications and contribute to best practices for the responsible development of domain-specific language models. Ensuring reliability, fairness, accuracy, and explainability remains a top priority for us.
|     Model Name    |             Perplexity Score            |         WEAT Score       |    Effect Size    |
|:-------------------:|:----------------------------------:|:----------------------------------:|:----------------------------------:|
|    **[BioChat5](https://huggingface.co/Indah1/BioChat5)**   | **4.5799** | **-0.00652** | **-0.4059** |
|    **[BioChat10](https://huggingface.co/Indah1/BioChat10)** | **4.5873** | **0.002351** | **0.06176** |
|    **[BioChat15](https://huggingface.co/Indah1/BioChat15)** | **4.8864** | **0.00859**  | **0.43890** |
### Framework versions

- PEFT 0.11.1