README.md

---

license: mit
---


# Merlin: Vision Language Foundation Model for 3D Computed Tomography

[![pypi](https://img.shields.io/pypi/v/merlin-vlm?style=for-the-badge)](https://pypi.org/project/merlin-vlm/)

Merlin is a 3D VLM for computed tomography that leverages both structured electronic health records (EHR) and unstructured radiology reports for pretraining. The huggingface repository here provides the model weights and an example image file.

[[💻 Github](https://github.com/StanfordMIMI/Merlin)] [[📄 Paper](https://arxiv.org/abs/2406.06512)]

## ⚡️ Installation

To install Merlin, you can simply run:

```python

pip install merlin-vlm

```

For an editable installation, use the following commands to clone and install this repository.
```python

git clone https://github.com/StanfordMIMI/Merlin.git

cd merlin

pip install -e .

```

For usage instructions, please visit the github [repository](https://github.com/StanfordMIMI/Merlin).

### 📁 Project Structure:

```

.

├── README.md

├── i3_resnet_clinical_longformer_best_clip_04-02-2024_23-21-36_epoch_99.pt <Merlin weights>

├── image1.nii.gz <Sample Image>

```

## 📎 Citation
If you find this repository useful for your work, please cite the cite the [original paper](https://arxiv.org/abs/2406.06512):

```bibtex

@article{blankemeier2024merlin,

  title={Merlin: A vision language foundation model for 3d computed tomography},

  author={Blankemeier, Louis and Cohen, Joseph Paul and Kumar, Ashwin and Van Veen, Dave and Gardezi, Syed Jamal Safdar and Paschali, Magdalini and Chen, Zhihong and Delbrouck, Jean-Benoit and Reis, Eduardo and Truyts, Cesar and others},

  journal={Research Square},

  pages={rs--3},

  year={2024}

}

```