metadata

license: mit
language:
  - multilingual
  - af
  - am
  - ar
  - as
  - az
  - be
  - bg
  - bn
  - br
  - bs
  - ca
  - cs
  - cy
  - da
  - de
  - el
  - en
  - eo
  - es
  - et
  - eu
  - fa
  - fi
  - fr
  - fy
  - ga
  - gd
  - gl
  - gu
  - ha
  - he
  - hi
  - hr
  - hu
  - hy
  - id
  - is
  - it
  - ja
  - jv
  - ka
  - kk
  - km
  - kn
  - ko
  - ku
  - ky
  - la
  - lo
  - lt
  - lv
  - mg
  - mk
  - ml
  - mn
  - mr
  - ms
  - my
  - ne
  - nl
  - 'no'
  - om
  - or
  - pa
  - pl
  - ps
  - pt
  - ro
  - ru
  - sa
  - sd
  - si
  - sk
  - sl
  - so
  - sq
  - sr
  - su
  - sv
  - sw
  - ta
  - te
  - th
  - tl
  - tr
  - ug
  - uk
  - ur
  - uz
  - vi
  - xh
  - yi
  - zh
  - yue
base_model:
  - FacebookAI/xlm-roberta-base
pipeline_tag: feature-extraction
tags:
  - music

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages

CLaMP 3 Overview

Overview

CLaMP 3 is a multimodal and multilingual framework for music information retrieval (MIR) that supports all major music formats—sheet music, audio, and performance signals—along with multilingual text. It is trained on 27 languages and can generalize to support 100 languages. Using contrastive learning, CLaMP 3 aligns these different formats into a shared representation space, making cross-modal retrieval seamless. Experiments show that it significantly outperforms previous strong baselines, setting a new state-of-the-art in multimodal and multilingual MIR.

Key Features

Multimodal Support:
- Sheet Music: Uses Interleaved ABC notation, with a context size of 512 bars.
- Performance Signals: Processes MIDI Text Format (MTF) data, with a context size of 512 MIDI messages.
- Audio Recordings: Works with features extracted by MERT, with a context size of 640 seconds of audio.
Multilingual Capabilities:
- Trained on 27 languages and generalizes to all 100 languages supported by XLM-R.
Datasets & Benchmarking:
- M4-RAG: A web-scale dataset of 2.31M high-quality music-text pairs across 27 languages and 194 countries.
- WikiMT-X: A MIR benchmark containing 1,000 triplets of sheet music, audio, and diverse text annotations.

What Can CLaMP 3 Do?

CLaMP 3 unifies diverse music data and text into a shared representation space, enabling the following key capabilities:

Text-to-Music Retrieval: Finds relevant music based on text descriptions in 100 languages.
Image-to-Music Retrieval: Matches music that aligns with the scene depicted in the image.
Cross-Modal Music Retrieval: Enables music retrieval and recommendation across different modalities.
Zero-Shot Music Classification: Identifies musical attributes such as genres, moods, and styles without labeled training data.
Music Semantic Similarity Evaluation: Measures semantic similarity between:
- Generated music and its text prompt, validating how well text-to-music models follow instructions.
- Generated music and reference music, assessing their semantic similarity, including aspects like style, instrumentation, and musicality.

For examples demonstrating these capabilities, visit CLaMP 3 Homepage.

Repository Structure

code/ → Training & feature extraction scripts.
classification/ → Linear classification training and prediction.
preprocessing/ → Convert data into Interleaved ABC, MTF, or MERT-extracted features.
retrieval/ → Semantic search, retrieval evaluation, and similarity calculations.

Note: Ensure the model weights are placed in the code/ folder, and verify the configuration hyperparameters before use.

Getting Started

Environment Setup

To set up the environment for CLaMP 3, run:

conda env create -f environment.yml
conda activate clamp3

Data Preparation

1. Convert Music Data to Compatible Formats

Before using CLaMP 3, preprocess MusicXML files into Interleaved ABC, MIDI files into MTF, and audio files into MERT-extracted features.

Note: Each script requires a manual edit of the input_dir variable at the top of the file before running, except for the MERT extraction script (extract_mert.py), which takes command-line arguments for input and output paths.

1.1 Convert MusicXML to Interleaved ABC Notation

CLaMP 3 requires Interleaved ABC notation for sheet music. To achieve this, first, convert MusicXML (.mxl, .xml, .musicxml) to standard ABC using batch_xml2abc.py:

python batch_xml2abc.py

Input: .mxl, .xml, .musicxml
Output: .abc (Standard ABC)

Next, process the standard ABC files into Interleaved ABC notation using batch_interleaved_abc.py:

python batch_interleaved_abc.py

Input: .abc (Standard ABC)
Output: .abc (Interleaved ABC for CLaMP 3)

1.2 Convert MIDI to MTF Format

CLaMP 3 processes performance signals in MIDI Text Format (MTF). Convert MIDI files (.mid, .midi) into MTF format using batch_midi2mtf.py:

python batch_midi2mtf.py

Input: .mid, .midi
Output: .mtf (MTF for CLaMP 3)

1.3 Extract Audio Features using MERT

For audio processing, CLaMP 3 uses MERT-extracted features instead of raw waveforms. Extract MERT-based features from raw audio (.mp3, .wav) using extract_mert.py:

python extract_mert.py --input_path <input_path> --output_path <output_path> --model_path m-a-p/MERT-v1-95M --mean_features

Input: .mp3, .wav
Output: .npy (Processed audio features for CLaMP 3)

Training and Feature Extraction

1. Training Models

CLaMP 3 is the most powerful music retrieval model, and in most cases, retraining is not needed. However, if necessary, follow these steps.

Modify config.py to adjust hyperparameters and data paths.
Train on your own data.

To train CLaMP 3 on symbolic music (e.g., sheet music, MIDI), run:

python -m torch.distributed.launch --nproc_per_node=<GPUs> --use_env train_clamp3_symbolic.py

For audio data, use:

python -m torch.distributed.launch --nproc_per_node=<GPUs> --use_env train_clamp3_audio.py

Using Pre-Trained Models (Recommended)

For most use cases, it's best to use pre-trained weights instead of training from scratch.

Version	Best for	Download Link
CLaMP 3 SAAS	Audio-based retrieval (Recommended)	Download SAAS
CLaMP 3 C2	Symbolic music retrieval (Sheet music, MIDI)	Download C2

How to Switch Between Versions?

By default, CLaMP 3 is configured for the SAAS version (optimized for audio).

If working with symbolic music (MIDI, sheet music), use the C2 version:
Modify line 66 in config.py from "saas" to "c2".

2. Feature Extraction

After training (or using pre-trained weights), extract features using extract_clamp3.py:

accelerate launch extract_clamp3.py --epoch <epoch> <input_dir> <output_dir> [--get_global]

--epoch <epoch>: (Optional) Specify the checkpoint epoch.
<input_dir>: Directory containing the input files.
<output_dir>: Destination folder for the output .npy features.
--get_global: (Required for retrieval!) Extracts a global semantic vector for each input.

All extracted features are stored as .npy files.

Note: For retrieval, --get_global must be used. Without it, CLaMP 3 will not work correctly for retrieval tasks. You only omit --get_global if you are performing downstream fine-tuning or need raw feature extraction for custom tasks.

Retrieval and Classification

1. Semantic Search

To perform semantic search with CLaMP 3, you first need to extract the features for both your query and reference data using extract_clamp3.py. The query is usually a text description, and the reference folder contains a large set of music data, such as audio or sheet music.

After extracting the features, you can perform the semantic search using the semantic_search.py script. This search can be used for various tasks.

python semantic_search.py <query_file> <reference_folder> [--top_k TOP_K]

<query_file>: Path to the query feature (e.g., ballad.npy).
<reference_folder>: Folder containing reference features for comparison.
--top_k: (Optional) Number of top similar items to display (default is 10).

CLaMP 3's semantic search enables various retrieval and evaluation tasks by comparing features extracted from queries and reference data. Generally, the larger and more diverse the reference music dataset, the higher the likelihood of retrieving relevant and accurately matched music.

1. Text-to-Music Retrieval

Query: Text description of the desired music.
Reference: Music data (e.g., audio files).
Output: Retrieves music that best matches the semantic meaning of the text description.

2. Image-to-Music Retrieval

Query: Generate an image caption using models like BLIP.
Reference: Music data (e.g., audio files).
Output: Finds music that semantically aligns with the image.

3. Cross-Modal and Same-Modal Music Retrieval

Cross-Modal Retrieval:
- Query: Music data from one modality (e.g., audio).
- Reference: Music data from another modality (e.g., MIDI, ABC notation).
- Output: Finds semantically similar music across different representations.
Same-Modal Retrieval (Semantic-Based Music Recommendation):
- Query & Reference: Both are from the same modality (e.g., audio-to-audio).
- Output: Recommends similar music based on semantic meaning.

4. Zero-Shot Music Classification

Query: Music data.
Reference: Class descriptions (e.g., "It is classical," "It is folk").
Output: Assigns the most relevant class based on feature similarity.

5. Music Semantic Similarity Evaluation

Query: High-quality music or music generation prompt.
Reference: Generated music.
Output: Ranks generated music based on semantic similarity to the query. For large-scale evaluation between generated music and reference music, it is recommended to use clamp3_score.py.

2. Classification

Train a linear classifier using train_cls.py:

python train_cls.py --train_folder <path> --eval_folder <path> [--num_epochs <int>] [--learning_rate <float>] [--balanced_training]

Run inference with inference_cls.py:

python inference_cls.py <weights_path> <feature_folder> <output_file>

Citation

If you find CLaMP 3 useful in your work, please consider citing our paper:

@misc{wu2025clamp3universalmusic,
  title={CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages}, 
  author={Shangda Wu and Zhancheng Guo and Ruibin Yuan and Junyan Jiang and Seungheon Doh and Gus Xia and Juhan Nam and Xiaobing Li and Feng Yu and Maosong Sun},
  year={2025},
  eprint={2502.10362},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2502.10362}
}