datasets:
- homebrewltd/instruction-speech-whispervq-v2
language:
- en
license: apache-2.0
tags:
- sound language model
Model Details
We have developed and released the family llama3s. This family is natively understanding audio and text input.
We expand the Semantic tokens experiment with WhisperVQ as a tokenizer for audio files from homebrewltd/llama3.1-s-base-v0.2 with nearly 1B tokens from Instruction Speech WhisperVQ v2 dataset.
Model developers Homebrew Research.
Input Text and sound.
Output Text.
Model Architecture Llama-3.
Language(s): English.
Intended Use
Intended Use Cases This family is primarily intended for research applications. This version aims to further improve the LLM on sound understanding capabilities.
Out-of-scope The use of llama3-s in any manner that violates applicable laws or regulations is strictly prohibited.
How to Get Started with the Model
First, we need to convert the audio file to sound tokens
Then, we can inference the model the same as any other LLM.
Training process
Training Metrics Image: Below is a snapshot of the training loss curve visualized.
Hardware
GPU Configuration: Cluster of 8x NVIDIA H100-SXM-80GB. GPU Usage:
- Continual Training: 6 hours.
Training Arguments
We utilize torchtune library for the latest FSDP2 training code implementation.
| Parameter | Continual Training |
|---|---|
| Epoch | 1 |
| Global batch size | 128 |
| Learning Rate | 0.5e-4 |
| Learning Scheduler | Cosine with warmup |
| Optimizer | Adam torch fused |
| Warmup Ratio | 0.01 |
| Weight Decay | 0.005 |
| Max Sequence Length | 512 |
Examples
- Good example:
Click to toggle Example 1
Click to toggle Example 2
- Misunderstanding example:
Click to toggle Example 3
- Off-tracked example:
Click to toggle Example 4
Citation Information
BibTeX:
@article{Llama3-S: Sound Instruction Language Model 2024,
title={Llama3-S},
author={Homebrew Research},
year=2024,
month=August},
url={https://huggingface.co/homebrewltd/llama3.1-s-2024-08-20}