SetFit with TaylorAI/gte-tiny

This is a SetFit model that can be used for Text Classification. This SetFit model uses TaylorAI/gte-tiny as the Sentence Transformer embedding model. A OneVsRestClassifier instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: TaylorAI/gte-tiny
Classification head: a OneVsRestClassifier instance
Maximum Sequence Length: 512 tokens

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("amplyfi/gte-tiny_annotations_cache_aggregated_multilabel")
# Run inference
preds = model("Intelligent Farming System Using Artificial Intelligence")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	3	11.0917	30

Training Hyperparameters

batch_size: (16, 2)
num_epochs: (10, 10)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (2e-05, 1e-05)
head_learning_rate: 0.01
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0017	1	0.2236	-
0.0833	50	0.3313	-
0.1667	100	0.3221	-
0.25	150	0.2733	-
0.3333	200	0.2331	-
0.4167	250	0.2129	-
0.5	300	0.1899	-
0.5833	350	0.1742	-
0.6667	400	0.1666	-
0.75	450	0.1653	-
0.8333	500	0.1531	-
0.9167	550	0.1453	-
1.0	600	0.1406	-
1.0833	650	0.118	-
1.1667	700	0.1247	-
1.25	750	0.124	-
1.3333	800	0.1195	-
1.4167	850	0.1227	-
1.5	900	0.1113	-
1.5833	950	0.1099	-
1.6667	1000	0.1188	-
1.75	1050	0.115	-
1.8333	1100	0.1045	-
1.9167	1150	0.1124	-
2.0	1200	0.1075	-
2.0833	1250	0.0935	-
2.1667	1300	0.1036	-
2.25	1350	0.0977	-
2.3333	1400	0.0974	-
2.4167	1450	0.1035	-
2.5	1500	0.0991	-
2.5833	1550	0.0906	-
2.6667	1600	0.0986	-
2.75	1650	0.0997	-
2.8333	1700	0.0964	-
2.9167	1750	0.0952	-
3.0	1800	0.0938	-
3.0833	1850	0.0825	-
3.1667	1900	0.0848	-
3.25	1950	0.0981	-
3.3333	2000	0.0861	-
3.4167	2050	0.0865	-
3.5	2100	0.0932	-
3.5833	2150	0.0879	-
3.6667	2200	0.0845	-
3.75	2250	0.09	-
3.8333	2300	0.0941	-
3.9167	2350	0.0922	-
4.0	2400	0.0855	-
4.0833	2450	0.0841	-
4.1667	2500	0.079	-
4.25	2550	0.09	-
4.3333	2600	0.0823	-
4.4167	2650	0.0776	-
4.5	2700	0.0832	-
4.5833	2750	0.088	-
4.6667	2800	0.0773	-
4.75	2850	0.0876	-
4.8333	2900	0.0827	-
4.9167	2950	0.0811	-
5.0	3000	0.0937	-
5.0833	3050	0.0856	-
5.1667	3100	0.0775	-
5.25	3150	0.0751	-
5.3333	3200	0.0784	-
5.4167	3250	0.0847	-
5.5	3300	0.0865	-
5.5833	3350	0.0851	-
5.6667	3400	0.0759	-
5.75	3450	0.0838	-
5.8333	3500	0.0778	-
5.9167	3550	0.0836	-
6.0	3600	0.0829	-
6.0833	3650	0.0767	-
6.1667	3700	0.0839	-
6.25	3750	0.0766	-
6.3333	3800	0.0752	-
6.4167	3850	0.0746	-
6.5	3900	0.076	-
6.5833	3950	0.0744	-
6.6667	4000	0.0802	-
6.75	4050	0.0786	-
6.8333	4100	0.0818	-
6.9167	4150	0.0859	-
7.0	4200	0.0714	-
7.0833	4250	0.074	-
7.1667	4300	0.0769	-
7.25	4350	0.0775	-
7.3333	4400	0.0753	-
7.4167	4450	0.0865	-
7.5	4500	0.0721	-
7.5833	4550	0.0803	-
7.6667	4600	0.0785	-
7.75	4650	0.0757	-
7.8333	4700	0.0788	-
7.9167	4750	0.0751	-
8.0	4800	0.0729	-
8.0833	4850	0.0696	-
8.1667	4900	0.073	-
8.25	4950	0.0771	-
8.3333	5000	0.0771	-
8.4167	5050	0.0758	-
8.5	5100	0.0747	-
8.5833	5150	0.0712	-
8.6667	5200	0.075	-
8.75	5250	0.0786	-
8.8333	5300	0.0817	-
8.9167	5350	0.076	-
9.0	5400	0.0742	-
9.0833	5450	0.0774	-
9.1667	5500	0.0698	-
9.25	5550	0.0803	-
9.3333	5600	0.0722	-
9.4167	5650	0.0727	-
9.5	5700	0.0704	-
9.5833	5750	0.0727	-
9.6667	5800	0.0768	-
9.75	5850	0.0752	-
9.8333	5900	0.0712	-
9.9167	5950	0.077	-
10.0	6000	0.0748	-

Framework Versions

Python: 3.10.12
SetFit: 1.1.1
Sentence Transformers: 3.3.1
Transformers: 4.48.0.dev0
PyTorch: 2.5.1+cu124
Datasets: 3.1.0
Tokenizers: 0.21.0

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

amplyfi
/

gte-tiny_annotations_cache_aggregated_multilabel