ethanteh/bge-finetuned-insurance-matryoshka

BGE Health Insurance Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5 on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ethanteh/bge-finetuned-insurance-matryoshka")
# Run inference
sentences = [
    'Pre-retirement Deductible 0; Post-retirement Deductible 0\n\nAge Male Female MSIII 200 MSIII 300 MSIII 500 MSIII 200 MSIII 300 28 1,166 1,590 2,324 1,246 1,755 29 1,184 1,598 2,336 1,264 1,763 30 1,212 1,609 2,345 1,284 1,772 31 1,241 1,648 2,403 1,305 1,783 32 1,272 1,693 2,469 1,336 1,791 33 1,305 1,744 2,544 1,366 1,829 34 1,340 1,799 2,625 1,396 1,873 35 1,384 1,865 2,723 1,426 1,918 36 1,430 1,932 2,821 1,457 1,962 37 1,490 2,016 2,944 1,490 2,004 38 1,555 2,103 3,072 1,528 2,046 39 1,658 2,239 3,272 1,567 2,087 40 1,794 2,419 3,536 1,610 2,134 41 1,930 2,600 3,801 1,674 2,214 MSIII 500 2,564 2,576 2,588 2,602 2,606 2,661 2,725 2,791 2,856 2,918 2,980 3,041 3,110 3,228\n\nNote:\n\nFor presentation purposes, the insurance charges are rounded up to the next ringgit.\n\nAnnual Insurance Charges for HLA MediShield III Rider\n\nOccupational Class 1 & 2',
    'What are the annual insurance charges for a 35-year-old male under HLA MediShield III Rider, Occupational Class 1 & 2, assuming pre and post retirement deductibles are 0?',
    'What types of post-hospitalization support are included in the medical coverage?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: dim_768, dim_512, dim_256, dim_128 and dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	dim_768	dim_512	dim_256	dim_128	dim_64
cosine_accuracy@1	0.0602	0.0843	0.0843	0.0964	0.0723
cosine_accuracy@3	0.5301	0.494	0.494	0.4578	0.4096
cosine_accuracy@5	0.6627	0.6386	0.6506	0.5783	0.5301
cosine_accuracy@10	0.7711	0.7831	0.7711	0.7952	0.7349
cosine_precision@1	0.0602	0.0843	0.0843	0.0964	0.0723
cosine_precision@3	0.1767	0.1647	0.1647	0.1526	0.1365
cosine_precision@5	0.1325	0.1277	0.1301	0.1157	0.106
cosine_precision@10	0.0771	0.0783	0.0771	0.0795	0.0735
cosine_recall@1	0.0602	0.0843	0.0843	0.0964	0.0723
cosine_recall@3	0.5301	0.494	0.494	0.4578	0.4096
cosine_recall@5	0.6627	0.6386	0.6506	0.5783	0.5301
cosine_recall@10	0.7711	0.7831	0.7711	0.7952	0.7349
cosine_ndcg@10	0.4436	0.4512	0.4432	0.4422	0.4003
cosine_mrr@10	0.3357	0.3434	0.337	0.331	0.2946
cosine_map@100	0.3482	0.3544	0.3477	0.3393	0.3079

Training Details

Training Dataset

json

• Dataset: json
• Size: 739 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 739 samples:

  	positive	anchor
  type	string	string
  details	min: 13 tokens mean: 262.79 tokens max: 512 tokens	min: 10 tokens mean: 20.63 tokens max: 48 tokens

• Samples:

  positive	anchor
  Summary Table of Critical Illness Coverage (Con’t) Benefit Payout No Critical Illnesses Early Stage 50% of Rider Sum Assured 36 40 46 Intermediate Stage 50% of Rider Sum Assured Advanced Stage 100% of Rider Sum Assured deficit with persisting clinical Third Degree Burns – of specified Summary Table of Critical Illness Coverage (Con’t) Critical Illnesses Early Stage 50% of Rider Sum Assured Human Immunodeficiency - Insertion of a Vena-cava filter Benefit Payout Intermediate Stage 50% of Rider Sum Assured Advanced Stage Hypertension – of specified No 100% of Rider Sum Assured Human Immunodeficiency Virus 50 in 56 60 Cardiomyopathy – of specified Summary Table of Critical Illness Coverage (Con’t) Critical Illnesses Early Stage 50% of Rider Sum Assured 61 62 63 64 65 66 67 68 69 70 Benefit Payout Intermediate Stage 50% of Rider Sum Assured Advanced Stage 100% of Rider Sum Assured deficit with persisting clinical No Special Benefit Special Benefit Diabetes Recovery Benefit...	What benefits are available for conditions like third-degree burns, HIV, or cardiomyopathy, according to this table?
  Q: Where can I purchase this plan? official website at greateasternlife.com/my/direct Q: How do I pay my premiums? A: You have the flexibility to pay your premium annually or half- Q: Am I covered for emergency medical assistance outside Malaysia? A: You are covered for emergency medical assistance received while travelling outside Malaysia but subject to a maximum of 120 consecutive days on any one trip. Note: Terms and conditions apply. Q: Am I covered for treatment outside Malaysia? attending physician, benefits in respect of the treatment for Q: How much will I get upon surrender of my policy? This plan has no surrender value. However, upon cancellation of the policy by policyowner, you will be entitled for a proportionate refund (where applicable) of the last premium paid as below: A: Policy purchased for a period Refund of Annual Refund of Half- less than or Premium Yearly Premium equivalent to 3 months 30% 30% 30% Note: The above refunds are only applicable provide...	What are the premium payment options for the GREAT Health Direct plan?
  66 Uremic Syndrome Syndrome Nil Nil Syndrome No. 67 68 69 70 71 72 73 74 75 76 Category Necrotising Fasciitis Respiratory Diphteria requiring mechanical ventilation Loss of limb(s) Cerebral Aneurysm Requiring Brain Surgery Multiple Root Avulsions of Brachial Plexus Rheumatic Fever with Heart Valve Disorders Resection of the whole small intestine (duodenum, jejunum and ileum) Severe Pulmonary Fibrosis Rabies Generalized • Nil • Nil • Nil • Nil • Nil • Nil • Nil • Nil • Nil • Nil A-Plus Beyond Early Critical Shield Early Intermediate Critical Illness Critical Illness • Nil • Nil • Nil • Nil • Nil • Nil • Nil • Nil • Nil • Nil A-Plus Beyond Critical Shield Advanced Critical Illness • Necrotising Fasciitis • Respiratory Diphteria requiring mechanical ventilation • Loss of limb(s) • Cerebral Aneurysm Requiring Brain Surgery • Multiple Root Avulsions of Brachial Plexus • Rheumatic Fever with Heart Valve Disorders • Resection of the whole small intestine (duodenum, jejunum and ileum) •...	What illnesses are listed under the 'Early Critical Illness' category?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 10
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• fp16: True
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.8602	5	-	0.4356	0.4285	0.4026	0.3877	0.3048
1.8602	10	46.1696	0.4348	0.4279	0.4035	0.3920	0.3294
2.8602	15	-	0.4340	0.4252	0.4032	0.4127	0.3518
3.8602	20	20.5361	0.4403	0.4493	0.4117	0.4156	0.3761
4.8602	25	-	0.4558	0.4413	0.4233	0.4166	0.3859
5.8602	30	15.2732	0.4407	0.4359	0.4259	0.4266	0.3974
6.8602	35	-	0.4416	0.4408	0.4389	0.4417	0.3983
7.8602	40	11.1479	0.4427	0.4512	0.4432	0.4427	0.397
8.8602	45	-	0.4460	0.4512	0.4432	0.4422	0.4003
9.8602	50	12.725	0.4436	0.4512	0.4432	0.4422	0.4003

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.48.3
• PyTorch: 2.5.1+cu124
• Accelerate: 1.3.0
• Datasets: 2.19.1
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}