Create README.md

README.md

@@ -0,0 +1,138 @@
+---
+license: apache-2.0
+datasets:
+- ehartford/dolphin
+- LinhDuong/chatdoctor-200k
+- sahil2801/code_instructions_120k
+- c-s-ale/dolly-15k-instruction-alpaca-format
+- tiiuae/falcon-refinedweb
+- bigcode/starcoderdata
+- togethercomputer/RedPajama-Data-1T
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- instruct
+- medical
+- code
+---
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+This model is an instruction-tuned LLaMa model with 33B parameters, with specialities in medical QA and code instruction.
+
+## Model Details
+
+<!-- Provide a longer summary of what this model is. -->
+
+- **Model type:** LlamaForCausalLM
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0
+- **Finetuned from model (QLoRA):** [huggyllama/llama-30b](https://huggingface.co/huggyllama/llama-30b)
+
+## Training Details
+
+### Training Data
+
+Converted the following datasets to alpaca:instruction format.
+
+1. [ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin)
+  - ORCA style dataset generously created by [Eric Hartford](https://huggingface.co/ehartford)
+  - Only used the 1 million GPT4 generated instructions file [flan1m-alpaca-uncensored.jsonl](https://huggingface.co/datasets/ehartford/dolphin/blob/main/flan1m-alpaca-uncensored.jsonl).
+2. [LinhDuong/chatdoctor-200k](https://huggingface.co/datasets/LinhDuong/chatdoctor-200k)
+  - Refined dataset sourced from icliniq medical QA forum
+3. [sahil2801/code_instructions_120k](https://huggingface.co/datasets/sahil2801/code_instructions_120k)
+  - Code instruction dataset generously created by Sahil Chaudhary from ThreeSixty AI
+4. [c-s-ale/dolly-15k-instruction-alpaca-format](https://huggingface.co/datasets/c-s-ale/dolly-15k-instruction-alpaca-format)
+  - Dolly 15k is a general instruction dataset generated by employees of Databricks.
+
+### Training Procedure 
+
+Trained using [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) QLoRa on [RunPod](https://www.runpod.io/console/gpu-cloud) 8x A6000 on Community Cloud for 1 epochs (~23 hours - ~$70).
+
+<details>
+  <summary>axolotl training config:</summary>
+  
+```yaml
+base_model: huggyllama/llama-30b
+base_model_config: huggyllama/llama-30b
+model_type: LlamaForCausalLM
+tokenizer_type: LlamaTokenizer
+load_in_8bit: false
+load_in_4bit: true
+strict: false
+
+push_dataset_to_hub:
+hub_model_id:
+hf_use_auth_token:
+
+datasets:
+  - path: ehartford/dolphin
+    type: alpaca
+    data_files:
+      - flan1m-alpaca-uncensored.jsonl
+      - flan5m-alpaca-uncensored.jsonl
+    shards: 25
+  - path: sahil2801/code_instructions_120k
+    type: alpaca
+  - path: LinhDuong/chatdoctor-200k
+    type: alpaca
+    shards: 2
+  - path: c-s-ale/dolly-15k-instruction-alpaca-format
+    type: alpaca
+
+dataset_prepared_path: last_run_prepared
+val_set_size: 0.01
+adapter: qlora
+lora_model_dir:
+sequence_len: 2048
+max_packed_sequence_len: 2048
+lora_r: 8
+lora_alpha: 32
+lora_dropout: 0.05
+lora_target_modules:
+lora_target_linear: true
+lora_fan_in_fan_out:
+
+wandb_mode: true
+wandb_project: med-orca-instruct-33b
+wandb_watch:
+wandb_run_id:
+wandb_log_model: 'openllama_checkpoint'
+output_dir: /disk/med-instruct-33b
+gradient_accumulation_steps: 1
+micro_batch_size: 4
+num_epochs: 1
+optimizer: paged_adamw_32bit
+torchdistx_path:
+lr_scheduler: cosine
+learning_rate: 0.0002
+train_on_inputs: false
+group_by_length: false
+bf16: true
+fp16: false
+tf32: true
+gradient_checkpointing: true
+early_stopping_patience:
+resume_from_checkpoint:
+local_rank:
+logging_steps: 2
+xformers_attention: true
+flash_attention:
+gptq_groupsize:
+gptq_model_v1:
+warmup_steps: 100
+eval_steps: 20
+save_steps:
+debug:
+deepspeed: true
+weight_decay: 0.00001
+special_tokens:
+  bos_token: "<s>"
+  eos_token: "</s>"
+  unk_token: "<unk>"
+
+```
+</details>