Upload folder using huggingface_hub

CRISPR_transformer_model/README.md

@@ -0,0 +1,88 @@
+---
+library_name: transformers
+tags:
+- generated_from_trainer
+datasets:
+- crispr_data
+model-index:
+- name: SX_spcas9_CRISPR_transformer
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# SX_spcas9_CRISPR_transformer
+
+This model is a fine-tuned version of [](https://huggingface.co/) on the crispr_data dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1334208.125
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.001
+- train_batch_size: 100
+- eval_batch_size: 100
+- seed: 63036
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.05
+- num_epochs: 30.0
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 13454687.4037 | 1.0   | 322  | 4939422.5       |
+| 4152085.8634  | 2.0   | 644  | 2624465.5       |
+| 2175050.1366  | 3.0   | 966  | 1416192.125     |
+| 1517438.0124  | 4.0   | 1288 | 1347445.125     |
+| 1400623.3043  | 5.0   | 1610 | 1365390.75      |
+| 1368684.5217  | 6.0   | 1932 | 1339115.375     |
+| 1355148.323   | 7.0   | 2254 | 1339799.375     |
+| 1348436.8696  | 8.0   | 2576 | 1337862.625     |
+| 1343765.2671  | 9.0   | 2898 | 1336787.625     |
+| 1342822.0621  | 10.0  | 3220 | 1339180.125     |
+| 1339066.1366  | 11.0  | 3542 | 1345115.5       |
+| 1340848.0994  | 12.0  | 3864 | 1335221.125     |
+| 1339262.5093  | 13.0  | 4186 | 1335443.125     |
+| 1338173.4161  | 14.0  | 4508 | 1336022.5       |
+| 1337596.7205  | 15.0  | 4830 | 1335901.625     |
+| 1336771.2795  | 16.0  | 5152 | 1337095.75      |
+| 1334854.8571  | 17.0  | 5474 | 1335969.5       |
+| 1333905.9876  | 18.0  | 5796 | 1334772.5       |
+| 1335469.3168  | 19.0  | 6118 | 1335592.125     |
+| 1333815.5528  | 20.0  | 6440 | 1335397.375     |
+| 1334275.8758  | 21.0  | 6762 | 1338478.125     |
+| 1333732.472   | 22.0  | 7084 | 1337926.5       |
+| 1331810.2857  | 23.0  | 7406 | 1334152.875     |
+| 1333228.1242  | 24.0  | 7728 | 1336674.75      |
+| 1332199.4534  | 25.0  | 8050 | 1336287.625     |
+| 1331443.8758  | 26.0  | 8372 | 1336826.0       |
+| 1331173.0683  | 27.0  | 8694 | 1333565.25      |
+| 1330894.6087  | 28.0  | 9016 | 1334126.875     |
+| 1331071.5031  | 29.0  | 9338 | 1333494.5       |
+| 1330270.6087  | 30.0  | 9660 | 1334208.125     |
+
+
+### Framework versions
+
+- Transformers 4.44.2
+- Pytorch 2.4.0+cu124
+- Datasets 2.21.0
+- Tokenizers 0.19.1

CRISPR_transformer_model/config.json

@@ -1,5 +1,4 @@
 {
-  "_name_or_path": "/home/ljw/sdc1/CRISPR_results/CRISPR_transformer/SX_spcas9_CRISPR_transformer",
   "architectures": [
     "CRISPRTransformerModel"
   ],

CRISPR_transformer_model/model.py

@@ -0,0 +1,85 @@
+from transformers import PreTrainedModel, RoFormerConfig, RoFormerModel
+import torch.nn as nn
+import torch
+import torch.nn.functional as F
+import numpy as np
+
+class CRISPRTransformerConfig(RoFormerConfig):
+    model_type = "CRISPR_transformer"
+    label_names = ["observation"]
+
+    def __init__(
+        self,
+        vocab_size = 4, # ACGT
+        hidden_size = 256, # model embedding dimension
+        num_hidden_layers = 3, # number of EncoderLayer
+        num_attention_heads = 4, # number of attention heads
+        intermediate_size = 1024, # FeedForward intermediate dimension size
+        hidden_dropout_prob = 0.1, # The dropout probability for all fully connected layers in the embeddings, encoder, and pooler
+        attention_probs_dropout_prob = 0.1, # The dropout ratio for the attention probabilities
+        max_position_embeddings = 256, # The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 1536).
+        ref1len = 127, # length of reference 1
+        ref2len = 127, # length of reference 2
+        seed = 63036, # random seed for intialization
+        **kwargs
+    ):
+        self.ref1len = ref1len
+        self.ref2len = ref2len
+        self.seed = seed
+        super().__init__(
+            vocab_size = vocab_size,
+            hidden_size = hidden_size,
+            num_hidden_layers = num_hidden_layers,
+            num_attention_heads = num_attention_heads,
+            intermediate_size = intermediate_size,
+            hidden_dropout_prob = hidden_dropout_prob,
+            attention_probs_dropout_prob = attention_probs_dropout_prob,
+            max_position_embeddings = max_position_embeddings,
+            **kwargs
+        )
+
+class CRISPRTransformerModel(PreTrainedModel):
+    config_class = CRISPRTransformerConfig
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.generator = torch.Generator().manual_seed(config.seed)
+        self.model = RoFormerModel(config)
+        self.mlp = nn.Linear(
+            in_features=config.hidden_size,
+            out_features=(config.ref1len + 1) * (config.ref2len + 1)
+        )
+        self.initialize_weights()
+
+    def initialize_weights(self):
+        for m in self.modules():
+            if isinstance(m, nn.Linear):
+                nn.init.normal_(m.weight, mean=0, std=1, generator=self.generator)
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+
+    def forward(self, refcode: torch.Tensor, observation: torch.Tensor=None):
+        # refcode (batch_size X sequence_length)
+        # model(refcode) (batch_size X sequence_length X hidden_size)
+        # model(refcode)[:, -1, :] arbitrary choose the last position to predict the logits
+        batch_size = refcode.shape[0]
+        logit = self.mlp(
+            self.model(
+                input_ids=refcode,
+                attention_mask=torch.ones(
+                    batch_size,
+                    self.config.ref1len + self.config.ref2len,
+                    dtype=torch.int64,
+                    device=self.model.device
+                )
+            ).last_hidden_state[:, -1, :]
+        ).view(batch_size, self.config.ref2len + 1, self.config.ref1len + 1)
+        if observation is not None:
+            return {
+                "logit": logit,
+                "loss": - (
+                    observation.flatten(start_dim=1) *
+                    F.log_softmax(logit.flatten(start_dim=1), dim=1)
+                ).sum()
+            }
+        return {"logit": logit}

CRISPR_transformer_model/runs/Nov14_11-57-39_ljw-System-Product-Name/events.out.tfevents.1731556660.ljw-System-Product-Name.1335450.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65f5400d207867d119030f71a27dd601f75079569a7f605073143eddc22dfe97
+size 19848

CRISPR_transformer_model/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50946c817480192af6ddb8ddbaa1677a28f005fe0ed7b77b3608f8ebee20fe1d
+size 5304