Upload StabilityPrediction

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,12 @@
+{
+  "architectures": [
+    "StabilityPrediction"
+  ],
+  "auto_map": {
+    "AutoConfig": "ddg_predictor_modeling.StabilityPredictionConfig",
+    "AutoModel": "ddg_predictor_modeling.StabilityPrediction"
+  },
+  "embed_dim": 1536,
+  "torch_dtype": "float32",
+  "transformers_version": "4.45.2"
+}

ddg_predictor_modeling.py

@@ -0,0 +1,58 @@
+from transformers import PreTrainedModel, PretrainedConfig
+from torch import nn
+from esm.pretrained import ESM3_sm_open_v0
+import torch
+
+
+class StabilityPredictionConfig(PretrainedConfig):
+    def __init__(self, embed_dim=1536, *args, **kwargs):
+        super().__init__(*args, embed_dim=1536, **kwargs)
+
+
+class SingleMutationPooler(nn.Module):
+    def __init__(self, embed_dim=1536):
+        super().__init__()
+        self.wt_weight = nn.Parameter(torch.ones((1, embed_dim)), requires_grad=True)
+        self.mut_weight = nn.Parameter(-1 * torch.ones((1, embed_dim)), requires_grad=True)
+        self.norm = nn.LayerNorm(embed_dim, bias=False)
+        
+    
+    def forward(self, wt_embedding, mut_embedding, positions):
+        embed_shape = wt_embedding.shape[-1]
+        positions = positions.view(-1, 1).unsqueeze(2).repeat(1, 1, embed_shape) + 1
+        wt_residues = torch.gather(wt_embedding, 1, positions).squeeze(1)
+        mut_residues = torch.gather(mut_embedding, 1, positions).squeeze(1)
+        wt_residues = wt_residues * self.wt_weight
+        mut_residues = mut_residues * self.mut_weight
+        return self.norm(wt_residues + mut_residues)
+    
+
+    
+class StabilityPrediction(PreTrainedModel):
+    config_class = StabilityPredictionConfig
+    def __init__(self, config=StabilityPredictionConfig()):
+        super().__init__(config=config)
+        self.backbone = ESM3_sm_open_v0(getattr(config, "device", "cpu"))
+        self.pooler = SingleMutationPooler()
+        self.regressor = nn.Linear(config.embed_dim, 1)
+        self.reset_parameters()
+
+    def reset_parameters(self):
+        nn.init.uniform_(self.regressor.weight, -0.01, 0.01)
+        nn.init.zeros_(self.regressor.bias)
+    
+    def compute_loss(self, logits, labels):
+        if labels is None:
+            return
+        return F.mse_loss(logits, labels)
+    
+    def forward(self, wt_input_ids, mut_input_ids, positions, labels=None):
+        wt_embeddings = self.backbone(sequence_tokens=wt_input_ids).embeddings
+        mut_embeddings = self.backbone(sequence_tokens=mut_input_ids).embeddings
+        aggregated_embeddings = self.pooler(wt_embeddings, mut_embeddings, positions)
+        logits = self.regressor(aggregated_embeddings)
+        loss = self.compute_loss(logits, labels)
+        return {
+            "loss": loss,
+            "logits": logits,
+        }

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f7b834c83ead9b4203f3d157b8b62ec0dcb698a7ba5d73de37b404bb9a90be7
+size 4973995160

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20f9699648ca3235501ce50a4254276330e3d3013b9163822535b1aea947f143
+size 633038580

model.safetensors.index.json

@@ -0,0 +1,551 @@
+{
+  "metadata": {
+    "total_size": 5606967572
+  },
+  "weight_map": {
+    "backbone.encoder.function_embed.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.function_embed.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.function_embed.2.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.function_embed.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.function_embed.4.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.function_embed.5.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.function_embed.6.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.function_embed.7.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.plddt_projection.bias": "model-00001-of-00002.safetensors",
+    "backbone.encoder.plddt_projection.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.residue_embed.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.sasa_embed.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.sequence_embed.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.ss8_embed.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.structure_per_res_plddt_projection.bias": "model-00001-of-00002.safetensors",
+    "backbone.encoder.structure_per_res_plddt_projection.weight": "model-00001-of-00002.safetensors",
+    "backbone.encoder.structure_tokens_embed.weight": "model-00001-of-00002.safetensors",
+    "backbone.output_heads.function_head.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.function_head.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.function_head.2.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.function_head.2.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.function_head.3.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.function_head.3.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.residue_head.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.residue_head.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.residue_head.2.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.residue_head.2.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.residue_head.3.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.residue_head.3.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sasa_head.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sasa_head.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sasa_head.2.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sasa_head.2.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sasa_head.3.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sasa_head.3.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sequence_head.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sequence_head.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sequence_head.2.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sequence_head.2.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sequence_head.3.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.sequence_head.3.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.ss8_head.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.ss8_head.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.ss8_head.2.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.ss8_head.2.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.ss8_head.3.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.ss8_head.3.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.structure_head.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.structure_head.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.structure_head.2.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.structure_head.2.weight": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.structure_head.3.bias": "model-00002-of-00002.safetensors",
+    "backbone.output_heads.structure_head.3.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.0.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.geom_attn.distance_scale_per_head": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.geom_attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.geom_attn.proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.geom_attn.rotation_scale_per_head": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.0.geom_attn.s_norm.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.1.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.1.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.1.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.1.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.1.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.1.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.1.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.1.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.1.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.1.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.10.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.10.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.10.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.10.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.10.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.10.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.10.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.10.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.10.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.10.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.11.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.11.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.11.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.11.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.11.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.11.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.11.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.11.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.11.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.11.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.12.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.12.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.12.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.12.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.12.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.12.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.12.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.12.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.12.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.12.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.13.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.13.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.13.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.13.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.13.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.13.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.13.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.13.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.13.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.13.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.14.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.14.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.14.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.14.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.14.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.14.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.14.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.14.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.14.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.14.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.15.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.15.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.15.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.15.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.15.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.15.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.15.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.15.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.15.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.15.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.16.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.16.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.16.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.16.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.16.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.16.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.16.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.16.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.16.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.16.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.17.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.17.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.17.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.17.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.17.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.17.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.17.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.17.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.17.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.17.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.18.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.18.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.18.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.18.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.18.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.18.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.18.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.18.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.18.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.18.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.19.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.19.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.19.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.19.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.19.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.19.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.19.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.19.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.19.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.19.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.2.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.2.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.2.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.2.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.2.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.2.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.2.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.2.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.2.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.2.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.20.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.20.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.20.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.20.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.20.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.20.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.20.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.20.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.20.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.20.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.21.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.21.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.21.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.21.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.21.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.21.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.21.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.21.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.21.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.21.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.22.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.22.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.22.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.22.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.22.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.22.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.22.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.22.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.22.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.22.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.23.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.23.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.23.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.23.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.23.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.23.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.23.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.23.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.23.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.23.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.24.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.24.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.24.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.24.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.24.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.24.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.24.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.24.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.24.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.24.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.25.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.25.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.25.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.25.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.25.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.25.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.25.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.25.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.25.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.25.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.26.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.26.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.26.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.26.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.26.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.26.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.26.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.26.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.26.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.26.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.27.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.27.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.27.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.27.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.27.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.27.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.27.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.27.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.27.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.27.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.28.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.28.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.28.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.28.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.28.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.28.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.28.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.28.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.28.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.28.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.29.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.29.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.29.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.29.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.29.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.29.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.29.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.29.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.29.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.29.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.3.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.3.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.3.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.3.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.3.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.3.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.3.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.3.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.3.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.3.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.30.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.30.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.30.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.30.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.30.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.30.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.30.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.30.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.30.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.30.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.31.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.31.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.31.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.31.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.31.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.31.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.31.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.31.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.31.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.31.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.32.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.32.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.32.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.32.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.32.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.32.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.32.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.32.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.32.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.32.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.33.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.33.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.33.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.33.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.33.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.33.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.33.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.33.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.33.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.33.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.34.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.34.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.34.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.34.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.34.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.34.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.34.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.34.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.34.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.34.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.35.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.35.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.35.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.35.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.35.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.35.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.35.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.35.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.35.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.35.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.36.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.36.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.36.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.36.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.36.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.36.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.36.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.36.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.36.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.36.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.37.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.37.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.37.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.37.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.37.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.37.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.37.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.37.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.37.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.37.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.38.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.38.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.38.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.38.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.38.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.38.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.38.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.38.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.38.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.38.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.39.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.39.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.39.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.39.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.39.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.39.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.39.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.39.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.39.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.39.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.4.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.4.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.4.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.4.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.4.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.4.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.4.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.4.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.4.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.4.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.40.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.40.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.40.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.40.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.40.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.40.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.40.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.40.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.40.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.40.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.41.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.41.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.41.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.41.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.41.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.41.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.41.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.41.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.41.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.41.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.42.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.42.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.42.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.42.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.42.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.42.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.42.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.42.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.42.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.42.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.43.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.43.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.43.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.43.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.43.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.43.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.43.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.43.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.43.ffn.1.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.43.ffn.3.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.44.attn.k_ln.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.44.attn.layernorm_qkv.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.44.attn.layernorm_qkv.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.44.attn.layernorm_qkv.1.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.44.attn.out_proj.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.44.attn.q_ln.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.44.ffn.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.44.ffn.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.44.ffn.1.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.44.ffn.3.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.45.attn.k_ln.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.45.attn.layernorm_qkv.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.45.attn.layernorm_qkv.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.45.attn.layernorm_qkv.1.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.45.attn.out_proj.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.45.attn.q_ln.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.45.ffn.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.45.ffn.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.45.ffn.1.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.45.ffn.3.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.46.attn.k_ln.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.46.attn.layernorm_qkv.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.46.attn.layernorm_qkv.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.46.attn.layernorm_qkv.1.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.46.attn.out_proj.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.46.attn.q_ln.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.46.ffn.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.46.ffn.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.46.ffn.1.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.46.ffn.3.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.47.attn.k_ln.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.47.attn.layernorm_qkv.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.47.attn.layernorm_qkv.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.47.attn.layernorm_qkv.1.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.47.attn.out_proj.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.47.attn.q_ln.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.47.ffn.0.bias": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.47.ffn.0.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.47.ffn.1.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.47.ffn.3.weight": "model-00002-of-00002.safetensors",
+    "backbone.transformer.blocks.5.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.5.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.5.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.5.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.5.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.5.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.5.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.5.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.5.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.5.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.6.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.6.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.6.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.6.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.6.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.6.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.6.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.6.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.6.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.6.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.7.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.7.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.7.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.7.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.7.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.7.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.7.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.7.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.7.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.7.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.8.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.8.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.8.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.8.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.8.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.8.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.8.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.8.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.8.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.8.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.9.attn.k_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.9.attn.layernorm_qkv.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.9.attn.layernorm_qkv.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.9.attn.layernorm_qkv.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.9.attn.out_proj.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.9.attn.q_ln.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.9.ffn.0.bias": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.9.ffn.0.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.9.ffn.1.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.blocks.9.ffn.3.weight": "model-00001-of-00002.safetensors",
+    "backbone.transformer.norm.weight": "model-00002-of-00002.safetensors",
+    "pooler.mut_weight": "model-00002-of-00002.safetensors",
+    "pooler.norm.weight": "model-00002-of-00002.safetensors",
+    "pooler.wt_weight": "model-00002-of-00002.safetensors",
+    "regressor.bias": "model-00002-of-00002.safetensors",
+    "regressor.weight": "model-00002-of-00002.safetensors"
+  }
+}