upload model

README.md

@@ -1,121 +0,0 @@
----
-tags:
-- transformers
-license: cc-by-nc-4.0
-pipeline_tag: feature-extraction
-language:
-- en
----
-
-<h1 align="center">Linq-AI-Research/Linq-Embed-Mistral</h1>
-
-**Linq-Embed-Mistral**
-
-Linq-Embed-Mistral has been developed by building upon the foundations of the [E5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct) and [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) models. We focus on improving text retrieval using advanced data refinement methods, including sophisticated data crafting, data filtering, and negative mining techniques. These methods are applied to both existing benchmark datasets and highly tailored synthetic datasets generated via LLMs. To enhance the quality of the synthetic data, we employ extensive prompt engineering and guidance from teacher models, ensuring these methods are specifically tailored to each task. Our efforts primarily aim to create high-quality triplet datasets (query, positive example, negative example), significantly improving text retrieval performance.
-
-Linq-Embed-Mistral performs exceptionally well in the MTEB benchmarks, achieving an average score of 68.1 across 56 datasets. This performance ranks it 1st among publicly accessible models on the MTEB leaderboard and 3rd overall among all evaluated models. The model excels in retrieval tasks, ranking 1st among all models listed on the MTEB leaderboard with a performance score of 60.0.
-
-This project is for research purposes only. Third-party datasets may be subject to additional terms and conditions under their associated licenses. Please refer to specific papers for more details:
-
-- [MTEB benchmark](https://arxiv.org/abs/2210.07316)
-- [Mistral](https://arxiv.org/abs/2310.06825)
-- [E5-mistral-7b-instruct](https://arxiv.org/pdf/2401.00368.pdf)
-
-For more details, refer to [this blog post](https://getlinq.com/blog/linq-embed-mistral/).
-
-## How to use
-
-### Transformers
-Here is an example of how to encode queries and passages from the Mr.TyDi training dataset.
-```python
-import torch
-import torch.nn.functional as F
-from torch import Tensor
-from transformers import AutoTokenizer, AutoModel
-
-def last_token_pool(last_hidden_states: Tensor,
-                 attention_mask: Tensor) -> Tensor:
-    left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
-    if left_padding:
-        return last_hidden_states[:, -1]
-    else:
-        sequence_lengths = attention_mask.sum(dim=1) - 1
-        batch_size = last_hidden_states.shape[0]
-        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]
-
-def get_detailed_instruct(task_description: str, query: str) -> str:
-    return f'Instruct: {task_description}\nQuery: {query}'
-
-# Each query must come with a one-sentence instruction that describes the task
-task = 'Given a question, retrieve Wikipedia passages that answer the question'
-queries = [
-    get_detailed_instruct(task, '최초의 원자력 발전소는 무엇인가?'),
-    get_detailed_instruct(task, 'Who invented Hangul?')
-]
-# No need to add instruction for retrieval documents
-passages = [
-    "현재 사용되는 핵분열 방식을 이용한 전력생산은 1948년 9월 미국 테네시주 오크리지에 설치된 X-10 흑연원자로에서 전구의 불을 밝히는 데 사용되면서 시작되었다. 그리고 1954년 6월에 구소련의 오브닌스크에 건설된 흑연감속 비등경수 압력관형 원자로를 사용한 오브닌스크 원자력 발전소가 시험적으로 전력생산을 시작하였고, 최초의 상업용 원자력 엉더이로를 사용한 영국 셀라필드 원자력 단지에 위치한 콜더 홀(Calder Hall) 원자력 발전소로, 1956년 10월 17일 상업 운전을 시작하였다.",
-    "Hangul was personally created and promulgated by the fourth king of the Joseon dynasty, Sejong the Great.[1][2] Sejong's scholarly institute, the Hall of Worthies, is often credited with the work, and at least one of its scholars was heavily involved in its creation, but it appears to have also been a personal project of Sejong."
-]
-
-# Load model and tokenizer
-tokenizer = AutoTokenizer.from_pretrained('Salesforce/SFR-Embedding-Mistral')
-model = AutoModel.from_pretrained('Salesforce/SFR-Embedding-Mistral')
-input_texts = [*queries, *passages]
-max_length = 4096
-# Tokenize the input texts
-batch_dict = tokenizer(input_texts, max_length=max_length, padding=True, truncation=True, return_tensors="pt")
-outputs = model(**batch_dict)
-embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
-
-# Normalize embeddings
-embeddings = F.normalize(embeddings, p=2, dim=1)
-scores = (embeddings[:2] @ embeddings[2:].T) * 100
-print(scores.tolist())
-# [[86.7153549194336, 36.64569091796875], [35.00493621826172, 82.0738525390625]]
-```
-
-### MTEB Benchmark Evaluation
-Check out [unilm/e5](https://github.com/microsoft/unilm/tree/master/e5) to reproduce evaluation results on the [BEIR](https://arxiv.org/abs/2104.08663) and [MTEB](https://arxiv.org/abs/2210.07316) benchmark.
-
-## Evaluation Result
-
-### MTEB
-
-
-|                                    Model Name                                    | Retrieval (15) | Average (56) |
-| :------------------------------------------------------------------------------: | :------------: | :----------: |
-| [Linq-Embed-Mistral](https://huggingface.co/Linq-AI-Research/Linq-Embed-Mistral) |      **60.0**      |     68.1     |
-|             [NV-Embed-v1](https://huggingface.co/nvidia/NV-Embed-v1)             |      59.4      |     69.3     |
-| [SFR-Embedding-Mistral](https://huggingface.co/Salesforce/SFR-Embedding-Mistral) |      59.0      |     67.6     |
-|                           [voyage-large-2-instruct]()                            |      58.3      |     68.3     |
-|                                  [GritLM-7B]()                                   |      57.4      |     66.8     |
-|                             voyage-lite-02-instruct                              |      56.6      |     67.1     |
-|[gte-Qwen1.5-7B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen1.5-7B-instruct)|      56.2      |     67.3     |
-| [e5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct) |      56.9      |     66.6     |
-|                     google-gecko.text-embedding-preview-0409                     |      55.7      |     66.3     |
-|                              text-embedding-3-large                              |      55.4      |     64.6     |
-|                            Cohere-embed-english-v3.0                             |      55.0      |     64.5     |
-
-Linq Research Team.
-
-- Junseong Kim
-- Seolhwa Lee
-- Jihoon Kwon
-- Sangmo Gu
-- Yejin Kim
-- Minkyung Cho
-- Jy-yong Sohn
-- Chanyeol Choi
-
-### Citation
-
-```bibtex
-@misc{LinqAIResearch2024,
-  title={Linq-Embed-Mistral:Elevating Text Retrieval with Improved GPT Data Through Task-Specific Control and Quality Refinement},
-  author={Junseong Kim, Seolhwa Lee, Jihoon Kwon, Sangmo Gu, Yejin Kim, Minkyung Cho, Jy-yong Sohn, Chanyeol Choi},
-  howpublished={Linq AI Research Blog},
-  year={2024},
-  url={https://getlinq.com/blog/linq-embed-mistral/}
-}
-```

config.json

@@ -0,0 +1,27 @@
+{
+  "_name_or_path": "intfloat/e5-mistral-7b-instruct",
+  "architectures": [
+    "MistralModel"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "model_type": "mistral",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.39.0",
+  "use_cache": false,
+  "vocab_size": 32000
+}

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:72a2e78bd39844eb2c4041525f8742ffddcceaf594082dadb7bd4085a62b3e3a
+size 4943161664

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff209caf16ced9cd45770a8cdeed50137a778569d6ce2908dc59e1c8aad26fb8
+size 4999818600

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5d64289b8791d2a13c4a5bbfb5246996c7f0f10ddaeaa8ae070ace37d68f33a
+size 4278371624

model.safetensors.index.json

@@ -0,0 +1,297 @@
+{
+  "metadata": {
+    "total_size": 14221320192
+  },
+  "weight_map": {
+    "embed_tokens.weight": "model-00001-of-00003.safetensors",
+    "layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.1.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.10.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.10.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.10.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.11.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.11.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.11.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.12.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.12.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.13.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.13.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.14.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.14.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.15.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.15.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.16.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.16.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.17.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.17.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.18.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.18.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.19.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.19.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.2.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.20.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.20.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.20.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.21.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.21.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.22.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.22.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.22.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.22.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.22.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.22.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.22.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.22.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.22.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.23.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.23.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.23.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.24.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.24.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.25.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.25.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.26.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.26.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.27.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.27.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.28.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.28.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.29.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.29.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.3.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.30.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.30.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.30.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.31.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.31.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.4.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.5.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.6.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.7.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.8.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.9.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.9.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "norm.weight": "model-00003-of-00003.safetensors"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,35 @@
+{
+  "additional_special_tokens": [
+    "<unk>",
+    "<s>",
+    "</s>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
+size 493443

tokenizer_config.json

@@ -0,0 +1,46 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<unk>",
+    "<s>",
+    "</s>"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}