update

README.md

@@ -0,0 +1,144 @@
+---
+thumbnail: https://github.com/rinnakk/japanese-pretrained-models/blob/master/rinna.png
+license: llama2
+language:
+- ja
+- en
+inference: false
+datasets:
+- databricks/databricks-dolly-15k
+- kunishou/databricks-dolly-15k-ja
+- izumi-lab/llm-japanese-dataset
+---
+
+# `rinna/youri-7b-instruction`
+
+![rinna-icon](./rinna.png)
+
+# Overview
+The model is the instruction-tuned version of [`rinna/youri-7b`](https://huggingface.co/rinna/youri-7b). It adopts the Alpaca input format.
+
+* **Model architecture**
+
+    A 32-layer, 4096-hidden-size transformer-based language model. Refer to the [llama2 paper](https://arxiv.org/abs/2307.09288) for architecture details.
+
+* **Fine-tuning**
+    
+    The fine-tuning data is the subset of the following datasets.
+    * [Databricks Dolly data](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
+    * [Japanese Databricks Dolly data](https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja)
+    * [FLAN Instruction Tuning data](https://github.com/google-research/FLAN) and its Japanese translation
+    * [Izumi lab LLM Japanese dataset](https://github.com/masanorihirano/llm-japanese-dataset/tree/main)
+      * The following sections are used
+        * alt
+        * aozora-txt
+        * CourseraParallel
+        * ParaNatCom
+        * Tab-delimited_Bilingual_Sentence_Pairs
+        * tanaka-corpus
+        * wikinews
+        * wordnet
+        * yasashi-japanese
+      * The [remaining sections](https://github.com/masanorihirano/llm-japanese-dataset/tree/main/datasets-cc-by-sa) contain commonly used evaluation corpora so they are skipped to prevent data leak.
+
+* **Authors**
+    
+    - [Tianyu Zhao](https://huggingface.co/tianyuz)
+    - [Kei Sawada](https://huggingface.co/keisawada)
+
+---
+
+# Benchmarking
+
+Evaluation experiments suggest that rinna's `youri-7b` series outperforms other open-source Japanese LLMs on Japanese tasks according to our runs. 
+
+| Model | Model type | 4-task score | 6-task score | 8-task score |
+| :-- | :-- | :-- | :-- | :-- | 
+| **rinna/youri-7b-instruction** | **SFT** | **83.88** | **80.93** | **63.63** |
+| rinna/youri-7b-chat | SFT | 78.29 | 78.47 | 62.18 |
+| matsuo-lab/weblab-10b-instruction-sft | SFT | 78.75 | 75.05 | 59.11 |
+| rinna/youri-7b | pre-trained | 73.32 | 74.58 | 58.87 |
+| stabilityai/japanese-stablelm-instruct-alpha-7b | SFT | 70.10 | 71.32 | 54.71 |
+| elyza/ELYZA-japanese-Llama-2-7b | pre-trained | 71.72 | 69.28 | 53.17 |
+| elyza/ELYZA-japanese-Llama-2-7b-instruct | SFT | 70.57 | 68.12 | 53.14 |
+| stabilityai/japanese-stablelm-base-alpha-7b | pre-trained | 61.03 | 65.83 | 51.05 |
+| matsuo-lab/weblab-10b | pre-trained | 66.33 | 65.58 | 50.74 |
+| meta/llama2-7b | pre-trained | 56.33 | 54.80 | 42.97 |
+| rinna/japanese-gpt-neox-3.6b | pre-trained | 47.20 | 54.68 | 41.80 |
+| rinna/bilingual-gpt-neox-4b | pre-trained | 46.60 | 52.04 | 40.03 |
+    
+---
+
+# How to use the model
+
+~~~~python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+tokenizer = AutoTokenizer.from_pretrained("rinna/youri-7b-instruction")
+model = AutoModelForCausalLM.from_pretrained("rinna/youri-7b-instruction")
+
+if torch.cuda.is_available():
+    model = model.to("cuda")
+
+instruction = "次の日本語を英語に翻訳してください。"
+input = "大規模言語モデル（だいきぼげんごモデル、英: large language model、LLM）は、多数のパラメータ（数千万から数十億）を持つ人工ニューラルネットワークで構成されるコンピュータ言語モデルで、膨大なラベルなしテキストを使用して自己教師あり学習または半教師あり学習によって訓練が行われる。"
+prompt = f"""
+以下は、タスクを説明する指示と、文脈のある入力の組み合わせです。要求を適切に満たす応答を書きなさい。
+
+### 指示:
+{instruction}
+
+### 入力:
+{input}
+
+### 応答:
+"""
+token_ids = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
+
+with torch.no_grad():
+    output_ids = model.generate(
+        token_ids.to(model.device),
+        max_new_tokens=200,
+        do_sample=True,
+        temperature=0.5,
+        pad_token_id=tokenizer.pad_token_id,
+        bos_token_id=tokenizer.bos_token_id,
+        eos_token_id=tokenizer.eos_token_id
+    )
+
+output = tokenizer.decode(output_ids.tolist()[0])
+print(output)
+"""
+以下は、タスクを説明する指示と、文脈のある入力の組み合わせです。要求を適切に満たす応答を書きなさい。
+
+### 指示:
+次の日本語を英語に翻訳してください。
+
+### 入力:
+大規模言語モデル（だいきぼげんごモデル、英: large language model、LLM）は、多数のパラメータ（数千万から数十億）を持つ人工ニューラルネットワークで構成されるコンピュータ言語モデルで、膨大なラベルなしテキストを使 用して自己教師あり学習または半教師あり学習によって訓練が行わ���る。
+
+### 応答:
+ Large language models (LLMs) are computer language models that are composed of artificial neural networks with millions to billions of parameters that are trained via self-supervised or semi-supervised learning using vast unlabeled text.</s>
+"""
+~~~~
+
+---
+
+# Tokenization
+The model uses the original llama-2 tokenizer.
+
+---
+
+# How to cite
+~~~
+@misc{RinnaYouri7bInstruction,, 
+    url={https://huggingface.co/rinna/youri-7b-instruction}, 
+    title={rinna/youri-7b-instruction}, 
+    author={Zhao, Tianyu and Sawada, Kei}
+}
+~~~
+---
+
+# License
+[The llama2 license](https://ai.meta.com/llama/license/)

config.json

@@ -0,0 +1,27 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 4096,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 32,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.34.0",
+  "use_cache": true,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "transformers_version": "4.34.0"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:04fe7e03e59d03717adf77a5cc88f8d90cb16a43ea1fcdd39de133cbc995ccce
+size 13476871854

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer_checklist.chk

@@ -0,0 +1 @@
+eeec4125e9c7560836b4873b6f8e3025  tokenizer.model

tokenizer_config.json

@@ -0,0 +1,35 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}