LLukas22 commited on
Commit
fcf6d02
1 Parent(s): 9118467

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +36 -17
README.md CHANGED
@@ -7,20 +7,9 @@ tags:
7
  - llm
8
  - ggml
9
  ---
10
-
11
-
12
  # GGML converted versions of [Mosaic's](https://huggingface.co/mosaicml) MPT Models
13
 
14
-
15
- ## CAUTION: MPT Development is still ongoing and not finished!
16
- - Rust & Python: Rustformers implementation see here: [Implement MPT Model](https://github.com/rustformers/llm/pull/218)
17
-
18
- If these implementations are complete i will add instructions on how to run the models and update them if necesary!
19
-
20
-
21
  ## Converted Models:
22
-
23
-
24
  | Name | Based on | Type | Container |
25
  |-|-|-|-|
26
  | [mpt-7b-f16.bin](https://huggingface.co/LLukas22/mpt-7b-ggml/blob/main/mpt-7b-f16.bin) | [mpt-7b](https://huggingface.co/mosaicml/mpt-7b) | fp16 | GGML |
@@ -36,16 +25,46 @@ If these implementations are complete i will add instructions on how to run the
36
  | [mpt-7b-storywriter-q4_0.bin](https://huggingface.co/LLukas22/mpt-7b-ggml/blob/main/mpt-7b-storywriter-q4_0.bin) | [mpt-7b-storywriter](https://huggingface.co/mosaicml/mpt-7b-storywriter) | int4 | GGML |
37
  | [mpt-7b-storywriter-q4_0-ggjt.bin](https://huggingface.co/LLukas22/mpt-7b-ggml/blob/main/mpt-7b-storywriter-q4_0-ggjt.bin) | [mpt-7b-storywriter](https://huggingface.co/mosaicml/mpt-7b-storywriter) | int4 | GGJT |
38
 
 
 
 
 
 
 
 
 
39
 
40
- ## Usage
 
 
41
 
42
- ### Rust & Python:
43
- #### TBD See above!
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
44
 
45
- ### Via GGML
46
  The `GGML` example only supports the ggml container type!
47
 
48
- ##### Installation
49
 
50
  ```
51
  git clone https://github.com/ggerganov/ggml
@@ -55,7 +74,7 @@ cmake ..
55
  make -j4 mpt
56
  ```
57
 
58
- ##### Run inference
59
 
60
  ```
61
  ./bin/mpt -m path/to/model.bin -p "The meaning of life is"
 
7
  - llm
8
  - ggml
9
  ---
 
 
10
  # GGML converted versions of [Mosaic's](https://huggingface.co/mosaicml) MPT Models
11
 
 
 
 
 
 
 
 
12
  ## Converted Models:
 
 
13
  | Name | Based on | Type | Container |
14
  |-|-|-|-|
15
  | [mpt-7b-f16.bin](https://huggingface.co/LLukas22/mpt-7b-ggml/blob/main/mpt-7b-f16.bin) | [mpt-7b](https://huggingface.co/mosaicml/mpt-7b) | fp16 | GGML |
 
25
  | [mpt-7b-storywriter-q4_0.bin](https://huggingface.co/LLukas22/mpt-7b-ggml/blob/main/mpt-7b-storywriter-q4_0.bin) | [mpt-7b-storywriter](https://huggingface.co/mosaicml/mpt-7b-storywriter) | int4 | GGML |
26
  | [mpt-7b-storywriter-q4_0-ggjt.bin](https://huggingface.co/LLukas22/mpt-7b-ggml/blob/main/mpt-7b-storywriter-q4_0-ggjt.bin) | [mpt-7b-storywriter](https://huggingface.co/mosaicml/mpt-7b-storywriter) | int4 | GGJT |
27
 
28
+ ⚠️Caution⚠️: mpt-7b-storywriter is still under development!
29
+
30
+ ## Usage
31
+
32
+ ### Python via [llm-rs](https://github.com/LLukas22/llm-rs-python):
33
+
34
+ #### Installation
35
+ Via pip: `pip install llm-rs huggingface_hub`
36
 
37
+ #### Run inference
38
+ ```python
39
+ from llm_rs import Mpt
40
 
41
+ #Download the model
42
+ hf_hub_download(repo_id="LLukas22/mpt-7b-ggml", filename="mpt-7b-q4_0-ggjt.bin", local_dir=".")
43
+
44
+ #Load the model
45
+ model = Mpt("mpt-7b-q4_0-ggjt.bin")
46
+
47
+ #Generate
48
+ print(model.generate("The meaning of life is"))
49
+ ```
50
+
51
+ ### Rust via [Rustformers/llm](https://github.com/rustformers/llm):
52
+
53
+ #### Installation
54
+ ```
55
+ git clone --recurse-submodules git@github.com:rustformers/llm.git
56
+ cargo build --release
57
+ ```
58
+
59
+ #### Run inference
60
+ ```
61
+ cargo run --release -- mpt infer -m path/to/model.bin -p "Tell me how cool the Rust programming language is:"
62
+ ```
63
 
64
+ ### C via [GGML](https://github.com/ggerganov/ggml)
65
  The `GGML` example only supports the ggml container type!
66
 
67
+ #### Installation
68
 
69
  ```
70
  git clone https://github.com/ggerganov/ggml
 
74
  make -j4 mpt
75
  ```
76
 
77
+ #### Run inference
78
 
79
  ```
80
  ./bin/mpt -m path/to/model.bin -p "The meaning of life is"