Xenova HF staff commited on
Commit
10f6b37
1 Parent(s): e455a4b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -6,7 +6,7 @@ tags:
6
  - tokenizers
7
  ---
8
 
9
- # GPT-4 Tokenizer
10
 
11
  A 🤗-compatible version of the **GPT-4o tokenizer** (adapted from [openai/tiktoken](https://github.com/openai/tiktoken)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).
12
 
@@ -24,6 +24,6 @@ assert tokenizer.encode('hello world') == [24912, 2375]
24
  ```js
25
  import { AutoTokenizer } from '@xenova/transformers';
26
 
27
- const tokenizer = await AutoTokenizer.from_pretrained('Xenova/gpt-4');
28
  const tokens = tokenizer.encode('hello world'); // [24912, 2375]
29
  ```
 
6
  - tokenizers
7
  ---
8
 
9
+ # GPT-4o Tokenizer
10
 
11
  A 🤗-compatible version of the **GPT-4o tokenizer** (adapted from [openai/tiktoken](https://github.com/openai/tiktoken)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).
12
 
 
24
  ```js
25
  import { AutoTokenizer } from '@xenova/transformers';
26
 
27
+ const tokenizer = await AutoTokenizer.from_pretrained('Xenova/gpt-4o');
28
  const tokens = tokenizer.encode('hello world'); // [24912, 2375]
29
  ```