Update README.md
Browse files
README.md
CHANGED
@@ -28,12 +28,12 @@ widget:
|
|
28 |
---
|
29 |
|
30 |
# NB-SBERT
|
31 |
-
|
32 |
|
33 |
-
The model maps sentences & paragraphs to a 768 dimensional dense vector space. This vector can be used for tasks like clustering and semantic search. Below we give some examples on how to use the model. The easiest way is to simply measure the cosine distance between two sentences. Sentences that are close to each other in meaning, will have a small cosine distance and a similarity close to 1. The model is trained in a way
|
34 |
|
35 |
## Keyword Extraction
|
36 |
-
The model can be used for extracting keywords from
|
37 |
|
38 |
```bash
|
39 |
pip install keybert
|
@@ -56,19 +56,41 @@ kw_model.extract_keywords(doc, stop_words=None)
|
|
56 |
# [('nasjonalbibliotek', 0.5242), ('bibliotek', 0.4342), ('samlinger', 0.3334), ('statsoverhode', 0.33), ('manuskripter', 0.3061)]
|
57 |
```
|
58 |
|
59 |
-
The [
|
60 |
|
61 |
## Keyword Extraction
|
62 |
[ToDo - Per Egil - https://github.com/MaartenGr/BERTopic]
|
63 |
|
64 |
|
65 |
## Similarity Search
|
66 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
67 |
|
68 |
|
69 |
## Embeddings and Sentence Similarity (Sentence-Transformers)
|
70 |
|
71 |
-
|
72 |
|
73 |
```bash
|
74 |
pip install -U sentence-transformers
|
@@ -98,10 +120,8 @@ print(scipy_cosine_scores)
|
|
98 |
```
|
99 |
|
100 |
|
101 |
-
|
102 |
-
|
103 |
## Embeddings and Sentence Similarity (HuggingFace Transformers)
|
104 |
-
Without [sentence-transformers](https://www.SBERT.net), you can use the model
|
105 |
|
106 |
```python
|
107 |
from transformers import AutoTokenizer, AutoModel
|
|
|
28 |
---
|
29 |
|
30 |
# NB-SBERT
|
31 |
+
NB-SBERT is a [SentenceTransformers](https://www.SBERT.net) model trained on a [machine translated version of the MNLI dataset](https://huggingface.co/datasets/NbAiLab/mnli-norwegian), starting from [nb-bert-base](https://huggingface.co/NbAiLab/nb-bert-base).
|
32 |
|
33 |
+
The model maps sentences & paragraphs to a 768 dimensional dense vector space. This vector can be used for tasks like clustering and semantic search. Below we give some examples on how to use the model. The easiest way is to simply measure the cosine distance between two sentences. Sentences that are close to each other in meaning, will have a small cosine distance and a similarity close to 1. The model is trained in such a way that similar sentences in different languages should also be close to each other. Ideally, an English-Norwegian sentence pair should have high similarity.
|
34 |
|
35 |
## Keyword Extraction
|
36 |
+
The model can be used for extracting keywords from text. The basic technique is to find the words that are most similar to the document. There are various frameworks for doing this. An easy way is to use [KeyBERT](https://github.com/MaartenGr/KeyBERT). This example shows how this can be done.
|
37 |
|
38 |
```bash
|
39 |
pip install keybert
|
|
|
56 |
# [('nasjonalbibliotek', 0.5242), ('bibliotek', 0.4342), ('samlinger', 0.3334), ('statsoverhode', 0.33), ('manuskripter', 0.3061)]
|
57 |
```
|
58 |
|
59 |
+
The [KeyBERT homepage](https://github.com/MaartenGr/KeyBERT) provides other several interesting examples: combining KeyBERT with stop words, extracting longer phrases, or directly producing highlighted text.
|
60 |
|
61 |
## Keyword Extraction
|
62 |
[ToDo - Per Egil - https://github.com/MaartenGr/BERTopic]
|
63 |
|
64 |
|
65 |
## Similarity Search
|
66 |
+
Another common use case for a SentenceTransformers model is to find relevant documents or passages of documents given a certain query text. In this scenario, it is pretty common to have a vector database that stores the embedding vectors for all our documents. Then, at runtime, an embedding for the query text is generated and compared efficiently against the vector database.
|
67 |
+
|
68 |
+
While production vector databases exist, a quick way to experiment with them is by using [`autofaiss`](https://github.com/criteo/autofaiss):
|
69 |
+
|
70 |
+
```bash
|
71 |
+
pip install autofaiss sentence-transformers
|
72 |
+
```
|
73 |
+
|
74 |
+
```python
|
75 |
+
from autofaiss import build_index
|
76 |
+
import numpy as np
|
77 |
+
|
78 |
+
from sentence_transformers import SentenceTransformer, util
|
79 |
+
sentences = ["This is a Norwegian boy", "Dette er en norsk gutt", "A red house"]
|
80 |
+
|
81 |
+
model = SentenceTransformer('NbAiLab/nb-sbert')
|
82 |
+
embeddings = model.encode(sentences)
|
83 |
+
index, index_infos = build_index(embeddings, save_on_disk=False)
|
84 |
+
|
85 |
+
query = model.encode(["A young boy"])
|
86 |
+
_, index_matches = index.search(query, 1)
|
87 |
+
print(index_matches)
|
88 |
+
```
|
89 |
|
90 |
|
91 |
## Embeddings and Sentence Similarity (Sentence-Transformers)
|
92 |
|
93 |
+
As seen above, using the library [sentence-transformers](https://www.SBERT.net) makes the use of these models quite convenient:
|
94 |
|
95 |
```bash
|
96 |
pip install -U sentence-transformers
|
|
|
120 |
```
|
121 |
|
122 |
|
|
|
|
|
123 |
## Embeddings and Sentence Similarity (HuggingFace Transformers)
|
124 |
+
Without [sentence-transformers](https://www.SBERT.net), you can still use the model. First, you pass in your input through the transformer model, then you have to apply the right pooling-operation on top of the contextualized word embeddings.
|
125 |
|
126 |
```python
|
127 |
from transformers import AutoTokenizer, AutoModel
|