NorBert 2

README.md

@@ -1,3 +1,27 @@
----
-license: cc-by-4.0
----
+---
+language: no
+license: cc-by-4.0
+pipeline_tag: fill-mask
+tags:
+- norwegian
+- bert
+thumbnail: https://raw.githubusercontent.com/ltgoslo/NorBERT/main/Norbert.png
+---
+## Quickstart
+**Release 2.0** (February 7, 2022)
+Trained on the very large corpus of Norwegian (C4 + NCC, about 15 billion word tokens).
+Features a 50 000 words vocabulary and was trained using Whole Word Masking.
+
+Download the model here:
+* Cased Norwegian BERT Base 2.0 (NorBERT 2): [221.zip](http://vectors.nlpl.eu/repository/20/221.zip)
+More about NorBERT training corpora, training procedure and evaluation benchmarks: http://norlm.nlpl.eu/
+
+Associated code: https://github.com/ltgoslo/NorBERT
+
+Check this paper for more details:
+_Andrey Kutuzov, Jeremy Barnes, Erik Velldal, Lilja Øvrelid, Stephan Oepen. [Large-Scale Contextualised Language Modelling for Norwegian](https://arxiv.org/abs/2104.06546), NoDaLiDa'21 (2021)_
+
+NorBERT was trained as a part of NorLM, a joint initiative of the projects [EOSC-Nordic](https://www.eosc-nordic.eu/) (European Open Science Cloud),
+coordinated by the [Language Technology Group](https://www.mn.uio.no/ifi/english/research/groups/ltg/) (LTG) at the University of Oslo.
+
+The computations were performed on resources provided by UNINETT Sigma2 - the National Infrastructure for High Performance Computing and Data Storage in Norway.

config.json

@@ -0,0 +1,17 @@
+{
+  "architectures": [
+  "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "type_vocab_size": 2,
+  "vocab_size": 50104
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c15dc6df9ed0cd6db1726ea7a2d5e8a7e73fa950065cb60436f09c1e7a6609c
+size 500746539

tf_model.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8741bc2e9666330a411b6e2d1d59280e31c047d141b73f82976ee3d2445b7449
+size 498346776

tokenizer_config.json

@@ -0,0 +1,3 @@
+{
+  "do_lower_case": false
+}