---
language:
- nl
datasets:
- yhavinga/mc4_nl_cleaned
tags:
- seq2seq
- lm-head
license: apache-2.0
inference: false
---

# Work in progress. Jan 2022

This model is a re-training of the original [t5-base-dutch](https://huggingface.co/flax-community/t5-base-dutch) model that was trained during the summer 2021 HuggingFace Flax/Jax community week. These two models are now the same. Eval accuracy 0.70 and loss 1,38.

NB! Consider using [yhavinga/t5-v1.1-base-dutch-cased](https://huggingface.co/yhavinga/t5-v1.1-base-dutch-cased) that has an eval accuracy of 0,78 and loss 0,96.

These models need to be finetuned, therefore the inference widget on the right has been turned off.

# A collection of Dutch T5 models

* Many thanks to the [Google TPU Research Cloud](https://sites.research.google/trc/about/) for providing access to a TPU cluster!
* Continuation of work started during the [Hugging Face community week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), organized by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google, for the project [Pre-train T5 from scratch in Dutch](https://discuss.huggingface.co/t/pretrain-t5-from-scratch-in-dutch/8109).
* Using improved training script - no more exceptions during training, so no restarting required.
* All models trained with tensorboard metrics.
* Thanks to @gsarti for creating the [t5-flax-gcp repository](https://github.com/gsarti/t5-flax-gcp)!