File size: 2,943 Bytes
b355cb3 b05df5b b355cb3 265db82 b355cb3 265db82 b355cb3 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 |
---
language:
- cs
- en
- de
tags:
- Czech
- English
- German
- trilingual
- KKY
- FAV
license: cc-by-nc-sa-4.0
---
# wav2vec2-base-cs-en-de-150k
This is a trilingual Wav2Vec 2.0 base model pre-trained from 150 thousand hours of speech (50 thousand hours of Czech, 50 thousand hours of English and 50 thousand hours of German).
It has been released along with a paper **A Comparative Analysis of Bilingual and Trilingual Wav2Vec Models for
Automatic Speech Recognition in Multilingual Oral History Archives** accepted to INTERSPEECH2024 conference.
## Paper
https://www.isca-archive.org/interspeech_2024/lehecka24_interspeech.pdf
Pre-print: http://arxiv.org/abs/2407.17160.
### All pre-trained models released along with the paper
- [fav-kky/wav2vec2-base-cs-50k](https://huggingface.co/fav-kky/wav2vec2-base-cs-50k) (monolingual Czech)
- [fav-kky/wav2vec2-base-de-50k](https://huggingface.co/fav-kky/wav2vec2-base-de-50k) (monolingual German)
- [fav-kky/wav2vec2-base-cs-en-100k](https://huggingface.co/fav-kky/wav2vec2-base-cs-en-100k) (bilingual Czech+English)
- [fav-kky/wav2vec2-base-cs-de-100k](https://huggingface.co/fav-kky/wav2vec2-base-cs-de-100k) (bilingual Czech+German)
- [fav-kky/wav2vec2-base-en-de-100k](https://huggingface.co/fav-kky/wav2vec2-base-en-de-100k) (bilingual English+German)
- [fav-kky/wav2vec2-base-cs-en-de-150k](https://huggingface.co/fav-kky/wav2vec2-base-cs-en-de-150k) (trilingual Czech+English+German)
## Citation
If you find this model useful, please cite our paper:
```
@inproceedings{lehecka24_interspeech,
title = {A Comparative Analysis of Bilingual and Trilingual Wav2Vec Models for Automatic Speech Recognition in Multilingual Oral History Archives},
author = {Jan Lehečka and Josef V. Psutka and Lubos Smidl and Pavel Ircing and Josef Psutka},
year = {2024},
booktitle = {Interspeech 2024},
pages = {1285--1289},
doi = {10.21437/Interspeech.2024-472},
issn = {2958-1796},
}
```
## Usage
This model does not have a tokenizer as it was pretrained on audio alone.
In order to use this model for speech recognition, a tokenizer should be created
and the model should be [fine-tuned](https://huggingface.co/blog/fine-tune-wav2vec2-english) on labeled ASR data.
Inputs must be 16kHz mono audio files.
This model can be used e.g., to extract per-frame contextual embeddings from audio:
```python
from transformers import Wav2Vec2Model, Wav2Vec2FeatureExtractor
import torchaudio
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("fav-kky/wav2vec2-base-cs-en-de-150k")
model = Wav2Vec2Model.from_pretrained("fav-kky/wav2vec2-base-cs-en-de-150k")
speech_array, sampling_rate = torchaudio.load("/path/to/audio/file.wav")
inputs = feature_extractor(
speech_array,
sampling_rate=16_000,
return_tensors="pt"
)["input_values"][0]
output = model(inputs)
embeddings = output.last_hidden_state.detach().numpy()[0]
```
## Related works |