shripadbhat

/

whisper-tiny-mr

like 0

I read some of your comments in the Whisper community. I am wondering if you achieved the objective of operating a number-detector on a small apparatus. Have you produced or obtained a whisper type model that can detect the 20 Marathi numbers in that language having a parameter count below 100 Million?

1. In the model, whisper-tiny-mr, it seems you have worked with {finetuned?} a pretrained model downloaded from somewhere else? Where did you get a pretrained "whisper-tiny-mr" ? URL?

I have some questions:
2) Why did you pretrain/finetune using a model having 50+thousand token vocabulary? It seems that you only needed a model having a little more than 20 token vocab (your 20 Marathi numbers). So, why deploy/finetune a model having a 50+ thousand token vocab? Did you answer the other guy's question: " Or is this in the context of speech recognition? i.e. someone speaks a full sentence, one word of which is a number that you want to transcribe correctly." You said [@sanchit-gandhi our case is simple earlier one of audio classification. This is the only thing we are doing with the model so don't know any other thing about the model.] Does that mean you only want a 20-logit classification model? Only for detecting 20 Marathi numbers

3. What are all of those two-letter "Special Tokens" in https://huggingface.co/shripadbhat/whisper-tiny-mr/blob/main/special_tokens_map.json for? Are those special tokens somehow representing your 20 Marathi numbers? If so, why did you implement the 20 Marathi numbers as Special Tokens?

4. Have you found, produced or obtained a whisper type model that can detect the 20 Marathi numbers in that language having a parameter count below 100 Million? Does this model also reliably refuse to output a number token when the input sound is not one of the the 20 Marathi numbers? (prevents False positives) Have you published code and weights for the small Whisper-type model that has both of these features?

5. Why did you stop training at Loss: 0.4618 ? That does not seem to be highly a reliable model checkpoint if you are only verifying loss for 20 tokens..... Why not train until a loss of below 0.01?

Loss: 0.4618
Wer: 41.6451

@@@@@@@@@@@@@@@@@@@@@@@@2

Copied from the other chat on Huggingface:

Hey @SameerMahajan ! To clarify, are you simply performing audio classification? i.e. you have an audio input where someone says a number, and you want to predict the number that they said. Or is this in the context of speech recognition? i.e. someone speaks a full sentence, one word of which is a number that you want to transcribe correctly.

Is the performance of the model otherwise good on Marathi? Am wondering whether you can fine-tune it for audio classification or speech recognition to boost Marathi performance as required (https://huggingface.co/blog/fine-tune-whisper)

SameerMahajan
Feb 17, 2023

@sanchit-gandhi our case is simple earlier one of audio classification. This is the only thing we are doing with the model so don't know any other thing about the model. Thanks for sharing your blog on fine tuning which we will take a look at to see whether it helps in our case.

sanchit-gandhi
Feb 22, 2023
•
edited Feb 22, 2023

Hey @SameerMahajan ! If you have a couple of hours of labelled audio-text data then you should definitely be able to fine-tune Whisper for this task!

One other thing you can try is using a Marathi model that has already been fine-tuned and see if it's any better: https://huggingface.co/spaces/whisper-event/leaderboard?dataset=mozilla-foundation%2Fcommon_voice_11_0&config=mr&split=test

The number one model has a demo built for it: https://huggingface.co/spaces/DrishtiSharma/Whisper-Marathi-Transcriber

IMO it's worth trying this out first and seeing how it performs.

You can also use the Python API to directly use the model in a Python script:

import torch
from transformers import pipeline

model_id = "DrishtiSharma/whisper-large-v2-marathi"

device = "cuda:0" if torch.cuda.is_available() else "cpu"

pipe = pipeline(
"automatic-speech-recognition",
model=model_id,
device=device,
)

audio = PATH/TO/YOUR/AUDIO

out = pipe(audio)

SameerMahajan
Feb 22, 2023

Thanks @sanchit-gandhi we will take a look. Do you have any example of retraining / tuning model with our own custom data (rather than your datasets which have somewhat complex structure)? The reason I am asking is that we have only numbers and some pre recorded audio files (30-40 samples per number). We can just label them as "1", "2", ... etc., try retraining / tuning and see what we get.

SameerMahajan
Feb 22, 2023

@sanchit-gandhi one problem with this tuned marathi model is that it is very large (6.17 GB as I see it during download). Our use case (https://youtu.be/L3L4mEszzTs) requires us to build an offline android app which typically cannot exceed a couple of hundred MBs...