| You must have the right Hub permissions to add configuration files to a model. | |
| thon | |
| from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig | |
| tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") | |
| model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-small") | |
| translation_generation_config = GenerationConfig( | |
| num_beams=4, | |
| early_stopping=True, | |
| decoder_start_token_id=0, | |
| eos_token_id=model.config.eos_token_id, | |
| pad_token=model.config.pad_token_id, | |
| ) | |
| Tip: add push_to_hub=True to push to the Hub | |
| translation_generation_config.save_pretrained("/tmp", "translation_generation_config.json") | |
| You could then use the named generation config file to parameterize generation | |
| generation_config = GenerationConfig.from_pretrained("/tmp", "translation_generation_config.json") | |
| inputs = tokenizer("translate English to French: Configuration files are easy to use!", return_tensors="pt") | |
| outputs = model.generate(**inputs, generation_config=generation_config) | |
| print(tokenizer.batch_decode(outputs, skip_special_tokens=True)) | |
| ['Les fichiers de configuration sont faciles à utiliser!'] | |
| Streaming | |
| The generate() supports streaming, through its streamer input. |