Update generation_config
#16
by
DeepStack
- opened
Thanks for making this model available.
The generation config may need to be updated to match the eos_token_id here:
https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/blob/main/generation_config.json
Hi @DeepStack ! I think you may be right, copying @alvarobartt to confirm with him on Monday.
Yes, you're right @DeepStack do you mind opening a PR with the fix or would you prefer us to go ahead and update it? Thanks in advance, also thanks @pcuenq for the ping!
For reference, it should be fairly similar to the changes made in https://huggingface.co/hugging-quants/Meta-Llama-3.1-405B-Instruct-GPTQ-INT4/discussions/9