intfloat

/

e5-mistral-7b-instruct

like 469

Please refer to the documentation at https://huggingface.co/docs/transformers/main/en/model_doc/mistral#combining-mistral-and-flash-attention-2 , simply add attn_implementation="flash_attention_2" to AutoModel.from_pretrained(...) will do the trick. Note that it requires A100 or newer GPUs.

FlashAttn is now added via torch.sdpa if you use torch>=2.2 and transformers>=4.37.1

Best Michael Feil