Model Description
LoRA adapter weights from fine-tuning BioMobileBERT on the MIMIC-III mortality prediction task. The PEFT was used and the model was trained for a maximum of 5 epochs with early stopping, full details can be found at the github repo.
- Model type: Language model LoRA adapter
- Language(s) (NLP): en
- License: apache-2.0
- Parent Model: BioMobileBERT
- Resources for more information:
How to Get Started with the Model
Use the code below to get started with the model.
Click to expand
from peft import AutoPeftModelForCausalLM, AutoPeftModelForSequenceClassification
from transformers import AutoTokenizer
model_name = "NTaylor/bio-mobilebert-mimic-mp-lora"
# load using AutoPeftModelForSequenceClassification
model = AutoPeftModelForSequenceClassification.from_pretrained(lora_id)
# use base llama tokenizer
tokenizer = AutoTokenizer.from_pretrained("nlpie/bio-mobilebert")
# example input
text = "Clinical note..."
inputs = tokenizer(text, return_tensors="pt")
outputs = reloaded_model(**inputs)
# extract prediction from outputs based on argmax of logits
pred = torch.argmax(outputs.logits, axis = -1)
print(f"Prediction is: {pred}") # binary classification: 1 for mortality
Out-of-Scope Use
This model and LoRA weights were trained on the MIMIC-III dataset and are not intended for use on other datasets, nor be used in any real clinical setting. The experiments were conducted as a means of exploring the potential of LoRA adapters for clinical NLP tasks, and the model should not be used for any other purpose.
Citation
BibTeX:
@misc{taylor2024efficiency,
title={Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks},
author={Niall Taylor and Upamanyu Ghose and Omid Rohanian and Mohammadmahdi Nouriborji and Andrey Kormilitzin and David Clifton and Alejo Nevado-Holgado},
year={2024},
eprint={2402.10597},
archivePrefix={arXiv},
primaryClass={cs.CL}
}