File size: 2,772 Bytes
eee4a69 19791d1 5af7312 19791d1 9a9773d 19791d1 23c0bfb 19791d1 9a9773d 19791d1 9a9773d 19791d1 9a9773d 19791d1 9a9773d 19791d1 6f28142 19791d1 eee4a69 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 |
---
license: mit
datasets:
- OleehyO/latex-formulas
metrics:
- bleu
pipeline_tag: image-to-text
tags:
- latex
- formula
---
# im2latex
This model is a base VisionEncoderDecoderModel fine-tuned on a dataset for generating LaTeX formulas from images.
## Model Details
- **Encoder**: Swin Transformer
- **Decoder**: GPT-2
- **Framework**: PyTorch
- **DDP (Distributed Data Parallel)**: Used for training
<img src="https://github.com/d-gurgurov/im2latex/blob/main/assets/im2latex.png?raw=true" alt="architecture" width="700"/>
## Training Data
The data is taken from [OleehyO/latex-formulas](https://huggingface.co/datasets/OleehyO/latex-formulas). The data was divided into 80:10:10 for train, val and test. The splits were made as follows:
```python
dataset = load_dataset(OleehyO/latex-formulas, cleaned_formulas)
train_val_split = dataset["train"].train_test_split(test_size=0.2, seed=42)
train_ds = train_val_split["train"]
val_test_split = train_val_split["test"].train_test_split(test_size=0.5, seed=42)
val_ds = val_test_split["train"]
test_ds = val_test_split["test"]
```
## Evaluation Metrics
The model was evaluated on a test set with the following results:
- **Test Loss**: 0.10
- **Test BLEU Score**: 0.67
## Usage
You can use the model directly with the `transformers` library:
```python
from transformers import VisionEncoderDecoderModel, AutoTokenizer, AutoFeatureExtractor
import torch
from PIL import Image
# load model, tokenizer, and feature extractor
model = VisionEncoderDecoderModel.from_pretrained("DGurgurov/im2latex")
tokenizer = AutoTokenizer.from_pretrained("DGurgurov/im2latex")
feature_extractor = AutoFeatureExtractor.from_pretrained("microsoft/swin-base-patch4-window7-224-in22k") # using the original feature extractor for now
# prepare an image
image = Image.open("path/to/your/image.png")
pixel_values = feature_extractor(images=image, return_tensors="pt").pixel_values
# generate LaTeX formula
generated_ids = model.generate(pixel_values)
generated_texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print("Generated LaTeX formula:", generated_texts[0])
```
## Training Script
The training script for this model can be found in the following repository: [GitHub](https://github.com/d-gurgurov/im2latex)
**Citation:**
- If you use this work in your research, please cite our paper:
```bibtex
@misc{gurgurov2024imagetolatexconvertermathematicalformulas,
title={Image-to-LaTeX Converter for Mathematical Formulas and Text},
author={Daniil Gurgurov and Aleksey Morshnev},
year={2024},
eprint={2408.04015},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2408.04015},
}
```
License
[MIT] |