|
--- |
|
license: cc-by-nc-sa-4.0 |
|
library_name: transformers |
|
pipeline_tag: image-text-to-text |
|
tags: |
|
- Eagle |
|
- VLM |
|
--- |
|
|
|
# Eagle Model Card |
|
|
|
## Model details |
|
|
|
**Model type:** |
|
Eagle is a family of Vision-Centric High-Resolution Multimodal LLMs. It presents a thorough exploration to strengthen multimodal LLM perception with a mixture of vision encoders and different input resolutions. The model contains a channel-concatenation-based "CLIP+X" fusion for vision experts with different architectures (ViT/ConvNets) and knowledge (detection/segmentation/OCR/SSL). The resulting family of Eagle models support up to over 1K input resolution and obtain strong results on multimodal LLM benchmarks, especially resolution-sensitive tasks such as optical character recognition and document understanding. |
|
|
|
|
|
 |
|
|
|
**Paper or resources for more information:** |
|
https://github.com/NVlabs/Eagle |
|
|
|
[arXiv](https://arxiv.org/pdf/2408.15998) / [Demo](https://huggingface.co/spaces/NVEagle/Eagle-X5-13B-Chat) / [Huggingface](https://huggingface.co/papers/2408.15998) |
|
|
|
``` |
|
@article{shi2024eagle, |
|
title = {Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders}, |
|
author={Min Shi and Fuxiao Liu and Shihao Wang and Shijia Liao and Subhashree Radhakrishnan and De-An Huang and Hongxu Yin and Karan Sapra and Yaser Yacoob and Humphrey Shi and Bryan Catanzaro and Andrew Tao and Jan Kautz and Zhiding Yu and Guilin Liu}, |
|
journal={arXiv:2408.15998}, |
|
year={2024} |
|
} |
|
``` |
|
|
|
## License |
|
- The code is released under the Apache 2.0 license as found in the [LICENSE](./LICENSE) file. |
|
- The pretrained weights are released under the [CC-BY-NC-SA-4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en). |
|
- The service is a research preview intended for non-commercial use only, and is subject to the following licenses and terms: |
|
- [Model License](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md) of LLaMA |
|
- [Terms of Use](https://openai.com/policies/terms-of-use) of the data generated by OpenAI |
|
- [Dataset Licenses](https://github.com/Efficient-Large-Model/VILA/blob/main/data_prepare/LICENSE) for each one used during training. |
|
|
|
**Where to send questions or comments about the model:** |
|
https://github.com/NVlabs/Eagle/issues |
|
|
|
## Model Architecture: |
|
|
|
**Architecture Type:** Transformer |
|
|
|
## Input: |
|
|
|
**Input Type:** Image, Text |
|
|
|
**Input Format:** Red, Green, Blue; String |
|
|
|
## Output: |
|
|
|
**Output Type:** Text |
|
|
|
**Output Format:** String |
|
|
|
## Inference: |
|
``` |
|
import os |
|
import torch |
|
import numpy as np |
|
from eagle import conversation as conversation_lib |
|
from eagle.constants import DEFAULT_IMAGE_TOKEN |
|
from eagle.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN, DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN |
|
from eagle.conversation import conv_templates, SeparatorStyle |
|
from eagle.model.builder import load_pretrained_model |
|
from eagle.utils import disable_torch_init |
|
from eagle.mm_utils import tokenizer_image_token, get_model_name_from_path, process_images, KeywordsStoppingCriteria |
|
from PIL import Image |
|
import argparse |
|
from transformers import TextIteratorStreamer |
|
from threading import Thread |
|
|
|
model_path = "NVEagle/Eagle-X5-13B-Chat" |
|
conv_mode = "vicuna_v1" |
|
image_path = "assets/georgia-tech.jpeg" |
|
input_prompt = "Describe this image." |
|
|
|
model_name = get_model_name_from_path(model_path) |
|
tokenizer, model, image_processor, context_len = load_pretrained_model(model_path,None,model_name,False,False) |
|
if model.config.mm_use_im_start_end: |
|
input_prompt = DEFAULT_IM_START_TOKEN + DEFAULT_IMAGE_TOKEN + DEFAULT_IM_END_TOKEN + '\n' + input_prompt |
|
else: |
|
input_prompt = DEFAULT_IMAGE_TOKEN + '\n' + input_prompt |
|
|
|
conv = conv_templates[conv_mode].copy() |
|
conv.append_message(conv.roles[0], input_prompt) |
|
conv.append_message(conv.roles[1], None) |
|
prompt = conv.get_prompt() |
|
|
|
image = Image.open(image_path).convert('RGB') |
|
image_tensor = process_images([image], image_processor, model.config)[0] |
|
input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt') |
|
|
|
input_ids = input_ids.to(device='cuda', non_blocking=True) |
|
image_tensor = image_tensor.to(dtype=torch.float16, device='cuda', non_blocking=True) |
|
|
|
with torch.inference_mode(): |
|
output_ids = model.generate( |
|
input_ids.unsqueeze(0), |
|
images=image_tensor.unsqueeze(0), |
|
image_sizes=[image.size], |
|
do_sample=True, |
|
temperature=0.2, |
|
top_p=0.5, |
|
num_beams=1, |
|
max_new_tokens=256, |
|
use_cache=True) |
|
|
|
outputs = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip() |
|
print(f"Image:{image_path} \nPrompt:{input_prompt} \nOutput:{outputs}") |
|
``` |
|
|
|
|
|
**[Preferred/Supported] Operating System(s):** <br> |
|
Linux |
|
|
|
## Intended use |
|
**Primary intended uses:** |
|
The primary use of Eagle is research on large multimodal models and chatbots. |
|
|
|
**Primary intended users:** |
|
The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence. |
|
|
|
## Ethical Considerations |
|
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. |
