it run without flash_attn
colab t4
git clone https://github.com/state-spaces/mamba.git
cd /content/mamba
/content/mamba/benchmarks/benchmark_generation_mamba_simple.py
in colabt4
!python /content/mamba/benchmarks/benchmark_generation_mamba_simple.py --model-name "nvidia/Hymba-1.5B-Base" --prompt "Who is Napoleon Bonaparte?" --minp 0.05 --topk 0 --temperature 0.7 --repetition-penalty 1.2
[' Who is Napoleon Bonaparte?\nNapoleon Bonaparte was a French military and political leader who rose to prominence during the Revolutionary Wars. He seized power in 1799 and became Emperor of France four years later, creating a lasting impact on history. His leadership marked the end of the French Revolution and ushered in an era known as the Napoleonic Era, characterized by significant changes in government structures worldwide and profound social reforms within France itself. The legacy of his']
!python /content/mamba/benchmarks/benchmark_generation_mamba_simple.py --model-name "nvidia/Hymba-1.5B-Base" --prompt "Who is Napoleon Bonaparte?" --minp 0.05 --topk 0 --temperature 0.7 --repetition-penalty 1.2
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/selective_scan_interface.py:164: FutureWarning: torch.cuda.amp.custom_fwd(args...)
is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda')
instead.
def forward(ctx, xz, conv1d_weight, conv1d_bias, x_proj_weight, delta_proj_weight,
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/selective_scan_interface.py:240: FutureWarning: torch.cuda.amp.custom_bwd(args...)
is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda')
instead.
def backward(ctx, dout):
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/triton/layer_norm.py:986: FutureWarning: torch.cuda.amp.custom_fwd(args...)
is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda')
instead.
def forward(
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/triton/layer_norm.py:1045: FutureWarning: torch.cuda.amp.custom_bwd(args...)
is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda')
instead.
def backward(ctx, dout, *args):
/usr/local/lib/python3.10/dist-packages/mamba_ssm/distributed/tensor_parallel.py:26: FutureWarning: torch.cuda.amp.custom_fwd(args...)
is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda')
instead.
def forward(ctx, x, weight, bias, process_group=None, sequence_parallel=True):
/usr/local/lib/python3.10/dist-packages/mamba_ssm/distributed/tensor_parallel.py:62: FutureWarning: torch.cuda.amp.custom_bwd(args...)
is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda')
instead.
def backward(ctx, grad_output):
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/triton/ssd_combined.py:758: FutureWarning: torch.cuda.amp.custom_fwd(args...)
is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda')
instead.
def forward(ctx, zxbcdt, conv1d_weight, conv1d_bias, dt_bias, A, D, chunk_size, initial_states=None, seq_idx=None, dt_limit=(0.0, float("inf")), return_final_states=False, activation="silu",
/usr/local/lib/python3.10/dist-packages/mamba_ssm/ops/triton/ssd_combined.py:836: FutureWarning: torch.cuda.amp.custom_bwd(args...)
is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda')
instead.
def backward(ctx, dout, *args):
Loading model nvidia/Hymba-1.5B-Base
The repository for nvidia/Hymba-1.5B-Base contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/nvidia/Hymba-1.5B-Base.
You can avoid this prompt in future by passing the argument trust_remote_code=True
.
Do you wish to run the custom code? [y/N] y
The repository for nvidia/Hymba-1.5B-Base contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/nvidia/Hymba-1.5B-Base.
You can avoid this prompt in future by passing the argument trust_remote_code=True
.
Do you wish to run the custom code? [y/N] y
2024-11-29 22:19:15.885095: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-11-29 22:19:15.902174: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-11-29 22:19:15.907074: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-11-29 22:19:17.584907: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Number of parameters: 1522797824
[' Who is Napoleon Bonaparte?\nNapoleon Bonaparte was a French military and political leader who rose to prominence during the Revolutionary Wars. He seized power in 1799 and became Emperor of France four years later, creating a lasting impact on history. His leadership marked the end of the French Revolution and ushered in an era known as the Napoleonic Era, characterized by significant changes in government structures worldwide and profound social reforms within France itself. The legacy of his']
Prompt length: 8, generation length: 100
nvidia/Hymba-1.5B-Base prompt processing + decoding time: 26364ms
!pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu121
!pip install mamba-ssm causal-conv1d
!huggingface-cli login
--genlen 1024 is Number of tokens
!python /content/mamba/benchmarks/benchmark_generation_mamba_simple.py --model-name "nvidia/Hymba-1.5B-Base" --prompt "Who is Napoleon Bonaparte?" --minp 0.05 --topk 0 --temperature 0.7 --repetition-penalty 1.2 --genlen 1024
umber of parameters: 1522797824
[' Who is Napoleon Bonaparte?\nNapoleon Bonaparte was a French military and political leader who rose to prominence during the Revolutionary Wars. He seized power in 1799 and became Emperor of France four years later, creating a lasting impact on history. His leadership marked the end of the French Revolution and ushered in an era known as the Napoleonic Era, characterized by significant changes in government structures worldwide and profound social reforms within France itself. The legacy of his influence can still be felt today through various aspects such as modern bureaucracy systems or legal frameworks derived from civil law tradition established under Napoleonic Code – making him one of those historical figures whose impacts are both far-reaching yet deeply rooted into our present times! In this article I will provide you with all necessary information regarding Famous Leader Biography napoleon bonaparte so that it becomes easier for anyone reading about them without any prior knowledge whatsoever! So let’s dive right into understanding more about how did he shape Europe forever through these ten facts below…\nWho Was Napoléon Bonaparte?\nNapoléon Bonaparte (15 August 1769 – 5 May 1821) was a French military and political leader who rose to prominence during the French Revolution and led several successful campaigns throughout Europe until being defeated at Waterloo in 1815. As emperor of France between 1804–1814/15, Napoléon implemented numerous legal and administrative reforms which remain influential today including modernizing education systems across Europe while also establishing the basis for much current civil laws used around the world even nowadays like code napolyoniole etc..\nHis early life saw him rise up quickly after serving as an artillery officer (by age 23!) before eventually becoming head of state following coup d’état against Louis XVI & Marie Antoinette back home again whereupon Napoleon led efforts towards expanding French territory abroad via conquest such as Egypt but ultimately failed due largely partisan politics within Europe itself along with other factors beyond their own control too though they managed manage some successes nonetheless despite setbacks suffered over time leading many historians believe if not exactly victorious then certainly never truly lost either…\nBorn On: August 15th , 1769 Place Of Birth : Ajaccio Island Corsica Region France\nDied At Age: 51 Years Old Date Of Death : May 5th , 1821 Location Of Burial : Chateau De Malmaison Near Paris City France\nNationality : French Nationals\nProfession : Military Officer Politician Diplomat Lawyer Writer Author Philosopher Scientist Historian Art Critic Music Composer Playwright Poet Novelist Essayist Journalist Publisher Editor Producer Director Performer Singer Songwriter Actress Model Presenter Entertainer Comedian Dancer Athlete Sportsperson Boxer Wrestler Fighter Uniform Player Football Soccer Baseball Hockey Basketball Tennis Volleyball Swimmer Surfer Scuba Diver Mountaineer Sea Explorer Astronomical Observer Space Traveler Science Communicator Teacher Student Scholar Thinker Intellectual Philanthropist Humanitarian Activist Advocate Civil Rights Movement Suffragette Peacekeeper Diplomat International Relations Navigator Cartographer Map Creator Geologist Paleontologist Archaeologist Anthropologist Ethnologist Psychiatrist Neurologist Cardiologist Ophthalmologist Otolaryngologist Plastic surgeon Orthopedist Endocrinologist Gastroenterologist Pediatrician Gynecologist Urologist Proctologist Colorectal specialist General practitioner Family physician Internist Hematologist Pulmonologist Radiologists Sonogram technicians X-ray technician Medical Laboratory Technologist Pharmacy technician Nurse Practitioners Counselor Social Worker Case Manager Health Information specialistetcetera etceterasetceteraseceterahteachers leaders educators mentors advisors coaches guides inspirational speakers motivators trendsetters pioneers innovators disruptors entrepreneurs startups ventures business strategies plans marketing communications advertising public relations sales promotion branding identity corporate culture strategy management finance accountancy auditing taxation economics politics government policy administration legislation regulation compliance monitoring surveillance oversight transparency openness accessibility affordability inclusivity diversity equality opportunity tolerance respect freedom liberty justice dignity integrity courage strength resilience perseverance determination ambition creativity imagination passion curiosity exploration discovery experimentation research development creation invention innovation advancement progress evolution transformation growth maturity wisdom experience learning teaching collaboration cooperation teamwork communication networking inspiration encouragement support']