Appreciate the model drop!

#1
by Nitral-AI - opened

But why is it only 4k? Its 2024 man, those are rookie numbers.

Nitral-AI changed discussion title from Appreciate the mode drop! to Appreciate the model drop!
Stability AI org

We will release the long context version soon, please stay tuned!

Is there a way to run it with Pytorch 2.0.0?

Got Error:
[nn.LayerNorm(dim, eps=eps, bias=bias) for _ in range(self.num_heads)]
File "../huggingface/modules/transformers_modules/stabilityai/stablelm-2-12b-chat/e2a306c50ac1900c408cc33c058ae48a33820cf1/modeling_stablelm.py", line 256, in
[nn.LayerNorm(dim, eps=eps, bias=bias) for _ in range(self.num_heads)]
TypeError: init() got an unexpected keyword argument 'bias'

Line 256:
[nn.LayerNorm(dim, eps=eps, bias=bias) for _ in range(self.num_heads)]

See: https://pytorch.org/docs/2.0/generated/torch.nn.LayerNorm.html#torch.nn.LayerNorm
torch.nn.LayerNorm(normalized_shape, eps=1e-05, elementwise_affine=True, device=None, dtype=None)
Parameter bias is missing.

image.png

image.png

Sign up or log in to comment