Hugging Face's logo

phanerozoic
/
Tiny-Knight-1.1b-v0.1

Text Generation
Transformers
Safetensors
English
llama
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
Tiny-Knight-1.1b-v0.1 / README.md
phanerozoic's picture
phanerozoic
Update README.md
89073ae verified
preview code
|
raw
history blame
3.3 kB
metadata
license: cc-by-nc-4.0
language:
  - en
widget:
  - text: |
      What kind of food do you enjoy eating?
    example_title: The Code of Chivalry

tinyknight.png

Tiny Knight-1.1b-v0.1

Tiny Knight-1.1b-v0.1 is a specialized language model crafted for generating knight and medieval-themed content. This iteration is built upon the foundations of TinyLlama-1.1B-Chat-v1.0, tailored to operate within environments constrained by computing resources.

Performance

This model excels in weaving narratives filled with chivalry, valor, and the intricate details of medieval life. Its specialization, however, limits its effectiveness in broader language tasks, especially those requiring detailed knowledge outside the medieval theme.

Direct Use

Tiny Knight-1.1b-v0.1 is particularly suited for generating content within medieval, knightly, or fantasy settings, ideal for storytelling, educational content, and thematic exploration. It is not recommended for general-purpose tasks or technical domains.

Context Setting and Interaction Guidelines

Given its specialized nature, Tiny Knight-1.1b-v0.1 benefits significantly from detailed context-setting. Providing a rich thematic backdrop in prompts enhances the model's performance, guiding it to generate more accurate and immersive content.

Initial Context Setting:

  • text: | Assistant: Hail and well met! I'm here to recount tales of valor and chivalry. What stories from the age of knights and castles would you like to explore? example_title: "Initiating Medieval Themed Conversation"

  • text: | Assistant: Greetings, traveler! The annals of history are open to us. Ask me anything about the realm of knights, the intricacies of medieval life, or the lore of legendary artifacts! example_title: "Engaging in Medieval Themed Dialogue"

Training Data

Incorporates a dataset focused on knightly tales, medieval history, and literature, derived from the foundational TinyLlama-1.1B model.

Custom Stopping Strings

Custom stopping strings were used to refine output quality:

  • "},"
  • "User:"
  • "You:"
  • "\nUser"
  • "\nUser:"
  • "me:"
  • "user"
  • "\n"

Training Hyperparameters and Fine-Tuning Details

  • Base Model Name: TinyLlama-1.1B-Chat-v1.0
  • Base Model Class: LlamaForCausalLM
  • Projections: gate, down, up, q, k, v, o
  • LoRA Rank: 16
  • LoRA Alpha: 32
  • True Batch Size: 32
  • Gradient Accumulation Steps: 1
  • Epochs: 0.18
  • Learning Rate: 3e-4
  • LR Scheduler: Linear
  • Step: 75
  • Loss: 1.87

Limitations

While adept at producing themed content, Tiny Knight-1.1b-v0.1's applicability is limited outside its specialized domain of knights and medieval themes.

Summary

Tiny Knight-1.1b-v0.1 represents a significant advancement in thematic language models, offering a specialized tool for exploring the medieval era. Its emphasis on context for optimal performance and the use of custom stopping strings make it a sophisticated asset for generating historically rich content.

Acknowledgments

Special thanks to the TinyLlama-1.1B team, whose pioneering work laid the groundwork for the creation of Tiny Knight-1.1b-v0.1.