metadata

license: apache-2.0

Quin Beta 1.1

Quin Beta 1.1 is an advanced Transformer-based language model with 2.7 billion parameters. It is an extension of Quin Beta 1.1, trained on an enhanced dataset that includes synthetic NLP texts and curated web content. The model demonstrates near state-of-the-art performance in benchmarks for common sense, language understanding, and logical reasoning within its parameter class. Quin Beta 1.1 is designed for QA, chat, and code formats and has a context length of 2048 tokens.

Key Features
Parameter Count: 2.7 billion
Training Data: 250 billion tokens from a combination of AOAI synthetic data and filtered web data
Training Tokens: 1.4 trillion tokens
Hardware Used: 96xA100-80G GPUs
Training Duration: 14 days
Frameworks: ONNX and JAX
Reinforcement Learning: Not applied (intended for enabling safety research)
Applications: QA, chat, and code formats
Context Length: 2048 tokens
Open-Source: Yes, aimed at enabling safety research like toxicity reduction and bias understanding

Performance

Quin Beta 1.1 has demonstrated near state-of-the-art performance on various benchmarks, specifically targeting:

Common sense reasoning
Language understanding
Logical reasoning

Usage

Installation

To use Quin Beta 1.1, you will need to have ONNX and JAX installed. Follow the installation instructions for each framework:

ONNX

pip install onnxruntime

JAX

pip install jax jaxlib

Loading the Model

Here is a basic example of how to load and use Quin Beta 1.1 using ONNX:

import onnxruntime as ort

# Load the ONNX model
ort_session = ort.InferenceSession("quin_beta_1.1.onnx")

# Prepare input
input_text = "Your input text here."
inputs = {
    "input_ids": [input_text],
}

# Run inference
outputs = ort_session.run(None, inputs)

# Get the result
result = outputs[0]
print(result)

Research and Safety

Quin Beta 1.1 is open-source and designed to enable research in safety domains such as:

Toxicity reduction
Bias understanding and mitigation

The model has not undergone reinforcement learning fine-tuning, providing a baseline for safety research efforts.

Contributing Contributions to Quin Beta 1.1 are welcome. If you would like to contribute, please follow these steps:

Fork the repository.
Create a new branch (git checkout -b feature-branch).
Make your changes.
Commit your changes (git commit -am 'Add new feature').
Push to the branch (git push origin feature-branch).
Create a new Pull Request.

License

Quin Beta 1.1 is released under the apache-2.0 License. See the LICENSE file for more details.

Contact

For any questions or issues, please contact drchamyoung@gmail.com