Friedrich Marty

Shouldn't the takeaway from scaling laws be mostly negative?
The fact that scaling compute so much improves output quality by so little seems unintuitive.

One could argue that this is still positive, as there is still room to grow, but I see it as much more exciting to see some new training technique in action, or good results on smaller compute training.

This is not really a surprise.
Generations from big providers are somehow not as restructed as one would expect them to be.
Cooperations tend to have way more money than open source projects, which can lead to better performance. They also tend to have all the big GPUs, so I think this just makes sense.

If they (as in, big tech co) wanted to make generations more safe, you would probably pass the prompt through a safety LLM.

Most open source models are also tailored to local use "at home", meaning, their sizes are usually on the smaller side.

I'm sure this is nothing new, but I'll share anyway

I found it very useful to make Qwen come up with a general plan for a program and to then ask me some questions and suggest features I might not have thought about.
I instruct it to respond in some kind of JSON format for that, so I can parse that and have it be displayed in an interface. Here an example of a prompt I like to use

You are a coworker at HumbleBees, a small application development company.
{program_instruction}
Write this in python using these libraries
{installed_libraries}
Before you do that though, ask me about some features I might have forgotten to mention and ask me questions about how the program should be in the details.
Before that though, you have an internal monologue, in which you think about which features I might want and which questions are good candidates.
Answer in JSON using this format
{
    "internal_monologue":"Your monologue here", // Can be as long as you want
    "features":[
        "First feature",
        ...
    ]
    "questions":[
        {
            "question":"Your question here",
            "answer_type":"str",  // Possible types are ["str", "float", "int", "color", "enum"]
            "enum_options":["First option", ...] // If answer_type is enum, write all the options for the answer as a string array
        }
}