If you run these on your own hardware can you take the guard-rails off (ie "I'm afraid I can't assist with that"), or are they baked into the model?
hnuser123456 · 42m ago
You need to find an abliterated finetune, where someone sends prompts that would hit the guardrails, traces the activated neurons, finds the pathway that leads to refusal, and deletes it.
generalizations · 24m ago
I've been hearing that in this case, there might not be anything underneath- that somehow OpenAI managed to train on exclusively sterilized synthetic data or something.
gostsamo · 18m ago
I jailbroke the smaller model with a virtual reality game where it was ready to give me instructions on making drugs, so there is some data which is edgy enough.
mattpavelle · 5m ago
Yes but the abliterated versions (those with partially removed guardrails) are significantly “dumber” so the trade off isn’t worthwhile imho.
stainablesteel · 43m ago
they're baked in but there's a community of people who crack and modify them
even chat gpt will help you crack them if you ask it nicely
tyfon · 1h ago
I have a 5950x with 128 gb ram and a 12 gb 3060 gpu.
The speed of generating tokens is excellent, the killer is that when the context grows even a little processing of it is super slow.
Hopefully someone smart will optimize this, but as it is now I keep using other models like qwen, mistral and gemma.
MaxikCZ · 59m ago
I would so appreciate concrete data instead of subjectivities like "excellent" and "super slow".
How many tokens is excellent?
How many is super slow? How many is non-filled context?
qrios · 45m ago
Some numbers are posted in the comments:
> … you can expect the speed to half when going from 4k to 16k long prompt …
> … it did slow down somewhat (from 25T/s to 18T/s) for very long context …
Depends on the hardware configuration (size of VRAM, speed of CPU and system RAM) and llama.cpp parameter settings, a bigger context prompt slows the T/s number significantly but not order of magnitudes.
Facit: gpt-oss 120B on a small GPU is not the proper setup for chat use cases.
tyfon · 50m ago
I'm not really timing it as I just use these models via open webui, nvim and a few things I've made like a discord bot, everything going via ollama.
But for comparison, it is generating tokens about 1.5 times as fast as gemma 3 27B qat or mistral-small 2506 q4.
Prompt processing/context however seems to be happening at about 1/4 of those models.
A bit more concrete of the "excellent", I can't really notice any difference between the speed of oss-120b once the context is processed and claude opus-4 via api.
HPsquared · 55m ago
People can read at a rate around 10 token/sec. So faster than that is pretty good, but it depends how wordy the response is (including chain of thought) and whether you'll be reading it all verbatim or just skimming.
captainregex · 45m ago
What are you aiming to do with these models that isn’t chat/text manipulation?
GTP · 32m ago
LLM noob here. Would this optimization work with any MoE model or is it specific for this one?
magicalhippo · 27m ago
It's just doing a regex on the layer names, so should work with other models as long as they have the expert layers named similarly.
It worked with Qwen 3 for me, for example.
The option is just a shortcut, you can provide your own regex to move specific layers to specific devices.
nativeit · 30m ago
…and yet a much more capable model (my own brain) still runs better than this on pop tarts.
NitpickLawyer · 7m ago
Give hydrogen a few billion years, and it starts making fun of the inefficiencies in silicon-based siblings.
MaxikCZ · 17m ago
Your comment will get donvoted to invisibility anyways (or mayhaps even flagged), but I have to ask: what are you trying to accomplish with comments such this? Just shitting at it because it isnt as good as youd like yet? You want the best of tomorrow today, and will only be rambling about how its not good enough yesterday?
gjsman-1000 · 15m ago
Well, now I have to ask, what your purpose on calling him out, is. Does it deeply offend you that non-believers exist, who do not believe the technology will improve substantially in usefulness from here?
even chat gpt will help you crack them if you ask it nicely
How many tokens is excellent? How many is super slow? How many is non-filled context?
> … you can expect the speed to half when going from 4k to 16k long prompt …
> … it did slow down somewhat (from 25T/s to 18T/s) for very long context …
Depends on the hardware configuration (size of VRAM, speed of CPU and system RAM) and llama.cpp parameter settings, a bigger context prompt slows the T/s number significantly but not order of magnitudes.
Facit: gpt-oss 120B on a small GPU is not the proper setup for chat use cases.
But for comparison, it is generating tokens about 1.5 times as fast as gemma 3 27B qat or mistral-small 2506 q4. Prompt processing/context however seems to be happening at about 1/4 of those models.
A bit more concrete of the "excellent", I can't really notice any difference between the speed of oss-120b once the context is processed and claude opus-4 via api.
It worked with Qwen 3 for me, for example.
The option is just a shortcut, you can provide your own regex to move specific layers to specific devices.