How does this sort of thing perform with 70b models?
hnuser123456 · 4h ago
273 GB/s / 70GB = 3.9 tokens/sec
3eb7988a1663 · 7h ago
If you are doing nothing but consuming models via llama.cpp, is the AMD chip an obstacle? Or is that more a problem for research/training where every CUDA feature needs to be present?
acheong08 · 5h ago
Llama.cpp works well on AMD, even for really outdated GPUs. Ollama refuses to work with my RX 570 from 2019 but llama.cpp supports it via Vulkan.
washadjeffmad · 27m ago
Don't you dare say anything unpositive about Ollama this close to whatever it is they're planning to distinguish themselves from llama.cpp.
They've been out hustling, handshaking, dealmaking, and big businessing their butts off, whether or not they clearly indicate the shoulders of the titans like Georgi Gerganov they're wrapping, and you are NO ONE to stand in their way.
Do NOT blow this for them. Understand? They've scooted under the radar successfully this far, and they will absolutely lose their shit if one more peon shrugs at how little they contribute upstream for what they've taken that could have gone to supporting their originator.
Ollama supports its own implementation of ggml, btw. gglm is a mysterious format that no one knows the origins of, which is all the more reason to support Ollama, imo.
Havoc · 3h ago
>Ollama refuses to work with my RX 570 from 2019 but llama.cpp supports it via Vulkan.
That's a bit odd given ollama utilizing llama to do the inference...
I would be very surprised. Typically LPDDR is soldered, as it takes too much power to run the traditional sockets, as well as being much slower.
Though there has been a modular option called LPCAMM[1]. However AFAIK it doesn't support the speed the specs of this box states.
Recently a newer connector, SOCAMM has been launched[2], which does support the high memory speeds, but it's just on the market and going into servers first AFAIK.
SOCAMM is also Nvidia-specific, not a wider standard. (At least, not yet.)
hnuser123456 · 4h ago
No, it's soldered, it would have to run around 6000 MT/s instead of 8533 if it was slotted DIMMs.
olddustytrail · 4h ago
That's an odd coincidence. I'd decided to get a new machine but I suspected we'd start seeing new releases with tons of GPU accessible RAM as people want to experiment with LLMs.
So I just got a cheap (~350 USD) mini PC to keep me going until the better stuff came out. Which was a 24GB, 6c/12t CPU from a company I'd not heard of called Bosgame (dunno why the article keeps calling them Bosman unless they have a different name in other countries. It's definitely https://www.bosgamepc.com/products/bosgame-m5-ai-mini-deskto... )
So my good machine might end up from the same place as my cheap one!
They've been out hustling, handshaking, dealmaking, and big businessing their butts off, whether or not they clearly indicate the shoulders of the titans like Georgi Gerganov they're wrapping, and you are NO ONE to stand in their way.
Do NOT blow this for them. Understand? They've scooted under the radar successfully this far, and they will absolutely lose their shit if one more peon shrugs at how little they contribute upstream for what they've taken that could have gone to supporting their originator.
Ollama supports its own implementation of ggml, btw. gglm is a mysterious format that no one knows the origins of, which is all the more reason to support Ollama, imo.
That's a bit odd given ollama utilizing llama to do the inference...
Though there has been a modular option called LPCAMM[1]. However AFAIK it doesn't support the speed the specs of this box states.
Recently a newer connector, SOCAMM has been launched[2], which does support the high memory speeds, but it's just on the market and going into servers first AFAIK.
[1]: https://www.anandtech.com/show/21069/modular-lpddr-becomes-a...
[2]: https://www.tomshardware.com/pc-components/ram/micron-and-sk...
So I just got a cheap (~350 USD) mini PC to keep me going until the better stuff came out. Which was a 24GB, 6c/12t CPU from a company I'd not heard of called Bosgame (dunno why the article keeps calling them Bosman unless they have a different name in other countries. It's definitely https://www.bosgamepc.com/products/bosgame-m5-ai-mini-deskto... )
So my good machine might end up from the same place as my cheap one!