Devstral

Any agentic dev software you could recommend that runs well with local models?

I’ve been using Cursor and I’m kind of disappointed. I get better results just going back and forth between the editor and ChatGPT

I tried localforge and aider, but they are kinda slow with local models

Yes, I agree. I've just ran the model locally and it's making a good impression. I've tested it with some ruby/rspec gotchas, which it handled nicely.

I'll give it a try with aider to test the large context as well.

They are referring to SWE bench lite. Just want to make sure you are too.

extraordinary.. or suspicious that the benchmarks aren't doing their job

Just to confirm, are you referring to Claude 3.7?

> For people without a 24GB RAM video card, I've got an 8GB RAM one running

What're you using for this? llama.cpp? Have a 12GB card (rtx 4070) i'd like to try it on.

M4 max 128gb ram.

LM studio MLX with full 128k context.

It works well but has a long 1 minute initial prompt processing time.

I wouldn’t buy a laptop for this, I would wait for the new AMD 32gb gpu coming out.

If you want a laptop I even consider my m4 max too slow to use more than just here or there.

It melts if you run this and battery goes down asap. Have to use it docked for full speed really

All Hands AI has instructions for running Devstral locally on a MacBook using LMStudio: https://docs.all-hands.dev/modules/usage/llms/local-llms#ser...

The same page also gives instructions for running the model through VLLM on a GPU, but it doesn't seem like it supports quantization, so it may require multiple GPUs since the instructions say "with at least 2 GPUs".

AMD is going to be off the beaten path, you're likely to have more success/less boring plumbing trouble with nVidia.

I think your options are generally:

0) A desktop PC with one or more graphics cards, or 1) A Mac with Apple Silicon

I've worked with other models from All Hands recently, and I believe they were based on Mistral.

My general impression so far is that they aren't quite up to Claude 3.7 Sonnet, but they're quite good. More than adequate for an "AI pair coding assistant", and suitable for larger architectural work as long as you break things into steps for it.

I don't believe them either. We really have to test these ourselves imo.

Qwen3 is a step backwards for me for example. And GLM4 is my current goto despite everyone saying it's "only good at html"

The 70b cogito model is also really good for me but doesn't get any attention.

I think it depends on our projects / languages we're using.

Still looking forward to trying this one though :)

This is basically the Mistral niche. If you are doing something generally perceived as ethical, you would use Gemma 3 IMO. When you aren't... well there are Apache licensed LLMs for you.

>Maybe the EU should cover the cost of creating this model

Wouldn't mind some of my taxpayer money flowing towards apache/mit licensed models.

Even if just to maintain a baseline alternative & keep everyone honest. Seems important that we don't have some large megacorps run away with this.

Pretty sure the EU paid for some supercomputers that AI startups can use and Mistral is partner in that program.

you can use constrained outptus for enforcing tool schemas any model can get it with a lil help

The best general model you can run locally is probably some version of Gemma 3 or the latest Mistral Small. On a Windows machine, this is limited by VRAM, since system RAM is too low-bandwidth to run models at usable speeds. On an M-series Mac, the system memory is on-die and fast enough to use. What you can run will be the total RAM, minus whatever MacOS uses and the space you want for other programs.

To determine how much space a model needs, you look at the size of the quantized (lower precision) model on HuggingFace or wherever it's hosted. Q4_K_M is a good default. As a rough rule of thumb, this will be a little over half the size of the parameters, if they were in gigabytes. For Devstral, that's 14.3GB. You will also need 1-8GB more than that, to store the context.

For example: A 32GB Macbook Air could use Devstral at 14.3+4GB, leaving ~14GB for the system and applications. A 16GB Macbook Air could use Gemma 3 12B at 7.3+2GB, leaving ~7GB for everything else. An 8GB Macbook could use Gemma 3 4B at 2.5GB+1GB, but this is probably not worth doing.

I use qwen3:30b-a3b-q4_K_M for coding support and spam filtering, qwen2.5vl:32b-q4_K_M for image recognition/tagging/describing and sometimes gemma3:27b-it-qat for writing. All through Ollama, as that provides a unified interface, and then accessed from Emacs, command-line llm tool or my Clojure programs.

There is no single "best" model yet, it seems.

That's on an M4 Max with 64GB of RAM. I wish I had gotten the 128GB model, though — given that I run large docker containers that consume ~24GB of my RAM, things can get tight.

I would recommend just trying it out! (as long as you have the disk space for a few models). llama.cpp[0] is pretty easy to download and build and has good support for M-series Macbook Airs. I usually just use LMStudio[1] though - it's got a nice and easy-to-use interface that looks like the ChatGPT or Claude webpage, and you can search for and download models from within the program. LMStudio would be the easiest way to get started and probably all you need. I use it a lot on my M2 Macbook Air and it's really handy.

[0] - https://github.com/ggml-org/llama.cpp

[1] - https://lmstudio.ai/

Eh, different usecase entirely. I don't really compare these.

from what I gather it's finetuned to use OpenHand specifically so shows value on thsoe benchmark that target a whole system as a blackbox (i.e. agent + llm) more than directly target the llm input/outputs

"Can run" is pretty easy, it's pretty small and quantized. It runs at 3.7 tokens/second on pure CPU with AMD 8945HS.

I can run it without issue on a 6800 XT with 64GB of RAM.

I don't think they're actually the same thing, I definitely feel like Claude is much better with code than ChatGPT is so there are clearly differences in the capabilities of these models. One analogy that I find helpful here is the idea that these AIs are like animals. Just like there are animals of the same family (meaning they're genetically related to some degree) they still adapt to different niches. I see all these AI companies ultimately creating models analogous to that.

Some AIs will be good at coding (perhaps in a particular language or ecosystem), some at analyzing information and churning out a report for you, and some will be better at operating in physical spaces.

Deepseek and Mistral are both more open source than Lllama

I think llama is less open source than this mistral release

> a single 4090 costs $3k

What a ripoff, considering that a 5090 with 32GB of VRAM also currently costs $3k ;)

(Source: I just received the one I ordered from Newegg a week ago for $2919. I used hotstocks.io to alert me that it was available, but I wasn’t super fast at clicking and still managed to get it. Things have cooled down a lot from the craziness of early February.)

If it runs on 4090, it also runs on 3090 which are available used for 600 EUR.

> a single 4090 costs $3k

I hope not. Mine was $1700 almost 2 years go, and the 5090 is out now...

Is there an equivalence between gpu vram and mac ram?