Yes, but from a business point of view, NLP based GUIs have been the holy grail of marketing and customer support, especially in STEM apps market.
Case in point, Wolfram Alpha is not much more than an attempt to market Mathematica to lazy and failing college students. If that cost, and localization, can be offloaded to LLMs as the universal front end to technical software, it'd free up SWE resources to focus on core functionality.
If Magma, my favorite math/cryptography tool, had an LLM frontend, I could save time wasted onboarding new cryptographers.
So, we’ve come full circle to symbolic AI! This article essentially suggests that LLMs could be effective translators of our requests to command-line code or input to symbolic AI software, which would then yield precise solutions. However, I feel this approach is overly mechanical, and I don’t believe AGI would be achieved by creating thousands, if not millions, of MCP servers on our machines. This is especially true because MCP lacks scalability, and anyone who has had to send more than three or four function schemas to a language model knows that excessive JSON schema complexity confuses the model and reduces its performance.
pona-a · 1h ago
I'm reminded of what happened in the later years of Cyc. They found their logical framework didn't address certain common problems, so they kept adding specialized hard-coded solutions in Lisp. LLMs are headed for AI autumn.
tpurves · 3h ago
Wonderfully cheeky but also helpfully informative writeup. Also appreciate the hat-tip to all the (as yet) unsolved security issues. Clearly MCP is onto something important, although undoubtedly the standard (or some replacement standard) will mature a fair bit before we're done with it. The flip side to that is, MCPs are probably as 'easier' to experiment with now than they are ever going to be.
arunbahl · 4h ago
Awesome stuff! We use a similar approach (without MCP) to great effect with Prolog currently and feels like we're only just starting to scratch the surface here.
A great paper from Nasim Borazjanizadeh and Steven Piantadosi at UC Berkeley for those interested: Reliable Reasoning Beyond Natural Language https://arxiv.org/abs/2407.11373
For anyone digging in who wants to hack on this: arun [at] aloe.inc
amelius · 3h ago
How does the LLM know that it can use the factor tool to factor integers? Just by looking at the string "factor an integer"?
svat · 36m ago
Yes, and I believe this is what the article is referring to when it says “a stochastic black box that communicates through a complex web of JSON schemas attached to docstring annotations”. Specifically, in the function definition:
They give it a list of tool commands it can use in the context I believe.
ash-ali · 3h ago
I think this is the proper way to use llms for tasks that require high fidelity. currently im working on binary analysis using llms for natural language and letting ghidra/codeql do the symbolic work. scalability is a massive issue, perhaps the biggest besides fidelity.
its interesting to see many people come to the same neuro-symbolic conclusion around the same time.
jgalt212 · 1h ago
> But let's not let a potential rootkit get in the way of a fun weekend experiment.
Great quote.
Iwan-Zotow · 32m ago
Curious if this could be done for Mathematica. SymPy is kind of ...
rjeli · 2h ago
the implementations have a distinctly "I wrote this at a 3 AM hackathon" vibe
The LLM handles the natural language interaction and orchestration, while the computer algebra system does what it does best ... exact symbolic manipulation.
Case in point, Wolfram Alpha is not much more than an attempt to market Mathematica to lazy and failing college students. If that cost, and localization, can be offloaded to LLMs as the universal front end to technical software, it'd free up SWE resources to focus on core functionality.
If Magma, my favorite math/cryptography tool, had an LLM frontend, I could save time wasted onboarding new cryptographers.
https://magma.maths.usyd.edu.au/calc/
A great paper from Nasim Borazjanizadeh and Steven Piantadosi at UC Berkeley for those interested: Reliable Reasoning Beyond Natural Language https://arxiv.org/abs/2407.11373
For anyone digging in who wants to hack on this: arun [at] aloe.inc
The documentation and source code seem to be:
- (official SDK): https://github.com/modelcontextprotocol/python-sdk/blob/e80c... -> using the function's docstring: https://github.com/modelcontextprotocol/python-sdk/blob/e80c...
- (v2?): https://gofastmcp.com/servers/tools#the-%40tool-decorator and https://github.com/jlowin/fastmcp/blob/998de22a6e76fc8ae323e... -> using the function's docstring: https://github.com/jlowin/fastmcp/blob/998de22a6e76fc8ae323e...
this is what the tools response for the mcp server looks like:
{ tools: [ 0: { name: "factor" description: "Factor an integer" inputSchema: { ... } 4 items } ] }
its interesting to see many people come to the same neuro-symbolic conclusion around the same time.
Great quote.
The LLM handles the natural language interaction and orchestration, while the computer algebra system does what it does best ... exact symbolic manipulation.
this smells like claude :D