A giant difference between eval in most languages versus lisp, is that eval in lisp doesn't just take in a string. Sounds somewhat subtle, but it can lead to some dramatic differences. And is, in at least a small part, why lisp people claim code is data. (I had a fun time exploring this a long time ago in https://taeric.github.io/CodeAsData.html)
RodgerTheGreat · 11h ago
Many practical applications of eval() boil down to accessing (and perhaps manipulating) scopes in a manner which the host language does not normally allow. Likewise, many potential dangers or pitfalls of eval() relate to how it allows (or does not allow) dynamically-assembled code to inherit access to some spatial or temporal scope of variables, definitions, "unsafe" libraries, and so forth. The interaction of eval() with scope is much a much more interesting design question than whether it accepts an AST, a string, or something in-between, except to the extent that non-stringy "programs" represent some form of reified closure.
taeric · 11h ago
I mean, you aren't wrong. But, little bobby tables is essentially what happens when you build up eval statements using a string, no?
RodgerTheGreat · 11h ago
if and only if a piece of eval'd code has the ability to produce harmful (let alone observable) side effects, which in a functional language will in turn largely be a consequence of what is imported into the code's scope.
taeric · 10h ago
Strictly, this isn't true, either? If you can influence what is returned from the evaluation, that may be enough to cause bugs, no? Without needing any access to scope or side effects. Consider a case where you use sql injection style modification to change an authz query so that it returns indicating the current user has admin rights. (Notably without making any external change in doing this.)
Granted, I fully cede that your point is largely right. Just feels like you are a bit too strong with the "if and only if" on it.
kevin_thibedeau · 12h ago
Lisp was supposed to ingest M-expressions and convert them internally. Eval would have been more like other languages had that plan not been abandoned. It is not meaningfully different for languages that construct an AST from a string before evaluation.
taeric · 11h ago
I'm not sure I follow? Presuming that the M-expressions were also made of atoms, they would still be meaningfully different than taking in the string of text. Closer to the same, agreed, but still not the same.
spauldo · 9h ago
IIRC M-expressions broke homoiconicity, so you couldn't just spit sexps at eval and expect anything sensible. I believe that's a lot of the reason they never got implemented.
beanjuiceII · 10h ago
is (1 2 3) code or data?
kazinator · 3h ago
It depends on the dialect. If we consider Common Lisp, it's a piece of data that doesn't constitute a valid form; it is calling 1 as a function. Some Lisp dialects off the mainstream path handle it. One that comes to mind is PicoLisp which admits (1 2 3) as an expression producing (1 2 3) (that object itself, essentially "auto-quoted", I think).
taeric · 10h ago
I don't understand the question? In particular, it feels out of place? The common claim is that code is data. This does not necessarily imply that data is code.
That said, amusingly, `(1 2 3)` can easily be considered code if you are able to evaluate permutation notation. No?
zem · 10h ago
the whole point of the "code is data" mantra is that it is both - the list `(1 2 3)` and the lisp code that evaluates to said list when parsed and interpreted.
beanjuiceII · 10h ago
wouldn't the data version be '(1 2 3) ? sorry I am just trying to understand
Xmd5a · 6h ago
Usually yes. To distinguish between lists and function calls. However if we are to pass code literals to eval, we need to quote them:
=>(eval '(1 2 3))
Cannot call 1 as a function.
=>(eval ''(1 2 3))
(1 2 3)
spauldo · 9h ago
You're basically asking if literals are code. I imagine you'll get varied opinions about that.
(And yes, that can be used unquoted in a few different contexts in Lisp, such as a special form or macro, or if you've managed to convince the reader that 1 is a function.)
neilv · 12h ago
I'm glad Matthew Flatt wrote this. There is a problem of newbies who read some academic textbook ("metacircular evaluator!") and then immediately try to use the eval feature when it doesn't make sense.
(Also, there's a more general problem of newbies trying to use the most advanced tool they've been exposed to. Why use the standard `if` statement, when you can use a proprietary high-powered pattern-matching form, now backed by a networked Kubernetes cluster of deep learning and LLMs.)
In the Racket community, one of my many attempts to discourage mistaken uses of `eval`:
In the all-in-one practitioner's book I was writing, eval wouldn't be introduced until almost the end, in the "Dangerous Last Resort" part of the book. (Maybe I should've planned a marketing gimmick around it, as "free bonus DLC", and you need to do some ritual to be a Certified Certifiable Racketeer, before you can read the eval secret scroll.)
The sandboxing of eval is something that can be left to a layer on top. You certainly don't want eval itself to be gutted of functionality; nothing less than he full language, thank you.
eval can be sandboxed with a combination of multiple approaches, including validating the code, and evaluating it in a restricted environment in which only certain symbols exist. The full library package is not visible.
Is my understanding correct that Lisp's powerful macro system stems from the ability to write the eval function in Lisp itself? From what I gather, Lisp starts with a small set of primitives and special forms—seven in the original Lisp, including lambda. I recall Paul Graham demonstrating in one of his essays that you can build an eval function using just these primitives. Those primitives are typically implemented in a host language like C, but once you have an eval function in Lisp, you can extend it with new rules. The underlying C interpreter only sees the primitives, but as a programmer, you can introduce new syntax rules via eval. This seems like a way to understand macros, where you effectively add new language rules. I know Lisp macros are typically defined using specific keywords like defmacro, but is the core idea similar—extending the language by building on the eval function with new rules?
kazinator · 3h ago
The relationship is this: the ease of writing a meta-circular eval in Lisp, and the ease of writing macros are related by common causes.
samth · 17h ago
No, macros and eval are quite different. You can see this for example in Python or JavaScript, which have eval but not macros.
That's a different meaning of first-class from Strachey's definition of a first-class citizen[1] - ie, one that can be passed as an argument, returned from a function, or assigned to a variable.
Syntactic macros are still second-class, like Lisp macros, but an improvement over text-replacement style macros.
For something macro-like which is first-class, there are fexprs[2] and operatives (from Kernel[3]) - these receive their operands verbatim, like macros, so they don't require quotation if we want to suppress evaluation. fexprs/Operatives can be passed around like any other value at runtime.
Stratchey defined "first-class objects". This was by analogy with "first-class citizens" in a legal/political sense, since they are treated just as well as any other object and have no additional limitations. If we extend the analogy to syntax then I think it's clear enough that it means that it is a piece of syntax which is treated the same as any other and does not require special treatment or impose additional restrictions.
Thank you for the clarification and the additional information, I think having macros as first-class objects is a cool (but separate) idea.
matheusmoreira · 9h ago
They aren't that different. Fexprs are essentially additional eval cases.
sparkie · 15h ago
> Is my understanding correct that Lisp's powerful macro system stems from the ability to write the eval function in Lisp itself?
I wouldn't say this is the case. Nearly any language could implement an `eval` for itself, but obviously, it is much simpler in Lisps because there's little syntax and few rules.
What makes Lisp macros different from say, C preprocessor macros, is the body of the macro is just Lisp code - so the "preprocessor" in this case has full access to the host languages facilities, which may include `eval`. The macros don't take textual input, but they take structured input in the form of S-expressions.
Implementing macros is obviously simpler due to eval, because we need to run the evaluator on the macro body, but it's not a strict requirement, as macro functionality could be provided by the implementation and could encapsulate its own evaluator.
Lisp macros are also simple due to the fact that Lisp code is just lists of data - you don't have to navigate a complex AST to walk through the code and emit particular syntax. You walk through the input with `car` and `cdr`, and you emit new syntax with `cons` (or `list`/`list*` which are derived from it). Macros can take code as their argument, and produce new code which is evaluated in place.
Macros still have hygiene issues though, because they're based on expanding code before evaluating it, variables used in macros can accidentally shadow variables in the scope of the macro's caller. There are workarounds (gensym) to navigate these hygiene problems.
> From what I gather, Lisp starts with a small set of primitives and special forms—seven in the original Lisp, including lambda. I recall Paul Graham demonstrating in one of his essays that you can build an eval function using just these primitives.
This is largely a theoretical demonstration but not real-world usage. In practice, Lisps have dozens or hundreds of "primitives". Common Lisp in particular is a big language and not trivial to implement. Scheme is a bit smaller, though r6rs started to also grow quite large, but this approach was revisited in r7rs (current), which aims for a small core, with additional functionality being provided through SRFIs (Scheme requests for implementation).
> Those primitives are typically implemented in a host language like C, but once you have an eval function in Lisp, you can extend it with new rules.
Using Scheme as an example, some SRFIs can be implemented purely in Scheme, as libraries, but others require the implementation to provide support, which often requires writing C code to provide them.
> This seems like a way to understand macros, where you effectively add new language rules. I know Lisp macros are typically defined using specific keywords like defmacro
As you note, it's `defmacro`, or `macro`, or `syntax-rules`, `syntax-case`, etc, which introduce new syntax - not eval in particular. Some macros will use `eval` in their bodies, which permits control of evaluation other than the regular applicative form of lambdas.
Macros are more than just `eval`. They're a multi-stage evaluation model where we first need to do some `macroexpand` (which will internally use `eval`), and afterwards the the resulting expression from the macro call is evaluated.
> but is the core idea similar—extending the language by building on the eval function with new rules?
There are some Lisps which still attempt this kind of minimalism.
One example is Ian Piumarta's Maru[1], which support extending `eval` (and `apply`) with new functionality at runtime based on the type being evaluated. Maru basically has global maps of type->evaluator and type->applicator, where we can add new pairs to at runtime and augment the behavior of `eval`/`apply`.
Kernel[2] also aims for the minimalist approach and does away with macros, quote and special-forms entirely, instead replacing them with a more general feature called an operative. The Kernel evaluator does not need to implement special rules for things like `lambda`, `cond`, `car`, `cdr` (as in Graham's "On Lips" essay) - but it just discriminates two forms - operative or applicative. Obviously, some kinds of operative are "primitive", but there's no difference from the PoV of the programmer. Which set of symbols you decide to implement as primitive is up to the implementation. The Kernel report suggests a small set of primitives and demonstrates the remaining standard features can be implemented using only the functionality provided so far.
In summary: The more layers of interpretation your `eval` function needs to do (assumption of source language, tokenization/parsing rules, symbol definitions, etc), the less well-defined the semantics are.
So on the less-well-defined end, you have something like JavaScript's `eval` function, in the middle you have lisp macros, which get an AST instead of just source code, and on the actually-pretty-well-defined end, you have a lambda that's already been parsed/compiled and just needs to be evaluated.
Granted, I fully cede that your point is largely right. Just feels like you are a bit too strong with the "if and only if" on it.
That said, amusingly, `(1 2 3)` can easily be considered code if you are able to evaluate permutation notation. No?
(And yes, that can be used unquoted in a few different contexts in Lisp, such as a special form or macro, or if you've managed to convince the reader that 1 is a function.)
(Also, there's a more general problem of newbies trying to use the most advanced tool they've been exposed to. Why use the standard `if` statement, when you can use a proprietary high-powered pattern-matching form, now backed by a networked Kubernetes cluster of deep learning and LLMs.)
In the Racket community, one of my many attempts to discourage mistaken uses of `eval`:
https://groups.google.com/g/racket-users/c/Z-IlF24RAKU/m/3h6...
In the all-in-one practitioner's book I was writing, eval wouldn't be introduced until almost the end, in the "Dangerous Last Resort" part of the book. (Maybe I should've planned a marketing gimmick around it, as "free bonus DLC", and you need to do some ritual to be a Certified Certifiable Racketeer, before you can read the eval secret scroll.)
- in Javascript: https://link.springer.com/chapter/10.1007/978-3-642-22655-7_...
- in R: https://dl.acm.org/doi/abs/10.1145/3485502
eval can be sandboxed with a combination of multiple approaches, including validating the code, and evaluating it in a restricted environment in which only certain symbols exist. The full library package is not visible.
On eval in dynamic languages generally and in Racket specifically (2011) - https://news.ycombinator.com/item?id=8098569 - July 2014 (18 comments)
There's also a PEP to make them first-class: https://peps.python.org/pep-0638/
Syntactic macros are still second-class, like Lisp macros, but an improvement over text-replacement style macros.
For something macro-like which is first-class, there are fexprs[2] and operatives (from Kernel[3]) - these receive their operands verbatim, like macros, so they don't require quotation if we want to suppress evaluation. fexprs/Operatives can be passed around like any other value at runtime.
[1]:https://en.wikipedia.org/wiki/First-class_citizen
[2]:https://en.wikipedia.org/wiki/Fexpr
[3]:https://web.cs.wpi.edu/~jshutt/kernel.html
Thank you for the clarification and the additional information, I think having macros as first-class objects is a cool (but separate) idea.
I wouldn't say this is the case. Nearly any language could implement an `eval` for itself, but obviously, it is much simpler in Lisps because there's little syntax and few rules.
What makes Lisp macros different from say, C preprocessor macros, is the body of the macro is just Lisp code - so the "preprocessor" in this case has full access to the host languages facilities, which may include `eval`. The macros don't take textual input, but they take structured input in the form of S-expressions.
Implementing macros is obviously simpler due to eval, because we need to run the evaluator on the macro body, but it's not a strict requirement, as macro functionality could be provided by the implementation and could encapsulate its own evaluator.
Lisp macros are also simple due to the fact that Lisp code is just lists of data - you don't have to navigate a complex AST to walk through the code and emit particular syntax. You walk through the input with `car` and `cdr`, and you emit new syntax with `cons` (or `list`/`list*` which are derived from it). Macros can take code as their argument, and produce new code which is evaluated in place.
Macros still have hygiene issues though, because they're based on expanding code before evaluating it, variables used in macros can accidentally shadow variables in the scope of the macro's caller. There are workarounds (gensym) to navigate these hygiene problems.
> From what I gather, Lisp starts with a small set of primitives and special forms—seven in the original Lisp, including lambda. I recall Paul Graham demonstrating in one of his essays that you can build an eval function using just these primitives.
This is largely a theoretical demonstration but not real-world usage. In practice, Lisps have dozens or hundreds of "primitives". Common Lisp in particular is a big language and not trivial to implement. Scheme is a bit smaller, though r6rs started to also grow quite large, but this approach was revisited in r7rs (current), which aims for a small core, with additional functionality being provided through SRFIs (Scheme requests for implementation).
> Those primitives are typically implemented in a host language like C, but once you have an eval function in Lisp, you can extend it with new rules.
Using Scheme as an example, some SRFIs can be implemented purely in Scheme, as libraries, but others require the implementation to provide support, which often requires writing C code to provide them.
> This seems like a way to understand macros, where you effectively add new language rules. I know Lisp macros are typically defined using specific keywords like defmacro
As you note, it's `defmacro`, or `macro`, or `syntax-rules`, `syntax-case`, etc, which introduce new syntax - not eval in particular. Some macros will use `eval` in their bodies, which permits control of evaluation other than the regular applicative form of lambdas.
Macros are more than just `eval`. They're a multi-stage evaluation model where we first need to do some `macroexpand` (which will internally use `eval`), and afterwards the the resulting expression from the macro call is evaluated.
> but is the core idea similar—extending the language by building on the eval function with new rules?
There are some Lisps which still attempt this kind of minimalism.
One example is Ian Piumarta's Maru[1], which support extending `eval` (and `apply`) with new functionality at runtime based on the type being evaluated. Maru basically has global maps of type->evaluator and type->applicator, where we can add new pairs to at runtime and augment the behavior of `eval`/`apply`.
Kernel[2] also aims for the minimalist approach and does away with macros, quote and special-forms entirely, instead replacing them with a more general feature called an operative. The Kernel evaluator does not need to implement special rules for things like `lambda`, `cond`, `car`, `cdr` (as in Graham's "On Lips" essay) - but it just discriminates two forms - operative or applicative. Obviously, some kinds of operative are "primitive", but there's no difference from the PoV of the programmer. Which set of symbols you decide to implement as primitive is up to the implementation. The Kernel report suggests a small set of primitives and demonstrates the remaining standard features can be implemented using only the functionality provided so far.
[1]:https://piumarta.com/software/maru/
[2]:https://web.cs.wpi.edu/~jshutt/kernel.html
So on the less-well-defined end, you have something like JavaScript's `eval` function, in the middle you have lisp macros, which get an AST instead of just source code, and on the actually-pretty-well-defined end, you have a lambda that's already been parsed/compiled and just needs to be evaluated.