Really, people should know by now that jq does point-free programming.
naniwaduni · 8h ago
Well you see, the author already knows common lisp, is familiar with their own ad hoc DSL by virtue of having just come up with it, and refuses to learn jq.
Personally, I probably would've written '[.root[][1]]' for that problem myself though—not a huge fan of map/1.
BoingBoomTschak · 5h ago
Just noticed this was submitted here.
1) I dislike that .[1] can be both an expression evaluated as a query and a "lambda". Really messes with my mind.
2) In my eyes, it's more intuitive because it looks like globbing and everybody knows globbing (this is the reason I use `**` too).
But yeah, this is a bit subjective. What isn't, though, is that I don't plan on adding much more than that; maybe merge, transform and an accessor using the same syntax. So if you know the host language, there's much less friction.
I really see this like Avisynth vs Vapoursynth.
cube2222 · 9h ago
The sentiment resonates with me.
Had similar thoughts a couple years ago, and wrote jql[0] as a jq alternative with a lispy syntax (custom, not Common Lisp), and I’ve been using it for command-line JSON processing ever since!
I learned the basics of jq and quite liked it, but since I discovered Nushell it has replaced nearly all my data processing I do at the cli. It really is good technology.
1oooqooq · 6h ago
site looks like a bashcompletion thing, how does it replace jq?
rafram · 5h ago
Not sure where you're getting the idea that it's a Bash completion extension from. It's a new shell (see name) that natively supports complex nested data structures, numbers, numbers with units, and so on. Compare with the classic POSIX shell model where everything is line-based (defined loosely), numerical operations rely on hacks, and splitting command output lines relies on hardcoded column indices.
bpshaver · 5h ago
I have nushell installed and use it sometimes. Does it have built-in JSON parsing like jq?
Edit: Well, I just found out about `cat some.json | from json` in nushell. Pretty cool! The nested tables are nice.
account-5 · 5h ago
No need for that even. It's just:
open some.json
account-5 · 4h ago
It's definitely not bash completion. But on the jq note you can read JSON, and various other formats, directly into nushell's data model (a table) and just start querying it.
forty · 6h ago
> I seriously dislike jq's convoluted, impossible-to-remember ad hoc DSL that instantly joined heaps of misery like CMake and gnuplot in my heart.
I like jq and gnuplot quite well. Makes me want to try CMake out ;)
Can convert between JSON<>EDN<>YAML<>Transit easily, plus includes a nifty little query language that is basically Clojure, so data transformations/extraction ends up really simple and concise.
I've always liked jq for simple things, but since I never sat down to actually work through the syntax, harder things tend to be too complicated to figure out quickly. Usually end up using Jet instead as if you already know Clojure, you already know the query language Jet uses.
One huge advantage of JQ is that it often is installed. I have jq in our Jenkins image, but I do not have this tool. The syntax is bit arcane, but once you invest bit of time into learning it, it starts to make sense (to a degree). It is a reasonable language for a stream processing.
ramses0 · 8h ago
There's a few jq patterns I've adopted:
echo "$SOME_JSON" | jq '.[]' --raw-output --compact-output | while read -r LINE ; do ... ; done
...lets you process stuff "record by record" pretty consistently. (and `( xxx ; yyy ; zzz ) | jq --slurp '.'` lets you do the reverse, "absorbing" multiple records into an array.
...lets you "load" json for merging, processing, formatting, etc. The leading "{}" is moderately necessary because `jq` technically _processes_ json, not generates it.
Finally, it's a huge cheat code for string formatting!!
$ echo "{}" | jq \
--arg FOO "hello \$world" \
--arg BAR "complicated \| chars" \
--arg ONE 1 \
--arg TWO 2 \
'"aaa \( $FOO ) and \( $BAR ) and \( ($ONE | tonumber) + ($TWO | tonumber) ) bbb"'
"aaa hello $world and complicated \\| chars and 3 bbb"
...optionally with `--raw-output` (un-json-quoted), and even supports some regex substitution in strings via `... | gsub(...)`.
Yes, yes... it's overly complicated compared to you and your fancy "programming languages", but sometimes with shell stuff, the ability to _CAPTURE_ arbitrary command output (eg: `--argjson LS_OUTPUT="$( ls -lart ... )"`), but then also use JSON/jq to _safely_ marshal/deaden the data into JSON is really helpful!
naniwaduni · 8h ago
> The leading "{}" is moderately necessary because `jq` technically _processes_ json, not generates it.
The --null-input/-n option is the "out-of-the-box" way to achieve this, and avoids a pipe (usually not a big deal, but leaves stdin free and sometimes saves a fork).
This lets you rewrite your first "pattern":
jq -cnr --argjson SOME_JSON "$SOME_JSON" '$SOME_JSON[]' | while read ...
We also have a "useless use of cat": --slurpfile does that job better:
jq -n --slurpfile FOO test.json '{bar: $FOO[]}'
(assuming you are assured that test.json contains one json value; --argjson will immediately fail if this is not the case, but with --slurpjson you may need to check that $FOO is a 1-item array.)
And of course, for exactly the single-file single-object case, you can just write:
jq '{bar: .}' test.json
ramses0 · 4m ago
Prefer long args in all cases, especially scripts and teaching.
Pipelines allow consistent syntax, but thanks for pointing out all the different variations of file support in jq.
great_wubwub · 8h ago
gron does 90% of what I need for json processing, it's a great first step and often the only necessary step.
I fail to see how your string-formatting example is better than using bash's printf?
ramses0 · 3m ago
You get a json-quoted (or json-quotable) string at the end. In bash, that's sometimes worth its weight in gold.
rafram · 8h ago
The `?` query operator is just a different, equally inscrutable DSL...
adamgordonbell · 5h ago
If you go through and learn some basic examples, JQ is a lot more understandable than seeing golfed examples in the wild might led you to believe. I wrote a tutorial once.
But it does also seem like a place where LLMs are handy. Why learn jq or regex or AWK, if you use them infrequently, when you can just ask an llm?
It feels like we're in danger of being stuck with the popular tools we have now for the foreseeable future. Why use a new tool when you can have the LLM figure it out using the old tools? Which means LLMs can't learn new tools because there's no human examples for them to learn from.
photonthug · 50m ago
Of course, it's different for tools that aren't open, but. Using best-in-class tools rather than making new ones is usually just a reasonable choice to avoid fragmentation of effort/interest that's ultimately pretty harmful to any software ecosystem.
As an example.. any candidate for replacing jq needs to be either faster or easier. If it's only a faster implementation, why change the query language? If it's only a different query language but not faster, then why not transpile the new query language into one that works with the old engine? Doing both at the same time without sacrificing completeness/expressiveness in the query language may warrant fragmentation of effort/interest, but that's a very high bar I would think..
True, this was initially for the sake of benchmarking. Very bad idea, especially since it doesn't really benefit here.
sauercrowd · 9h ago
Why's that? What does it do?
aidenn0 · 8h ago
The Lisp standard allows it to be implementation dependent, but in SBCL (which I believe the author of TFA is using), it disables all runtime type and bounds checking. This usually speeds things up negligibly[1] in exchange for allowing all kinds of undefined behavior.
1: If it speeds things up non-negligibly, there's almost always a way to get a similar speedup without setting safety to 0; e.g. if you check your types outside of your hot loops, the compiler is smart enough to omit type-checks inside the loop.
jlarocco · 6h ago
Honestly it's more nuanced than that, and probably not such a big deal.
It's kind of like building in Debug mode in other languages. Internally and for testing, use (safety 3). If the code in question doesn't trigger any errors or warnings, then in most cases it's safe to turn (safety 0) and get the tiny performance boost.
I wouldn't recommend (safety 0) globally, but it's probably fine locally in performance critical code that's been tested well, but I do agree it's probably not worth going to (safety 0) in most cases.
The best solution is a compiler who's (speed 3) optimization level is smart enough to optimize out the unnecessary safety checks from (safety 3). I think SBCL can do that in some cases (the safety checks get optimized for speed, at least).
aidenn0 · 6h ago
> If the code in question doesn't trigger any errors or warnings, then in most cases it's safe to turn (safety 0) and get the tiny performance boost.
This is trivially not true. Consider:
(defun foo (x)
(declare (safety 0)
(type x (array fixnum (4)))
[Lots of code that doesn't trigger any warnings])
Then in a different source file doing e.g:
(foo nil)
Nothing good will come of that.
> I wouldn't recommend (safety 0) globally, but it's probably fine locally in performance critical code that's been tested well, but I do agree it's probably not worth going to (safety 0) in most cases.
> The best solution is a compiler who's (speed 3) optimization level is smart enough to optimize out the unnecessary safety checks from (safety 3). I think SBCL can do that in some cases (the safety checks get optimized for speed, at least).
The only thing I can think of is that I communicated things poorly in my comment, because this is nearly exactly what I was saying in my comment.
jlarocco · 6h ago
Sure, that code fails with a memory fault with (safety 0). But I explicitly recommended testing and debugging with (safety 3) first, and in that case it gets a type-error and doesn't crash. Once the broken `(foo nil)` is fixed and it's time to ship a release build to the user, it should be safe to drop down to (safety 0).
I think we both agree that 99.9% of the time it's not worth using (safety 0), though.
naniwaduni · 8h ago
This is a bit of a nonsense benchmark—the author is basically benchmarking process startup time, and finding that if you only start one lisp process, it runs faster than if you start one jq process per input file, which runs faster than if you start one lisp process per input file. This becomes quite clear at the end when they compare multiple jq implementation (which are known to have nontrivially different parsing performance characteristics, which is the bulk of the "real work" for this task lies) and come up with execution times on the same order of magnitude.
A more apples-to-apples comparison would be to use find {} + to pass multiple filenames to jq and output using input_filename.
selkin · 6h ago
I am old enough to remember when creating a new DSL for every task was in vogue. What used to be an issue with that approach, which I've seen hinted to in another comment here, is that I can only become proficient in so many programming languages. Achieving proficiency requires a lot of practice, so using that specific DSL needs to have a high value to justify the time investment in practicing.
This issue is almost negated today: I find myself no longer writing jq queries, or regular expressions (both I am quite proficient in,) but having AI write those for me. This is exactly where the so-called "vibe coding" shines, and why I no longer care about tool specific DSLs.
aidenn0 · 8h ago
I'm guessing there's supposed to be something other than an empty black box after "already an improvement, in my eyes:"? I'm just seeing a black box (on firefox).
[edit]
Removing "Unifont" from the font-family list fixes the problem, so I must have an issue with my unifont install?
For those that like that style, on Linux both Xfce and KDE have themes that replicate it for their window decorations (recommending the desktop environment would be a bit too much)
ramses0 · 8h ago
He's got an "about..." section which is a treat and gives some details on it. It works surprisingly well and is a bit of a blast from the past! :-D
pletnes · 8h ago
Pyjq is the python package for jq, in case you want to use jq as a library. The article claims that this doesn’t exist.
cAtte_ · 8h ago
no, what the author claims is that jq should've been designed as a regular python library, using python syntax, rather than as a bespoke DSL (that happens to have parser bindings for python)
Personally, I probably would've written '[.root[][1]]' for that problem myself though—not a huge fan of map/1.
1) I dislike that .[1] can be both an expression evaluated as a query and a "lambda". Really messes with my mind.
2) In my eyes, it's more intuitive because it looks like globbing and everybody knows globbing (this is the reason I use `**` too).
But yeah, this is a bit subjective. What isn't, though, is that I don't plan on adding much more than that; maybe merge, transform and an accessor using the same syntax. So if you know the host language, there's much less friction.
I really see this like Avisynth vs Vapoursynth.
Had similar thoughts a couple years ago, and wrote jql[0] as a jq alternative with a lispy syntax (custom, not Common Lisp), and I’ve been using it for command-line JSON processing ever since!
[0]: https://github.com/cube2222/jql
Edit: Well, I just found out about `cat some.json | from json` in nushell. Pretty cool! The nested tables are nice.
open some.json
I like jq and gnuplot quite well. Makes me want to try CMake out ;)
Can convert between JSON<>EDN<>YAML<>Transit easily, plus includes a nifty little query language that is basically Clojure, so data transformations/extraction ends up really simple and concise.
I've always liked jq for simple things, but since I never sat down to actually work through the syntax, harder things tend to be too complicated to figure out quickly. Usually end up using Jet instead as if you already know Clojure, you already know the query language Jet uses.
https://github.com/inconvergent/lqn
* [lqn](https://github.com/inconvergent/lqn) - query language and terminal utility for querying and transforming Lisp, JSON and other text files.
(by this person doing nice generative art: https://inconvergent.net/)
Don't forget `--argjson`
...lets you "load" json for merging, processing, formatting, etc. The leading "{}" is moderately necessary because `jq` technically _processes_ json, not generates it.Finally, it's a huge cheat code for string formatting!!
...optionally with `--raw-output` (un-json-quoted), and even supports some regex substitution in strings via `... | gsub(...)`.Yes, yes... it's overly complicated compared to you and your fancy "programming languages", but sometimes with shell stuff, the ability to _CAPTURE_ arbitrary command output (eg: `--argjson LS_OUTPUT="$( ls -lart ... )"`), but then also use JSON/jq to _safely_ marshal/deaden the data into JSON is really helpful!
The --null-input/-n option is the "out-of-the-box" way to achieve this, and avoids a pipe (usually not a big deal, but leaves stdin free and sometimes saves a fork).
This lets you rewrite your first "pattern":
We also have a "useless use of cat": --slurpfile does that job better: (assuming you are assured that test.json contains one json value; --argjson will immediately fail if this is not the case, but with --slurpjson you may need to check that $FOO is a 1-item array.)And of course, for exactly the single-file single-object case, you can just write:
Pipelines allow consistent syntax, but thanks for pointing out all the different variations of file support in jq.
https://github.com/tomnomnom/gron
But it does also seem like a place where LLMs are handy. Why learn jq or regex or AWK, if you use them infrequently, when you can just ask an llm?
Edit: tutorial: https://earthly.dev/blog/jq-select/
As an example.. any candidate for replacing jq needs to be either faster or easier. If it's only a faster implementation, why change the query language? If it's only a different query language but not faster, then why not transpile the new query language into one that works with the old engine? Doing both at the same time without sacrificing completeness/expressiveness in the query language may warrant fragmentation of effort/interest, but that's a very high bar I would think..
Please, don't do that!
1: If it speeds things up non-negligibly, there's almost always a way to get a similar speedup without setting safety to 0; e.g. if you check your types outside of your hot loops, the compiler is smart enough to omit type-checks inside the loop.
It's kind of like building in Debug mode in other languages. Internally and for testing, use (safety 3). If the code in question doesn't trigger any errors or warnings, then in most cases it's safe to turn (safety 0) and get the tiny performance boost.
I wouldn't recommend (safety 0) globally, but it's probably fine locally in performance critical code that's been tested well, but I do agree it's probably not worth going to (safety 0) in most cases.
The best solution is a compiler who's (speed 3) optimization level is smart enough to optimize out the unnecessary safety checks from (safety 3). I think SBCL can do that in some cases (the safety checks get optimized for speed, at least).
This is trivially not true. Consider:
Then in a different source file doing e.g: Nothing good will come of that.> I wouldn't recommend (safety 0) globally, but it's probably fine locally in performance critical code that's been tested well, but I do agree it's probably not worth going to (safety 0) in most cases.
> The best solution is a compiler who's (speed 3) optimization level is smart enough to optimize out the unnecessary safety checks from (safety 3). I think SBCL can do that in some cases (the safety checks get optimized for speed, at least).
The only thing I can think of is that I communicated things poorly in my comment, because this is nearly exactly what I was saying in my comment.
I think we both agree that 99.9% of the time it's not worth using (safety 0), though.
A more apples-to-apples comparison would be to use find {} + to pass multiple filenames to jq and output using input_filename.
This issue is almost negated today: I find myself no longer writing jq queries, or regular expressions (both I am quite proficient in,) but having AI write those for me. This is exactly where the so-called "vibe coding" shines, and why I no longer care about tool specific DSLs.
[edit]
Removing "Unifont" from the font-family list fixes the problem, so I must have an issue with my unifont install?
For those that like that style, on Linux both Xfce and KDE have themes that replicate it for their window decorations (recommending the desktop environment would be a bit too much)