I think the project itself is really cool, that said I really don't like the trend of having LLMs regurgitate content back to us. That said, this kinda makes me think of Browsh, who took the opposite approach and tries to render the HTML in the terminal (without LLMs as far as I know)
That would also keep your wallet or GPU rag coller
qsort · 2h ago
This is actually very cool. Not really replacing a browser, but it could enable an alternative way of browsing the web with a combination of deterministic search and prompts. It would probably work even better as a command line tool.
A natural next step could be doing things with multiple "tabs" at once, e.g: tab 1 contains news outlet A's coverage of a story, tab 2 has outlet B's coverage, tab 3 has Wikipedia; summarize and provide references. I guess the problem at that point is whether the underlying model can support this type of workflow, which doesn't really seem to be the case even with SOTA models.
simedw · 2h ago
Thank you.
I was thinking of showing multiple tabs/views at the same time, but only from the same source.
Maybe we could have one tab with the original content optimised for cli viewing, and another tab just doing fact checking (can ground it with google search or brave). Would be a fun experiment.
nextaccountic · 1h ago
In your cleanup step, after cleaning obvious junk, I think you should do whatever Firefox's reader mode does to further clean up, and if that fails bail out to the current output. That should reduce the number of tokens you send to the LLM even more
You should also have some way for the LLM to indicate there is no useful output because perhaps the page is supposed to be a SPA. This would force you to execute Javascript to render that particular page though
simedw · 1h ago
Just had a look and three is quite a lot going into Firefox's reader mode.
Interestingly, the original idea of what we call a "browser" nowadays – the "user agent" – was built on the premise that each user has specific needs and preferences. The user agent was designed to act on their behalf, negotiating data transfers and resolving conflicts between content author and user (content consumer) preferences according to "strengths" and various reconciliation mechanisms.
(The fact that browsers nowadays are usually expected to represent something "pixel-perfect" to everyone with similar devices is utterly against the original intention.)
Yet the original idea was (due to the state of technical possibilities) primarily about design and interactivity. The fact that we now have tools to extend this concept to core language and content processing is… huge.
It seems we're approaching the moment when our individual personal agent, when asked about a new page, will tell us:
Well, there's nothing new of interest for you, frankly:
All information presented there was present on pages visited recently.
-- or --
You've already learned everything mentioned there. (*)
Here's a brief summary: …
(Do you want to dig deeper, see the content verbatim, or anything else?)
Because its "browsing history" will also contain a notion of what we "know" from chats or what we had previously marked as "known".
ffsm8 · 7m ago
> Well, there's nothing new of interest for you, frankly
For this to work like a user would want, the model would have to be sentient.
But you could try to get there with current models, it'd just be very untrustworthy to the point of being pointless beyond a novelty
wrsh07 · 1h ago
Would really love to see more functionality built into this. Handling post requests, enabling scripting, etc could all be super powerful
Bluestein · 6m ago
Gosh. Lovely project and cool, and - likewise - a bit scary: This is where the "bubble" seals itself "from the inside" and custom (or cloud, biased) LLMs sear the "bubble" in.-
The ultimate rose (or red, or blue or black ...) coloured glasses.-
__MatrixMan__ · 42m ago
It would be cool of it were smart enough to figure out whether it was necessary to rewrite the page on every visit. There's a large chunk of the web where one of us could visit once, rewrite to markdown, and then serve the cleaned up version to each other without requiring a distinct rebuild on each visit.
myfonj · 32m ago
Each user have distinct needs, and has a distinct prior knowledge about the topic, so even the "raw" super clean source form will probably be eventually adjusted differently for most users.
But yes, having some global shared redundant P2P cache (of the "raw" data), like IPFS (?) could possibly help and save some processing power and help with availability and data preservation.
simedw · 31m ago
If the goal is to have a more consistent layout on each visit, I think we could save the last page's markdown and send it to the model as a one-shot example...
pmxi · 34m ago
The author says this is for “personalized views using your own prompts.” Though, I suppose it’s still useful to cache the outputs for the default prompt.
__MatrixMan__ · 31m ago
Or to cache the output for whatever prompt your peers think is most appropriate for that particular site.
mromanuk · 1h ago
I definitely like the LLM in the middle, it’s a nice way to circumvent the SEO machine and how Google has optimized writing in recent years. Removing all the cruft from a recipe is a brilliant case for an LLM. And I suspect more of this is coming: LLMs to filter. I mean, it would be nice to just read the recipe from HTML, but SEO has turned everything into an arms race.
yellow_lead · 22m ago
LLM adds cruft, LLM removes cruft, never a miscommunication
hirako2000 · 26m ago
Do you also like what it costs you to browse the web via an LLM potentially swallowing millions of tokens per minutes ?
prophesi · 17m ago
This seems like a suitable job for a small language model. Bit biased since I just read this paper[0]
I wonder if you could use a less sophisticated model (maybe even something based on LSTMs) to walk over the DOM and extract just the chunks that should be emitted and collected into the browsable data structure, but doing it all locally. I feel like it'd be straightforward to generate training data for this, using an LLM-based toolchain like what the author wrote to be used directly.
kelsey98765431 · 59m ago
People here are not realizing that html is just the start. If you can render a webpage into a view, you can render any input the model accepts. PDF to this view. Zip file of images to this view. Giant json file into this view. Whatever. The view is the product here, not the html input.
bubblyworld · 2h ago
Classic that the first example is for parsing the goddamn recipe from the goddamn recipe site. Instant thumbs up from me haha, looks like a neat little project.
hyperific · 50m ago
Why not use pandoc to convert html to markdown and have the LLM condense from there?
coder543 · 35m ago
Just a typo note: the flow diagram in the article says "Gemini 2.5 Pro Lite", but there is no such thing.
simedw · 15m ago
You are right, it's Gemini 2.5 Flash Lite
cyrillite · 45m ago
I have been thinking of a project extremely similar to this for a totally different purpose. It’s lovely to see something like this. Thank you for sharing it, inspiring
amelius · 42m ago
Curious about that different purpose ...
clbrmbr · 2h ago
Suggestion: add a -p option:
spegel -p "extract only the product reviews" > REVIEWS.md
cheevly · 2h ago
Very cool! My retired AI agent transformed live webpage content, here's an old video clip of transforming HN to My Little Pony (with some annoying sounds): https://www.youtube.com/watch?v=1_j6cYeByOU. Skip to ~37 seconds for the outcome. I made an open-source standalone Chrome extension as well, it should probably still work for anyone curious: https://github.com/joshgriffith/ChromeGPT
neocodesoftware · 34m ago
Does it fail cloudflare captcha?
ohadron · 2h ago
This is a terrific idea and could also have a lot of value with regards to accessibility.
deepdarkforest · 1h ago
The main problem with these approaches is that most sites now are useless without JS or having access to the accessibility tree. Projects like browser-use or other DOM based approaches at least see the DOM(and screenshots).
I wonder if you could turn this into a chrome extension that at least filters and parses the DOM
jadbox · 1h ago
I actually made a CLI tool recently that uses Puppeteer to render the page including JS, summarizes key info and actions, and enables simple form filling all from a CLI menu. I built it for my own use-cases (checking and paying power bills from CLI), but I'd love to get feedback on the core concept: https://github.com/jadbox/solomonagent
willsmith72 · 1h ago
True for stuff requiring interaction, but to help their LCP/SEO lots of sites these days render plain html first. It's not "usable" but for viewing it's pretty good
adrianpike · 1h ago
Super neat - I did something similar on a lark to enable useful "web browsing" over 1200 baud packet - I have Starlink back at my camp but might be a few miles away, so as long as I can get line of sight I can Google up stuff, albeit slow. Worked well but I never really productionalized it beyond some weekend tinkering.
stared · 1h ago
Any chance it would work for pages like Facebook or LinkedIn? I would love to have a distraction-free way of searching information there.
Obviously, against wishes of these social networks, which want us to be addicted... I mean, engaged.
aydyn · 1h ago
Does anyone really get addicted to linkedin? Its so sanitized and clinical. Nobody acts real on there or even pretends to.
encom · 45m ago
The worst[1] part about losing my job last month was having to take LinkedIn seriously, and the best[2] part about now having found a new job is logging off LinkedIn, for a very long time hopefully. The self-aggrandising, pretentious, occasionally virtue signalling, performance-posting make me want to throw up. It takes a considerable amount of effort on my part to not make sarcastic shitposts, but in the interest of self preservation, I restrain myself. My header picture, however, is my extremely messy desk, full of electronics, tools, test equipment, drawings, computers and coffee cups. Because that's just how I work when I'm in the zone, and it serves as a quiet counterpoint to the polished self-promotion people do.
And I didn't even get the new job through LinkedIn, though it did yield one interview.
[1] Not the actual worst.
[2] Not the actual best.
simedw · 1h ago
We’ll probably have to add some custom code to log in, get an auth token, and then browse with it. Not sure if LinkedIn would like that, but I certainly would.
nashashmi · 1h ago
You should call this software a lens and filter instead of a mirror. It takes the essential information and transforms it into another medium.
WD-42 · 38m ago
Does anyone know why LLMs love emojis so much?
web3aj · 1h ago
Very cool. I’ve been interested in browsing the web directly from my terminal; this feels accessible.
anonu · 2h ago
Don't you need javascript to make most webpages useful?
inetknght · 1h ago
Good sir, no.
The web has existed for long before javascript was around.
The web was useful for long before javascript was around.
I literally hate javascript -- not the language itself but the way it is used. It has enabled some pretty cool things, yes. But javascript is not required to make useful webpages.
pmxi · 26m ago
I think you misunderstood him. Yes, it’s possible to CREATE a useful webpage without JavaScript, but many EXISTING webpages rely on JavaScript to be functional.
eniac111 · 1h ago
Cool! It would be even better if it was able to create simple web pages for vintage browsers.
stronglikedan · 1h ago
That would violate the do-one-thing-and-do-it-well principle for no apparent benefit. There are plenty of tools to convert markdown to basic HTML already.
098799 · 1h ago
You could also use headless selenium under the hood and pipe to the model the entire Dom of the document after the JavaScript was loaded. Of course it would make it much slower but also would amend the main worry people have which is many websites will flat out not show anything in the initial GET request.
busssard · 1h ago
can you flesh this out a tiny bit?
because for indy-crawlers the javascript rendering is the main problem.
pepperonipboy · 2h ago
Could work great with emacs' eww!
sammy0910 · 2h ago
I built a project that basically does this for emacs
Have you considered making an MCP for this? Would be great for use in vibe-coding
amelius · 1h ago
Can it strip ads?
tossandthrow · 59m ago
It can inject its own!
amelius · 21m ago
You have a point as it uses Gemini under the hood. However, the moment Google introduces ads in the model users will run away. So Google really has no opportunity here to inject ads.
And wouldn't it be ironic if Gemini was used to strip ads from webpages?
fzaninotto · 1h ago
Congrats! Now you need an entire datacenter to visualize a web page.
juujian · 1h ago
Couldn't this time reasonably well on a local machine is you have some kind of neutral processing chip and enough ram? Conversion to MD shouldn't require a huge model.
busssard · 1h ago
only if you use an API and not a dedicated distill/tune for html to MD conversion.
But the question of Javascript remains
insane_dreamer · 1h ago
Interesting, but why round-trip through an LLM just to convert HTML to Markdown?
markstos · 59m ago
Because the modern web isn't reliably HTML, it's "web apps" with heavy use of JavaScript and API calls. To first display the HTML that you see in your browser, you need a user agent that runs JavaScript and makes all the backend calls that Chrome would make to put together some HTML.
Some websites may still return some static upfront that could be usefully understood without JavaScript processing, but a lot don't.
That's not to say you need an LLM, there are projects like Puppeteer that are like headless browsers that can return the rendered HTML, which can then be sent through an HTML to Markdown filter. That would be less computationally intensive.
Klaster_1 · 1h ago
Now that's a user agent!
ktpsns · 2h ago
Reminds me of https://www.brow.sh/ which is not AI related at all but just a very powerful terminal browser which in fact supports JS, even videos.
No comments yet
b0a04gl · 1h ago
this is another layer of abstraction on top of an already broken system. you're running html through an llm to get markdown that gets rendered in a terminal browser. that's like... three format conversions just to read text. the original web had simple html that was readable in any terminal browser already. now they arent designed as documents anymore but rather designed as applications that happen to deliver some content as a side effect
MangoToupe · 1h ago
That's the world we live in. You can either not have access to content or you must accept abstractions to remove all the bad decisions browser vendors have forced on us the last 30 years to support ad-browsing.
jrm4 · 24m ago
I 100% agree -- but still I find this a feature and not a bug. It's always an arms race, and I like this shot fired.
worldsayshi · 58m ago
If the web site is a SPA that is hydrated using an API it would be conceivable that the LLM can build a reusable interface around the API while taking inspiration from the original page. That interface can then be stored in some cache.
I'm not saying it's necessarily a good idea but perhaps a bad/fun idea that can inspire good ideas?
_joel · 1h ago
> this is another layer of abstraction on top of an already broken system
pretty much like all modern computing then, hey.
nashashmi · 1h ago
Think of it as a secretary that is transforming and formatting information. You may desire for the original medium to be something like what you want but you don’t get that so you can get a cheap dumber secretary instead.
amelius · 43m ago
I take it you never use "Reader mode" in your browser?
https://github.com/browsh-org/browsh https://www.youtube.com/watch?v=HZq86XfBoRo
A natural next step could be doing things with multiple "tabs" at once, e.g: tab 1 contains news outlet A's coverage of a story, tab 2 has outlet B's coverage, tab 3 has Wikipedia; summarize and provide references. I guess the problem at that point is whether the underlying model can support this type of workflow, which doesn't really seem to be the case even with SOTA models.
I was thinking of showing multiple tabs/views at the same time, but only from the same source.
Maybe we could have one tab with the original content optimised for cli viewing, and another tab just doing fact checking (can ground it with google search or brave). Would be a fun experiment.
You should also have some way for the LLM to indicate there is no useful output because perhaps the page is supposed to be a SPA. This would force you to execute Javascript to render that particular page though
https://github.com/mozilla/readability
almost unrelated, but you can also compare spegel to https://www.brow.sh/
(The fact that browsers nowadays are usually expected to represent something "pixel-perfect" to everyone with similar devices is utterly against the original intention.)
Yet the original idea was (due to the state of technical possibilities) primarily about design and interactivity. The fact that we now have tools to extend this concept to core language and content processing is… huge.
It seems we're approaching the moment when our individual personal agent, when asked about a new page, will tell us:
Because its "browsing history" will also contain a notion of what we "know" from chats or what we had previously marked as "known".For this to work like a user would want, the model would have to be sentient.
But you could try to get there with current models, it'd just be very untrustworthy to the point of being pointless beyond a novelty
The ultimate rose (or red, or blue or black ...) coloured glasses.-
But yes, having some global shared redundant P2P cache (of the "raw" data), like IPFS (?) could possibly help and save some processing power and help with availability and data preservation.
[0] https://research.nvidia.com/labs/lpr/slm-agents/
I wonder if you could turn this into a chrome extension that at least filters and parses the DOM
Obviously, against wishes of these social networks, which want us to be addicted... I mean, engaged.
And I didn't even get the new job through LinkedIn, though it did yield one interview.
[1] Not the actual worst.
[2] Not the actual best.
The web has existed for long before javascript was around.
The web was useful for long before javascript was around.
I literally hate javascript -- not the language itself but the way it is used. It has enabled some pretty cool things, yes. But javascript is not required to make useful webpages.
https://github.com/sstraust/simpleweb
And wouldn't it be ironic if Gemini was used to strip ads from webpages?
But the question of Javascript remains
Some websites may still return some static upfront that could be usefully understood without JavaScript processing, but a lot don't.
That's not to say you need an LLM, there are projects like Puppeteer that are like headless browsers that can return the rendered HTML, which can then be sent through an HTML to Markdown filter. That would be less computationally intensive.
No comments yet
I'm not saying it's necessarily a good idea but perhaps a bad/fun idea that can inspire good ideas?
pretty much like all modern computing then, hey.
https://github.com/sstraust/simpleweb
I'm curious how you tackled that problem
Not the answer to your question but here's the prompt
I think most of it comes down to Flash-Lite being really fast, and the fact that I'm only outputting markdown, which is fairly easy and streams well.
No comments yet