Show HN: POC to scrape and structure HTML into JSON for RAG
8 nirvanist 4 4/29/2025, 4:29:14 AM structured.pages.dev ↗
Hey all,
I built a quick PoC that scrapes a webpage, sends the content to Gemini Flash, and outputs a clean, structured JSON — ready for RAG workflows.
In my case, I’ll use this structured data to enhance models by integrating external knowledge sources during the generation process.
Curious if you think this has potential or if there are any use cases I might have missed. Happy to share more details if there's interest!
Comments (4)
mahi_novice · 20h ago
Do you mind sharing more about the implementation details? Any safeguards you have for the urls and all?
nirvanist · 18h ago
Basically, I use a headless Chromium with Puppeteer to render the page. Then, some logic extracts and cleans the HTML content. Finally, I use Gemini with a specific schema to return a JSON response.
mahi_novice · 20h ago
How do you plan to use it?
nirvanist · 18h ago
At the moment, I use it in client projects to build agents for their chat systems by adding RAG to models