Ask HN: How do you find and format public data for your apps?

1 AdobiWanKenobi 1 7/1/2025, 3:49:16 PM

I'm working on a project that sources a lot of publically accessible but unprocessed, unformatted and unconsolidated data. Currently I'm just running Brave API + Crawlee (Playwright) + AWS Bedrock in Python for realtime processing. This is fine but not exactly scalable.

I have also tried to do some web scraping myself building my own data processing pipeline from scratch and honestly this is a bit of a pain.

So I am asking HN how do you find and process information online. Especially if its not something you can plan ahead for.

I know there's tavily search as well as the new perplexity api but they don't quite fulfil what I need.

Comments (1)

sargstuff · 14h ago

Finding (Note: where data from & terms of use!) : general search engine search on "publically available data for " <insert topics/areas of interest aka computer science>

> ... building my own data processing pipeline from scratch and honestly this is a bit of a pain.

Yup, only thing really changed over years is more ways to access data / volume of data / faster ways to process data.

Show HN: Spegel, a Terminal Browser That Uses LLMs to Rewrite Webpages (simedw.com)

Show HN: I made a 2D game engine in Dart (bullseye2d.org)

Show HN: Jobs by Referral: Find jobs in your LinkedIn network (jobsbyreferral.com)

Show HN: Core – open source memory graph for LLMs – shareable, user owned (github.com)

Show HN: A modern C++20 AI SDK (GPT‑4o, Claude 3.5, tool‑calling)

Show HN: Learning Rust with Neovim, Rust-Analyzer and LLMs (lorentz.app)

Show HN: I built the tool I wished existed for moving Stripe between countries (stripemove.com)

Show HN: HackerNewt – Breadth-first exploring HN client for iOS (apps.apple.com)

Show HN: A continuation of IRS Direct File that can be self-hosted (github.com)

Show HN: Arch-Router – 1.5B model for LLM routing by preferences, not benchmarks

Show HN: A local secrets manager with easy backup (github.com)

Show HN: Exceptional free templates, component libraries and boilerplates (htmlrev.com)

Show HN: Conduit – Turn large text files into listenable audio (conduit-landing-page-git-master-tobys-projects-a638df7e.vercel.app)

Show HN: Procedurally generated 3DGS splats powered by Spark (github.com)

Show HN: Just a Line: Resurrected (github.com)

Show HN: ToplingDB - A Persistent Key-Value Store for External Storage (github.com)

Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken (github.com)

Show HN: Open-Source International Space Station Tracker ESP32/Arduino for $20 (github.com)

Show HN: tapable-tracer Trace the connections and flows between tapable hooks (github.com)

Show HN: New Ensō – first public beta (untested.sonnet.io)

Show HN: We're two coffee nerds who built an AI app to track beans and recipes (beanbook.app)

Show HN: Terminal in Browser (github.com)

Show HN: I built a VPN management UI for my non-technical family (github.com)

Show HN: Local LLM Notepad – run a GPT-style model from a USB stick (github.com)

Show HN: Generate presentation slides from Jira backlog. + Demo-playground (agileplus.io)

Show HN: “Text me when my Congress member votes” – daily alerts, (not a app) (sites.google.com)

Show HN: Octelium – FOSS Alternative to Teleport, Cloudflare, Tailscale, Ngrok (github.com)

Show HN: I'm an airline pilot – I built interactive graphs/globes of my flights (jameshard.ing)

Show HN: Runik – Turn fan wikis into e-reader dictionaries (github.com)

Show HN: A tool to minimize echo chambers on social media (tandfonline.com)

Show HN: Lifp – A Lisp Built on Bun (github.com)

Show HN: I built a daily sunlight tracker (lumehealth.io)

Show HN: Private real-time dictation app for Mac (github.com)

Show HN: Get off your a** and do push-ups (timm.so)

Show HN: C.O.R.E – Opensource, user owned, shareable memory for Claude, Cursor (github.com)

Show HN: Subwise – A Smart Subscription Tracker with Browser Integration (usesubwise.com)

Show HN: Fixstars AIBooster – Accelerate AI Training and Cut GPU Costs (fixstars.com)

Show HN: Sink – Sync any directory with any device on your local network (github.com)

Show HN: Vet – A tool for safely running remote shell scripts (getvet.sh)

Show HN: Summle – A little maths Game (summle.net)

Show HN: Sharpe Ratio Calculation Tool (fundratios.com)

Show HN: Magnitude – Open-source AI browser automation framework (github.com)

Show HN: A cross-platform terminal emulator written in Java (github.com)

Show HN: SpeechAnalyzerDylib – C-FFI for macOS 26.0 SpeechAnalyzer API (github.com)

Show HN: Zenta – Mindfulness for Terminal Users (github.com)

Show HN: Attach Gateway – one-command OIDC/DID auth for local LLMs (github.com)

Show HN: I built an AI dataset generator (github.com)

Show HN: A tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted) (llmapitest.com)

Show HN: Memva – Sell your software without leaving the terminal (memva.com)

Show HN: A different kind of AI Video generation

Ask HN: How do you find and format public data for your apps?

Comments (1)