Ask HN: How can I trace what user queries make AI bots crawl my site?

Comments (1)

benoau · 4h ago

It might be present in your web server logs, with the caveat that you may not have them or have access to them and the bots may not pass the HTTP_REFERER header and the logs may not record it.

It might be present in client-side logs, with the caveat the crawler might block tracker URLs or not open a page in a way that executes their scripts.

Either of these may reveal what the first-page was a crawler reached, or the pages they directly-access without navigating your site.

You might be able to correlate timing for when a "user query" occurs and subsequent scraping.

Google will allow only apps from verified developers to be installed on Android (9to5google.com)

Google's Liquid Cooling (chipsandcheese.com)

The Limits of NTP Accuracy on Linux (scottstuff.net)

The Annotated Transformer (2022) (nlp.seas.harvard.edu)

Show HN: Base, an SQLite database editor for macOS (menial.co.uk)

A visual introduction to big O notation (samwho.dev)

WiFi-3D-Fusion – Real-time 3D motion sensing with Wi-Fi (github.com)

Capture Checking in Scala (nrinaudo.github.io)

Climbing catfish filmed scaling waterfalls (science.org)

macOS 26 Tahoe's Dead Canary Utility App Icons (daringfireball.net)

How RubyGems.org protects OSS infrastructure (blog.rubygems.org)

Turning a Decommissioned iPhone into a UniFi Protect Camera (caseyliss.com)

DeepWiki: Understand Any Codebase (aitidbits.ai)

US retail giants raise prices due to tariffs (english.elpais.com)

Reverse Engineering All the Raspberry Pis (jeffgeerling.com)

Launch HN: April (YC S25) – Voice AI to manage your email and calendar

A small change to improve browsers for keyboard navigation (b.43z.one)

Mob Programming (2018) (mobprogramming.org)

Fenster: Most minimal cross-platform GUI library (github.com)

Building the mouse Logitech won't make (samwilkinson.io)

Neural Nets vs. Cellular Automata (nets-vs-automata.net)

Blacksky grew to millions of users without spending a dollar (newpublic.substack.com)

What is a color space? (makingsoftware.com)

Show HN: Timep – A next-gen profiler and flamegraph-generator for bash code (github.com)

How to Fix Your Context (dbreunig.com)

Playing every game of Wordle simultaneously (chriskw.xyz)

What are OKLCH colors? (jakub.kr)

Apple vs. Facebook Is Kayfabe (infrequently.org)

Exploring the tragedy of the Counter-Strike 2 server browser (bphilip.uk)

FCC bars providers for non-compliance with robocall protections (docs.fcc.gov)

How to make things slower so they go faster (gojiberries.io)

Llama Fund: Crowdfund AI Models (llama.fund)

Australia Post halts most parcels to US as tariff chaos hits global carriers (abc.net.au)

A short proof of the Hairy Ball theorem (2016) [pdf] (www2.math.upenn.edu)

An illustrated guide to OAuth (ducktyped.org)

The CTO Was ChatGPT (ehandbook.com)

Study finds gaps in evidence for air-cleaning technologies to prevent infections (news.cuanschutz.edu)

The MiniPC Revolution (jadarma.github.io)

Japan's Creepiest Station (tokyocowboy.co)

Gonzo: A Go-based TUI for log analysis (OpenTelemetry/OTLP support) (github.com)

Intel Xeon Clearwater Forest with 288 Cores on Intel 18A at Hot Chips 2025 (servethehome.com)

The Unix-Haters Handbook (1994) [pdf] (simson.net)

Ban me at the IP level if you don't like me (boston.conman.org)

Anonymous Structavaganza in Zig (lirk.top)

Show HN: Stagewise – frontend coding agent for real codebases (stagewise.io)

You're loading fonts wrong (and it's crippling your performance) (jonoalderson.com)

Default Methods in Go (mcyoung.xyz)

Scamlexity: When agentic AI browsers get scammed (guard.io)

MetaScope Build 10: Revolutionary File Processing and External Drive Support (zalodesignstudio.com)

Prison isn’t set up for today’s tech so we have to do legal work the old way (prisonjournalismproject.org)

Ask HN: How can I trace what user queries make AI bots crawl my site?

Comments (1)