Firehose

Filtered to Hacker News, tagged “web crawling” · clear filters

Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives

All People Companies Papers Podcasts Hacker News

Browse by tag

artificial intelligence 43 AI 11 cybersecurity 6 finance 6 tech 6 productivity 4 machine learning 3 open-source 3 software development 3

2 AUG 2026 · Hacker News · 40 pts · 59 comments ↗

Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

Only 8.9% of websites block AI crawlers, but a significant majority (94.8%) are never named in AI answers, with 10 out of 193 sites being named in at least one answer across 5,978 assistant answers. AI summary

AI web crawling