The crawl before the fall of referrals: AI's impact on content providers