Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

For what it's worth, Brave Search lies about their User Agent too. I found it fishy as well, but they claim that many websites only allow Googlebot to crawl and ban other UAs. I remember searching for alternative search engines and finding an article that said most new engines face this exact problem: they can't crawl because any unusual bots are blocked.

I have tried programming scrappers in the past and one thing I noticed is that there doesn't seem to be a guide in how to make a "good" bot, since there are so few bots with legitimate use cases. Most people use Chrome, too. So I guess now UA is pointless as the only valid UA is going to be Chrome or Googlebot.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: