Can a scraping bot have JavaScript enabled?

Can a scraping bot have JavaScript enabled?

I've got a few thousand request that seem to be coming from a client with JavaScript enabled and I'm wondering if that client could be a bot.

答案1

Yes - a bot can do whatever it wants with the resources that it's gathering, which could certainly include interpreting javascript.

Take, for instance, Google's new little thumbnail display of pages in search results; many sites rely on javascript for basic display functionality, so interpretation of js would be a must for that to come up with previews that are close to accurate.

The easiest way to tell may be to just check the user-agent string; well behaved bots will just tell you what they are.

相关内容