Perl Harbour

Nik Barron - www.virus.org - 20 January 2010Security

An errant web search script from Bing, Microsoft's search engine

An errant web search script from Bing, Microsoft's search engine, managed to seriously degrade the performance of a major Perl resource recently, the Comprehensive Perl Archive Network (CPAN). CPAN is a central repository of Perl scripts and modules, the "swiss army chainsaw" used by countless web sites and other online applications. CPAN has been online since 1995.

Scripts that "spider" web sites are common for search engines, but are supposed to obey the "robots.txt" mechanism, where a site's owner can specify what they want exposed on search engines. Database-driven web sites such as CPAN often restrict their sites from excessive spidering which can degrade performance and often result in broken search engine links due to the dynamic nature of the site.

Microsoft's servers were apparently hitting the site with 20-30 scripts every few seconds until site admins blocked access. To be fair to Microsoft, one of Bing's managers contacted CPAN via their blog and offered out of hours assistance to resolve the problem.

More information: