Skip to main content

Polyglotbot

In order to source some parallel training data for machine translation we are using a web crawler named "Polyglotbot". The bot identifies itself with the user agent string polyglotbot/1.0 (+https://polyglot.technology/p/polyglotbot.html). Polyglotbot respects the robots.txt standard, so if you don't want the bot to crawl all or parts of your site, you can prevent it from crawling using this method. The bot also tries to be friendly and rate limits itself not to overload a site with requests. If you have questions or concerns about Polyglotbot, please email support@polyglot.technology