Recently we build one Magento 2 online store but I notice there are thousands of spam search terms, e.g. “大奖娱乐88tb88手机版+Q82019309.com.com“, after I deleted all these spam search terms, it will produce one by one soon. So how to stop spam search terms produced?
I checked the server access log, which there are thousands followed records:
18.104.22.168 - - [12/Feb/2017:13:16:46 +0000] "GET /robots.txt HTTP/1.0" 301 749 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
22.214.171.124 - - [12/Feb/2017:13:19:03 +0000] "GET /catalogsearch/result/?q=888%E5%A4%A7%E5%A5%96%E5%A8%B1%E4%B9%9088pt88+Q82019309.com.com HTTP/1.0" 200 8587 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
The log records show the bot try to get robots.txt, but it get 301, which means there is no robots.txt, then the bot try to visit Magento’s search page so add the spam search terms.
After understanding this, it will be easy to solve. create robots.txt file under website root folder and add the followed command: