- "GET /sitemap.xml HTTP/1.1" 302 5 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
复制代码- "GET /sitemap.xml HTTP/1.1" 302 5 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181;Bytespider;https://zhanzhang.toutiao.com/"
复制代码- "GET /wp-sitemap-users-1.xml HTTP/1.1" 200 190 "/wp-sitemap-users-1.xml" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider"
复制代码
被视为扫描拦截,这是不是搜索引擎蜘蛛啊? |
|