commonly used text browser is lynx, you can install it in the Linux environment, it is in your web browser, and search engine spiders almost see the content of the agreement, so we often use it to test web crawl, if the link is important if your pages use JS and AJAX technology. The lynx browser is not to see these links, the same search engines do not grab these links. So in the web site before the online, you can first use the browser to check the main text ", whether to use the technical
1, use a text browser to check your web page, see whether it is an important link of URL text browser grab
of each site had better set a robots text >
access behavior for tracking users is very effective, but for search engines, it is not good practice, if the use of these techniques, you may be "index is not complete, because the search engine program also cannot be ruled out that some from the URL look different, but" the content is exactly the same ". Is not friendly to included.
2, allows the search engine spiders do not use in tracking its website access path session ID or parameters in the case of crawling your site
3, ensure that your network server supports If-Modified-Since HTTP header
not friendly to the search engine Shanghai dragon
if your server supports if-modified-since-HTTP header, when the search engine spiders crawl to your web page, will first check the If-Modified-Since HTTP header values, through this value, search engine crawler can tell your web page since its last crawl after any change, if there is no change, it will can not download the same page content, save your spider resources, but also save the bandwidth of the server, so that spiders can grab more other pages.
and technology are closely related, many people think that is not the Shanghai dragon, in fact this is a misunderstanding, if you do not understand the technology, it is not possible to Shanghai dragon is very successful, a problem can not make scientific decisions, in many people’s eyes, Shanghai the dragon is nothing more than day to send some articles, send some outside the chain, if you do understand Shanghai Longfeng, your level is still in a very early stage, even if the search engine is a very good thing, so we want to get the corresponding flow on it, we must have some of the technology, it should at least understand the principle, then the web page is in accordance with the requirements of Shanghai Dragon technology, we should pay attention to what the problem?
Some session ID
4, setting a reasonable robots file