Search engine web crawlers constantly check for new content to index on web sites, which can affect a site’s ranking in natural search results. Thus, it pays to ensure that sites are presenting crawlers with new and unduplicated information, experts say.
Search engine web crawlers constantly check for new content to index on web sites, which can affect a site’s ranking in natural search results. Thus, it pays to ensure that sites are presenting crawlers with new and unduplicated information, experts say.
“When you design the architecture of your site, you want search engines crawling content that is good and new,” says Detlev Johnson, a Chicago-based search marketing expert who operates SearchReturn.com. Detlev spoke on natural search marketing strategies this week at the ACCM direct-marketing industry conference and trade show in Orlando, FL.
When designing a web page for the holiday shopping season, for example, a retailer might arrange products and promotions in the middle of a page. That might get indexed by a web crawler to show up high in natural search rankings related to holiday gifts. But if the outside edges of the same page contain extensive sections of unrelated information—a retailer’s privacy policy, for example—the page could get blocked from appearing in rankings of holiday gifts, Detlev says.
Retailers can manage how their pages get crawled, he adds, by using “redirects” to forward crawlers to temporary or permanent pages with more appropriate content. But they must use the appropriate redirect, such as a 301 redirect for forwarding to a new page on a permanent basis, or a 302 redirect for forwarding to a temporary page that will eventually revert back to the original page.
In addition, a web site designed with multiple URLs for the same product can confuse web crawlers trying to index a site for natural search, says Matt Bailey, president of search engine optimization firm Site Logic, who also spoke at ACCM. One quick way to test how well a site’s URLs are designed is to first navigate to a site by entering its home page URL in a browser, then click the site’s brand logo to see if it pulls up a different URL. “If it’s a different URL producing a duplicate copy of the same page, search engines try to figure out which is the right page. If you’re not controlling your duplicate pages, you’re causing problems with search engine optimization.”
Getting URLs under control is a good way to help search engines record a page, and it helps customers find products, he adds. A URL that contains many extra characters not related to a product category or brand makes it more difficult for shoppers to understand where they are in a search and navigation trail, he adds.
Bailey points to Target.com as an example of a site with long URLs that can negatively affect both natural search optimization and customer browsing. When a shopper clicks on a link for a Canon digital camera, for example, the URL shows a long series of numbers and other characters unrelated to cameras in addition to words like “camera” and “Canon.” An example of a site with cleaner URLs is BabySuperMall.com, where a click on Bananafish baby crib bedding shows a relatively short URL designed with useful navigation and product terms like “browse/by-brand-bananafish,” Bailey says.
Detlev advises site operators to use tools available from major search engines to improve how search engine crawlers—also called spiders, robots or bots—index their pages and to look for problems such as duplicate pages that may confuse crawlers. Such tools, often free, are available at sites including Robots.txt.org, Google.com/webmasters and SiteExplorer.search.yahoo.com.
Back...
