May 10, 2004, 12:00 AM

How helping spiders reveals the web’s dynamic content to search engines

Frequently changing content is more difficult for search engines to index. In fact, 23% of attendees at a Oneupweb web seminar called the challenge of helping spiders find dynamic content “overwhelming” -- but a new product bridges the gap.

Dynamic – frequently changing – content constitutes much of the “invisible web,” so-called because it represents material less easily found and indexed by search engine crawlers. Yet as web sites’ use of dynamic content becomes more prevalent, the problems it can pose relative to search engine marketing grow. A recent web seminar by search engine marketing firm Oneupweb found that 23% of site operators attending who used dynamic content found the challenge of making that content visible to the major engine’s spiders “overwhelming,” while 15% reported being inundated with the maintenance requirements of such attempts. Part of the challenge facing sites using dynamic content is that content management systems used to manage data such as product information internally aren’t necessarily search engine-friendly.


Oneupweb’s recently-released BridgeWorx product is one of the latest to tackle the issue from a technology perspective. The product essentially bridges the gap between search engine spiders and dynamic content to guide spiders through pages of frequently-changing content while still allowing site operators to use their existing content management systems internally, the company says.

Oneupweb offers a way online marketers can get a quick initial read on whether their dynamic content is invisible. An advanced search that displays all the indexed pages of a site can be compared with the actual number of pages on the sties. A huge difference between the numbers or a large number of missing products indicates that search engine crawlers aren’t finding much of the site’s dynamic content.

Web sites with such “invisible” content miss customers and brand surfers because the dynamic content doesn’t make it into search engine indexes and therefore, search results, according to Lisa Wehr, Oneupweb’s president.

comments powered by Disqus




From The IR Blog


Anna Johansson / E-Commerce

Why is social proof big for niche brands?

A small online retailer that lacks brand recognition can get a big boost from high ...


Donn Davis / E-Commerce

Technology takeover: The fashion industry is next

We are now entering the third decade of the Amazon effect, and it is just ...

Research Guides