May 10, 2004, 12:00 AM

How helping spiders reveals the web’s dynamic content to search engines

Frequently changing content is more difficult for search engines to index. In fact, 23% of attendees at a Oneupweb web seminar called the challenge of helping spiders find dynamic content “overwhelming” -- but a new product bridges the gap.

Dynamic – frequently changing – content constitutes much of the “invisible web,” so-called because it represents material less easily found and indexed by search engine crawlers. Yet as web sites’ use of dynamic content becomes more prevalent, the problems it can pose relative to search engine marketing grow. A recent web seminar by search engine marketing firm Oneupweb found that 23% of site operators attending who used dynamic content found the challenge of making that content visible to the major engine’s spiders “overwhelming,” while 15% reported being inundated with the maintenance requirements of such attempts. Part of the challenge facing sites using dynamic content is that content management systems used to manage data such as product information internally aren’t necessarily search engine-friendly.


Oneupweb’s recently-released BridgeWorx product is one of the latest to tackle the issue from a technology perspective. The product essentially bridges the gap between search engine spiders and dynamic content to guide spiders through pages of frequently-changing content while still allowing site operators to use their existing content management systems internally, the company says.

Oneupweb offers a way online marketers can get a quick initial read on whether their dynamic content is invisible. An advanced search that displays all the indexed pages of a site can be compared with the actual number of pages on the sties. A huge difference between the numbers or a large number of missing products indicates that search engine crawlers aren’t finding much of the site’s dynamic content.

Web sites with such “invisible” content miss customers and brand surfers because the dynamic content doesn’t make it into search engine indexes and therefore, search results, according to Lisa Wehr, Oneupweb’s president.


Sign In to Make a Comment

Comments are moderated by Internet Retailer and can be removed.

Not a member? Signup for free today!




Relevant Commentary


Jason Squardo / Mobile Commerce

Five tips for achieving high mobile search rankings

Searches on mobile devices will soon exceed those on computers, Google says. Retailers that keep ...


Sergio Pereira / B2B E-Commerce

Quill turns to its B2B customers for new ideas

Coming in April is a new section of that will let customers and Quill ...