When Giant Directories Roamed the Earth

Written by guest contributor Dave Jansik

Back in the day, directories were the definitive sources for finding quality content on the Web. Following the success of Yahoo, several companies and volunteer-driven efforts sprung up to get a piece of the action. The concept: we do all the searching ahead of time for our users, we establish proprietary traffic, and then the inevitable follow up: we charge businesses for inclusion in the directory. Brilliant. Simple.

Over the last six years, I have been involved in building one of the largest editorially and volunteer driven directory of Web sites. LookSmart’s founders recognized the need to “sort through the crap” that search engines were spewing out in the late 90s. At the time, their intentions were pure and good. The bubble was growing when Microsoft contracted LookSmart to create comprehensive directories of all the best sites out there. With all that money flowing in, we could afford to hire over 100 editors and ontologists worldwide. With the acquisition of Zeal.com we enabled a Web-based platform to harness the skills of hundreds of volunteers to add non-commercial sites and manage categories. Some were in it to get traffic for their own sites while others truly wanted to help build a great search experience. They searched for sites that were determined to be the authority on a topic. Descriptions and titles were hand-written one at a time. The platform enabled users to add numerous types of metadata, link sites and categories, perform mass edit functions, and easily find transgressions such as empty categories. Soon after, we began a paid inclusion service that played off the huge traffic numbers LookSmart received through its distribution on MSN. Thousands of companies were given the right to have as many product pages as they wanted placed in the directory. Suffice it to say that from a quality standpoint, directory search results were heavy on commercial and light on the quality we first intended. I was on the Ontology team whose task was to create and manage an ontology that accommodated what ended up being over 2 million commercial and non-commercial sites neatly organized into a highly intuitive 250,000 category ontology.

When Google entered the fray, directories came to be considered limited in scope. Many specific queries – the quick answers – couldn’t be answered by a directory. Google got you what you wanted in that crucial first five site returns. LookSmart responded by acquiring WiseNut and Grub.org to position it in the search arena. MSN recognized the need to compete with Google, dropped our contract in favor of building their own search engine from scratch, and we were sent scrambling to find a way to replace seventy percent of our revenue because MSN accounted for the bulk of our user traffic. Without Microsoft, resources to maintain our search products were pared. Editorial took a big hit, and with such a large directory on our hands the remaining staff and dwindling number of Zeal volunteers couldn’t keep up with the exponential growth of the Web.

It’s 2005 and most of the editors are gone. I have spent much time clearing out most of the commercial sites that once challenged our credibility. A handful of passionate and dedicated volunteers continue to shape and develop the directory – some oversee its vast landscape while others continue to focus on the topics they hold dear. It is their hobby and their community of friends. It’s their baby. Still, fly-by-night contributors and savvy SEOs endure the Zeal quiz solely to submit their sites knowing that being in LookSmart’s directory will boost Google PageRank.

So, what is the future of the directory? The web is a vast, vast place. Will search engines be able to deliver specific content out of the billions of pages they are able to index? Those of us on the directory bandwagon feel that contextual search and browse tools will always be an essential tool for researching the Web. Consider how useful it is for a searcher looking for everything on the paper and packaging industry to have neatly organized categories of guides, associations, major companies, professional journals, and equipment suppliers already found for him. Kids can search for information on mummies and then climb back up or across the directory to find facts on King Tut, the pyramids, and Cleopatra. People who aren’t sure where to go on vacation can look up official travel guides to hundreds of cities worldwide and see what kind of hotels, entertainment, activities, and restaurants are there before booking.

Here are a few other uses and features that can keep directories relevant, interesting, and even profitable:
– Search engine seeding. From a technical standpoint, it is widely acknowledged that search engines can use directories “seeded” with relevant URLs to improve relevancy.
– Marketing to a specific segment of Web users by creating verticals. The best of the Web has already been categorized – it’s a great place to start zeroing in on an audience and advertisers. Freely distribute the category structure to other verticals to build traffic.
– Making them more consumer-friendly for users and contributors. Add simple site rating and ranking tools that bring the best sites to the top of category site lists, incorporate message boards to share ideas and suggest other sites to fellow users, make it easy to submit a new site, offer basic search engine features such as preview panes and highlighted keywords.
– Address staleness issue. By including topical RSS feeds, news feeds, or links to community bookmarking services such as Furl. Once a directory is built it is fairly easy for a small staff to maintain and add new content.

It will be interesting for me to see if directories last as a useful research tool. Yahoo’s directory, once their crown jewel, now occupies a smidge of real estate towards the bottom of their homepage. There are other players popping up solely for the submission money, though just about all of them are so thin on content that one visit is plenty. Bookmarking services like del.icio.us and Furl are helping us dynamically learn about what people find interesting on the Web, yet there are issues with the free-wheeling and personalized metadata, or folksonomies, that users of these services employ. No one has invented the perfect, omniscient search tool yet. And ya know, I wouldn’t be surprised if the person who creates one gets their idea while in a directory.

***
Dave Jansik is a Senior Ontologist and Content Producer at LookSmart.com. He has worked on Web directories for 6 years.
[Editor’s note: I’d like to thank my friend Dave for taking the time to put this commentary together. He has been a quiet advocate for helping people find information on the Web for years and it’s great to have him sharing his thoughts with us.]