All NewsEducationTV
Equities & FundsCrypto & Digital AssetsAI & TechnologyBusiness & CorporateUS Politics & PolicyGeopolitics & Global RiskMacro, Rates & FXCommodities & EnergyEuropean Politics & MarketsAsia-PacificReal Estate & Property
Story archiveAll categories
← All Stories

Remembering the pre-Google web, when search was an experiment

Created at 7 Aug · 11:06 AM1 source↑ Market-relevant
IN SHORT

Before Google's dominance, early search engines like AltaVista, Lycos, and Ask Jeeves attempted to organize the chaotic early web. This era relied heavily on directories, manual curation, and a sense of exploration rather than a single, dominant search interface.

✉Newsletter

PiQ Daily

Pick your topics. Get only what matters, on your cadence.

Who's Involved

Nicole M. Radziwill
Recalls Usenet and webrings from her experience as a systems administrator, programmer, and project manager in the '90s.
Mark Friend
Director of IT support firm Classroom365 Limited, who worked as a systems operator in the late '90s.
Remembering the pre-Google web, when search was an experiment

↳ Why This Matters

Understanding the evolution of search engines provides context for the current digital information landscape and highlights how user behavior and technology have shaped our interaction with the internet.

Key facts

  • In the mid-1990s, the internet was chaotic, and finding information relied on proto-search engines and directories.
  • Early search engines like AltaVista, Lycos, Excite, HotBot, and Ask Jeeves attempted to organize the burgeoning web.
  • Yahoo initially functioned as a human-edited directory, categorizing websites.
  • Webrings and Usenet were important tools for discovering content and forming online communities before Google's rise.
  • Crawlers and indexing software became crucial as the web's scale outpaced human curation.

In the mid-1990s, the internet was a vastly different landscape, characterized by a sense of exploration and a struggle to find valuable content. Unlike today's experience dominated by Google, early search engines such as AltaVista, Lycos, Excite, HotBot, and Ask Jeeves offered their own unique approaches to navigating the web's growing chaos.

The pre-Google internet did not assume universal searchability. Users frequently discovered pages through human-curated directories like Yahoo, personal bookmarks, newsgroups, and email signatures. This era, described as 'Wild West-style discoverability,' saw search engines as just one component of a larger ecosystem blending curation and chance.

Resources like Usenet, established in 1979, played a significant role in organizing and discovering information. Usenet provided topic-based newsgroups and fostered a culture of distributed discussion, making online information feel social and communal. Nicole M. Radziwill, who worked in IT during the '90s, recalls Usenet as a key portal for unearthing interesting content and discovering websites through links shared in post footers.

Webrings later emerged as another artifact of this era, allowing microcommunities to form around shared niche interests. These were often managed by individuals who curated lists of related sites, serving as an alternative to monolithic aggregators like Reddit and providing a way to find topic-focused websites.

Yahoo exemplified the early logic of human-edited guides, organized into nested subject categories. This approach was effective for the young web, enabling human editors to identify quality sites and impose order. Radziwill recounted submitting websites to Yahoo, likening it to the Yellow Pages for websites, and the rigorous process of justifying a site's legitimacy and importance to human moderators.

However, human curation faced scalability issues as the web expanded. The sheer volume of content outpaced the capacity for manual classification, paving the way for crawlers and automated ranking systems. Engines like AltaVista, Lycos, Excite, and HotBot adopted software-based approaches, using crawlers and indexes to map the rapidly growing web.

AltaVista, in particular, represented a significant advancement with its fast crawler and scalable indexing software. It handled millions of HTTP requests daily, making search feel more like querying a machine than browsing a catalog. Its speed, breadth, and capabilities made it a defining search engine of the 1990s, offering users a sense that the web could be approached as an indexable whole, even if results were imperfect.

Frequently asked questions

Some of the prominent search engines before Google included AltaVista, Lycos, Excite, HotBot, and Ask Jeeves.

People found information through human-edited directories like Yahoo, webrings, Usenet newsgroups, bookmarks, and links shared between websites.

Yahoo began as a human-edited guide to the web, organized into subject categories, helping users find quality sites and imposing order on the growing internet.

Human curation became unsustainable as the number of web pages grew exponentially, outstripping the capacity of human editors to classify them in a timely and cost-effective manner.

Get the newsletter.

Pick the topics you actually care about. We'll email when there's news worth your time, on the cadence you choose. Cancel any time from your account.

Cadence

How It Developed

The mid-1990s web was chaotic, with users finding content through directories, bookmarks, and links.
Early search engines like AltaVista, Lycos, and Ask Jeeves aimed to organize the web.
Yahoo began as a human-edited directory, organizing the web into subject categories.
Webrings emerged as a way for microcommunities to share niche interests and increase exposure.
Search engines like AltaVista, Lycos, and HotBot used crawlers and indexes to map the web.
AltaVista became a defining search engine due to its speed, breadth, and capabilities.
Searching the web in the late '90s was considered both a technical skill and an art form.

Sources

T1
Remembering the pre-Google web, when search was an experimentvar abtest_2163326 = new ABTest(2163326, 'impression');Ars Technica

Related Stories

Cloudflare open-sources AI agent platform for non-coders
6 Aug · 4:21 PM
Jeff Dean's new AI startup, Discovery Loop, secures funding from Khosla Ventures, Radical Ventures
6 Aug · 2:30 PM
Anthropic to Build In-House Silicon Team
6 Aug · 8:11 PM
AI Title Searches Miss Key Issues in 40.8% of Files, Report Finds
6 Aug · 7:46 PM
AI models design novel bacterial viruses
6 Aug · 7:11 PM