This piece was written by guest author Nitin Karandikar, who writes Software Abstractions. It is the fifth guest installment in the Rising Star Dream Team Future of Search series.

I’ve been writing about Search technologies for a while now, so when Kaila Colbin of VortexDNA offered me the chance to participate in a future-of-search marathon, I jumped at the chance!

What will the search for information look like in the future - in five years, ten, twenty? Is it just more of the same, or will it look radically different?

Looking Back

Before looking to the future, let us first look at how far we have come. Danny Sullivan has a great post looking at a decade of search history and the various tribulations of past and present search engines - AltaVista, Ask Jeeves, Microsoft, Yahoo! and of course, the early Google. We owe a huge debt of gratitude to the tremendous contributions of these and other early pioneers of Search; Google, in particular, deserves a great deal of the credit for making web search ubiquitous outside the tech community. Indeed, “to Google” as a verb has become virtually synonymous with the idea of Web Search, much as the Xerox brand became synonymous with the idea of the photocopier in a bygone era.

Google’s venerable PageRank algorithm is certainly best-of-breed for the present, and Google keeps tweaking its results continuously. Given this progress in the Search area, can we still expect to see major improvements in search in the forseeable future? As an analogy, consider the DC-3 airplane - the first truly modern airliner, it was powerful, safe, reliable and economical (indeed, some of these are still flying today). It revolutionized air travel, and with its introduction, many considered the aviation age to have arrived for the general public. And yet, early jet aircraft had already appeared on the horizon, so to speak; within a decade, this reliable workhorse was obsolete, overtaken by jet aircraft in the competition for public air travel.

It could easily be the same with search. The key question that a search engine addresses is: what results do the maximum number of users find most useful for a given search query? PageRank is simply an approximation of the Wisdom of Crowds to answer this question. Is there a richer abstraction? Is Engagement the new black? Whatever the new approach is - in order to provide accurate results, it must work as implicitly as possible.

We have only to envision the possibilities …

Looking Forward

So let us take a speculative look at search, circa 2015 . To look at it systematically, we can separate the search engine into the following components from a user perspective:   (To see this breakdown in visual format, check out my earlier post on an abstract architecture for search )

• Query specification
• Base Index
• Relevance Algorithm
• Results Visualization
• Ongoing Interest

Let us consider the possible future directions for each one in turn.

Future Directions

Query Specification
(Input)

Google pioneered the keyword-centric, minimalist approach for specifying the search query, and all the major search engines follow that lead. But the search criteria could be so much richer ; instead of experimenting with different types of keyword searches to find the information they need, users could simply provide additional criteria up front to qualify their request.

Admittedly, this approach does not work for everyone. The casual user would get reasonable defaults, which would automatically get updated with regular use to their favorite values; the topical researcher, on the other hand, would actively tinker with these widgets in a “power user” mode. (Google already supports this type of functionality in a limited fashion.)

Some possible advanced features for specifying the query are given below:

1. Content Spec:  Enabling the user to dynamically specify the data sources to be included, based on domain, reputation, social network, and so on
   
2. Scope:  Input for seamlessly limiting the scope of the search, to Enterprise or personal data
3. Qualifiers:  Allow the user to add more information to disambiguate result matches, e.g. qualifying if “Java” means the programming language or the island
4. Parameter ranges:  Domain-specific parameters can be extremely valuable even to a general-purpose engine (see #5 in the section on Relevance Algorithms below)
5. UI paradigms:  Text keywords are a limited form of input. The actual input mechanisms could be more visual, in the form of sliders, buttons, fields and other UI widgets. Imagine, for example, that as you move a slider, the search results change or an increasing number of results appear on the page!
6. Multiple Profiles:  Personalization does not always have to be implicit. A user could explicitly set up profiles to represent different interests - professional, hobby, personal and so on, so that switching the profile would quickly change the areas of interest

Base Index
(Content)

This is a core area of concern for search engines: what is the scope of content to be considered when searching for information?

The standard approach currently is to build web crawlers that continuously scan as many web sites and web pages as possible; the scanned content is then used to build a master content index that is then updated regularly. This index is then used as the basis when searching for information.

For the base index, the big changes in the future are likely to involve both the scope and understanding of the content; here is a short list:

1. Rich media search, e.g. true indexing of audio and video content
2. Dynamic content search (searching the invisible web )
3. Integration of personal, web and corporate information
4. Perspective-based search, e.g. conservative vs liberal, hard news vs opinion, and so on
5. Subset creation, on-the-fly, e.g. to search for domain-specific data

Relevance Algorithm
(Mechanism)

This is, of course, the most-debated topic when discussing the future of search engines. Clearly, many different approaches and technologies show promise; some of these are noted below:

1. Personalization (but without storing personal info )
2. Social Input / Wisdom-of-Crowds (which has its pitfalls )
3. Social Graph: where your selected network of people help improve search results (Robert Scoble has recently gotten religion about this concept; Danny Sullivan rebuts)
4. Semantic Processing: of both, the query AND the content   (will this let the Search Engine find answers that we never knew we had?)

5. Parametric Search:  Vertical search engines already routinely offer domain-specific parametric search; for example, job search engines allow the user to specify the all-important location of the job as a primary criterion. Can this type of feature be generalized, so that as a user drills down deeper into search results, an increasing number of parameters can be offered?

6. Human-powered Search, for either the short head or the long tail of search
7. Swarm Intelligence: Mimicking biological search, such as Ant colony optimization, particle swarm optimization, and so on

Results Visualization
(Output)

Again, Google leads the way with its minimalist approach: simple headings, links and snippets of text. This is slowly changing, with the new “Universal Search” approach from Microsoft, Google and others; Ask.com is a leader in this area.

Search engines of the future will likely implement completely new paradigms for users to navigate and view search results. Often, meta-results - representing information about the results - are as important as the results themselves: users can figure out where a given result fits into the overall universe of results, and find the related results to an item of interest that has been found.

Some possibilities for results display are given below:

1. Tag clusters is not a new concept, but has yet to gain traction among the majors. Quintura, with its dynamic tag cloud display, has one of the best examples.
2. Organize results information by content type, is something every search engine will have to think about in the future. For example, should news stories be presented in an “overview capsule” fashion, or organized as a timeline-based view? Dale Dougherty at O’Reilly Radar has a brilliant article on this topic: Journalism is burning.
3. Follow-up actions - on viewing search engine results, a very common user action (as Greg Linden points out ) is to modify the current query, either to drill-down further or to try a different approach to find the required information. Google’s “did you mean …” feature is a step in this direction (although it leaves much to be desired).

4. Domain-specific visualization can significantly enhance the understanding of results. This is similar to the data organization point above, but focused on the display itself; results from different vertical domains may require very different visualization techniques, such as colors, graphs, images, trend lines, heat maps, topographic charts, and so on. [For a list of the more exotic variations, check out this amazing list from Smashing Magazine.]

5. Dynamic scoping - enabling users to widen or narrow search results, based on different criteria - such as geography (local or global), site authority, timeliness, point of view, domain, and so on - is a powerful feature, that will continue to grow in importance.

Ongoing Interest
(Notification)

This can be best explained as a Reverse search, where it is the content that finds the user - thus turning the concept of search on its head.

Most of us have ongoing interests in certain areas; they could be professional, social or personal. It makes a great deal of sense for the search engine to keep track of these interests and pro-actively notify the user at some periodic interval of new items that fit those interests. Google Alerts is an early example in this direction. But enhancements to its functionality in the future could significantly boost its utility.

Some day, search engine notifications could support the following features:

1. Diverse Mediums:  Many search engines already support email notifications. What’s to stop them from adding support for many additional delivery mechanisms, such as IM, SMS, widgets, the twitter API, and so on?

2. Levels of Detail:  Allowing users to set the scope and organization of information presented.

3. Prioritization:  This is a key feature! Once users are able to set priorities for different types of searches and for different areas, this can be used to drive the other features. For example, send me the headline about a breaking news event directly relevant to my blog, as an instant message, but email me a digest of the day’s results for baseball scores.
4. Schedules:  Some search results make sense only at certain times of the day; e.g. traffic search results are only relevant at commute times on work days.
5. Dynamic Control:  Finally, empowering users to assert dynamic, granular control over their search alerts would make this functionality truly powerful. For example, once I’ve been notified about a breaking news story, I might want to artificially boost its priority and delivery method to continually get updates quickly and efficiently.

Power and Responsibility

As search engines start including a few or many of the features described above, search will grow increasingly more powerful. It will get easier to find any information we want, quickly and easily. Whether the information is high-level or detailed, global or local, general or specific, past or present, in any domain - no nugget of human knowledge shall escape this relentless spotlight.

Is shining a light on the darkest corners of the web always a good thing? As a webbed superhero once told me (and a few billion others) - “With great power comes great responsibility!”. Privacy advocates are rightly concerned with the growing power of global web search engines; ongoing efforts from official and community channels are essential in minimizing abuse. A related issue is that web content can be archived and searched in perpetuity - the societal effects of this phenomenon have not yet begun to be understood.  A recent New York Times column highlighted this issue (paid content; here’s a perspective on it from Slate magazine ).

Conclusion

Clearly, search engines will continue to evolve, and a future engine might well have many of the improvements described above, in the next ten years. But how about even further out - say, 2020 or 2030? Will disruptive changes in networking, computing and information technologies radically change the way search engines operate? A change in the nature of human thinking, interaction and social customs would be even more dramatic, and could cause a change in the nature of search itself.

This is, of course, a fertile area of speculation more in the realm of Science Fiction (for now): for example, will we one day need a galactic search engine? Can we create microscopic information-matching agents, either biological or atomic? Results that suddenly become available to the user as knowledge in the brain? An “implicit” search engine that finds information as we need it? Why not?

This entry was posted on Friday, September 7th, 2007 at 7:32 pm and is filed under Search. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.