Search Agents

I like the idea of using the weekends, which are usually a time for the most boring of “weekly roundup” type posts to do some idle speculation on the future of the web. Today, I am going to take you ten years forward, where we can look at a few technologies that will help ring in Web 3.0.

Web 3.0 In A Nutshell

As I said last week, Web 3.0 is can loosely be defined as a set of:
Highly specialized information silos, moderated by a cult of personality, validated by the community, and put into context with the inclusion of meta-data through widgets.

To break it down even further, this new web will turn the internet into a giant database and our place in it will be to organize this well-spring of information into slices that are palatable to us.

How we do this will be through widgets and ever more powerful data management technologies. Most of which are here today, in one form or another.

RSS. A Web 3.0 Driver

In ten years RSS and its related technologies will be seen as the single most important internet technology since Tim Berners-Lee and Robert Cailliau created the World Wide Web at CERN around 17 years ago.

Real Simple Syndication is crucial to the development of the new web because it’s just that, really simple. Anyone with a Wordpress account or a tiny bit of coding knowledge can generate an extensible, standards based database of information that can be transferred to almost any other modern web site.

If Web 3.0 is the Semantic Web, where computer agents read content like human beings do — then RSS will be its eyes (or at least its corrective lenses). Already, entire business models are being created around aggregating meta-data. Netvibes allows you to create your own personal homepage, drawing much of its content from RSS feeds that you select. iGoogle does the exact same thing, and a host of others are jumping on the concept that the easiest way to give users relevant content is to give them the ability to define relevance for themselves.

In this future, RSS will be extended to include a host of data-points it currently does not. Each blog post (or microblogging feed), every picture, every video clip will have searchable, taggable, XML based syndication around it.

But wait, there is more.

People Search

The web as a database means that your online persona is apt to become an entry in it. If you look at technologies like FOAF you will see what I mean. FOAF is a project founded by Libby Miller and Dan Brickley and you can think of it as RSS but for Social Networks. It takes common profile data and puts it into a form that makes it cross-compatible with other social networks. Once Search Engines are properly able to manage meta-data like RSS, FOAF and the half-dozen other protocols out there and present it more intuitively the concept of a truly universal internet is well without our grasp.

Defining Context

Finally, RSS enables users to define their own contexts for information. Imagine a word where creating a mashup between Google maps and your Twitter account was no more difficult than sticking a few widgets together. This type of widgetizing of the web is not too far off, already Yahoo has a mashup creator — Yahoo Pipes that lets you do just this. Web 3.0’s real power will be in the ability to create data and transfer it effectively, even now we are well on our way.

Related Products: iGoogle, Netvibes, Yahoo Pipes

Software Agents And Expert Systems

Human beings are intrinsically lazy creatures. That might not sit well with you, but intuitively you know its true. OK, fine, for the sake of discussion lets exchange the word lazy for efficient. Feel better now?

Now for a few definitions to seed our discussion:

Expert System: An expert system, also known as a knowledge based system, is a computer program that contains some of the subject-specific knowledge, and contains the knowledge and analytical skills of one or more human experts.

Software Agent: In computer science, a software agent is a piece of software that acts for a user or other program in a relationship of agency. Such “action on behalf of” implies the authority to decide when (and if) action is appropriate. The idea is that agents are not strictly invoked for a task, but activate themselves.

Software Agents and Expert systems will be our off line access point to Web 3.0.

Expert Systems

If you have ever had a sniffle and gone to WebMD for advice, then you understand what an Expert System is. The short version is that it is a software agent that takes user input, runs it through a knowledge database and then spits out an output using fancy technologies like neural nets (which since this is not a hard science blog, is well beyond the scope of this post).

Ten years from now, Expert Systems won’t just be designed for general cases, but will be able to be easily generated to understand individuals tastes. Already we see contextual advertising and contextual search, but what if you could extend this concepts to a web browser, or to your mobile phone. Imagine a world where your computer would generate a profile, a meme map about you based on your interactions with the web and refine your experience based on this map.

If you used a search engine, your results would be weighted based not only on the standard Web 3.0 metrics, but also on “what you care about” as defined by all your previous interactions with this particular search engine and all of this would be completely transparent.

It is a world defined not by the strength of a arbitrary search algorithm, but one of mass personalization where every search that you make and every result that you decide to follow up on means that your next search will be more and more personalized. You push all of this data into your FOAF, and you really have something.

Related Products: Google Search History, WebMD, Contextual Advertising

Software Agents

Programs that surf the web for you will become more and more powerful. In a world where your personal profile containing your likes, dislikes and search history is as easy to upload as it is to add a feed to your RSS reader, it is no surprise that a major industry will be software that does this searching for you.

Imagine a scenario where you want to find a new camera. Since your personal meme map containing a listing of all the cameras you have ever searched for as ordered by frequency is available, you can set your software agent to continue this search for you in your absence. When you return home you would be presented with a list of sites ordered by price, relevance (to you) and features that have been found based on your preference. What you do with this list is fed back into the system, improving future searches.

Related Projects: MIT Media Lab

The Privacy Caveat

Reputation management, Meme management and Data privacy will be the major issues of the day. When you have a world where everything that you do is being written into an RSS feed (in one form or another), the ability to protect this feed will be crucial. New industries that are currently being developed will be expanded on. Professional and Semi-Professional netizens will hire SEO experts to ensure that their reputations are being properly managed.

Where once there was only an industry for corporate level intelligence and brand protection, bloggers with a vested interest in how they are perceived online (the Robert Scobles and Mike Arringtons of the world) will join into the mix.

Also, lets not forget the improvement in privacy features. The ability to block certain actions from being indexed, or limit the access to your profile by third party sources will be the next big push in internet security and privacy.

Related Products: RepuTrace

Summary

  • Information will be pushed into feed services like FOAF and RSS transparently.
  • Information will be aggregated by Search Engines and indexed in a human readable format.
  • Expert systems will be able to act as a proxy for you online.
  • Managed search through software agents will become cheap and available.
  • Privacy protection technologies will come to the forefront.

Related

How To Define Web 3.0