Fighting For The Loebner Prize

By Steve Spalding October 16th, 2008
Under: In The News

In 1950 Alan Turing proposed a test for artificial intelligence. The short version is that you put a person in the a room and have him communicate with either a computer or a human across a text-only channel. If that person can’t distinguish whether he is talking to the machine or the human, repeat the experiment a few times.

The machine wins if it can fool at least 30% of the interrogators.

The Loebner prize is a competition started in 1991 that aims to find a machine that can pass the test. What you see above is yet another attempt at the $100,000 gold medal.

As it stands, no one has managed to fool quite enough people to be crowned a winner. During this competition, a chatbot called L convinced 25% of the participants that it was a real, flesh and blood (bytes and bits) human.

Futurists, like Ray Kurzweil, say that we should have the problem cracked by 2029, but others also said it would be solved by the year 2000. More than being a question of processing power or talent, the lack of a winner could boil down to motivation. Much of the most rigorous AI research strays away from tackling Turing’s thought experiment because at the end of the day it doesn’t really prove anything.

It’s easier to test whether a particular “intelligent” system performs the task it was designed for rather than something this arbitrary and while it would be interesting to have a computer that could fool a human being, even if it passes the AI hasn’t proved that it is what we’d classically call intelligent (which involves something more rigorous than imitation).

That being said, let’s say you’re sitting around and think Turing might have just given you a quick way to win 100k. Before you get too excited, here are a few parameters –

The chat bot needs to be able to fool the human for 5 minutes.

The conversation is text only.

The human being knows that there is a chance that this is a computer program (many “Turing Tests” leave this part out).

So read Turing’s paper, plunk down some code, and spin the wheel. Who knows, you might just make the history books.

(No Ratings Yet)

Loading ...

Print This Post

If you liked that little story, here's what else you can do.

Check Out My Projects | Submit Your Own | Read My Book

Lance

Not to deride Turing in any way, but I think passing the test would amount to little more than a clever parlour trick. A chatbot might concievably fool a high enough number of people to pass the test, but what sort of creative problem solving abilities would it have?

When a computer can consistently match an average human score on a standard IQ test, then we'll have real artificial intelligence instead of a fun, but relatively useless word-game.

sbspalding

Agreed. I like the spirit of it but I think it falls short of proving
intelligence. I don't even think Turing -wanted- it to act as a proxy for
intelligence. If you look at the paper, it's approached as a philosophical
experiment as much as anything else.

Brianmce

That's the beauty of the test. If creative problem solving abilities allow you to distinguish human from computer, then that is a reasonable thing for the interrogator to talk about - ask the computer to solve a creative problem.

The Loebner is a very watered down version of the Turing test (short timespan, only 30% of people etc) - the real test involves an interrogator actively trying to distinguish who is what. Because questions are completely open, the qualities required to deal with such a conversation ultimately covers everything we think of as our mental faculties.

It would be much *easier* to write a computer that matches or surpasses a human on an IQ test than it would to pass the turing test. IQ tests are a fairly restricted domain of questions after all. I suspect that such a program could easily be written today.

sbspalding

Excellent point. I think most standard tests for cognitive ability could be
surpassed by a computer. I do think that the problem you'll find will be in
the arena of creative problem solving. No matter how "smart" a program is,
as it stands we don't have sufficiently advanced models for creativity (even
with genetic algorithms and other AI black magic) to create a system that
we'd view as overly intelligent.

Brianmce

The problem is that a lot of the time it can still fake it. *We* may solve a particular task by deep reasoning, clever ideas and creativity, only to be surpassed by a computer which merely brute forces the problem (eg Chess). Hence all tests like this are subject to the "but its not *really* thinking" complaint. This is why an open-ended test like Turing's is so important. Rather than trying to devise a test to act as a proxy for intelligence, just compare using exactly the same criteria we would use to judge people (excluding irrelevant giveaways like non being an organic humanoid lifeform etc). If they're truly indistinguishable, then I think it merits the intelligence claim.

The only problem with the test is that people often seem to mistake it for a direction, rather than a finish line. I don't believe we'll ever pass a properly conducted Turing test merely by constructing ever more elaborate Eliza clones any more than we could reach the moon by building a really *good* wooden ladder. You could build a hundred meter one and call it progress, but compared to the guy playing with rockets who's yet to get a centimetre off the ground you've accomplished nothing.

Events like the Loebner are thus ultimately a sideshow - building a better chatbot might fool some people, but it will never get you over the finish line - don't mistake it for progress.

sbspalding

Couldn't have said it better, thanks for the exchange mate. Do you work in AI or are you just interested?

Brianmce

No, I'm purely an interested observer. I think AI is one of the most fascinating and significant problems going, but unfortunately its also one of the most difficult ones. I don't hold much hope for it being solved in the near future, but it's interesting to speculate about.

amichail

Check out this Web 2.0 approach to chatbots: http://chatbotgame.com.

Just as Deep Blue brute-forced it in chess with speed, the idea behind the Chatbot Game is to brute-force it with a huge number of user-submitted Google-like chat rules.

sbspalding

That's actually really cool, thanks for sharing.

ITtradeOnline

I watched this video really careful. I found interesting the fact that some humans could not differentiate between computer and another human; and I think the test really showed that a computer could fool a lot of people (me included perhaps); as long as you do not know who is on the other side, could be really tricky!

blog comments powered by Disqus

On FriendFeed, this post was liked by 0 people and commented on 0 times hide

View this post on FriendFeed

Submit Your Idea Today!

Exploring The Intersections Of Technology and Society

Subscribe via RSS, Or select your favorite Reader:

Projects

Join The Network

Partners

Conduits and Filters - There is no Pulitzer prize for web content. I am not talking about an award with the same prestige ...

Mick Jagger And The Secret Of Freelance Pricing - “You can’t always get what you want. You can’t always get what you want, but if y...

Viralogy Experts - A few weeks I ago I sat down for an interview with Jun Loayza, one of the founders of Viralogy R...

Don’t You Wish You Could Do Everything? - But you’re an entrepreneur and you realize that focus is finite. Every single day you are tra...

On The Death Of PR - I spend a lot of time with the Internet. I live it, I breathe it, I bet there is an alternate reali...

On Disclosure - I thought I’d share my response to an interesting question raised by Ted Murphy of IZEA. Ted ...

Transmedia Storytelling - When developing any kind of story arc, whether for a piece of content or for a marketing campaign, i...

Robot Asks For Directions In Munich - This one came from New Scientist and it’s a pretty interesting experiment in human/robot comm...

How Telling Stories Helped The World Bank - This is the story of Steve Denning and how he used a simple story to help change the direction of th...

Deconstructing Berkshire Hathaway - It’s always interesting to look into the mind of the extraordinarily successful. This time up ...

The Future Of Flu Vaccines - The clever people at Georgia Tech are conducting research which might make your next trip to the doc...

The New Battlefield: Robots At War - A compelling and somewhat stunning piece by the Wilson Center on military robots and how they are ta...

How To Make Money On The Internet - Here are some “AHA” Moments that happened to cross my mind while I was on my daily bike...

Twitter And Magpie: Point And Counterpoint - I have been a user of the micro-blogging site Twitter for over 18 months and I love the social netw...

Purgatory - Today I had two experiences, each involving a blog. First, I read a post in a blog today that gave ...

The Fun Fair - How many bookmarks do you have? I have over 10 folders and each one holds an average of 100 bookmar...

True Believers - Your site is in Beta and you need users to test the site, so how do you get the audience into your ...

Look Out iTunes, Lala Is Here - OK, to start off, I am not an iTunes user (I don’t even own an iPod), however, I have used iTunes...

Use eChirpy to Tweet Blog from Your Email (Huh?) - What if you have something to say but don’t want to blog? What if you prefer Twitter but fin...

Dig Your Well Before You’re Thirsty - One of my favorite networking books is Harvey Mackay’s Dig Your Well Before You’re Thi...

Russians Most Engaged in Social Media - According to ComScore , Russia is the most socially engaged country, with folks spending an average ...

Real Estate Porn: I am Condo Curious - Pushing the print ad envelope. With the decline in print ads, I guess they figure no one is looking....

Building A Company With Social Media - via WedDesignerDepot (take the jump for more net porn) Further Reading: 50 Ways to Use Social Media...

Effective Meetings on a Budget - As online project manager you should be using virtual meetings. The first reason is that it’s...