AI Zone Admin Forum Add your forum

Chatbot Conference London On Oct 3th Erwin will be moderating on the Chatbots conference in London. Why don't you meet him there? As a Chatbots.org visitor, you'll get a discount as well: Why don't we meet there? get a discount

We do experience some trouble with our forms and some details page. We'll fix this ASAP. Sorry for the inconvenience!
Important Announcements RE: 2017. (inc. dates, and protocol)
 
 

The organising committee of the AISB Loebner Prize met to discuss issues arising around this year’s competition and have made the following decisions.

- The deadline for submission for the 2017 competition has been pushed back to Monday 24 July 2017.

- The date of announcement of finalists has been pushed back to Tuesday 15 August 2017.

- For this year, submissions using either protocol will be accepted. Differences in the protocols, such as the old protocol being character-based and the new protocol being line-based, do not effect the finalist selection procedure, so neither protocol has an advantage over the other.

- Finalists will be offered help on making their entry compatible with the new protocol, for the final.

- Entrants are advised to either, attend, or to have a representative attend, such that they may address any technical issues arising with their entries. In future years this may be a requirement.

- Discretionary funding for travel to the final at Bletchley Park is available for finalist, or their representatives. Please send requests for funding to .(JavaScript must be enabled to view this email address).

The organising committee would like to thank the AISB Loebner Prize community for their engagement and helpful feedback.

 

 
  [ # 1 ]

Hi Andrew,

Good to see the old protocol being accepted for entries. Can I advise updating the AISB entry page with the new dates and details: http://aisb.org.uk/events/loebner-prize as not all entrants are members of chatbots.org

I assume from your post that the final will be using the new protocol only. The finest minds we have at chatbots.org have struggled with getting the new protocol to work. What would happen if an entry that gets through to the final 4 works with the old protocol but cannot be made to work with the new protocol for finals day?

 

 
  [ # 2 ]

Steve,

I’ve sent the announcement to the aisb.org.uk webmaster, but chatbots.org updates a little quicker smile

Unofficially, I’m not sure if we’re utterly fixed on all of the 2017 finalists having to be on the new protocol by the final. I’d say our priority is on making sure that all submissions are checked fairly and all finalists are judged fairly. This could lead to all sorts of outcomes, maybe every finalist uses the new protocol, maybe none of them do, maybe the ones that have the old protocol are compatible with a “bridge” system.

Anything we commit to now might be forced by circumstance, but our priority is to keep the competition running with a happy community.

 

 
  [ # 3 ]

Great news Andrew. I’ll get to work on my entry!

 

 
  [ # 4 ]

As far as I’m concerned I wouldn’t even mind if the judges interacted directly with my program if for whichever reason a protocol fails to work. To quote Hugh Loebner from “Parsing the Turing Test”:

In the years that I directed the Loebner Prize Contest, the winning entries were clearly not humans. The task was not to distinguish which was the human and which was the program, but rather which program was the ‘most human’ even if it clearly was not a human. I am confident that with or without the presence of human confederates in the Loebner Prize Competition, the same programs would have won.

 

 
  [ # 5 ]

Andrew,
When can we expect a final version of the New LPP server software to be available so we can test against it?

As an aside, I did create an “Auto Judge”, which reads a text file and automatically asks questions if you are interested.

 

 
  [ # 6 ]

Merlin,

Last time I spoke to the protocol developer he said he’d made some changes recently. If there are outstanding issues, then you should contact them directly. I’ll PM you their email, to help avoid it being published on the public web, but any other people reading this can ask me for it.

Auto Judge sounds cool. Tell me more smile

Cheers,

Andrew

 

 
  [ # 7 ]

Thanks Andrew,
Last I checked the LPP up on github was about 2 months old (and I thought it just included Dave’s updates).
It would be good to get the latest and greatest so we could use and test against it.

I built “Auto Judge” to test my bot. It is easier to have a bunch of test questions loaded and automatically input than it is to type them in.

I know in the qualifier round it is 1 bot 1 judge and there have been concerns about inconsistency in typed input. “Auto Judge” eliminates the possibility of typing errors.

It can run with line by line input or in full auto mode where it will ask a new question after each response.

Image Attachments
auto_judge.JPG
 

 
  [ # 8 ]

Looks like the server and html files were updated for bugs 28 days ago (can’t say I noticed): https://github.com/jhudsy/LoebnerPrizeProtocol

That auto-judge looks interesting, and I can attest to typos in previous years so I’d certainly recommend automating the procedure. Mind that some bots sometimes fail to respond at all, so I’d put the feed at intervals of 30 seconds or so. (My own AI sometimes spends 6 seconds thinking when a question requires introspection. A human may or may not be faster)
Also, my AI sometimes answers with several consecutive responses followed by newlines, as do humans, which could clash if a full-auto feed asks a new question immediately after the first response.

 

 
  [ # 9 ]

I may have missed the last update, so I will download and test that version this week. But, if there is a newer version I would rather be testing on the latest.

Auto-judge has 2 modes.
Manual= sends an input line each time you click the auto button.
Full Auto= Instead of polling with a certain time frame, it waits for a response before it sends the next input. As long as the newlines come as part of the message i think it would handle it. If there are multiple messages, you would have to use manual mode.

You can toggle between the 2 modes and can still manually input a line if need be.

The other thing I built was “echo bot”, which did nothing more than send back the input to test network connections/configuration.

 

 
  [ # 10 ]

I mean multiple messages, so manual mode would certainly be preferable. I recall Steve remarking that no bot would do this, so I’ll clarify with some examples where confederates, and thus bots as well, could reasonably send multiple messages:

judge: Count to 3.
1
2
3

judge: Count to five, then say hello
*5 seconds pass*
confederate: hello

judge: Calculate 123 x 456
confederate: Okay, hang on while I get out my calculator.
confederate: 124132

confederate: I am a humon
confederate: *human

confederate: Have you ever danced wi (confederate accidentally hits the return key)
confederate: - with the devil in the pale moonlight?

Echo bot sounds neat, but would also interfere with my program’s learning ability were it to be used. I’ll be asking the testers to use greetings and questions only to test the network setup, and hopefully they’ll follow that instruction as well as they did last year.

 

 
  [ # 11 ]

Echo is really only useful for checking the network configuration.

Auto-judge is good for testing and only for the qualifying round.
Any delays and multiple messages are lost in that round because the scoring is done on the transcript.


Issues with the current version:

No indication of Ip address
Remote network clients must use IP of the server to connect.

Going past round 3 crashes the server
If server crashes, how does contest continue?

If judge sends a message before the round starts, server crashes

 

 
  [ # 12 ]

Andrew said, “neither protocol has an advantage over the other.”  However, this decision by the organising committee of the AISB Loebner Prize gives an enormous advantage to the old protocol contestants over the new protocol contestants. 

New protocol contestants deserve a special mention as recognition by the AISB.  The time, effort and skill required to successfully implement support for the new protocol was substantial.

 

 
  [ # 13 ]

Thanks Andrew

Good news that the old LPP is back in the game.
I hope this means there will be a full compliment of chatbots in this years contest.
Bang goes my chance of getting into the final!
I would settle for improving my score from previous contests.
We need to fit the test to the chatbots, not fit the chatbots to the test.
Good luck to all.

 

 
  [ # 14 ]

Andrew said, “Finalists will be offered help on making their entry compatible with the new protocol,

In other words, the old protocol contestants will be offered help beating the new protocol contestants.

My comments express disappointment as a new protocol contestant, but it a calm, sensible manner.

 

 
  [ # 15 ]

Let me say that again, “but in a calm, sensible manner.” to emphasize my intentions to the AISB, (and to to correct my “it” typo above).  I have no intention of jeopardizing the contest like the old protocol contestants did.

 

 

 1 2 3 > 
1 of 3
 
  login or register to react
‹‹ New LPP on Linux      AISB FAQ ››