AI Zone Admin Forum Add your forum

NEWS: Chatbots.org survey on 3000 US and UK consumers shows it is time for chatbot integration in customer service!read more..

How would you detect a chatbot?
 
 

To the experts here who compete to create chatbots that can act like people, let me ask: what would you do to detect a chatbot? How would you avoid being fooled?

Are the Russian political “bots” detectable? How would you police Twitter?

Is there a single kind of question that would trap all but the most sophisticated bots?

 

 
  [ # 1 ]

The main thing to realise is that there is a great diversity of bots, but each is limited in domain, so your best tactic would be to try very varied questions.

The average chatbot or spambot won’t be able to handle deliberately abusive misspelling, but a commercial chatbot will, because they can afford the resources involved in spellchecking. This is still the most effective tactic though.

Second-best is asking word games, like “How many vowels in the xth word of your last response?”. Because there is no reasonable purpose for word games and there are many variations possible, few bots are made to answer them.

However, with chatbots I generally just look at how often they return generic all-purpose replies like “yes, I do” (do what?), or if they keep referring to what I said as “it” for several bouts without topic-relevant details. A lot of bots don’t handle pronouns because that requires resource-heavy parsing.

As for troll bots, whenever I found insinuative vile spewing posts they typically turned out to be from real Americans. Bots’ subject matter seemed more focused on controverse than provocation. The following features are common:
- Outlandishly foreign-seeming usernames with awkward and long combinations of syllables (generated)
- Repetitive subjects and frequent posting history (only equalled by some obsessive autistics)
- Lack of interactive activities (no replies, no following, followers, interests or subscriptions)
- Bad grammar (similar to Indian and Russian, with verbs and prepositions out of order)
- Bare-bones profile information is a bonus
For the time being, troll bots are best detectable through meta-data and posting history. Since a bot’s purpose is to spam, it should post as often as it can, without for instance the inescapable 8-hour period of inactivity known as sleep. I heard there’s a famous Russian bot in America that starts anger-Tweeting at 3AM in the morning every day. That’s definitely not human.

 

 
  [ # 2 ]

A wonderful list. Thanks.

I was thinking not too many bots can answer simple questions that combine generalization, arithmetic, and pronouns:
“I have 3 apples and Jon has 4 oranges: how many pieces of fruit does he have?”

 

 
  [ # 3 ]

You’ll also find some interesting suggestions here: https://www.chatbots.org/ai_zone/viewthread/1776/

I’m afraid the questions that one intuitively considers difficult, often turn out to be easily sidestepped. Mostly because everybody else also thinks they are good questions, so chatbots will have encountered them before. Fruit arithmetic has been so regular in the Loebner Prize that one can do with recognising two numbers and “how many” and ignore everything else for a pretty good guess. Of course then you think, “What if I ask how many elephants?” and you’d probably find that that also occurred to the chatbot creators.
But, I should mention that this is only true for chatbots and most particularly Turing Test participants. It is highly unlikely that spambots or commercial bots would be equipped for that line of questioning, because it is not relevant to their domain.

 

 
  [ # 4 ]

Here’s a relevant image of bots at work. Again, it’s the meta-data like time that’s a clearer sign than the things they say.

 

 
  [ # 5 ]

Don:
Why doesn’t someone come up with a product concept for Twitter/Facebook users, where they can do a quick meta tag analysis? Or why doesn’t some company (duh: Twitter or Facebook) incorporate these simple truths you have been telling about and flag or remove the obvious bots?

 

 
  [ # 6 ]

That is the right question.
Twitter and Facebook have only begun tackling the problem this year, and only due to pressure from the public and US congress, who have been looking for someone else to blame for their recent voting behaviour.

According to e.g. this article, the executives of these Social media are “freedom of speech” fanatics, the American way. The last thing they want is apply any sort of censorship. Policing is not in their interests. It would require an ongoing research branch like anti-virus measures, and they’re already not making a profit. But mostly they’re ideologically pigheaded.

Independent researchers are showing a lot more interest. Last year there was a guy that released a browser add-on that automatically traced article links back to unreliable sources, shortly after Facebook claimed they couldn’t do anything about disinformation.

 

 
  login or register to react