AI Zone Admin Forum Add your forum

NEWS: Chatbots.org survey on 3000 US and UK consumers shows it is time for chatbot integration in customer service!read more..

Tokenize change in 6.2c
 
 

I am doing a lot of date parsing in my bot and having just updated to 6.2c, I have noticed a difference in the tokenization process.

In 6.2, :prepare monday 1st april would tokenize into

monday  1st  april 

which would then get date merged into

Monday  April_1 

But with 6.2c, the tokenizer produces

monday  1  st  april 

and this is name merged (not date merged) into

monday  1  St_april 

with the consequence that I’ll have to adjust several rules.

I can do that, but if this is an undesired side-effect of the changes in FindEndWord, then I’ll hold off upgrading.

 

 

 
  [ # 1 ]

sounds undesired. let me fix

 

 
  [ # 2 ]

Thank you.

 

 
  login or register to react