AI Zone Admin Forum Add your forum

Chatbot Conference in the Netherlands on Tue 24, 2018, with international line-up! Check it out

Chinese opening and closing double quotes are revised to ASCII

I am running into some problem when my top file contains Chinese opening double quote.

My top file is UTF-8 encoding in order to handle Chinese. If there is any Chinese double quote in the file, :build mybot will raise a warning.

line 11 of UTF8 closing double quote revised to Ascii

If I encode the same file with UTF8 without BOM, the above warning goes away, but I get another file level warning:
File has no utf8 BOM but has character>127

I also tried escape the double quotes, and ^original(), but it won’t help. Is there anyway to solve this problem?

For testing purpose:
Chinese opening double quote is “
Chinese closing double double is ”

Thank you so much for your help.


  [ # 1 ]

CS only tokenizes strings and processes them correctly using simple ascii doublequotes, hence why it translates other utf8 quotes to ascii simple.  I understand your files have utf8 quotes.  Why is it important that they NOT be changed internally. When you output such string to user, Chinese want to see the curly quotes only?


  [ # 2 ]

Yes, the curly quotes (directional) is expected in Chinese, since the vertical quotes are not officially in Chinese. Is it possible that we do not tokenize Chinese quotes at all and treat it as a regular utf8 character? Or is it possible that after tokenization, the output is converted back to the original quote format?


  [ # 3 ]

So, is it sufficient if I provide an output control that has CS automatically convert normal quotes on output to the user into curly quotes?


  [ # 4 ]

Yes, absolutely.  It’s better to have two output control for the right and left curly quotes. grin

Thank you so much.


  [ # 5 ]

done in next release


  login or register to react