| Store | Cart

Re: [TCLCORE] Customers with "Bengali" as keyboard input language

From: Robert Neumann <RNeu...@vizrt.com>
Thu, 18 Sep 2014 08:31:32 +0000
Hello Jeff,

I tried replacing all the message pump functions with their W counterparts. After doing this the TCL/TK code still needs to be adapted further to handle the windows messages containing different data, e.g. WM_CHAR will contain Unicode chars.

Do you think there will be a lot more changes required which I don’t see yet? Do we have a chance of doing this ourselfs in our 8.4 code base?

Regards,
Robert Neumann.

Von: Jeff Hobbs [mailto:jef...@activestate.com]
Gesendet: Donnerstag, 18. September 2014 02:10
An: Robert Neumann
Cc: Dipl. Ing. Sergey G. Brester; tcl-...@lists.sourceforge.net; Francesca Kreuz
Betreff: Re: [TCLCORE] Customers with "Bengali" as keyboard input language

I would actually lean towards believing there are a few ascii commands in the Tcl/Tk code that could be permanently set over to wide, causing issues like this.  I thought about doing this a few years ago (when NT became the solid supported base), but never finished the work.  I believe there is no more need for the A/W command jumping on Windows.  That can all be simplified to pure use of W functions.

Jeff

On Wed, Sep 17, 2014 at 7:44 AM, Robert Neumann <RNeu...@vizrt.com<mailto:RNeu...@vizrt.com>> wrote:
Hello sebres,

thank you for your answer.

I tested on Windows 8 where it is easy to install multiple keyboard input languages and switch between them on the fly using “Win+Space”. Windows will sent a WM_INPUTLANGCHANGE message to an application when the keyboard is switched to a different language. The language of Windows itself (user interface, message boxes etc.) remains German all the time. Only the keyboard is switched to Bengali.

If I understand you right, you doubt that TK is translating input using codepages the way I presume and my system is simply setup in a way that does not work for Bengali. But Bengali *does* work on my machine and that of our customer in every textbox (Notepad, Notepad++, Word etc.) except for TK text boxes and it seems to be because of the code path processing keyboard input being not fully Unicode.

And this already starts on the Windows side before the keycode even reaches TK:
It seems that in first place Windows is applying codepages because of TK creating its windows in a way which makes them *ANSI* for the windows API.
What I could determine with the debugger and using Spy++ is that TK is in fact receiving only 8-bit character codes in WM_CHAR message, as an ANSI window would receive.

This code is not always the same for the “A” key, but it is always 8-bit:

-          With German keyboard language the code sent for “A” is 0x41

-          With Arabian keyboard language the code sent for “A” is 0xF6

-          With Bengali keyboard language the code sent for *every* character is 0x3F (=”?”)
TK will then translate those codes to UTF-8, for which it needs a codepage again – and it has one for Arabian and Russian. But it hasn’t for Bengali and if it always receives 0x3F, there is nothing to be done anyway.

From my current level of research this happens because Windows uses codepages to convert keyboard input to ANSI codes if the receiver is an ANSI window. Windows has no codepage for Bengali so it sends “?” (http://www.unicode.org/mail-arch/unicode-ml/y2001-m04/0148.html).

That TK creates the window using CreateWindowW() is not enough to make it receive Unicode messages:

Only if you create a window using CreateWindowW() *and* also use GetMessageW() and DispatchMessageW() the character codes in WM_CHAR become real Unicode:

-          With German keyboard language the code sent for “A” will be 0x41

-          With Arabian keyboard language the code sent for “A” will be 0x650

-          With Bengali keyboard language the code sent for “A” will be 0x993

This is reproducible on every Windows 7 and 8 machine by installing additional keyboard languages (you don’t need to change the language of Windows itself).

Best regards,
Robert.

Von: Dipl. Ing. Sergey G. Brester [mailto:seb...@users.sourceforge.net<mailto:seb...@users.sourceforge.net>]
Gesendet: Mittwoch, 17. September 2014 16:15
An: Robert Neumann
Cc: tcl-...@lists.sourceforge.net<mailto:tcl-...@lists.sourceforge.net>
Betreff: Re: [TCLCORE] Customers with "Bengali" as keyboard input language


@all: sorry for german letter, just to explain easier ...

---

Hallo Herr Neumann,

it seems that all input is translated using codepages.

Ich denke nicht, dass das richtig ist - denn wäre es so, sollte dann mein folgendes Test scheitern:

Getestet habe ich mit Standart-Demo-Widget "Entries without scrollbars" aus TK-Demos von Activestate-Standardinstallation in tk8.5 und tk8.6 (leider habe ich 8.4 momentan nicht zur Hand).
Ich habe da in ein leereres Editbox ein zweisprachiges Text eingegeben, un zwar habe ich German und Russian keyboard layout parallel getestet. Mein System-Encoding ist German (cp1252). So ich könnte z.B. mit dem Umschalten folgenden Satz eintippen:
Привет - äöüß

Hätte tk (wie Sie es vermuten) es zwischenkonvertiert, so hätte ich dann folgendes eingetippt:

Für system encoding cp1252:
Ïðèâåò - äöüß

Oder für system encoding cp1251:
Привет - дцьЯ

Ich denke es liegt an irgendwas anderem - sei es Treiber oder "Microsoft Indic Language Input Tool" oder etwas noch.

Haben Sie es mit Activestate-Standardinstallation schon mal ausprobiert?
Denn wenn es da gehen wird - könnte es dann an Ihrem Compiler bzw. Compiler-Einstellungen liegen.

Leider habe ich keine Bengali/Windows zur Verfügung.
Könnten Sie die Problem-Umgebung etwas präziser beschreiben: Windows-Version, System-Encoding, etc.
Oder könnte man evtl. das "Kundensystem" als Virtuelle Maschine (VM) einrichten?

--

MFG,

sebres.



Am 17.09.2014 12:52, schrieb Robert Neumann:
Hello everyone,

we have been using TCL/TK for the user interface of our modelling tool from the very beginning 16 years ago:
http://www.vizrt.com/products/viz_artist/
This is the worldwide most used application for creation of realtime broadcast graphics.

For our first customers in Asia we implemented a custom IME editor, but the custom editor was optional and customers worldwide have long stopped using it, since TK textbox has sufficient unicode handling and Windows provides input methods for every language.

This works for nearly every language (including Thai, Arabian etc.) but we have some issues with customers using some indic languages like “Bengali” as the keyboard input language. Their input arrives as “???” in the TK text widget (that’s even true for the TCL console itself). You can paste Bengali characters from other applications though and they will be displayed correctly, so it seems to be a problem with processing of keyboard input.

I have dug into the TK source code (we are still working on 8.4) a bit and it seems that all input is translated using codepages. But there are no Windows codepages for Bengali (http://www.unicode.org/mail-arch/unicode-ml/y2001-m04/0148.html).

If you create a window as a unicode window (CreateWindowW) *and* use the wide character variants of *all* functions driving the message loop you will receive unicode WM_CHAR events, which already contain Unicode character codes and don’t need codepages anymore.
It seems TK windows are Unicode, but the message pump is implemented with the ANSI variants of GetMessaget() etc.
I have laid out some details at comp.lang.tcl (https://groups.google.com/forum/#!topic/comp.lang.tcl/3HaLG78feUE)

Of course I may be on the wrong track here, because I don’t have a perfect overview of all the settings and code paths in TK.

Any hint or any help would be very much appreciated.

We don’t want to activate the workaround with the custom IME editor again, because it implies hosting a single standard windows textbox in our TK GUI just for this purpose, which caused all kinds of problems.

We have implemented numerous modifications to the TCL/TK 8.4 source code to adapt it for our needs so we need to stick to this version for now. (But a short test with the TCL console of ActiveState TCL 8.6 revealed the same problems there.)

Kind regards,
Robert Neumann.

Vizrt Austria GmbH


------------------------------------------------------------------------------

Want excitement?

Manually upgrade your production database.

When you want reliability, choose Perforce

Perforce version control. Predictably reliable.

http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk


_______________________________________________

Tcl-Core mailing list

Tcl-...@lists.sourceforge.net<mailto:Tcl-...@lists.sourceforge.net>

https://lists.sourceforge.net/lists/listinfo/tcl-core



------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Tcl-Core mailing list
Tcl-...@lists.sourceforge.net<mailto:Tcl-...@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/tcl-core

------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Tcl-Core mailing list
Tcl-...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tcl-core

Recent Messages in this Thread
Robert Neumann Sep 17, 2014 10:52 am
Dipl. Ing. Sergey G. Brester Sep 17, 2014 02:14 pm
Robert Neumann Sep 17, 2014 02:44 pm
Jeff Hobbs Sep 18, 2014 12:10 am
Martin Lemburg Sep 18, 2014 06:56 am
Robert Neumann Sep 18, 2014 08:31 am
Jan Nijtmans Sep 18, 2014 09:46 am
Robert Neumann Sep 18, 2014 11:14 am
Donal K. Fellows Sep 20, 2014 12:44 pm
Don Porter Sep 20, 2014 12:49 pm
Don Porter Sep 20, 2014 12:58 pm
Messages in this thread