The Keyboard Layout Editor
(this entry is a repost, the original was lost in a database mishap.)
As part of the 2008 GSoC program, I worked on a Keyboard Layout Editor for the X.Org Foundation.
The Keyboard Layout Editor (KLE) is an application that allows you to create keyboard layouts for the X.Org server, commonly found in the Linux, OpenSolaris, *BSD, etc Desktops.
My mentor was Sergey Udaltsov, maintainer of xkeyboard-config, the Keyboard Indicator applet in GNOME, supporting libraries for keyboard layouts and much more. I had great help and Sergey was very supportive. Highly recommended mentor for your GSoC’09 project.
The screenshot above shows the main window of the program; a keyboard with blank layout (keys are empty), a section Add to layout with items that can be used to populate the layout, and a section for the description of the layout (Layout details).
There are typically two workflows; first you start off with a blank layout and you add Unicode characters, dead keys, include files, then you save.
The other workflow is to start with an appropriate existing layout as a base, then add more characters, make changes, etc.
It might be strange to talk about different workflows, but in terms of usability it’s important provide assistance for such cases. For example, having tooltips is important when a person starts off with a new layout.
Here we started with a blank layout; we click on Start Character Map, then locate the characters you need, and drag and drop them to the appropriate keys. Each key is composed of four parts, and we number these from 1 to 4. The way we count is quite peculiar,
- bottom left, when you press the key as is (key)
- top left, when you press the key with Shift (Shift + key)
- bottom right, when you press the key with AltGr (AltGr + key)
- top right, when you press the key with Shilft+AltGr (Shift + AltGr + key)
This is my entry to the most engineered diagram competition.
The dead keys relate to diacritic marks such as grave and acute. Since they are too small to see, we present them next to a D letter (D for Dead key). In some cases I could not find a character equivalent to the diacritic mark, so I put ?, therefore it looks like D?. If you put the mouse pointer over the key, you can see the full details in the tooltip.
In many cases, there exist layouts/variants that contain most of the characters you want to add. In this case, you add and enable in the Include files section. You can then override any of those characters by dragging and dropping to the layout.
At this stage in the blog post, it is important to clarify the notions of a layout and a variant. The two are quite similar and the distinction is messy when trying to explain to the end-user. The French layout file is fr, which contains several variants (distinct groups of mappings of physical keys to Unicode characters). When you are actually talking about a French keyboard layout, you are actually referring to the default variant of the «fr» file. Oftentimes people refer to the «fr» file as a whole as the French layout. You can also pick a non-default variant of the layout file, and call it your layout.
The way I would like to define layout and variant is this: a layout refers to the default variant of the layout file. This is consistent to the fact that distributions pick the default variant in the settings so it’s what get the most visibility, or when users select a new layout, they are presented with the default setting first. Regarding layouts in general, it is important for different languages/scripts to make effort that the default layout is updated and includes extra useful and relevant characters.
This is the updated Greek keyboard layout, and is the near-final version that we are planning to submit to xkeyboard-config. It adds Greek Polytonic to the existing Greek layout. It does not make changes to the previous default layout, so users will not be unpleasantly surprised. It also adds all sort of characters that are found in the Greek Unicode block.
In this post I simplified some of the terms/description. If I went a bit too far, please correct me and I’ll update in-place.
Update 8th Sep 08: What are the plans for further development of the layout editor;
- Increase the user base and get more people trying out the editor. This requires some more cleanup of the code, more instructions on how to run it youselves, and get people to provide feedback. An open-source project without users is not a successful project.
- Make it easier for developers to contribute on the project. If you use Eclipse, you can install pydev, antlr3ide, mylyn, subclipse, and you can do the full development from within the cozy Eclipse environment. These need documentation.
- The Issues page at the project has about ten items. This list needs to be reduced.
- The natural place for users of the layout editor is the http://listserv.bat.ru/xkb/List.html mailing list. We need to promote the editor there, and get examples of users actually using it to maintain layouts.
- An issue that plagues some users is when they need compose sequences to generate characters that no pre-composed forms exist. If users really need this (mainly Latin and Cyrillic scripts, complex scripts), it can be adapted to the UI.
- It is technically easy to adapt the editor so that it produces XML layouts. Considering the state of XKB-atkins, this may not be a top priority at the moment. libxml2 comes with the MIT license, so in license terms it would be OK. Not sure if it is OK to link libxml2 to the X.org server. It might actually solve the slow parsing of the configurations files and the issues of xkbcomp.
- At the moment the default geometry is a somewhat generic keyboard. In addition, I deactivated several keys (such as the function keys), in order not to confuse users (you can activate with a small change in the code). The keyboard can be expanded to a full 105-keys style. A related project would be to figure out an efficient way to edit those geometry files, and make the keyboard customised. If people start creating layouts with the editor, they will certainly love to edit geometry files!
ANTLR grammar for XKB, and Relax NG schema (draft)
I completed the ANTLRv3 grammar for symbols/ configuration files of XKB. The grammar can parse and create the abstract syntax tree (AST) for all keyboard layouts in xkeyboard-config.
ANTLRv3 helps you create parsers for domain specific languages (DSL), an example of which is the configuration files in XKB.
Having the ANTLRv3 grammar for a configuration file allows to generate code in any of the supported target lagnuages (C, C++, Java, Python, C#, etc), so that you easily include a parser that reads those files. Essentially you avoid using custom parsers which can be difficult to maintain, or parsers that were generated with flex/bison.
On a similar note, here is the grammar to parse Compose files (such as en_US.UTF-8/Compose.pre). I am not going to be using in the project for now, but it was fun writing it. The Python target takes 18s to create the AST for the >5500 lines of the en_US.UTF-8 compose file, on a typical modern laptop.
I am also working on creating a RelaxNG schema for the XKB configuration files (those under symbols/). There is a draft available, which needs much more work.The Relax NG book by Eric van de Vlist is very useful here.
The immediate goal is to use the code generated by ANTLR to parse the XKB files and create XML files based on the Relax NG schema. I am using Python, and there are a few options; the libxml2 bindings for Python, and PyXML. The latter has more visible documentation, but I think that I should better be using the former.
Update: lxml appears to be the nice way to use libxml2 (instead of using directly libxml2).
Looking into the symbol files
In the previous post, we talked about the ANTLR grammar that parses the XKB layout files.
The grammar is available at http://code.google.com/p/keyboardlayouteditor/source/browse. I’ll rather push to the freedesktop repository once the project is completed. Now it’s too easy for me, just doing svn commit -m something.
Below you can see the relevant layout files for each country (and in some cases, language), and how the grammar deals with them. First column is filenames from the CVS XKB symbols subdirectory (to be moved eminently to GIT). Last’s week discussion with Sergey helped me figure out issues with the symbol files, simplify what information is needed, and what can be eliminated. Second column has Not OK if something is wrong. Third column tries to explain what was wrong.
| ad | ||
| af | ||
| al | ||
| altwin | ||
| am | ||
| ara | ||
| az | ||
| ba | ||
| bd | ||
| be | ||
| bg | ||
| br | ||
| braille | ||
| bt | ||
| by | ||
| ca | ||
| capslock | ||
| cd | ||
| ch | ||
| cn | ||
| compose | ||
| ctrl | ||
| cz | ||
| de | ||
| dk | ||
| ee | ||
| epo | ||
| es | ||
| et | ||
| eurosign | ||
| fi | ||
| fo | ||
| fr | ||
| gb | NOK | Non-UTF8 |
| ge | ||
| gh | ||
| gn | ||
| gr | ||
| group | NOK | virtualMods= AltGr |
| hr | ||
| hu | NOK | Non-UTF8 |
| ie | ||
| il | NOK | key.type=”FOUR_LEVEL” (typically: key.type[something]=….) |
| in | NOK | key.type=”FOUR_LEVEL” (typically: key.type[something]=….) |
| inet | ||
| iq | ||
| ir | ||
| is | ||
| it | ||
| jp | NOK | key <BKSP> { type=”", // empty? symbols[Group1]= [ bracketright, braceright ] }; |
| keypad | NOK | overlay1=<KO7> }; // what’s “overlay”? |
| kg | ||
| kh | ||
| kpdl | ||
| kr | ||
| kz | ||
| la | ||
| latam | ||
| latin | ||
| level3 | NOK | virtual_modifiers LAlt, AlGr; virtualMods= Lalt |
| level5 | ||
| lk | ||
| lt | ||
| lv | ||
| ma | ||
| mao | ||
| me | ||
| mk | ||
| mm | ||
| mn | ||
| mt | ||
| mv | ||
| nbsp | NOK | Non-UTF8 |
| ng | ||
| nl | ||
| no | ||
| np | ||
| olpc | ||
| pc | NOK | key <AA00> { type=”SOMETHING” } instead of { type[Group1]=”SOMETHING” } |
| pk | ||
| pl | ||
| pt | ||
| ro | ||
| rs | ||
| ru | ||
| se | ||
| shift | NOK | actions [Group1] = [ |
| si | ||
| sk | ||
| srvr_ctrl | NOK | key <AA00> { type=”SOMETHING” } instead of { type[Group1]=”SOMETHING” } |
| sy | ||
| th | ||
| tj | ||
| tr | ||
| ua |
Non-UTF-8 are the files that have characters that are not UTF-8 (are iso-8859-1).
Some layouts have key.type = “something” and others key.type[SomeGroup] = “something”. Apparently, the format allows to infer which is the group that the type acts upon? That’s weird. Would it be better to put the group information? Is it required that the group is not set?
Some files have virtualMods, which I do not know what it is. Is it used?
Parsing XKB files with antlr
antlr (well, antlr3) is an amazing tool that replaces lex/flex, yacc/bison.
One would use antlr3 if they want to deal with Domain-Specific Languages (DSL), an example of which are the text configuration files.
In our case, we use antlr3 to parse some of the XKB configuration files, those found in /etc/X11/xkb/symbols/??.
Our aim is to be able to easily read and write those configuration files. Of course, once we have them read, we do all sorts of processing.
The stable version of antlr3 is 3.0.1, which happened to give lots of internal errors. It has not been very useful, so I tried a few times the latest beta version 3.1b, and eventually managed to get it to work. If I am not mistaken, 3.1 stable should be announced in a few days.
When using antlr, you have the choice of several target languages, such as Java, C, C++ and Python. I am using the Python target, and the latest version that is available from the antlr3 repository.
Here is the tree of the gb layout file,
tree = (SECTION (MAPTYPE (MAPOPTIONS partial default alphanumeric_keys xkb_symbols) (MAPNAME “basic”)) (MAPMATERIAL (TOKEN_INCLUDE “latin”) (TOKEN_NAME Group1 (VALUE “United Kingdom”)) (TOKEN_KEY (KEYCODEX AE02) (KEYSYMS 2 quotedbl twosuperior oneeighth)) (TOKEN_KEY (KEYCODEX AE03) (KEYSYMS 3 sterling threesuperior sterling)) (TOKEN_KEY (KEYCODEX AE04) (KEYSYMS 4 dollar EuroSign onequarter)) (TOKEN_KEY (KEYCODEX AC11) (KEYSYMS apostrophe at dead_circumflex dead_caron)) (TOKEN_KEY (KEYCODEX TLDE) (KEYSYMS grave notsign bar bar)) (TOKEN_KEY (KEYCODEX BKSL) (KEYSYMS numbersign asciitilde dead_grave dead_breve)) (TOKEN_KEY (KEYCODEX LSGT) (KEYSYMS backslash bar bar brokenbar)) (TOKEN_INCLUDE “level3(ralt_switch_multikey)”))) (SECTION (MAPTYPE (MAPOPTIONS partial alphanumeric_keys xkb_symbols) (MAPNAME “intl”)) (MAPMATERIAL (TOKEN_INCLUDE “latin”) (TOKEN_NAME Group1 (VALUE “United Kingdom – International (with dead keys)”)) (TOKEN_KEY (KEYCODEX AE02) (KEYSYMS 2 dead_diaeresis twosuperior onehalf)) (TOKEN_KEY (KEYCODEX AE03) (KEYSYMS 3 sterling threesuperior onethird)) (TOKEN_KEY (KEYCODEX AE04) (KEYSYMS 4 dollar EuroSign onequarter)) (TOKEN_KEY (KEYCODEX AE06) (KEYSYMS 6 dead_circumflex NoSymbol onesixth)) (TOKEN_KEY (KEYCODEX AC11) (KEYSYMS dead_acute at apostrophe bar)) (TOKEN_KEY (KEYCODEX TLDE) (KEYSYMS dead_grave notsign bar bar)) (TOKEN_KEY (KEYCODEX BKSL) (KEYSYMS numbersign dead_tilde bar bar)) (TOKEN_KEY (KEYCODEX LSGT) (KEYSYMS backslash bar bar bar)) (TOKEN_INCLUDE “level3(ralt_switch)”))) (SECTION (MAPTYPE (MAPOPTIONS partial alphanumeric_keys xkb_symbols) (MAPNAME “dvorak”)) (MAPMATERIAL (TOKEN_INCLUDE “us(dvorak)”) (TOKEN_NAME Group1 (VALUE “United Kingdom – Dvorak”)) (TOKEN_KEY (KEYCODEX BKSL) (KEYSYMS numbersign asciitilde)) (TOKEN_KEY (KEYCODEX AE02) (KEYSYMS 2 quotedbl twosuperior NoSymbol)) (TOKEN_KEY (KEYCODEX AE03) (KEYSYMS 3 sterling threesuperior NoSymbol)) (TOKEN_KEY (KEYCODEX AE04) (KEYSYMS 4 dollar EuroSign NoSymbol)) (TOKEN_KEY (KEYCODEX LSGT) (KEYSYMS backslash bar)) (TOKEN_KEY (KEYCODEX AD01) (KEYSYMS apostrophe at)))) (SECTION (MAPTYPE (MAPOPTIONS partial alphanumeric_keys xkb_symbols) (MAPNAME “mac”)) (MAPMATERIAL (TOKEN_INCLUDE “latin”) (TOKEN_NAME Group1 (VALUE “United Kingdom – Macintosh”)) (TOKEN_KEY (KEYCODEX AE02) (KEYSYMS 2 at EuroSign)) (TOKEN_KEY (KEYCODEX AE03) (KEYSYMS 3 sterling numbersign)) (TOKEN_INCLUDE “level3(ralt_switch)”)))
When traversing the tree, we can then pretty-print the layout at wish:
partial default alphanumeric_keys xkb_symbols “basic” {
name[Group1] = “United Kingdom”;
include “latin”
include “level3(ralt_switch_multikey)”
key <AE02> = { [ 2 , quotedbl , twosuperior , oneeighth ] };
key <AE03> = { [ 3 , sterling , threesuperior , sterling ] };
key <AE04> = { [ 4 , dollar , EuroSign , onequarter ] };
key <AC11> = { [ apostrophe , at , dead_circumflex , dead_caron ] };
key <TLDE> = { [ grave , notsign , bar , bar ] };
key <BKSL> = { [ numbersign , asciitilde , dead_grave , dead_breve ] };
key <LSGT> = { [ backslash , bar , bar , brokenbar ] };
};
… snip …
The code is currently hosted at code.google.com (keyboardlayouteditor) and I intend to move it shortly to FDO.




