Crusoe Transmeta CMS Bug

A known bug is better than an unknown feature.
Manoj Sati

Summary of the problem

Quite a long time ago, I found something strange on my Crusoe laptop. Something which was looking like a bug, smelling like a bug and taste just like a bug. So, I did spend some time to investigate it a bit. I am running a Debian GNU/Linux and I have a laptop Vaio C1MZX with a Transmeta Crusoe TM5800 (see the lspci).

From time to time the Xserver stop to work properly and refuse new connections from any Xclients. Each time a client try to connect, the following error message is displayed:

 X Error of failed request:  BadLength (poly request too large or internal Xlib length error)
 Major opcode of failed request:  18 (X_ChangeProperty)
 Serial number of failed request:  15
 Current serial number in output stream:  18

The Xserver itself is still standing but any Xclient trying to connect will terminates with a BadLenght error. It seems that during the initialisation of the network connection of the Xclient on the Xserver, the Xserver is sending a wrong message and makes the Xclient to terminate (see the gdb log xlogo_bug.log). The 9th reply from the Xserver seems the one that makes the Xclients crash (the normal negotiation between the Xserver and the Xclient is traced in xlogo_nobug.log). I also noticed that stopping the Xserver and starting it again was not helping to remove the bug. The Xserver will still behave the same (i.e. since the bug start it stays consistent for a while). More surprisingly, if you do a copy of the binary file of the "buggy" Xserver and run it, then it will work without bug, but when running again the original binary file that started with the bug, the bug will appear once again.

All Xservers (whatever version number in the 4.3.x serie, compiled with optimization level 2, with debug or not) will crash at some point and behave as mentioned previously. I tried to go inside the Xserver by attaching gdb to the Xserver process but the Xserver start to use a lot of cpu time and nothing happen. When interrupted inside gdb by a Ctrl-C, gdb give a prompt again but a "bt" gives some nonsense informations (when the bug is not present attaching gdb to the process works normally).

I and other people have been reporting this problem in the GNU/Linux Debian X-strike force mailing-list (see bug #216933) and also to the freedesktop bugzilla (see here)

After posting the description of the bug on the Linux Kernel Mailing-list I've been contacted by a guy from Transmeta which kindly offered to try to look for this bug on his own and try to see if he can do something. Thanks to the discussion I had with him, I think I got a deeper understanding of what was going on.

As you should all know, the Transmeta Crusoe is using a CMS (Code Morphing Software) to translate the x86 binaries into VLIW (Very Long Instruction Words) instructions. For performance reasons, once the translation of one binary is done it is kept into a cache in order to not retranslate it too often.

[Note: More precisely, the CMS is first interpreting each block of the binary that are executed several times and collect some informations about it meanwhile the execution. When a certain threshold is reached and enough informations gathered, the CMS store an optimized translation of the block of binary into the CMS cache. So, the code is not cached immediately after the first encounter with the CMS.]

Now you have to remember that we've noticed that once we enter in a particular mode (with unknown conditions) the Xserver was always failing even if we stop it and run it again. But, when running a copy of the binary, it was working fine.

According to the guy from Transmeta (which was working on the development of the CMS for the Crusoe and the Efficeon), this technique of copying binaries was a work around to the cache of the CMS and seems to be an usual technique used by the developers of the CMS. Now, Lets make the hypothesis that from time to time the CMS is getting crazy for some strange reasons and translate something wrong in the binary of the Xserver.

Here we are !!! The combination of the CMS cache and the presence of a bug in the CMS can perfectly explains the behaviour of the bug (both the persistence once it occurs and that a copy of the exact same file can work properly). So my guess is that the bug is somewhere in the CMS...

Another particularity of the bug is that it seems to appear only in very special conditions (Henry Pan reported that it was occuring 2.5% of the time). For example, the people from Transmeta didn't manage to reproduce it on their development board (PDB). So, it makes it very difficult for them to fix it. That's why, I try to gather a list of people that encounter this bug and what version of the CMS they are running, what hardware they are using, and what Linux distribution they have installed (most of them are running Debian). I hope to convince people from Transmeta of the importance of this bug and have them to fix it.

Identify Your CMS Revision

The revision of the CMS can be taken from the dmesg or from the BIOS. The following is taken from the dmesg:

CPU: After generic identify, caps: 0080893f 0081813f 00000000 00000000
CPU: After vendor identify, caps:  0080893f 0081813f 0000004e 00000000
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (32 bytes/line)
CPU: L2 Cache: 512K (128 bytes/line)
CPU: Processor revision 1.4.1.0, 933 MHz
CPU: Code Morphing Software revision 4.3.2-9-343
CPU: 20020426 17:54 official release 4.3.2#2
CPU: After all inits, caps: 0080893f 0081813f 0000004e 00000000
CPU: Transmeta(tm) Crusoe(tm) Processor TM5800 stepping 03

CMS revision = 4.3.2-9-343

How to upgrade the CMS from Linux ?

Ok, lets be optimistic. Let say that one day we'll find the bug and find a patch for it. What will be next ?

Well, it would be fine if we could have a way to ugrade our CMS with the last version of it, bug free, faster, nicer, ...

Hum, looks like things are not that easy !!!

I know that HP did this for his tablet PC (see here). But it has been the only one to do so. It even leads to some discussion about other hardwares (see here). And some reverse engineering on the CMS and the Transmeta Crusoe (see here and here and you can get a disassembled image of the CMS 4.4.0, here). Since this, Transmeta didn't give any update of the CMS (at least, I didn't heard about it). Until now the requisite of the users was to improve the performance of the laptop (see here), but now we have to face (what I believe is) a real bug. So, one can imagine that this might change things slightly.

The problem is that Transmeta has totally dropped the effort on the development of the Crusoe CMS and as they have hard time to reproduce the bug on a development board, so we have to come with a way for them to reproduce the bug in order to get a fix.

And even if we achieve this, the way the CMS is built-in on the hardware allow one way to upgrade the firmware. Each upgrade has to be signed with a certain private key (this private key is known by the seller of the laptop and by Transmeta itself... well I assume that in fact each entity has a partial knowledge of the private key which makes impossible to one of these to do something without the agreement of the other one).

[Note: Even inside Transmeta, it seems to takes 4 or 8 people to sign an upgrade. So, even if the developers want to help us, there is no way for them to do so (so, don't be rude to them ! ;).]

At the end, the process for upgrading the CMS firmware is the following:

  1. Load the upgrade in memory,
  2. Check the signature of the upgrade with the public key stored in the ROM.
  3. If the signature match with the upgrade, then apply the upgrade.

As you see, an upgrade (for Linux or Windows, whatever) requires the agreement of both Transmeta AND the laptop seller. And you cannot easily hack your way through.

I see only two solutions to do our own upgrade of the CMS:

  1. Take the EPROM out and write our own public key on it... (risky and need a lot of hardware. I wouldn't recommand this way)
  2. Crack the public/private keys of the hardware. (this is a known plaintext attack for the HP tablet-PC, and for the other hardwares we can only have access to the public-key which is making it more difficult but nothing that can resist to a brute force attack in the case we have enough Seti-like softwares running).

I have no idea if it is legal or not... it is not my concern now. I'm just seeking for solutions ! :) I guess it depends if you are in the States, in Japan or in Europe. But, after all, I am just an unsatisfied customer who try to use properly a piece of hardware that I own because I paid for it... Am I a criminal because of this ???

So, I am about here in my investigations and I still have this annoying bug with the Xserver... Moreover, it seems that Transmeta is fully on the Efficeon now and does not want to invest time and money on looking for a bug in the Crusoe CMS (except if one of its customer is specifically asking for it, which is very unlikely from Sony and/or Fujitsu).

Not so much hope, don't you think ? Well, I won't give up so easily (for once that we have a nice bug to fight with) !!! :)

Find a Workaround ?

The bug is probably due to some code optimization done by the compiler or more likely to a serie of instructions. A lot of people noticed that from time to time after an update of the X server in their distribution, the bug was appearing more (or less) often.

The idea is to find a suitable set of compilation options which do not enable the bug. Chris Lange reported that he compiled his own Xserver with the options -march=i386 -mcpu=i386 -O0 -fomit-frame-pointer -finline-functions. And didn't experiment any crash since (note that all optimization is disabled, only part of the memory managment is optimized). He kindly put his custom Xserver RPMs at http://tardis.ath.cx/misc/xf86_4.3.0_cust/

I tried by myself with the options -O0 -fomit-frame-pointer and the server seems to be stable like this, I didn't experiment any occurence of the bug since then. For the Debian users, Ilja Gerhardt and I did a small documentation and a patch to set the compiling options right. Or just download the xserver-xfree86 package or the xserver-xorg package that I compiled. Feel free to use it and to send me your comments/improvements.

Moreover, as said a guy from Transmeta, getting rid of the optimization is not really a problem as the CMS is doing some optimizations by its own.

Conclusion

So, if anybody which has encounter this bug could mail me his CMS revision and/or add some informations about this bug or on how to upgrade the CMS, I would be very pleased to gather all this and mail Transmeta about it.

The nicest solution would be to fix this bug in the CMS itself, but finding a way to compile the Xserver in order to avoid this bug would be also a good start.

List of platforms with the bug

Reverse Engineering of the Transmeta Crusoe CMS