New x86_64 Toolchain for TS72xx

I just finished putting together an x86_64 'compatibility' toolchain for the TS72xx boards. This is an addition to what I've already done, fully documented on the Gentoo Wiki. See my toolchains directory under x86_64-pc-linux-gnu/arm-unknown-linux-gnu.

The binaries can be installed following these instructions.

Njoy!

Follow Up to EABI / Maverick Userland

Hi Everyone,

I left off with instructions for creating an eabi / maverick toolchain for the ts72xx boards from EmbeddedARM.com, and am still sifting my way through some pretty dense decisions.

First of all, Gentoo's Portage build system makes it really easy to cross-compile packages and maintain dependencies. I think it's fantastic. The ony tricky part is in how the package database is maintained on-board. It's not even a database actually, it's right in the filesystem. Depending on the block size, even a 2-byte file could take up 4 kB of space, which is way beyond anything conceptual for an on-board package database for embedded devices.

I've considered writing in some sqlite code into the 'qmerge' app that comes with portage-utils. The benefit is that the sqlite file would save potentially doezens of kB of space. Furthermore, the sqlite file could be compressed in the filesystem after updates have taken place. When any package maintenance needs to be done, the sqlite database could be extracted to a tmpfs-mounted directory, and all of the operations could be performed there.

An alternative to that, is actually just to use the current filesystem-based package db and simply compress that directory without the sqlite backend. Then when updates were necessary, it would be possible to extract that to tmpfs, and re-compress when done.

Time is of the essence on my latest project, so it's likely that I won't be able to fully code that into portage-utils until my current project is matured and nearing completion. The matured stage is probably 1 month from now, and the completion stage is probably closer to 2 months away.

I'll post my instructions, as planned on gentoo-wiki.com for doing cross-compilation and building up to a certain stage, but more instructions on how to maintain packages on a live system may be delayed for some time.

More Maverick/EABI fun ;-)

I've just finished cross-compiling an entire root filesystem from scratch for the TS72xx boards with the EABI and it's working out very well! The softfp floating-point ABI has indeed shown a dramatic increase in speed! I've tested out using emqbit-bench and the performance improvements match those found in the article "Why ARM's EABI Matters". I haven't yet been able to try out some hard floating point calculations, such as those done by Martin Guy, but hopefully I'll get to that soon.

The entire HowTo is written up right now and I've tested the procedure with both the arm-softfloat-linux-gnu toolchain/kernel/filesystem and the armv4t-maverick-linux-gnueabi toolchain/kernel/filesystem.

The kernel config /patches are working perfectly for linux-2.6.23.1 . I should have an article up on Gentoo-Wiki.com specifically assisting TS-72xx users in creating a manageable build environment for their boards.

After I get the article up, my next goal will be to determine a suitable binary package manager for live system updates. Thus far, I've looked at Paludis and iPkg [ 20080207 Update: ] and Qmerge which is one of the 'q' tools that comes with app-portage/portage-utils.

After reading an interview with Ciaran McCreesh, I've really gained an appreciation for what Paludis is doing with ebuild syntax modification. Some of those modifications are just so intuitive, convenient, and easily implemented, that I really wish they were already in Gentoo's Portage system. Actually, the article explains in some detail why that isn't already the case.

Paludis also requires a C++ runtime, just because it would have been quite difficult to code it entirely in C, although a rewrite is still possible I suppose. I'd rather not get into C++ right now on production systems. Also it seems that there is still a dependency for the entire portage / paludis ebuild tree to reside on the host, and that is definitely not happening.

On the other hand, iPkg has been in use for quite some time for embedded Linux package management and takes up only a few tens of kilobytes. It's recently been completely rewritten in C, which is fantastic news.

If I were to go with iPkg, then the other issue would be in conversion between Gentoo's tbz2 binary package format and iPkg's .ipkg binary format. Such a converter exists, but still requires some fine tuning.

[ 20080207 Update1: ] I've 95% decided to go with qmerge as my preferred on-board package manager. There are only a few things I need to check out before I go ahead with it fully. Namely, I need to do some test to make sure that it maintains a package database as I would expect, and that it can somehow report exact package versions when told to list installed packages (that might require some work on my part).

[ 20080207 Update2: ] Oh Gawd! Gentoo strikes again... I swear, I've never known a distro that so vagrantly throws around resources as if they were nothing... I think the dependency on /var/db/pkg might have bj@rkskt my idea of using qmerge as an embedded package manager. I'd rather not have to do any feature-adding if i can avoid it... qmerge was 95 percent there... maybe I should just write in some sqlite code and get it over with... I wonder how much that would actually improve things, if at all.

What puzzles me these days about all of the different Linux distro's having different package managers, is that the fundamental issue that they're all trying to solve is the same. Some of them handle that issue better than others, some of them lack certain features that others have, and some lean toward binary package distribution more than others (actually most). Fundamentally, the entire problem lies in solving dependency issues and tieing that together with the package database.

With embedded systems, the size of the package DB has to be extremely small, or else it needs to be maintained off-board.

[ Update 20080207: ] I think that I might be mistaken on the size of the portage db - it's much larger than expected!! For instance du -hs /var/db/pkg/net-wireless/wireless-tools is 128 kB !! That's WAY too big. Maybe I'm looking in the wrong place here - was I wrong to expect an actual database and not a filesystem for recording all of the relevent package information?

Maverick-Crunch / EABI support for Crossdev

Since I've been using the TS-7xxx boards from Technologic Systems, I've been wanting to experiment a bit more with the Maverick-Crunch floating point coprocessor in order to squeeze a bit more performance in number-crunching apps.

Here are some simple instructions for creating an armv4tl-maverick-linux-gnueabi toolchain using Gentoo/Portage/Crossdev. These instructions are current as of 26. February 2008.

[ Update: 26. Feb, 2008 ]

The procedure is tested and I have yet to experience any bugs in any normal userspace applications. I know that lame now produces mp3s 25 times faster, and there were no audible errors, which basically confirms with Martin Guy's results that the Maverick FPU is being used.

The only thing that I am lacking is a better stress test - paranoia.c just segfaults, and I'm not really sure why.

I've moved the instructions, as well as instructions for creating an eabi Linux kernel and a full userland, to the Gentoo Wiki.

My experiences with Qemu - Part II

So I've managed to get this far with Qemu:

(qemu) Uncompressing Linux......................................................... done, booting the kernel.
Linux version 2.6.20.6 (cfriedt@sith) (gcc version 3.4.4) #35 Thu May 31 21:57:50 CEST 2007
CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00003137
Machine: ARM-Versatile PB
Memory policy: ECC disabled, Data cache writeback
CPU0: D VIVT write-through cache
CPU0: I cache: 4096 bytes, associativity 4, 32 byte lines, 32 sets
CPU0: D cache: 65536 bytes, associativity 4, 32 byte lines, 512 sets
Built 1 zonelists. Total pages: 65024
Kernel command line: root=/dev/nfs nfsroot=192.168.7.1:/usr/gentoo_root,uid=0,gid=0,rsize=32768,wsize=32768,timeo=14,nfsvers=3,rw rw ip=192.168.7.2:192.168.7.1:192.168.7.1:255.255.255.0:qemu console=ttyAMA0,115200
PID hash table entries: 1024 (order: 10, 4096 bytes)
Console: colour dummy device 80x30
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 256MB = 256MB total
Memory: 258048KB available (1516K code, 174K data, 100K init)
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
NET: Registered protocol family 16
PCI core found (slot 11)
PCI: bus0: Fast back to back transfers disabled
PCI map irq: slot 0, pin 1, devslot 12, irq: 27
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 8192 bind 4096)
TCP reno registered
NetWinder Floating Point Emulator V0.97 (double precision)
CLCD: unknown LCD panel ID 0x00001000, using VGA
CLCD: Versatile hardware, VGA display
Clock CLCDCLK: setting VCO reg params: S=1 R=99 V=98
Console: switching to colour frame buffer device 80x30
Serial: AMBA PL011 UART driver
dev:f1: ttyAMA0 at MMIO 0x101f1000 (irq = 12) is a AMBA/PL011
dev:f2: ttyAMA1 at MMIO 0x101f2000 (irq = 13) is a AMBA/PL011
dev:f3: ttyAMA2 at MMIO 0x101f3000 (irq = 14) is a AMBA/PL011
fpga:09: ttyAMA3 at MMIO 0x10009000 (irq = 38) is a AMBA/PL011
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky
smc91x.c: v1.1, sep 22 2004 by Nicolas Pitre
eth0: SMC91C11xFD (rev 1) at d080a000 IRQ 25 [nowait]
eth0: Ethernet addr: 52:54:00:12:34:56
mice: PS/2 mouse device common for all mice
TCP cubic registered
ieee80211: 802.11 data/management/control stack, git-1.1.13
ieee80211: Copyright (C) 2004-2005 Intel Corporation
input: AT Raw Set 2 keyboard as /class/input/input0
eth0: link up
input: ImExPS/2 Generic Explorer Mouse as /class/input/input1
IP-Config: Complete:
device=eth0, addr=192.168.7.2, mask=255.255.255.0, gw=192.168.7.1,
host=qemu, domain=, nis-domain=(none),
bootserver=192.168.7.1, rootserver=192.168.7.1, rootpath=
Root-NFS: unknown option: uid=0
Looking up port of RPC 100003/2 on 192.168.7.1
Looking up port of RPC 100005/1 on 192.168.7.1
VFS: Mounted root (nfs filesystem).
Freeing init memory: 100K
init: must be superuser.
Kernel panic - not syncing: Attempted to kill init!

It's looking fairly bright now ;-) All I have to do is figure out why (this has absolutely no real reason) Linux would start up as someone who isn't the superuser
... hmmf..

Weeee !!! I just added an init= option to the 'append' section of Qemu!!! sahweet!!! She's ready to roll ;-)

The option to enable 16-bit UID system calls must be selected as well, otherwise root (0) will get mapped to 0xffffffff (-1 in 2's compliment == -EPERM).

My experiences with Qemu - part I ... was "Building GNU Classpath for the ARM (Cont'd)"

Ok, so it looks like the external USB hard disk idea was not as good as I was hoping. The problem is that the board just has too little RAM !!! (to compile such a package as the GNU Classpath)

What happens in either the NFS case or the USB HD case, is that eventually the swapd and the jikes process enter this vicious cycle, where jikes caches the first half of what it needs to compile, swaps it out, and then caches the second half of what it needs to compile - clearly if it doesn't have 100% of the things it needs to compile in RAM (this occurred even without using -pipe) then it will enter an infinite loop (it would be nice if that was detected by jikes - i think that's CRC32 or something?).

In any event, with the suggestion that my boss Dave gave me, I'm now working on emulating an appropriate kernel, and using the appropriate uclibc-softfloat userspace for the ARM, using Qemu. This should solve all of my compilation woes because I would then be able to have enough actual physical RAM to compile everything natively. I believe that the VersatilePB platform available in Qemu has a limit of 256MB RAM though. That should be enough (I assume).

Aside from the target having a different name, the main other differences between the TS-7xxx boards and the Versatile PB are:

• The Versatile PB uses an ARM926T chip instead of an ARM920T
• The Versatile PB supports an SMC 91c111 ethernet device instead of the ep93xx device.
• The Versatile PB names its serial devices /dev/ttyAMA[0-n] instead of /dev/ttyAM[0-n]

All of the exact details are here:

http://fabrice.bellard.free.fr/qemu/qemu-doc.html#SEC53

Now, a brief overview of the differences and what they will mean.

1. The ARM926T chip vs the ARM920T chip. To quote from the linux kernel help page:
   

   This is a variant of the ARM920. It has slightly different instruction sequences for cache and TLB operations. Curiously, there is no documentation on it at the ARM corporate website [EDIT: There is now].
   
   Say Y if you want support for the ARM926T processor. Otherwise, say N.

   
   I tried compiling for the Versatile PB without ARM926T support and that didn't even compile - it would be nice to have the ARM920T as an option here too. Anyways, that's not really a problem. As long as I can run Qemu with an ARM chip emulated and use an NFS root to compile, that's all that matters.
   
   
2. eth0 is eth0 ... I doubt it matters what hardware is underlying for my purposes.
   
   
3. Some scripts and applications will have to be aware of the difference of platform, if i end up coding scripts and things of the like on the emulator. I just as soon wouldn't, but it could be quite beneficial. In any event, that's not a real issue, it's as simple as replacing a string from my perspective. The cool part about that is that we could end up simulating our hardware, including the EM noise model - cool ;-)
   

I'll keep you posted!

Building GNU Classpath for the ARM (Cont'd ...)

I spontaneously decided to copy over the entire Gentoo stage3 filesystem to my external hard-disk, which is recognized as a general USB storage device under linux. NFS was taking waaaaay too long for this compilation. I had checked everything out with top, under linux, played with the swappiness parameter in /proc/sys/vm, and also attempted to renice the jikes processes' priority to a whopping 17! Non of those had a major effect on the cpu-usage of the jikes process. The highest that I'd seen it was 10% for a whole 2 seconds.

The problem lies with NFS root and the overhead associate not only with RPC, but also the fact that my swap was also on the NFS root, effectively squaring the complexity of the overhead. With my Gentoo stage 3 root filesystem located on a locally attached usb storage device, along with the swap file, the speed of compilation increased dramatically! The jikes process has just started and its already achieved 50% of the cpu time! That's up from an average of maybe 3% with the tweaking I mentioned previously.

I'd better mention this to Dave as well, so that he's not stuck ever repeating my stupid mistake of compiling on an NFS root!

Building the GNU Classpath for ARM

My preferred method now for building packages on the TS-7xxx series of single board computers (SBC's) is to use the gentoo stage3 filesystem as an NFS root, which comes with a C/C++ compiler and most of the necessary utilities. It is much slower, but this way I don't have to work around any strange cross-compiling issues or annoying pkgconfig native-executable requirements which tend to pollute configure scripts for most higher level applications.

The target I'm building for is an ARM 920T processor from cirrus - the EP93xx series of processors - which have only recently acquired a decent level of support in the Linux kernel. Also, as of gcc-4.1.1 the Thumb instruction set is supported as well, although I don't really use it. The ARM EABI is quite useful for those that require floating point functionality, but most of what my company does requires integer math for the most part. For that reason, the stage3 filesystem I use for my NFS root is the arm-softfloat-linux-uclibc stage3 filesystem located here:

http://adelie.polymtl.ca/experimental/arm/embedded/stages/stage3-arm-uclibc-softfloat-20050811.tar.bz2

However, please feel free to select any Gentoo mirror from this site:

http://www.gentoo.org/main/en/mirrors.xml

I've posted several sets of instructions for those that would also like to use the 2.6 kernel I've compiled for the TS-7xxx boards as well. If you're interested, please see the mailing list archives:

http://tech.groups.yahoo.com/group/ts-7000/message/6574

You can also download the kernel directly from here:

http://vaiprime.visibleassets.com/~cfriedt/zImage-2.6

and the config, here:

http://vaiprime.visibleassets.com/~cfriedt/linux-2.6.20.6.config

I recently changed my kernel configuration to include swap functionality, specifically because compiling the gnu classpath for the arm used over 3 times as much memory than was available on the board.

The tricky part though, is that Linux doesn't play very well with swap files on an NFS filesystem. But after doing a bit of googling, I found that there is 1 commonly used work-around.

# dd if=/dev/zero of=/mnt/swapfile bs=1024 count=$((1024*256))
# losetup /dev/loop0 /mnt/swapfile
# mkswap /dev/loop0
# swapon /dev/loop0

In the above section, I've created a 256MB swap file for the linux kernel, and it is accessed by the kernel through a loop device, which offers some apparently greater level of control than simply using swapon /mnt/swapfile.

I'm still compiling the gnu classpath, and I'm not sure exactly how far along the process is, but here are some of the memory usage figures:

=========================
MemTotal: 62192 kB
MemFree: 2728 kB
SwapCached: 12172 kB
SwapTotal: 262136 kB
SwapFree: 127772 kB
=========================

I'll continue to update this blog post as the compilation proceeds, but it's already been going for 2 hours steadily...

Progress with Work

I've accomplished several things for work - that is several programs compiled for my new uclibc root for the arm boards / TS-7xxx series - including

• getting nfs-mounting to work
• building / testing OpenVPN
• building troll-ftpd (needs testing)

I've also started a document to release to the ts-7000 group on how to do this from scratch. The cool part is that all of the configuration data, patches, binpkgs, etc are simply stored in /tinygentoo/etc/portage /tinygentoo/etc/make.conf /usr/portage/local/overlay and /usr/portage/local/binpkgs.

Troll-ftpd was compiled all-right, but I can't sign in as root, even though I could with the TS distro. I think I might have to write another patch for ftpd.c so that it properly checks passwords without crypt().

I also re-built openssl, but it seems a little bit bloated with the gentoo installation, and with the new 'engines' structure. I'm hopefully going to get rid of all of that, because really, we don't need all of the ciphers and we definitely don't need any of the engines because we have absolutely no crypto-specific hardware. Have to figure out what's up with that, so i'm waiting to hear back on the openssl list.

Still need to create a custom ebuild for JamVM - or even get it to build for the arm under uClibc for that matter :p

Also need to create a custom ebuild for the GNU Classpath, considering 90% of it goes to waste on our boards (and it's just fat anyway).