My experiences with Qemu - Part II

So I've managed to get this far with Qemu:

(qemu) Uncompressing Linux......................................................... done, booting the kernel.
Linux version 2.6.20.6 (cfriedt@sith) (gcc version 3.4.4) #35 Thu May 31 21:57:50 CEST 2007
CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00003137
Machine: ARM-Versatile PB
Memory policy: ECC disabled, Data cache writeback
CPU0: D VIVT write-through cache
CPU0: I cache: 4096 bytes, associativity 4, 32 byte lines, 32 sets
CPU0: D cache: 65536 bytes, associativity 4, 32 byte lines, 512 sets
Built 1 zonelists. Total pages: 65024
Kernel command line: root=/dev/nfs nfsroot=192.168.7.1:/usr/gentoo_root,uid=0,gid=0,rsize=32768,wsize=32768,timeo=14,nfsvers=3,rw rw ip=192.168.7.2:192.168.7.1:192.168.7.1:255.255.255.0:qemu console=ttyAMA0,115200
PID hash table entries: 1024 (order: 10, 4096 bytes)
Console: colour dummy device 80x30
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 256MB = 256MB total
Memory: 258048KB available (1516K code, 174K data, 100K init)
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
NET: Registered protocol family 16
PCI core found (slot 11)
PCI: bus0: Fast back to back transfers disabled
PCI map irq: slot 0, pin 1, devslot 12, irq: 27
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 8192 bind 4096)
TCP reno registered
NetWinder Floating Point Emulator V0.97 (double precision)
CLCD: unknown LCD panel ID 0x00001000, using VGA
CLCD: Versatile hardware, VGA display
Clock CLCDCLK: setting VCO reg params: S=1 R=99 V=98
Console: switching to colour frame buffer device 80x30
Serial: AMBA PL011 UART driver
dev:f1: ttyAMA0 at MMIO 0x101f1000 (irq = 12) is a AMBA/PL011
dev:f2: ttyAMA1 at MMIO 0x101f2000 (irq = 13) is a AMBA/PL011
dev:f3: ttyAMA2 at MMIO 0x101f3000 (irq = 14) is a AMBA/PL011
fpga:09: ttyAMA3 at MMIO 0x10009000 (irq = 38) is a AMBA/PL011
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky
smc91x.c: v1.1, sep 22 2004 by Nicolas Pitre
eth0: SMC91C11xFD (rev 1) at d080a000 IRQ 25 [nowait]
eth0: Ethernet addr: 52:54:00:12:34:56
mice: PS/2 mouse device common for all mice
TCP cubic registered
ieee80211: 802.11 data/management/control stack, git-1.1.13
ieee80211: Copyright (C) 2004-2005 Intel Corporation
input: AT Raw Set 2 keyboard as /class/input/input0
eth0: link up
input: ImExPS/2 Generic Explorer Mouse as /class/input/input1
IP-Config: Complete:
device=eth0, addr=192.168.7.2, mask=255.255.255.0, gw=192.168.7.1,
host=qemu, domain=, nis-domain=(none),
bootserver=192.168.7.1, rootserver=192.168.7.1, rootpath=
Root-NFS: unknown option: uid=0
Looking up port of RPC 100003/2 on 192.168.7.1
Looking up port of RPC 100005/1 on 192.168.7.1
VFS: Mounted root (nfs filesystem).
Freeing init memory: 100K
init: must be superuser.
Kernel panic - not syncing: Attempted to kill init!

It's looking fairly bright now ;-) All I have to do is figure out why (this has absolutely no real reason) Linux would start up as someone who isn't the superuser
... hmmf..

Weeee !!! I just added an init= option to the 'append' section of Qemu!!! sahweet!!! She's ready to roll ;-)

The option to enable 16-bit UID system calls must be selected as well, otherwise root (0) will get mapped to 0xffffffff (-1 in 2's compliment == -EPERM).

My experiences with Qemu - part I ... was "Building GNU Classpath for the ARM (Cont'd)"

Ok, so it looks like the external USB hard disk idea was not as good as I was hoping. The problem is that the board just has too little RAM !!! (to compile such a package as the GNU Classpath)

What happens in either the NFS case or the USB HD case, is that eventually the swapd and the jikes process enter this vicious cycle, where jikes caches the first half of what it needs to compile, swaps it out, and then caches the second half of what it needs to compile - clearly if it doesn't have 100% of the things it needs to compile in RAM (this occurred even without using -pipe) then it will enter an infinite loop (it would be nice if that was detected by jikes - i think that's CRC32 or something?).

In any event, with the suggestion that my boss Dave gave me, I'm now working on emulating an appropriate kernel, and using the appropriate uclibc-softfloat userspace for the ARM, using Qemu. This should solve all of my compilation woes because I would then be able to have enough actual physical RAM to compile everything natively. I believe that the VersatilePB platform available in Qemu has a limit of 256MB RAM though. That should be enough (I assume).

Aside from the target having a different name, the main other differences between the TS-7xxx boards and the Versatile PB are:

• The Versatile PB uses an ARM926T chip instead of an ARM920T
• The Versatile PB supports an SMC 91c111 ethernet device instead of the ep93xx device.
• The Versatile PB names its serial devices /dev/ttyAMA[0-n] instead of /dev/ttyAM[0-n]

All of the exact details are here:

http://fabrice.bellard.free.fr/qemu/qemu-doc.html#SEC53

Now, a brief overview of the differences and what they will mean.

1. The ARM926T chip vs the ARM920T chip. To quote from the linux kernel help page:
   

   This is a variant of the ARM920. It has slightly different instruction sequences for cache and TLB operations. Curiously, there is no documentation on it at the ARM corporate website [EDIT: There is now].
   
   Say Y if you want support for the ARM926T processor. Otherwise, say N.

   
   I tried compiling for the Versatile PB without ARM926T support and that didn't even compile - it would be nice to have the ARM920T as an option here too. Anyways, that's not really a problem. As long as I can run Qemu with an ARM chip emulated and use an NFS root to compile, that's all that matters.
   
   
2. eth0 is eth0 ... I doubt it matters what hardware is underlying for my purposes.
   
   
3. Some scripts and applications will have to be aware of the difference of platform, if i end up coding scripts and things of the like on the emulator. I just as soon wouldn't, but it could be quite beneficial. In any event, that's not a real issue, it's as simple as replacing a string from my perspective. The cool part about that is that we could end up simulating our hardware, including the EM noise model - cool ;-)
   

I'll keep you posted!

Building GNU Classpath for the ARM (Cont'd ...)

I spontaneously decided to copy over the entire Gentoo stage3 filesystem to my external hard-disk, which is recognized as a general USB storage device under linux. NFS was taking waaaaay too long for this compilation. I had checked everything out with top, under linux, played with the swappiness parameter in /proc/sys/vm, and also attempted to renice the jikes processes' priority to a whopping 17! Non of those had a major effect on the cpu-usage of the jikes process. The highest that I'd seen it was 10% for a whole 2 seconds.

The problem lies with NFS root and the overhead associate not only with RPC, but also the fact that my swap was also on the NFS root, effectively squaring the complexity of the overhead. With my Gentoo stage 3 root filesystem located on a locally attached usb storage device, along with the swap file, the speed of compilation increased dramatically! The jikes process has just started and its already achieved 50% of the cpu time! That's up from an average of maybe 3% with the tweaking I mentioned previously.

I'd better mention this to Dave as well, so that he's not stuck ever repeating my stupid mistake of compiling on an NFS root!

Building the GNU Classpath for ARM

My preferred method now for building packages on the TS-7xxx series of single board computers (SBC's) is to use the gentoo stage3 filesystem as an NFS root, which comes with a C/C++ compiler and most of the necessary utilities. It is much slower, but this way I don't have to work around any strange cross-compiling issues or annoying pkgconfig native-executable requirements which tend to pollute configure scripts for most higher level applications.

The target I'm building for is an ARM 920T processor from cirrus - the EP93xx series of processors - which have only recently acquired a decent level of support in the Linux kernel. Also, as of gcc-4.1.1 the Thumb instruction set is supported as well, although I don't really use it. The ARM EABI is quite useful for those that require floating point functionality, but most of what my company does requires integer math for the most part. For that reason, the stage3 filesystem I use for my NFS root is the arm-softfloat-linux-uclibc stage3 filesystem located here:

http://adelie.polymtl.ca/experimental/arm/embedded/stages/stage3-arm-uclibc-softfloat-20050811.tar.bz2

However, please feel free to select any Gentoo mirror from this site:

http://www.gentoo.org/main/en/mirrors.xml

I've posted several sets of instructions for those that would also like to use the 2.6 kernel I've compiled for the TS-7xxx boards as well. If you're interested, please see the mailing list archives:

http://tech.groups.yahoo.com/group/ts-7000/message/6574

You can also download the kernel directly from here:

http://vaiprime.visibleassets.com/~cfriedt/zImage-2.6

and the config, here:

http://vaiprime.visibleassets.com/~cfriedt/linux-2.6.20.6.config

I recently changed my kernel configuration to include swap functionality, specifically because compiling the gnu classpath for the arm used over 3 times as much memory than was available on the board.

The tricky part though, is that Linux doesn't play very well with swap files on an NFS filesystem. But after doing a bit of googling, I found that there is 1 commonly used work-around.

# dd if=/dev/zero of=/mnt/swapfile bs=1024 count=$((1024*256))
# losetup /dev/loop0 /mnt/swapfile
# mkswap /dev/loop0
# swapon /dev/loop0

In the above section, I've created a 256MB swap file for the linux kernel, and it is accessed by the kernel through a loop device, which offers some apparently greater level of control than simply using swapon /mnt/swapfile.

I'm still compiling the gnu classpath, and I'm not sure exactly how far along the process is, but here are some of the memory usage figures:

=========================
MemTotal: 62192 kB
MemFree: 2728 kB
SwapCached: 12172 kB
SwapTotal: 262136 kB
SwapFree: 127772 kB
=========================

I'll continue to update this blog post as the compilation proceeds, but it's already been going for 2 hours steadily...

Gun-Related Deaths in the Toronto Area

Yesterday at 2:30pm EST, a 14-year-old boy was shot in the chest close to his high school. Here is a link to one the Toronto Star articles:

http://www.thestar.com/News/article/216948

He died only one and a half hours later at Sunnybrook hospital.

I don't really know if what the mayor of Toronto has suggested is even a solution. It might even just aggravate such violence even more, and force it to occur elsewhere, where a kid wouldn't even have a chance to be treated at the hospital.

When I lived right downtown, on Gerrard & Yonge, there was a guy who sold weed 2 floors below me. I never met him, but people say he used to enjoy having a lot of parties. He had a lot of expensive stuff in his apartment, like a plasma TV, stereo system, etc. I guess rumors spread to the wrong people that he kept a lot of weed and money at his place too. Then some kids decided to break in - only he was there when they expected him to be out. He tried to resist and was shot and killed instantly, while his girlfriend sat in the bathroom, terrified.

http://theeyeopener.cfhosting.ca/storydetail.cfm?storyid=1940

I was at the office late, working, but I remember coming home and seeing the entire street filled with police cars. The police vehicles were still casting this eerie periodic blue and red strobe on on all of the nearby buildings. Then I walked in to the building just as the guy's girlfriend was being escorted out by police, and she was in complete shock. Her eyes were all red from tears. I felt so sad for her.

My next instinct when I got home was to lock my door as soon as I got in, fearing that the shooters would be waiting somewhere in the stairwells for a chance to escape. I usually never lock my door - granted, I usually also don't have anything to steal either.

Then only later that year - perhaps at the worst possible time it could happen - a 15 year old girl was killed in a shooting the day after Christmas.

http://www.thestar.com/article/206901
This happened right around the corner from my house, in perhaps the busiest shopping area in the entire city - on boxing day no less. She was only out shopping with her mother when some guy across the street started unloading trying to kill this young gang member that she just happened to be standing close to. The target actually escaped unharmed, but the little girl was killed instantly. Another bystander was taken to hospital for a gunshot wound.

After that, I knew that I no longer liked living in my neighborhood, so I moved to little italy. You'd think that gun violence like that only happens in the under-priviliged areas of a city - but that's wrong. Although I believe that the most gun violence in Toronto occurs in the suburbs. Where I lived at Yonge & Gerrard, the rent was not cheap and I wouldn't call it an underpriviliged area by any means. Same with in Scarborough. I wouldn't call that a project. This sort of thing doesn't happen only in publicly subsidized housing areas, which is what I would consider stereotypical.

Then there was the college in Montreal that was shot up a la Columbine, although that's a completely different story.

What I find particularly disturbing about many of the shootings in Toronto, is that the victims or aggressors are quite often teenage kids who've gotten themselves involved in gangs or selling pot or whatever. The bottom line is that kids need to be discouraged from getting involved. They need to be shown that getting involved in this kind of thing is dangerous. Kids don't care that whatever is punishable by 6 months in juvenile hall, but when you show them a picture of some 14 year old kid who was killed over turf or selling nickel-bags or something, and then you show them pictures of that kids family, and how empty they become... that's when they realize that getting involved in gangs or dealing has repercussions.

I'm not saying that the victim of the latest shooting was involved in something illegal. I have no idea. I've never met the kid. What's obvious, however, is that the person who shot him wanted him dead for one reason or another.

Kids today, and especially those that are underprivileged (in a _local_ sense - relative to their peers), need to have other things to occupy their time... I would encourage the city of Toronto to concentrate on that. Spend some money - buy a couple of used Playstations or something like that for the local community center, where you know families can't afford one. Buy 20 soccer balls and show up one day at the school to give them away. Maybe buy a few violins and start an after school music program. I'm sure that there are 1,000 better things a kid could be doing with his or her time than trying to make money selling pot, or looking somewhere to buy a gun.

I mean, maybe putting 5 police officers in the hallway at these schools would entice people to keep violence away from the school, but that doesn't mean it's not just going to happen somewhere else. These kids just need to realize that they have better things that they could be doing with their time.

I highly doubt that the family and loved-ones of the boy that died will read this, but if you do, you have my condolences.

Gnome Issues

I'm still having this frustrating thing with gnome when I try to open certain file types with a double-click. Here is what I posted to the Gentoo Forums.

I just upgraded to =x11-misc/shared-mime-info-0.20 and the problem persists with gnome-2.16.2 (the 2007.0 profile).

Specifically, I still have issues with "OpenDocument Text" and "PDF document", but likely many more.

Here is a list of extensions, expected mime-types, and actual mime-types

Extension	'Filename Suggests'	'Contents Suggest'
.odt	odt document	OpenDocument Text
.pdf	pdf document	PDF document

(The above would actually display as a table, but for some reason I'm still considered a n00b in these forums - NaN := Not a n00b <=> me )
[Edit: The table works perfectly with BlogSpot ;-)]

In any event, double-clicking on any of the above file types will generate an error of the form:

Cannot Open {insert filename}

The filename "{insert filename}" indicates that this file is of type "{insert from column 1}". The contents of the file indicate that the file is of type "{insert from column 2}". If you open this file, the file might present a security risk to your system.

Do not open the file unless you created the file yourself, or received the file from a trusted source. To open the file, rename the file to the correct extension for "{insert from column 2}", then open the file normally. Alternatively, use the Open With menu to choose a specific application for the file.

Is there some special gnome magic that needs to be performed for gnome to realize that it should be using the updated x11-misc/shared-mime-info ? Or is this some obscure preference that only exists in a gconf schema?

I will pay 50,000,000 * 1 / 1,000,000 ths of a peso to anyone who can solve this issue, and that is a promise!! I have the cash sitting on top of my desk as I write this!!, and that is a promise!! I have the cash sitting on top of my desk as I write this!!

Progress with Work

I've accomplished several things for work - that is several programs compiled for my new uclibc root for the arm boards / TS-7xxx series - including

• getting nfs-mounting to work
• building / testing OpenVPN
• building troll-ftpd (needs testing)

I've also started a document to release to the ts-7000 group on how to do this from scratch. The cool part is that all of the configuration data, patches, binpkgs, etc are simply stored in /tinygentoo/etc/portage /tinygentoo/etc/make.conf /usr/portage/local/overlay and /usr/portage/local/binpkgs.

Troll-ftpd was compiled all-right, but I can't sign in as root, even though I could with the TS distro. I think I might have to write another patch for ftpd.c so that it properly checks passwords without crypt().

I also re-built openssl, but it seems a little bit bloated with the gentoo installation, and with the new 'engines' structure. I'm hopefully going to get rid of all of that, because really, we don't need all of the ciphers and we definitely don't need any of the engines because we have absolutely no crypto-specific hardware. Have to figure out what's up with that, so i'm waiting to hear back on the openssl list.

Still need to create a custom ebuild for JamVM - or even get it to build for the arm under uClibc for that matter :p

Also need to create a custom ebuild for the GNU Classpath, considering 90% of it goes to waste on our boards (and it's just fat anyway).

Things on the To-Do List for work

I have to push a new filesystem and kernel, as well as installation instructions for work.

Currently, we're using the TS-7xxx boards, with a Cirrus EP93xx / ARM 920T processor, for our embedded devices. The trouble is, that the boards ship with a non-standard 2.4 kernel and the bloated glibc library. I've successfully compiled and installed the 2.6.20.6 linux kernel as well as a uClibc userland for the boards, but now I'm in the process of working out bugs and installing other software that is necessary for the basic functionality of our boards.

To-Do:

• Make nfs-mounting work on the boards (recompile uClibc with full RPC, and then busybox)
• Build OpenVPN
• Build an FTP service
• Build JamVM
• Copy over the GNU Classpath that we have
• Build Avahi
• (etc, etc)

You may have guessed that this isn't such a small job. Anyway, I'm going to try to do quite a bit of it today.

Ciao!

Switching to Digital Communications

So, I promised that I would post some newer going-ons about what's happening in Kiel. I have an offer to change my course of study from Informatik (read: computer science, not your local IT department) to Digital Communications (once an engineer always an engineer).

There are a couple of key benefits for me here, the first being that the M.Sc. in Digital Communications is all taught in English. Now, don't get me wrong - I don't want to give you the impression that I'm flaking out because all of my courses at the moment are in German. I really like the courses that I'm taking in Informatik, those which are related to my field of study. In general, however, I've found that Informatik is a bit on the boring and theoretical side. I'm an electrical guy - I actually like to get my hands dirty and _apply_ my knowledge once in a while.

The other major reason why I chose to switch is because having a full time job _and_ studying for a graduate degree is _incredibly_ draining. I ended up having stress-related health problems about a month ago. Since then I made the decision to not kill myself by working too hard. This also means that I've cut down my course load and applied for the DigiComm program. I tend to work about 20 to 30 hours per week. If you add the number of lecture hours I have on top of that, as well as approximately 1-5 times the amount of time required for native-German speakers to study, it adds up to quite a lot.

I'm looking forward to the change.

First 'blogger' entry

"Hello World!"
- anonymous programmer

It was a very hard decision, but I just decided to move my blog to a remote service. Formerly, I had done all of the web-hosting and setup of the blog engine myself from home, using no-ip.com's service and a linux box (www.perpetual-notion.myftp.org) - you wouldn't believe how easy that is. However, now that I'm living my digital life out of my laptop from virtually wherever I am in the world, I feel that having something static would probably be a much wiser idea.

I suppose that there are a few other methods I could have pursued, such as:

1. Hacking the linksys router to incorporate a flash memory reader / writer
2. Hacking the linksys router to incorporate a USB port / external hard disk

Aside from the possibility that I could wreck my Linksys router, or that the flash memory would die in a matter of months, or the hard drive would die because of the crappy external enclosures that I bought, I believe the university is blocking all incoming traffic on port 80 & 443 (http, https). Therefore hopes of hosting my own blog have been somewhat extinguished.

Eric had mentioned that I should continue with my blog, and I guess there are good grounds for that request (the primary reason being that I'm anti-social and don't call home nearly enough). In spite of my anti-social nature, I too sometimes feel somewhat disconnected from the rest of 'the world' back home.

Thus, I went with blogger.com Since Google already has the right to search through all of my email, I would imagine that they would love the opportunity to search through my blog postings as well. However this will probably also be an easier way to publish photos for everyone to see and also to get some feedback from my friends & family abroad.

My current method to communicate is skype, but mainly with Erin, and mailing photos as postcards by simply writing an address and fixing a stamp to the back - that's one of the best parts about digital photography (Erin is really a way better photographer than me, but I still like sending her pictures as postcards too ;-)).

In any event, I'll write something fairly soon so that you can all have some idea about what's happening in my life. That might be tomorrow, because I'm a bit busy with schoolwork and work tonight - I would have loved to go to the danish film that's playing at the university tonight, but I thought that travelling to an island and walking a kilometer out into the north sea was enough fun for one weekend!! heheeh...

ciao 4 now