20150217

MappedByteBuffer.hurray()!: Programming the Linux Framebuffer in Java & VMFlexArray Explained

I recently travelled to Belgium to participate in FOSDEM. This year, I gave two presentations:
  • MappedByteBuffer.hurray()!: Programming the Linux FrameBuffer in Java & VMFlexArray Explained. See here.
  • Internet of #allthethings: Using GNURadio Companion to Interact with an IEEE 802.15.4 Network. See here.
The topic of this post will focus on the former. Specifically, VMFlexArray.

Currently, there is no Java Virtual Machine in existence that allows a developer to reference off-heap memory regions as Java arrays - e.g. via byte[] or int[]. That is, all java arrays that the VM deals with must be contiguously allocated at instantiation time.

What I mean by that, is that when the VM instantiates an integer array object, e.g. int[] x = new int[ length ], it typically allocates memory (now careful, I'm going to use some C teriminology here) for an object struct (2 uintptr_t in JamVM) which represents the instance of the int[] object, followed by 1 uintptr_t, which represents the length of the int[] object, followed by exactly length uintptr_t items (on a 32-bit machine) or length / 2 uintptr_t items (on a 64-bit machine) to represent the data.

VMFlexArray


VMFlexArrays are slightly different. For the same case as above, where a new int[] is allocated on the Java heap, the VM would allocate an object struct (2 uintptr_t in JamVM) which represents the instance of the int[] object, followed by 1 uintptr_t to represent the length of the int[] object, followed by 1 uintptr_t to point to the int[] data, followed by the data itself.

What makes VMFlexArrays different, and what makes them flexible (and arguably way better than what most JVMs use today) is that they include that extra uintptr_t to point to the data which could exist anywhere in virtual memory. That means, obviously, VMFlexArrays can point to contiguous data that the JVM would allocate for a regular array, but it also means that it can point to an arbitrary location - and still cooperate with the garbage collector. Indeed, the object lifecycle remains unchanged for VMFlexArrays if the garbage collector avoids releasing memory regions with free(3) if the VMFlexArray pointer does not point to the next contiguous memory address.


VMFlexArray is a solution I came up with that allows one integrate off-heap memory regions into the Java Virtual Machine - e.g. a native external thread that allocates memory using malloc(3), or pages mapped from a device such as /dev/video0 using mmap(2).

Buffer Views


Perhaps the aspect of VMFlexArrays that I found most useful, that I somehow forgot to mention during my talk, is that they rather trivially allow the following code snippet to work as expected. Specifically, an IntBuffer derived from a ByteBuffer with a backing array should be able to provide an int[] backing array view of the same virtual memory.


Currently this code, which should work pretty seamlessly, fails miserably.


ByteBuffers allow themselves to be viewed as IntBuffers or LongBuffers or ShortBuffers. Pretty brilliant! Well... if it worked it would be brilliant. The fact is, as shown above, one cannot wrap a byte[] into a ByteBuffer, view it as an IntBuffer, and then call IntBuffer.array() to get an int[] view of the original byte[]. That would make the NIO API complete, in my opinion, and this feature is sadly lacking.

With VMFlexArrays, that problem is solved.


I've even used this code to memory map the Linux FrameBuffer and animate a bunch of bouncing balls :-) It works quite well.


There's even a massive speedup associated with access to the underlying byte[] from a ByteBuffer and even more so viewing the ByteBuffer as an IntBuffer, with access to the underlying int[].


I am definitely interested in enabling these changes to make it into OpenJDK, and I feel that the community at large would benefit greatly from them. As my time is rather limited these days, I might prefer to mentor a student to make these changes in the Google Summer of Code, 2015, if OpenJDK was a mentoring organization. Otherwise, I would be open to mentoring a student under the umbrella of JamVM or GNU Classpath as a mentoring organization.

20150208

Internet of #allthethings: Using GNURadio Companion to Interact with an IEEE 802.15.4 Network

I recently travelled to Belgium to participate in FOSDEM. This year, I gave two presentations:

  • MappedByteBuffer.hurray()! Programming the Linux Framebuffer in Java & VMFlexArray Explained. See here.
  • Internet of #allthethings: Using GNURadio Companion to Interact with an IEEE 802.15.4 Network. See here.
The topic of this post will focus on the latter.
The gist of my talk was essentially that we have all of the tools available for us to quickly prototype all sorts of 802.15.4 devices. All that is needed is to integrate the following:

  • FreakZ: A BSD-licensed ZigBee stack (for non-commercial purposes)
    • Easily modified to communicate to a GNURadio device via UDP (github)
    • Note: this stack is not certified.
  • GNURadio
  • GNURadio IEEE 802.15.4 Out Of Tree (OOT) module
    • gr-ieee-802_15_4 is available today
    • based on work originally from UCLA
    • unofficially meets all of the mandatory requirements for the IEEE 802.15.4 PHY layer
    • meets some of mandatory requirements for the IEEE 802.15.4 MAC layer
    • lacking mandatory MAC features such as
      • Beacon Management
      • Receive Beacons
      • Channel Access Mechanism
        • Carrier Sense Multiple Access with Collision Avoidance (CSMA-CA)
      • ACK Delivery
      • Security
      • Orphan Scanning
      • Store One Transaction
The primary barrier-to-entry for developers & researchers is most likely going to be the cost of an SDR. Even after buying an SDR that is capable of sampling at a sufficiently high rate around 2.4GHz, it still requires some minimal amount of investment in other 802.15.4 equipment such as ZigBee enabled thermostats, light bulbs, or gateways (I am only aware of ZigBee consumer products in the IEEE 802.15.4 market today).

To assist would-be developers in overcoming that hurdle, what I have done is simply used my USRP B200 to record real-world 802.15.4 traffic produced from an EM370 in NodeTest mode. This should easily facilitate offline signal processing using e.g. GNURadio (see File Source block), Matlab, Octave, or any other programming language. The block diagram for doing so is depicted below. I have intentionally made all of my variables directly obvious.


Keep in mind, that the the files are rather large (433 MB compressed with LZMA2) as the samples are complex-float32 and I have oversampled at a rate of 4x (8M samples per second) intentionally to better facilitate SDR receiver design. You may find them here.

The files are listed and described below:

  • ieee802154-channel14-txtone-complex-float32.dat
    • simply recording a tone at 2420 MHz in the presence of noise
    • note: there is a slight frequency offset which will need to be corrected


  • ieee802154-channel14-txstream-complex-float32.dat
    • a continuous random stream of valid channel symbols

  • ieee802154-channel14-tx-complex-float32.dat
    • a stream containing intermittent & full IEEE 802.15.4 frames
    • frames are sent once every 25500 us

A Few Notes About the Current State of IEEE 802.15.4 in GNURadio

All of the open-source PHY implementations assume that Symbol and Timing Recovery (STR) is already performed. This is fine for simulation (depicted below).

Indeed, clock recovery, frequency offset compensation, and phase offset compensation are often the most complicated part of real-world wireless receiver architectures. Without frequency compensation, the constellation diagram appears to move around the unit circle, as shown below.


For those who would like to get started quickly, you may follow the PSK Symbol Recovery tutorial. Use a Polyphase Clock Sync block, followed by the "blind" Constant Modulus Algorithm (CMA) equalizer, followed by Costas Loop. The Polyphase Clock Sync takes the taps for the matched filter as an argument, the filter length, as well as the number of samples per channel symbol, and returns a configurable number of samples per symbol with a (somewhat) frequency-corrected clock. The CMA equalizer then forces all of the samples onto the unit circle, and finally the Costas Loop corrects the phase of the signal. This approach works well enough but it has some associated complexity. The clock recovery, frequency, and phase compensation work independently. It has been suggested that using the Least Mean Squared Decision-Directed (LMS-DD) equalizer could improve performance. The Polyphase Clock Sync block can be more accurate, sacrificing time in the receive chain due to increased complexity.

However, since we already know the preamble of an IEEE 802.15.4 packet in the 2450 MHz ISM region, we are at liberty to implement a more sophisticated coherent architecture in our receiver. Specifically, due to the known preamble, we may use a Correlate and Sync block.

I will be doing a bit more experimenting in the coming days and will post an update once available.

20120927

Two Jedi-Fu Links for Installing Gentoo Quickly



  1. Install LiveDVD to Hard Drive
    (works with 12.1 too)
  2. Remove Pam
    (so passwordless root logins over SSH work)
    (I typically use this for VMs when hacking Gentoo-ish stuff)
    (... this is not the IP address you're looking for...)

20120615

What an Amazing Time Warp!

So when I was a teenager (years and years ago), my cousin John Muller & I played with Matt Howatt and a guy from Australia named Miles, and we covered the song Tinfoil by Limblifter. I played the bass, and Matt & John switched up for rhythm & lead guitar for every other song.

Tonight I had the privilege of seeing some of the original band from Limblifter playing in Hamilton.

It was a fantastic experience to be brought back to the music I loved in my youth and continue to love in adulthood.

Particularly, when I used to play this song, I played the bass, and I was certainly happy to meet the new female bass player tonight, who is certainly overqualified for this set. She has an MA I. Music specializing in the double bass. Aside from the cool factor of a 'chick' bassist, she was doing an amazing job with backup (and in some cases, lead) vocals.

This has probably been the best 'feel good' show for me this summer, thus far.

Thanks again  to Limblifter for putting on a full energy show!


20120611

Cook Beans: Check ;-)


To Do: Cook Beans!


For anyone who thinks that going vegan is expensive... totally not the case.

If you are vegan and don't already own one, or if you're considering going vegan and have no idea where to start, investment #1 should be a pressure cooker. Not very many people (in Canada?) own one, but they're soooo common in India, and I've got my indian friends to thank for turning me on to mine.

You can buy massive dried bags of lentils of various colours (dal), kidney beans (rajma), chick peas (chana), or black beans for next to nothing. Compare the contents of one bag ($4 maybe?) to the same mass of beans in 20 (?) cans of water for >= $1 each!

And due to the simple equation from high school that most have forgotten, PV=nRT [1], you can cook dried gram at a much lower temperature if the pressure is increased (V, n, & R are held constant). This saves on electricity (or gas as the case may be). I'll be the first to admit, I used to cook dried beans like a complete idiot and soak them overnight, and then boil them for 5 hours or something. With a pressure cooker, there is no need to soak legumes overnight and cooking them takes about 1 hr. Most importantly, it completely seals in the flavour for whatever meal you have planned: burritos (like me), kaali, masoor, or tadka dal - or good, old-fashioned soups (e.g. lentil, split-pea)!

A nice side effect: dried legumes take far less energy to transport than cans of legumes that are 60% salt and water; you're going green [2]!

Naturally, not everyone has the proper body chemistry to be able to go vegan with great success, but here is one attractive figure that might catch your eye:

Marathon runners should consume about 1.5 grams of protein per kg of body weight per day [3].

For a 180 pound guy like me (~80kg according to Google's conversion), that's about 50g of chick peas. What is that for a total protein bill per day - about 25¢ ??

Not bad at all :)

[1] The Ideal Gas Law (Influences amazing things! E.g. how fast a pot of water takes to boil)
[2] Vegetarian Diets Help Protect the Environment
(As if you didn't already know how green a vegan diet is!!)
[3] Sources of Protein for Runners: Spring Marathon Training Plan Part 5




Disclaimer: I'm not totally vegan these days (I now eat fish & other seafood), but I was for ~8 years or so. It was personal preference and the evolution of my diet. I like the great perspective it gave me on international foods though, along with environmental and even spiritual perspectives. I just want to feed my family well, and sometimes (really just for my little man) that does include some animal products 

20111216

Yum ;-)

A nearly-vegan pad thai (no eggs, but crab & otherwise some fish sauce).


20111129

Two Thumbs Up to Koush

I just installed & tweeked Koush's Cyanogen Mod 9 alpha 11 for my Nexus S running Ice Cream Sandwich from AOSP. You can read his original message on the XDA forums.

For an engineering build it's running quite smoothly. I'm certainly looking forward to mirroring his git and building a userdebug variant. Typically this speeds up most aspects of the UI and other software since debug messages are less frequent.

A few pointers:

  • If you feel that the boot animation is continuing infinitely, you probably forgot to erase the data partition (I did, originally)
  • If you like Google Car Home, which is a great home-replacement while driving, then install it manually from a backed-up apk and install the 3rd party Car Mode Control app. The original google car home will install, but it is claimed not to work with ICS and does not show up in the launcher so there is no way to use it unless Car Mode Control is installed.
  • If you encounter "Unfortunately, Google TTS Engine has stopped" Select PicoTTS to get Navigation and My Tracks to work.

20111111

An Update

Just to dispel any confusion that might arise if people google me to dig up 'dirt'.

Yes, I am in fact single again. Yes, there may be photos of me & my ex floating around on the interwebs that seem to be very recent - and they are! I only did become single again recently, and it was a bit sudden for me too!

So is life, however, and it does go on, as they say.

It's actually really surprising how little I'm blogging these days, with the major shift changing from blogging to micro-blogging via Twitter, FaceBook, and Google+ . Sorry if I haven't been terribly vocal.

Just thought I would post an update.

Ciao!

C

20110831

Google Nexus S Android Phone Suffers USB Death

Today I am a very lucky guy, in spite of the fact that the USB OTG functionality on my Google Nexus S has just vanished. Why does that make me lucky? Well, it doesn't, but luckily I had ordered a second Nexus S that works properly with Canadian HSDPA frequencies on the Rogers network, and it arrived literally the same second that my old Nexus S stopped working. Total coincidence... I think.

Actually, I should be more specific - nothing on the old phone has stopped working except for the USB OTG controller. So when I plug my in my phone to my workstation, I receive "unable to enumerate device on port... " under Linux and "USB Device Not Recognized" under Windows 7. In layman's terms, my phone no longer works as a USB disk. 

Being a clever hacker, I managed to get the dmesg output on my Nexus S, which I hope will be of some use for people at either Google or Samsung. The source code for the Nexus S (codenamed crespo) is available, so I might look into it further some time later. 

In reference to the dmesg output below, 

"The FSA9480 chip is used on some Samsung phones to detect various accessories using sensing resistors on the ID pin of the USB port." [1]
It's likely an authentication chip - there are several vendors that provide something similar. The datasheet is only available under NDA of course. Naturally, I've tried several cables, and several different workstations, to no avail. There are many layers of software and electronics at work (or rather not at work) here, so without some chip documentation / resources, there's not really a way that I can debug this just using the source code of the Linux driver.

But just to clarify - anytime an 'err -6' appears, that corresponds to with -ENXIO "No such device or address", and anytime 'err -5' appears, it corresponds to  -EIO "I/O error", so the outlook is not good. It's likely that the chip in question had a bad solder joint and is not powering up with the rest of the phone, or it experienced a "massive" current spike from my laptop that it couldn't handle.

I believe my device is still under warranty though, so I'll certainly be shipping it back for repair soon. 

The strangest part of this whole thing is that I've kept my phone in pristine condition, which (sadly) suggests that the design quality of this Nexus S was lacking, in spite of what Google has said in various marketing videos. It could also be Fairchild's fault, if in fact it turned out to be an ESD issue. Anything that's connected on the USB should be able to handle 'typical' levels of ESD from a PC's USB pins. 

[ 2066.560942] fsa9480 7-0025: dev1: 0x0, dev2: 0x0
[ 2068.234132] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2068.362855] fsa9480 7-0025: fsa9480_irq_thread: err -6
[ 2068.363813] fsa9480 7-0025: fsa9480_detect_dev: err -6
[ 2068.363919] fsa9480 7-0025: dev1: 0xfa, dev2: 0xff
[ 2068.374642] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2068.377227] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2068.377957] i2c i2c-7: sendbytes: NAK bailout.
[ 2068.378051] fsa9480 7-0025: fsa9480_reg_init: err -5
[ 2068.379090] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2068.975317] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2068.977052] fsa9480 7-0025: dev1: 0x1f, dev2: 0xff
[ 2068.978903] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2068.991221] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2068.992016] i2c i2c-7: sendbytes: NAK bailout.
[ 2068.992109] fsa9480 7-0025: fsa9480_reg_init: err -5
[ 2068.993178] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2068.994968] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2069.003127] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2069.004982] fsa9480 7-0025: dev1: 0x10, dev2: 0x3
[ 2069.006708] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2069.008463] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2069.009962] i2c i2c-7: sendbytes: NAK bailout.
[ 2069.010054] fsa9480 7-0025: fsa9480_reg_init: err -5
[ 2069.010498] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2069.012253] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2069.014000] fsa9480 7-0025: dev1: 0x10, dev2: 0x1f
[ 2069.015793] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2069.035247] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2069.037069] fsa9480 7-0025: dev1: 0xff, dev2: 0xff
[ 2069.038796] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2069.041289] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2069.043017] fsa9480 7-0025: dev1: 0x10, dev2: 0xff
[ 2069.044765] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2069.048313] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2069.050040] fsa9480 7-0025: dev1: 0x17, dev2: 0xff
[ 2069.051853] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2069.060308] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2069.062035] fsa9480 7-0025: dev1: 0x13, dev2: 0xff
[ 2069.063858] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2069.067409] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2069.068963] i2c i2c-7: sendbytes: NAK bailout.
[ 2069.069124] fsa9480 7-0025: fsa9480_detect_dev: err -5
[ 2069.069209] fsa9480 7-0025: dev1: 0xfb, dev2: 0xff
[ 2069.070971] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2069.091186] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2069.092983] fsa9480 7-0025: dev1: 0x10, dev2: 0x3
[ 2069.094731] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2069.104292] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2069.105018] i2c i2c-7: sendbytes: NAK bailout.
[ 2069.105108] fsa9480 7-0025: fsa9480_reg_init: err -5
[ 2069.106102] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2070.150280] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2070.152079] fsa9480 7-0025: dev1: 0xff, dev2: 0xff
[ 2070.153836] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2070.166528] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2070.167026] i2c i2c-7: sendbytes: NAK bailout.
[ 2070.167188] fsa9480 7-0025: fsa9480_reg_init: err -5
[ 2070.168389] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2070.192257] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
[ 2070.193986] fsa9480 7-0025: dev1: 0x10, dev2: 0x1
[ 2070.195786] fsa9480 7-0025: dev1: 0x10, dev2: 0x0
...
[ 2070.210682] fsa9480 7-0025: dev1: 0x10, dev2: 0x0


[1] http://electronics.stackexchange.com/questions/13041/how-do-i-make-the-fairchild-fsa9280-fsa9480-fsa880-boot-pin-trigger-samsung-gal