Running YourKit UI remotely

I got into a somewhat unique situation few weeks ago: I had a JVM heap dump 16GB in size that I had to analyze using a memory profiler. It was obviously hopeless to analyze it on my laptop which only has 4GB of RAM. Fortunately, I was given access to a computer half the world away that had 64GB of RAM. What I did was I moved my YourKit profiler instance onto that machine and tried to run it using X.11 tunneled through SSH.

The good news is: it works.

The bad news is: it needs tweaking to work.

First thing I run into was that the GUI was hideously slow. I would click on a button in the profiler, and feedback would take several minutes. I'm not joking. Several minutes. Googling revealed a Sun Bug Database item "Antialiasing/Compositing is slow on remote display". It suggests that to run GUI Java apps over X.11, one should add

-Dsun.java2d.pmoffscreen=false

as a JVM startup flag. I added it to profiler's yjp.sh launch script, and Holy Moly, it became much better. Granted, it was now rendering screens progressively from top to bottom, but it did it in matter of seconds, not minutes. It now felt like something out of 1999 instead of 1993. Further Googling for various combinations of "X.11" and "slow" and "ssh" turned out sites that suggested it's possible to make x.11 over a bit faster, by telling SSH to use Arcfour or Blowfish ciphers instead of AES, as well as to compress the traffic:

ssh -c arcfour,blowfish-cbc -C -X host

This led to further improvements; the progressive rendering got faster by another factor of two. That's as good as it gets. Now it feels like something out of circa 2001. I can work with that.

I work for Twitter now

I work for Twitter now.

"Why?"

Twitter is an amazing company that provides an amazing service to the humankind. I see it as a fundamental new modus of communication that's quickly becoming part of the global human culture. Whether the earthquake in Haiti or the post-election uprising in Iran, Twitter has proven to be a reliable and prompt medium for getting the latest news on what's happening in the world, directly from the people most involved in the events. (While simultaneously still allowing you to to tell your friends what you had for breakfast, of course.) I appreciate its singular focus on doing one thing right, and that it managed to remain simple to use. Twitter is experiencing a 16% month-over-month (that's exponential!) traffic increase, and I'm very excited at the prospect of throwing my weight behind making it able to cope with this welcome challenge. The company is chock full of extremely bright people that I will no doubt be a delight to work with, and maybe some of their awesome will rub on to me too...

I want to change the world for the better and make a positive influence on people's lives. Often, the most significant changes are the ones people don't even notice, because they make Things Work As They Are Supposed To. How big a change? I have no idea yet, but it could be vital. If it makes the difference between a smooth operation and a Fail Whale for someone caught in the next civil unrest who's using a momentarily available internet connection to let his family know he's okay, it can mean the world to them.

I appreciate Twitter's open-source fostering culture. Today, the barrier to entry for influencing lots of people's lives is actually pretty low. It's called Open Source Software, and it's easy to participate and create serious value. I count that for every hour I spent on my OSS projects, I saved in thousands of man hours to my users, a number that will keep growing until the last of my code goes out of use (will be a while...). The value leverage is just insane. I just never figured out how to make it pay the bills, so I can't devote myself 100% to it. Twitter plays great in the OSS ecosystem, so I hope that I'll be able to create value in two streams: improving Twitter through the OSS it relies on, and as a consequence, improving the said OSS.

"So are you moving to Bay Area?"

Yes. We're working on my visa, and once that's approved (hopefully around October), I'll move with my wife and children to San Francisco to join the company in person. Until then, I'm working remotely from home in Hungary. I'm absolutely thrilled at the prospect of living there, and I'm very much looking forward to this new adventure.

Windows Live nightmare

Last week I purchased a copy of Gears of War - it cost me some 12 bucks in local currency, one of the benefits of only buying 2-3 year old games. Yup, I'm the guy from this comic. So it happened that this weekend I decided to play a bit with it, so went to install it on a Sunday afternoon.

As installs go, it was uneventful, and I got to launching the game.

First snag, it tells me that it needs to install an "AMD Processor Update" otherwise it will crash during gameplay. Okay, let it install it. Reboot machine. Restart the game. I can play now, right?

Wrong.

It again tells me it needs to install the same update. Uh-oh. Okay, give it another try. Reboot machine. Restart the game.

I get to the main screen. Fine. I can play the game now, right?

Wrong.

I attempt to start the campaign, and it tells me that unless I have a Windows Live profile, I won't be able to save games!

Excuse me?!

*Pause*

Oh well, let's create a Windows Live account, I can live with that. It opens a browser, I fill out few forms, and off we go. I can play the game now, right?

Wrong.

Next it tells me that now it needs to update the local Windows Live client binaries. It tells me that I can choose to not update now, but I won't be able to play the game until I do. Yeah, that's great choice, indeed, how very thoughtful of them to give me options.

We're now at about one hour mark from the moment I inserted the install DVD. But once it got updated, I can finally log in and play the game, right?

Wrong.

It now tells me that Windows Live is "not supported in my region." How splendid! If I live in Hungary, I'm not allowed to save my bloody games, is that what you are telling me? Apparently, yes.

Another hour of googling and forum reading ensues. I end up on some dead links at sites named xbox.com and gamesforwindows.com and the likes where I can supposedly change my region but in reality I can't. I end up on a site where finally I can do it (I think it's some long domain name with words "windows" and "account" and "services" in it in some order). I lie that I'm in UK. It seems to accept it. So, I can now play the game, right?

Wrong.

They still claim I'm in "unsupported region". Turns out, they're protecting against such smartypants as I am, naive people who believed that Internet is global and makes it irrelevant where do you sit physically. Nope. No matter what you change your location to, the initial location when you created your account sticks with you. At one forum, a user tells me that if I were to create a different account now (forget the one I already have, it's tainted forever he tells me), and initially log in through some obscure UK site, and supply plausibly looking UK address and phone number then I could... sheesh.

I give up, play Bioshock for 15 minutes (all I had left of my evening), go to sleep, cursing my actions that led me to being exposed to a Microsoft technology again.

Next morning, more forum trawling. Turns out that you can create a "local" or "offline" profile, and that it lets you save the game locally. Hey, that's all I really ever wanted! So, how do I do it? Because, sure enough, I didn't see this anywhere on the UI when they were gently pushing me in the direction of creating a Live account.

Turns out, you need to click some kind of a "Learn More" button during the account creation process, and the help document will contain a link (only if you scroll to the bottom of the page) that will open a dialog to create a local account. At least it didn't say "Beware of the tiger" on the door...

So, let me get this straight: some Microsoft genius, in an attempt to drive as many users as possible into signing up for their Live service thoroughly hid the offline account creation, even if this will cause genuine grief for a subset of those users that are in an "unsupported region".

To use an analogy (it'll be a car analogy, of course), it's as if you'd have a four-lane highway, and all traffic signs point you to drive there to reach your destination, and of course you do, but after a bit of a drive, there's suddenly a blockade on the road, and since the license plates on your car are from an "unsupported region", you aren't allowed to drive further, and the grumpy officers won't tell you how you're supposed to get to your destination, only that they have specific instructions to not let pass the lower castes such as you are. So you drive back, and eventually find some secret vagabond scratchings on a wall (that'd be the forum posts) that point you to an otherwise unmarked dirt path through a forest, that will let you eventually reach your destination.

In the end, I have my local profile, so I can play the game now, right?

Right. Except I feel like I've been kicked in teeth by the whole experience so far, so it'll be some time before I'll have any desire to go back to this. Honestly, yesterday evening I was an inch from ritually burning the DVD. And I know that next time I hear that a game requires Windows Live in any capacity, that'll be a good sign for me to steer clear of it.

November moment

Oh, the sudden urgency in the people’s steps in the street as the raindrops start hitting the pavement. I don’t have an umbrella, so I’m drawing close to the buildings on the way home for some little shelter their walls and balconies can provide. I thought these buildings familiar, but either because I’m passing so close to them, or because of the rain, they smell different. It’s surprising, but pleasant and triggers some previously deeply buried childhood memories.

Review of the Commodore 64 emulator for the iPhone

UPDATE: Apple yanked the app because of the BASIC interpreter hole, and the developers have plugged it and resubmitted the app. Way to go, Apple. I mean, what harm could that BASIC interpreter do? It has no means of loading external code - no access to local filesystem of the underlying OS, no network connectivity, nothing. Are they afraid I'll manually type-in a program from a listing published in a magazine or something? I got tons more sense of childlike wonder from toying with the interpreter than I could ever from playing Jack Attack and Dragon's Den combined! Sheesh... On to the original review:

Against better judgment, I ended up taking my iPhone with me to the bathtub yesterday evening. I needed to check something really urgent, like the IMDB rating of "Watchmen" or similar (just finished the book). You know, stuff that you risk water damage to your high-tech gadget. I ended up browsing my RSS feeds instead though and spotted news about Commodore 64 emulator being approved by Apple and available on the App Store.
Now, you need to know that the first home computer I've ever been exposed to was a Commodore 64, at around age of eleven. I've got an enormous amount of emotional attachment to it, being enthusiast hacker for it as a kid. So, I was sitting there in my bathtub thinking, "this would be nice to get" and then realized "wait a minute, I can get it; right here, right now."
Let's pause for a moment to think about it folks, just how big enabler Internet is in our lives. I mean, there I was, purchasing a Commodore 64 simulator from my bathtub, and a minute later, I launched it and started toying around with the UI.

That's where disappointment started to set in.

Oh, don't get me wrong. Developers of this application (and the excellent open source Frodo C64 simulator they based it on) deserve every praise possible. I've used Frodo several years earlier, and it is painstakingly precise reproduction of a Commodore 64, up to the point of hardware side effects. The iPhone application is tasteful, snappy, responsive, and the UI is okay.
I do actually have one gripe with the UI, though. See, Commodore 64 horizontal resolution is 320 pixels, exactly the same as that of the iPhone. Since the developers added a graphic around it resembling the venerable Commodore 1701 monitor, the mask obscures some of the pixels at the edge. This is very annoying actually, since lots of games display important information around the top and bottom edges. These machines back then emitted an image that had two distinct areas named "border" and "paper". Border was the outermost part of the image and had a single attribute: color. Paper was a smaller rectangle within the border where the image was actually displayed. The purpose of the "border" was specifically to act as a disposable margin that could be clipped to size by your monitor (in many cases, a TV screen) so that you don't lose paper pixels. The developers of the iPhone emulator thus improved aesthetics at expense of functionality.

But that's not what I'm disappointed with.

Honestly, I could be disappointed by the initial selection of games; they're all ancient simplistic titles even by Commodore 64 standards, old 1983 titles where developers still didn't know how to utilize the platform to its potential. There will be more games available later, so I'm not too worried about that. (I hope they will release Elite...)

That's not what I'm disappointed with.

What I'm disappointed with is the lack of the Commodore 64 essence, as I experienced it. That essence, ladies and gentlemen, is hacking. Even as a kid, I disliked it when people were using the machine exclusively as a game console. As a bit of personal history, let me tell you that my family was not exactly wealthy when I was a kid, and that too is an understatement. After I became interested in computers (following a series of articles in a monthly magazine for school kids), I would write my little programs (initially BASIC, later 6510 machine code) in a notebook, then annoying one of the three people in our village who had a machine to give me a bit of a time to try them out. I saw the potential for creating my own stuff in these machines, and couldn't imagine why would someone, who is fortunate enough to actually be able to afford one, use it only to play games on it. I disliked people who didn't see beyond gaming (I'm not saying it was justified and think the same today, I was an eleven year old kid then, remember?).

Commodore 64 was an utterly open platform. Even from BASIC, you could POKE around with it (for those not in the know, "POKE" is a basic command for directly setting contents of a memory location). Even before I would have a Commodore 64 of my own (which I had to wait all the way until 1989 to be able to afford one; cool kids were playing with Amigas by then) I would have a book that contained a description of the BASIC, of the contents of both BASIC and System ROM (complete with all system calls and memory locations it used for settings, buffers, and so on), of all peripheral chips (VIC, SID, CIA1, CIA2 etc.), their mapped memory locations and how to program them, a complete reference of 6510 machine code, and finally, complete hardware schematics for the machine. If you had this book, you could do anything with the machine. Anything. And believe me, I did a lot. Won't go into details now as I'm well into get-off-my-lawn territory here.

That's why I'm disappointed that I won't be able to do a lot with this one, except play old games on my phone. I don't imagine that under Apple's regime I would ever be able to fire up Turbo Assembler and write some rasters, if you know what I mean. And that's what makes me sad.

Apple even only approved the emulator once the developer has disabled the BASIC, so it's not even the familiar C64 screen that welcomes you on startup. It looks like this instead:



However, turns out there's a workaround; the BASIC ain't disabled, it's just hidden. If you go to "Advanced" and turn on "Always show full keyboard", then launch a game, go to "EXTRA" keys, and tap "RESET", you end up with the familiar screen:



(Mind you, the problem with monitor mask clipping the edge pixels is particularly articulated in this picture - you don't really see the first character in the line!) I had a bit more fun with it though now that I had a command line. For starters, let's see if there's something on the "floppy disk":



Sure there is! Looks like a floppy for "Dragon's Den" (one of the games included with the app). Interestingly, it does not matter what game I started before resetting the emulator: the "floppy" in device 8 was always that of "Dragon's Den". I didn't find other numbered devices containing other games "floppies" either. For now. However, here's a kicker. (Or an easter egg if it's intentional, although I don't think it is.) If you load the program from the floppy and run it manually:



OMG, YOU GET AN INTRO!:



That's correct ladies and gentlemen - the developers seem to had trouble locating a genuine copy of the game, so they bundled one that was distributed by a cracker group. Back in the day it was customary for people who cracked games for, erm, "unofficial distribution" to prepend a small program to it called "intro" which was a colorful, blinking screen with greetings to fellow members of the trade and good chiptunes. They also usually led to menus for enabling cheats and sometimes on-screen instructions for the game in case the player didn't have the original game manual at hand. Like, if you're playing on an iPhone, right? (Honestly, if that wasn't there I would have never figured out this particular game on my own. So the intro actually added value for me. Also, the fact someone had this game in cracked form but not in uncracked form strongly suggests that if it weren't for crackers, we wouldn't have this game today. Just sayin'.)

Needless to say, the intro isn't there if you launch the game from the iPhone app menu.

Verdict? Nice easter egg with the intro, folks. Beautiful, polished iPhone application. Potential for more game titles to come. But if you were more than a game player on Commodore 64, don't hold your breath. You'll likely never be able to upload any of your your own old time C64 creations on the device to show it off. For that, you'll still need an emulator on your desktop OS.

I hope at least they'll release Elite, though. That'd be awesome.

Relativity of simultaneity

A problem with staying in a profession for long years is that there's less and less things that can truly excite or surprise you as the time passes. That's why I was particularly delighted to experience a moment of sudden enlightenment while listening to Rich Hickey talk about "Persistent Data Structures and Managed References" last thursday at QCon in London. The title of the talk might not sound exciting, but I have to tell you, if this were the only talk I attended, it alone would have been worth attending. I don't say this often. Slides are here, although you're missing out Rich's incredibly good verbal presenting style if you just read them, obviously.

Rich's view (with which I have to agree; in the time I knew Rich, I have yet to hear the first thing from him I'd disagree with, whether it be concurrency, memory models, or ideas for what to do in San Francisco if you only have a single day to explore it); so, Rich's view is that the idea of variables in traditional programming languages is broken, because they are based on a single thread of control at a time. This used to be true in single-threaded programs, and even to a degree for multi-threaded programs running on single-CPU systems.

He argues that we need well defined time models for state changes in our concurrent computational systems. A lack of a time model fails to capture the situation where two observers (threads, CPUs) can observe two events (changes of state) in different order.

Then it suddenly hit me! I learned this stuff in school! Relative timelines, observers... Rich is talking about relativity of simultaneity!

(Which is one of the simpler conseqeunces of Einstein's special theory of relativity.)

Wait a moment. Finding such a surprising parallel analogy between physical world surrounding us and our digital computational systems seems at first unlikely, but thinking more of it, it makes perfect sense. For event timelines to appear differently to different observers, we only need two prerequisites: truly parallel processing, and finite speed of change propagation. Both are true for the real world (quite massively parallel, and the changes propagate at at most the speed of light), and for digital computational systems (parallel already at two CPU cores, with changes occurring in CPU registers and needing time to propagate downstream the hardware memory model to be observable by other CPUs).

One of the wonderful things about Rich is that he's able to express these notions very clearly. It is all really obvious when you think about it, and I couldn't even say I wasn't aware of it for at least three years now, it's just that before hearing this talk, I always had a very slight hope that someday, someone (infinitely wiser than me) will somehow be able to, you know, solve this. Eliminate this problem. The thing I realize now is that it's inherent. You can no sooner eliminate relativity of simultaneity and the rest of consequences it brings to the table from our computational systems than you could cancel its effects in the physical world.

Rich does the only sane thing to do with a problem you can't eliminate - he embraces it. His programming language, Clojure, is an attempt at being the easiest way to write correct concurrent programs. Emphasis on all of "easy to write", "concurrent", and "correct". I learned about Clojure sometime last year when it came up at the jvm-languages mailing list. I read up the documentation on the website, and came away thoroughly impressed. Then I met Rich and saw a 30-minute Clojure presentation last year at the JVM language summit, saw the 60-minute version of the same talk now at QCon, and I'm still impressed. Clojure embraces the problem of concurrent state modifications by encapsulating the state into "references" - the only mutable data types in Clojure, which all point to immutable values. The references can change in time though to point to new values, and the big idea here is that each reference has defined concurrency semantics for how and when change occurs.

Now, there's likely a huge number of temporal semantics of how can some entity's state change. Clojure identifies several usual semantic categories though, and provides them for ease of use. We have synchronous coordinated (Clojure has a STM for that), synchronous uncoordinated, asynchronous uncoordinated, as well as isolated (thread local). I think you can build whatever you need using those.

Anyhow, during Rich's talk I had a moment of enlightenment. Not because I heard something entirely new, but because the pieces of the jigsaw puzzle finally fell into place and revealed a picture. The picture does away with a false hope I had, which is good as it is a clear and definite answer to a question, and having accepted it it is at least clear to me what's the direction forward. During the rest of the day, I was telling pretty much everyone I met and their dog about how I was shown the inherent manifestation of relativity in finite-speed concurrent systems. (Joe Armstrong was just smirking at it, probably thinking "you wet-behind-the-ears kids, this should've been obvious to you for decades", and then we proceeded making fun of all n-phase commit protocols, for any finite value of n).

On multicore systems, there's no longer absolute time. Every core runs on its own relative time and there's no definite sequencing of events. It's time we build this assumption into our code, and not use tools and methodologies that are ignorant of this fact. And as Rich would point it out, writing your code in Clojure is a good way towards the goal.

Speaking at QCon

Just a heads up that I'll be speaking at QCon in London this March, about using JavaScript in the enterprise. Looking forward to meet you all there.

IP telephony on iPhone

I spend quite a lot of time on calling my colleagues in US or UK (I am in Hungary), which can result in quite exorbitant phone bills. I tried to fix this lately by looking for IP telephony solutions for iPhone - I often have conference calls lasting upward of half an hour, and up to even two hours, so I don't want to be confined to my desk or any other physical location. I will often walk up and down in the apartment, subconsciously tracing patterns of the carpets with my feet (it's funny when I consciously realize I'm doing it), staring out of various windows at the sky, sometimes sprawling on a sofa and so forth. (Hey, working from home has its perks!). So, I need mobility within the apartment, hence I still want to use the phone.

Now, Skype would be the obvious choice - for a flat fee of 10.29 EUR/month I can get unlimited calls to both US and UK numbers. Major hurdle is that Skype ships no client for the iPhone. Fring actually does the job, it is a multiprotocol chat/voice client for iPhone (and some other phones) that supports Skype accounts too. Voice quality is not particularly good, but it's within tolerable. Fring however lacks an essential (for me) feature - it does not provide a DTMF keypad. I always dial into the company conference center, and must be able to punch in the conference number I wish to join. Fring can't do it. I can still use it to call direct extensions, but the bulk of my calls is through the conference system, and it's a no-go. I asked Fring support, and all they told me was:

Currently, it is not available to dial DTMF sounds on fring, this should be implemented in future versions.

For the record, Skype desktop client does have a DTMF keypad.

Next choice is TruPhone. They offer a client for iPhone (and also a bunch of other makers' devices). TruPhone operates their own call service, much like Skype does, and while there's no flat-fee subscription model, their 0.06 USD/minute plan is sure quite attractive (heck, even calling numbers within my cell provider's network, T-Mobile Hungary, is more expensive than that!). The voice quality is very good, even better than cellphone (one coworker remarked after I first used it to dial in to a conference that I sounded much clearer than I usually do). The client app also has a muting button and a DTMF keypad, the first useful, the second essential for conferencing. It would all be rosy, if only the client weren't extremely buggy. Sometime, with iPhone OS 2.1, the UI would slow down (you could see it animate scrolls pixel by pixel) and it'll then claim there's not enough memory. iPhone OS 2.2 seem to have fixed that, but now it instead often complains that it can't find "Wi-Fi coverage" (even though the iPhone itself clearly sees the network). The aforementioned DTMF keypad has the tendency to not react to touch, or conversely, to get jammed as if I touched the key but didn't lift my finger, sending a prolonged shrieking "beeeeep" of that particular number key tone through the earphones into my skull, full volume. (The iPhone built-in phone app nicely tones down the DTMF beeps on my end of the line, so I still have an audio feedback that I'm typing them, but don't hear them full-volume). Regardless, the "low memory" and "unable to find wi-fi" are the most critical problems - even turning the phone off and on again doesn't always help it with its wi-fi disorientation syndrome (and mind you, I should never be forced to power cycle my phone just to use an app!).

So, it's a mixed bag. I'd use Fring if it had a DTMF keypad because of a flat-rate unlimited calls provided by Skype. The second best right now is TruPhone, which provides very good voice quality and otherwise pleasant service with low pay-as-you-go rates, but their iPhone client is bordering on unusable because of all the bugs. I'll be eagerly awaiting app updates to both.

World of Goo

I found a fantastic game. It's named World of Goo. Here's a trailer. And here's the Wikipedia entry. Check those ratings - 9/10, 9.5/10, 10/10, 93%, 96%.

It's hard to summarize what's right about this game. Most everything is.

Let me start with few obscure details that might appeal specifically to a computer geek like me.

First, this game runs on Windows, on Nintendo Wii, and on Mac OS X. Linux version comes soon. And guess what, the Mac OS X version is not a lame "Win32 API linked with Wine" kind of port. Proof that it isn't is that Wine-linked Win32 API using Mac software only runs on Intel CPUs. Not so this game. It runs on Macs with PowerPC CPUs (either a G4 or a G5). It's pretty much a native version on Mac OS X. It runs crisply on my iMac G5. I asked the developers in e-mail about their development methodology, and the developer Ron Carmel was kind to reply. (The whole company seems to be just two guys, working from wi-fi equipped coffee houses in San Francisco. They have years of experience working for big name game companies, though.) Anyway, Ron's response was:

at some point, when we realized we're going to go multi-platform (wii/pc at the time) we simply abstracted away anything and everything platform dependent from the code. this meant graphics, sound, input handling, window creation, threading, that sort of thing. the game only deals with a set of abstract classs that it gets from a singleton "environment" class. those classes provide a set of services (like graphics, sound, etc) and we have different implementations of those classes for windows, mac, wii, and soon linux.

I can't say just how much I respect people who take this approach. It's one of hallmarks of professionalism in my book. More often than not, even much more complex software systems can (and should) be developed to be really multiplatform. At least on unices, it is fairly common to maintain software as CPU-agnostic (and often OS-specifics-agnostic) source code with small CPU&OS adapters. These guys however didn't only do it across different unices, which are all closely related OSes. They did it across Windows, Mac OS X, and whatever is the OS of the Wii. That's 3 operating systems, and likely 3 different CPUs too (Intel, PPC, and whatever the Wii CPU is, although AFAIK, it's also some PPC variant). Consider how most large software houses can't be bothered to do this - they just target Windows primarily, even though they'd have lots of internal benefits if they cleaned up their source code to be multiplatform.

Second obscure detail, the game has no DRM. None. These guys were smart enough to realize it'd be a waste of resources to bother with it. Another big tip o'the hat to them.

What this game does have though is: captive, immersive, innovative gameplay based on building structures out of gooey balls that bind to each other, and leveraging mechanics in a very realistic physics simulation (on various levels, you'll need to use tension, elastics, gravity, wind, flotation, and so on). It's tremendous fun. Immersion is further helped by the fact that there's no UI aside from the mouse pointer - all interaction with the game world is by directly dragging and dropping the otherwise aimlessly wandering goo balls with the mouse.

There's lot of humor and cuteness baked in. You can get emotionally attached to little goo balls as you could to those lemmings back in the day.

And then there's the beautiful visual art (that at times reminds me of Tim Burton movies, particulary The Corpse Bride), and music that perfectly matches it to create the immersive atmosphere. There are overarching motives and hints of a backstory that are intriguing on their own.

It's ideal for a casual gamer who needs to blow of steam for an hour after a hard day's work. My son is playing it every evening. My daughter is playing it every evening. I am playing it every evening. You can play it too in minutes - there's a downloadable demo with the full first chapter of the game (the full game has four chapters and an "epilogue").

FreeCC, the modern JavaCC

If you ever needed to write a parser, you hopefully didn't do it by hand but rather used a parser generator. The two predominant parser generators in Java are ANTLR and JavaCC. JavaCC project however suffers from paralysis for too many years now. The original developers aren't present for too many years now, and the current set of active committers didn't make any significant improvements for years. My FreeMarker colleague Jon Revusky has recently taken interest in it (FreeMarker having used JavaCC in the last six years, when Jon ditched the hand-written parser of the 1.x versions). Since his improvements to the JavaCC codebase weren't accepted by mainstream (largely dormant) JavaCC project, he did the only correct move: he forked it.

Enter FreeCC.

FreeCC is backwards compatible on source level with JavaCC - it can generate parsers from existing JavaCC sources, and the parsers behave identically (since it's a continuation of the original codebase with different name, it's not too surprising, but doing the work under fresh name, it doesn't hurt to emphasize), FreeCC, however has some modern new amenities, such as:

• typesafe collections in its own code, instead of Java 1.1 Vector and Hashtable


• instead of hardcoded prints in the source code, the parsers are generated using editable templates (can you say "multiple target languages support?")


• other long overdue source code sanitization


• generated parsers don't rely on static singletons anymore

These are all nice and dandy; code reworks in particular improve the project's long term maintainability by making its comprehension more easy for newly joining developers (in this regard, FreeCC is much more welcoming than JavaCC for a newbie hacker). These however don't really give you much of an advantage when you're writing your grammars. The following ones do, and they're the real kickers:

• code injection feature eliminates (or at least strongly reduces) the need for manual post-editing of files


• grammar include feature, which allow smaller grammar files to be reused in larger ones (with JavaCC, you had to copy-paste), as well as allow you to organize larger grammars into separate smaller files.

These features are also the sign of what kind of features are still to come: features that provide you with modern conveniences a programmer in a need of a parser generator would wish to have. These two new features above help you create maintainable grammar sources, and they are just the start. If you adopt these features however, there's no going back. Since JavaCC doesn't have these features, your grammar files will no longer be compatible with JavaCC, only with FreeCC. But you'll hardly want to go back. FreeCC has taken this particular parser generator codebase much further in its few months of existence than the JavaCC project did in years, and is gaining traction. Given the fact that FreeCC is a continuation of the current JavaCC codebase (which didn't really progress further since the fork), it is really risk-free to try it out for your next project (or even in current project!) instead of JavaCC. You can also expect that the developer will be more open to your feature requests, as Jon has a good track record of listening to community wishes in FreeMarker.

FreeCC is the JavaCC you'd want to use in 21st century.

At FreeMarker project, we've already switched to FreeCC; FreeMarker 2.4.0 and 2.3.15 will both have a parser built using FreeCC. Since Jon also works on FreeMarker, he's truly eating his own dog food. (Actually, there's even more to that. Since FreeCC in turn uses FreeMarker as the templating system for its output, he's eating his own dog food doubly! This seemingly creates a circular dependency between FreeMarker and FreeCC, except that luckily all that was needed was a FreeMarker JAR with parser built still using JavaCC to bootstrap the process initially; the projects are self-sufficient since and don't need JavaCC anymore.)

HTML encapsulated JSON

Continuing the early morning post on having an XSL-like solution for JSON, where your webapp only outputs JSON files, and has attached stylesheet(s) that the browsers can use to display it as a nicely formatted HTML it intended for human audience. All solutions I outlined there needed an active change to existing technologies: custom HTTP header, or extension to JSON, but in any case they would need explicit support from browsers. 

In other words, they would never get widely adopted. 

But then, just as I went to sleep, it hit me that there's a solution that operates completely within the currently existing technologies. I'll call this solution "HTML encapsulated JSON". The premise is that you create a simple HTML page that has the JSON payload in its body, and has script elements that pull in the JSON-to-DOM transforming script. Something like this:

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <script src="/carlist-json-to-html.js" type="text/javascript" />
    <link rel="stylesheet" type="text/css" href="carlist.css" />
  </head>
  <body>
    <pre> 
        {   "cars": [
                {   "color": "red",
                    "make": "Ford",
                    "model": "Mustang",
                    "year": 1986,
                    "price": { 
                        "amount": 250,
                        "currency": "USD"
                    }
                },
                {   "color": "yellow",
                    "make": "Mazda",
                    "model": "6",
                    "type": "GT",
                    "year": 2002,
                    "price": { 
                        "amount": 14000,
                        "currency": "USD"
                    }
                }
            ],
            "dealer": {
                "name": "Honest Joe",
                "address": {
                    "street": "123 7th Avenue",
                    "city": "Dustfield"
                },
                "phone": "1-456-789789"
            }
        }
      </pre>
    </body>
</html>

As you can see, the JSON is put into html/body/pre (using "pre" to be fully DTD conformant). A machine client should be able to easily parse it out from there. Granted, it'll need to use both an XML parser and then a JSON parser, but that shouldn't be too big deal.

But the best part is definitely that you can very easily transform this into a nicely formatted HTML, by including a script (emphasized in red) to build a DOM from the JSON extracted from the body. Actually, you can just generate a fairly naked HTML, and then use a separate CSS stylesheet (as shown above) to format the HTML for presentation! And it's all standards compliant, and works with any modern browser!

Next, I'll try to come up with typical code for a JavaScript code template that would be used for a JSON-to-DOM conversion.

CSS for JSON

The back-in-the-day promise of XSL stylesheets was that you could output an XML document from a HTTP request, and have it processed as-is by a machine recipient, or transformed into a pretty HTML for human viewing automatically. 

Seeing how I (and probably many others) do prefer JSON to XML nowadays for my machine-to-machine communication, I'm thinking of an equivalent for JSON. In similar vein, XSL is almost entirely replaced by CSS today, and I keep thinking that we'd need an XSL or CSS equivalent for JSON.

Actually, we probably don't even need a new language for describing the transform. JavaScript would probably do just fine - read the JSON, build the DOM.

The other problem, assuming we already have the language for transforming JSON to pretty formatted HTML: how would you declare a transform in a JSON document?

In XML, you can use a processing instruction. In HTML, you can use the <link> element. Now, unlike XML's processing instructions, or HTML's <link> element, JSON notoriously lacks any support for any out-of-band information that isn't the actual content. Heck, it doesn't even support comments!

I can see three possible ways around it:

1. Declare the transform in the transport header, i.e. just slap on an "X-JSON-Stylesheet" or similar header into the HTTP response. The obvious problem is that this ties the solution to the transport.
2. Add it to a conventional location into the JSON data itself. I.e. have the top-level object literal have a "__metadata__" object, and put this and similar other information into it. i.e.:

   { "__metadata__": { "links": [{"url": "myStyleSheet.cssjs", "type": "stylesheet"}]} ... }

   The obvious problems are: a) the top-level value in a JSON document ain't necessarily an object (can be an array, or even a primitive value) b) some JSON based formats won't be happy about having extra elements with magic names inside of them.
3. Extend JSON to add a syntactic equivalent of XML processing instructions. Honestly, I prefer this approach. Just adding single-line comment support, and then having a special prefix for the "processing instruction" comment (i.e. using a question mark, to keep it similar with XML PE) would do, I think:
   

   //?stylesheet myStyleSheet.cssjs
   { real payload goes here ... }

In any case, it is up to browser vendors to get the ball rolling on this - both the declaration mechanism, and the transform language.

I think this would be a great technology to have, where I could just output JSON from my web applications, and have it both consumed by software, and presented in nicely human readable form in browsers.

Package private access in Open Source code

I recently got in a same situation three times: someone wanted to use code I wrote in an Open Source project written in Java, and they couldn't, because the class/method in question had package private ("default") access, rendering it inaccessible outside of its package.

• First, Charlie Nutter needs access to package-private class BeanMetaobjectProtocol in Dynalang-MOP.
  
• Next, John Arkley needs access to package-private class AllHttpScopesHashModel in FreeMarker to help push it into Spring.
  
• Finally, Steve Yegge needs access to the package-private constructor of Context class in Rhino.
  

Now, you know, when the same thing hits me three times in a row in short timeframe (incidentally, all three coming from people with high geek cred), that gets me thinking: What good is package-private access in an Open Source project anyway?!

It's a very valid question, really. Think about it: people can see the code - it's open. People want to use the code - it's normal. And when they want to, you frustrate them by declaratively preventing them from linking their code to your open source code. They can look, but they can't touch.

That seems wrong.

Of course, you could argue for package private access' validity. Here are some arguments you could come up with:

Package private helps implementation hiding on a package level! Well, duh, it does. However, if a class/method is useful to another class you wrote and that lives in the same package, it might also be useful to some poor outsider schmuck too! It must already conform to quite rigorous coding standards, as it will be used by some other classes, ones you wrote in the same package, so it'd better maintain state invariants and so on. Does it really make a difference if it's another class of yours, or another class of another developer, living in another package? I say: it shouldn't. If you think the class is just an auxiliary, and it's just cluttering the JavaDoc, just move it to a *.support subpackage instead (Spring does that extensively).

Package private helps you hide bits you don't want to be tied down by backwards compatibility! This is really a corollary of the first one. I used to be big on this one (that's probably why I got asked to loosen up access restrictions in the first place - because I used to place them there in the first place). Common wisdom is that once people start using your publicly available API, you'd better not break it on the next  release. By making lots of stuff public, you increase the surface area that needs to be kept backwards compatible, right? Right, sort of.

Now, how about this instead: make it public anyway, just note in a JavaDoc entry (or even better, in an annotation, say @SubjectToChange) that there's no backwards compatibility guarantee on this method. If it's an annotation, people can even have an automated tool for checking for its use before they upgrade. Hell, you can even have a @BreaksCompatibility annotation! What it boils down to is: don't treat users of your library as children and decide what's good for them. Inform them that the API is volatile, but open it up, don't close it. It's not really closed anyway, as they can see the source. You're just erecting a glass wall in front of them; they can look but they can't touch. 

And they'll come to bug you about it anyway. 

My point of view right now is that it's better to provide a suggestive hint that an API is volatile in the documentation or annotation and let anyone use them at their own risk rather than build a non-negotiable restriction into the code (as in, can't negotiate it with a compiler; you can still definitely negotiate it with me).

Also, I'm not saying either that package private access is completely inadequate for Open Source libraries. I'll admit there might be valid use cases for it, but if there are, they're very few.

Now, before you'd accuse me of being a "make everything public" proponent, let me say that this reasoning doesn't apply at all to private access, and only partially applies to protected access. Let's see why:

Private access is completely justified. Let me point out one clear distinction between packages and classes: classes and instances of classes can have state. Packages can not. That's pretty much what makes the difference. Through refactorings, you can end up with methods in a class that violate its state invariants. Other methods in the class can call these methods as steps to transform the object from one valid state to another valid state, but it might be invalid in the interim. You would never expose such a method publicly. Well, you shouldn't expose it even package-privately either; that'd be sloppy programming (see my above remark about package-privately accessible methods having to be just as rigorously coded as public ones). A class (and class instance) is a unit of state encapsulation, a package is not. Hence, classes need private access.

Protected methods remain usable by 3rd party code, as long as it has classes that extend your base  class. It mightn't be too ideal a constraint (forcing subclassing in order to access functionality), and you might rather want to design your libraries to favor composition (has-a) over subclassing (is-a) architecture. But if you must have classes intended to be subclassed, and there's functionality that's only ever used/extended/implemented by a subclass, make it protected. Especially when you have abstract protected methods that act as poor Java programmer's mixins (are intended to be called by code in the base class). Sometimes you'll allow (or downright expect) such abstract methods to violate an invariant in the object's state temporarily, as it'll be called as a step in a more complex state transition implemented by an algorithm in the base class method. In these cases, protected access is justified. But in other cases though, you might still consider making some protected methods public.

In conclusion: I'm currently fairly convinced that what package private access is good for  is preventing your users from linking their code to useful bits of your code for purposes you didn't anticipate up front; it's often nothing more than an unnecessary garden wall.

Undeclared, Undefined, Null in JavaScript

So, a coworker of mine run into a situation recently where they were testing for an expression in JavaScript as:

if(obj.prop != null && obj.prop.subprop > 0) ...

This basically guards you against dereferencing "subprop" if "obj.prop" is itself undefined or null. They were running into a situation where the first predicate would pass, and the second would fail. However, "obj.prop" always printed "undefined". Turns out, it actually had the string "undefined" as its value.

Ouch.

Anyway, said coworker pointed me to a blog entry titled Three common mistakes in JavaScript / EcmaScript. Said blog entry says:

if (SomeObject != null) {

Well, in JavaScript, which is a dynamic language, something that has not been assigned to is not null, it's undefined. Undefined is different from null. Why? Don't ask me. Well, anyways, you can use typeof to explicitly check for undefined, or use other more or less clean tricks, but the best way to deal with that is probably to just rely on the type-sloppiness of JavaScript and count on it to evaluate null and undefined as false in a boolean expression, like this:

if (SomeObject) {

It looks uglier, but it's more robust.

I have to disagree. It's not more robust. It will also catch the cases of SomeObject having numeric value of 0, or string value of empty string. Because their boolean coercion is also false. To make matters worse, the original example, using SomeObject != null actually works and is in most cases actually the most appropriate!

See, in JavaScript, null and undefined are actually equal according to the == and != operator! (ECMA-262 standard, section 11.9.3 tells us so.) In vast majority of cases, you don't care about the difference at all, so using someExpr != null is good enough.

If you really-truly must distinguish between undefined and null, you have some options. Curiously, while there is an actual built-in language literal for the null value (namely, null), the sole value of the Null type, there is no built-in language literal for undefined, the sole value of the Undefined type. The identifier "undefined" can be assigned to, so you can't write something simple as if(x === undefined):

var undefined = "I'm defined now";
var x; // he's really undefined
print(x === undefined); // prints false

Inconvenient, huh? So, how to test for undefined? Well, the common practice found in most books and tutorials on JavaScript seems to be using the JS built-in typeof() function, but I really don't like it, because this is implemented by way of a string comparison:

var x;
print(typeof(x) == "undefined");

will actually print true. But as I said, I think it's ugly.

My solution instead relies on the fact that undefined is equal to null, but is not strictly equal to null, therefore this expression also works:

var x;
print(x == null && x !== null);

will also print true, and it involes only two simple comparisons.

Which brings us to the question of what is the actual difference in JavaScript between an undeclared variable, a variable with undefined value, and a variable with null value. Let's see:

var x = {}; // empty object
var u; // declared, but undefined
var n = null; // declared, defined to be null

function isUndefined(x) { return x == null && x !== null; }

print(isUndefined(x.x)); // prints true - access to undefined property on an object yields undefined
print(isUndefined(u)); // prints true - declared, but undefined value
print(isUndefined(n)); // prints false - the value is null, not undefined
print(isUndefined(z)); // runtime error -- z is undeclared

So, it's an error to dereference an undeclared variable, "z" in above example (it's okay to assign to it, which creates a new global variable). It is not an error to dereference a declared variable with undefined value, "u" in above example. Its value is the undefined value. Further, access to any undefined properties on objects also result in undefined value, duh. As for null, well, null is just a value like true, or false, or 4.66920166091; it's the single value of the type Null.

Hope this clears up the whole topic of undefined/null values (and undeclared variables) somewhat.

Objective-J

So, here's some news: people at 280 Slides created a web application that allows you to build presentations in your web browser. It does look very nice, people are comparing it to Apple's Keynote application. All in all, yet another webapp out there; what's the big deal, right?

Well, the people who created 280 Slides were previously Apple employees. 280 Slides wasn't just written in JavaScript. No. These people created something called Objective-J, which is to JavaScript as Objective-C is to C. And then they implemented part of Apple's Cocoa application framework atop of it (named it Cappuccino), and finally implemented the application atop of it.

Now that's quite amazing.

Dion Almaer writes that

Objective-J is the language that takes JavaScript and makes it Objective (as Obj-C did to C). Lots of square brackets. When the browser gets served .j files, it preprocesses them on the fly. This means that you can do things like, use standard JavaScript in places.

Interesting. Objective-J will eventually be open sourced at objective-j.org, and I'll be quite curious to see what did they do. I suspect they have a transformer from Objective-J source code to plain JavaScript (presumably itself written in JS), and then the browser's JS runtime converts the source code to JS when it downloads it. But I might be wrong.

Then there's the interesting issue that Objective-C improved C with OO features. But what did Objective-J improve? JavaScript is extremely object-oriented to begin with, so this sounds more as if they wanted to bring the actual Objective-C flavor of OO to JavaScript instead, because that's what they're comfortable doing. They need to drive nails into a different wall now, and they'd still prefer to do it with their old hammer!

Don't get me wrong, I'm not making fun of them. Shaping one's tools in a new environment after ones you knew and loved in a previous environment is a valid activity if you percieve it as the path that allows you to be most productive. To build a new language atop of JS and then build an application framework atop of it, and then build a very usable and visually appealing application on top of it (very cross-browser compatible too) gets you a metric shitload of geek cred in my circles. It might turn out to be a catalyst for getting a lot of similarly nice future webapps out there. It might turn out to be the next big thing for JavaScript in browser.

I'm eagerly waiting for content to start popping up at objective-j.org, although of course the Objective-J.js can be readily inspected.

So, WebKit JS used an AST walker before?

Color me flabbergasted.

WebKit recently announced a new JavaScript interpreter, named SquirellFish, claiming it is much faster than the previous interpreter in WebKit.

That's good and all, but in the "Why it's fast" section of the linked article, they say:

Like the interpreters for many scripting languages, WebKit’s previous JavaScript interpreter was a simple syntax tree walker.

It was? Oh my... They also say that:

SquirrelFish’s bytecode engine elegantly eliminates almost all of the overhead of a tree-walking interpreter. First, a bytecode stream exactly describes the operations needed to execute a program. Compiling to bytecode implicitly strips away irrelevant grammatical structure. Second, a bytecode dispatch is a single direct memory read, followed by a single indirect branch. Therefore, executing a bytecode instruction is much faster than visiting a syntax tree node. Third, with the syntax tree gone, the interpreter no longer needs to propagate execution state between syntax tree nodes.

Why, yes, indeed!
For the record, Rhino has been doing this for ages - AST is compiled to internal "JS bytecode" format that strips away grammar, and then interprets it. This works like this since, well, around the turn of the millenium. (Actually, Rhino can even kick it another notch and can also optionally compile the JS bytecode to Java bytecode, eliminating the interpreter altogether. Which bytecode then the JVM JIT compiler can further compile to machine code at its own discretion.)

Anyway, I digress. All I wanted to say is that I'm honestly amazed that a supposedly professional implementation of JavaScript (the one shipped in WebKit before SquirellFish came along, and by consequence, shipped in Apple's Safari browser) would use an AST walking interpreter. Yeah, I know, you'll say "most scripts run once on the page, so optimization is overkill", but with AJAX this is no longer true, and apparently, the WebKit team also thinks so.

On the positive side, they're finally moving on to a better solution, and I congratulate them on the undoubtedly hard decision to finally take the leap toward a more modern execution architecture.

"Mixed Language Environments" interview

in case you're interested, the video interview where Kresten Krab Thorup interviewed Charlie Nutter, Erik Meijer, and myself about mixed language environments on virtual machines is now online. This was shot at the JAOO conference last September.

Interview about Rhino

Floyd Marinescu interviewed me about Rhino at the JAOO conference in Aarhus, Denmark last September. You can watch the interview here. Yes, I know I said "scratch to itch" :-)

I actually stepped down from the Rhino maintainer role since this interview was shot, although I do remain with the project as a committer.

Airport Extreme disappoints

UPDATE: Apparently, firmware upgrade 7.3.2 fixed the disk access problems. I'm now able to use a USB-attached disk to AEBS just fine. Skype also started working fine some time ago (even before the firmware upgrade). The speed issue (not being able to exceed 8 Mbit/sec on 802.11g with all-Apple equipment) unfortunately still remains.

I got a brand new Airport Extreme Base Station (AEBS) this Friday. (Kirk hauled it for me from Las Vegas to Hungary - Apple gear costs roughly twice as much here than it does in US.)

I already own a 250GB external HDD, so I wanted to attach it to AEBS for either file sharing or, hopefully, wireless Time Machine backups.

(Too bad Apple officially said only three days later that they don't support Time Machine backups with AEBS. Meaning, if it works for you, you're lucky, but if it doesn't, you're on your own.)

Anyway...

My experience is that even when Time Machine is not involved, AEBS can't handle a USB drive at all. Just attaching the drive to the base station will cause it to come online much more slowly (needs several minutes after plugged in). Also, whenever I wanted to mount the drive through Finder on any of the Macs in the house and transfer a file to/from it, AEBS would become inaccessible in a minute or two, then spontaneously reboot. With no drive attached, it's stable as a rock.

Of course, you could claim that the drive is at fault. Now, the same drive works perfectly when attached to either my MacBook Pro, or to my iMac G5, both running Mac OS X 10.5.2. It was formatted and partitioned (single partition, GUID scheme, HFS+ journalled filesystem) on the iMac G5. As in, freshly reformatted and repartitioned once again in a desperate attempt to make it work with AEBS. Run Disk Utility's "Repair Disk" on it too, just in case.

Now, when the drive is attached to either Mac and shared from it, the other Mac can mount it wirelessly and even do Time Machine backups to it, without any problems. So, you know, the drive itself seems in order. But if I connect it to the AEBS, it kills the AEBS. AEBS firmware is also upgraded to latest, 7.3.1. I wanted to reduce the clutter on my desk by moving the HDD and its wires to the AEBS and am fairly frustrated with the inability to do so.

I repeat: the symptoms have nothing to do with Time Machine. With Time Machine off, and trying to access the drive over AEBS as a vanilla networked volume, AEBS dies in a minute or so. I suspect a firmware bug.

And then there's the speed.

My MacBook Pro is a first-gen Core Duo, meaning it can only use 802.11g, and can't be upgraded to use 802.11n. Still, 802.11g is 54 MBit/sec, right? Well, the fastest data transfer rate I could achieve moving data to the AEBS-attached HDD (before it crapped out) was a bit over 8 MBit/sec. On a distance of few feet between the devices. Not happy. I can understand various protocol overheads would chip off the bandwidth but I can't understand getting 14% of advertised bandwidth. Not even if it were half-duplex.

Okay, what about Ethernet access? I tried connecting over Ethernet to see whether the AEBS crashing problem is inherent to Wi-Fi, or is also present over Ethernet (result: it is present over Ethernet too). The MacBook Pro said the connection is 1000MBit/sec - fair enough, as both it and the AEBS are supposedly Gigabit Ethernet. Both devices use speed and duplex autoselect based on the cable actually plugged in. So, with 1000MBit/sec theoretical maximum speed, what effective throughput could I get? 72MBit/sec, tops. It ain't the HDD that's slow - it's a 7200RPM drive with USB 2.0, and when connected directly to the computer it is pretty much screaming fast.

And then there are application compatibility problems.

My wife Kriszti speaks quite a lot with her aunt over Skype, and with AEBS she experiences much more stalled audio/video over Skype. I actually subscribe to two broadband connections: a 4MBit cable and also a 1MBit ADSL as a backup. I moved the old router to the 1MBit ADSL (which previously wasn't available over wireless, being only a backup). When Kriszti had repeated Skype problems, I told her to switch to using the old wireless router, and lo and behold Skype works okay for her again, so her Mac tends to remain connected to the old router's wireless network (so our machines aren't on the same network, which is also a source of inconveniences). So: cable ISP with old router = Skype ok; cable ISP with AEBS = Skype not okay; ADSL ISP with old router = Skype okay. There's a pattern here, although in order to be able to claim with 100% certainity that it's a problem with AEBS, I'd also need to test Skype with AEBS connected to the ADSL line which I won't do now as it'd require too much uncabling/hauling/recabling.

So, we have a base station that crashes when you try to use an USB HDD with it, with disappointing speed figures, and apparently also causes problems for Skype traffic. The only really good point is that contrary to the old taiwanese no-name router I replaced, it has a better range and implements WPA/WPA2 in a manner that Windows laptops can also understand; both of these don't really mean anything to me, but it certainly makes the nice girl next door (who we give free Internet access in exchange for occasionally babysitting our kids) lot happier.

All in all, a disappointment.

Xstream with Hibernate

People have been asking in the comments to my post on XStream about how to properly integrate Hibernate with XStream, as there are few pieces of puzzle that don't necessarily fit together perfectly. I answered in the comments, but Blogger annoyingly doesn't allow a lot of formatting in comments, so I decided to write a separate follow-up post; here it goes.

• First of all, I use Hibernate 2. I know, it's old, but it does the work and I won't fix it unless it is broken. I expect most of the advice will also apply to Hibernate 3.


• I'm aliasing all classes using XStream.alias(). Reason being that on the server side, some data model classes are actually subclassed for additional (although non-Hibernate) related functionality. You might or might not need this.


• This is however essential: We need to ensure that XStream will treat Hibernate lists, sets, and maps as Java lists, sets, and maps. I believe this is what helps avoid the infamous "com.thoughtworks.xstream.converters.ConversionException: Cannot handle CGLIB enhanced proxies with multiple callbacks..." problem. There are three things that need to be done:
  • use Xstream.addDefaultImplementation() to tell XStream to treat all Hibernate-enhanced collection classes as plain Java collections:

    xstream.addDefaultImplementation(
            net.sf.hibernate.collection.List.class, java.util.List.class);
    xstream.addDefaultImplementation(
            net.sf.hibernate.collection.Map.class, java.util.Map.class);
    xstream.addDefaultImplementation(
            net.sf.hibernate.collection.Set.class, java.util.Set.class);

  
  
  • Finally, in order for XStream to actually handle these collections I needed to define and register some custom converters that are able to handle Hibernate collections as Java collections:

    Mapper mapper = xstream.getMapper();
    xstream.registerConverter(new HibernateCollectionConverter(mapper));
    xstream.registerConverter(new HibernateMapConverter(mapper));

    These custom converter classes are rather trivial, here are their definitions. All they really do is extend the XStream built-in collection and map converters, and declare their ability to handle Hibernate lists, sets, and maps:

    import net.sf.hibernate.collection.List;
    import net.sf.hibernate.collection.Set;
    import com.thoughtworks.xstream.converters.collections.CollectionConverter;
    import com.thoughtworks.xstream.mapper.Mapper;
    
    class HibernateCollectionConverter extends CollectionConverter {
        HibernateCollectionConverter(Mapper mapper) {
            super(mapper);
        }
    
        public boolean canConvert(Class type) {
            return super.canConvert(type) || type == List.class || type == Set.class; 
        }
    }

    and

    import net.sf.hibernate.collection.Map;
    import com.thoughtworks.xstream.converters.collections.MapConverter;
    import com.thoughtworks.xstream.mapper.Mapper;
    
    class HibernateMapConverter extends MapConverter {
    
        HibernateMapConverter(Mapper mapper) {
            super(mapper);
        }
    
        public boolean canConvert(Class type) {
            return super.canConvert(type) || type == Map.class; 
        }
    }

That's all I did and it eliminated all of my Hibernate+XStream problems - hope it will also help you.