Hysterical Raisins

Why Alan Moore is right, and why he's wrong

lalo — Sat, 28 Feb 2009 05:23:00 -0300

Excerpt from Alan Moore's interview at Wired:

One thing is that with the comics medium, it has been proven—I believe by Pentagon tests in the late '80s—that comics are actually the best medium for imparting information to somebody in a form that they will retain and remember. That's not just me saying that, that's the Pentagon. I personally feel—and this is just pseudo-scientific hippie bullshit—I feel this might be because the unit of currency of what used to be called our left brain is the word. Our left brain is what goes about speech and rationality. The unit of currency for our right brain, conversely, would be the image, because the right brain is preverbal.

So perhaps it is because of the combination of words and images in a readable form that comics does have this unique power. Now, of course, movies are a combination of words and images, but they have a completely different structure and completely different way of working. With a movie you are being dragged through the scenario at a relentless 24 frames a second. With a comic book you can dart your eyes back to a previous panel, or you can flip back a couple of pages to check whether there is some reference in the dialog to a scene that happened earlier.

You can also spend as much time as you want absorbing every image. This is especially true of something like Watchmen, where I was trying to take advantage of Dave Gibbons' brilliant capacity as a former surveyor for including incredible amounts of detail in every tiny panel, so we could choreograph every little thing. The little symbols and signs appearing in the background, every little touch could be choreographed to the last detail, and we knew that the audience—because they'd be reading at their own pace—would be able to study each panel and to take in these almost subliminal details. Even the best director in the world, even a person as talented as Terry Gilliam, could not possibly get that amount of information into a few frames of a movie. Even if they did, it would have zipped past far too quickly. Because the audience at the movie theater is not in control of the experience in the same way somebody reading is.

One of my big objections to film as a medium is that it's much too immersive, and I think that it turns us into a population of lazy and unimaginative drones. The absurd lengths that modern cinema and its CGI capabilities will go in order to save the audience the bother of imagining anything themselves is probably having a crippling effect on the mass imagination. You don't have to do anything. With a comic, you're having to do quite a lot. Even though you've got pictures there for you, you're having to fill in all the gaps between the panels, you're having to imagine characters voices. You're having to do quite a lot of work. Not quite as much work as with a straight unillustrated book, but you're still going to do quite a lot of work.

And I have to agree. Comics are comics; you read it at your own pace, you analyse the details like it held the secrets of the universe, you enjoy the little hidden things both in the art and story, and you fill in the gaps. That's what makes it great, and that's why, no matter how well the movie is done, the comic will always be better.

(Well. In normal circumstances at least. I've seen mediocre books or short stories become great movies, but that's a separate story altogether, and to date I haven't seen it done with comics yet.)

On the other hand, I think dismissing the flick like he does is a waste of good entertainment as well. Time to quote from Dave Gibbons' interview in the same issue:

The most bizarre thing was to actually be inside the Owlship, you know? As I kind of implied in an earlier answer I've always loved drawings and measured plans of things. I went to a lot of trouble to make the Owlship convincing and make room for everything that we saw inside it. So, to actually be inside this thing—the thing that had been inside my head, I was now inside that. It felt exactly like the space that I'd felt when I'd done the drawings. I think that was really the strangest thing, to sit in the command chair and play with the joystick and press the buttons and watch all the lights flash on.

And that's where the magic really is. That's why those geeky movies are so great. It's like, well, going to a theme park, except usually with higher quality results. These things have lived in our imaginations for years, and now we get to see them there, big and real-looking. It's, well, fun.

Another important thing missed there is that movies can be a social experience. Comics, by the very merit of being read at your own pace, are solitary; you can get together with people to read comics, but you don't actually read together — well, you can, but it kind of ruins the experience. That's what is (well, used to be) so great about Heroes; it's kind of like reading a comic book, only I do it with my girlfriend, and we react together.

Short version? Absolutely do go watch the Watchmen, but not if you haven't read the comic yet. :-)

How to use bzr-svn with SourceForge

lalo — Tue, 17 Feb 2009 17:58:00 -0300

Some projects I work with haven't yet abandoned Subversion. I try to tolerate it as much as I can, but sometimes (if I need local commits, or if there is heavy merging involved) it just won't do. Thankfully, I have bzr-svn to make my life less miserable.

The thing is, though; how to do the initial branching (“checkout” for those still stuck in svn terminology)? Because bzr-svn tries too hard at being atomic, and we all know SourceForge's Subversion server is made of purest fail. If the server decides to disconnect you in the middle of the operation, you lose all the (potentially hours of) work until that point.

After much frustration, I figured out the way to go with that.

First, branch from revision 0: bzr branch -r 0 https://crossfire.svn.sourceforge.net/svnroot/crossfire/server/trunk server-svn-trunk. Now you have a local branch that is already usable (well, usable as a branch, there will obviously be nothing in the working tree). Here lies the greatest trick, because while getting revision 0 doesn't actually pull any revisions, it makes bzr-svn do most of its hard mapping work.

Then get inside the branch, and pull the revisions in batches of (in my experience) no more than 500: bzr pull -r 500 && bzr pull -r 1000 && bzr pull -r 1500 etc. If something fails, you don't have much left to recover from.

You may be asking, if it's that painful, why do I bother? Simple: because it's only painful in the initial branching. After it's all up and running, it will be a lot less messy than dealing with svn, especially if I have non-trivial merging to perform. (Which, in this case, I do.)

The Muse: Bipolar disorder type 2?

lalo — Fri, 13 Feb 2009 01:15:00 -0200

Here's my latest archeo-neurological or archeo-psychiatric theory: “muse” is really an old term for bipolar disorder type 2.

It is my impression that old artists would sit around (or walk around) for days, weeks, doing nothing remarkable, or sometimes doing the hard, mechanical work of polishing up their creations. They would wonder where their “muses” are, why their genius is dormant.

And then one day, without warning, they feel that creativity, that exhilaration, that burst of awesome ideas, and a touch of insanity, that we've come to call genius; and they would attribute its less-than-constant presence to an invisible entity, the “muse”.

I believe, in our age, we call it hypomania, instead. A symptom of bipolar disorder type 2, a very common affliction that is frequently found in creative people. Paraphrasing Wikipedia, it's unknown whether creative types are more prone to be bipolar, or bipolars are more prone to be creative, or both are caused by a third, unidentified factor. What we do know is that the overlap is too great to be a coincidence.

Why is this interesting? Well, bipolar 2/creative people also tend to be romantics. So if you don't mind, I think I'll start referring to my hypomania as a muse, thankyouverymuch. It just sounds so much more desirable that way.

A reflection on game styles in CRPGs

lalo — Thu, 15 Jan 2009 03:20:00 -0200

This is an attempt at a taxonomy of style choices I've observed in CRPGs (I still refuse to call the game genre “RPG”; a game is not an RPG unless it in any way involves playing a role). This is based on “old-school” games I've been playing most of my life, on “massively social” games that have been popping up recently (and a nod to those that aren't CRPGs as well), and on second-hand accounts of MMORPGs, which I personally haven't played enough to form opinions on.

Special attention is given to how these choices affect the design of Crossfire 2.0, and other future projects I have planned.

If you first need an introduction, refresher, or fact-checker about the “scene”, I'll make a somewhat heterodox recommendation: Sluggy Freelance's Years of Yarncraft storyline is an in-depth, insightful, and accurate, if not serious, look on the whole thing. It doesn't cover the “massively social” phenomenon, but I hope I will do that in the article.

Player styles

I'll start with player and playing styles, because I believe understanding how people play a game and how/why they enjoy it is necessary before you can even discuss the rest.

Bear in mind, like most taxonomies when applied to people, most of my classifications below are talking about primary traits; most people will be a combination of different types, either by combination, or over time (like, on different days you may be an explorer or a role-player).

Primary activity interest

A very, very long time ago, people who played actual RPGs (nowadays relegated to being called “tabletop RPGs”, sigh) first classified players in two basic camps: the hack-and-slash camp, and the roleplay camp.

Over the years, and as we got acquainted with CRPGs and adventures and computer strategy games, this was refined, expanded upon, simplified again, and distilled. Today, while there are many ways to make this distinction, which are certainly valid in their own ways, I believe a simple split in four groups is the most useful, in terms of understanding and designing games.

So I call players either: gamer, hack-and-slasher (H&Ser), role-player (RPer), or explorer.

Hack-and-slashers are in it to fight. Nowadays, that doesn't necessarily mean actual in-game fights; rather, H&Sers like to win. They get fun from the game primarily by enjoying the rush of victory and superiority. In a classic CRPG, the easiest and most rewarding way to do that is by combat.

The key aspect to please H&Sers is challenge/reward: it must be relatively easy to find something that challenges them, the challenge can't be too easy, it can't be impossible, and the reward after winning must be proportional and appropriate, in order to trigger the reward pleasure centres.

You can alienate H&Sers with limits. If, for example, they can only run so many fights in a day, your H&Sers will probably just create second/third/fourth characters, or abandon the game after a short time. If they defeat everything you throw at them and then there's nothing more to fight and they need to wait a day, or three, or a week before there's more, they will probably not log in at all during this period, and there's a chance they will have discovered something else in the interval, which means you lost the player.

Role-players are run by imagination. They want to “be” someone else and do things they wouldn't do in real life. They really care about the character. They also form a mental opinion of the character's personality and preferences, and try to make choices according to that.

The key aspect to please RPers is believability: they must be able to suspend their disbelief when they start playing, and stay in their imaginary world until they decide to leave. Anything in the game that pulls them back to the real world is a disruption to that “flow”. Believability is not the same thing as realism; realism is just one of the ways of doing it, but it would be boring if all games were realistic; many (most?) RPers go to the games precisely in search of some “believable” fantasy. My personal “three pillars of believability” are internal consistency, depth, and detail; I'll expand on another post if there is interest, but I think it's pretty self-explanatory as it is.

You can alienate RPers (other than the lack of the above) with simple discouragement. RPers are a relative minority, and many other players think they are a bit weird. Every time you remove a feature that benefits RPers, or add one that hinders them, their interest will wane. If players are allowed to openly discriminate against them, they will leave. A good strategy some games adopt is to offer separate, RP-focused servers where RP is mandatory.

Gamers see the whole thing as a game. Which, technically, it is ;-) A gamer enjoys looking for edges and tricks to optimise the character, the overall strategy, battle tactics, economics, party/guild workings, everything.

The key aspect to please gamers is strategy: there must be cool, useful, and difficult to find tricks. There must be a wide range of possible strategies. But to avoid alienating the other players, it must be possible to play the game without them. It's also a good idea to “shake things up” occasionally; change the rules so that some strategies get nerfed, others spring up, and the gamers have to look for the new ones. They'll complain, but it won't be sincere; while they do have an attachment to their painstakingly-develop tricks, they have more pleasure in finding new ones than using what they have.

You can alienate gamers with ephemeral rewards. For example: say you offer a way to greatly improve weapons. Say it's either hard to do, or expensive; and say it's relatively hard to find. That's the kind of thing gamers love. But then, after a lot of effort to improve his sabre, he discovers that your game mechanics require changing weapons every few days. So now, all that effort is wasted.

Explorers want to see your game. They want to see everything, try everything the game has to offer. There are subgroups, of course; some want to see the whole world, some want to play every single last quest, some “collect” items, some want to understand the underlying story, some want to play many times with every possible class (or profession or whatever the equivalent).

The key aspect to please explorers is rich content: this one is a no-brainer :-) the more there is to see, the more they'll like it. But it has to be interesting; if every town looks the same, they'll stop by the 3rd or 4th.

You can alienate explorers with too much or too little obstacle. If I can simply create a character and explore the whole world, I'll have fun and maybe even write you a good review, but I will also drop the game after a few days. On the other hand, if I can't explore anything new for a whole week, I'll get bored and leave as well.

An interesting way to put it is that each of those 4 groups “plays with” a different portion of the brain. The H&Ser plays with the instinctive pleasure centres, the RPer with the imagination, the gamer with the intelligence, and the explorer with the curiosity.

Finally, there is of course a fifth group: those that aren't interested in the game itself, at all. There are many possible cases here; some people play MMORPGs and “massively social” games primarily to “meet” new people, or to chat. Others like looking at the game, because it's cute or cool or whatever; although those either don't stick around for long, or evolve into a specialised sort of “explorer”. Some play because they like the setting; this is especially the case for games that are adaptations of known fictional settings. Again, those players either leave after a while, or drift into one of the other categories.

To quest or not to quest

Good quests are one of the best tools of the trade. It's what turns a game into a medium for interactive storytelling. However, the plain, harsh truth is that some people don't really care about your story! Some people want to be able to enjoy the “regular” things to do in the game, and stay away from the quests, or do them just as much as absolutely required. Again, it's a spectrum; on one end, some players completely avoid quests, while on the other, they play for the quest and only do other stuff in order to “support” the quest requirements.

Wait, what? Some people like doing the day-to-day character maintenance more than quests, or even to their exclusion? It's easy to fall in the trap of assuming the day-to-day stuff is boring; but in reality, people are different and have different tastes. True quote from a real person in my tweeter feed today: “I hate when my routine gets disrupted. Life without routine is totally boring and not worth living.” I was amazed at first, but upon reflection, it just proves my old core belief that people are different; and it matches my observations of game players. Some really do come back every day for the maintenance routine, and find quests a disruption.

Worker vs. fighter

In the “massively social” CRPGs that are popping up online recently (I'll offer Travians as an example that I actually like playing), you have other things to do than going around bashing people's teeth in.

Now, on many MMORPGs you can do other things as well. The stories of “gold farmers” in WoW are a famous example; real-life sweatshops with people connected to the game making in-game items in an in-game sweatshop. But in most games, those activities are second-class; you can't be a level 70 tailor or miner, for example.

To explore the case of Travians, it started up not so much as a CRPG, but a kind of RPG-ified “The Sims”; just like “The Sims” was originally a toy to spy in more detail on the life of citizens from Sim Cities, and later grew to oversell its “parent” game by far, also Travians grew out of another game by the same company, called Travian (yeah confusing... but the company is German, and in German Travian is called Travian, but Travians is called Travianer, which is... well, slightly better). Travian is a strategy game, a kind of massively-multiplayer Catan/Civ. Then along came Travianer/Travians, where you get to play one of the actual citizens.

The funny side-effect of this story is that, although there is fighting aplenty and a quest, the main focus of Travians is not to fight, but rather to be a productive citizen. You harvest one of the primary resources from Travian (clay, wood, ore, or grain), or you process them into secondary resources (bricks, boards, coal, iron, flour, bread). Fighting doesn't reward you with money, only experience; to get money you need to work. (But if you really, really want to be a bum, you can live off of digging up treasure in the swamp.)

This experiment (and the success of Travians) brought a realisation: some people like doing things other than fighting. Some people like putting the character to work, or figuring out the complexities of trade.

And frankly? After the success of The Sims, we really should have figured this out earlier.

Simple vs. rich (or, dull vs. complex)

Complexity in a game is a tough issue. Complexity comes in all levels: quest, rules, user interface. Make it too simple, and many players will be bored; and, of course, you won't attract any “gamer” players. Make it too complex, and many players will be bored, or confused.

Personally, I'm a follower of the school of making complexity optional. Offer a very simple version; the easy-to-find quests, the basic rules, the default UI. But also offer layers and layers of extra complexity the player can opt in to: extra quests or side-quests that, preferably, tie into the main quest and enrich its story; additional corner rules for special, rare items, or for advanced classes, or something; optional elements in the UI that can be configured.

And then there are wizards. This is complete guesswork on my part, but from observation, I believe people who prefer to play magic-users, also prefer a little more complexity. It makes sense; understanding large spell lists, with the reward of having more options of action, as opposed to getting a weapon and bashing stuff — that's clearly a complexity choice.

The trade of trading

Trading is a human pleasure. Like most human pleasures, some people abhor it, others can't live without it.

In most CRPGs and CRPG-alikes, some form of trading is required. In Crossfire, WoW, or even old-school Diablo and Ultima, you'll get random stuff from questing, stuff that you don't want but that is worth money; on the other hand, some items you'll never find, or you won't find often enough, so you'll have to buy instead. In Travians and others, you need to actually produce and sell things in order to make game money.

But from that, sprung a funny tangent: some people like doing it. Some people enjoy looking for the best price, or even finding ways to make profit out of buying stuff in one place and selling it elsewhere.

I'm told in the Korean MMORPG Ragnarok Online, playing a trader is actually an option. It's still somewhat limited, as you still need to fight in order to level up; but it's an added choice, and apparently, lots of people actually go for it. I'd like to see a CRPG where fully playing a trader is an option; your goal is actually to make profit, and you level by making good deals. Or rather: I'd like it even more to write it ;-)

Game model styles

Time is money

A trend that is integral part of the “massively social games” phenomenon is limiting how much you can do based on time. Generally, that's done using some sort of points that regenerate through time, and which you spend to do some of the things in the game; Travians has “occupational points”, and Imperial Galaxy has “command bandwidth”, for example. There are ways to get bonuses to produce these points faster; in Travians you get more OP if you sleep in better beds, while in Imperial Galaxy it's basically a matter of keeping your “home sector” clean.

Strategy and building games use more resource-centred techniques; in Nile Online and Travian, your limit is how fast you can harvest resources that are needed as materials to build stuff. Also, each building takes a fixed time to build, and you can only have one construction going on (per city) at any given time; so if my current palace upgrade takes 12h, that means I can't build anything else on that city for the next 12h.

The purpose of these limits seems to be twofold. On one side, you want to avoid giving compulsive players that have no lives a very big advantage, because that drives everyone else away. But also, these daily limits are an encouragement for players to come back every day and do a little more.

This is a hard choice to make. And personally, I think there's an enormous risk of limiting it too much and becoming annoying; everyone I know who plays Travians agrees there aren't enough OPs in a day.

In some games, the limit can also serve to encourage players to explore other aspects; in particular, the more social aspects. But doing that well requires your limits to be partial (not affect the social aspects — Travians fails in this point), and more importantly, it requires the other aspects you want to emphasise as an alternative to be well-developed and interesting.

Specialise or balance?

This one has been an ongoing argument in Crossfire for as long as I can remember.

Some games strongly encourage you to build an all-rounded character. In Travians, for instance, I tried to ignore the combat aspects at first, but quickly found myself unable to complete quests. Then, still unwilling to split my (limited) attention, I decided to focus on combat; but after a few days, weapon upgrades became too expensive for me, and now I need to spend a few more days upgrading my tools. So their system essentially forbids focusing.

On the other side, MMORPGs usually encourage teamplay by giving strong benefits to cooperation between differently-focused characters; the typical successful WoW party requires (at least) one or two tanks, one or two damage-dealers, a spell caster, and a healer. (The typical D&D party adds to that recipe one trap finder and one leader.)

In Crossfire, focusing is good on lower levels, but the usual complaint is that pretty much all level 100 characters are identical (or possibly, can be divided into dragons and non-dragons).

It's all about choice

There are different kinds of players, with different preferences and styles. Do you try to allow all styles, or do you focus your efforts on pleasing one category? Both are valid choices, especially if development resources are very limited.

If you think this section is too short, feel free to re-read the first section of this post, Player styles, and reflect on all the game model choices hidden in there ;-)

Starting from zero we've got nothing to lose

In Years of Yarncraft, Pete Abrams pokes fun at this aspect, by having Torg's character start the game armed only with a stick, and having to kill a bunch of salamanders as his first quest.

How low do you want characters to start? The traditional answer, until relatively recent times, was that a starting character had to be reasonably capable, enough to entice and interest the player. But now, the trend seems to be starting very close to zero. And if the game offers a lot of choice, that may actually be a good thing, because then you won't have to make some of those choices until you have a better understanding of their effects.

Of particular interest to this aspect is the trend of making the first few quests a tutorial of sorts, teaching some of the most important things in the game. Travian plays this card heavily (possibly a little too heavily; you don't want people to feel patronised).

This also ties in to game mechanics choices (which I'm covering in the next post) about character growth and the meaning/algorithms of attributes and skills. It's common in simpler games to start all attributes at 0, which means normal human average, and then simply add to them a number of points per level, without any real final cap.

Those pesky economics

All RPG-like games must include some sort of an economy. In single-player games, that's traditionally just a question of making sure more powerful items are more expensive, while at the same time your ability to acquire game-money increases as you progress. Additionally, different items would be available for sale in different areas of the game world.

Then, simulation-heavy games and procedural content came along, and it became fashionable to try to achieve a “real economy” in the game world. I have personally not played a game that does that, or even seen positive reviews about one, so the benefit seems questionable.

But there's something better than a simulated “real” economy: a real real economy. Travian, Travians and Nile Online have all succeeded in doing that, Nile to a much greater extent. Quite simply, if most (or all) trading is with other players, and you have a sufficiently large number of them, soon real market factors will emerge, and in a way that players can understand without too much effort.

Maybe the ideal best is a mixed, real/simulated “real” economy, where player purchases have a strong (primary even) influence, but where items that players aren't really interested on can also fluctuate, as NPC demand for them grows and shrinks due to simulated or player-caused events. But the reason those web-based games have succeeded in creating an actual economy is one that may defeat this point: simply, those games have staggering numbers of players — thousands upon thousands online simultaneously at any given moment — which is enough population to simulate the economy of a small village.

On a side note, someone linked a very funny (but true) article about RPG economics to #crossfire earlier in the week. While the article is about D&D (and other “tabletop” fantasy RPGs), it mostly applies to CRPG as well.

To be continued

All right... I ran out of time and brain power :-) and I'm afraid nobody will bother to read this if it gets any longer than it already is. In a few days time, I'll write about game mechanics styles, and story/setting styles.

Automagically-translating chat thingy

lalo — Fri, 19 Dec 2008 20:23:00 -0200

Usually, I have to communicate with the people in the building's management office via Google Translate. It works, but it's awfully painful to be constantly flipping the language drop-downs back and forth. (It's two drop-downs, one for source and one for target language.)

So I wrote a little javascript gadget that does the hard work for me, and also keeps a “log” of the conversation. You can peruse it at http://lalomartins.info/transchat.html

(Attention though: this is not a chat app, not in the modern sense. It's “chat” in the old-school sense, of actually talking to a person that's in front of you. It's... an interpreter widget, not a chatbox :-) enjoy and spread if you wish...)

kill -FKOFFDAMMIT 25208

lalo — Sun, 09 Nov 2008 05:23:00 -0200

You know, we (Unix-y people) need a new signal, stronger than SIGKILL. As satisfying as it can be to type “killall -KILL firefox” (we just know that's the actual reason you still love the command line), there are a number of situations where that will still not get rid of the damn process; for example, if it's in the middle of some syscalls, specially nfs (grr!) or swapping (which is precisely when you need to kill it). So I'd like to propose a new signal which, let's say, waits for half a second, and if the process really doesn't respond, then gets rid of it for good, regardless of what else it was doing. In the middle of your quality toilet time, with pants down and all? Who cares, just get out. (If the process does respond, then I suppose do a -TERM... or rather, the other way around; send a -TERM, wait half a second, and if nothing seems to be happening, then bring out the ultraviolence?)

Here's a list of suggested names for the new signal.

SIGFKOFFDAMMIT
SIGKTHXBYE
SIGWITHCRUELTY
SIGNUKE
SIGLEAVENOTRACE
SIGCAPISCE
SIGNOSRSLY
SIGEXTREMEPREJUDICE
SIGGETTHEHELLOUTOFOURGALAXY
SIGCHUCKNORRIS
SIGEXTRACRISPY
SIGHULKSMASH

and my personal favourite:

SIGGOBACKINTIMEANDSTRANGLEITINTHECRIB

XML considered harmful, or,

lalo — Sat, 25 Oct 2008 15:37:00 -0200

I have, on a number of occasions, stated that XML is harmful, and should be taken out and shot. So here I am today, to explain why I think that, and offer alternatives.

Not good for humans

The main problem is, of course, that XML was never intended for humans. It's not designed so that we can efficiently write it, read it, understand it at a glance, or maintain it. But many tools that use XML today tend to forget that, leading to hours of wasted time and lots of frustration. (XML for configuration files, anyone? Zope's ZCML and .Net's configs and all those Java frameworks?)

Then, of course, that's not XML's fault; it was never designed to succeed at that task. The fault lies with developers who misuse it. Well, yes and no. The reason people misuse it is because it's overhyped; XML is the new peanut butter (or garlic butter, according to Pete Abrams) — adding it to anything makes it taste better and sell more. (I don't even like peanut butter.)

Not good for machines

What it was designed for is communication between programs; an unified, extensible format for data transmission. By having libraries to handle it in most languages and environments, you'd make it easy for developers to deal with it, and as a consequence, to make their programs communicate.

However, after roughly ten years of working with it, it is my informed opinion that XML fails at that, too. I'm not saying it got supplanted by better technology which we invented later. It did, to be fair. But what I'm saying is that it was wrong from the beginning. And if it's not good for us and it's not good for our programs, why are we still using it? (Peanut butter, I know.)

So let's try to break out of the hype and prove that it's bad for our programs.

The perceived problem with XML can be summarised in one sentence: XML is costly to parse. But that's too superficial; let's go deeper, look at the specifics, and the flaws in philosophy/design that lead to this perception.

Parsing XML: layers

I usually tell my co-workers that there's two “layers” to parsing XML. While that is true, it's only true in the context of our data; if I were to make that statement more generic, I'd say: there's always at least two “layers” to parsing XML.

The first, the “bottom” layer if you want, is syntactic parsing. This means reading XML itself: tags, entities, attributes, comments, CDATA, PCDATA, white space, the works. The input to syntactic parsing is a string or stream of bytes; the “output” is an API — SAX, DOM, ElementTree, you name it.

On the opposite end of the stack, the “top” layer so to speak, is semantic parsing, or extracting the data you're actually interested in. The “input” here is a generic API; in the typical case of two layers, the API from syntactic parsing. The “output” is a domain-specific API or, more commonly, a collection of structured data (usually objects, nowadays).

An example where you may have more than two layers is when you're using something else built on top of XML; the most common case being feeds. So at the bottom layer something will parse XML, then another chunk of code will parse that as RSS or Atom, and then your semantic layer will actually extract the data. At work, we initially made our data available as RDF; so we had a second, “middle” layer (we actually used a JavaScript RDF library) which would parse the RDF, and then we did our semantic parsing by using the RDF library's API. That made our code a lot simpler, but it also made it a lot slower; so we later switched to ignoring the RDF and simply treating it as XML. (Even later, we switched to a JSON format.)

Syntactic parsing: too much structure

Syntactic parsing is what XML is supposedly “all about”; the point being, you don't see it. In our case, at work, it's done by the browser (which gives us DOM with a touch of XPath). In pretty much any other case, it will still be done by your environment (the browser, in our case; JBoss and .Net are other examples), or by a standard library.

Well, that's great, right?

It is, yeah. But it hides the fact that those libraries (even if it's “hidden” in the environment, it's still at some level done by a library) tend to be huge and ridiculously complex. The XML syntax is designed to cover an enormous universe of cases that your program will concretely never encounter, and yet, you have to pay the complexity cost for them.

Semantic parsing: not enough structure

XML shines on xHTML: a markup language for text, where you have arbitrary streams of text sparkled with special instructions about it. Some of those “instructions” are really containers, which have more text and instructions. XML does that really well.

It shines a little less on something like SVG, where it represents arbitrary streams of heterogeneous objects. Some of those contain other objects, and XML does help there.

But the truth is that, for representing your program's data? It probably sucks. Its model is very different from the object model of most (all?) popular languages and frameworks today. In the end, we find ourselves designing our data structures as many as three times: once in the language in which we're actually writing it, one in a relational database, and one in XML. The mappings between them are often poor, since the semantics of the three models are so poorly matched.

Sadly, it would be relatively trivial to pick a lowest-common-denominator model that would fit all of today's popular languages. But XML didn't even try.

That's not the whole of my objection, though. Due to the MASSIVE FAIL in the syntactic layer, we get a semantic layer that's only marginally simpler than it would be to parse a DSL (domain-specific language); maybe less simple, if you use a good library for your DSL. There are about half a dozen XML APIs in wide use; smart people are frequently getting annoyed at the ones already there and coming up with a new, better one. And although a modern offering like, say, ElementTree can be light-years ahead of SAX or DOM, it can't help being clumsy and feeling unnatural to the language; at the bottom line, what it's doing is dressing up a rotting corpse.

Conclusion

Here's a better phrasing then, for the problem of XML as I see it:

XML has too much structure where it doesn't help, and not enough where it matters. One of the reasons I love JSON is that it's not designed to mark-up text, or to transfer “streams of data”; it's designed to transfer objects (JSON means “JavaScript Object Notation”), which means it maps nicely to my code on both ends, whether that code is JavaScript, Python, C++, or even C. (It maps nicely to Java as well, but who cares.)

Alternatives (existing and ideal)

Right now, for real-life code, most places where you're using (or thinking of using) XML would probably be better served with JSON. A few more complex cases may justify a DSL, but I would hesitate a lot before going down that route.

Ideally, I'd like to propose a new format; an “active” derivative of JSON, inspired by the modern practise of “JSON with callback”. Essentially, I'd like to replace JSON's “flat” object notation ({'attr1': 'value', 'attr2': 'value'}) with something which looks like a Python constructor (MyClass(attr1='value', attr2='value')). The pseudo-classes (or pseudo-functions if you're looking at it from C) would play the role that tag names play in XML elements, which would make it even more straightforward to map this data to actual objects on each end.

This would, of course, lose the benefit that “JSON with callback” can simply be executed in a browser. But then again, “JSON with callback” is not formally correct JSON anyway, so we already sacrificed some portability for that ability. “Real” JSON is usually converted to “JSON with callback” by a simple routine on the server side. A similar transformation could convert the format I'm proposing into JavaScript; the fragment above would become: MyClass({attr1: 'value', attr2: 'value'}).

Review: Sanctuary

lalo — Mon, 06 Oct 2008 23:23:00 -0300

This weekend I watched the first double-episode of Sanctuary, the new series in the Sci-Fi Channel. If you're a self-respecting sci-fi geek, you probably know that Sanctuary was created by one actor, one writer, and one producer of Stargate: SG-1, and that it started off as a web-based series. The double-episode is, in fact, the first “season” of the web series, with the tiniest bit of re-shooting and, dare I say it?, “re-post-production”.

The writing isn't bad, the acting is decent (great in some cases, but unfortunately not the lead), and the special effects are pretty good.

Still, I give it a “FAIL”. Sorry, but it's just not interesting. There's nothing new, there's nothing that happens there to keep me interested. Supernatural creatures living in secret in our world? Yawn, that was cool in the early 90s. What, so the big secret of the “mysterious” Doctor Magnus is that? Sorry, that was already old in the early 90s when the rest of the premise was cool. Also, you just ruined the “mysterious” part by revealing it so soon.

It's also too slow on the first half, lots of talking heads and little plot progress, with the second half having too much action and little plot progress. In fact, plot progress tends to happen in “bursts”, which is, sorry, not good at all.

Good try, but I won't be coming back for the next one.

Freedom for Whom?

lalo — Wed, 24 Sep 2008 16:50:00 -0300

I think I've seen this argument for the first time in a Slashdot comment, years ago. I've since adopted it, refined it, and used it a lot myself; but now in light of the Android release, I think it's worth mentioning again.

The big problem I see with “Open Source” is that there are, in fact, two groups there. Fortunately the same is not true of Free Software, but even our arguing that it's about freedom still doesn't help... well, read on.

The thing with “Open Source” is: who is it open to?

Arguably, Open Source, as a vague, undefined thing, has existed for decades. But as a conscious, named movement with its own marketing, it spun off from the Free Software movement in the late 1990s, after the “open-sourcing” of Mozilla and the publishing of The Cathedral and the Bazaar. (Or, according to some, it spun off a few weeks later, when RMS noticed those guys were talking about something else and split off from the Open Source initiative.) Still, in hindsight, one can say things like the BSDs, and even the original Unix, were done more in the spirit of Open Source than of Free Software.

Now Free Software, with all its GNU/FSF writings, has always been very clear about its goals. We're here for the freedoms of the user. If you get a piece of software, you have a bunch of inalienable rights, rights that aren't being respected by most software, and which we intend to uphold and defend. Nice, eh?

Open Source people, on the other hand, seem to be a little confused about this. It's like watching two madmen (or drunks) arguing, each founding an argument on an entirely different premise. Some, perhaps still in touch with the “origins” of Open Source in the 90s, believe it's about being “open” to the users of the software. Others have adopted the belief (from BSD maybe?) that it's all about “openness” to the developers.

(More importantly, some of them don't realise Free Software ≠ Open Source, and mistakenly argue this in even more confusing terms; like the old fallacy that the GPL, and viral licenses in general, are bad for Free Software because they give “less freedom” than BSD-style licenses. They do, if you're thinking of other developers, who will then have the “freedom” to “steal” my software and use it in their own closed software, and not give back to the project in any way. I don't care the least about those; I'm writing software for the freedom of my users, and those have their freedoms enforced by a viral license. Now are viral licenses bad for Open Source? Honestly, I couldn't care less.)

The Android platform seems to be firmly planted in the latter camp, sadly. (Or maybe not so sadly; I rejoice with every Java-based product that fails.) It's “open”, first and foremost, for handset makers and network operators, and a distant second, to application developers. “Openness” for the end-user doesn't seem to even be a consideration. Now of course, both things are pretty much incompatible; being “open” to the operators means, really, “open” for them to “close” it in whatever ways they want; so yeah, no VOIP.

Oh well. At least I don't need to be conflicted about whether I want an Android device, whether I can stand Java long enough to actually like the OS. Clearly, that won't be a consideration, and OpenMoko — or, if they fail, someone else, probably using LiMo or FSO stacks — will be the mobile phone for me. Eventually :-)

Creationists, I feel your plight

lalo — Mon, 15 Sep 2008 06:46:00 -0300

You know what... I do understand and respect a family's right to believe any idiotic nonsense they want to. Who knows; it's a seriously messed up universe we live in, they could even be right.

But whether or not they're factually right is not what matters most in my book (which is not a holy book). Rather... their theories are grounded in a larger belief system, which has at its core a moral and ethical code. Telling the children in absolute terms that their parents beliefs are wrong, understandably, undermines the whole system, which in turn undermines the code.

(I'm not saying American creationists have a great code. Those are often the same people who are racists, xenophobes, or quick to judge someone by the bank account. But it's built on a good stem of what Americans call "work ethic", and arguably, it's better than no code at all.)

At the root of the argument is, in fact, a completely different wrong, which is the one I actually wanted to address in this post. The problem actually does lie with the (majority of) science teachers, who fall for the trap I call "the religion of science".

Good, respectable scientists almost never talk about facts. A fact, or rather an absolute rule, is anathema to good science; the only facts useful to scientists are the results of their experiments, everything else is theory.

What about bad scientists? Oh, those are frequently 100% sure of obvious truths, like, "the atom is indivisible" and "heat is a fluid" and "there's no such a thing as a memristor". (They're particularly eager to label things "impossible".)

The pattern here is: an open mind is the first pre-requisite for science, even more important than genius.

But in school -- and not only America, but many other places -- you only start hearing this kind of talk in college, or if you're really lucky, high school. Basic science, the teachers who were supposed to be giving children the basics of science and awakening potential future scientists, generally fall for "the religion of science". They "teach" absolutes; this is so, because that is so. Which is specially funny since some things they teach in basic school are known to be untrue, like Newtonian physics.

We wouldn't be even having this argument if school science teachers could behave like good scientists and formulate their teachings like, "evidence suggests that..." and "it is believed that..." and "apparently, ..." and, often, "this and that evidence points to the conclusion that...".

What, actually teach creationism in schools? No, now you're pushing it. Come on, the other parents also have the right to keep their children safely away from your nonsensical idiocy.