Comics Online and Interactive Music (Not particularly interesting as a title)

My brother sent me this link, largely as a way of bragging about IE9 and HTML5 because he drank the Kool-Aid some time ago, but then we got into a conversation, which I wanted to share because it lead to a short but interesting discourse.

Basically, I thought the site was pretty cool. I wouldn’t want to read comic books that way. I actually haven’t found an online format that “works” so to speak for the traditional graphic novel approach. Being able to embed supplemental information is neat though. I rather particularly didn’t like the custom cursor, but I forgot it was there relatively quickly.

Interesting idea from a music/programming standpoint would be trying to cue the music to match roughly where the “camera” is along the path of the comic, but with a smooth transition. The programming side of it isn’t complicated if you write the music well-enough to do the transition smoothly at basically any point, but it would be a really neat script if it could use essentially “blocks” of music to create an interesting transition based on the speed of the reader.

To which Harris responded.

Perhaps I’m over simplifying it but it sounds like what you’re referring to is pre-canned loops of music, similar to what’s used in a video game…For reference – the only concrete one I have – take Halo: Combat Evolved and my favorite level Assault on the Control Room (thank you Halopedia ;).  Essentially, you’re loops would be queued up at various points in the story as the reader progresses through the plot.  Having the various cue points embedded in the story to trigger the music wouldn’t be that difficult…

Perhaps I’m making it more simplistic than it needs to be…

Which prompted the following thoughts:

The problem is how long you make the loops. Halo is actually an extra crazy cool example because Marty and Jay and the gang came up with ways of phasing music in and out for specific sections. Halo 3: ODST is a great example of this (and totally worth picking up for $15 or whatever it is now), partially because the storyline is so compelling, but, in general, Halo pays attention to the number of enemies that are active and the amount of approximate damage flying around. So if you just sit in a corner shooting somewhere, it doesn’t care. But if there are 40 to 50 enemies in an area and they become aware of you, a general “battle motif” starts playing, and then at a specific triggered time, the main melody kicks in and can progress for quite awhile before it runs out, but in general, the battle motif sticks around until you move on to the next section, or, alternatively, will swell and then die once you kill all the bad guys. If you take way too long, it will eventually kill itself off.

The problem in the story version is how long you make the loops – in order to do it your way, they have to be super short so that you can account for any reader’s reading pace and jump to the next piece at the appropriate moment. Too short, as you know from listening to looped music, and it’s super bland and boring. Too long, and you’ll miss a cue, and then possibly another one as the reader zips through a page that should have taken longer to read (possibly because they’re not doing it particularly well).

Then what if they go backwards? That’s a problem games don’t face. In games, there is always the linear, forward progression of time. It’s not uncommon for me to flip back sometimes and check out a panel again, re-read some dialogue, or even observe the layout and try to infer the timing from it and the different ways they could have laid out the panels. That’s part of what reading comics is all about – accepting this visual presentation in just about any order you want to. Games, TV, and Movies don’t have this problem. They control the flow of time through their story, and should exert that control in interesting ways.

This isn’t always true of games, of course. Prince of Persia: The Sands of Time (I am officially adopting the grammatical quality of italicization to denote names of specific games. I have decided it is appropriate, and thus it is!) literally lets you rewind time, but they just play a cute sound effect and spool the music backwards. With a comic, particularly the Tron: Legacy comic you presented there, the user/reader/interactive participant has explicit control of “time” in the context of the story. If I want the stadium where Sam and Rinzler fight to rise slowly to the teeming, cheering hordes, I can do that! But the music has to know (or somehow be able to sense) that I’m performing a more dramatic pan through this panel.

Now, what I really liked about that panel is that it’s impossible in a traditional comic. What I didn’t like about it was that it was impossible in a traditional comic. It’s weird, but it was very much a motion-comics thing that works really well as a single panel within a bunch of other simple, non-moving panels.

I got a little carried away with this stuff.