Note: this isn't really a review of Serenity, more a reflection of the fact that the film finally allowed me to crystalize some of my thoughts about Mal's character. As such, there are plenty of spoilers ahead. If you're looking for reviews, here are some interesting ones from Gary Westfahl at Locus Online, Strange Horizons (a double review, from the perspective of a Firefly fan and a newbie), fellow Readerville-ian Peter Cashwell, and The Washington Post. Also see John Scalzi and Alan DeNiro on what Serenity's box office means for the chances of a sequel.
Like teenage girls with superpowers, Joss Whedon can't seem to keep away from the concept of heroism. With his first effort, Buffy the Vampire Slayer, he dealt with the most conventional form of hero--the chosen one. There were twists to his version of the story in that his chosen hero was a flighty blonde girl who would have preferred a trip to the mall to saving the world, but ultimately Buffy's story boils down to a person who is chosen by destiny to save the world. With Angel, Whedon moved one step away from tradition by positing a hero who was, fundamentally, just a guy. Although gifted with superhuman strength and something nearing immortality, Angel was in temperament and inclination an ordinary person who chose to shoulder a burden he was unequal to. However, Whedon chose to set Angel's story in a universe in which good and evil were distinct, identifiable entities--they may have been called The Powers That Be and The First instead of God and The Devil (for an emphatically self-proclaimed atheist, Whedon sure took a long time to wean himself away from these notions of powerful protective and destructive entities), but their function remained unchanged. Within the Buffy and Angel universe, there existed forces for good, and a battle for good, that our heroes, although they could never understand or hope to win them completely, could join with the knowledge that they had some flavor of benign divinity on their side.
When we first meet Mal Reynolds (or, when we were first supposed to meet him, for those of us who watched "The Train Job" before "Serenity"), he genuinely believes that he lives in the Buffyverse. Despite being in the middle of a war zone, Mal is exuberant, cramming more joy into a few minutes than we see from him in the entire run of the show and the movie put together, and he might tell his frightened soldier that it's because "We're too pretty to die", or himself that it's because of his faith in God, but what it all boils down to is that this Mal believes in a moral universe. He might never put it this way, but he believes that there exists a force for good, and that a person who gives their all to a good and worthy cause will find the universe on their side (or at least on their cause's side). But of course, with Firefly, Whedon has moved away from the notion of any sort of moral authority that exists outside of ourselves. Firefly's universe mirrors our own in that it is fundamentally unfair, which is the lesson that Mal learns when the Independents surrender and his heart breaks into a thousand tiny, jagged pieces.
And from that point until the final shots of Serenity and probably after, Mal is furious. He's enraged at the universe for disabusing him of his belief that he lived in a world in which heroism was possible. Any other man would have been defeated, would have succumbed to nihilism and hopelessness (as we saw Angel do more than once) but Mal's anger won't let him do that. He vacillates between heroism and selfishness, never quite facing up to the notion that it is possible, albeit extremely difficult and ultimately unsatisfying, to be a force for morality in an amoral universe (something that Book tries to teach him in their penultimate scene together). Mal's broken heart tells him that heroism doesn't exist, that acting selflessly will have no meaningful consequence, and that he should take care of himself alone, and he listens to that heart quite often. But whatever it was that once made Mal believe in some good greater than himself sank its roots deep, and Mal constantly returns, furious and reluctant, to a fight that he doesn't truly believe in. As Inara so accurately puts it in Serenity, Mal's character is impossible to nail down because he keeps changing his mind about it. It's not just, as my brother put it after seeing the film, that we can't decide whether Mal is a hero or a complete bastard--we can't decide what kind of hero, or what kind of complete bastard, he is.
More than any other Whedon show, Firefly played with our stereotypical notions of heroism--mostly because it was grounded in the forms of a genre, the Western, that even more than epic fantasy is synonymous with a stark distinction between shining white good and deep black evil (it'll be interesting, one of these days, to talk about Firefly in relationship to that other reimagined modern Western, Deadwood), and with very codified notions of how heroes should behave. Although the writers usually used this breaking of stereotypes for comic relief (while rescuing Mal from a gangster who's been torturing him, Zoe announces that the crew should hang back, as Mal needs to deal with the man on his own. Mal: "No, I don't!"), it's also a reflection of the show's central theme--the impossibility of heroism in the mundane world.
Possibly the most fascinating and tangled aspect of Serenity is the parallel drawn between Mal and The Operative. Whereas Mal has discarded his idealism--allowed it to be burned up by his rage--thus turning himself into the kind of man who would choose a stolen payroll over a man's life, The Operative has sublimated himself--discarded his humanity--in the service of an ideal, and it's a hallmark of Firefly's twisted take on morality that the former is the hero and the latter the monster. They're both wrong, of course, and yet they both have an insight into the very aspect of humanity that they've chosen to turn away from--love. Serenity begins and ends with these two men musing over the power and danger of love, and the fact is that it is Simon's love for River, and her love for him, that enable the truth about what happened on Miranda to be known--a love that neither The Operative, nor Mal, are capable of expressing.
Because, of course, Mal is afraid. Like most people burned by love, he's wary of letting it back into his heart. And because he is fundamentally a big, melodramatic baby, he's incapable of sucking in that fear and confronting it. Having sunk his emotional capital into an ideal and watched it fade away, Mal turns around and gives his heart to something he can touch and control--a ship. Serenity is a piece of Mal's soul, and his only meaningful interactions with other humans go through that steel buffer. The people Mal loves are his crew, not his family, and although they are no less holy and important to him for that fact, he can never let himself interact with them in any capacity other than as their captain (his final benediction to Book is to tell the dying man that Mal still considers him a member of his crew).
All of which boils down to the fact that Mal Reynolds really needs a big hug, and a sharp swift kick to the head, not necessarily in that order. But more importantly, someone needs to tell Mal to get over himself and grow up.
But will he? If Serenity has a theme (beyond 'we'd really like to let everyone know what the deal is with River and why the Alliance is after her'), it is the fundamental wrongness of trying to force people to be better than they are. Is this possibly a sign to the viewers, who were looking forward to a long, cathartic journey for Mal, at the end of which he'd be happy and well-adjusted and would marry Inara and have lots of babies? Unlike Buffy or Angel, most of Firefly's characters are adults whose character, for better or worse, is already fixed. There will be no journeys into adulthood for Mal, which is probably for the best--it would be a shame to lose one of the most fascinating, conflicted characters in modern SF.