A few days ago, I was looking at a recent Blog Watch post on Jeffro's Space Gaming Blog, and found a post in which the author argues for abolishing the concept of experience points as a metric of advancement in D&D. I won't recapitulate his case here; suffice it to say that although I strongly disagree with his conclusions, it's worth a read. (The Time for Experience Points Has Come and Gone)
I found the points raised against XP to be worth considering, but ultimately I can dispense with most of them fairly easily. The problem of tedious accounting has never been a big deal for me. Rather than the fifteen minutes of boredom described in the post, in my experience calculating XP is pretty quick and straightforward with a pocket calculator, and for the five or (at most) ten minutes it takes, the players are happy to talk amongst themselves about how great the climax of the adventure was, or hashing out the distribution of treasure.
The problem of sacrificing adventure quality for insuring that the "proper" amount of XP is earned is really only a problem in games in which the pacing is dictated by story concerns rather than player choice. Far be it from me to condemn a style of play that someone enjoys, but for my own preferences I cordially despise story-oriented games. In the sandbox, whether it takes the characters ten sessions or two to gain a level, there are always appropriate challenges for them. This is aided by the old school philosophy that "encounter balance" is at best a very loose guideline rather than an ironclad rule.
The charge that XP provides incentives inimical to "good" play is one that deserves a more in-depth response. First, I want to talk about what's right about XP.
Why I like XP
Alternative methods generally involve advancement after some number of sessions or adventures, as determined either by the DM or by group consensus. While this makes advancement very consistent and predictable, it also divorces it from player choice and performance during the game itself. No matter how well or how poorly they played, how bold or how craven, how clever or how obtuse, characters are guaranteed advancement, so long as they survive. I'm not saying this is categorically a bad way of handling advancement (especially if you're one who enjoys a tight story arc rather than a freewheeling sandbox), but if you're looking for problems of incentives, this is a big one. It incentives very cautious, risk-averse play.
An XP-based system, on the other hand, allows characters to gain in level organically within the capaign setting, dependent upon skill of play and risk tolerance. Yes, the amount of XP available is to some degree determined by the DM, but whether the player characters actually walk away with any given chunk of it depends on whether and how effectively the players decide to face the challenge. If you do it well, they can't just chew through one encounter after the next; they'll have to exercise discretion and cunning, and leave some stones unturned. The shrewdest players know the limits of their characters and the signs that a situation is more than they care to take on.
On the problem of incentives
XP for monsters and treasure, says the author, encourages the players to fight every monster and grind through the entire dungeon room by room, even when it's counterproductive to story considerations.
The author uses the example of rescuing a princess, and posits that an XP system incentivizes the PCs to ignore quick solutions like teleporting directly into the chamber where she is held prisoner, thereby bypassing all the XP-rich enemies and treasure hoards. Now, if your villain is foolish enough to let everyone know where the princess is being held, and doesn't even take precautions to prevent that teleport-quick grab scenario, I don't really know what you're doing designing adventures in the first place. The more likely scenario - and frankly, in my estimation a desirable one - is that the PCs will have to do some reconnaissance and investigation to find out where the princess is and determine the type and strength of the villain's defenses, and in the course of that, they're going to have the opportunity to fight and discover loot. Gaining of XP through monsters and treasure can happen organically, even within the context of a story goal.
I think it needs to be said that XP is not the only
incentive in play in a game session. Survival is another important
consideration. The sandbox almost by definition contains places and
creatures beyond the strength of characters of any given level, and discretion is a must. Over
the long haul of an adventure, supplies of hit points, spells, light,
and rations can also conspire to deter the systematic room-by-room
looting of a dungeon. Charging headlong into danger and pressing resources to the limit to scrape up every last bit of XP must be weighed against the likelihood of character death or even TPK. XP aren't of much value to a dead character. (Unless death is cheap due to prevalence of resurrection magic - which in my mind makes a much stronger case for abolishing or sharply limiting such magic, not XP.)
Other non-XP rewards for skillful adventuring include boons and benefits within the campaign world. Rescuing the princess earns the gratitude of the royal family. Rediscovering a lost trade route unlocks new goods and services for purchase. Turning the orc queen and her hordes against the evil duke earns the respect and admiration of the villagers who were victimized by both. What precisely any of these things mean for the campaign and the PCs is open to interpretation, but this kind of reward can be every bit as satisfying and as useful as XP, sometimes even more so.
In fact, I would go so far as to say that if your players are motivated solely by acquisition of loot and XP, you might want to look in the mirror before you point your finger. When players don't have any reason to care about in-fiction rewards and consequences, it's natural that they'll default to the only thing that still has meaning for them, which is accumulating power by leveling up. It's on you, as DM, to provide a dynamic, compelling setting, and give the players reason to care about that world and its people. One way to do that is by making it clear that the players' choices have a real, noticeable effect on the long-term course of the campaign setting, whether it be a village, a megadungeon, a kingdom, or a whole world. When the status quo prevails whether they rescue the princess or not, you're providing no incentive to do so, and your hopes must hang entirely on them being "good sports" and going along with your story in which they have no personal investment.
There's nothing inherently wrong with a "kill monster, grab treasure" game, but if you want it to be more than that, in most cases XP is not the culprit. XP is but one motivator of several. It's the easiest to utilize in play, because it requires little thought beyond adding numbers to a total. The others are more difficult, because they require careful consideration of the consequences of player actions, and openness to allowing those changes to happen to your campaign setting, even if they cause it to deviate wildly from your original vision.
Finally, there may be some players who just refuse to care about anything but powering up their characters, but it's still a mistake to point the finger at XP per se. A game without XP simply hands them what they want without their even having to work for it, and my guess is that will bore them pretty quickly. If it doesn't, then they're not really that type - they can be motivated by things other than XP, and we're right back to your responsibility as DM to provide that motivation. Simply removing XP from the equation doesn't make your setting and NPCs interesting. If your players really are incorrigible power-mongers, your options are to learn to love the game as they like to play it, or find yourself some players whose idea of fun gaming aligns more closely with your own.