In a way, this post is a continuation of my ponderings on the skills of the thief class, but it necessarily encompasses the much broader topic of surprise in encounters too.
Let us stipulate that the thief class does not have a monopoly on stealth. We've all amused ourselves at some time or other by sneaking up behind a friend or a sibling and then startling the holy bejeebus out of him or her, and most of us have not trained as thieves. Anyone can try to move quietly, and anyone can hide by ducking behind a curtain or a tree or around a corner. Moving with absolute silence and hiding in nothing more substantive than shadows are the province of the thief, and to some extent, the halfling, class. (Edit: Here's where this brilliantly simple idea came from. Thanks, Brendan.)
That leaves us with the questions of the game effects of moving quietly, moving silently, hiding, and hiding in shadows, and how they relate to surprise in encounters.
According to the Basic Set rules (either Moldvay or Mentzer edition,) each side in an encounter has a 2 in 6 chance of being surprised. There are all sorts of circumstances that could affect the chances of someone being surprised, including how vigilant or distracted he is, how stealthy the opposition is, and how much ambient noise there is in the location that would not alarm the listener but would mask warning sounds. When some of those factors are unknown, the dice roll can be considered oracular with regard to the side whose precise current actions and circumstances are unknown. If the roll indicates the monsters were surprised, perhaps they were arguing amongst themselves, intent upon a game of dice, or devouring a recent kill. If the characters are surprised, it might mean that the monsters were taking great pains to be quiet. Quite often in published adventures, some of these factors are made explicit, and explicit adjustments are made - the orcs are noted to be rip-roaring drunk and automatically surprised, or the kobolds have set an ambush and surprise on a roll of 1-4 instead of 1-2, or the goblin guards are watching the only entrance and surprised only on a 1. There's nothing wrong with this, but if you don't care to figure out these details for every encounter, the usual 1 or 2 on 1d6 mechanic can reasonably be considered to subsume all those possibilities.
With regard to the player characters' party, the 2 in 6 chance to surprise and to be surprised represents a typical party, including a couple of fighters in clunky metal armor, exercising typical caution and typical vigilance for a foray into dangerous environs. If the players specify that their characters are taking greater pains to be stealthy or vigilant, it may warrant some adjustments.
In many ways, I think the official rules leave a lot to be desired when it comes to surprise and stealth and the relationship between the two. What follows are my own interpretations of the rules and, where needed, house rules to supplement them.
Moving normally is the default assumption for both characters and monsters, with the default odds of surprise. Characters or creatures who are alert and listening, according to the rules for making Hear Noise checks, will detect the noise from the party automatically, and will probably be surprised only in exceptional circumstances (e.g. invisible opponent.)
Moving quietly for a non-thief should require some sort of check. Let's keep it simple and make it a d6 check, successful on a 1-3. Modify it by the character's Dexterity adjustment to individual initiative. A penalty should apply to characters in metal armor - say, +2. Movement rate is reduced by half while moving quietly. Thieves succeed automatically, and characters or monsters standing or sitting still and not talking are also considered to be automatically quiet. Characters or creatures who are alert and listening will detect the noise from those moving quietly with a standard Hear Noise roll (1 on 1d6 for human, 1 or 2 for demihuman or monster, Hear Noise percentage for thieves) and are unlikely to be surprised. Moving quietly and undetected increases the chance of surprising creatures encountered by 1, i.e. to a 1-3 on 1d6.
Moving silently can only be done by a thief or other character with the thief's Move Silently skill. A thief can automatically move quietly whenever he or she chooses, and with no penalty to movement. A thief may also attempt to move with utter silence, making no sound discernible by human hearing. Listening for noise is useless. Chance of surprising encountered creatures is 1-3 on 1d6. A thief who fails to move silently is still moving quietly, with all the benefits described above. A thief who remains motionless, as in hiding normally or in shadows, is automatically silent.
Note that either kind of stealthy movement requires everyone in the party to be using it successfully in order to gain any benefit. It does no good for the thief to move with absolute silence if the clumsy fighter in plate mail is rattling like a wagon load of tin pots on a bumpy road.
Hiding can be done by anyone, provided there is some sort of cover or concealment. Hiding in a completely bare, square room is futile, but a square room with alcoves offers some possibilities, and a square room packed with crates and barrels is rife with potential hiding places. Characters or creatures in hiding are considered to be quiet, and may be detected by a successful Hear Noise check, though the listener will not necessarily know the exact source of the sounds nor what produces them. Hidden characters or creatures, should they choose to spring out before they are found, surprise their opponents on a roll of 1-4 on 1d6.
Hiding in shadows can only be accomplished by a thief or other with the Hide in Shadows skill. A thief could try to hide in an empty square room, as long as there were some shadows or patches of darkness within it. A thief striking from the shadows surprises on a 1-5, and the attack counts as a backstab. This ability may be used in combat, but requires that the target's attention not be on the thief, that nobody else warns the target, and that the thief remains hidden for one full round prior to making his backstab attempt, forgoing all other action for that round. In all editions that I've read, only the Mentzer edition Basic Rules allow a thief to move while hiding in shadows. All other editions specify that the character must remain perfectly still; this makes the most sense to me.
Note that unlike quiet and silent movement, hiding and hiding in shadows are mutually exclusive; a thief may attempt one or the other but not both at the same time, and a failed attempt at the latter does not default to the former. Naturally, either kind of hiding will be automatically unsuccessful against anyone watching the character at the time of the attempt.
A creature's level of alertness will also affect its chance of being surprised. Being completely absorbed in some task or activity makes it 1 point more likely to be surprised. Being drunk, drowsy, or otherwise impaired likewise merits a 1-point penalty. Doing nothing but watching and listening for approaching threats, as a vigilant sentry would do, garners a 1-point bonus if watching all directions, and a 2-point bonus if watching one direction specifically but a 2-point penalty if approached from the opposite direction. If no extraneous noises are present, such a sentry should automatically detect normal movement, and receive a Hear Noise check to detect quiet movement as well, further reducing the chances of being surprised.
One other aspect of surprise and stealth that I've found sorely lacking is the distinction between observing the other party without being observed yourself and simply catching the other off-guard. If you surprise the goblins, can you choose to slip away unnoticed, or do you just have a few seconds to get an uncontested first strike or a head start on them before they can react? The Expert Rules clearly state, with regard to wilderness encounters, that, "Any group may always avoid an encounter if it surprises another group," which very heavily implies that the surprising group notices the other group without being noticed itself. I'm tentatively going to say that this could be applied in most dungeon encounters as well, except for those where it would be obviously ridiculous. If you surprise a group of goblins in the corridor, you've spotted them first, and can choose whether to attack or quickly duck back around the corner. If you surprise a group of goblins by bashing down the door to their room, you can get the drop on them or run like hell while they stand gawking, but they certainly know you were there!
Whew! Definitely my most long-winded post in quite a while, but hopefully illuminating and/or useful. Comments and critiques are welcome.