For all its many strengths, there are a few areas in which D&D fails in epic fashion. Either there's a glaring hole in the rules where something that desperately needs to be addressed is not, or there's some bizarre kludge dropped into the hole which is either inconsistent with the rest of the game, absurdly complex to the point of being virtually unplayable, or both. Among those epic fails are the game's attempts to model unarmed combat.
I've only briefly glanced at the 1st and 2nd edition AD&D unarmed combat rules, so I can't comment much on them but to say that even at a glance they seemed convoluted and bizarre. What I've read in the OSR blogosphere from people much more familiar with them only reinforces that initial impression. Classic D&D has kept it simpler, at least relative to AD&D, but it's still pretty weird.
Frank Mentzer's Companion Set contains the first attempt, so far as I'm aware, at an official unarmed combat system for Classic. It includes such mechanics as rolling against Constitution to check for stuns, and saving throws to avoid being knocked out. Of course, in real brawls and fist fights, people are sometimes stunned and knocked out. Tacked on to an already existing system in which people beat on each other with clubs and hammers with no chance for stuns and knockouts, though, it looks pretty absurd. You can knock out a guy with 50 hp, or leave him reeling and unable to defend himself for a few rounds, by striking him with a fist, but whack him over the head with a mace and he takes 1d6 damage and keeps fighting? That's just silly.
There's really no reason that unarmed strikes need a different system at all. A knockout is aptly modeled by reducing an opponent to 0 hp with unarmed strikes. Just as the 50 hp fighter doesn't get run through every time he's successfully attacked with a sword, he doesn't take it square on the chin every time someone lands a haymaker on him, either. All the damage leading up to the final blow, whether lethal or non-lethal, is about wearing him down, rattling his confidence, using up his luck, or whatever the fashionable explanation of hit points is these days. It's the final blow that finally strikes true and lays him out.
How much damage an unarmed strike should do is open to debate, and varies from source to source. Some prescribe a base damage of 0 or 1 point, modified by Strength bonuses. Others say 1d2 points, or 1d2 for a punch and 1d3 for a kick, plus Strength bonuses. Still others start at some base amount, like 1 point, and increase it by one die size per Strength bonus - 1d2 for a +1, 1d3 for a +2, and 1d4 for a +3, for example.
Personally, I like the punch 1d2+St/kick 1d3+St formula. I'd probably apply a -2 penalty to hit with a kick, unless the target is much lower to the ground or the attacker is standing on a higher level relative to the target, e.g. a brawler standing on top of a table kicking someone standing beside it.
Stuns are a little more problematic. They could be hand-waved as just an expected part of the 10 second (or 6 second, or one minute, or whatever your edition uses) combat round, like feints and parries and jockeying for position. Alternatively, on any critical hit with a weapon that the DM rules capable of stunning, the target must making a saving throw (vs. paralysis seems an appropriate old school save) or be stunned for 1 round per 2 points of damage inflicted by the attack. It seems reasonable to limit stuns to creatures twice the mass of the attacker or less.
Wrestling or grappling attacks, which attempt to grasp and restrain an opponent rather than causing damage, are another problem. You could handle it in the same way as unarmed strikes, wearing the opponent down and capturing it when its hp reach 0, but that produces some odd and counterintuitive results. For one, what happens to the opponent's hp when the attacker releases it? It seems weird to have it remain at 0 hp, but equally weird to suddenly regain all of its hit points. It also runs counter to common sense that simply grabbing hold of someone should require completely wearing down his resistance.
Mentzer's wrestling rules from the Companion Set are serviceable, but add a new game mechanic (Wrestling Rating, which is used for nothing else, and sometimes breaks down in strange ways, especially when calculating it for monsters) and a lot of complexity to combat. Below is my tentative attempt at integrating grappling into the usual combat mechanics.
A character attempting to grapple an opponent makes a normal attack roll vs. the opponent's AC sans armor. Dexterity and magic still apply, but physical body armor does not. Success indicates that the opponent has been grabbed. If desired, the attacker may attempt to grab a specific body part (e.g. the opponent's sword arm) by accepting a -4 penalty to the attack. An opponent using a conventional attack, either armed or unarmed, automatically wins initiative and may strike first against the would-be grappler. If the attack hits, the grappling attempt is fended off for that round, and the grappling character may not take another action. Note that if the opponent is attacking a different target entirely, and ignores
the grappler, actions are resolved according to normal initiative procedures, and the opponent does not automatically win initiative over the grappler.
A grabbed opponent may be limited in its movement (opposed Strength check to see who determines the direction of movement, but the winner is still encumbered as if carrying the other.) It may attack with a small or medium weapon or natural attacks, unless the specific limb holding the weapon was the target of the initial grab. Attacks against the grabbing character are +2 to hit, since the character cannot evade them effectively while maintaining his hold. Alternately, the opponent may try to escape the hold by making a grappling attack of his own; if it succeeds he may choose between throwing off the grappler or establishing a grip of his own (both combatants are now holding onto each other.)
A second successful grappling attack results in the opponent's attack being neutralized completely, and the grappling character may inflict 1d6+Strength bonus of subdual damage for each round that he maintains this hold. This is the equivalent of placing the opponent in a hammer lock, full nelson, choke hold, or similar. A successful armed or unarmed strike by the opponent once again fends off the grappler's attack, although the initial grab is not broken unless the grappler chooses to break it; only the improved grip and subdual hold are prevented. Note that two characters both grappling and trying for such a hold against each other cannot both succeed; in this case the attacker who hits the best AC gains the upper hand. The held opponent may attack with his off hand, either unarmed or with a small weapon, at -4 to hit. Anyone else may attack either combatant at +4 to hit, as neither can dodge while the hold is maintained.
In the case of multiple attackers trying to grapple a single opponent, each attacker rolls separately. Unless the opponent has multiple attacks, only one grappling attack may be fended off by a successful hit. Each attacker beyond the first that successfully grapples adds +2 to the Strength score of the strongest of the group for purposes of determining movement, and adds to the encumbrance of the load on the grappled opponent. Conversely, the opponent's encumbrance is divided amongst the grapplers. Up to four attackers can grapple an opponent of equal size, eight can grapple an opponent of twice their size, and twelve can grapple one of three times their size. When a subdual hold is established, each additional grappler can completely neutralize one attack of the held creature.
A character or creature who is a part of such a "dog pile" may be pulled away by a new combatant making a successful grappling attack against it.
A subdual hold, and the resulting damage, may not be inflicted on an opponent greater than twice the mass of the attacker. In the case of multiple attackers, if a number of them whose combined mass is greater than half that of their opponent succeed in a second grappling attack in the same round, they may establish such a hold.
Whenever the Strength score or mass of a creature are not known, the DM may assign them according to his own judgment. Large or huge creatures, like horses, giants, and dragons, should certainly have Strength scores far outstripping the human range; that and their great masses (an average horse weighs around 1,000 pounds) should serve to curtail the most absurd abuses of the grappling rules.
And there you have it. Hopefully it's less complicated and easier to apply than its length would indicate. It may very well have some holes or be prone to breaking under certain circumstances, so please feel free to weigh in if you spot a problem.