Grappling is one of those things that's always a nightmare in RPG combat. It's almost invariably a fiddly mess that doesn't mesh or scale well with standard combat rules. To an extent, grappling is a part of normal combat, as a lot of grabbing, pushing, tripping, and so on takes place even in a sword fight. Sometimes, though, a combatant will expressly attempt to grasp and hold on to an opponent, and some grappling-specific mechanics come in handy.
What I'm going for here is something reasonably simple, without too many conditions and modifiers, which complements rather than contradicts the basic combat rules and minimizes absurd outcomes such as a high-level fighter wrestling a war horse or a giant to the ground.
For the basics of my fantasy heartbreaker combat system, see here and here.
Without further ado, here's what I've come up with for grappling.
The first phase of a grappling attack is resolved with standard combat rolls. The combatant initiating the grapple must either have at least one hand free or use a natural attack capable of grasping, such as a crocodile's jaws. A defender may be either unarmed or armed. (Included in standard G&G combat, if I haven't mentioned it already, is a rule that attacking unarmed against an armed opponent incurs a -2 penalty to AC; thus grappling an opponent with deadly weaponry is more hazardous than grappling an unarmed one.) A grappling combat roll is always modified by Agility; if the defender chooses to attack normally and try to avoid being grappled, it uses whatever modifier it normally would for its own mode of attack.
Both combat rolls have their normal effects, inflicting damage if they exceed the opponent's AC. If the attacker's roll is higher than the defender's (including all modifiers for both) the attacker has achieved a solid grip on the defender. If the defender's roll is higher, it eludes the attacker's grasp and remains free. Note that neither necessarily needs to inflict damage; the higher roll wins, even if it's otherwise dismal. Ties always go to the defender; i.e. no change in status.
Once a grip is established, in subsequent rounds the attacker may simply hold on, or may try to overpower the opponent. Regardless of which action is chosen, each combatant may continue to attack the other, making normal combat rolls each round and inflicting damage as appropriate.
Holding on to an opponent means that the combatant is maintaining its grip on the opponent and avoiding the opponent's attacks. One common tactic is to hold the opponent from behind, or in the case of a larger opponent, to climb on and cling to its back. While holding, the grappling character's combat rolls against the opponent are made at +2, while the opponent's combat rolls against the grappler suffer -2. Additionally, attacks by other creatures or characters against either grappler or grappled are made at +2 to the combat roll, as their ability to dodge and parry is limited.
The opponent chooses the direction of movement, if any, possibly at a reduced movement rate if the grappler is heavy enough to encumber it.
Each round, new combat rolls are made, and if the opponent wins a combat roll against the grappler, it may either throw him off or establish a grapple of its own, which leads to an overpowering contest.
Only small, medium, or natural weapons may be used while in this stage.
At the GM's option, holding onto certain opponents may render some attack forms impossible and others more likely to succeed. For instance, a grappled medusa may be unable to turn and use her gaze attack on the grappler, but the grappler would be extremely vulnerable to the bites of the writhing snakes on her head.
Overpowering occurs when one combatant tries to restrain, subdue, or move the other. When two combatants are both grappling each other, it automatically becomes a contest of overpowering. Overpowering requires a contested roll of 1d6, with each combatant adding its Combat Rating and Might adjustment. For monsters without a Might score, use the creature's unadjusted Hit Dice. Characters not using both hands to grapple (holding a weapon or other item in one hand) are penalized by -1.
The one with the higher total is in control, and may do one of the following:
Automatically inflict 1 point of damage per point of difference in the rolls
Reduce damage done by the opponent by 1 point per point of difference in the rolls
Drag/carry the opponent 5 feet per point of difference.
Break free, if its combat roll also exceeds that of the opponent.
Ties result in a stalemate, with no movement possible.
A new overpowering roll is made each round that the grapple continues, until one or the other combatant escapes or is subdued. A combatant reduced to 0 hp in a grappling contest is considered subdued and unable to resist further. A subdued opponent may be automatically slain if desired.
Only small or natural weapons may be used while involved in an overpowering contest.
Gort the fighter wishes to grapple and subdue an outlaw without killing him in order to bring the miscreant to justice. He has a Combat Rating of 4 and a Might adjustment of +2, and an Armor Class of 14. The outlaw has a CR of 3 and Might +1, and AC 14 also. Gort is unarmed, and so suffers a -2 penalty to his AC against the outlaw's dagger.
Gort scores a total of 17 on his first try, beating the outlaw's AC by 3 points, doing the maximum unarmed damage of 2 points. The outlaw, however, gets an 18, tagging Gort for 4 points of damage and preventing him from getting a good grip.
Next round, however, Gort gets a 12. This isn't enough to deal any damage, but it beats the outlaw's roll of 8, and Gort grabs hold of the cad. He immediately tries to overpower his foe, aiming to subdue him. Gort rolls 1d6 and adds 5 for his combat skill and strength, achieving a total of 10! The outlaw rolls and adds his total bonus of +4, with a -1 penalty because he's not letting go of his dagger; he only scores a 6. Gort could inflict an automatic 4 points of damage, or he could reduce the damage done by the outlaw by 4 points, or he could drag the bastard up to 20 feet. Since the outlaw failed to do any damage with his combat roll, the second option is moot. Gort chooses to wear him down with a punishing choke hold, doing 4 points of damage.
In the third round, the outlaw's combat roll is a resounding 18, but Gort outdoes him with a 21. Gort does 2 points of damage with his (unarmed) combat roll, but will take 4 points from the outlaw's dagger. Since Gort has maintained his grasp on his opponent, another overpowering check is made, which he again wins, 9 to 7. Gort could choose to reduce the damage the outlaw inflicts on him by 2 points, to increase his own damage against his foe by 2 points, or to move him 10 feet. He chooses to keep the pressure on his choke hold rather than fend off the outlaw's weapon hand, so he takes the full 4 points of damage but inflicts 4 points of his own.
So we leave their struggle, and move on to...
Sera the thief gets caught picking the pocket of a burly fighter. With no easy escape, she decides her best bet is to grab and hold on so he can't skewer her with his sword. Sera's CR is 1, her Agility bonus is +1, and her AC is 13. The fighter has an AC of 14, +1 Might, and CR of 3. Sera has one free hand and her club in the other, so she suffers no penalty to AC.
Her adjusted combat roll is a 10 - not enough to hurt her opponent, but the fighter rolls only a 7, and she seizes him by his sword belt and jumps on his back.
Next round she rolls a 15 and the fighter gets a 16, but since she has him grappled, she gets +2 to her roll, and his roll is penalized by -2. Their adjusted totals are 17 and 14. Sera wins, holding her relatively safe position on the fighter's back. He does 1 point of damage to her, and she does 3 points to him. If she hadn't successfully grappled him, she would have taken 3 points and done only 1 to him.
If the fighter should beat Sera's combat roll next round, he could choose either to throw her off, or to attempt to overpower her. As the thief hangs on for dear life, we move on to...
Dorn the Overly Inquisitive prods a crocodile with a stick and gets the result any person with basic common sense would expect. The croc attacks and scores a total roll of 13, not good enough to pierce Dorn's plate armor (AC 16) but enough to beat his pitiful flailing roll of 7.
The croc naturally tries to overpower him. It rolls a 4 on 1d6, plus its Combat Rating of 3, plus its unadjusted Hit Dice (also 3) for a total of 10. Dorn rolls a 5, plus his CR of 2 and Might adjustment of +1, coming up short with an 8. The croc chooses to drag him 10', into the water.
The next round the croc wins again with a combat roll of 8 over Dorn's 5, neither scoring any damage, and wins the overpowering roll 11 to 6. Now the croc goes into its "death roll" and deals 5 points of damage to Dorn, who is in serious trouble...
Multiple attackers: If more than one combatant tries to grapple the same target, any that is not counterattacked succeeds as long as its combat roll is not a natural 1. For example, four goblins try to grapple a fighter wielding a mace, with which he can sweep up to three opponents. The first three goblins succeed only if they beat the fighter's combat rolls against them. The fourth succeeds unless it rolls a 1; the fighter doesn't have enough actions to oppose it.
When multiple attackers attempt to overpower a grappled opponent, only one 1d6 roll is made for the entire side. Add the most powerful member's CR and Might or HD adjustments and +1 for each additional member.
Multiple defenders: An attacker with more than one available grasping appendage may attempt to grapple more than one target. This is treated as a sweep, with combat rolls modified accordingly. Overpowering rolls are made, dividing the attacker's bonus from CR and Might or HD evenly among the opponents held, rounding down. For example, a fighter with CR 4 and Might adjustment +2 who grapples two goblins rolls 1d6+3 for each of his overpowering rolls. The goblins roll without penalty.
Holding onto a grappled opponent while attacking or fending off others is likewise considered a sweep and combat rolls should be adjusted accordingly.
Mutual grappling: If two combatants both wish to grapple, both are automatically successful. Combat rolls determine only whether any damage is scored. A mutual grapple is always a contest of overpowering; proceed to the overpowering roll.
Disarm: If a combatant wins two overpowering rolls in a row and chooses the "reduce damage" option (whether or not the opponent would actually score any damage), the opponent is disarmed if desired (and applicable.)
Tackle: Charging or leaping onto an opponent adds +2 to the initial grappling attempt. For example, a panther leaping from a tree branch to grapple a target on the ground would get this bonus.
Adapting to standard D&D
Use Dexterity adjustment instead of Agility, and Strength instead of Might. Use (20-THAC0) instead of Combat Rating. When Combat Rating is increased or penalized due to grappling positions, increase or reduce any damage caused by rolling twice and taking the higher or lower roll, respectively.
If I've made any mathematical or logic errors, or if you find a serious (or even not-so-serious) rules expoit that could result in an unbalanced advantage or an absurd result, please point them out!