In the comments of a fairly recent post, I was asked to describe how I judge the challenge level of a combat encounter. Short answer is that I don't have any hard-and-fast formula, and more or less eyeball it. But of course that only evades the real question, which is what do I look at when eyeballing in order to make a judgment?
Firstly, let me say that encounter balance, as fetishized in some recent editions, is abhorrent to me. Crafting every encounter to be "level appropriate" takes the fun out of the game, in my estimation. I don't like railroady scenarios with scripted highs and lows that require easy victories over mooks and dramatic nail-biting climaxes. And no encounter should be a mandatory combat - if the players can talk, bribe, sneak, or trick their way past something, the DM absolutely should not nullify that to force them into Exciting Combat!™
That said, I also don't want to create dungeons full of killer encounters or boring cake-walks. An adventure should include plenty of opportunities for winnable combats, as well as encounters that demand more discretion. Balance, to me, takes place at the level of the entire dungeon or scenario, not zoomed in to the individual encounter.
The amount of damage a group of creatures can do in a round is, I think, the most important consideration in judging their threat level. Two primary factors contribute to this: the damage potential of each attack, and the total number of attacks per round of the group.
I would weight the total number of attacks more heavily. A fight against four goblins is likely to be more deadly than a fight against a single ogre because the goblins get four chances to hit every round, while the ogre only gets one. Against a party of 1st level characters, the goblins could potentially kill four in a round, while the ogre can't finish more than a single one at a time, no matter how much damage it rolls.
Damage per attack I would judged based on how many attacks it would take to kill a typical PC. The more rounds the party can stand against it, the less dangerous it is to them. Can it kill a PC with a single attack? An attack that just hurts a character is a prompt to make a choice: withdraw or fight on. This is an important consideration because character death, if it happens, is more likely to be seen as a result of player choice. A one-hit kill is just a one-hit kill. If you want to give a 1st-level party a fighting chance to engage in combat without too much anxiety, make sure to have some monsters that do 1d3 or 1d4 damage around.
Hit Dice are the usual unit of Monster Toughness in old school games. Dungeon levels are supposed roughly to correspond to the average HD of the monsters found in them, and BECMI and Rules Cyclopedia D&D even have an encounter balancing formula which is based on Hit Dice. However, Hit Dice are a distant third in my estimation of monster threat level. What they are is a fairly good indication of how long a monster will last in combat, and thus how long it will get to keep making its attack and damage rolls against the PCs.
Hit Dice also affect the chances of successfully attacking, but only by about 5% per HD, which is a lot less than extra attacks do. A 1 HD creature needs to roll a 14 or better to hit AC 5, which translates to a 35% chance. A second attack increases the odds of getting at least one hit to 57.75%, better than having 4 extra Hit Dice!
Quite often, big HD correlate with big damage, but not always, so it's still a good idea to gauge them separately.
All else being equal, a single high HD monster with multiple attacks is usually more dangerous than a group of lesser creatures whose HD and attacks approximate it. Consider a pack of five 1 HD orcs with 1d8-damage weapons and a 5 HD owlbear with three 1d8 attacks. Each orc will take 5 or so points of damage to kill, and each one killed reduces their number of attacks by one. The owlbear gets its three attacks per round until it's dead.
Armor Class rates a little bit lower than Hit Dice in my Monster Threat Assessment. It too affects how long the monster remains alive and fighting, but unlike a big pile of hit points, which can only be whittled away, AC is bypassed by a good attack roll, and a d20 is a lot swingier than most dice you might roll for damage.
Special attacks are trickier to assess. For low-level parties, their effect is actually minimal. A failed save vs. poison might kill you, but so will a single good solid physical blow, and dead is dead. The odds of missing a save and getting hit in combat are only a little bit different. A save-or-suck power is similar to a low damage attack: it's a cue to rethink one's position, not instant Game Over.
Save-or-die and save-or-incapacitated powers become a big deal at mid
levels, not because they become more absolutely dangerous, but because
they're now a lot more dangerous relative to attacks that do hp damage.
An orc with a sword and a medusa with a petrifying gaze can both take
down a 1st level character in one round. Only the medusa is likely to
be able to do the same to a 4th or 8th level character. The character's
hp increases almost always greatly outstrip saving throw improvements,
and while healing spells and potions are by now fairly common, spells
that will reverse the effects of poison and petrification are still
either out of reach or in short supply.
By high levels, saving throws have improved enough to make the success rate of save-or-die effects acceptably low, and the party probably has access to remedies, which makes them an inconvenience rather than a deadly peril.
Of course, the number of attacks principle also applies to special attacks. If a monster can direct its special attack form against multiple targets at once, or there are several of the monsters in the encounter, it magnifies the threat level in much the same way.
Lots of attacks > big damage per attack > high HD > high AC
Combinations of those factors are possible, naturally. So, lots of attacks + big damage per attack generally trumps big damage and high HD; big damage and high HD beats high HD and high AC, etc. What about lots of attacks and high AC vs. big damage and high HD? How do special powers change things? Use your own judgment. It's an art; not a science.
Putting It In Play
In practice, I try to balance a dungeon as a whole, with a lot of encounters that I think the party could handle without too much trouble, some that would be quite risky, and a few that would almost certainly be disastrous if handled with straight-up combat.
Remember that a good fight is one that costs the party some resources to win, not necessarily one in which the sides are evenly matched. To a wise adventuring party, evenly matched battles
are to be scrupulously avoided except in the most dire need or the richest
possibility of reward. Evenly matched means that both sides are likely
to suffer losses, and either side could end up defeating the other. That might sound exciting, and it is sometimes, but an adventure is more than one fight. It's a war of attrition.
In a good dungeon, the PCs might be able to win any given fight, but they probably can't win them all, one after another, in a gauntlet match. Even winnable fights must often be avoided, in order to conserve resources for fights that are not merely winnable, but important and/or profitable. "Can we win this fight?" is an important question, but so is "What about the next one?"
1st level parties are the toughest to plan for because, with their low hit point totals, outcomes depend heavily on the very swingy d20 attack roll. 1st-level characters and 1 HD monsters can go from full strength to dead with a single attack. A seemingly even battle can become lopsided very quickly if one side has a run of lucky or unlucky rolls. Additionally, if the party includes a spell-caster, the outcome may hinge heavily on which spell he has ready and whether he still has it or has already used it.
If you want to design a dungeon for low-level characters in which direct combat will play a big part, encounters should be predominantly with numbers of monsters smaller than the PC party, with 1 HD or less, low damage potential, and poor ACs. A few encounters can be with even smaller numbers of creatures with 2-4 HD and better damage or AC. Anything bigger, tougher, or more numerous is a serious risk -- but don't let that discourage you from including a couple such encounters. It ain't all about chopping things up with swords, and part of the fun is knowing the difference.
As characters gain in levels and hit points, they generally survive longer, and overall, combat becomes less swingy. The more rounds they can go without dying, the more rolls are made, and the more rolls made, the more the cumulative result is pulled toward the average. Characters with a few levels under their belts can go toe-to-toe with monsters well above their weight class more successfully than low-level ones can because hit points increase a lot faster than damage potential. That doesn't necessarily mean they'll win, but an ill-chosen fight is less likely to wipe them out before they can even realize their mistake and run away.
Still, I think it's good to follow the rule of thumb that the majority
of encounters should be ones that the party could handily best in combat, but
will take a bite out of their resources. The more fights they pick, the more their resources and hit points get ground down, the closer they get to being that 1st-level party all over again, when life and death can hinge on a single misguided choice or bad roll of the dice...