In fact, the B/X reaction roll system doesn't produce absurd results so much as it lacks guidance on how to interpret the results. I have just one minor house rule that clarifies things a bit, and from there it's all about interpretation.
The house rule: Whenever a reaction roll returns a result of "attack," immediately roll a morale check. If the check is failed, the creatures encountered do not physically assault the party, though they are still hostile and wish to do the characters harm. This prevents, say, a pack of cowardly kobolds from being as overtly belligerent as a mighty red dragon. The kobolds, with a morale score of 6, will openly attack less than 42% of the time the reaction dice alone say they will. The red dragon, with a morale of 10, is almost 92% likely to attack once the reaction dice return an "attack" result. (Probabilities from anydice.com.)
Of course this morale check may be adjusted if the odds heavily favor (or appear to heavily favor) one side or the other. A +1 or -1 adjustment should be sufficient in all but the most extreme cases.
Interpretation: So, what happens when the reaction dice say "Attack!" but the morale check says the monsters hold back? The answer is that they will still do their best to cause grief, or at least inconvenience, to the PCs, even though they don't like their chances in open battle. Some possible non-combat "attacks" include:
- Give the party faulty information with the intent of leading them into danger. For instance, the kobolds might plead for their lives with information about a valuable treasure, while actually giving the party directions to the lair of an owlbear.
- Flee, but lead the party into danger, such as a trapped corridor, where the monsters know how to avoid the hazard.
- Retreat to gather reinforcements. Just because the monsters don't feel confident enough to attack at the initial encounter doesn't mean they won't with a few dozen buddies at their side.
- Bribe the party with worthless, cursed, or dangerous items.
- Steal from the party, either by stealth or openly grabbing whatever they can and hightailing it.
- Destroy important resources, such as rations and light sources.
- Set traps or ambushes in the party's path.
- Bluff to scare the party away.
- Challenge the party to a contest of skill or chance, such as a dice or riddle game, for high but non-lethal stakes.
Another scenario that always troubled me, and for which some suggestions would have greatly clarified things, was the evil or Chaotic monster rolling a "friendly" reaction to good or Lawful PCs, or the good or Lawful monster rolling an "attack" reaction against good or Lawful PCs. I don't know if anyone else had difficulty with this, but it was quite incongruous in my mind that orcs or kobolds should want to be best buds with a gang of heroes.
The obvious answer to me now, in the case of opposite alignment "friendlies" is that this is more often than not merely an alliance of convenience, not a genuine friendship. The monsters will act friendly, because it suits their purposes at the moment. They may offer aid and information. The difference between this and the "non-attack attack" above is that the creatures are making the offer sincerely - they're really giving the PCs something they believe the PCs will actually value. The kobolds, for instance, might helpfully point out that the orcs in the cave across the ravine are much wealthier than they are, and that the chieftain's chamber is down a corridor to the left from the entrance. Friendliness might also indicate simply a desire not to risk their necks in a fight - the kobolds might offer food or directions in order to buy some good will and save their own necks. In either case, kobolds being Chaotic, the PCs can't necessarily count on such friendship the next time they meet...
When the dice say that good creatures attack good PCs, it's a case of paranoia, over-zealousness, or mistaken identity. You can't always tell who's evil or intends harm just by looking, after all, and in the dangerous realm of the dungeon, even good creatures may take a shoot first, ask questions later attitude. Of course, being good, they'll likely be open to parley should the PCs proclaim their innocence or lack of malice toward them.