Over the years of reading RPG blogs, I've seen a lot of musings about the portrayal of certain monsters in D&D. I've seen the "baby orc" moral dilemma hashed over quite a lot. A few times, people have argued (with very little justification, in my opinion) that orcs and other monstrous humanoids represent human cultures, and therefore the use of them as Always Chaotic Evil opponents was in some way an expression of latent racism in RPGs and their players. I've seen arguments (with a lot more justification) that the monstrous humanoids, at least as typically represented in adventure modules and supplements, could easily be reskinned as barbaric humans, and thus aren't terribly interesting.
Gygaxian naturalism is a strong force for this sort of anthropomorphizing of creatures. Indeed, as much affection as I have for module B2: The Keep on the Borderlands, the goblins, orcs, kobolds, and other inhabitants of the Caves of Chaos are virtually indistinguishable from savage humans. They reproduce and rear young like humans, they eat and drink like humans, the form social structures like humans. They're just mean and ugly. One gets the feeling that if they only cleaned up a bit, learned some rudimentary etiquette, and abandoned their prejudices toward humans and demihumans (and vice versa), an orc or goblin from the Caves could stroll into the tavern, order a pint, and swap tall tales with the other patrons and not be too far out of place.
I don't believe there's anything inherently wrong in running a game with these interpretations of monsters, but it's not to my taste at this stage in my gaming life, nor do I wish to reskin them as various sorts of human antagonists. What I'd rather do, in most cases, is make monsters truly distinct from humans. An encounter with goblins or orcs should be substantively different from an encounter with brigands or bandits, and each humanoid kindred should be different from the others. Moreover, there should be reasons why these creatures are so often in conflict with humans and demihumans besides mere prejudices and cultural differences.
How do we do that?
One way is to give monsters biological needs and drives that are dangerous or abhorrent to humans. Take orcs as an example. Perhaps orcs have a physiological need to consume the flesh of sapient humanoids. Orc whelps aren't innocent babies but voracious little bastards who need human or demihuman meat to grow. Even orc adults, if they don't taste man-flesh once a month or so, begin to lose even their already limited capacity for intelligence and cooperative behavior, becoming more and more feral and savage, eventually resorting to cannibalizing their own kind to sate their hunger. Whether you consider this to be evil or "just the way they are," it's pretty clear that orcs are going to have a hard time coexisting peaceably with men, elves, and dwarves, and often even with each other.
Another way is to avoid ascribing human emotions to monsters. Humanoid creatures need not experience the same range of feelings, nor to the same degrees or for the same reasons, that humans do. For example, they might be constitutionally incapable of empathy or compassion, similar to human psychopaths. Such creatures could have an instinctual drive to preserve their own species, while being utterly indifferent or even malevolent toward others. This seems especially appropriate for non-mammalian humanoids, such as lizardmen or troglodytes. Lizardmen seem like good candidates for an apathetic-psychopathic personality, while troglodytes could be the more actively sadistic serial killer type profile.
Emotions could be even more radically different in various ways. Loss may induce laughter, affection might be dysphoric, hostility breed attraction, or kindness lead to revulsion.
A monster's alien biology may not be inherently hostile to human life, but may give it a radically different perspective. Consider "dark fey" type creatures, which are not born of two parents and reared from infancy to adulthood, but instead are spawned from elemental, magical, or psychic forces, awakening from the oblivion of their previous nonexistence fully formed. Such creatures have no concept or understanding of family bonds, no experiences of the innocence of childhood, maybe not even of adulthood or old age or stages of life generally. Not only are there no "young" monsters with which to impose the tired "baby orc" dilemma on players, these creatures would perceive no difference between children, adults, or elderly humans other than sizes and outward appearances.
In my dark fey series, I reimagined several creatures as embodying specific negative human traits, such as envy and avarice. Coupling this with the abiological origins described above, there's not only a very strong reason why these creatures behave in ways that seem irrational, or at least irrationally intense, to humans, but also a reason why creatures of the same type have the same very limited range of ethics and personalities. A goblin isn't mean and envious because it's a goblin; it took the form of a goblin in the first place because its spirit is mean and envious. Spirits of different origins and character might become hobgoblins, kobolds, pixies, or sprites instead.
Or, consider monsters who are born as humans, but through magic or curses or just the natural order of a fantasy world become monstrous in body and soul. Maybe ogres were once beautiful humans who were manipulative, spiteful, and ugly inside, and their bodies transformed to match their twisted souls. As hideous ogres, they're banished from society, lacking both people to manipulate to fulfill their emotional needs and the charisma to do so. Consequently, they become even more hateful and cruel, and probably extremely jealous of attractive humans. Beauty, to them, becomes a reminder of what they've lost, and thus a heinous sin.
Radically different physiology and psychological and emotional experiences might give monsters a blue and orange (as opposed to black and white or shades of grey) moral outlook. To dragons with a pathological (by human standards) love of treasure, theft, or even (gasp!) destruction of treasure may be a far more serious offense than murder. A gentle touch among orcs may be a grave insult, while a punch in the gut is a jovial greeting. Maybe destroying corpses is seen by some monsters as blasphemous, while reanimating them as undead is honorable. Capriciousness may be valued, while steadiness is disdained. A monster may be forbidden by its moral code to lie to a being wearing a wicker hat.
It's possible (and fun!) to combine some of these distinguishing characteristics. Take the doppelganger, a faceless, featureless, genderless being: Maybe the reason it takes the forms of humanoids is because it craves the experience of having an identity. It's not that it feels happy when it takes the form of an individual or sad and empty when it doesn't; it's more like an addiction or compulsion. The creature literally feels a euphoric high by duplicating someone and disposing of that person. While in its stolen form, it does things that seem cruel and evil to humans not out of some desire to be evil or inflict pain, but because its mind is totally alien to the human experience. It may keep its cover for a while by aping their behaviors, but it's apt to apply them inappropriately, and eventually is exposed or driven away, loses its grip on its new "identity" and starts the cycle again.
Whatever traits you apply to any particular monster, players encountering it should feel they're dealing with something inhuman. Negotiating with monsters may be an exercise in thinking way outside the box, and employing them as henchmen is an adventure in itself. It's certainly a challenge, as a human DM, to play the roles of beings who think, act, and feel in ways humans habitually don't, but it can go a long way toward making monsters more than just another faction to deal with or one more set of stats and powers to best in combat.
