When I was new to D&D, and perusing the bestiary section of the rules, the very idea of undead monsters was chilling, creepy, fascinating. These were things that existed outside the laws that govern living creatures. Except, of course, in the rules they really weren't. They had Armor Classes and Hit Dice and Morale scores just like every other monster, and over time the veneer of wonder wore away and they started to feel like just another hit point total to whittle down in armed combat.
If you're lucky enough to have relatively new and inexperienced players, or at least some who haven't read and internalized the monster descriptions and stats from the rules, this may not even be an issue. That actually is the case for me right now. I could drop a wight or a wraith into a dungeon room, and when the PCs blundered into that room they'd be freaked out, just as my original group was at the first appearance of the wight in the crypt in the Caves of Chaos. But if I did that, they'd eventually become jaded too, and since I'd prefer to forestall that and preserve as much of the mysterious creepy-weird-scary vibe of the undead as possible, I'd prefer to avoid portraying undead as just so much sword-fodder. Here are a few things I can think of to make encounters with undead unique and compelling.
Description: The appearance of undead monsters is as diverse as that of the humans and demi-humans from which they arise. Giving them unique descriptions where appropriate helps make them more than just bundles of stats. What do they wear? Are they male or female or too far gone to even tell? Are they skinny, emaciated, fat, muscular and hulking? Do they have hair, facial hair, or other features that would have distinguished them in life? How do they move? Shambling like Hollywood zombies, on all fours like feral things, crouching and leaping, gliding...? Don't forget the other senses besides sight. Do they moan, gurgle, snarl, scream, hiss, mutter incoherently? Undead with a rotting physical body might well smell like a charnel pit, but even incorporeal wraiths, specters, and ghosts might have a particular scent that betrays their presence, whether that be wood smoke, a salty sea breeze, whiskey, roses, or something else associated with the life or death of the creature's former living self.
Motives: Undead aren't necessarily motivated by the same things that mortal men and beasts are, or perhaps more accurately, they often are but in twisted and distorted ways. Often, undead have one all-consuming motivation. Unlike living creatures, they have no biological needs. They are never distracted by the demands of mere survival, either as individuals or species. They do not need food, drink, or shelter, and they have no biological drive to reproduce. That means that, whatever it is that does drive them, they may pursue it with a single-mindedness (or single-mindlessness) far beyond the most powerful mortal obsession. Ghouls are consumed by a desire to feed on flesh. Malice toward the living is a common motive, as is guarding some place, object, or person. Sure, those are classics, but why limit yourself? Why couldn't an undead creature be driven by a primal need to create more of its own kind - to reproduce, as it were? How about other human motivations, taken to extremes, like self-preservation (maybe the creature isn't even aware that it's dead!), companionship, knowledge, greed, envy, vanity, lust, longing for something lost, compassion, religious fervor, bigotry...Play it straight up, or subvert it in ironic and disturbing ways.
Powers: The stock powers of undead by the book aren't all that scary if only described in game terms. You're paralyzed. You're diseased. You lose a level. At the very least, they should get colorful, unsettling descriptions in-game. What does being energy drained feel like? Does it chill you to the bone or create a temporary link between you and the undead, flooding your mind with the monster's ghastly tormented thoughts?
Change up powers, if you think of something more suitable to a particular monster, that really fits its history and motives. You can also add side effects to powers, things that have trivial or no mechanical effects, but that unnerve the players. Residual dreams or visions, phobias, minor disfigurements like white hair or ashen skin, inexplicable cravings for raw meat, chills, and other effects can be either temporary or permanent while imposing no mechanical hindrances on a character.
Tactics: How many scary stories have the heroes simply slugging it out toe-to-toe with a supernatural adversary immediately upon encountering it? Not many. In most cases, spooks and spirits are elusive and devious opponents. Often the undead don't engage in direct combat at all, but entice or frighten their victims into stumbling into other hazards. Even when they do directly attack characters, it shouldn't feel as if the PCs are just fighting another warrior who happens to be rotting or translucent. There should be something uncanny about the way the creature fights, and it should use its powers - including things that aren't actually listed as powers, such as being incorporeal or impervious to pain - to maximum effect, even if that effect is only descriptive rather than mechanical. A zombie fights relentlessly, and couldn't care less about being menaced, or even stabbed, with a sword. A wraith or spectre is the ultimate hit-and-run attacker, being able to pass through solid matter with absolute silence to surprise its foes and then quickly retreating through walls or into solid ground where the PCs can't follow, only to strike again at a time of its choosing. A pack of ghouls might share a telepathic link that enables them to utilize tactics seemingly beyond their simple feral intellects.
- A wraith child, with large tearful eyes, who is horribly lonely and desperately wants companionship. She does not attack at first, but plays on the party's sympathy, desperately trying to hold hands or embrace one of them. Her touch inflicts a chilling energy drain. When the victim recoils from her, she cries pitifully and "attacks," seeking to reestablish physical contact.
- A spectre who was once a reclusive scholar, and now haunts the ruins of his library. He jealously guards his collected knowledge, and still thirsts for more. His touch inflicts the usual double energy drain, by actually siphoning off the victim's knowledge; after a battle, he attempts to transcribe this newly acquired knowledge into books, though in his undead state it produces only bizarre scribblings intelligible only to him. If presented with a book or other source of written information, and not currently threatened, he will at once immerse himself in it for 2d4 turns, to the point of being oblivious to all else but a direct attack against him.
- A wight who was once a painter is now driven to gather models so that he may continue his artistic pursuits. He targets particularly attractive women; those he successfully drains become his companions and subjects. When encountered, they may be nude or dressed in some tattered finery. He still "paints," smearing blood, mud, and ashes on stone walls or scraps of canvas; the results are grotesque and frightening parodies of life and beauty.
- A revenant, formerly a soldier and battlefield medic driven to madness and suicide by the suffering he witnessed. He ignores healthy folk, but can sense pain within 100 yards, whether from injury or illness, and is driven to end the misery of those poor suffering individuals - even those who are certain to recover if left alone. He attacks with a poisonous touch that also causes numbness, paradoxically speaking words of soothing comfort as he does so.
- A wraith, a coward in life, still exists in terror of pain and death. When encountered, he warns the characters to stay back and leave him alone, becoming more hysterical the longer they remain. If anyone advances toward him, no matter how non-threateningly, he attacks with the desperate fury of a cornered animal. Unaware of his true condition, he is still desperately afraid of weapons, even those that can't actually harm his insubstantial form. If struck by a weapon that can't hurt him, he is only 25% likely to notice (he can't feel pain), but if he does, he screams, "I'm hit! I'm hit!" and flees immediately. A hit from a weapon that can harm him automatically causes him to flee.
- A ghost, wraith, or specter driven by a desire for vengeance against its murderer. Those who survive its attack are thereafter afflicted with nightmares of the creature's last moments of life. A dispel evil spell will remove the effect; otherwise only revenge against the murderer will end the nightmares. The dreams contain clues to the killer's identity, perhaps as obvious as a clear view of his face, or perhaps more subtle. The spirit may desire the killer's death, or may be appeased by exposing and disgracing the murderer (especially if the killer is himself deceased.)
- A pack of ghouls, the remnants of a notorious band of highwaymen. Despite their ghoulish need to feed on human flesh, they still instinctively prefer to attack wealthy-looking individuals or parties, ignoring the destitute and impoverished. Curiously, they have no actual interest in the treasure of their victims, only in their meat and the marrow in their bones. Travelers are warned to let these abandoned troves lie, for the ghouls may still be nearby, and anyone picking up the loot may become their next target. In an area where the rich gain their wealth by oppressing the lower classes, these ghouls might even gain a reputation as ghastly champions of the common folk - undead Robin Hoods.
Sometimes using undead that aren't, or weren't, human can evoke horror and revulsion when players might be accustomed to the human variety. Consider a horde of zombie halflings, or dwarf-wraiths who perished in a mining accident, still guarding the vein of ore. How shocked might the players be to come upon an ogre, its back toward them, hunched over a meal, only to have it turn and reveal a face like one of the walkers from The Walking Dead, rotten teeth dripping with gore? Would the gaze of a ghost-medusa still petrify, or would it take on new powers? If ghouls hunt in packs, why not a pack of ghoul-wolves? There are even things like undead dragons to be found in the pages of official rules and supplements.
The rules specifically mention making zombies and skeletons of dead things other than humans. (See animate dead spell description.) Other standard undead can be modified as well, perhaps adding a Hit Die or two to corporeal forms to represent the fact that their nerves and vital organs are no longer functional or vulnerable. Incorporeal ones could either use the standard stats for the undead form (bulk makes little difference when you no longer have a body!) or the Hit Dice of its original form (maybe that represented its strength of spirit as well as physical endurance, and so still applies in ethereal undeath.) Whether you allow bugbear wights or grizzly bear spectres might depend on the underpinnings and assumptions of your campaign setting, e.g. whether those creatures have "souls" that can live on after death, or whether their corpses can be animated.
The goal here is not to simply make bigger undead with more HD and higher damage potential, but to horrify the players with things outside their experience and expectations.