The most important ingredient, I think, is for the race to represent and exemplify some concept or ideal. Embodying some particular facet of humanity gives a non-human race a sharper identity than simply packaging the whole spectrum of human diversity in different bodies with different powers. It doesn't matter how awesome your new race's appearance or how cool the powers you give it; if it doesn't connect with something deep in a player's psyche, it's not likely to gain much traction. To illustrate this point, let's consider the races that are considered an integral part of the core game, the demi-humans.
Dwarves (or dwarfs) and elves have a long and storied record in folklore. In everything from Tolkien to Snow White, dwarves are strongly associated with mining, metal, and stone, and greed for gems and precious metals. More abstractly, the dwarf race represents the stability and endurance of those materials, but also the rigidity and coarseness of stone, and craftsmanship, a desire to create things of beauty and value that long outlast their creators.
Elves are avatars of immortality and timelessness, beauty and grace. They are often portrayed as lacking or transcending the faults and frailties of humanity, but also lacking its drive and ambition, whiling away their eternal lifespans in song and communion with nature.
Halflings lack the ancient pedigree of dwarves and elves, but the archetype is as old as humanity itself. The halfling is the everyman who lives a simple honest life, aspiring to nothing greater than comfort for himself and those he cares for, of whom nothing great is expected. Sometimes he represents childlike naivete and wonder also. In heroic fantasy, the halfling is the underdog, the fish out of water, the unlikely hero who succeeds through sheer determination, whether he does so with wide-eyed enthusiasm or forever wishing that he was instead asleep in the plush chair by the hearth, well-stuffed with honey-cakes and tea.
I don't know how well I've done capturing the essences of those races, but if I say the word "dwarf" or "elf" or "halfling" it means something to you beyond "short person with beard" or "pointy ears."
Perhaps one reason the gnome, despite having a much firmer grounding in myth and legend than the halfling, is seen as a second-tier PC race at best (it doesn't even appear as a player character race/class in classic D&D) is that the archetype it's meant to portray is so nebulous. Depending on your interpretation, it steps on the toes of the dwarf, the elf, and the halfling to some degree without offering anything distinctly gnomish for a player to latch onto. Sure, they can be illusionists, but what does it mean to be a gnome?
So, here are my thoughts on what it takes to succeed as a PC race:
- Embody some concept, philosophy, or archetype, such that it informs every aspect of the race's personality and behavior. This can be a tricky, because it needs to be focused enough to be distinct and meaningful, yet not so narrow as to make the race a cliche or caricature. (The much overused "warrior race" template is a good example of this. There's a reason why the Star Trek writers gave Klingons an obsession with "honor" - to be credible as anything other than one-dimensional enemies, they needed to be more than just warriors.)
- The archetype must resonate with some inner yearning of the human psyche. Who doesn't want to create something that outlasts his mortal life as dwarves do? Who hasn't dreamed of being unconstrained by the shackles of time like an elf? Who doesn't feel divided between the desire for simple comforts and the drive to achieve great things, as a halfling? If a player can't grok what the race stands for on a deep level, it's unlikely to engender anything more than the much maligned "human with funny ears" type of roleplaying.
- Players must be aware of what the race is all about. The easiest way to do this is to adapt a creature from popular myth or folklore which they've already heard of. Even if the race's character and personality have to be further focused and refined for the game, at least you have a starting point for understanding. Whether a new race has mythological roots or is purely the spawn of your own warped imagination, it needs to be established in the campaign. Players should have the chance to interact with members of the race and get a feel for it before it's thrust in front of them as a character option. Some bloggers have mentioned the practice of requiring the player character party to establish contact and relations with new races to "unlock" them as playable races.
I should mention that when I refer to the success of a character race, I don't necessarily mean that it catches on universally, with the RPG hobby as a whole. Probably far more often, something will catch on in someone's personal game, because of the way the GM runs it and the way he or she breathes life into a race in the context of the campaign. The game design industry as a whole necessarily operates in broad strokes, where my campaign can be tailored to resonate with my individual players.
Sometimes it's a player who falls into a role with such ease and enthusiasm that it makes the race come alive for others. I remember a cousin of mine who did exactly that with the gremlin from the Top Ballista supplement, lifting the class from a mere novelty to full legitimacy in the campaign. My wife did the same for the dryad (modified from the official Tall Tales of the Wee Folk version) in our current game.