If the DM feels that a player is not keeping to a character's chosen alignment, the DM may suggest a change of alignment or give the character a punishment or penalty.Other editions suggest similar, or even more strict schemes of enforcement. The obvious question is how do you fairly enforce something that can't even be unambiguously defined?
The Classic D&D single axis alignment system of Law, Neutrality, and Chaos tries to amalgamate several distinct concepts under each alignment. Law is described as a commitment to order, respect for rules, good will toward living creatures, honesty, and a sort of collectivist "community spirit" of elevating the group over the individual. Chaos is written as a combination of belief in randomness and luck, disregard of rules, malice or malevolent indifference toward others, dishonesty, and individualism. Neutrality lies at some unspecified midpoint between the two.
I'll use myself as an example to illustrate the myriad contradictions in these assumptions. I am, politically, an anarchist (specifically, the anarcho-capitalist philosophy of Murray N. Rothbard.) Your first impression might well be that my alignment is Chaotic. Hold on a second, though. It's true, I reject the authoritarian order of the state, but I believe very strongly in a natural, spontaneous order that arises among individuals in a free society. I believe in a moral code deriving from man's nature as a rational being. Am I just another species of Lawful, then, and if so, how do you distinguish me from the authoritarian? I'm also an individualist - the group is nothing but an abstraction that has no existence, identity, or needs of its own separate from those of the individuals who comprise it, and to elevate it above them is an absurdity. Chaotic, then? I abhor violence against non-aggressive individuals, and advocate the freedom of every individual to pursue his own happiness without violent interference of others. That smacks of good to me - Lawful? Or am I Neutral, because I have some characteristics from each end of the spectrum? If so, isn't nearly the entirety of humanity Neutral?
Now let's suppose a two-axis model, with a good-evil component. Having separated Good and Evil from Law and Chaos, we can assume that the latter refers only to a being's relationship with law and order. But what kind of order, that imposed artificially by intelligent beings, or the underlying order of nature? We're left with the same dilemma from above, that of the spontaneous order anarchist vs. the advocate of the authority of a strong government. Are those really to be considered the same? Also, does a creature's position on the Law axis describe his beliefs or his behavior? A character who acts spontaneously may nonetheless believe that there is an underlying order to reality, while one living a life of stultifying routine might think that in the grand scheme there's no purpose or sense to anything.
As to Good and Evil, are we relying on the practical results of a person's ethos, or his or her intentions? Many people would say that my free market anarchism would result in widespread human suffering, and label it an evil ideology, but I believe it to be the surest path to human flourishing. My intentions are unquestionably good. Likewise, I can accept that the authoritarian statist may have nothing but the best of intentions, but it's my conviction that to put his methods into practice is a drag on human liberty, creativity, productivity, happiness, and ultimately opposed to the welfare of life. Are we both Evil, both Good, or is one of us Good and the other Evil? (And if the latter, how do you determine which is which?)
The alignment system, I think, is intended to be an aide to role playing, not a course in philosophy and politics, but it nonetheless must be definable, comprehensible, unambiguous, and non-contradictory if it is to be genuinely useful to players and DMs as a guide to playing the roles of characters and monsters. With all their vagaries and contradictions, I consider the systems as written in both Classic D&D and AD&D to be hopelessly broken.
For my own campaign, I'm going to drop the Law-Chaos axis. Having no unambiguous definitions with which I feel comfortable, I consider it unworkable. Instead, I'll replace it with a single axis of Good and Evil. A creature's position on the axis is determined by its intentions and motivations.
A Good creature has an active concern for the well-being of others (sometimes with the provision that those others mean no harm.) It is not barred from pursuing its own individual self-interest, but will actively avoid inflicting harm in the process. Killing is only permissible to prevent evil from being done, though it may often be assumed that irredeemably evil creatures present a constant threat and may be dispatched on principle. A Good creature will prefer win-win outcomes in its interactions with others, and when that is not possible may purposely take the short end of the stick itself.
A Neutral creature is driven by self-interest, and is indifferent toward the well-being of others, but not actively malevolent. It will not actively pursue their harm or suffering, but will have little interest in alleviating it either, unless it perceives some clear benefit to itself. Such a creature prefers to win in its interactions with others, and is indifferent to whether the outcome for the other is win, lose, or draw.
An Evil creature is driven by a desire to harm others. It may well act out of a desire for personal wealth or gain, but it enjoys the discomfiture and suffering of others as much as or more than its own gain. When interacting with others, the most important thing is that they lose, though of course the Evil creature generally prefers for that loss to be accompanied by its own gain.
This system should serve for adjudicating effects that rely on a determination of Good or Evil, such as a Detect Evil spell or an aligned magic sword. Aside from those considerations, sanctions for playing alignments are strictly according to the reactions of other creatures and characters in-game, i.e. good creatures will feel affinity for other good creatures, be repulsed and horrified by evil acts and refuse to associate with the perpetrators, and see neutrals as incomprehensibly cold or callous.
Whether a creature in the game prefers to pursue its goals within the existing local framework of law, irrespective of it, or in defiance of it, or whether it believes in an orderly or a random universe is left to the discretion of the person controlling it, and need not be codified in the alignment structure.