You're probably getting sick of these thinking-out-loud posts in quest of a good, elegant, non-Vancian magic system. Actually, so am I, but I have one more idea that came from digesting a few comments and a few posts in other blogs linked in said comments. Last one, I promise. Fingers crossed that this one's a keeper.
What I've come up with is an attempt at a more or less seamless hybrid spell point/hit point casting system. Spell casters will have a pool of magical power that they can tap without ill effects, and this increases with experience. When they've used that up, they can continue casting by fueling their spells with their hit points.
This requires a new stat, similar to hit points. For lack of a better term, I'll call it magic points, or mp. For each level of experience up to 9th, a magic-user gains 1d4 mp, plus his Intelligence bonus. Every level after 9th, 2 more mp are gained, and Int bonus no longer applies. Clerics and druids get 1d4 mp per level beginning at 2nd, adjusted by Wisdom, until 9th level, and 1 mp per level thereafter. Elves gain 1d3 points per level, adjusted by Intelligence, and a final 2 points at 10th level.
Spells cost mp equal to double the spell level. A first level spell costs 2 mp, a second level spell costs 4, and so on.
A magic-user may continue to cast spells after his mp are depleted by using his hit points, but this is an arduous and hazardous procedure. Channeling one's own life force to form a spell requires a successful saving throw vs. spells with a penalty equal to the level of the spell attempted. If successful, the hit point cost is paid and the spell is cast. If unsuccessful, the spell fails, but the attempt still costs half the normal hp drain. In either case, a save vs. death ray must be made or the caster is dazed and cannot attack or attempt another spell for the next round.
Occasionally, a spell may be cast with insufficient mp - for example, casting a level 2 spell (cost 4 mp) with only 2 mp remaining. In that case, the balance is taken from hit points. If more than half the cost is paid in hit points, the saving throw vs. spells must be made to successfully cast. If half or more of the cost is paid with mp, only the saving throw vs. death ray to avoid being dazed by the effort is required.
Magic points can be restored through rest. For each eight hours of rest, a magic-user regains 1 mp per level of experience. Magic may also be restored by certain uncommon alchemical preparations (see below.)
Using this method, low level casters, especially highly intelligent ones, have a little more power. Going beyond the limits of magic points by dipping into hit points after mp are exhausted is pretty difficult at low levels, though, so power inflation isn't too great.
High level casters have less total mojo than they do in the standard rules. They can't go flinging 8th and 9th level spells all day. A 36th level wizard with Intelligence of 16 will have an average of about 95 mp. That's enough for five 9th level spells, and not much else. He could conceivably cast 47 1st level spells, in the rather unlikely event that he chose to do so. Casting from the "reserve account" of hit points is easier than for lower level mages, somewhat offsetting his reduced capacity relative to his standard Vancian counterpart.
Some additional rules for balance:
Damage-dealing spells that normally scale by level, such as magic missile and fire ball, don't automatically scale up.
Instead, they cost an extra 1 mp per additional die of damage, at the
caster's option. A magic-user wishing to throw a 20-die fire ball must
pay 21 mp, 6 for the base 3rd level spell and its five dice of damage,
and 15 for the additional damage dice. He could opt to spend only the
base 6 points for the default 5d6 damage, or some number in between. Maximum damage is still limited by level in the usual way; a 10th level mage can't do more than 10d6, no matter how many mp he spends.
Healing spells can be cast normally from the mp pool. Casting from hit points, however, causes a transfer of hit points from the caster to the recipient, on a one-to-one basis. If a cleric casts cure light wounds from her hit points to heal a front-line fighter, and heals 5 points of damage, she loses 5 hit points herself. This is instead of, not in addition to, the usual casting cost.
Some additional optional rules for fun:
I see what you did there: Magic-users (but not clerics) may try to cast a spell they see another magic-user casting. To do so requires the mage to witness another magic-user casting the spell within the last turn. A saving throw vs. spells, with a penalty of twice the spell level, is rolled. Making the save indicates a successfully improvised casting. An Intelligence check, with a penalty equal to the spell level, allows the caster to scribe the spell into his spell book, provided that he does so within one day. Otherwise, it's a one-off feat. (This wonderful idea shamelessly lifted from Untimately and bolted onto this system, where I must say it looks quite handsome indeed.)
Counterspell: A spell caster who notices another caster of the same type beginning an incantation may attempt to nullify it with a counterspell. This costs the same number of mp or hp as the spell being countered. Each caster rolls 1d20 and adds his level and Intelligence bonus. High roll prevails; either the spell takes effect as normal or is completely nullified except perhaps for a token visual "fizzle."
Bonus: Magic items!
Thaumaturgic spirits: This potent magical liquor must be kept in a tightly stoppered glass container, for it will evaporate within 1 turn if left uncovered. It replenishes an arcane caster's reserves of magical power. Upon drinking a dose, a magic-user feels a euphoric rush, gaining mp but losing Wisdom. There are three grades of spirits: lesser, greater, and select. Lesser imparts 1d6+1 mp at the cost of 1 point of Wisdom. Greater gives 2d6+2 mp and drains 1d3 points of Wisdom. Select liquor grants 4d6+4 mp, but drains 1d6 points of Wisdom. It is possible for the user to exceed his normal maximum mp. If not used within one hour, the excess points are lost. Lost points of Wisdom return at this time also.
Non-spell casters are affected similarly, but since they know no spells, the pool of magical power dissipates after an hour. This is usually even more intoxicating to them than to people accustomed to having magic at their disposal, and may cause delusions or hallucinations.
Thinking cap: This magical hat may take nearly any form, but the standard pointed wizard's hat is typical. When worn, it grants exceptional clarity of thought, boosting the wearer's Intelligence by 4 points, to a maximum of 18. An arcane spell caster will gain any bonus mp that accrue from the increased score. The hat will function in this way once per day, and the effect lasts for 1d6 turns.
And with that, I think this topic is pretty well resolved in my mind. That's the last one for a long, long time, I swear.