Magic is an integral - some would say indispensible - part of fantasy stories and fantasy role-playing games. Sure, you can have a medieval game without magic, but it loses a significant element of the fantastic. Even games that bar player characters from being spell-casters often do so not to expunge magic from the game entirely, but to keep it beyond the understanding of the players - to keep it wild and fantastic and fearsome.
I like magic in my game. I like for the players to be able to choose to run spell-casting characters if they want to. But I also like for magic to be magical - wild and fantastic and fearsome - as much as it can be without making it the province of DM and NPCs only. That's why I like the tight spell lists of B/X and BECMI D&D - at least as a foundation upon which to build.
There is, it's true, a lot to like about the massive variety of spells in AD&D, as well as various supplements. They can add a lot of flavor to the campaign milieu, and utility to characters, both PC and NPC. They can serve this purpose without being added to the standard spell lists.
The tight lists of 12 magic-user spells and 8 cleric spells per spell level are the ones that are most commonly known. Not every spell caster will know every spell, but most are at least aware of the existence of these spells. If you don't know Phantasmal Force, you at least know that there is such a spell, and that with a little determination you can probably ferret out a source from which to learn it. They're the magical meat and potatoes of the campaign. They allow for a good diversity of functions, and the campaign will survive just fine on a steady diet of them.
Beyond those lists is the whole kitchen sink of spells, every one that's ever caught your eye in another rule set, or an adventure module or supplement, everything that you might devise from your own imagination, whatever you might fancy dropping into your current game world. Rather than dumping them into the mix wholesale, you carefully pick and choose which ones fit, and where they'll be found.
There could be any number of explanations why those non-list spells are so rare and obscure. Perhaps the civilization that invented them fell and the knowledge was lost. Perhaps they're leftover "beta" versions of common spells that fell out of favor with the discovery of new versions, with surprising bugs and maybe even a few forgotten utilities. Maybe they were invented by wizards who keep their secrets close to the vest. Maybe they're banned by the king, the church, or the mages' guild, for reasons ethical, spiritual, or commercial. (The flimsiest pretense will do - look at the historical reasons in our real world for banning all kinds of things.) Maybe they can be learned only by dangerous rituals or pilgrimages to sacred or magical sites, or by using ancient devices that project knowledge directly into the caster's mind. Some of them might even have inhuman origins, and can be learned only from dragons or fairies or demons or what-have-you; humans might be able to understand them well enough to memorize and cast them, but not well enough to teach them to another human.
These are the spells that you give judiciously to NPC opponents or allies to make them more menacing or mysterious. These are the ones you place very rarely in treasure troves to get your spell-casting PCs excited. These are the ones the players might hear about in rumors, motivating them to undertake expeditions and quests to obtain them. These are the spells that might convey all sorts of interesting implications about the campaign world and its societies and history. These are the spice that you add to the dish of meat and potatoes. They're not essential, but a sprinkling of them adds interest and versatility.
From a pragmatic perspective, a scheme consisting of a small staple list and a universe of supplemental stuff provides the ease and convenience of the former, while allowing you the freedom to tempt or bedevil your players with more exotic stuff as needed, and as suits the particulars of the campaign and the world in which it takes place.