People love a good urban fantasy. Suddenly, books about the fantastic are everywhere--it'd be really hard not to notice the sudden kick in popularity. (See, for example: Twilight. But look away before you go blind.) What's also noticeable, however, is that people mostly like urban fantasies when they're packaged as urban fantasies.
Take Liar by Justine Larbalestier, a story about a young girl in the city dealing with the death of a boy in her class through several filters of narrative unreliability. Larbalestier's a rising YA author who's made her mark through fantasy stories. Taking it by pattern alone, it should have been no surprise that she'd write another. But when Liar was released, despite a strong narrative voice and an intriguing story, reviews poured in that were lukewarm at best. There was even a little bit of an outcry. I read it myself and felt annoyed rather than impressed by the twist.
In a similar vein, Jennifer Crusie--typecast as a witty and remarkable beach read romance writer--wrote a novel called Maybe, This Time: a retelling of Henry James' The Turn of the Screw. Prior to its publication, she was terribly popular, and for good reason. Maybe, This Time might even have been a strong entry into her list of works. But, once again, readers made unsure faces at the content. They loved the romance; they loved the characters; they loved the creepy old house. But... and there it was. But.
Neither had been marketed as urban fantasy.
Liar was classed as general young adult and Maybe, This Time as romance. In both cases, the supernatural elements aren't casual, subtle little asides in the story--they slam into the plot and carry it away into definite fantasy territory: the transformed land for which there is no ignorance. And their results actually did feel a little bit cheap. Not because the magic had been badly shoehorned in, but because the inherent reader expectation in a non-genre book is for the book to stick to the rules laid down by the beginning of the book--the rules of normalcy, the rules of reality. Readers felt betrayed because the answer had come to them, and it wasn't one they could have ever deduced.
Authors don't often break genres for the same reason that mystery writers don't often end their books with, "oh, by the way, the culprit was this guy nobody ever mentioned who just happened to spot an opportunity and wandered in to commit the crime because he could." Readers like a fighting chance at solving the narrative and, frankly, nobody expects the unexpected. (That's what being unexpected means.)
The problem here is, of course, obvious: nobody complains when a surprise mystery slides into their literature. It's a rare book nowadays that goes entirely without mention of romance. Even science fiction gets away with a little bit of plausible deniability now and then if people slip the tech in discreetly enough. Speculative fiction is very likely the only genre that has to experience this bit of exclusion from the mainstream. And how do you get around that without betraying the reader's expectations?