Many older fans grew up with Elmo too; they just remember when the show was more of an ensemble piece, instead of Elmo taking most of the attention.
That's the one thing I hate about when a character gets so popular... when he's on a show as a host for so long, you don't see any actual character. You just see him talking to the audience. I always bring this up, but watch Elmo in Grouchland VS Elmo's World. In EIG, Elmo goes through a character journey, in which the actions of an over the top villain mimic his own selfishness. It's almost like he grew up and changed during the course of the movie. And you really see a lot of the Elmo and Zoe friendship unfold.
Now look at Elmo's world... Elmo talks about an object or idea, and he shows how it's expressed in different ways, and how a Birthday cake can't do whatever the item or idea can do... that's not character development. That's not tension... that's someone talking about something.
Even the Rocko stuff is brilliant. You get to see Elmo's darker side... Elmo almost saying to the audience, "look! I like Zoe... she's my closest friend. But if she doesn't knock this Rocko stuff out, I'm never talking to this crazy brat again!" The problem is there's a difference between a feeling, sympathetic character, and the same character dressed up in different sweaters and sold at the Disney Store to 3 year olds and middle aged housewives. And unfortunately, Elmo's World puts Elmo in that latter category.