In my dissection of the series (yes, I've done that, lol), I've come to the conclusion that it's just Eddy the other cul-de-sac kids really hated. On those rare occasions where just Ed and/or Double D are by themselves, you'll notice the other kids don't even seem to mind them at all - there's even been a couple of occasions where Kevin engaged in pleasant/friendly conversation with just Double D. I think it's mainly Eddy's influence, because he's the little pest of the group, and Ed and Double D mainly just go along with him because he's the leader of their group.
That being said, I think it's really only Kevin who held contempt for them (well, again, mostly Eddy), and not really the other kids: Rolf, Jonny, and Nazz never seem bothered by the Eds for the most part, but whenever Kevin's around, suddenly the other kids act like they don't like the Eds either, I think it's because Kevin's the "cool" kid, and the others would rather hang with him to seem like "cool" kids as well, which I can actually remember kids actually doing back when I was a kid, lol.
I mean, I give Danny Antonucci and his writers that, they really captured childhood exploits really well on this show, if even in an exaggerated, fantastic way - Danny even admitted that a lot of what happened on the show came from things that his own kids and their friends did. The Kanker Sisters were based on a pair of girls who stalked him in middle school.