No, the Fraggle Five don't just ignore the other Fraggles. They quite get along with them!
For example, Wembley develops a crush for a Fraggle named Lou in "We Love you Wembley", and Boober seems to have a little "thing" for a Fraggle named Tosh (I LOVE Tosh!!! ^^ She's so adorable. And her beast is supah coo'!) in some later episodes (of course, this is just my analysis... I just think they'd make a FINE couple). And there's more! I just don't feel like writing it out.
And how do we REALLY know that the Fraggle Five are the MAIN Fraggles? They are the main Fraggles of the show, but for all we know the other Fraggles are off having some zany adventures too! The way I think of it is the Fraggle Five are like a clique, but they go off and talk to other Fraggles, play with other Fraggles, blah, blah, blah... but those other Fraggles have cliques of their own. So everyone's happy!
^_^ Debo