Well...as Cantus put it "Invisibility is in the eye of the beholder."
Yes, Doc does see Gobo, but it isn't until a connection is made between them that Doc and Gobo both interpret Cantus's way of seeing. This was established in "The Honk of Honks". In the beginning of the episode, Doc looks straight at Gobo (Sprocket even points to him!) and Doc says:
"I don't see anything."
Later, Gobo makes the connection of "touch" between him and Doc.

"I felt bad for you. That's how you touched me, and I touched
you. You aren't just a Silly Creature! You're a
you, aren't ya?"
Doc: "I'm a...what?"
Gobo: "A
you, like
me!"
As for Matt, yes a lot of Silly Creatures, young and old, see him. However, the grown-ups (IMHO) would just mistake him for something they could interpret and understand, logically a little old man with a big nose. In "You Can't Do That Without A Hat", an old lady comes to him at a donut shop and says "Oh, you poor man. You look like you could use this" and she gives him a quarter.
This underlying aspect would seem to further underline the Silly Creature's inability to truly "see" (to understand) as Cantus puts it.
So to perhaps put it as simply as I can, Matt was noticed by many, but not seen for what he truly was. So in a way, his true nature as a Fraggle was "invisible" to the Silly Creatures. Adults merely saw this vague Mark Twain-like, short old man in explorer gear stumbling his way around the world.
Convincing John