Yeah, because performing a Muppet is more than just doing a voice, you have to act through your arm and your hand and transfer your feelings and emotions into the puppet, and then the voices comes from that. This is why most of the time, it was important with Jim that if you were going to voice a character, you had to actually phyiscally perform the character as well.
Conversely, Sid and Marty Krofft would have little people inside their suits, then bring in voice actors to dub all of their voices.