Yeah, turns out all TV airings of that song have the dialogue intact. It appears home media releases often trim/mute it….. for whatever reason.
EDIT: Okay, so turns out for whatever reason Sesame Workshop uploaded this clip on their website with his dialogue trimmed/muted next to the full, unedited clip, which later on carried over to its YouTube upload. I’m guessing that version must have just not been used on the show.
I know there's also a version of Gospel Alphabet that augments the chorus with additional vocals, though I can't quite tell if this is that version. Without it augmentation though, you can clearly make out on-set vocals from certain performers like Frank Oz, Fran Brill, and Jerry Nelson.