I think it's always a mistake to say, "it doesn't have to be good, the audience will just think it's funny" etc., etc. To me this is just an insult. If the singing doesn't have to be good because "they came to see a puppet show, not a singing show", then by the same logic the puppets in Avenue Q don't have to be good because they came to see a broadway musical, not a puppet show. The context of the singing is what determines if it needs to be sung well, not the fact that it's a puppet show.
Now this is not to say that you have to have a stellar voice. For instance, if Randy Jackson were to critiquing Jim Henson's voice he'd say something like, "Check it out, dawg, it's like this, baby. Your voice is like, OK and everything, but it just didn't hit it for me, know what I'm sayin? Yer harmonies are eeeehhhhhh, just a little flat, kinda pitchy..."
All this to say that Jim Henson had a decent sense of pitch, but perhaps not the best "instrument" as far as singing. He was far from unpleasant to listen to, because he was on pitch. Perhaps not always in the dead-center of the pitch, but not off-key. Really the same could be said of Dave Goelz and Frank Oz. Richard Hunt and Jerry Nelson had very nice singing voices. If you track their singing over the years you'll notice that they ALL improved immensely. Except maybe Jerry who has always been amazing...
I'd day if you're going to sing, do so to the best of your abilities, and in fact taking some voice lessons would only be helpful, even if they don't turn you into a professional singer. I studied singing for 13 years straight and to this day count it as one of the smartest things I've ever done. not only for singing, but for performance in general. But if you're only willing to "go through the motions because no one will really care", then you're doing a disservice, both to yourself and your audiences.
Now perhaps the most important thing to remember about theatrical singing is this: character always wins. If you sing in the character's voice, and are honest to the character and the story, then it is hard to go wrong. Even if you don't have the best singing voice in the world. The context will carry you, not the singing skill (or lack thereof).