I think what it comes down to is whether or not we even like the Guest Star to begin with. Personally, I can't stand Mark Hamil. He could recite Shakespear dressed as Queen Victoria and riding a flaming armadillo and I wouldn't be impressed. His over acting, mugging and attempts at comedy were nothing short of embarrasing (but the "When You Wish Upon A Star" bit at the end WAS pretty cool!). I have to just edit him out as I watch the show.
On the other hand, I thought that Bruce Forsyth was brilliant, and perfect for the show. But being an American, I have almost no knowledge of who he is (except for a supporting roll in Disney's "Bedknobs & Broomsticks").
Also, can the Guest Star even do televison variety/musical comedy while relating to a chunk of foam and fabric on a grown person's arm? Not an easy thing to do. (I puppeteered a pilot where I had to ad-lib with 8 celebrities-only 2 could do it and they were both non-actors!).
Finally, can the Guest Star bring something to the show that's different AND can work with The Muppets? Big difference between Jsomeone like Julie Andrews and say Debbie Harry (They give her "Rainbow" to "sing"? I'd rather listen to teeth being drilled.)
Any Guest that can't do this would be my vote for "worst".