A few more...
1 The episode sponsored by all ten numbers from 1 to 10 was indeed the original version of Elmo In Numberland. That was a great one, which sadly got butchered in subsequent airings.
2 There was an episode featuring the entire alphabet, actually there are three of them I can remember...
a. The first one is the one with the Alphabet Cast Introduction as I've dubbed it for the Muppet List, since I don't have the episode number.
This was the episode with Muppet and human cast members introducing themselves by name, with one for each letter.
Grover was there to state Y is for You.
And Big Bird at the end saying that though noone named with the letter Z lived on Sesame Street, it brought us to the end of the alphabet.
This episode, from discussions here, I think is from at least the first ten seasons worth.
b. There was one where Uncle Wally was doing a sort of softshoe routine throughout the entire SS set from the corner at the Fix-It Shop to the other corner past Big Bird's nesting area. As he danced, he pointed to different parts of the street, and an effect would trigger showing a different lowercase letter.
c. There was from 1992-1995 I believe where Elmo and Telly were on a sort of scavenger hunt, looking for one thing that started with each letter, bringing it back to Luis and María at the Fix-It Shop.
This one I only remember partially, at least the introduction of a Mary & Little Lamb who spoke with a Scottish accent not seen again till recently in Episode 4093. This the character I've denoted as Mary Scot & Her Little Lamb on the list.
Hope this helps and have a good day.