Learning About Letters is my favorite Sesame Street video about the alphabet. Though it was made in 1986, none of the segments are from the 1980s. Most are either from 1969 or the early 1970s. And it features almost every main Muppet character from 1986, with the exceptions of The Count and Prairie Dawn (and possibly Elmo). It also had an early Martin Robinson performance as Telly.
Most of the skits are really great. I really liekd Kermit drawing an M on-screen, letters of the alphabet appearing in several signs, Mmmmonster Meal, La La La, C is for Cookie, and The Villian in the pantimine Hat. I would have liked it if this video had something with Harvey Kneeslapper, as well as some segments with more obscure characters like Guy Smiley (What's my Letter? or Mystery Guest should have been on here), Lefty, or Herbert Birdsfoot.