In the last few years, it seems like there's been a number of celebrity segments that include a brief mention of a letter of the alphabet but aren't really about that letter. Tori Kelly's "Try a Little Kindness" begins with her singing about K standing for Kindness, Sia's "Song About Songs" begins with her talking about S being the word that starts song, Gwen Stefani's "Be a Good Friend" began with her singing about the letter F being for friendship, and this season had Keegan Michael Keys selling newspapers in a segment that ends with him saying that N is for newspaper (and Sesame Workshop's official upload titles it "N is for Newspaper").
I think this started after the show was shortened to a half hour, so maybe it's to accommodate the shorter form, ostensibly including a letter segment while also giving a little more variety (I guess similar to how some early segments are barely about what they teach, best example being Octopuses Garden, which works in a lesson counting to 8 and the CTW archives first season show content lists it among segments about the number 8). Though this isn't that new. The season 17 segment with Big Bird and Snuffy doing a vaudeville song is technically about the letter S, but it doesn't teach the letter that much - just two questions about S words where the "punchline" is that they all start with S, and a bunch of jokes heavily relying on S words (which I didn't really notice until recently that they emphasized S words).
And most of the segments I've mentioned here have appeared on the recent Awesome Alphabet Collection.