That's a big 10 for BlueAM! Ah ha, I love numerical puns!
Yep, the video Count It Higher: The Count Counts Sesame Street Music Videos did indeed come with a little booklet. See, this was part of a series of SS videos that were I believe 18 in total and all had their specific number on the outside lateral sides of the video box. The video boxes were in a range of colors: red, orange, maybe yellow, green, and blue; and the number on the little square box on the sides had the video's volume number in that same color. The front of the video cassette's case had an image for each, the one in question showed my namesake in front of what could be the music den and some gold records behind the mantle.
Additionally, each video in this series came with an activity booklet. Got six videos of the series myself: the one in question, Learning About Letters, Learning About Numbers, Sing!, Bedtime Songs & Stories, and The Alphabet Game.
The booklet for Count It higher did have blank uncolored images of stuff related to the songs featured on the video, can't recall what they were you know my blind batty eyes. There were also lyrics to all the songs featured on the video.
Hope this helps and have a good night.