Listen, I know Frank performed Brown Grover quite a bit, but I'll swear on a stack of bibles Jim performed him this time. During the first season, when most of the monsters didn't have distinct personalities yet, there was quite a lot of mix-and-matching going on.
Case-in-point, there was a skit with six monsters lined up along the brick wall
whispering "C" to each other. Cookie Monster was there, but instead of Frank, Carroll Spinney voiced him.
There's evidence in either the Sesame Street Unpaved book, or Jim Henson: The Works that when Frank and Jim performed Bert and Ernie for the first time, Jim was Bert and Frank was Ernie, until they realized they were more comfortable switching roles.
In the song Up and Down, Jim performed the prototype Herry Monster puppet, which was later inherited by Jerry Nelson.
So it's not beyond the realm of possiblity Jim could've been Brown Grover, if only for one skit, before passing it on to someone else.
I think I missed my calling. I should've been a lawyer. Or maybe not.