No, you're really not, and I speak from both sides of the aisle. I've confused people in the past whenever I'd change my av, but conversely, there apparently was a period where I was changing my av so much, people were commenting on how strange it was that I actually kept a certain one after so long.
At the same time, I'm also guilty of almost instantly recognizing members by their avs, and, well, yes, if you changed yours (particular if it wasn't a tiny little piggy), I'd probably not realize it's you at first.