The issue with Frank was his obnoxious laugh hurt his throat; while, yes, Harvey was a one-joke character, it was mostly the former on why he was dropped.
Far as I know, I haven't seen too many fans say Harvey was annoying. As a kid, I found him to be kind of strange, but I love him more as an adult because of the corny jokes that come from Frank.