What D'Snowth said.
I mean, if you lose even just a single follower, is it such a big deal? I've even had moments where I thought I wasn't popular and felt unhappy about it (such as when I've made threads that got little to no attention). Maybe that one person wasn't interested in following your posts anymore, but I don't think you should take it so seriously.