It is better than being a naughty booger.
I had to google it to make sure she didn't call you a dirty name.
Being American I could be wrong, but my understanding of the term is:
In the English (as in the country) it is a semi polite term, but has an equalivancy of fuck.
In past generations (as in the 1950's and prior) a "bugger" was a sodomite, but came to be synonymous with gay, at a time when that was an insult.
Being old has advantages, you learn some things.