Where it started:
"Well, in 1820 the word 'nut' was a slang term for your head.
It eventually evolved into meaning not just 'head' but 'something wrong in the head', which is one of the ways we use the phrase today, saying someone is nuts, or that you are being driven nuts.
There are multiple forms of this phrase that uses different words, but all of them mean the same thing. You are frustrated to the point of madness"
taken from: Know Your Phrase!