One of my neighbours mentioned to me today that he finds the squirrels that scamper around here annoying, and although he didn't say so I suspect it's just a matter of time before he asks me to stop putting out nuts for the birds & squirrels.

I don't subscribe to this way of thinking and feel quite strongly about this.

If I am feeding the birds and squirrels in my own garden (privately owned and fenced in) does he have any right to stop me (i.e. by complaining to the council etc)?

He has no right to stop you feeding wild animals. You should however recognise that squirrels are not all sweetness and light, and that he has a proper right and reason to be upset.
Could you consider feeding only the birds, from a squirrel-proof table, because their needs are much greater than the needs of the squirrels.