It isn't racsit because he hasn't said anything negative about white people that isn't true. Throughout history black people were treated like shit and actually owned by a lot of white people who used that word negatively a lot. Even after the abolition of slavely black people were treated like shit by a lot of white people.
You being a good person has nothing to do with you being able to use that word or not, the history of the word and the way it's use by white people still makes black people feel is the reason you shouldn't use it.
It wasn't even that long ago there were billboards in the uk saying "If you want a n***** for a neighbour vote labour" so it isn't even like it's ancient history.