I think what annoys football fans is the snobbery and acceptance of poor behaviors ,such as pissing / spewing and fighting in the streets at rugby events , compared to the almost zero tolerance for football behaviors.
I have a train driver and police friends who dread the six nations and the unacceptable behaviors that job carrying adults ( not kids ) display , they have no where near the same concerns for the many Cardiff city home games .