Turns out the passenger was at fault all along, if the pilot asks you to leave the aircraft you have no choice.