That's fair.

I would check your online account (personal tax account) to check if there's any outstanding returns. If there are, it's likely they've estimated the tax due and applied penalties for non-filing, and are collecting it all through PAYE.

Self assessment is a little out of my area of expertise, but this is what I assume is happening.

The estimated tax you should be able to sort out because it's just a case of filing nil returns for the years you weren't self employed, but the penalties you're likely going to need to appeal. Should you win, you will probably get it all back.