I’m just speaking from a town planning (law) perspective. Land ownership is a completely separate matter.

The Council cannot make them knock it down or make alterations to it if it’s been there (without planning permission) more than 4 years.

Any land ownership matters will be between the OP and his neighbour and ultimately for the courts to decide and cost £££. The council will not get involved.