I believe Tosh would have been very happy to return but as you suggested some within the club were not keen.
Also worth recalling that whilst there has always been a fans rivalry with the Swans we had had many players who had started there careers at Swansea or were from that area and vice versa. His going west was a much easier decision to make then that it would probably be now.