As long as you have life saving appliances at hand you will be okay. Immersion suits are usually carried on yachts. The answer is always be prepared for the worst (weather and sea state) when out at sea. And of course be drilled to react. I.e. know how to don an immersion suit quickly if required to.
Spedger