I would agree with that to a point, though sometimes a few freak results will cause that as well.
In terms of a 46-game season, 1609 teams have finished top half with a positive goal difference; 249 have finished top half with a negative goal difference, though this includes the seasons where goal average was used and goal difference has just been calculated.
Obviously, a close goal difference of -1, -2 will be far more common for teams finishing in the top half than much lower goal differences. Ours is currently -14. Only 6 sides finishing in the top half of a division played over 46 games have had a goal difference of -14 or worse, but 947 top half sides have finished with a goal difference of +14 or better.
In 2002/03, Millwall finished 9th in this division with a goal difference of -10. Much of this was down to defeats at home to Portsmouth and Rotherham of 0-5 and 0-6 respectively. Preston finished last season with a goal difference of -14. They won and lost 17 games over the season but their defeats were generally heavier than their wins, which weren't often by more than a goal.
While only 6 sides have finished in the top half with a goal difference of -14 or worse, only 4 sides have finished in the bottom half with a goal difference of +14 or better. All were comfortably midtable. One leapt at me from my spreadsheet, showing Peterborough being relegated in 1968 with a goal difference of +12, which seemed very unusual. On further inspection, they had been deducted 19 points that season for paying illegal bonuses to players.
Only 2 sides have ever been relegated with a positive goal difference; Huddersfield remain the only team ever to win promotion (in 2017) with a negative goal difference.