My grandparents were from Deptford, London, and when they came to Australia they formed a large extended family with other ex pats. So, I grew up with lots of accents from the UK, and it never occurred to me that they could be unintelligible to other English speakers until I was over in the UK with some Aussie friends who were totally stumped - particularly by strong Irish or Scottish accents.
Then, to my even greater surprise, my wife and I were in Scotland and neither of us could understand some people.