This sounds similar to myself. I actually got the highest year 6 SAT test scores my primary school had ever achieved. It was the scummiest primary school in the town, right in the heart of the worst streets on the biggest council estate.
While certainly not being the smartest kid in my secondary school, I was up there. That school was an odd one in terms of demographic because it was the state comprehensive but it was in the posher part of town. As a result, most of the council estate kids went there but a lot of lower middle class kids and posher village kids went there as well.
Growing up in a very closed working class environment is what made me think so much of myself, I think. When all the other kids on the estate were out scaring grannies and smashing up park benches for no reason whatsoever, I was sat at home reading books on medieval history and watching programmes about architecture!
After leaving school and entering the real world and starting to travel a lot, I was certainly humbled when suddenly I was meeting people who not only were into the same things as me but knew masses and masses more.