To support your point, I grew up in a poor urban area, and over 30% of the gifted kids didn’t graduate. Another 30% developed addiction or had teen pregnancies. Being poor is much more disadvantaged than being smart is advantageous.
Being smart means you're bored a lot. Being bored means you find other things to occupy your time. Being poor means you have fewer good options, and are more likely to settle on the bad ones.
I think being smart and being poor multiply each other's effects. Most of the troublemakers were really smart, and got into shit because they had nothing better to do. Outsmarting the teachers was fun, making jokes meant you had to know exactly what was going on in order to skewer it, sneaking around was a thrill and a show of skill.
Anecdotally, one of the most brilliant kids I went to school with was also a teen mom, because yeah, gotta keep busy somehow. And that was in one of the most affluent and highest-rated districts in the state if not the country. I can only imagine it's much worse where there are fewer resources.
The problem isn’t access.