It's simple and it works. It's cheap, it's responsive (button presses are responded to instantaneously, all of my game controllers take several seconds to power on and connect before they're useful), it's completely passive when not in use and gives you great battery life, in regular use (i.e., one TV per room) there's no practical issues with pairing or interference.
The only limitation is that you need line of sight to the TV.
Which might be an issue except generally if you're using a TV, you have line of sight to it because... it's a TV and that's its function.
Line of site is often a pain with things in front of the TV or appliances in TV stands behind a cabinet.
The responsiveness is often slow (granted this is the fault of the TV's UI/software more than it is the fault of the infrared).
Having to point the remote at the TV is annoying - often it doesn't register unless you're sticking your arm out (takes a few tries for me most of the time).
The Xbox controllers work instantly and it doesn't matter what's blocking them. Latency is also low (latency matters in game consoles).
The latency is low once the controller is on and connected. How long does that take? With all of my game controllers (switch, ps3, xbox 360, steam link) it's several seconds.
Latency to register a button press at the other end doesn't really matter for a TV remote unless it goes to extremes. It doesn't matter if you try and turn the volume up and it takes 50ms instead of 10ms. It does matter if that jumps to several seconds.
And since your remote can now travel through walls (and your TV can now receive remote controls through walls!) you now need to worry about security and pairing since there's no longer isolation being provided just by the simple act of having walls. I definitely don't want to try and support my mother in pairing her remote after the batteries need a change every three months.
Have you tried replacing the batteries in your remote? Is your entire room covered in mirrors? It could be something silly like that or maybe that your remote is just terrible. The basic technology has worked for pretty much everyone for decades now without much issue.
I generally just leave my remote sitting on the floor when I'm down there playing with the kid or resting on the arm rest of the chair or something and press buttons on it without picking it up. If it's pointing anywhere within like 60 degrees on either side of my TV whether at the floor or the ceiling it registers every time. This has consistently been my experience across pretty much every TV I've dealt with in 20 years (and I still have a 20 year old LCD TV here I use frequently whose batteries I've replaced maybe 3 times now).
The only limitation is that you need line of sight to the TV.
Which might be an issue except generally if you're using a TV, you have line of sight to it because... it's a TV and that's its function.