E2E encryption can be really resource consuming, especially with the amount of traffic used for video chat. You might not want this just because of a worse user experience.
Video chat is something like 1.5Mbps. Even phones can do chacha20 at >1000Mbps these days. It's not that expensive.
Where people start to feel it is if you want the server in the middle to be decrypting and re-encrypting everything, because it's not that much for one user but it starts to add up for a million users. But the obvious solution there is to use E2EE and then it isn't doing that.