I have a pinephone; it is not good enough. Very unstable; everything (apps/wm) just keep crashing randomly; I send in crash reports. And I try the latest images every month.
But yes, I think it would work. I would be happy with a phone that can run Linux desktop software with HDMI. It would need to be powerful enough (which the pinephone is very much not at all), not eat so much battery when the USB dock is plugged in with HDMI and, most importantly as you say, be stable as Linux. The pinephone feels very far removed from any of this.
Personally, I don't really care about the size of the phone; I gladly give up weight for more battery life (I will have to carry a battery anyway otherwise so why not inside the phone). I guess I would rather the (6.5inch) phone have an eInk display but plug into a screen for a full Linux desktop. But sure, there is no real money in that for phone makers as only people like me would buy them. With Android they are next to useless for me though, which is annoying as there are nice Android (eInk) phones out there if they would run Linux.
A friend has a pine phone and itβs just not good enough as a daily driver. Forget UI. It needs a reboot regularly.