That can happen indeed. Compensating for lack of system or domain understanding with ML can result in mediocre results. I've seen this repeatedly with ML teams struggling to get their models adjusted to what was fundamentally not so great data that needed a simple cleanup. Failing to understand the data was dirty, which was easy to address, led to a wild goose chase extracting this and that feature in attempts to make the magic work better.

Once you have deep understanding of your domain and system, finding the places where ML truly adds value is a lot easier. Also, you'll have a basic understanding of how things are without it and you'll know whether it is working better or not and whether that's worth the trouble.