This article really resonates with me: I'm currently working on bootstrapping an applied AI service and faced most of the challenges mentioned, to the point where I was doubting myself as I was not as efficient or productive as other Saas / Software founders. So this article is kind of reassuring to me.
A few more comments:
Cloud infra
For a traditional web app you can quickly deploy it on a cheap AWS on Heroku VM for a few dollars/mo and later scale as you get more traffic. With AI you now need expensive training VMs. There are free options such as Google Collab but it doesn't scale for anything else than toy projects or prototyping. AWS entry point GPU instance (p2.xlarge) is at $0.900/hour i.e. $648/mo, and a more performant one (p3.2xlarge) at $2160/mo. Yes, you should shut them down when you are done with training but still. You can also use spot instances to reduce cost but it's not straightforward to set up.
For inference, you also need a VM with enough memory for your model to fit in, so again an expensive VM from day one.
Datasets
if you rely on a publicly available dataset, chances are there are already 10 startups doing the same product. In order to have a somewhat unique and differentiated product, you need a way to acquire and label a private dataset.
Humans in the loop
The labeling part is very tedious and costly both in terms of time and money. You can hire experts or do it yourself at great cost, or you can hire cheap outsourced labor who will deliver low-quality annotations that you will spend a lot of time controlling, filtering, sending back, etc.
For inference, depending on your domain, even with state-of-the-art performance you may end up with say 90% accuracy, ie 1 angry customer out of 10. that's probably not acceptable, but it gets worse: chances are you will attract early-adopter customers who are faced with hard cases, whose current solution doesn't work so that's why they want to use your fancy AI in the first place. In that context, your accuracy for this kind of customer might actually be much worse. So again you need significant human resources to control inferences in production. It will be hard to offer real-time results, so you may have to design your product to be async and introduce some delay in responses, which is maybe not the UX you initially had in mind.
I still think there are tremendous opportunities in applied AI products and services, but it's important to have these challenges in mind when planning a new product or startup.
How does it compare with building your own GPU-heavy computers? I'm not too familiar with it or how consumer-grade GPUs fare in these workloads, but training sounds like it in theory could happen locally easier than any consumer facing parts.
A few more comments:
Cloud infra For a traditional web app you can quickly deploy it on a cheap AWS on Heroku VM for a few dollars/mo and later scale as you get more traffic. With AI you now need expensive training VMs. There are free options such as Google Collab but it doesn't scale for anything else than toy projects or prototyping. AWS entry point GPU instance (p2.xlarge) is at $0.900/hour i.e. $648/mo, and a more performant one (p3.2xlarge) at $2160/mo. Yes, you should shut them down when you are done with training but still. You can also use spot instances to reduce cost but it's not straightforward to set up.
For inference, you also need a VM with enough memory for your model to fit in, so again an expensive VM from day one.
Datasets if you rely on a publicly available dataset, chances are there are already 10 startups doing the same product. In order to have a somewhat unique and differentiated product, you need a way to acquire and label a private dataset.
Humans in the loop The labeling part is very tedious and costly both in terms of time and money. You can hire experts or do it yourself at great cost, or you can hire cheap outsourced labor who will deliver low-quality annotations that you will spend a lot of time controlling, filtering, sending back, etc.
For inference, depending on your domain, even with state-of-the-art performance you may end up with say 90% accuracy, ie 1 angry customer out of 10. that's probably not acceptable, but it gets worse: chances are you will attract early-adopter customers who are faced with hard cases, whose current solution doesn't work so that's why they want to use your fancy AI in the first place. In that context, your accuracy for this kind of customer might actually be much worse. So again you need significant human resources to control inferences in production. It will be hard to offer real-time results, so you may have to design your product to be async and introduce some delay in responses, which is maybe not the UX you initially had in mind.
I still think there are tremendous opportunities in applied AI products and services, but it's important to have these challenges in mind when planning a new product or startup.