This is part of a series of markets about things I could do in 2024. See also: /Mira/which-of-mira-s-cool-ideas-will-mir
Summary
Tired of spinning up expensive GPU VMs to handle your erratic AI model usage? Upload your model(LLM, image generation, etc.), deposit $20, and use a standard API endpoint as little or as often as you like without worrying about scaling VMs up/down.
There is a minimal charge for storage, but mostly you pay per token or per generation.
If you upload a model that others use, you can keep 70% of the API profits paid out to your Stripe account. So, if you finetune a fancy new language model, you can sell exclusive access to it on our site, or upload it for free on HuggingFace as usual and use my site as a way for people to easily test it without setting anything up. You keep full rights to your models by default.
Do you play around with a lot of AI art or language models? Sign up for a subscription and get unlimited use(for ordinary or even heavy human users, but not exploitative) of any model on the site.
Do you have a custom model for business use? (Manifold might have a lora for the Manifold logo or Mana-chan integrated into every market banner) Upload it to our site and use it just like standard OpenAI APIs. A complicated ComfyUI workflow could also be turned into an API, if there was sufficient demand.
Points against
This is kind of an obvious idea. Everybody everywhere knows how to wrap a model in an API. While nobody has this exact pay-per-use scheme with custom models( I checked https://airtable.com/appdROzVqLVH3K1NP/shreLdrTqHNgcR4n0/tblwZmslxCJBaYyXr/viwEkeyBtQU9SzS6S ), it's the kind of thing that somebody could clone as a feature for their model-hosting service.
If the models are public(and even if not, there are enough public models out there), somebody could scrape them all, clone my API, buy Google Ads redirecting everyone to their site instead of mine, and steal all the attention and get more funding and name recognition.
The kinds of people that hesitate to spin up an expensive GPU server are probably cheap customers. Mid-sized companies might also be interested, but they would also be willing to have their infrastructure people manage a fleet of servers.
Market Mechanics
Trigger condition: Deploying any application website to a domain name. Planning, writing prototype code, a static placeholder website, etc. will not be sufficient as minimal effort for this market.
Resolves NA if the trigger condition is not met. Otherwise, resolves to the logarithmic interpolation between nearest bounding answers(or to $1 if I start this but don't make any positive profit for the entire year). Units are "profit in USD". I'll add new options at multiples of 10 as needed before resolving.
Example: I make $500 in annual profit. The nearest options are $100 and $1000. Then this market resolves (ln(500) - ln(100)) / (ln(1000) - ln(100)) = 70% $1000 and 30% 100.
"Profit" is income minus expenses. I'll have to calculate expenses for my taxes, so e.g. OpEx will count 100% as an expense, CapEx will probably be weighted 20%. A salary to myself would count as an expense. Equity doesn't count as income.
Related questions
@Mira Are you talking about a fully serverless model or just a cloud service with autoscaling?
Either way, you're going to have a lot of competitors. All cloud providers offer tools for deploying AI inference pipeleines with autoscaling. If for some reason messing with the VMs themselves is too much, there are several startups that do that: https://ramsrigoutham.medium.com/5-startups-solving-for-ml-serverless-gpu-inferencing-bab35a0f7fc2
I don't know if they have the exact pricing model but pricing custom models per-token doesn't make much sense. Some models run much faster than others. The only reasonable way to price custom models is by how much resources they end up using.
@Shump You can charge per-token or per request for arbitrary models by changing the price based on the model to account for estimated resources. I would've billed per request not per VM instance. Or if it's somehow hard to infer, make the developer choose some parameters. I don't think generating a pricing table is difficult.
I'm probably not going to pursue this idea, so it will likely resolve NA. But I wouldn't be worried about those competitors. If you look at the actual people at those companies, like their LinkedIn and background and everything, only Replicate gives me pause. Everybody else is nothing to me, I have an edge even if they have 10x as much money as me; but Replicate, every idea I think of they have somebody with the right background that predictably would think of it, and I would be slower than their team of 27 people, and they have more money than me.
I wouldn't worry about Amazon, Microsoft, or Google's clouds: They also have good people, but they spend most of their time thinking about big fish. I know exactly how I would get people to prefer my stuff over theirs.
runpod.io is also good enough. That's basically what I would've made first. I had heard of them before, but didn't realize they did the same API thing until a friend mentioned them on Twitter.
Actually, I wrote the above looking at that banana one, thinking "These people are nothing. I could easily beat them", and it looks like that already happened because I see this on their website:
Feb 1, 2024
Hello everyone, today we’re announcing the sunsetting of the Banana Serverless GPU platform.
On March 31st, in two months, Banana infrastructure will be shut down at noon PST. Please ensure that your GPU services are migrated to a new provider by this time.
Later in this article, I’ll provide a guide for a clean migration.
We wish we could have made it work.
And the stated reason is incompetence, just like I thought:
Unfortunately, the realities of business have turned their head. Given current runway, traction, retention, shifting AI macro trends, supply-constrained GPU markets, and a deeper understanding of the engineering required, we’ve realized that we do not have the time and resources to hit that spec.
So don't link me companies that are shutting down from incompetence and tell me they're blocking me. I'm not like them.
Replicate ( https://replicate.com/ ) has basically all of this, including uploading custom models, except AFAIK revenue sharing for custom models, but that's only business code (and possibly legal issues), not a technical moat your prospective startup would have. Also efficient inferencing and serving lots of customer is non trivial.
@CamillePerrin I'll have to look at that one closer, but it sure does look like it kills the idea. That means this is a successful market - because I browsed through a list of "LLMOps Startups" to see if something else was close enough and couldn't find it. So you gave me new information.
You probably want to bet NO on the "Will I do it?" market, since this will likely NA.
Unlike public models, you’ll pay for boot and idle time in addition to the time it spends processing your requests.
I just noticed the public models are a dream, but the private models charge you for a VM and have slow startup just the same.
Definitely a solvable problem, but still they could stop putting it off and do it at any time.