Each GPU cluster has lifeless time. Coaching jobs end, workloads shift and {hardware} sits darkish whereas energy and cooling prices maintain working. For neocloud operators, these empty cycles are misplaced margin.
The apparent workaround is spot GPU markets — renting spare capability to whoever wants it. However spot cases imply the cloud vendor continues to be the one doing the renting, and engineers shopping for that capability are nonetheless paying for uncooked compute with no inference stack hooked up.
FriendliAI's reply is completely different: run inference instantly on the unused {hardware}, optimize for token throughput, and cut up the income with the operator. FriendliAI was based by Byung-Gon Chun, the researcher whose paper on steady batching grew to become foundational to vLLM, the open supply inference engine used throughout most manufacturing deployments in the present day.
Chun spent over a decade as a professor at Seoul Nationwide College learning environment friendly execution of machine studying fashions at scale. That analysis produced a paper known as Orca, which launched steady batching. The method processes inference requests dynamically somewhat than ready to fill a hard and fast batch earlier than executing. It’s now business customary and is the core mechanism inside vLLM.
This week, FriendliAI is launching a brand new platform known as InferenceSense. Simply as publishers use Google AdSense to monetize unsold advert stock, neocloud operators can use InferenceSense to fill unused GPU cycles with paid AI inference workloads and accumulate a share of the token income. The operator's personal jobs at all times take precedence — the second a scheduler reclaims a GPU, InferenceSense yields.
"What we’re offering is that as an alternative of letting GPUs be idle, by working inferences they will monetize these idle GPUs," Chun informed VentureBeat.
How a Seoul Nationwide College lab constructed the engine inside vLLM
Chun based FriendliAI in 2021, earlier than many of the business had shifted consideration from coaching to inference. The corporate's main product is a devoted inference endpoint service for AI startups and enterprises working open-weight fashions. FriendliAI additionally seems as a deployment possibility on Hugging Face alongside Azure, AWS and GCP, and at present helps greater than 500,000 open-weight fashions from the platform.
InferenceSense now extends that inference engine to the capability drawback GPU operators face between workloads.
The way it works
InferenceSense runs on high of Kubernetes, which most neocloud operators are already utilizing for useful resource orchestration. An operator allocates a pool of GPUs to a Kubernetes cluster managed by FriendliAI — declaring which nodes can be found and beneath what circumstances they are often reclaimed. Idle detection runs by Kubernetes itself.
"We’ve got our personal orchestrator that runs on the GPUs of those neocloud — or simply cloud — distributors," Chun mentioned. "We positively make the most of Kubernetes, however the software program working on high is a extremely extremely optimized inference stack."
When GPUs are unused, InferenceSense spins up remoted containers serving paid inference workloads on open-weight fashions together with DeepSeek, Qwen, Kimi, GLM and MiniMax. When the operator's scheduler wants {hardware} again, the inference workloads are preempted and GPUs are returned. FriendliAI says the handoff occurs inside seconds.
Demand is aggregated by FriendliAI's direct shoppers and thru inference aggregators like OpenRouter. The operator provides the capability; FriendliAI handles the demand pipeline, mannequin optimization and serving stack. There are not any upfront charges and no minimal commitments. An actual-time dashboard reveals operators which fashions are working, tokens being processed and income accrued.
Why token throughput beats uncooked capability rental
Spot GPU markets from suppliers like CoreWeave, Lambda Labs and RunPod contain the cloud vendor renting out its personal {hardware} to a 3rd social gathering. InferenceSense runs on {hardware} the neocloud operator already owns, with the operator defining which nodes take part and setting scheduling agreements with FriendliAI upfront. The excellence issues: spot markets monetize capability, InferenceSense monetizes tokens.
Token throughput per GPU-hour determines how a lot InferenceSense can really earn throughout unused home windows. FriendliAI claims its engine delivers two to 3 occasions the throughput of a typical vLLM deployment, although Chun notes the determine varies by workload kind.
Most competing inference stacks are constructed on Python-based open supply frameworks. FriendliAI's engine is written in C++ and makes use of customized GPU kernels somewhat than Nvidia's cuDNN library. The corporate has constructed its personal mannequin illustration layer for partitioning and executing fashions throughout {hardware}, with its personal implementations of speculative decoding, quantization and KV-cache administration.
Since FriendliAI's engine processes extra tokens per GPU-hour than a typical vLLM stack, operators ought to generate extra income per unused cycle than they might by standing up their very own inference service.
What AI engineers evaluating inference prices ought to watch
For AI engineers evaluating the place to run inference workloads, the neocloud versus hyperscaler choice has usually come down to cost and availability.
InferenceSense provides a brand new consideration: if neoclouds can monetize idle capability by inference, they’ve extra financial incentive to maintain token costs aggressive.
That’s not a cause to vary infrastructure selections in the present day — it’s nonetheless early. However engineers monitoring complete inference value ought to watch whether or not neocloud adoption of platforms like InferenceSense places downward stress on API pricing for fashions like DeepSeek and Qwen over the subsequent 12 months.
"When we’ve got extra environment friendly suppliers, the general value will go down," Chun mentioned. "With InferenceSense we will contribute to creating these fashions cheaper."
