AI-as-a-Service: An Open Source Alternative to the Nvidia AI Enterprise Stack

I started this because the numbers in front of me did not work, and because I could not find anyone who had checked whether they had to. Nobody asked me to do it. I never finished it either, as the architecture was still being ratified when my role was made redundant.

The immediate problem was licensing cost. Nvidia AI Enterprise carried a per-GPU licence, and it scaled badly in exactly the configuration our reference design called for. A minimum of eight GPUs per system, and a system in each of two sites for resiliency, meant sixteen licences before a single workload ran. On accelerated compute, where the hardware is already an expensive line item, that software layer was material enough to affect whether the proposition was competitive at all.

What made me think it was worth the effort was a distinction most people miss. Nvidia gives away a great deal. CUDA, NVML, DCGM, NGC, Triton, NeMo, RAPIDS, TensorRT, DALI, DeepStream, Magnum IO and the Nsight tools are all free. What AI Enterprise licenses is the enterprise wrapper around that free tooling, principally Base Command Manager, plus the support, certification and commercial terms around it. Once you see that, the question stops being "can we replace Nvidia" and becomes "which layer of Nvidia are we actually paying for". Only the second question has a useful answer.

Behind the licensing sat something larger. The AI infrastructure proposition had no managed services layer at all, and customers buying an AI platform, from us or anyone else, would not have the in-house expertise to operate the underlying fabric.

I built it as a capability matrix rather than a product comparison. Every Base Command Manager feature listed, the open source or free equivalent against it, what that substitute actually did, and the steps to implement it. Component level was a deliberate choice and it cost me time. A platform-level argument is quick to make and impossible to defend, because an opponent only has to name one capability you missed. A component-level one is slow to build and hard to dismiss, because every claim in it is individually checkable. I wrote it expecting someone to try.

Two matrices resulted, eleven infrastructure areas and eleven MLOps. Slurm and Argo Workflows for scheduling, split deliberately because HPC and container-native orchestration are not the same problem and one scheduler does one of them badly. Slurm again for GPU allocation including MIG, with GPUDirect, NVML and DCGM alongside it, all free. WekaIO and Cumulus Linux on Spectrum switches for the fabric, with Magnum IO for data movement. PostgreSQL with TimescaleDB, and InfluxDB noted where ingest frequency exceeded it. Prometheus and Grafana bridged to GPU telemetry through the DCGM Exporter. ELK for logging. Canonical MaaS with Terraform and Ansible for provisioning, the same approach I had already taken to a billable feature in the Bare Metal work. Kubernetes with Helm, the GPU and Network Operators, KubeVirt and Singularity. Vault for secrets, KubeCost for cost visibility, which matters far more on GPU infrastructure than general compute.

The weakest version of this analysis concludes the proprietary suite offers nothing. It offers a good deal, and I built the matrix to say so, because a business case that claims more than it can defend gets taken apart in the room, and the person presenting it goes down with it. What the licence buys is largely not unique capability. It is support, certified hardware compatibility, long-term patching, a pre-integrated stack, security and compliance tooling, and commercial licensing. That is a transfer of responsibility, not a purchase of function. So I never framed this as free. I framed it as: this cost moves, here is where it moves to, and here is the evidence we can carry it. NetQ for Spectrum Ethernet and Unified Fabric Manager for Quantum InfiniBand had no substitute, and both are named openly in the matrix rather than left out.

Then the part most substitution exercises leave out. None of this works on paper alone, so I went to the teams who would actually run it and asked what would break. I wanted the objections, not agreement, because they would surface in front of an executive eventually either way. Slurm and Kubernetes skills were already there, which is what turned this from a credible idea into something buildable.

• A row-by-row matrix setting every Nvidia Base Command Manager feature against its open source or free equivalent, eleven infrastructure areas and eleven MLOps areas, with what each substitute does and how to deploy it
• Infrastructure substitutions: Slurm and Argo Workflows, GPUDirect, NVML and DCGM, WekaIO, Cumulus Linux, Magnum IO, PostgreSQL with TimescaleDB, Prometheus and Grafana with the DCGM Exporter, ELK, Canonical MaaS with Terraform and Ansible, Kubernetes with Helm, the GPU and Network Operators, KubeVirt, Singularity, Vault, KubeCost, Slurm accounting and LDAP
• MLOps substitutions: Triton for inference, NeMo, DeepStream, Clara Parabricks, RAPIDS, Nsight Systems and Nsight Compute, Jenkins X with MLflow, TensorRT, DALI and the CUDA-X AI library set
• Slurm dependency analysis testing each scheduling and GPU management capability individually against the licence, covering allocation, fine-grained GPU assignment, elastic compute, GPU-aware scheduling, preemption and backfilling, job arrays, fairshare, fault tolerance, checkpointing and job profiling
• Nine-part assessment of what the Nvidia AI Enterprise licence actually provides, separating genuine capability from support, certification, pre-integration, compliance and commercial licensing
• NetQ and Unified Fabric Manager listed in the matrix as the two capabilities with no substitute, rather than left out
• Use case to technology mapping across eleven AI workload types, tying each to its frameworks and Nvidia tooling
• Checked every assumption against the teams who would actually run it, rather than taking in-house capability on faith
• Managed services definition at component level with the support and managed services lead, reading the matrix from both directions
• The cost case taken to leadership, and a separate AI managed services proposal taken to the services lead and the go-to-market team

• Two people in leadership heard the pitch and received it well. What carried it was not the saving itself, any of us could have estimated that, it was proof the skills to run it already sat inside the company
• Licence cost avoidance was substantial in the reference configuration, where a per-GPU licence across a minimum eight-GPU system, duplicated across two sites, meant sixteen licences before any workload ran
• The longest-lived result of this work was not the licensing argument. It was noticing that nothing in the AI proposition covered managed services, and taking a proposal for it to the services lead and the AI go-to-market team. That gap outlasts whatever happens to the open source stack itself
• Left unfinished. Redundancy came while several substitutions were still being signed off, so nothing was piloted or built, and I do not know whether any of it continued after I left
• The method is the transferable result: substitution argued at component level so it could be checked rather than asserted, validated against internal capability rather than vendor claims, and explicit about what could not be replaced
• This was the third time I have run this same build-versus-buy test, after Bare Metal as a Service and NetFoundry, and the first time the answer came out in favour of building. In-house skills, mainstream components and a licence that scales per GPU all pulled in that direction this time. If the test kept landing the same way regardless of the inputs, that would worry me more than the result itself