On-premise AI: what it is, how it works and when you need it.

Most people meet AI through a cloud service: a browser tab, a subscription, and a model running in a data centre somewhere. On-premise AI is the other way to have it. This page explains what it is, how it works, how it compares with cloud and private cloud options, when a business needs it, and what it takes to run. It is written for people deciding, so it says where cloud AI is the better answer as well.

What “on-premise” means

The term comes from ordinary IT. Software is “on-premises” when it runs on computers at your own site rather than on a provider’s. Applied to AI, it means three things are inside your building: the language model that generates answers, the documents you want it to work from, and the index that lets it find the right excerpts of those documents. Staff reach it over the office network, the way they reach a file server. The model can be strong: the open models that run on-premise today are capable at document work, drafting and search, and they improve every few months.

How it works

On-premise, cloud and private cloud

The useful question is where your documents are processed and who holds the keys to that machine.

Our on-premise vs cloud comparison goes row by row across all four deployment options, and the private cloud comparison explains why “private” and “on-premise” are different questions.

When a business needs on-premise AI

• Obligations that turn on where information goes. Legal professional privilege, health records law, professional confidentiality rules for accountants and solicitors, the Aged Care Act’s protection of residents’ information, and the TFN Rule all ask who received the information. With processing inside the building, no third party did. Our compliance guides cover each.
• Client terms that require it. Government, defence-adjacent, health and financial clients increasingly write onshore or on-premises handling into engagement terms.
• A policy that says so. Many practices have already banned client material in cloud AI tools and then watched staff use them anyway. On-premise AI makes the approved tool the convenient one.
• Work that must continue offline. A unit keeps answering with the internet switched off, which matters for continuity and for the most sensitive matters.
• Knowledge that should compound privately. A model that learns your practice’s language and documents is more useful every month, and on a unit that improvement belongs to you alone.

When cloud AI is the better answer

If the work carries no confidential material, if your obligations permit processing by an offshore provider under contract, or if you need frontier-model capability for open-ended general reasoning, cloud AI is capable, well-run and quick to start. Many businesses run both: cloud AI for general work, a unit for anything touching client matters. If your documents already live in Microsoft 365 or Google Drive and you want Australian handling without hardware, a hosted private option such as Ermos Edge sits between the two.

What it takes to run

Historically, on-premise AI meant building it yourself: choosing hardware, installing models, writing the retrieval layer, and maintaining all of it. That suits businesses with an IT team and very few others. The alternative is a managed unit, which a provider installs, updates and monitors, so nobody in the business touches a terminal. That combination, hardware at your premises, keys held by you, and a provider keeping it running, is what we call Resident AI. From the business’s side, what it takes is a power point, a network port, somewhere secure to put the unit, and an installation visit.

What it costs, in structure

Cloud AI is priced per seat, so the bill grows with every hire. On-premise AI is priced per unit, and a unit serves the whole practice regardless of headcount. Small teams usually find per-seat cheaper; the lines cross somewhere in the tens of staff, and past that point per-seat keeps climbing. Our cost comparison works through the structures without quoting figures, because exact pricing depends on sizing.

Common misconceptions

• “Local models are too weak.” For open-ended general knowledge, frontier cloud models lead. For work grounded in your own documents, matters and precedents, a local model answers from material no cloud model has seen, and most day-to-day professional work is the second kind.
• “It needs a server room.” A modern unit is quiet enough for an office and the size of a desktop machine.
• “We would have to maintain it.” Only if you build it yourself. A managed unit is maintained by the provider through sessions you start and supervise.
• “Can I run ChatGPT on-premise?” No. OpenAI delivers ChatGPT as a cloud service. Running ChatGPT-style AI on your premises means running an open model on hardware you control. Our ChatGPT comparison covers the difference.

See it in a real document

The clearest way to understand what on-premise AI changes is to look at what leaves a business when a document goes into a cloud tool. Our what leaks to AI series marks up real document types, item by item, and shows the same document on a unit inside the building. Ermos Dominion is our on-premise unit.

Common questions