On-Premises AI for Law Firms: How It Actually Works

On-premises AI used to mean a six-figure GPU cluster in a server room with a cooling budget. It does not mean that anymore. A modern firm can run a useful, productive private AI system on a single appliance that fits on a shelf, costs less than a year of cloud AI subscriptions for a mid-size team, and never sends a prompt outside the firewall. Here is how the stack actually works.

What "On-Premises" Means in 2026

When we say on-premises AI, we mean all four of these things are true:

• The model weights live on hardware the firm owns or controls.
• Inference (the step where the model produces an output) happens on that same hardware.
• Any firm documents the model reads stay on that hardware.
• No prompts, outputs, or document content are transmitted to a third-party AI vendor during normal use.

That is a stricter definition than some vendors use. You will see "private AI" used to describe a cloud deployment that runs in a single tenant's virtual environment. That is better than multi-tenant SaaS AI, but the data still leaves the firm. The definition above is the one that answers the compliance questions cleanly, because it is the only one where the data never moves.

The Hardware Story

Running a capable modern language model used to require an expensive GPU server. The model architecture and the hardware have both moved. A few things happened in parallel that changed the economics.

Quantization techniques (compressing model weights to smaller numeric formats with minimal quality loss) shrank the memory footprint of frontier-quality open-source models by 2-4x. Apple silicon (M-series chips) unified CPU, GPU, and neural engine access to a single large pool of fast memory, which turns out to be exactly what language model inference needs. Consumer GPUs (the latest NVIDIA RTX cards) carry enough VRAM to host mid-sized models. Specialized inference frameworks (llama.cpp, MLX, vLLM, and others) squeezed more throughput out of the same hardware.

The net result is that a firm has real choices at multiple price points.

• Entry appliance (Mac Studio, high-end Mac Mini, or equivalent PC workstation): host a strong mid-sized model, serve a small team, handle most day-to-day workflows. Sits under a desk.
• Mid-tier appliance (multi-GPU workstation, Mac Studio Ultra, or similar): host larger models, serve a mid-size team concurrently, handle longer documents and richer retrieval contexts.
• Firm-wide server (rack-mounted server with enterprise GPUs): for larger firms, shared across offices, with redundancy and higher concurrency.

The right tier depends on headcount, workload mix, and whether the firm wants a single-office appliance or a central system that remote offices connect to over a VPN. All three tiers ship today, and the delta between them is real but not unreasonable.

The Model Story

The models that matter for on-prem deployment are open-source or open-weight. Llama, Qwen, Mistral, DeepSeek, and a handful of others now ship models with permissive licenses and quality that is in the same conversation as frontier cloud models for the tasks professional services firms actually care about: summarization, drafting, extraction, classification, retrieval-based Q&A, and structured output generation.

"Same conversation as frontier" is not "identical to GPT-5 on every benchmark." For tasks that need long-tail general knowledge, bleeding-edge reasoning, or exotic multimodal capabilities, frontier cloud models still win. For the 80% of actual firm workflows (intake, drafting, reviewing documents the firm already owns, generating structured outputs against firm templates), open-source models are more than sufficient.

The practical choice at deployment time is not "one perfect model." It is usually two or three models on the same box, routed by task. A smaller fast model for interactive chat and quick drafting. A mid-sized model for longer documents and more complex reasoning. A specialized embedding model for document retrieval. All running locally. All swappable as better open-source models are released.

The Software Stack

Zoom out and the on-prem stack has four layers. Each one is a real engineering choice, and each one has mature open-source options in 2026.

Layer 1: Hardware (the box itself). Covered above. Pick the tier.

Layer 2: Model serving. This is the software that loads the model weights into memory and serves inference requests. Frameworks like llama.cpp, Ollama, MLX, and vLLM each have their strengths. The model serving layer exposes a local API (usually an OpenAI-compatible endpoint) that the rest of the stack talks to. Nothing on this layer calls out to the internet during inference.

Layer 3: Retrieval and firm knowledge. This is where the firm's documents become useful to the model. Documents (engagement letters, past memos, firm templates, research files) are indexed into a local vector database. When a staff member asks a question, the retrieval layer pulls the relevant firm documents and hands them to the model as context. This is the layer that turns a generic model into a firm-specific assistant. Everything stays on the box. No document content is transmitted anywhere.

Layer 4: The agent and app layer. This is what staff actually see. The prebuilt agents (drafting, intake, reconciliation, document review) that we wrote about in a companion post. The user interfaces, the inboxes, the dashboards. This layer calls the model serving layer, which calls the model. Simple in principle, a real engineering job in practice, but every piece of it can run on the same box.

Each layer is independently upgradeable. A new open-source model ships, you swap it. A better embedding model comes out, you reindex. A new agent is needed, you add it to Layer 4. The firm's investment is in the stack and the firm-specific data in Layer 3, not in any single model.

What Runs Locally, and What Doesn't

"Everything on the box" is the goal, but it is useful to be precise about what that means in practice.

• Inference runs locally. Always. Every prompt, every output. No exceptions during normal use.
• Firm document retrieval runs locally. The vector index and the documents themselves live on the box.
• Model weight updates are optional and explicit. When a better open-source model ships, the firm can choose to download it. This is a known, auditable network event, not a background data flow.
• Software updates are also optional and explicit. Same pattern. The firm decides when to update and what to update.
• The internet is not required for day-to-day use. You can unplug the network and the system keeps working.

That last point matters. If the box works offline, you know the client data is not traveling. That is a testable property, not a promise.

Guardrails Move With the Box

Private AI is not a license to skip the guardrails that cloud AI forces you to think about. It just means the guardrails are under the firm's control.

Access controls on who can use which agents. Audit logs of who did what, when. Data retention policies for prompts and outputs. Role-specific model routing (partners can do things paralegals cannot). Ethical walls across matters. All of these still have to be implemented. They just run on the same box as the model.

The advantage is that the firm writes the policy, the firm enforces it, and the firm has the logs. There is no vendor in the middle whose policy can change in a future terms-of-service update.

The Cost Picture

A rough, deliberately conservative three-year comparison for a 10-person firm:

• Cloud AI (per-seat, enterprise tier): $30 to $60 per user per month, times 10 users, times 36 months = $10,800 to $21,600. Before add-ons for document search, retrieval, or specialized tools.
• On-premises appliance (entry tier): Hardware in the $5,000 to $15,000 range, plus a software and support layer, plus an initial setup. Most firms at this size land in the $20,000 to $40,000 range all-in for three years.

On pure dollar cost, on-prem is roughly comparable to cloud for a firm of this size. The cost argument is not "private AI is dramatically cheaper." The cost argument is "private AI is in the same ballpark, and it also removes the compliance work, the vendor risk, the data disclosure question, and the ongoing negotiation over what the vendor does with your data." Those are not line items on an invoice, but they are real.

For larger firms, the math tilts more clearly toward on-prem. Cloud AI costs scale linearly with headcount. On-prem costs scale with hardware and support, not per user.

The Trajectory

Two things are true at the same time. Cloud AI is getting better, faster, and more sophisticated in ways that are hard to replicate on a single appliance. Open-source models are getting better, faster, and more efficient in ways that make on-prem more capable every quarter.

For a professional services firm, the second curve is the one that matters. A firm does not need bleeding-edge capability. It needs reliably good capability on private infrastructure. That is exactly what the open-source trajectory is delivering.

The firms buying private AI appliances today are not doing it because they think the technology is frozen. They are doing it because the architecture (their data, their hardware, their rules) does not become less right over time. It becomes more right as AI becomes more embedded in the work.

If you are thinking about what private AI should look like inside your firm, let's talk. And if you have not yet read the companion piece on why cloud AI is a problem for professional services, start there.