When a client asks how I decide which AI models to trust with company data, my first move is to slow down the phrase “company data,” because it covers a lot of ground. Some of it never belongs near a model at all. If you are building an app, you do not send API keys or passwords over the wire unencrypted; those live in a local environment file. And sensitive personally identifiable information, PII, should not be handed to an external model either.
But “do not share it” is not the same as “cannot use it.” If you are careful about encryption and limit what leaves your walls, you can do a lot of the heavy lifting locally. Say you are querying a dataset full of medical or dental patients, or financial records with real PII in it. You run the query locally, use retrieval-augmented generation against your own database, and aggregate the result first: sum it, average it, pull the trend. Once the data is rolled up, no individual’s record is in it anymore, and that is what goes to the external model. You get the analytical horsepower without sending a single client’s data point over the wire.
The governance gaps I see are rarely exotic. Companies get lazy about where data comes to rest. They run an analysis and leave the output in a file that is not safe, especially when it still has PII in it. They forget to delete old proprietary data that is now a liability, and then it leaks, by accident or through a hacker. Most of it comes down to not being strict enough about aggressively cleaning up data once you are done with it.
On open versus closed models for a typical mid-market business, here is my honest read, and it is a little counterintuitive. A managed closed-source model is usually good enough to get going. But understand the tradeoff: closed means your data travels to someone else’s cloud to be processed. If the data is truly sensitive, an open-source model running locally, on machines or encrypted servers you control, is safer, because that data never enters a system that could absorb it. It stays inside your walls. And the one governance basic almost nobody has actually done? Inventory your data. Sit down as an organization, map what you have, and rate each set by sensitivity to the business. That rating tells you what to protect first.
That instinct, governance before hype, is exactly where the serious operators have landed. Here is the fuller framework.
From performance hype to governance reality
The model-by-model arms race has become a strategic distraction. While the industry fixates on which LLM is currently “smartest,” strategic operators recognize that performance in a vacuum is secondary to data control. The modern priority is not chasing the marginal reasoning gains of a new frontier release; it is establishing a defensible policy where data residency and infrastructure sovereignty come first.
The thesis: the choice between open-weight and closed models should be driven by data governance and infrastructure sovereignty, not the hype cycle. Organizations that prioritize resilience over frontier status focus on preventing vendor logic-lock and enforcing data residency at the architectural level.
Open-source vs. closed-source in a business context
“Open” and “closed” are not just technical labels. They are two philosophies of infrastructure ownership.
- Closed-source LLMs: proprietary systems accessed via managed APIs. The vendor controls the weights, the infrastructure, and the environment where your data is processed.
- Open-source (open-weight) LLMs: models whose weights you can deploy locally, in a private VPC or on-premise, giving you full autonomy over execution.
| Category | Closed-Source (API) | Open-Source (Self-Hosted) |
|---|---|---|
| Deployment Velocity | High: instant API access. | Moderate: requires infrastructure setup. |
| Infrastructure Ownership | None: dependent on vendor uptime. | Full: sovereignty over the stack. |
| Transparency | Opaque: a black box. | High: auditable inference, weights, logs. |
| Customization | Limited to vendor tuning. | Extensive: full fine-tuning. |
Closed models offer the fastest path to deployment, but the true cost of that speed is often a hidden loss of data residency control.
The security tradeoff: where does your data actually go?
Residency is no longer about where a file sits at rest. With generative AI, residency is decided at the AI gateway, the control plane where inference routing happens.
The gateway trap: an LLM gateway sitting in US-East that routes to EU-region endpoints still violates processing residency, because the gateway reads the prompt and processes metadata in the US before routing. For true compliance, the gateway itself must be deployed regionally. Operators should evaluate three tiers:
- Storage residency: data at rest sits in a specific region.
- Processing residency: the inference endpoint sits in the required region. This is more restrictive, since model availability in some regions lags the US.
- End-to-end residency: the entire stack, gateway, model, logs, and key management, is isolated within the region. This is the bar for national security or sensitive healthcare workloads.
Without controls that disable cross-region fallbacks during traffic spikes, organizations risk accidental violations of GDPR, the Australian Privacy Act, or Quebec Law 25.
The economic reality
A persistent myth says self-hosting is prohibitively expensive. In reality, electricity for local runs is less than 1% of equivalent API fees, and once hardware is amortized, local inference can offer roughly a 20x cost reduction over per-token billing. Break-even depends on scale:
- Small enterprises using 30B-class models on consumer hardware: near-instant break-even, 0.3 to 3 months.
- Medium and large enterprises using 70B-class models: a 6 to 24 month window.
- Frontier-scale open models needing $200k+ clusters: 2 to 5 years, viable only at extreme volumes.
Establishing the governance floor
A governed stack is worth more than a frontier stack with no oversight. The basics every operator needs:
- Data entry permissiveness: define which data types (PHI, PII, proprietary code) are allowed in which residency tier.
- De-identification mapping: send pseudonymized tokens rather than direct PII, which reduces “right to erasure” obligations because logs no longer constitute personal data.
- Access and regional audit: implement region-specific logging and identity-governed access so every request is traceable to an approved boundary.
Decision checklist for non-technical leaders
- Can the AI gateway be deployed inside our own VPC or on-premise to avoid the US-processing trap?
- Are cross-region fallbacks disabled by default, even during spikes?
- Does the gateway provide region-specific, identity-governed access for all users and agents?
- Does the model show consistent reasoning and tool-use reliability for its task?
- Are prompts, responses, and audit logs stored only within approved boundaries?
- Does a 70B-class model get within 10% of a frontier model, enough to justify the sovereignty gains?
Conclusion
The ultimate advantage in AI does not come from holding the most expensive API key. It comes from the discipline of the “good-enough” stack: a high-performing open-weight model behind a regionally controlled gateway delivers most of a frontier model’s intelligence with all of the governance. That is the architectural version of what I tell every operator. Aggregate before you send, keep the sensitive stuff on infrastructure you control, and clean up your data instead of leaving it in unsafe files.
But the gateway and the residency tiers are step two. Step one is the thing almost nobody has done: inventory your data and rate it by sensitivity. You cannot govern what you have not mapped, and you cannot choose a model wisely until you know exactly what you would be trusting it with. Do that first, and every model decision after it gets easier.
Governance starts with knowing what you have. Start with a JLytics data assessment to inventory your data, rate it by sensitivity, and decide which models you can trust with it.
