Choosing a self-hosted AI workflow platform

Evaluate the operating model, not the demo

Every platform in this category demos well, because the demo shows the reasoning step. The differences appear in the parts that are unglamorous: what happens on restart, what happens when authorization cannot be evaluated, and what the upgrade path looks like eighteen months in.

Deployment and lifecycle checklist

Ask for documented installation, upgrade, rollback, backup and disaster-recovery procedures — and ask whether they have been tested, not just written. Ask how component versions are pinned and how compatibility between CLI, API and runners is enforced across an upgrade.

A platform that cannot be rolled back is a platform you cannot safely patch.

Correctness questions worth asking

Are run and task state transitions durable and recoverable? Can a retry duplicate a non-idempotent external action? Do authorization failures fail closed? Are invalid dependency graphs rejected before execution rather than discovered halfway through?

Does a runner restart lose work silently, or does the platform detect and recover it? These answers separate a product from a promising internal tool.

What your security review will ask for

Prepare for questions about authentication and authorization models, secret storage and rotation, execution isolation, supply-chain artifacts such as SBOM and provenance, prompt-injection posture and tool access control.

Also ask what the vendor does *not* claim. A platform that clearly states it has no published attestation is easier to evaluate than one that implies certification it has not received.

What "self-hosted" needs to mean specifically

The term covers a wide range, and the range matters. At one end, the software runs in your infrastructure and holds its own state. At the other, a locally installed component talks to a vendor control plane that makes the decisions — which is a hosted product with an on-premises client.

The questions that separate them: where does run state live? Where are authorization decisions evaluated? If the vendor's infrastructure is unreachable, does execution continue? Can the deployment run without egress to the vendor at all?

None of these have universally correct answers, and a managed control plane is a legitimate model with real operational advantages. But it is a different risk profile from the one people usually mean by self-hosted, and the difference should be established early rather than discovered during a security review.

Data residency, and the part that leaves anyway

Self-hosting settles where the platform's own data lives. It does not settle where the data in a run goes, because a run that calls an external model provider sends content to that provider by definition.

So the honest question is not whether data leaves but which data, to which endpoint, under whose approval. A platform that treats provider selection as governed configuration — approved endpoints, scoped credentials, per-profile choice — gives you an answer. One that treats it as a per-pipeline environment variable does not.

This is also where locally hosted models become an architectural option rather than an ideological one. If provider choice is configuration, running some workloads against a local model and others against a hosted one is a routing decision, not a rewrite.

The bootstrap boundary

Every self-hosted platform has a small set of material it cannot manage itself: database authentication, the root key that decrypts everything else, service identity and transport trust, and whatever is needed to restore an encrypted backup.

Ask for that set to be enumerated explicitly. It should be short, it should come from your deployment platform's own secret mechanism, and — importantly — it should not be writable through the application's configuration interface or setup UI. A system that can be reconfigured through the interface it protects has a circular trust boundary.

The corollary is a question worth asking directly: if the root key is lost, what is recoverable? The answer is usually "nothing encrypted with it", and it is better to hear that during evaluation than during an incident.

Configuration as reviewable artifact

A platform whose operational definitions live only in its own database has an upgrade problem, a review problem and a disaster-recovery problem simultaneously. You cannot diff a database row in a pull request, promote it between environments, or reconstruct it from a repository after a failure.

What to look for: definitions expressed as files, an explicit sync direction, and drift detection that tells you when running state and repository state disagree. Ask which direction wins on conflict, and ask what happens to a change someone made through the UI while a sync was pending.

Ownership boundaries matter too. It is a good sign if some configuration is restricted to a system-level repository while teams manage their own — that separation is what stops a team-level change from widening a platform-level surface.

Identity integration, honestly assessed

Most evaluations check the box for SSO and stop. The harder questions are about what happens between the identity provider and the platform's own model.

Can the provider contribute roles and group memberships, and are those grants distinguishable from locally managed ones? When a directory sync runs, does it prune only the grants it owns, or can it remove access someone granted deliberately in the platform? Does removing a user from a group in the directory actually revoke their reach, and how quickly?

Ask about the failure mode too. If the identity provider is unreachable, does the platform fail closed for new sessions, and can an operator still reach it to fix the problem? Both answers should exist, and they should have been decided rather than defaulted.

Execution isolation and placement

Where a run executes stops being an implementation detail as soon as some work is more sensitive than other work. The question is whether placement is something you control declaratively or something that happens to be true.

Look for per-run isolation rather than a shared long-lived worker, and for placement selected by scope, capacity and affinity rather than by whichever executor was free. The practical test: can you guarantee that work touching production data runs only on runners inside a specific boundary, and is that guarantee configuration or convention?

Ask about runner identity and revocation as well. Removing a runner should disconnect it promptly, and there should be a deliberate way to keep an identity permanently revoked — distinct from removing it in a way that lets the same name reconnect later.

Upgrades, compatibility and the eighteen-month question

The demo is the best the product will ever look. The upgrade is where you find out what you bought. Ask how API, CLI and runner versions are pinned relative to each other, and what happens when they drift — specifically, whether a runner one version behind fails loudly or behaves subtly differently.

Ask what a rollback actually restores. Configuration is usually straightforward; database migrations frequently are not. A platform with forward-only migrations and no tested rollback path is one you cannot safely patch under time pressure, which is exactly when you will need to.

And ask how long a version is supported, because the answer determines whether upgrades are something you schedule or something that happens to you.

Operational visibility before you need it

Ask what the platform tells you about itself, not just about runs. Component health, dispatcher and runner state, queue depth, and per-workflow cost attribution are the difference between diagnosing a problem and guessing at one.

Log access deserves a specific question, because it is a common gap. Can an operator read component logs through the platform with the same authorization model as everything else, scoped per source, redacted, with the access itself recorded? A platform that requires shell access to its own containers for routine diagnosis has pushed a governance problem into your infrastructure.

Cost attribution is worth asking about explicitly. Aggregate spend tells you there is a problem. Spend per pipeline, per run, per model and per step tells you which workflow to look at.

Reading the compliance answer carefully

There is a meaningful difference between "we are certified", "we are in the process", "we are built to support your certification" and "we have no attestation". All four can be acceptable depending on what you are buying and when. Only one of them is often implied without being said.

The genuinely useful signal is specificity. A vendor who names their current status plainly, including the parts that are not finished, is easier to evaluate and generally easier to work with than one whose materials are ambiguous in a consistently favourable direction.

For a self-hosted product, a substantial share of the control environment is yours anyway. What you are evaluating is whether the platform gives you the evidence to demonstrate your own controls — which is a question about audit records and access models, not about the vendor's certificate.

A shortlist scoring approach

If you are comparing options, resist scoring features. Feature lists converge and tell you little about how a system behaves under stress. Score the operating model instead, in five areas: configuration ownership, authorization model, execution placement, evidence durability, and lifecycle maturity.

For each, write down what the platform does natively, what you would have to build around it, and who would own that in two years. The last column is the one that predicts regret. Anything you would build around a platform is something you will maintain, review and eventually explain to an auditor.

Then run one real workflow — not the demo — end to end on the shortlist. Pick one that already crosses three systems and requires an approval. It will surface more than any questionnaire, and it costs less than choosing wrong.

Backup and restore, tested rather than documented

Every vendor documents backup. Fewer can describe a tested restore, and the gap between the two is where self-hosting goes wrong. The question to ask is not whether backups exist but what a restore actually reconstitutes.

Work through it concretely. Run history and evidence records — restored, or lost? Encrypted credentials — restorable, and with which key? Configuration — from the backup, or re-synced from the repository, and what happens if those disagree about which is newer? In-flight runs at the moment of failure — resumed, failed cleanly, or left in an indeterminate state that requires manual repair?

The last one is worth pressing on. Durable, recoverable state transitions are the difference between a restore that produces a working system and one that produces a system nobody trusts until it has been manually audited.

Multi-tenancy inside your own organisation

Self-hosted does not mean single-team. Within a year, a platform that works will have several teams on it, and the question of how they are separated arrives whether or not it was designed for.

The things to look for: resource ownership that a team can actually administer, scopes that constrain both configuration and execution, and inheritance rules you can reason about. Ask specifically what a team owner can do without a platform administrator, because if the answer is "very little", the platform team becomes a queue.

Ask the inverse too. What can a team owner do that affects other teams? Shared credentials, shared runners and platform-level configuration are the usual coupling points, and knowing where they are is what lets you decide which ones are acceptable.

The cost that is not on the invoice

Self-hosting trades licence cost for operational cost, and the second is easier to underestimate because it arrives distributed across people rather than as a line item.

Count it honestly: infrastructure, upgrades, backup verification, certificate and credential rotation, identity integration maintenance, and someone on call for a platform other teams now depend on. Then count the offsetting side — data residency satisfied without negotiation, no per-seat pricing conversation as usage grows, and no dependency on a vendor's uptime for your incident response to function.

The comparison that matters is against the alternative you would actually build. Teams that reject a platform on cost frequently end up assembling the same capabilities in-house, where the cost is equally real and considerably harder to see because it is spread across projects that each look reasonable on their own.