Skip to content

Blog

Deployment failures should leave a recovery path

A failed deployment should explain where it stopped, what already happened, which actions are safe, and what verified target can be restored.

AppaloftDeployment controlRecoveryRollbackAI

When a deployment fails, the easiest message for a platform to return is: Deployment failed.

It is not wrong. It is just not very useful. Did the failure happen while creating the plan, preparing a resource, starting the runtime, running a health check, or reading back the final URL? Did the earlier stages complete? Will retrying create the same resources again? Should the operator change configuration, redeploy, or roll back to the last verified version?

Appaloft treats deployment as a control-plane path shared by the CLI, Web Console, GitHub workflows, and AI agents. That model requires successful deployments to leave clear evidence. It also requires failed deployments to leave enough facts for a safe next step. Failure is not the end of the workflow, and it should not be just a red patch on a log page.

This post follows From GitHub Actions to Appaloft: what evidence a deployment should leave behind, Install progress should not be a pile of disconnected logs, and What a deploy URL should promise. The question here is narrower: what should one failed deployment leave behind so that someone can recover it?

“Failed” is not a diagnosis

A deployment is rarely one action. It may include project and environment selection, configuration parsing, artifact preparation, resource binding, runtime execution, health checks, and external reachability checks.

If every error collapses into failed, the platform loses the context that makes the error actionable. The user sees the same red state, but the response is different in each case:

  • a missing configuration value needs input before retrying;
  • a server that is not ready needs time or infrastructure work;
  • a running image with a failed health check needs runtime diagnosis;
  • a healthy deployment with an unreachable URL needs route or DNS investigation;
  • a new version that is unhealthy may need to return to the previous verified version.

All of these can be called a failed deployment. They should not share the same recovery hint. An error should answer three questions: where did it stop, what already happened, and which action is currently allowed?

Preserve the facts before cleaning up

After a failure, the first useful action is not always to delete everything. The platform needs to preserve enough evidence for someone to understand and classify what happened.

At minimum, the deployment should be associated with:

  • a source revision, such as a commit SHA or source snapshot;
  • the artifact or image digest actually used;
  • the target project, environment, resource, and server;
  • a redacted configuration summary, not secret values;
  • dependencies that were created, reused, or not created;
  • runtime, health-check, and HTTP readback results;
  • the failed stage, structured error, and timestamp;
  • the previous verified deployment, or an explicit manual recovery gap.

That is why deployment provenance cannot live only in a GitHub Actions log. CI logs are good for recording a process. The control plane needs to preserve a deployment state that still makes sense later. A week from now, an operator should not have to reconstruct production state from a hundred lines of shell output.

Preserving facts does not mean storing every raw response. Provider tokens, environment values, database URLs, and unredacted request bodies should not end up in deployment details. Recovery needs evidence, and it needs a clear redaction boundary.

Retry, redeploy, and rollback are different actions

Failure pages often contain one button: “Retry.” The underlying meanings are different.

Retry is for a transient failure or a stage that can be repeated safely, such as waiting for readiness, reading state again, or continuing an operation with a clear idempotency boundary. It should not silently recreate every dependency.

Redeploy starts a new attempt with new input. The operator may have changed an environment value, selected a different source revision, or chosen another target. The new attempt should be recorded as a new deployment, not overwrite the failed one.

Rollback moves a mutable entrypoint back to a previous verified target. It needs to know which artifact, configuration snapshot, and route state to restore, and it needs evidence that the target is still available. Without an explicit target, the platform should not guess with latest or “whatever looked successful last time.”

In short:

Action Problem it solves Evidence it needs
Retry Can this attempt safely continue? Current stage, idempotency boundary, remaining state
Redeploy Can we publish new input? New source, config summary, target environment
Rollback Can we restore a verified target? Digest, config snapshot, route readback

When an action does not meet those conditions, the product should say that it is unavailable instead of turning a risky operation into a convenient button.

AI agents must be even stricter about success

The most dangerous failure behavior for an AI agent is not returning an error. It is describing partial success as complete success.

An uploaded image is not proof that the runtime is ready. A container returning 200 is not proof that a custom domain has completed TLS. A provider accepting a request is not proof that the final route readback passed.

After deployment, an agent should read structured status, health, diagnostics, and recovery readiness before deciding whether to continue, retry, ask for more input, or stop. It can say:

The artifact was created and the runtime started, but HTTP readback has not passed. Check the route target; there is not enough evidence to claim the deployment is complete.

It should not say “it is live” merely because it received a URL. As What a deploy URL should promise argues, a URL is part of the deployment contract. An immutable evidence URL, a mutable alias, a custom domain, and a temporarily blocked route make different promises.

Recoverability belongs in product state

Users do not need every internal detail, but they do need the platform’s judgment about the next step:

  • keep waiting: the platform is still performing checks without new side effects;
  • change and retry: required input is missing or invalid;
  • redeploy: this attempt is over and needs new source or target input;
  • roll back: the new version failed while a verified target remains available;
  • handle manually: the system lacks enough evidence to recover safely.

That is more useful than “check the logs,” and safer than retrying forever. Logs still matter, but they should support an organized state rather than force the user to infer the state from raw output.

The sign of a mature deployment system is not that it never fails. That is not a realistic standard. A better one is this: after a failure, people can tell what happened, what already exists, which action will not widen the blast radius, and how to return to a verified state.

For Appaloft, the deployment control plane is not a button that stays green forever. It is a readable path through success, failure, and recovery.