Decide what happens when the workflow fails

A workflow demonstration is easy when every field is complete and every system responds. Real work starts when one of those assumptions fails.

Start with the blocked job. Use the missing asset reference, the photograph nobody can interpret or the approval nobody has authority to make. Decide who needs to act and keep the evidence with the blockage.

This is exception-path-first design: define abnormal states, decision authority, evidence preservation, containment, escalation and recovery before optimising the normal route.

I would rather put a smaller workflow into use that handles those cases than build a larger one that only behaves in a demonstration.

Map failure before automation

Every workflow is a sequence of controlled state changes. Before automating one, answer the same questions at each important transition:

1. What must be true before this item may advance?
2. Which evidence proves it?
3. Which action is permitted?
4. How can the transition fail?
5. Who owns each failure state?
6. How long may the item remain there?
7. What restores it to a controlled route?

Walk through recent examples where the normal route stopped. Examine them with their original gaps and workarounds still visible.

For each one, record:

• what the workflow expected;
• what was missing or contradictory;
• how the problem became visible;
• who noticed it;
• where the evidence went;
• who was able to decide;
• how the final action returned to the record.

This becomes an exception register. It separates a genuine exception from an unwritten part of the normal process. If the same “exception” happens every day, the workflow has been described incorrectly.

A useful register does not need to be complicated:

• Required photograph missing: the evidence rule holds the submission while retaining everything valid. The field supervisor owns it until the missing item is supplied and checked.
• Source system unavailable: the timeout rule moves the work into a visible queue. The system owner retains responsibility until the source responds and the queued work is reconciled.
• Approval authority absent: the authority rule prevents the transition and escalates the case. The operations manager owns it until an authorised decision is recorded.
• Conflicting status: the reconciliation rule preserves both values. The data owner resolves the disagreement by establishing source authority or correcting the record.

The value is not the table. It is the agreement about what the organisation will do when the normal route is unavailable.

Control the main exception types

Incomplete evidence

The system should identify what is missing and what the person needs to do next. “Invalid submission” is not enough.

Keep the valid evidence already supplied and ask for the failed item, rather than making somebody repeat the whole job.

An unavailable system

Decide whether work stops, queues safely or continues under a controlled offline process. Record which answer was based on cached information and when the source becomes available again.

A retry can help with a brief technical failure. It should not hide a business decision or leave the team unsure whether the job was accepted.

The governing rule is:

Retry uncertainty about availability. Do not blindly retry uncertainty about outcome.

If a request may have succeeded before the connection failed, sending it again may duplicate an order, approval or payment. Amazon’s guidance on idempotent APIs explains how a stable request identifier can make a repeated request return the same result instead of performing the action twice.

Retries should also be bounded. Google’s Site Reliability Engineering guidance warns that aggressive retries can amplify an overloaded service. Use time limits, randomized exponential backoff and a controlled queue or degraded mode.

Conflicting records

If two systems disagree, keep both records, show which source has authority for the fact in question and create work to resolve the disagreement. Arrival order does not confer authority.

Ambiguous evidence

Some evidence cannot be validated by a simple rule. A person may need to interpret a photograph, site note or customer response.

The workflow should present the original evidence, the question being decided and the available actions. Record who made the judgement and what they chose.

Missing authority

An approval step needs more than a button. Define who has authority, what they are approving and what happens when they are unavailable.

Escalation should move the decision to another authorised role. It should not quietly reduce the level of control because a deadline is close.

Stale data

Time-sensitive evidence needs an age limit. If the source is older than that limit, the workflow should stop making the claim that depended on it.

Showing the last known answer can be useful, but only when its age and status are unmistakable.

Work arrives out of sequence

Events do not always arrive in the order the process expects. A completion message may appear before the evidence upload, or an old update may arrive after a correction.

Give each event an identity and time. Define which transitions are legal from the current state. Hold or reject events that cannot be applied safely rather than allowing arrival order to rewrite the operation.

Work completes only in part

A workflow can update one system and fail before updating the next. The user sees one action; the organisation is left with two truths.

Decide which side effects can be reversed, which must be reconciled and who owns the repair. Rolling back application code does not restore a permit, message or payment already created elsewhere.

Choose the right failure response

Not every exception needs the same control.

The response should be chosen from the consequence of being wrong, not from what is easiest to code.

Give every blockage a route

A useful exception record answers six questions:

1. What prevented the normal route?
2. Which evidence supports that conclusion?
3. What is the operational consequence while it remains blocked?
4. Who owns the next action?
5. When must they act?
6. What condition allows the workflow to continue?

Without the final answer, an exception becomes a holding area. Work accumulates because nobody knows what “resolved” means.

The owner should receive the job, the failed rule and the relevant evidence together. Avoid sending a generic notification that forces them to reconstruct the case across other systems.

An exception queue should make the operation visible. Show state, owner, age, consequence and next action. Sort by management consequence and age, not merely by the time the technical error was logged.

Design escalation as authority

Escalation is not the act of sending another notification. It is a controlled transfer of decision authority.

The evidence should stay with the case while authority moves to the next named role. The receiving person needs:

• the decision required;
• the complete evidence package;
• the rule or control that stopped the item;
• the actions they are authorised to take;
• the consequence of delay;
• the time remaining;
• the record of earlier attempts.

Define what happens when the first reviewer does not respond. The next step may be reassignment, a higher authority, a controlled fallback or an explicit stop. It should not be an endless series of reminders.

If the workflow relies on one person being available, that dependency is part of the design and should be tested like any other.

Preserve the decision

Many workflows lose control at the moment a person intervenes. The issue is resolved on a call or in chat, and the system records only that the job moved forward.

Keep:

• the original evidence and its version;
• the automated or rule-based conclusion;
• the exception and state transition;
• the person or service that acted;
• the final decision;
• the reason or correction where it matters;
• the rule or workflow version;
• the time of the action;
• the side effects created in other systems.

Use one stable correlation identifier across the workflow, messages, external calls and exception record. Technical telemetry can then explain which service failed. Workflow telemetry can explain which job stopped, what management consequence followed and whether it recovered.

Missing audit evidence is itself an exception. If the system cannot prove who authorised a consequential transition, the safe answer is not to assume that somebody did.

This is particularly important when AI helps classify, summarise or suggest an action. The model can assist with the evidence. A named person remains responsible for a consequential decision.

The Synapse portfolio record shows approval points, blocked work and human intervention in a working Strataflow system. It is portfolio engineering with representative work, not a client deployment.

Test the awkward version

A pilot should not be accepted because the happy path completes. Test at least:

• a required field is missing;
• a source responds late or not at all;
• a request succeeds but its response is lost;
• two records disagree;
• events arrive twice or out of order;
• an unauthorised person attempts approval;
• the authorised person does not respond;
• evidence exceeds its freshness limit;
• an item is corrected and resubmitted;
• only some external updates complete;
• a person changes the suggested outcome;
• the workflow restarts after interruption.

For each test, confirm the job is not lost, duplicated or allowed to advance without the required decision.

Set pilot controls before the first live case:

• a bounded group of users, work types or locations;
• a maximum acceptable exception backlog and age;
• a stop condition for errors, unsafe states or missing evidence;
• a named person with authority to pause the pilot;
• a rollback plan for code and operational state;
• a reconciliation method for anything already changed elsewhere;
• an agreed recovery test before work resumes.

Rollback is not complete when the old software version is running. The organisation must also know which jobs, approvals, messages and external records were touched, then restore or reconcile them to a controlled state.

The 2012 Knight Capital failure is an extreme example of why containment comes before scale. The US Securities and Exchange Commission reported that the firm sent more than four million orders and lost more than $460 million in 45 minutes after deploying defective trading software without adequate safeguards. The industry is different; the operational lesson is not. A fast automated route needs a fast, tested way to stop it.

Measure the exception path

Normal-route throughput does not tell management whether abnormal work is under control.

Track a small set of measures that lead to action:

Do not reward a team simply for closing exceptions quickly. A case can be closed by bypassing the control that created it. Review the evidence, decision and resulting state.

The objective is not zero exceptions. A workflow that reports none may simply be hiding them. The objective is visible, owned and recoverable abnormal work.

Keep the first release narrow

Choose one workflow with a known owner and a clear result. Put the exception route into use with the people who handle the real cases. Measure whether the intervention shortens the blockage, improves evidence or removes a repeated manual reconciliation.

Add volume only when the team can answer:

• Which abnormal states can occur?
• Who has authority in each one?
• Where is the evidence kept?
• How does the work return to a controlled route?
• What stops the workflow when its assumptions fail?
• How will external state be reconciled after recovery?

That is the real acceptance test. The normal route proves the workflow can run. The exception path proves the organisation can operate it.

The Workflow Automation Pilot is designed around one production workflow and an agreed test for success. If the process itself is still unclear, Operational Improvement should come first.