The missing piece was downstream reactions:

• a read model that stays current when orders change
• search index updates without polling
• cross-service notifications that don’t add writes inside the request

Marten and EventStoreDB are proper event stores. I like both in the right system.

But this app didn’t need a new persistence model. I wanted a smaller move: keep PostgreSQL as the source of truth and listen to its write-ahead log.

That is what this sample does. Debezium reads the WAL, Kafka carries the change events, and a .NET consumer maps the raw envelope into OrderCreated, OrderUpdated, OrderDeleted output. Practical CDC for an existing relational app, not a replacement for event sourcing.

You can access the entire code from my GitHub Repo

How It Fits Together

PostgreSQL WAL -> Debezium Connect -> Apache Kafka -> .NET consumer

PostgreSQL already writes every change to its WAL for crash recovery. Setting wal_level=logical tells Postgres to include enough detail for logical replication. Debezium connects as a replication client, reads those changes, wraps them in a before/after envelope, and publishes to a Kafka topic named after the table.

The app that writes orders does not publish anything. It only writes to PostgreSQL. CDC sits beside it.

The Docker Setup

Four containers: PostgreSQL with logical replication, Kafka in KRaft mode, Debezium Connect, and Kafka UI for debugging.

Every image is free and open-source. Kafka uses the official Apache image (Apache 2.0), not the Confluent distribution. Works identically with Podman or Rancher Desktop.

services:
  postgres:
    image: postgres:16-alpine
    container_name: cdc-postgres
    ports:
      - "5432:5432"
    environment:
      POSTGRES_DB: orders
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    command: >
      postgres
      -c wal_level=logical
      -c max_replication_slots=4
      -c max_wal_senders=4
    volumes:
      - ./setup.sql:/docker-entrypoint-initdb.d/setup.sql

  kafka:
    image: apache/kafka:3.7.0
    container_name: cdc-kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  debezium:
    image: debezium/connect:2.5
    container_name: cdc-debezium
    depends_on:
      kafka:
        condition: service_started
      postgres:
        condition: service_started
    ports:
      - "8083:8083"
    environment:
      BOOTSTRAP_SERVERS: kafka:29092
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: debezium_configs
      OFFSET_STORAGE_TOPIC: debezium_offsets
      STATUS_STORAGE_TOPIC: debezium_statuses
      CONFIG_STORAGE_REPLICATION_FACTOR: 1
      OFFSET_STORAGE_REPLICATION_FACTOR: 1
      STATUS_STORAGE_REPLICATION_FACTOR: 1

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: cdc-kafka-ui
    depends_on:
      - kafka
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
      KAFKA_CLUSTERS_0_KAFKACONNECT_0_NAME: debezium
      KAFKA_CLUSTERS_0_KAFKACONNECT_0_ADDRESS: http://debezium:8083

The command override on PostgreSQL is the key part. Without wal_level=logical, Debezium cannot connect. max_replication_slots=4 and max_wal_senders=4 give enough room for the connector plus any other replication you might add later.

The setup.sql file (mounted into the init directory) creates the orders table and seeds three rows:

CREATE TABLE IF NOT EXISTS orders (
    id SERIAL PRIMARY KEY,
    customer TEXT NOT NULL,
    product TEXT NOT NULL,
    quantity INT NOT NULL DEFAULT 1,
    total_amount DECIMAL(10,2) NOT NULL,
    status TEXT NOT NULL DEFAULT 'pending',
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

INSERT INTO orders (customer, product, quantity, total_amount, status) VALUES
    ('alice', 'Widget A', 2, 49.98, 'pending'),
    ('bob', 'Widget B', 1, 24.99, 'confirmed'),
    ('carol', 'Widget C', 5, 124.95, 'shipped');

ALTER TABLE orders REPLICA IDENTITY FULL;

One detail matters for updates and deletes:

ALTER TABLE orders REPLICA IDENTITY FULL;

PostgreSQL’s default replica identity only sends the primary key for update/delete before-images. Debezium can’t show previous status, customer, or total_amount without it.

FULL includes the entire previous row. Small line, big difference: -> shipped vs pending -> shipped.

Start everything:

docker-compose up -d

Wait for Kafka and Debezium to stabilize, then register the connector.

Registering the Debezium Connector

Debezium Connect exposes a REST API. You POST a connector config:

curl.exe -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d @register-connector.json

On macOS/Linux, curl is fine. On Windows PowerShell, use curl.exe so PowerShell does not route the command through its Invoke-WebRequest alias.

The connector config tells Debezium which database to watch and which tables to capture:

{
  "name": "orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres",
    "database.dbname": "orders",
    "topic.prefix": "orders",
    "table.include.list": "public.orders",
    "plugin.name": "pgoutput",
    "slot.name": "orders_slot",
    "publication.name": "orders_pub",
    "decimal.handling.mode": "string",
    "schema.history.internal.kafka.bootstrap.servers": "kafka:29092",
    "schema.history.internal.kafka.topic": "schema-changes"
  }
}

decimal.handling.mode: string tells Debezium to serialize NUMERIC/DECIMAL columns as plain strings instead of base64-encoded bytes. Without this, a DECIMAL(10,2) value of 49.98 arrives as "E4Y=" and your consumer has to decode it manually. With string, it arrives as "49.98" and System.Text.Json handles the rest.

topic.prefix combined with the schema and table name gives you the Kafka topic: orders.public.orders. Once registered, Debezium starts streaming. You can verify in Kafka UI at http://localhost:8080.

What Debezium Events Look Like

Every CDC event has a payload with before, after, and op fields. For an INSERT:

{
  "payload": {
    "before": null,
    "after": {
      "id": 4,
      "customer": "dave",
      "product": "Widget D",
      "quantity": 3,
      "total_amount": "74.97",
      "status": "pending"
    },
    "op": "c",
    "ts_ms": 1713600000000
  }
}

op: "c" means create, "u" means update, "d" means delete, and "r" means read (the initial snapshot). For updates, both before and after are populated so you can see exactly what changed.

The .NET Consumer

A console app using Confluent.Kafka. The deserializer maps the Debezium envelope into domain event records.

The envelope type:

using System.Text.Json;
using System.Text.Json.Serialization;

public sealed class DebeziumEnvelope
{
    [JsonPropertyName("before")]
    public JsonElement? Before { get; set; }

    [JsonPropertyName("after")]
    public JsonElement? After { get; set; }

    [JsonPropertyName("op")]
    public string Operation { get; set; } = string.Empty;

    [JsonPropertyName("ts_ms")]
    public long TimestampMs { get; set; }

    public bool IsCreate => Operation == "c" || Operation == "r";
    public bool IsUpdate => Operation == "u";
    public bool IsDelete => Operation == "d";
}

public sealed class DebeziumMessage
{
    [JsonPropertyName("payload")]
    public DebeziumEnvelope? Payload { get; set; }
}

The domain events:

public record OrderCreated(
    int Id,
    string Customer,
    string Product,
    int Quantity,
    decimal TotalAmount,
    string Status,
    DateTimeOffset CreatedAt);

public record OrderUpdated(
    int Id,
    string? PreviousStatus,
    string CurrentStatus,
    string Customer,
    string Product,
    decimal TotalAmount,
    DateTimeOffset UpdatedAt);

And the consumer loop in Program.cs:

using System.Text.Json;
using System.Text.Json.Serialization;
using AnimatLabs.CdcEventSourcing;
using AnimatLabs.CdcEventSourcing.DomainEvents;
using Confluent.Kafka;

var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "cdc-consumer",
    AutoOffsetReset = AutoOffsetReset.Earliest,
    EnableAutoCommit = true
};

var topic = "orders.public.orders";
using var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe(topic);

Console.WriteLine($"Listening on {topic}. Insert or update rows in the orders table to see events.");
Console.WriteLine("Press Ctrl+C to stop.");
Console.WriteLine();

var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };

try
{
    while (!cts.Token.IsCancellationRequested)
    {
        var result = consumer.Consume(cts.Token);
        if (result?.Message?.Value is null) continue;

        var message = JsonSerializer.Deserialize<DebeziumMessage>(result.Message.Value);
        var envelope = message?.Payload;
        if (envelope is null) continue;

        if (envelope.IsCreate)
        {
            var row = envelope.After?.Deserialize<OrderRow>();
            if (row is null) continue;

            var created = new OrderCreated(
                row.Id, row.Customer, row.Product,
                row.Quantity, row.TotalAmount, row.Status,
                DateTimeOffset.FromUnixTimeMilliseconds(envelope.TimestampMs));

            Console.WriteLine($"[OrderCreated] #{created.Id} {created.Customer} " +
                $"bought {created.Quantity}x {created.Product} for {created.TotalAmount:C}");
        }
        else if (envelope.IsUpdate)
        {
            var before = envelope.Before?.Deserialize<OrderRow>();
            var after = envelope.After?.Deserialize<OrderRow>();
            if (after is null) continue;

            var updated = new OrderUpdated(
                after.Id, before?.Status, after.Status,
                after.Customer, after.Product, after.TotalAmount,
                DateTimeOffset.FromUnixTimeMilliseconds(envelope.TimestampMs));

            Console.WriteLine($"[OrderUpdated] #{updated.Id} " +
                $"{updated.PreviousStatus} -> {updated.CurrentStatus} ({updated.Customer})");
        }
        else if (envelope.IsDelete)
        {
            var row = envelope.Before?.Deserialize<OrderRow>();
            Console.WriteLine($"[OrderDeleted] #{row?.Id} {row?.Customer}");
        }
    }
}
catch (OperationCanceledException) { }
finally { consumer.Close(); }

The OrderRow maps the JSON payload to a C# object. Debezium sends numeric columns as strings (because of decimal.handling.mode: string), so JsonNumberHandling.AllowReadingFromString handles the conversion:

[JsonNumberHandling(JsonNumberHandling.AllowReadingFromString)]
internal sealed class OrderRow
{
    [JsonPropertyName("id")]
    public int Id { get; set; }

    [JsonPropertyName("customer")]
    public string Customer { get; set; } = "";

    [JsonPropertyName("product")]
    public string Product { get; set; } = "";

    [JsonPropertyName("quantity")]
    public int Quantity { get; set; }

    [JsonPropertyName("total_amount")]
    public decimal TotalAmount { get; set; }

    [JsonPropertyName("status")]
    public string Status { get; set; } = "";
}

Run the consumer:

cd AnimatLabs.CdcEventSourcing
dotnet run

It immediately reads the initial snapshot:

Listening on orders.public.orders. Insert or update rows in the orders table to see events.
Press Ctrl+C to stop.

[OrderCreated] #1 alice bought 2x Widget A for $49.98
[OrderCreated] #2 bob bought 1x Widget B for $24.99
[OrderCreated] #3 carol bought 5x Widget C for $124.95

Open another terminal and insert a row:

docker exec cdc-postgres psql -U postgres -d orders -c \
  "INSERT INTO orders (customer, product, quantity, total_amount) VALUES ('dave', 'Widget D', 3, 74.97);"

Update it:

docker exec cdc-postgres psql -U postgres -d orders -c \
  "UPDATE orders SET status = 'shipped' WHERE customer = 'dave';"

Delete it:

docker exec cdc-postgres psql -U postgres -d orders -c \
  "DELETE FROM orders WHERE customer = 'dave';"

This is the verified output from my local run:

[OrderCreated] #4 dave bought 3x Widget D for $74.97
[OrderUpdated] #4 pending -> shipped (dave)
[OrderDeleted] #4 dave

What This Gives You

The same database write can now feed a few separate jobs:

• Keep raw events in Kafka for audit and replay
• Evict Redis entries when the source row changes instead of guessing TTLs
• Feed a search index or read model from the same topic

These are all independent consumers on the same Kafka topic. Add them as you need them. The database does not care.

What I Would Not Use This For

I would not use CDC as a shortcut around domain modeling. If your system needs event-sourced aggregates, explicit commands, versioned domain events, and business-time replay, start with a real event store or a framework designed for that model.

CDC is strongest when:

• the relational schema already exists
• other services need to react to committed changes
• you want the application write path to stay simple
• eventual consistency is acceptable

It is weaker when consumers need perfect domain intent. A row update can tell you that status changed from pending to shipped; it cannot tell you whether that happened because a warehouse scan completed, a support agent overrode the order, or a migration script fixed old data. If that distinction matters, publish an explicit domain event.

Things to Watch For

Connector lag. If Debezium falls behind, changes pile up in the WAL. Watch replication slot lag in PostgreSQL and the Debezium metrics endpoint.

Replica identity. If you need previous row values for updates/deletes, set REPLICA IDENTITY FULL on the captured table. Otherwise delete events may only include the primary key.

Schema changes. Adding a column is usually fine. Renaming or removing columns can break consumers. Version the events you expose from your consumer if other teams depend on them.

At-least-once delivery. Kafka consumers can see duplicates. Make handlers idempotent with the row primary key, Debezium metadata, or your own processed-event table.

Startup order. Kafka and Debezium take a few seconds to settle. The README keeps the commands separate on purpose so you can see each moving part.

A follow-up post covers piping these same CDC events to the browser over SignalR and SSE. Same Kafka topic, different consumer.

HTMX has an SSE extension that opens an EventSource from attributes. The server sends HTML fragments, HTMX swaps them into the page. I paired it with WorkflowForge 2.1.1 for a five-step order workflow with automatic compensation. Zero custom JavaScript.

You can access the entire code from my GitHub Repo

If you’re new to SSE in .NET, I wrote about the three SSE patterns from scratch in the previous post.

The Flow

Two endpoints, one flow:

Browser                                 Server
  │                                        │
  ├─ hx-get="/workflow/reset" ────────────>│ Returns HTML fragment
  │                                        │   with sse-connect attribute
  │<────── <div sse-connect="/stream"> ────┤
  │                                        │
  ├─ HTMX opens EventSource ─────────────>│ WorkflowForge runs 5 ops
  │                                        │   Channel pipes events
  │<──────── event: step ──────────────────┤
  │<──────── event: step ──────────────────┤
  │<──────── event: done ──────────────────┤

Button click fetches an HTML fragment with SSE attributes. HTMX opens the stream and the server pushes step updates as HTML. Click “Run with Failure” and ChargePayment throws. WorkflowForge then runs compensations in reverse order for every step that had completed up to the failure—including ChargePayment’s own RestoreAsync (which logs that there is no charge to reverse when the gateway never completed), then ReserveStock, then ValidateOrder - all streamed live.

The HTML

The entire interactive UI:

<div>
    <button hx-get="/workflow/reset"
            hx-target="#output"
            hx-swap="innerHTML">Run Order Workflow</button>
    <button hx-get="/workflow/reset?fail=true"
            hx-target="#output"
            hx-swap="innerHTML">Run with Failure</button>
</div>

<div id="output">
    <p>Click a button to start a workflow.</p>
</div>

No onclick. No EventSource in JavaScript. The <head> loads HTMX and the SSE extension. hx-ext="sse" goes on the <body> tag. Everything else is server-driven.

The Server Side

The reset endpoint returns an HTML fragment pre-wired for SSE. When HTMX inserts this into the page, it sees the sse-connect attribute and opens an EventSource to the stream URL:

app.MapGet("/workflow/reset", (bool fail = false) =>
{
    var html = $"""
        <div sse-connect="/workflow/stream?fail={fail.ToString().ToLowerInvariant()}" sse-close="close">
            <div id="steps" sse-swap="step" hx-swap="beforeend"></div>
            <div id="final-status" sse-swap="done" hx-swap="innerHTML"></div>
        </div>
        """;
    return Results.Content(html, "text/html");
});

sse-swap="step" means “when an SSE event named step arrives, swap this div.” hx-swap="beforeend" appends each step instead of replacing. sse-close="close" shuts down the connection when the workflow finishes.

The stream endpoint runs the workflow on a background task and pushes events through a Channel:

app.MapGet("/workflow/stream", async (HttpContext ctx, bool fail = false) =>
{
    var bufferingFeature = ctx.Features.Get<IHttpResponseBodyFeature>();
    bufferingFeature?.DisableBuffering();

    ctx.Response.ContentType = "text/event-stream";
    ctx.Response.Headers.CacheControl = "no-cache";
    ctx.Response.Headers.Connection = "keep-alive";
    ctx.Response.Headers["X-Accel-Buffering"] = "no";

    var sink = new ChannelEventSink();
    var ct = ctx.RequestAborted;
    var workflow = OrderProcessingWorkflow.Build(sink, fail);

    using var foundry = WF.CreateFoundry(
        workflowName: workflow.Name,
        initialProperties: new Dictionary<string, object?>
        {
            [OrderKeys.ShouldFail] = fail
        });

    using var smith = WF.CreateSmith(new ConsoleLogger("WF"));
    string? finalHtml = null;

    try
    {
        _ = Task.Run(async () =>
        {
            try
            {
                await smith.ForgeAsync(workflow, foundry, ct).ConfigureAwait(false);
                finalHtml = "<p><strong>All steps completed successfully.</strong></p>";
            }
            catch (OperationCanceledException) { }
            catch
            {
                finalHtml = "<p><strong>Workflow failed -- compensation complete.</strong></p>";
            }
            finally
            {
                sink.Complete();
            }
        }, ct);

        await foreach (var evt in sink.Reader.ReadAllAsync(ct))
        {
            var html = BuildStepHtml(evt);
            await SendSseAsync(ctx, "step", html, ct).ConfigureAwait(false);
        }

        if (finalHtml is not null)
        {
            await SendSseAsync(ctx, "done", finalHtml, ct).ConfigureAwait(false);
            await SendSseAsync(ctx, "close", "", ct).ConfigureAwait(false);
        }
    }
    catch (OperationCanceledException) { }
});

The close event triggers sse-close on the client, shutting down the EventSource so it doesn’t auto-reconnect.

Bridging WorkflowForge to SSE

System.Threading.Channels connects the two. Workflow operations write events, the SSE loop reads them:

public sealed class ChannelEventSink : IWorkflowEventSink
{
    private readonly Channel<WorkflowEvent> _channel =
        Channel.CreateUnbounded<WorkflowEvent>(new UnboundedChannelOptions
        {
            SingleReader = true,
            SingleWriter = false
        });

    public ChannelReader<WorkflowEvent> Reader => _channel.Reader;

    public void Report(string operationName, string status, string detail)
    {
        _channel.Writer.TryWrite(new WorkflowEvent
        {
            OperationName = operationName,
            Status = status,
            Detail = detail
        });
    }

    public void Complete() => _channel.Writer.TryComplete();
}

SingleWriter = false because multiple operations can report concurrently. The reader is the SSE loop: one consumer, serialized writes to the response stream.

The Workflow and Compensation

Five operations with WorkflowForge 2.1.1:

public static IWorkflow Build(IWorkflowEventSink sink, bool shouldFail)
{
    return WF
        .CreateWorkflow("OrderProcessing")
        .AddOperation(new ValidateOrderOperation(sink))
        .AddOperation(new ReserveStockOperation(sink))
        .AddOperation(new ChargePaymentOperation(sink, shouldFail))
        .AddOperation(new CreateShipmentOperation(sink))
        .AddOperation(new SendNotificationOperation(sink))
        .Build();
}

Each operation implements ForgeAsyncCore (do the work) and RestoreAsync (undo it). When any step throws, WorkflowForge walks backward and invokes RestoreAsync for each completed step in reverse order (including the step that failed, when it has cleanup or a no-op path). The ChargePaymentOperation has a shouldFail flag that simulates a payment gateway timeout:

protected override async Task<object?> ForgeAsyncCore(
    object? inputData, IWorkflowFoundry foundry, CancellationToken ct)
{
    sink.Report(Name, "running", "Charging payment method...");
    await Task.Delay(1200, ct).ConfigureAwait(false);

    if (shouldFail)
    {
        sink.Report(Name, "failed", "Payment gateway timeout -- triggering compensation");
        throw new InvalidOperationException("Payment gateway timeout");
    }

    foundry.SetProperty(OrderKeys.PaymentCharged, true);
    sink.Report(Name, "completed", "$149.99 charged to card ending 4242");
    return inputData;
}

In RestoreAsync, it checks if a charge actually happened via GetPropertyOrDefault. No charge means no refund (the kind of detail that breaks things in production if you skip it).

Each step event becomes plain HTML:

static string BuildStepHtml(WorkflowEvent evt)
{
    var label = evt.Status.ToUpperInvariant();
    return $"""<div class="step">[{label}] <strong>{evt.OperationName}</strong> {WebUtility.HtmlEncode(evt.Detail)}</div>""";
}

No client-side rendering. The browser inserts HTML fragments directly.

Where This Fits

HTMX + SSE works well for admin dashboards, status monitors, internal tools: anything where the server owns the state and the client displays it. The browser handles reconnection.

Chat or collaborative editing? Wrong tool. WebSockets or SignalR when you need traffic both ways, which is a longer story than this write-up has room for but you already know when you need it.

To try it:

cd playground/WorkflowForge/AnimatLabs.WorkflowForge.HtmxDashboard
dotnet run

Open http://localhost:5075. Two buttons: one happy path, one failure. On failure, compensations run newest-first across all completed steps (including ChargePayment’s rollback hook), and you can see each step stream in.

If WorkflowForge has been useful to you, a ⭐ star on GitHub helps it reach more .NET developers. And if you'd like to support the work behind it, Ko-fi is always open!

⭐ Star on GitHub Support on Ko-fi

More on This Topic

• Server-Sent Events in ASP.NET Core
• WorkflowForge introduction
• MassTransit saga with WorkflowForge

You can access the entire code from my GitHub Repo

What SSE Actually Is

HTTP. Set Content-Type: text/event-stream, keep the connection open, write data: something\n\n. Each double-newline ends an event.

The browser’s EventSource API parses it. That’s the whole spec.

No WebSocket upgrade. Proxies treat it like a long-lived HTTP response; most already allow it. EventSource reconnects automatically and sends Last-Event-ID so the server knows where to resume.

Three optional fields: data (payload), event (name for multiple types on one stream), id (for reconnection).

If your UI only needs the server to talk and the browser to listen, you skip the WebSocket handshake, you skip hub abstractions, and you stay on plain HTTP semantics that ops already know how to load-balance and cache-policy, which is why I reach for SSE first on internal dashboards.

The Simplest Stream: A Clock

Start with the bare minimum: one event type, one value, fire every second:

app.MapGet("/events/clock", async (HttpContext ctx) =>
{
    ctx.Response.ContentType = "text/event-stream";
    ctx.Response.Headers.CacheControl = "no-cache";
    ctx.Response.Headers.Connection = "keep-alive";

    var ct = ctx.RequestAborted;

    await foreach (var tick in StreamClock(ct))
    {
        await ctx.Response.WriteAsync($"data: {tick:HH:mm:ss}\n\n", ct);
        await ctx.Response.Body.FlushAsync(ct);
    }
});

static async IAsyncEnumerable<DateTime> StreamClock(
    [EnumeratorCancellation] CancellationToken ct)
{
    while (!ct.IsCancellationRequested)
    {
        yield return DateTime.UtcNow;
        await Task.Delay(1000, ct);
    }
}

IAsyncEnumerable keeps the stream lazy. Each tick becomes a data: line, \n\n ends the event. Flush after each write or the client sees nothing until the buffer fills (I forgot this the first time and stared at a blank page for five minutes).

On the browser side, EventSource is native. No library:

const clock = new EventSource('/events/clock');
clock.onmessage = e => {
    document.getElementById('clock').textContent = e.data;
};
clock.onopen = () => document.getElementById('clock-status').classList.remove('off');
clock.onerror = () => document.getElementById('clock-status').classList.add('off');

Named Events

What if you need multiple event types on one connection? The event: field handles that. Order stream example: it pushes both placed and cancelled events.

app.MapGet("/events/orders", async (HttpContext ctx) =>
{
    ctx.Response.ContentType = "text/event-stream";
    ctx.Response.Headers.CacheControl = "no-cache";
    ctx.Response.Headers.Connection = "keep-alive";

    var ct = ctx.RequestAborted;
    var orderNum = 1000;

    while (!ct.IsCancellationRequested)
    {
        await Task.Delay(Random.Shared.Next(1500, 4000), ct);

        orderNum++;
        var amount = Math.Round(Random.Shared.NextDouble() * 500 + 10, 2);
        var status = Random.Shared.Next(10) < 8 ? "placed" : "cancelled";

        await ctx.Response.WriteAsync($"event: {status}\n", ct);
        await ctx.Response.WriteAsync($"data: Order #{orderNum} — ${amount}\n\n", ct);
        await ctx.Response.Body.FlushAsync(ct);
    }
});

The event: line goes before data:. On the client, you use addEventListener instead of onmessage. That’s the key difference. onmessage only fires for unnamed events:

const orders = new EventSource('/events/orders');
orders.addEventListener('placed', e => addOrderLine(e.data, 'placed'));
orders.addEventListener('cancelled', e => addOrderLine(e.data, 'cancelled'));

One connection, multiple handlers.

Reconnection with Event IDs

The server sends id: with each event. Connection drops, the browser reconnects and sends Last-Event-ID in the header. Server picks up where it left off. No custom retry logic.

app.MapGet("/events/metrics", async (HttpContext ctx) =>
{
    ctx.Response.ContentType = "text/event-stream";
    ctx.Response.Headers.CacheControl = "no-cache";
    ctx.Response.Headers.Connection = "keep-alive";

    var ct = ctx.RequestAborted;
    var lastId = 0;

    if (ctx.Request.Headers.TryGetValue("Last-Event-ID", out var lastEventId)
        && int.TryParse(lastEventId, out var parsed))
    {
        lastId = parsed;
    }

    var seq = lastId;
    while (!ct.IsCancellationRequested)
    {
        await Task.Delay(2000, ct);
        seq++;

        var cpu = Math.Round(Random.Shared.NextDouble() * 60 + 10, 1);
        var mem = Random.Shared.Next(40, 85);

        await ctx.Response.WriteAsync($"id: {seq}\n", ct);
        await ctx.Response.WriteAsync($"data: {{\"cpu\":{cpu},\"mem\":{mem},\"seq\":{seq}}}\n\n", ct);
        await ctx.Response.Body.FlushAsync(ct);
    }
});

Client side is just JSON parsing:

const metrics = new EventSource('/events/metrics');
metrics.onmessage = e => {
    const d = JSON.parse(e.data);
    document.getElementById('cpu').textContent = d.cpu + '%';
    document.getElementById('mem').textContent = d.mem + '%';
    document.getElementById('seq').textContent = '#' + d.seq;
};

The browser handles the reconnection loop and sends Last-Event-ID automatically. You don’t write that code.

When to Use What

Dashboards, notifications, live logs, progress bars? Server-Sent Events. Chat, collaborative editing, anything where clients push back heavily? SignalR. If you need binary frames or a custom wire protocol, drop to WebSockets.

Really comes down to data direction and what your infra team will allow through the proxy.

Gotchas

HTTP/1.1 connection limits. Browsers cap connections per domain at ~6. Open 6 SSE streams and your next fetch queues.

HTTP/2 multiplexes over one connection, so the problem disappears. If you’re on HTTP/1.1, keep streams under the limit.

Text only. No binary. Base64 if you must, but at that point you probably want WebSockets.

The playground has all three patterns running on http://localhost:5074:

cd playground/ServerSentEvents/AnimatLabs.ServerSentEvents
dotnet run

No NuGet packages, no hub classes: just a GET endpoint and a loop. Full code and setup in the playground README.

Follow-up piece: SSE plus HTMX for a workflow dashboard where the server pushes HTML fragments. No custom JavaScript. I wanted that article to exist mostly so I’d stop re-explaining EventSource to myself every six months.

Related Reading

• HTMX dashboard in .NET
• Reactive programming with System.Reactive

Couldn’t find one I liked. Built this instead. MassTransit handles message routing, WorkflowForge handles the rollback logic. (I’ve sat through enough saga talks where the speaker waved at a box labeled “compensate” and moved on. This is the opposite of that.)

You can access the entire code from my GitHub Repo

Prerequisites

You need .NET 8 and the WorkflowForge solution. The demo runs InMemory transport by default; no RabbitMQ required.

Want RabbitMQ instead? The project has a docker-compose.yml:

cd playground/WorkflowForge/AnimatLabs.WorkflowForge.MassTransitSaga.OrderService
docker-compose up -d

That starts RabbitMQ on port 5672 with the management UI on 15672 (guest/guest). Swap UsingInMemory for UsingRabbitMq in Program.cs and you’re set.

The Problem

You’ve got an order flow: reserve stock, charge payment, create shipment. If any step fails, you undo the previous ones. Saga pattern.

In real life those steps spread across buses, databases, and vendors that all fail in creatively annoying ways at 2 a.m., but this demo stays in one process so you can watch compensation without blaming the network.

The hard part (the part most tutorials skip) is the compensation logic. When a payment gateway times out, who releases the stock? Who refunds the charge? How do you keep that logic in one place instead of scattered across consumers?

The Split: MassTransit + WorkflowForge

MassTransit distributes the messages. WorkflowForge 2.1.1 compensates the failures. One library does pub/sub. The other does the orchestration and rollback.

The flow:

SubmitOrder → ReserveStock → ChargePayment → CreateShipment → OrderAccepted
                                  ↓ (failure)
                            ChargePayment.Compensate → ReserveStock.Compensate → OrderFailed

Orders over $500 simulate a payment gateway timeout. When that happens, WorkflowForge runs the compensation in reverse order: refund payment, then release stock.

This Is a Single-Service Demo

This demo runs in one process (OrderService). No separate Inventory, Payments, or Shipping services.

The step-level events (ReserveStock, ChargePayment, CreateShipment) are published to the bus but no consumers handle them. They’re fire-and-forget; the workflow steps do the work directly and publish for visibility or future use.

In a real system you’d add consumers in separate services. For this demo, the goal is to show the compensation pattern without the extra moving parts.

Messages

Contracts are plain records. No shared state, only events.

namespace AnimatLabs.WorkflowForge.Workflows.Sample.OrderSaga.Contracts;

public record SubmitOrder(Guid OrderId, string CustomerEmail, decimal Amount);

public record OrderAccepted(Guid OrderId);
public record OrderFailed(Guid OrderId, string Reason);

public record ReserveStock(Guid OrderId, int Quantity);
public record ChargePayment(Guid OrderId, decimal Amount);
public record CreateShipment(Guid OrderId, string CustomerEmail);

These live in the shared Workflows.Sample library so any execution project can reference them. The file also defines StockReserved, PaymentFailed, and similar; those are there for a future multi-service setup. In this demo the workflow steps publish ReserveStock, ChargePayment, CreateShipment to the bus, but nothing consumes them yet.

The Consumer

OrderSubmittedConsumer receives the message, builds a workflow, and runs it. The key: WF.CreateFoundry holds the saga state (OrderId, Amount, CustomerEmail). WF.CreateSmith runs the workflow.

public sealed class OrderSubmittedConsumer(IBus bus, ILogger<OrderSubmittedConsumer> logger) : IConsumer<SubmitOrder>
{
    public async Task Consume(ConsumeContext<SubmitOrder> context)
    {
        var order = context.Message;
        logger.LogInformation("Received order {OrderId} for ${Amount}", order.OrderId, order.Amount);

        var shouldFail = order.Amount > 500;
        var massTransitBus = new MassTransitBus(bus);
        var workflow = OrderSagaWorkflow.Build(massTransitBus, shouldFail);

        using var foundry = WF.CreateFoundry(
            workflowName: workflow.Name,
            initialProperties: new Dictionary<string, object?>
            {
                [SagaKeys.OrderId] = order.OrderId,
                [SagaKeys.Amount] = order.Amount,
                [SagaKeys.CustomerEmail] = order.CustomerEmail
            });

        using var smith = WF.CreateSmith(new ConsoleLogger("WF-Saga"));

        try
        {
            await smith.ForgeAsync(workflow, foundry, context.CancellationToken).ConfigureAwait(false);
            await context.Publish(new OrderAccepted(order.OrderId)).ConfigureAwait(false);
            logger.LogInformation("Order {OrderId} completed successfully", order.OrderId);
        }
        catch (Exception ex)
        {
            await context.Publish(new OrderFailed(order.OrderId, ex.Message)).ConfigureAwait(false);
            logger.LogError(ex, "Order {OrderId} failed -- compensation executed", order.OrderId);
        }
    }
}

Workflow steps depend on IMessageBus (async publish). The OrderService project adapts MassTransit’s IBus with a thin wrapper:

public interface IMessageBus
{
    Task PublishAsync<T>(T message, CancellationToken ct) where T : class;
}

public sealed class MassTransitBus(IBus bus) : IMessageBus
{
    public Task PublishAsync<T>(T message, CancellationToken ct) where T : class
        => bus.Publish(message, ct);
}

IMessageBus lives in the shared Workflows.Sample library; MassTransitBus lives next to the consumer in AnimatLabs.WorkflowForge.MassTransitSaga.OrderService.

The Workflow

Each step extends WorkflowOperationBase. Forge does the work. Restore does the rollback.

public static class OrderSagaWorkflow
{
    public static IWorkflow Build(IMessageBus bus, bool simulatePaymentFailure = false)
    {
        return WF
            .CreateWorkflow("OrderSaga")
            .AddOperation(new ReserveStockStep(bus))
            .AddOperation(new ChargePaymentStep(bus, simulatePaymentFailure))
            .AddOperation(new CreateShipmentStep(bus))
            .Build();
    }
}

The Steps

public sealed class ReserveStockStep(IMessageBus bus) : WorkflowOperationBase
{
    public override string Name => "ReserveStock";

    protected override async Task<object?> ForgeAsyncCore(
        object? inputData, IWorkflowFoundry foundry, CancellationToken ct)
    {
        var orderId = foundry.GetPropertyOrDefault<Guid>(SagaKeys.OrderId);
        foundry.Logger.LogInformation("[ReserveStock] Reserving stock for order {OrderId}", orderId);

        await bus.PublishAsync(new ReserveStock(orderId, 1), ct).ConfigureAwait(false);
        await Task.Delay(500, ct).ConfigureAwait(false);

        foundry.Logger.LogInformation("[ReserveStock] Stock reserved for order {OrderId}", orderId);
        return inputData;
    }

    public override async Task RestoreAsync(
        object? outputData, IWorkflowFoundry foundry, CancellationToken ct)
    {
        var orderId = foundry.GetPropertyOrDefault<Guid>(SagaKeys.OrderId);
        foundry.Logger.LogWarning("[ReserveStock] COMPENSATING: Releasing stock for order {OrderId}", orderId);
        await Task.Delay(300, ct).ConfigureAwait(false);
        foundry.Logger.LogWarning("[ReserveStock] Stock released for order {OrderId}", orderId);
    }
}

public sealed class ChargePaymentStep(IMessageBus bus, bool simulateFailure) : WorkflowOperationBase
{
    public override string Name => "ChargePayment";

    protected override async Task<object?> ForgeAsyncCore(
        object? inputData, IWorkflowFoundry foundry, CancellationToken ct)
    {
        var orderId = foundry.GetPropertyOrDefault<Guid>(SagaKeys.OrderId);
        var amount = foundry.GetPropertyOrDefault<decimal>(SagaKeys.Amount);
        foundry.Logger.LogInformation("[ChargePayment] Charging ${Amount} for order {OrderId}", amount, orderId);

        await bus.PublishAsync(new ChargePayment(orderId, amount), ct).ConfigureAwait(false);
        await Task.Delay(800, ct).ConfigureAwait(false);

        if (simulateFailure)
        {
            foundry.Logger.LogError("[ChargePayment] Payment gateway timeout for order {OrderId}", orderId);
            throw new InvalidOperationException($"Payment gateway timeout for order {orderId}");
        }

        var txId = $"TXN-{Random.Shared.Next(10000, 99999)}";
        foundry.SetProperty(SagaKeys.TransactionId, txId);
        foundry.Logger.LogInformation("[ChargePayment] Payment {TransactionId} charged for order {OrderId}", txId, orderId);

        return inputData;
    }

    public override async Task RestoreAsync(
        object? outputData, IWorkflowFoundry foundry, CancellationToken ct)
    {
        var orderId = foundry.GetPropertyOrDefault<Guid>(SagaKeys.OrderId);
        var txId = foundry.GetPropertyOrDefault<string>(SagaKeys.TransactionId);

        if (!string.IsNullOrEmpty(txId))
        {
            foundry.Logger.LogWarning("[ChargePayment] COMPENSATING: Refunding {TransactionId} for order {OrderId}", txId, orderId);
            await Task.Delay(500, ct).ConfigureAwait(false);
            foundry.Logger.LogWarning("[ChargePayment] Refund issued for {TransactionId}", txId);
        }
        else
        {
            foundry.Logger.LogInformation("[ChargePayment] No charge to reverse for order {OrderId}", orderId);
        }
    }
}

ChargePaymentStep.RestoreAsync checks TransactionId before refunding. If payment never succeeded, there’s nothing to reverse. That’s the kind of detail that breaks production sagas.

Wiring It Up

InMemory transport means no RabbitMQ needed. Run it, watch two orders: $99 succeeds, $999 fails and triggers compensation.

builder.Services.AddMassTransit(cfg =>
{
    cfg.AddConsumer<OrderSubmittedConsumer>();

    cfg.UsingInMemory((context, inmem) =>
    {
        inmem.ConfigureEndpoints(context);
    });
});

// Fire a test order after the bus starts
_ = Task.Run(async () =>
{
    await Task.Delay(2000);
    var bus = host.Services.GetRequiredService<IBus>();

    await bus.Publish(new SubmitOrder(Guid.NewGuid(), "happy@example.com", 99.99m));
    await Task.Delay(5000);

    await bus.Publish(new SubmitOrder(Guid.NewGuid(), "sad@example.com", 999.99m));
});

What I Learned Building This

The part I like most about this approach: compensation logic lives in one place: the RestoreAsync methods. No scattered event handlers, no “if payment failed then fire ReleaseStock” across multiple consumers. WorkflowForge runs the compensation cascade automatically when any step throws.

For RabbitMQ, swap UsingInMemory for UsingRabbitMq in Program.cs. Same code, same workflow, different transport. The project has a docker-compose.yml with RabbitMQ ready to go if you want to test it.

To try it out:

cd playground/WorkflowForge
dotnet run --project AnimatLabs.WorkflowForge.MassTransitSaga.OrderService

The app auto-submits two orders. Watch the logs for the $999 failure and the compensation cascade running in reverse.

If WorkflowForge has been useful to you, a ⭐ star on GitHub helps it reach more .NET developers. And if you'd like to support the work behind it, Ko-fi is always open!

⭐ Star on GitHub Support on Ko-fi

See Also

• WorkflowForge introduction
• WorkflowForge with Coravel
• HTMX dashboard in .NET
• Polly v8 resilience patterns

Five .NET services, a React frontend, Redis, and PostgreSQL. I had a text file. Service A on 5000. Service B on 5001. Frontend on 3000.

Someone would clone the repo, run dotnet run, and Service A would crash because their machine already had something on 5000. Every. Single. Time.

I added instructions to the README: “Change the port in launchSettings.json.” Nobody read them. I changed the ports. Someone else changed them back. I gave up.

Traefik fixed all of this. Every service runs on the same internal port, and routing happens by hostname instead of port number.

You can access the entire code from my GitHub Repo

The docker-compose

services:
  traefik:
    image: traefik:v3.6
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entryPoints.web.address=:80"
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - traefik-net

  api-service:
    build: ./ApiService
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.api.rule=Host(`api.localhost`)"
      - "traefik.http.routers.api.entrypoints=web"
      - "traefik.http.services.api.loadbalancer.server.port=8080"
    networks:
      - traefik-net

  web-service:
    build: ./WebService
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.web.rule=Host(`web.localhost`)"
      - "traefik.http.routers.web.entrypoints=web"
      - "traefik.http.services.web.loadbalancer.server.port=8080"
    networks:
      - traefik-net

networks:
  traefik-net:
    driver: bridge

That’s the full thing. Traefik plus two .NET services. docker-compose up --build and you get:

• http://api.localhost (API)
• http://web.localhost (Web frontend)
• http://localhost:8080 (Traefik dashboard)

Neither service publishes a port to the host. Third service? Add another block with a different hostname. Traefik picks it up. You don’t touch any ports.

All images are free and open-source (Traefik is MIT). The compose commands work with Docker, Podman, or Rancher Desktop.

The Dockerfile

Both services share the same Dockerfile pattern. ApiService/Dockerfile:

FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS base
WORKDIR /app
EXPOSE 8080

FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build
WORKDIR /src
COPY ["ApiService.csproj", "."]
RUN dotnet restore
COPY . .
RUN dotnet build -c Release -o /app/build

FROM build AS publish
RUN dotnet publish -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "ApiService.dll"]

WebService/Dockerfile is identical (swap ApiService for WebService). Both expose 8080.

How Traefik Routes

Three concepts: entrypoints (ports Traefik listens on), routers (rules that match requests), and services (the backends that handle them).

In our compose, web is an entrypoint on port 80. The router rule Host(api.localhost) matches the HTTP Host header. And loadbalancer.server.port=8080 tells Traefik which port inside the container to hit.

So http://api.localhost/orders goes to Traefik on port 80, Host header matches, Traefik forwards to api-service:8080. No port published on the host. No coordination lookup.

You can combine matchers too: Host(api.localhost) && PathPrefix(/v2) for path-based routing on top of hostnames.

HTTPS With Let’s Encrypt

For staging or production, Traefik handles certs. You don’t install certbot, you don’t set up cron jobs, you don’t think about renewal.

services:
  traefik:
    image: traefik:v3.6
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entryPoints.web.address=:80"
      - "--entryPoints.websecure.address=:443"
      - "--certificatesresolvers.le.acme.httpchallenge=true"
      - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.le.acme.email=you@example.com"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--entryPoints.web.http.redirections.entrypoint.to=websecure"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./letsencrypt:/letsencrypt

And the service labels:

labels:
  - "traefik.http.routers.api.rule=Host(`api.yourdomain.com`)"
  - "traefik.http.routers.api.entrypoints=websecure"
  - "traefik.http.routers.api.tls.certresolver=le"

Issues the cert. Renews before expiry. Redirects HTTP to HTTPS. I set this up once six months ago and haven’t touched it since.

Debugging

I hit all of these at least once.

The 502 I burned the most time on was a port mismatch. My loadbalancer.server.port label said 5000 but the container was listening on 8080. Dashboard at localhost:8080 showed the router in red. Obvious in hindsight. Check docker ps, verify the container is alive, then look at the dashboard.

The 404-on-everything problem? Missing traefik.enable=true. We set exposedbydefault=false in the compose (you should), which means every service needs that label. I missed it on a third service and spent twenty minutes reading Traefik docs before I noticed.

Then there’s the backtick thing:

# Broken
- "traefik.http.routers.api.rule=Host('api.localhost')"

# Works
- "traefik.http.routers.api.rule=Host(`api.localhost`)"

Single quotes vs backticks. The error message doesn’t help. You just have to know.

One more: if service A calls service B, skip Traefik entirely. Use the compose service name and internal port:

var client = new HttpClient { BaseAddress = new Uri("http://api-service:8080") };

That stays inside the Docker network.

How I Actually Use It

I don’t run everything in Docker during development. Too slow for hot reload. What I do: start Traefik once with docker-compose up -d traefik, then run individual services with dotnet watch. When I need the full containerized setup (integration testing, demo), docker-compose up --build brings everything up.

Staging gets TLS and health checks. Production adds rate limiting and access logs on top. Same compose file, different labels. Haven’t changed the core config in months.

The playground has two .NET services behind Traefik:

cd playground/TraefikDotNet
docker-compose up --build

Open http://api.localhost, http://web.localhost, and http://localhost:8080 (dashboard). Both services run on 8080 internally. No port conflicts.

Playground: TraefikDotNet

Related

• WSL2 on Windows
• Docker on WSL2 with Visual Studio
• Shipping a quality .NET OSS release

Polly v8 does this properly. I want to walk through how I set it up, what the defaults give you, and where I’ve had to go beyond them. Retries without jitter are basically a distributed denial of service you aimed at yourself, which is a mouthful, but it is also the fastest way to turn a brownout into an outage when every pod wakes up and slams the same downstream at the same millisecond.

Start Here

Microsoft.Extensions.Http.Resilience gives you a pre-configured pipeline with one line:

dotnet add package Microsoft.Extensions.Http.Resilience

builder.Services.AddHttpClient("MyApi")
    .AddStandardResilienceHandler();

That gets you exponential backoff with jitter, a circuit breaker, and a timeout. Sensible defaults. For most HTTP clients in a typical web app, this is enough. Honestly, I wish more teams would start here instead of building custom retry logic from scratch.

If this covers your case, stop reading. What follows is for the other 20%.

Going Beyond the Defaults

I reach for custom Polly pipelines when the one-liner doesn’t fit. In my experience that’s usually one of these situations:

• A payment gateway needs 2 retries with a 5-second timeout while an analytics endpoint is fine with 5 retries and 30 seconds
• Database calls, message queues, file I/O. Anything that isn’t IHttpClientFactory
• The SLA is tight enough (say 500ms) that the default circuit breaker settings are too generous
• I need OnRetry or OnOpened callbacks wired into our structured logging

Retries

If something fails, try again. Simple idea, easy to get wrong. Naive retries with fixed delays can turn a struggling service into a dead one because every instance retries at the same interval and you get a thundering herd.

The fix is jitter. UseJitter = true adds randomness to the backoff so retries spread out instead of hitting the downstream service in waves. I’ve seen this make the difference between a service recovering on its own and a full cascading failure.

var retryPipeline = new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        Delay = TimeSpan.FromSeconds(1),
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,
        OnRetry = args =>
        {
            Console.WriteLine($"Retry {args.AttemptNumber} after {args.RetryDelay}");
            return ValueTask.CompletedTask;
        }
    })
    .Build();

var result = await retryPipeline.ExecuteAsync(async ct =>
{
    return await _httpClient.GetAsync("/api/data", ct);
});

One thing to watch: don’t retry non-idempotent operations without safeguards. Retrying a payment charge can double-bill someone. Retrying a 400 is pointless because it’ll fail the same way every time.

Circuit Breaker

When a downstream service is unhealthy, retrying just delays the inevitable. Circuit breakers flip the approach: once failures cross a threshold, stop trying altogether and fail fast.

The circuit has three states. Closed (normal flow), Open (requests fail immediately), and Half-Open (a few test requests probe whether the service recovered). Pretty standard pattern, but there’s a setting most people miss:

var circuitBreakerPipeline = new ResiliencePipelineBuilder()
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions
    {
        FailureRatio = 0.5,
        SamplingDuration = TimeSpan.FromSeconds(10),
        MinimumThroughput = 8,
        BreakDuration = TimeSpan.FromSeconds(30),
        OnOpened = args =>
        {
            Console.WriteLine("Circuit opened - failing fast");
            return ValueTask.CompletedTask;
        },
        OnClosed = args =>
        {
            Console.WriteLine("Circuit closed - back to normal");
            return ValueTask.CompletedTask;
        }
    })
    .Build();

MinimumThroughput is the one. Without it, a single failure during low-traffic hours (say, 1 request in 10 seconds) would open the circuit because 1/1 = 100% failure. I set it to 8 so the circuit only evaluates after enough requests to be meaningful.

Timeouts

This is the one that bit us the hardest. Without timeouts, a slow downstream service silently consumes your connection pool and thread pool until the whole app stalls. No errors, no exceptions. Just increasing latency and then everything stops.

var timeoutPipeline = new ResiliencePipelineBuilder()
    .AddTimeout(new TimeoutStrategyOptions
    {
        Timeout = TimeSpan.FromSeconds(10),
        OnTimeout = args =>
        {
            Console.WriteLine($"Timeout after {args.Timeout}");
            return ValueTask.CompletedTask;
        }
    })
    .Build();

I use two levels: an overall timeout for the entire operation (including retries) and a per-attempt timeout for individual calls. A common mistake is setting the timeout to 30 seconds when the SLA requires a 2-second response. By the time it fires, you’ve already violated the SLA.

Putting It Together

Order matters when you compose strategies. They apply outer to inner:

var resiliencePipeline = new ResiliencePipelineBuilder()
    .AddTimeout(TimeSpan.FromSeconds(30))
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        Delay = TimeSpan.FromSeconds(1),
        BackoffType = DelayBackoffType.Exponential
    })
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions
    {
        FailureRatio = 0.5,
        BreakDuration = TimeSpan.FromSeconds(30)
    })
    .AddTimeout(TimeSpan.FromSeconds(5))
    .Build();

So each HTTP call gets 5 seconds max. Failures retry up to 3 times with exponential backoff. If 50% of calls fail within the sampling window, the circuit opens for 30 seconds. And the whole thing wraps in a 30-second overall timeout.

Per-Client Configuration

When AddStandardResilienceHandler() isn’t enough, AddResilienceHandler gives you full control per named client:

builder.Services.AddHttpClient("PaymentGateway")
    .AddResilienceHandler("strict", builder =>
    {
        builder
            .AddTimeout(TimeSpan.FromSeconds(5))
            .AddRetry(new HttpRetryStrategyOptions
            {
                MaxRetryAttempts = 2,
                BackoffType = DelayBackoffType.Exponential
            })
            .AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
            {
                FailureRatio = 0.3,
                SamplingDuration = TimeSpan.FromSeconds(10),
                BreakDuration = TimeSpan.FromSeconds(30)
            });
    });

builder.Services.AddHttpClient("AnalyticsApi")
    .AddStandardResilienceHandler();

Payment gateway gets tight timeouts, fewer retries, and an aggressive circuit breaker. Analytics API is fine with defaults. HttpRetryStrategyOptions and HttpCircuitBreakerStrategyOptions handle transient HTTP errors (5xx, network failures) so you don’t have to specify which exceptions to catch.

WorkflowForge Integration

If you’re using WorkflowForge for workflow orchestration, the Polly extension adds resilience as middleware on the foundry:

// dotnet add package WorkflowForge.Extensions.Resilience.Polly
using WorkflowForge.Extensions.Resilience.Polly;

using var foundry = WorkflowForge.CreateFoundry("OrderProcessing");
foundry.UsePollyRetry(maxRetryAttempts: 3, baseDelay: TimeSpan.FromSeconds(1));

Or go all-in with retry, circuit breaker, and timeout together:

foundry.UsePollyComprehensive(
    maxRetryAttempts: 3,
    circuitBreakerThreshold: 5,
    timeoutDuration: TimeSpan.FromSeconds(30));

You can also wrap individual operations instead of the entire foundry:

var resilientOp = PollyRetryOperation.WithRetryPolicy(
    new ActionWorkflowOperation("CallApi", async (input, foundry, ct) =>
    {
        // call external API
    }),
    maxRetryAttempts: 3,
    baseDelay: TimeSpan.FromSeconds(1));

What I Actually Run in Production

The patterns above are what I’ve settled on after running these in production for a while. Circuit breakers on every external HTTP client, structured logging in OnRetry and OnOpened so we actually know when things degrade, and separate configs for critical vs best-effort services.

The snippets above are standalone. Copy them into any .NET 8+ project with dotnet add package Polly (or Microsoft.Extensions.Http.Resilience for the IHttpClientFactory integration). For WorkflowForge, add WorkflowForge.Extensions.Resilience.Polly.

If WorkflowForge has been useful to you, a ⭐ star on GitHub helps it reach more .NET developers. And if you'd like to support the work behind it, Ko-fi is always open!

⭐ Star on GitHub Support on Ko-fi

More on This Topic

• Redis distributed locking in .NET
• Refit API clients
• MassTransit saga with WorkflowForge

WorkflowForge 2.0 shipped from my laptop. dotnet pack, dotnet nuget push, done. Thirteen packages on NuGet.org, no CI to speak of.

For the 2.1 release (sixty issues on a feature branch, thirteen packages targeting .NET Framework 4.8, .NET 8.0, and .NET 10.0), I wanted to do it properly. Static analysis. Automated testing across all three frameworks. Supply chain attestation. A software bill of materials. Gated publish with human approval.

What follows is every piece of infrastructure that went into that transition. Not a changelog, not a feature announcement. Just the tooling, the configuration, and the places where things broke in ways I did not expect. Some of those breaks were embarrassing.

If you maintain a .NET open-source project and ship to NuGet, this is the checklist I wish I had before I started.

You can access the entire code from my GitHub Repo

Static Analysis: Free for Public Repos

SonarCloud is free for any public GitHub repository. There is no reason not to use it.

Setting it up requires a Java runtime for the scanner, a project token stored as a GitHub Actions secret, and three commands wrapping the build:

- name: Install SonarScanner for .NET
  run: dotnet tool install --global dotnet-sonarscanner

- name: Build and Test with SonarCloud
  run: |
    dotnet sonarscanner begin \
      /k:"animatlabs_workflow-forge" \
      /o:"animatlabs" \
      /d:sonar.token="$SONAR_TOKEN" \
      /d:sonar.host.url="https://sonarcloud.io" \
      /d:sonar.cs.opencover.reportsPaths="**/TestResults/**/coverage.opencover.xml"
    dotnet build WorkflowForge.sln --configuration Release
    dotnet test WorkflowForge.sln --configuration Release \
      --collect:"XPlat Code Coverage" \
      -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover
    dotnet sonarscanner end /d:sonar.token="$SONAR_TOKEN"

The scanner wraps the build, collects coverage data, and uploads results. First run on the WorkflowForge codebase: quality gate passed, 88.9% coverage on new code, 2.2% duplication. But it also flagged dozens of issues I had not noticed.

What It Found

The most common finding was structured logging violations:

// Looks innocent. Breaks structured logging.
_logger.LogInformation($"Processing order {orderId}");

// Template + args: OrderId is a real field in Seq, App Insights, etc.
_logger.LogInformation("Processing order {OrderId}", orderId);

The interpolated version compiles and runs. The log output looks identical. But orderId gets baked into the message string instead of being a separate structured field. In production, you cannot filter or aggregate by order ID. SonarCloud flagged every instance across the codebase.

It also caught missing sealed modifiers on classes that were not designed for inheritance, inconsistent visibility on internal constants, and unused event declarations in test doubles. Legitimate code smells that manual review had missed.

What It Did Not Find

The hardest bugs in this release required reasoning about concurrency, shared state, and disposal patterns. No static analyzer handles those well today.

Thread safety in conditional operations:

// Shared bool without a memory barrier
private bool _lastConditionResult;

// volatile so compensation does not read another run's stale flag
private volatile bool _lastConditionResult;

ConditionalWorkflowOperation evaluates a condition during execution and reads the result during compensation. Without volatile, one workflow’s compensation could read another workflow’s stale condition result. The fix is one keyword, but finding the bug requires understanding the concurrent execution model.

O(n^2) indexing in persistence middleware:

// O(n) scan on every middleware hop
var index = workflow.Operations.ToList().IndexOf(currentOperation);

// Read the index the foundry already tracks
private static int GetCurrentOperationIndex(IWorkflowFoundry foundry)
{
    if (foundry.Properties.TryGetValue(
        FoundryPropertyKeys.CurrentOperationIndex, out var obj) && obj is int idx)
        return idx;
    // ...fallback counter logic
}

The foundry now sets CurrentOperationIndex before each operation executes. The middleware reads it directly instead of scanning.

Allocation on every property access:

// New ReadOnlyCollection every time someone touches Operations
internal IReadOnlyList<IWorkflowOperation> Operations =>
    new ReadOnlyCollection<IWorkflowOperation>(_operations);

// Cache until the underlying list changes
private ReadOnlyCollection<IWorkflowOperation>? _cachedOperations;

internal IReadOnlyList<IWorkflowOperation> Operations =>
    _cachedOperations ??= new ReadOnlyCollection<IWorkflowOperation>(_operations);

In hot paths that inspect the operation list, the old version created a new ReadOnlyCollection wrapper on every access. The fix caches it and clears the cache when operations are added.

Event handler memory leak:

// Dispose() never dropped event subscriptions
public void Dispose()
{
    if (_disposed) return;
    _disposed = true;
    _concurrencyLimiter?.Dispose();
}

// ...same, but clear delegates so GC can collect peers
public void Dispose()
{
    if (_disposed) return;
    _disposed = true;

    _concurrencyLimiter?.Dispose();

    WorkflowStarted = null;
    WorkflowCompleted = null;
    WorkflowFailed = null;
    CompensationTriggered = null;
    CompensationCompleted = null;
    // ...all event fields set to null
}

WorkflowSmith and WorkflowFoundry subscribed to each other’s events but never unsubscribed on dispose. In long-running applications, disposed objects stayed alive through event handler references.

Silent recovery failures:

The recovery extension caught exceptions during resume and returned successfully, silently swallowing errors. The persistence middleware overwrote restored operation outputs with input data during the resume path, undoing the point of checkpointing. Both required reading the code paths carefully and writing tests for the failure scenarios.

SonarCloud earned its keep. It is also not enough. The badge is not a substitute for reading your own code.

Multi-Target Testing

WorkflowForge targets .NET Framework 4.8, .NET 8.0, and .NET 10.0. Each framework has different runtime behavior, and the differences can be subtle.

The CI pipeline runs the full test suite against all three:

dotnet test WorkflowForge.sln --framework net8.0 \
  --collect:"XPlat Code Coverage"
dotnet test WorkflowForge.sln --framework net10.0 \
  --collect:"XPlat Code Coverage"
dotnet test WorkflowForge.sln --framework net48

Coverage is collected on .NET 8.0 and 10.0 (OpenCover format for SonarCloud ingestion). .NET Framework 4.8 runs the tests but without coverage instrumentation, since the XPlat collector does not support it.

This caught real issues. .NET Framework 4.8 enforces strong-name validation strictly. If assembly A is signed and references unsigned assembly B, the runtime throws a FileLoadException. .NET Core and later ignore strong names entirely. Without multi-TFM testing, the benchmark suite worked on .NET 8.0 and 10.0 but failed on 4.8 due to a SignAssembly inheritance issue in Directory.Build.props.

Coverage reports are uploaded as separate artifacts for independent auditing, alongside the .trx test result files.

The Pipeline

This is the full CI/CD workflow that runs on every push, every PR, and on-demand for publish:

┌──────────────────────────────────────────────────────────────────┐
│  Build Job                                                       │
│                                                                  │
│  Checkout ─► Setup SDKs ─► Restore ─► SonarScanner Begin        │
│  ─► Build ─► Test (net8.0) ─► Test (net10.0) ─► Test (net48)   │
│  ─► SonarScanner End ─► Upload Results                          │
│  ─► Pack ─► Generate SBOM ─► Upload Packages                   │
└──────────────────────┬───────────────────────────────────────────┘
                       │ (requires manual approval)
┌──────────────────────▼───────────────────────────────────────────┐
│  Publish Job  (nuget-publish environment)                        │
│                                                                  │
│  Download Packages ─► Sign (optional) ─► Attest Provenance      │
│  ─► Attest SBOM ─► Push to NuGet.org                            │
└──────────────────────────────────────────────────────────────────┘

A few design decisions worth explaining.

Environment gate. The publish job runs in a nuget-publish GitHub Environment that requires manual approval. No accidental publishes. The concurrency group prevents parallel publish attempts to the same branch:

publish:
  needs: build
  environment: nuget-publish
  concurrency:
    group: publish-${{ github.ref }}
    cancel-in-progress: false

Minimal permissions. The top-level permissions: {} drops all default GitHub token permissions. Each job requests only what it needs: contents: read for the build, id-token: write and attestations: write for the publish.

The full workflow is on GitHub.

Supply Chain Hardening

Four things happen before any package reaches NuGet.org.

SHA-Pinned Actions

Every GitHub Action in the workflow is pinned to a full commit SHA instead of a mutable version tag:

- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6
- uses: actions/setup-dotnet@c2fa09f4bde5ebb9d1777cf28262a3eb3db3ced7  # v5
- uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32  # v4.1.0

Tags like v6 can be moved to point to a different commit at any time. A compromised action author could push malicious code under the same tag. SHAs are immutable. If the hash does not match, the step fails.

The tradeoff is maintenance: when actions release new versions, you need to update the hash. Dependabot handles this automatically with weekly update PRs for both GitHub Actions and NuGet dependencies.

One gotcha: precision matters. During the 2.1 publish, the attest-build-provenance step failed because the SHA had a single-character typo (ecc instead of ecd). The entire pipeline stopped. There is no “close enough” with SHA-pinning.

Sigstore Build Attestation

Every .nupkg, .snupkg, and SBOM file gets a cryptographic attestation before reaching NuGet.org:

- name: Attest NuGet Package Provenance
  uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32  # v4.1.0
  with:
    subject-path: './packages/*.nupkg'

- name: Attest Symbol Package Provenance
  uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32  # v4.1.0
  with:
    subject-path: './packages/*.snupkg'

- name: Attest SBOM
  uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32  # v4.1.0
  with:
    subject-path: './packages/bom.json'

This proves, cryptographically, that each artifact was built by a specific GitHub Actions workflow, from a specific commit, in a specific repository. Consumers can verify with gh attestation verify. No long-lived signing keys to manage. GitHub’s Sigstore integration uses short-lived OIDC tokens.

CycloneDX SBOM

One command generates a complete dependency manifest in CycloneDX JSON format:

- name: Generate CycloneDX SBOM
  run: |
    dotnet tool install --global CycloneDX
    dotnet cyclonedx WorkflowForge.sln -o ./packages --json

The resulting bom.json lists every dependency, its version, and its license. Increasingly required by enterprise consumers doing compliance reviews. The SBOM is uploaded alongside the packages and gets its own build attestation.

NuGet Vulnerability Auditing

In Directory.Build.props, three properties make every dotnet restore an audit:

<NuGetAudit>true</NuGetAudit>
<NuGetAuditMode>all</NuGetAuditMode>
<NuGetAuditLevel>low</NuGetAuditLevel>

all checks both direct and transitive dependencies. low means any known vulnerability at any severity level fails the build. No silent CVEs in the dependency tree.

NuGet Packaging: The DebugType Trap

This is the part that broke.

Directory.Build.props centralizes build settings for the entire solution. Two of those settings handle debug symbols and NuGet symbol packages:

<DebugType>portable</DebugType>
<IncludeSymbols>true</IncludeSymbols>
<SymbolPackageFormat>snupkg</SymbolPackageFormat>

portable produces a separate .pdb file alongside the DLL. snupkg packages that .pdb for NuGet.org’s symbol server, enabling step-through debugging for consumers.

The problem: nine of thirteen project .csproj files also had this line:

<DebugType>embedded</DebugType>

embedded bakes the PDB directly into the DLL. No separate .pdb file. The .csproj setting overrides Directory.Build.props, so dotnet pack generated .snupkg files containing nothing. NuGet.org rejected them:

BadRequest https://www.nuget.org/api/v2/symbolpackage/ 77ms
error: Response status code does not indicate success:
  400 (The package does not contain any symbol (.pdb) files.)

Nine of thirteen packages failed. The four that succeeded happened not to have the DebugType=embedded override.

Fix: remove DebugType=embedded from every individual .csproj and let all projects inherit portable from Directory.Build.props. One property, nine projects, fourteen files.

The consequence was not. NuGet.org does not allow re-pushing the same package version. The thirteen .nupkg files had already been accepted. So we had to bump every project to 2.1.1, update version references across documentation and CI, add a CHANGELOG entry, and deprecate the thirteen partially-published 2.1.0 packages individually.

If you are setting up NuGet packaging for a multi-project solution: check your DebugType before your first publish.

SourceLink and Symbol Packages

Getting debug symbols right required centralizing several properties in Directory.Build.props:

<DebugType>portable</DebugType>
<EmbedUntrackedSources>true</EmbedUntrackedSources>
<PublishRepositoryUrl>true</PublishRepositoryUrl>
<IncludeSymbols>true</IncludeSymbols>
<SymbolPackageFormat>snupkg</SymbolPackageFormat>

Plus a SourceLink package reference:

<PackageReference Include="Microsoft.SourceLink.GitHub" Version="8.0.0" PrivateAssets="All"/>

Together, these give consumers of the NuGet package the ability to step into the source code directly from their debugger. EmbedUntrackedSources ensures generated files are included. PublishRepositoryUrl embeds the repository URL in the package metadata.

The key lesson: these properties must live in one place. The moment individual .csproj files start overriding DebugType, the symbol package pipeline breaks silently (dotnet pack does not warn you that the .snupkg is empty).

What Actually Shipped

Beyond the infrastructure, three user-facing changes made it into 2.1.

Inline compensation: attach a restore delegate directly to AddOperation instead of writing a separate class:

var workflow = WorkflowForge.CreateWorkflow("OrderProcessing")
    .AddOperation("ProcessPayment", async (foundry, ct) =>
    {
        foundry.Properties["payment_id"] = "PAY-123";
    },
    restoreAction: async (foundry, ct) =>
    {
        var paymentId = foundry.Properties["payment_id"];
        // Issue refund...
    })
    .Build();

For workflows where the compensation logic is simple, this eliminates an entire class per operation.

GetOperationOutput: inspect what any completed operation returned, by name or index, from the orchestrator level.

Multi-target test validation: all tests run across .NET Framework 4.8, .NET 8.0, and .NET 10.0 on every CI build. API compatibility issues get caught before they ship.

The Checklist

If I were setting up a new .NET OSS project today, this is what I would add before the first NuGet publish:

• Static analysis: SonarCloud or equivalent, free for public repos
• Code coverage with quality gate (coverage on new code, not just overall)
• Multi-target testing across every framework you ship
• CI/CD pipeline with environment-gated publish and manual approval
• SHA-pinned GitHub Actions with Dependabot for automated updates
• Build provenance attestation via Sigstore
• SBOM generation (CycloneDX or SPDX)
• NuGet vulnerability auditing (NuGetAudit=true with all mode)
• SourceLink + symbol packages (check DebugType before first publish)
• SDK version pinning via global.json
• Strong-name signing with key in Directory.Build.props

None of this is individually complex. The complexity is in getting all of it working together without one setting silently breaking another. That is what took sixty issues to sort out.

Resources

If WorkflowForge has been useful to you, a ⭐ star on GitHub helps it reach more .NET developers. And if you'd like to support the work behind it, Ko-fi is always open!

⭐ Star on GitHub Support on Ko-fi

Related Reading

• WorkflowForge 2.0 benchmarks
• Traefik with .NET Docker services
• Polly v8 resilience patterns

You need to run a multi-step business process on a schedule: nightly order reconciliation, hourly data syncs, daily report generation. The typical approach? Hangfire for scheduling, custom code for the workflow logic, and manual error handling for each step. That’s three concerns tangled together, plus a Redis or SQL dependency just for scheduling.

What if you could have:

• Zero external dependencies for scheduling (no Redis, no SQL)
• Automatic compensation if any step fails (saga pattern, built-in)
• Microsecond execution instead of milliseconds
• Clean separation between “when to run” and “what to run”

That’s exactly what happens when you pair Coravel (scheduling) with WorkflowForge (workflow orchestration). I like that split. Took me a while to stop stuffing everything into one scheduler-shaped lump.

I’ve built a complete runnable sample that demonstrates this pattern end-to-end:

You can access the entire code from my GitHub Repo

The Architecture

┌─────────────────────────────────────────────────────────┐
│                  .NET Console Host                      │
├─────────────────────────────────────────────────────────┤
│  Coravel Scheduler                                      │
│  └─ "Run ReconciliationJob every N seconds"             │
│       │                                                 │
│       ▼                                                 │
│  WorkflowForge Workflow                                 │
│  └─ Step 1: Fetch unprocessed orders                    │
│  └─ Step 2: Process payments    ← (refund on failure)   │
│  └─ Step 3: Reserve inventory   ← (release on failure)  │
│  └─ Step 4: [MaybeFail]         ← toggle to demo saga   │
│  └─ Step 5: Send confirmations                          │
└─────────────────────────────────────────────────────────┘

Coravel decides when to run. WorkflowForge handles what to run, including automatic rollback of completed steps when something fails downstream.

Solution Layout

The sample splits concerns cleanly across two projects:

WorkflowForge/
├── AnimatLabs.WorkflowForge.Workflows.Sample/        # Business logic
│   └── NightlyReconciliation/
│       ├── Workflow.cs                                # Workflow builder
│       ├── ReconciliationKeys.cs                      # Shared state keys
│       ├── Models/
│       │   ├── Order.cs
│       │   └── PaymentTransaction.cs
│       ├── Operations/
│       │   ├── FetchUnprocessedOrdersOperation.cs
│       │   ├── ProcessPaymentsOperation.cs
│       │   ├── UpdateInventoryOperation.cs
│       │   ├── MaybeFailOperation.cs
│       │   └── SendConfirmationEmailsOperation.cs
│       └── Services/                                  # Interfaces only
│           ├── IOrderRepository.cs
│           ├── IPaymentService.cs
│           ├── IInventoryService.cs
│           └── IEmailSender.cs
├── AnimatLabs.WorkflowForge.CoravelScheduledWorkflows/  # Host + fakes
│   ├── Program.cs
│   ├── appsettings.json
│   ├── Jobs/ReconciliationJob.cs
│   ├── Options/ReconciliationJobOptions.cs
│   └── Services/
│       ├── InMemoryOrderRepository.cs
│       ├── FakePaymentService.cs
│       ├── FakeInventoryService.cs
│       └── FakeEmailSender.cs
└── AnimatLabs.WorkflowForge.sln

The workflows project contains only business logic: operations, models, and service interfaces. It has a single dependency: WorkflowForge.

The host project wires Coravel scheduling, provides fake service implementations for demo purposes, and depends on Coravel, WorkflowForge, and Microsoft.Extensions.Hosting.

Setup: Two NuGet Packages

dotnet add package Coravel
dotnet add package WorkflowForge

No database migrations. No connection strings for the scheduler.

The Workflow Definition

The entire workflow is defined in one method. Chain the operations and build:

using AnimatLabs.WorkflowForge.Workflows.Sample.NightlyReconciliation.Operations;
using AnimatLabs.WorkflowForge.Workflows.Sample.NightlyReconciliation.Services;
using WorkflowForge.Abstractions;
using WF = WorkflowForge.WorkflowForge;

namespace AnimatLabs.WorkflowForge.Workflows.Sample.NightlyReconciliation;

public static class NightlyReconciliationWorkflow
{
    public static IWorkflow Build(
        IOrderRepository orderRepository,
        IPaymentService paymentService,
        IInventoryService inventoryService,
        IEmailSender emailSender)
    {
        return WF
            .CreateWorkflow("NightlyReconciliation")
            .AddOperation(new FetchUnprocessedOrdersOperation(orderRepository))
            .AddOperation(new ProcessPaymentsOperation(paymentService))
            .AddOperation(new UpdateInventoryOperation(inventoryService))
            .AddOperation(new MaybeFailOperation())
            .AddOperation(new SendConfirmationEmailsOperation(emailSender))
            .Build();
    }
}

Dependencies are injected into operations via constructors (clean, testable, no service locator).

The Models

Simple domain objects that flow through the workflow:

public sealed class Order
{
    public required string Id { get; init; } = null!;
    public decimal Amount { get; init; }
    public required IReadOnlyList<string> Items { get; init; } = null!;
}

public sealed class PaymentTransaction
{
    public required string TransactionId { get; init; } = null!;
    public required string OrderId { get; init; } = null!;
    public decimal Amount { get; init; }
}

Shared State Keys

Operations communicate through the foundry’s property bag. The keys are simple constants, no magic strings scattered across the codebase:

public static class ReconciliationKeys
{
    public const string BatchSize = "recon.batch_size";
    public const string DemoFailure = "recon.demo_failure";
    public const string Orders = "recon.orders";
    public const string PaymentTransactions = "recon.payment_transactions";
}

Operations with Automatic Compensation

This is where WorkflowForge shines. Each operation extends WorkflowOperationBase and can implement both execution (ForgeAsyncCore) and compensation (RestoreAsync). If a downstream step fails, completed operations are automatically rolled back in reverse order.

Fetching Orders

The first step loads unprocessed orders and stores them in the foundry for downstream operations:

public sealed class FetchUnprocessedOrdersOperation : WorkflowOperationBase
{
    private readonly IOrderRepository _orderRepository;

    public FetchUnprocessedOrdersOperation(IOrderRepository orderRepository)
    {
        _orderRepository = orderRepository;
    }

    public override string Name => "FetchUnprocessedOrders";

    protected override async Task<object?> ForgeAsyncCore(
        object? inputData, IWorkflowFoundry foundry, CancellationToken cancellationToken)
    {
        var batchSize = foundry.GetPropertyOrDefault<int>(ReconciliationKeys.BatchSize, 3);
        foundry.Logger.LogInformation("Fetching up to {BatchSize} order(s)", batchSize);

        var orders = await _orderRepository
            .GetUnprocessedOrdersAsync(batchSize, cancellationToken)
            .ConfigureAwait(false);

        foundry.SetProperty(ReconciliationKeys.Orders, orders);
        foundry.Logger.LogInformation("Fetched {Count} order(s)", orders.Count);
        return inputData;
    }

    public override Task RestoreAsync(
        object? outputData, IWorkflowFoundry foundry, CancellationToken cancellationToken)
    {
        foundry.Properties.TryRemove(ReconciliationKeys.Orders, out _);
        return Task.CompletedTask;
    }
}

Processing Payments (with Refund Compensation)

This is the core compensation pattern. ForgeAsyncCore charges each order and stores the transactions. RestoreAsync refunds them if anything fails downstream:

public sealed class ProcessPaymentsOperation : WorkflowOperationBase
{
    private readonly IPaymentService _paymentService;

    public ProcessPaymentsOperation(IPaymentService paymentService)
    {
        _paymentService = paymentService;
    }

    public override string Name => "ProcessPayments";

    protected override async Task<object?> ForgeAsyncCore(
        object? inputData, IWorkflowFoundry foundry, CancellationToken cancellationToken)
    {
        var orders = foundry.GetPropertyOrDefault<IReadOnlyList<Order>>(ReconciliationKeys.Orders)
            ?? Array.Empty<Order>();

        foundry.Logger.LogInformation("Processing payments for {Count} order(s)", orders.Count);

        var transactions = new List<PaymentTransaction>(orders.Count);
        foreach (var order in orders)
        {
            cancellationToken.ThrowIfCancellationRequested();
            var tx = await _paymentService.ChargeAsync(order, cancellationToken).ConfigureAwait(false);
            transactions.Add(tx);
        }

        foundry.SetProperty(ReconciliationKeys.PaymentTransactions, transactions);
        foundry.Logger.LogInformation("Processed {Count} payment(s)", transactions.Count);
        return inputData;
    }

    public override async Task RestoreAsync(
        object? outputData, IWorkflowFoundry foundry, CancellationToken cancellationToken)
    {
        var transactions = foundry.GetPropertyOrDefault<IReadOnlyList<PaymentTransaction>>(
            ReconciliationKeys.PaymentTransactions) ?? Array.Empty<PaymentTransaction>();

        foundry.Logger.LogWarning("Compensating payments: refunding {Count} transaction(s)",
            transactions.Count);

        foreach (var tx in transactions)
        {
            cancellationToken.ThrowIfCancellationRequested();
            await _paymentService.RefundAsync(tx, cancellationToken).ConfigureAwait(false);
        }
    }
}

Updating Inventory (with Release Compensation)

Same pattern: reserve on execute, release on compensate:

public sealed class UpdateInventoryOperation : WorkflowOperationBase
{
    private readonly IInventoryService _inventoryService;

    public UpdateInventoryOperation(IInventoryService inventoryService)
    {
        _inventoryService = inventoryService;
    }

    public override string Name => "UpdateInventory";

    protected override async Task<object?> ForgeAsyncCore(
        object? inputData, IWorkflowFoundry foundry, CancellationToken cancellationToken)
    {
        var orders = foundry.GetPropertyOrDefault<IReadOnlyList<Order>>(ReconciliationKeys.Orders)
            ?? Array.Empty<Order>();

        foundry.Logger.LogInformation("Updating inventory for {Count} order(s)", orders.Count);

        foreach (var order in orders)
        {
            cancellationToken.ThrowIfCancellationRequested();
            await _inventoryService.ReserveAsync(order.Id, order.Items, cancellationToken)
                .ConfigureAwait(false);
        }

        return inputData;
    }

    public override async Task RestoreAsync(
        object? outputData, IWorkflowFoundry foundry, CancellationToken cancellationToken)
    {
        var orders = foundry.GetPropertyOrDefault<IReadOnlyList<Order>>(ReconciliationKeys.Orders)
            ?? Array.Empty<Order>();

        foundry.Logger.LogWarning("Compensating inventory: releasing {Count} reservation(s)",
            orders.Count);

        foreach (var order in orders)
        {
            cancellationToken.ThrowIfCancellationRequested();
            await _inventoryService.ReleaseAsync(order.Id, order.Items, cancellationToken)
                .ConfigureAwait(false);
        }
    }
}

The Simulated Failure Toggle

This operation exists purely to demonstrate compensation. When DemoFailure is set to true, it throws, which makes WorkflowForge roll back payments and inventory automatically:

public sealed class MaybeFailOperation : WorkflowOperationBase
{
    public override string Name => "MaybeFail";

    protected override Task<object?> ForgeAsyncCore(
        object? inputData, IWorkflowFoundry foundry, CancellationToken cancellationToken)
    {
        var demoFailure = foundry.GetPropertyOrDefault<bool>(ReconciliationKeys.DemoFailure, false);

        if (demoFailure)
        {
            foundry.Logger.LogError("Simulated failure triggered to demonstrate compensation");
            throw new InvalidOperationException("Simulated failure (DemoFailure=true)");
        }

        return Task.FromResult(inputData);
    }

    public override Task RestoreAsync(
        object? outputData, IWorkflowFoundry foundry, CancellationToken cancellationToken)
        => Task.CompletedTask;
}

Sending Confirmations

The final step. No meaningful compensation needed (you can’t unsend an email):

public sealed class SendConfirmationEmailsOperation : WorkflowOperationBase
{
    private readonly IEmailSender _emailSender;

    public SendConfirmationEmailsOperation(IEmailSender emailSender)
    {
        _emailSender = emailSender;
    }

    public override string Name => "SendConfirmationEmails";

    protected override async Task<object?> ForgeAsyncCore(
        object? inputData, IWorkflowFoundry foundry, CancellationToken cancellationToken)
    {
        var orders = foundry.GetPropertyOrDefault<IReadOnlyList<Order>>(ReconciliationKeys.Orders)
            ?? Array.Empty<Order>();

        foundry.Logger.LogInformation("Sending confirmations for {Count} order(s)", orders.Count);

        foreach (var order in orders)
        {
            cancellationToken.ThrowIfCancellationRequested();
            await _emailSender.SendConfirmationAsync(order.Id, cancellationToken)
                .ConfigureAwait(false);
        }

        return inputData;
    }

    public override Task RestoreAsync(
        object? outputData, IWorkflowFoundry foundry, CancellationToken cancellationToken)
        => Task.CompletedTask;
}

Why do non-mutating operations override RestoreAsync? WorkflowForge triggers compensation when any operation that overrides RestoreAsync has completed before a failure. Even no-op restore implementations (like MaybeFail and SendConfirmationEmails) should override it so the engine knows they participated in the workflow’s compensation chain.

The Coravel Scheduling Layer

The Job

The ReconciliationJob implements Coravel’s IInvocable. It builds the workflow, creates a foundry with initial properties, and runs it:

public sealed class ReconciliationJob : IInvocable
{
    private readonly IOrderRepository _orderRepository;
    private readonly IPaymentService _paymentService;
    private readonly IInventoryService _inventoryService;
    private readonly IEmailSender _emailSender;
    private readonly IOptions<ReconciliationJobOptions> _options;
    private readonly ILogger<ReconciliationJob> _logger;

    public ReconciliationJob(
        IOrderRepository orderRepository,
        IPaymentService paymentService,
        IInventoryService inventoryService,
        IEmailSender emailSender,
        IOptions<ReconciliationJobOptions> options,
        ILogger<ReconciliationJob> logger)
    {
        _orderRepository = orderRepository;
        _paymentService = paymentService;
        _inventoryService = inventoryService;
        _emailSender = emailSender;
        _options = options;
        _logger = logger;
    }

    public async Task Invoke()
    {
        var options = _options.Value;
        _logger.LogInformation(
            "Starting scheduled reconciliation workflow (DemoFailure={DemoFailure})",
            options.DemoFailure);

        var workflow = NightlyReconciliationWorkflow.Build(
            _orderRepository, _paymentService, _inventoryService, _emailSender);

        using var foundry = WF.CreateFoundry(
            workflowName: workflow.Name,
            initialProperties: new Dictionary<string, object?>
            {
                [ReconciliationKeys.BatchSize] = 3,
                [ReconciliationKeys.DemoFailure] = options.DemoFailure
            });

        using var smith = WF.CreateSmith(new ConsoleLogger("WF"));

        try
        {
            await smith.ForgeAsync(workflow, foundry).ConfigureAwait(false);
            _logger.LogInformation("Reconciliation workflow finished successfully");
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
                "Reconciliation workflow failed (compensation should have run for completed steps)");
        }
    }
}

Configuration

{
  "ReconciliationJob": {
    "ScheduleSeconds": 10,
    "DemoFailure": false
  }
}

public sealed class ReconciliationJobOptions
{
    public const string SectionName = "ReconciliationJob";
    public int ScheduleSeconds { get; set; } = 10;
    public bool DemoFailure { get; set; } = false;
}

Host Setup (Program.cs)

Standard .NET hosting with Coravel scheduler. The entire host in one file:

var builder = Host.CreateApplicationBuilder(args);

builder.Logging.ClearProviders();
builder.Logging.AddSimpleConsole(options =>
{
    options.SingleLine = true;
    options.TimestampFormat = "HH:mm:ss ";
});

builder.Services.Configure<ReconciliationJobOptions>(
    builder.Configuration.GetSection(ReconciliationJobOptions.SectionName));

builder.Services.AddScheduler();
builder.Services.AddTransient<ReconciliationJob>();

builder.Services.AddSingleton<IOrderRepository, InMemoryOrderRepository>();
builder.Services.AddSingleton<IPaymentService, FakePaymentService>();
builder.Services.AddSingleton<IInventoryService, FakeInventoryService>();
builder.Services.AddSingleton<IEmailSender, FakeEmailSender>();

var host = builder.Build();

host.Services.UseScheduler(scheduler =>
{
    var options = host.Services.GetRequiredService<IOptions<ReconciliationJobOptions>>().Value;

    scheduler
        .Schedule<ReconciliationJob>()
        .EverySeconds(options.ScheduleSeconds)
        .PreventOverlapping(nameof(ReconciliationJob));
});

await host.RunAsync();

PreventOverlapping ensures a new run doesn’t start if the previous one is still going. Important for workflows that touch external systems.

Try It Yourself

# Happy path -- all steps succeed
dotnet run --project AnimatLabs.WorkflowForge.CoravelScheduledWorkflows

# Toggle simulated failure to see compensation in action
dotnet run --project AnimatLabs.WorkflowForge.CoravelScheduledWorkflows -- \
    ReconciliationJob:DemoFailure=true

With DemoFailure=true, the MaybeFailOperation throws after payments and inventory are processed. Watch the logs. You’ll see WorkflowForge automatically refund payments and release inventory reservations. The saga pattern, without writing a single line of orchestration code.

Why This Combination Works

Each tool does one thing well. No overlap, no conflict.

When to Use This Pattern

Good fit:

• Multi-step business processes on a schedule
• Operations that need rollback if later steps fail (payments, inventory, external APIs)
• Teams that want to avoid Hangfire/Quartz complexity
• Applications where in-memory scheduling is sufficient

Not ideal:

• Jobs that must survive application restarts (use Hangfire with persistence)
• Distributed job coordination across instances (use Hangfire or Quartz)
• Visual workflow designers for business users (use Elsa)

Resources

If you’ve tried a different scheduling approach with WorkflowForge, I’d be curious how it compared.

If WorkflowForge has been useful to you, a ⭐ star on GitHub helps it reach more .NET developers. And if you'd like to support the work behind it, Ko-fi is always open!

⭐ Star on GitHub Support on Ko-fi

See Also

• WorkflowForge introduction
• WorkflowForge 2.0 benchmarks
• MassTransit saga with WorkflowForge

Every .NET developer has been here. You’re building a system with 50 DTOs. Same five chores, every class. For each one you need:

• A mapper to convert between entities and DTOs
• Validation logic
• Equality comparisons
• ToString() overrides for debugging
• JSON serialization hints

That’s 5 pieces of boilerplate per class. 250 methods that are 90% identical, differing only in property names. You write them by hand, copy-paste errors creep in, and when a property changes, you forget to update the mapper.

The traditional solutions all have problems:

Source generators offer a different path: generate the code at compile time, as if you wrote it yourself.

What Source Generators Actually Do

Source generators hook into the Roslyn compiler. They:

1. Inspect your code via syntax trees and semantic models
2. Generate new C# source files
3. Add them to your compilation

The generated code is real C# - visible in your IDE, navigable with “Go to Definition”, fully debuggable. There’s zero runtime overhead for the generation itself. Took me a while to trust that, but the debugger does not lie.

They sit in the build pipeline like this:

flowchart TD
    A[Your Code] --> B[Roslyn Parser]
    B --> C[Syntax Trees]
    C --> D[Source Generators]
    D --> E[Additional Code]
    E --> F[Compilation]
    F --> G[IL]

The key difference from reflection: by the time your app runs, the generated code is already compiled. No Type.GetProperties(), no Activator.CreateInstance(), no JIT compilation of dynamic methods.

The Code: A Complete Playground Repo

I’ve put all the samples into a runnable playground solution:

You can access the full sample code here: GitHub Repo

It includes both Roslyn source generators and the equivalent T4-generated output (checked in) so you can compare the two approaches.

What Gets Generated (In This Repo)

The playground focuses on four patterns that show up constantly in real .NET codebases.

1) Strongly-Typed Configuration Binding

Define a POCO and mark it:

using AnimatLabs.SourceGenerators.Attributes;

namespace AnimatLabs.SourceGenerators.Demo.Models;

[GenerateConfiguration(SectionName = "Database")]
public partial class DatabaseSettings
{
    public string ConnectionString { get; set; } = string.Empty;
    public int Timeout { get; set; } = 30;
    public RetrySettings Retry { get; set; } = new();
}

public class RetrySettings
{
    public int MaxAttempts { get; set; } = 3;
    public int DelayMs { get; set; } = 1000;
}

The generator emits Bind(IConfiguration) plus nested binders for complex properties.

// Generated (shape): DatabaseSettings.Configuration.g.cs
public static global::AnimatLabs.SourceGenerators.Demo.Models.DatabaseSettings Bind(
    global::Microsoft.Extensions.Configuration.IConfiguration configuration)
{
    var section = configuration.GetSection("Database");
    return BindSection_AnimatLabs_SourceGenerators_Demo_Models_DatabaseSettings(section);
}

Supported property types: string, int, bool, double, decimal, enums, and nested classes. Unsupported types are skipped.

2) Enum Extensions (Display Names + Parsing)

using System.ComponentModel.DataAnnotations;
using AnimatLabs.SourceGenerators.Attributes;

namespace AnimatLabs.SourceGenerators.Demo.Models;

[GenerateEnumExtensions]
public enum OrderStatus
{
    [Display(Name = "Pending Approval")] Pending,
    [Display(Name = "In Progress")] Processing,
    Shipped,
    Delivered,
    Cancelled
}

The generator emits:

• ToDisplayName() (uses DisplayAttribute.Name when present)
• TryParse(string, out OrderStatus) (accepts both member name and display name)
• GetAll()

3) DTO Mapping Without Reflection

using AnimatLabs.SourceGenerators.Attributes;
using AnimatLabs.SourceGenerators.Demo.Models;

namespace AnimatLabs.SourceGenerators.Demo.Mappers;

[GenerateMapper]
public partial class UserMapper
{
    public partial UserDto ToDto(User entity);
    public partial User ToEntity(UserDto dto);
}

The generator implements each method by assigning properties where name and type match.

4) Auto-Generated ToString()

using AnimatLabs.SourceGenerators.Attributes;

namespace AnimatLabs.SourceGenerators.Demo.Models;

[AutoToString]
public partial class Person
{
    public string FirstName { get; set; } = string.Empty;
    public string LastName { get; set; } = string.Empty;
    public int Age { get; set; }
}