Span Links for Batch and Fan-Out Jobs

A span can have only one parent. That fits a request that enqueues one job, but job systems often do other things: a consumer processes 100 messages in one batch, a single job fans out into 10,000 child jobs, or a callback runs after a group of jobs finishes. Forcing these into parent-child relationships produces misleading or unusable traces. Span links express "this work is related to those spans" without pretending there is one parent. This guide shows where to use them, as part of Distributed Tracing for Async Jobs in Observability & Monitoring for Job Queues.

Problem Statement

A data platform has three patterns that tracing handles badly. An SQS consumer receives batches of up to 10 messages and writes them to the warehouse in one insert; its span is attached to whichever message came first, so the other nine requests appear to end at the enqueue. A nightly job fans out into 20,000 per-customer jobs as children of one trace, producing a trace so large the tracing backend refuses to display it. A Celery chord's callback aggregates results from 50 tasks, but its trace shows only one of them as the parent. The team wants every original request to lead to the work done for it, bounded trace sizes, and a way to navigate from a batch or callback to all of its inputs.

Prerequisites

  • OpenTelemetry SDKs in the producer and consumer services, with W3C trace context propagated in message metadata.
  • A tracing backend that displays span links (Jaeger, Tempo, Honeycomb, Datadog, and most others do).
  • Familiarity with basic job tracing — see OpenTelemetry tracing for BullMQ.

Step 1 — Recognise When Parent-Child Is Wrong

Parent-child means "this span is part of that operation, and that operation waits for or contains it." Links mean "this span is causally related to those spans." Use links when:

  • Many inputs, one operation. A batch consumer processes messages from many producers. There is no single parent.
  • One input, very many outputs. A fan-out creates thousands of jobs. Making them all children creates one enormous trace.
  • Long or unbounded gaps. A job runs hours after its trigger. Keeping it in the same trace stretches the trace over hours.
  • Aggregation. A fan-in callback depends on many earlier jobs.
A batch of ten messages: parent-child vs links On the left, the batch processing span is made a child of the first message's producer span. That message's trace continues into the warehouse insert, but the other nine traces end at their enqueue and never show what happened next. On the right, the batch span starts its own trace and carries ten links, one to each producer span. Every producer trace can navigate to the batch, and the batch shows all its inputs. One batch, ten inputs parent-child batch insert 9 traces end here links batch insert own trace, 10 links Links preserve every input's path without choosing a favourite.

Step 2 — Link a Batch Consumer to Every Message

When a consumer receives a batch, extract the trace context from each message and pass them all as links when starting the processing span. In Python with SQS:

from opentelemetry import trace, propagate
from opentelemetry.trace import Link, SpanKind

tracer = trace.get_tracer("warehouse-loader")

def handle_batch(messages: list[dict]) -> None:
    links = []
    for m in messages:
        carrier = {k: v["StringValue"] for k, v in m.get("MessageAttributes", {}).items()}
        ctx = trace.get_current_span(propagate.extract(carrier)).get_span_context()
        if ctx.is_valid:
            links.append(Link(ctx, attributes={"messaging.message.id": m["MessageId"]}))

    with tracer.start_as_current_span("orders batch process", kind=SpanKind.CONSUMER, links=links,
                                      attributes={"messaging.batch.message_count": len(messages)}):
        insert_rows([parse(m) for m in messages])

Links must be supplied when the span starts (newer SDKs also allow add_link afterwards, but not all backends handle late links). The processing span begins a new trace. From any producer's trace, the backend shows "linked from" and lets you jump to the batch; from the batch, you see all ten producers.

If per-message work happens inside the batch (validation, enrichment), create a short child span per message and give each one a link to its own producer. That shows exactly which message was slow without mixing them up.

Step 3 — Bound Fan-Out Traces with Links

A job that creates thousands of child jobs should not make them all part of its own trace. Give each child job its own trace with a link back to the fan-out span:

import { trace, context, propagation, SpanKind } from "@opentelemetry/api";
const tracer = trace.getTracer("billing");

async function fanOut(customerIds: string[]) {
  await tracer.startActiveSpan("invoice fan-out", async (span) => {
    const carrier: Record<string, string> = {};
    propagation.inject(context.active(), carrier);
    await invoiceQueue.addBulk(customerIds.map((id) => ({
      name: "invoice", data: { id, _link: carrier },
    })));
    span.setAttribute("fanout.count", customerIds.length);
    span.end();
  });
}

// worker: new root, linked to the fan-out
function processInvoice(job: Job) {
  const parentCtx = trace.getSpanContext(propagation.extract(context.active(), job.data._link));
  return tracer.startActiveSpan("invoice process",
    { kind: SpanKind.CONSUMER, root: true, links: parentCtx ? [{ context: parentCtx }] : [] },
    async (span) => { try { await buildInvoice(job.data.id); } finally { span.end(); } });
}

root: true starts a new trace even though a context is available. Each invoice trace is small and fast to load; the fan-out span records how many jobs it created. To answer "how did the whole nightly run go?", use metrics or query linked spans in bulk (Step 5) rather than loading one giant trace.

Fan-out without a giant trace The nightly job's trace contains one fan-out span that records it created 20,000 jobs. Each invoice job starts its own small trace with a link back to the fan-out span. The nightly trace stays a handful of spans, each invoice trace is a few spans, and the relationship is preserved through links rather than by nesting 20,000 children under one parent. 20,000 jobs, 20,001 small traces invoice fan-out fanout.count = 20,000 invoice process (cust 1) invoice process (cust 2) ⋮ invoice process (cust 20,000) each invoice trace links back to the fan-out span

Step 4 — Link Fan-In Callbacks to Their Inputs

A callback that runs after a group completes — a Celery chord body, a BullMQ flow parent, a Sidekiq batch callback — depends on all the group's jobs. It cannot link to all of them if there are thousands (see Step 6), but it can link to the fan-out span and record the group's size and outcome:

@app.task(bind=True)
def summarise(self, results, fanout_carrier):
    fanout_ctx = trace.get_current_span(propagate.extract(fanout_carrier)).get_span_context()
    with tracer.start_as_current_span("report summarise", links=[Link(fanout_ctx)],
                                      attributes={"group.size": len(results),
                                                  "group.failed": sum(1 for r in results if r is None)}):
        write_summary(results)
Linking a fan-in callback, small vs large groups For a small group of around fifty jobs, the callback span carries one link per member, so the slowest or failed member is one click away. For a large group of twenty thousand jobs, that would exceed link limits, so the callback links only to the fan-out span and records group size and failure count as attributes; members are found by querying the shared run ID. Callback links scale with group size ~50 jobs: link each callback 20,000 jobs: link the fan-out fan-out callback group.size, group.failed Members of a large group are found by the shared run ID, not by links.

For small groups (tens of jobs), linking to each member span is practical and makes it easy to find the slowest one. For large groups, link to the fan-out and rely on metrics or attribute queries for the members. The chord mechanics are covered in Celery chains, groups and chords.

Step 5 — Query Linked Traces

Links are only useful if you can navigate and query them. Most backends show links in the span detail view; several also let you query by linked trace ID. Add attributes that make bulk questions answerable without following links one by one:

  • fanout.id (a stable ID for the run) on the fan-out span and on every child — then "all invoice spans for last night's run" is a simple attribute query.
  • messaging.batch.message_count on batch spans, to find unusually large or small batches.
  • group.size and group.failed on callbacks.

With fanout.id on every child, a single query shows the slowest invoices of a run, their error rate, and their duration distribution — the overview that one giant trace would have tried and failed to provide.

Step 6 — Respect Link Limits

SDKs cap the number of links per span (128 by default in most SDKs, configurable with OTEL_SPAN_LINK_COUNT_LIMIT). Backends may impose their own limits. Link the batch span to every message only when batches are small (SQS allows at most 10 per receive; Kafka consumers can receive hundreds). For large batches, link to a sample, or create per-message child spans each with one link. Links also add a little size to each span; that matters only at very high volumes.

Verification

  • A batch of ten test messages produces one batch span with ten links, and each producer trace shows a link to it.
  • A fan-out of 1,000 test jobs produces 1,000 small traces, each linked to the fan-out span, and the fan-out trace loads quickly.
  • A query by fanout.id returns all child spans for a run.
  • A chord callback span shows the link to its fan-out and the group size and failure count.

Gotchas & Edge Cases

Sampling and links. A linked span makes its own sampling decision; a sampled producer can link to an unsampled consumer and vice versa. Use tail sampling to keep error traces, and accept that some links point to traces that were not kept.

Backend support varies. Some backends display links but cannot search by them. Test navigation in yours before relying on it for incident response.

Do not mix models per job type. If some invoice jobs are children and others are linked, queries and dashboards become confusing. Choose one model per job type and document it.

Retries inside batches. If one message in a batch fails and is retried alone, its retry span should link to its own producer, not to the original batch.

FAQ

Are links the same as follows-from references? Yes, conceptually. OpenTracing's "follows from" relationship became span links in OpenTelemetry.

Should every queue job use links instead of parent-child? No. For a request that enqueues one job that runs soon, parent-child gives the clearest single trace. Use links for batches, large fan-outs, long delays, and aggregation.

Do metrics need links? No, but exemplars on metrics can point to linked traces, so a slow bucket in a heatmap leads to a specific job's trace.

Can a span have both a parent and links? Yes. A per-message child span inside a batch has the batch span as its parent and a link to its own producer. Parent describes where the work happens; links describe what caused it.

Related