zintara.io

Reply measurement · Practical guide

Cold email experiment stopping rules: decide when to read the result

Choose the analysis point before launching, while continuing to monitor operational problems. Stopping for a safety or sending issue is different from stopping because a variant briefly looks better.

Reviewed · Examples are illustrative

Who this helps: Campaign owners and analysts evaluating attribution, comparisons and experiment evidence.

Define the decision

Early rates can swing sharply when only a few outcomes have arrived. Repeatedly checking for a favorable difference and ending the test at that moment changes the statistical behavior of an ordinary fixed-sample analysis. A planned sequential method can handle repeated looks, but it must actually be part of the design.

Work through the procedure

  1. Set the required sample and reply observation window using the chosen design.
  2. Define operational interruption rules, such as a connection failure or an exclusion mistake.
  3. Specify who may stop the experiment and how interrupted runs will be reported.
  4. Analyze at the planned point or use a properly designed sequential method instead of improvising a significance rule.

Worked example

The following is a synthetic example for this procedure, not a customer result or performance benchmark.

Day 2: A has 2 positives; B has 0
Day 4: A has 2 positives; B has 3
Neither snapshot is the planned final analysis.
Separate event: one sender stops authenticating.
Action: pause affected operations, document the interruption and assess comparability before continuing.

Read the result

The operational pause protects the process without pretending the early outcome counts identified a winner. An interrupted test may need to be reported as inconclusive or redesigned. Restarting while keeping only favorable earlier observations would create a new, undocumented analysis rule.

Check before moving on

  1. Keep a dated record of the original plan.
  2. Allow replies from the final send cohort to mature.
  3. Report deviations and missing data instead of hiding them in the final percentage.

Limits and next action

This guide does not supply sequential-testing boundaries or a universal stopping threshold. Zintara campaign controls are operational tools; pausing a campaign does not validate an experiment's statistics. Use a method matched to the actual monitoring plan.

Source: Zintara product context; analytical methods and examples are defined in the guide

Source references

Worked examples are illustrative. Editorial procedures are suggested methods, not measured performance claims or promises of additional product features.

Related guides

Explore the Zintara workflow