Quantum measurement results are samples, not exact probabilities. This guide shows how to convert shot counts into probabilities, calculate uncertainty, compare repeated runs, plot error bars, and record enough context to tell a meaningful algorithm change from ordinary sampling variation.
Overview
A quantum circuit produces a probability distribution, but a device or simulator usually reveals that distribution through repeated measurements called shots. If an outcome appears 620 times in 1,000 shots, the observed probability is 0.62. It is an estimate of the underlying probability, not proof that the probability is exactly 0.62.
For a single binary outcome, the basic model is binomial: each shot either produces the outcome of interest or it does not. If k successes are observed in n shots, then:
p_hat = k / n
The estimate becomes more stable as n increases, but increasing shots does not remove systematic hardware errors, calibration drift, readout bias, gate errors, or an incorrectly specified circuit. Treat shot count as one part of the experiment record, not as a substitute for controlling the experiment.
This workflow applies across common quantum computing tutorials and SDKs. Qiskit commonly exposes integer-keyed count dictionaries, Cirq can return measurement samples or histograms, and PennyLane commonly returns probability vectors or sampled measurement arrays. The statistical analysis is the same after you normalize those outputs into counts.
What to track
1. Counts, shots, and the event definition
Store the complete count dictionary whenever possible, not just the most frequent result. Define the event being analyzed precisely. For example, “qubit 0 is measured as 1” is different from “the bitstring is 01.” For a multi-bit event, sum counts for every bitstring that satisfies the event before calculating its probability.
from collections import Counter
def probability_for(counts, event):
shots = sum(counts.values())
successes = sum(count for outcome, count in counts.items()
if event(outcome))
return successes / shots, successes, shots
counts = {"00": 382, "01": 118, "10": 121, "11": 379}
p, successes, shots = probability_for(
counts,
lambda bitstring: bitstring[-1] == "1"
)
print(p, successes, shots)
Bit-string ordering differs across tools and measurement layouts. Confirm which character maps to which qubit before interpreting a result. Also record whether unused classical bits, measurement keys, or register names affect the returned labels.
2. Confidence intervals, not only point estimates
A 95% confidence interval communicates sampling uncertainty around a measured proportion. For small counts or probabilities near zero and one, use a Wilson or exact binomial interval rather than relying on the simple normal approximation.
from statsmodels.stats.proportion import proportion_confint
def wilson_interval(successes, shots):
return proportion_confint(
successes, shots, alpha=0.05, method="wilson"
)
low, high = wilson_interval(successes, shots)
print(f"p={successes/shots:.3f}, 95% CI=({low:.3f}, {high:.3f})")
If you do not want an additional dependency, SciPy's binomial distribution can support an exact interval, depending on the installed version and chosen method. Pin and record package versions in reproducible projects because statistical function signatures and SDK result objects can change between releases.
3. Repeated runs and paired conditions
When comparing two circuits or two parameter settings, record k_a, n_a, k_b, and n_b. A difference in estimates is:
delta = (k_a / n_a) - (k_b / n_b)
For a formal comparison, use a two-proportion method or a confidence interval for the difference. If the same shots or matched experimental units are not used, treat the samples as independent. If runs are paired by design, use a paired analysis instead; do not assume that repeating a circuit automatically creates paired observations.
4. Experimental metadata
For every run, keep the circuit identifier, SDK and package versions, backend or simulator name, shot count, transpilation settings, optimization level, measurement mapping, circuit parameters, execution timestamp, seed where supported, and relevant noise or mitigation settings. A simulator seed can make sampling reproducible, but it does not make a hardware run deterministic. Backend calibration and queue-time conditions may also matter when investigating changes.
| Experiment | Useful summary | Recommended uncertainty method |
|---|---|---|
| One binary outcome | Successes and shots | Wilson or exact binomial interval |
| Two independent circuits | Two proportions and their difference | Interval for a difference or two-proportion test |
| Many bitstrings | Counts for each category | Multinomial intervals or a planned distribution comparison |
| Repeated parameter sweep | Estimate and interval per parameter | Intervals with multiple-comparison awareness |
| Hardware versus simulator | Matched event probabilities | Difference analysis plus an explicit noise model |
Cadence and checkpoints
Use a checkpoint whenever a result will influence a development decision. A practical cadence has three levels:
- Every run: validate that counts sum to shots, confirm expected keys or measurement shapes, calculate probabilities and intervals, and save metadata.
- At each circuit or algorithm change: compare the new result with a fixed baseline using the same event definition, shot budget, and analysis code.
- Monthly or quarterly: rerun a small measurement-validation suite after SDK, dependency, backend, transpiler, noise-model, or hardware-access changes. Review whether bit ordering, result schemas, and seed behavior still match your assumptions.
Choose shot counts before looking at the result. For a rough binary-proportion margin of error, the normal approximation gives 1.96 * sqrt(p(1-p)/n) for a 95% interval. When planning without a reliable value of p, using p=0.5 is conservative for this approximation. Use the resulting number as a planning estimate, then check the final interval with a Wilson or exact method.
Keep a baseline notebook or machine-readable record. A CSV or JSON row containing the event, counts, shots, interval, circuit hash, backend, seed, and timestamp is often more useful than a screenshot of a histogram.
How to interpret changes
Do not call a change meaningful simply because one probability is numerically larger. First check whether the intervals overlap, but remember that interval overlap is only a quick visual heuristic, not a complete significance test. For a decision, calculate an interval for the difference or use a test appropriate to the design.
Separate three questions:
- Is the change larger than sampling noise? Compare the estimated difference with its uncertainty.
- Is it practically important? Define a minimum effect size before the experiment, such as the improvement needed to justify additional circuit depth.
- Is it reproducible? Repeat under controlled conditions and check whether the direction and approximate size persist.
If a result changes after increasing shots, determine whether the original estimate was simply noisy. If it changes after moving from a simulator to hardware, consider readout and gate errors, connectivity, transpilation, and calibration conditions. If it changes after an SDK upgrade, rerun a fixed regression circuit and inspect result conventions before changing the algorithm. A useful companion is a pytest-based measurement suite such as Testing Quantum Circuits in Python.
For many bitstrings, avoid interpreting dozens of isolated bars without a plan. Define the outcomes that matter, aggregate rare categories where scientifically justified, and use a distribution-level comparison when the full output distribution is the object of interest. Error bars on individual categories can become misleading when many categories are inspected simultaneously.
When to revisit
Revisit this analysis monthly or quarterly for active projects, and immediately when a recurring data point changes. Trigger a fresh review after an SDK or dependency upgrade, a backend change, a new transpilation configuration, a modified noise model, a new error-mitigation step, or a change in measurement mapping.
Before accepting a new result, run this checklist:
- Do all counts sum to the declared shot count?
- Is the event definition and bit ordering documented?
- Are probabilities accompanied by confidence intervals?
- Was the shot budget selected before inspecting the outcome?
- Are baseline and candidate runs comparable, or is the difference explained?
- Are SDK versions, backend conditions, seeds, circuit parameters, and timestamps saved?
- Has the measurement analysis been rerun after relevant toolchain changes?
For broader reproducibility work, pair this measurement workflow with a circuit resource record covering qubits and depth, and review SDK-specific changes before migrating analysis code. The goal is not to eliminate variation; it is to make variation visible, quantified, and actionable.