Correlation-based waveform QC¶
When you have many repeats of the same measurement (a lot of cables, a day of production S21, a temperature chamber full of the same DUT), pairwise Pearson correlation is a simple, unit-aware way to find traces that do not look like the rest.
Pearson r is the linear similarity of two y-arrays: −1 (anti-correlated) to +1 (same shape, up to a shift and scale). For S-parameter QC we correlate dB magnitude traces, not complex samples and not linear |S|.
Why pairwise, not “vs the average”¶
Do not average first and then compare each trace to the average. Outliers pull the average toward themselves and hide in the residual. Score every trace against every other trace first; average only the traces that survive.
Algorithm¶
Need ≥ 3 traces. Pairwise statistics degenerate below that.
1. Choose one trace type (usually S21) and the dataset filter (lot, day, project).
2. Align x (interpolate to a common grid, or trim to overlap).
3. Convert S-parameters to dB: y = 20 * log10(|S|).
4. Build y_matrix with shape (n_traces, n_points).
5. r[i, j] = pearson(y_i, y_j); r[i, i] = 1.
6. score_i = mean(r[i, j] for j != i) # exclude the diagonal
7. threshold = mean(scores) - sigma * std(scores) # default sigma = 2
8. outlier if score_i < threshold
9. clean average = mean of non-outlier rows (in dB if that is the published statistic;
or convert back — say which)
Always print each trace’s score. The distribution is the result; a boolean flag is a cut on that distribution.
Interpreting r (shape, same family of traces)¶
| r | Reading |
|---|---|
| ≥ 0.99 | Nearly identical |
| 0.95–0.99 | Similar, natural variation |
| 0.90–0.95 | Noticeable differences |
| < 0.90 | Meaningfully different — likely an outlier in this set |
These bands are rules of thumb for overlay-similar S21 families. A filter or an antenna measurement can be “good” at lower r. Prefer the 2σ-below-the-mean cut on this batch’s scores over a universal 0.90 line.
After scoring¶
- Tag outliers (
outlier). Do not delete them from the project. - Publish the clean average as a dataset in the same project.
- Notebook: outliers + average, not every passing trace unless the review asks for it.
If every trace flags as an outlier, σ is too tight or the batch is not one family (mixed lots, mixed trace types). Say so; do not publish an empty average.