Measurement
Part of Measuring multilingual social marketing without confusing attribution with cause
Testing a multilingual benchmark protocol on synthetic records
Multilingual social marketing benchmark research: test the protocol on synthetic records with two analysts, language versions and seeded edge cases.
A multilingual social marketing benchmark study is only as sound as its protocol. Before anyone is recruited, the protocol needs a test on synthetic records. If it fails there, it will fail on real participants' data.
The test asks whether two analysts, working from the frozen specification alone, reach the same units, denominators and strata. It also shows where the protocol breaks first, while a fix still costs only a revised specification.
What to take away
- A protocol is ready for recruitment when two analysts working from the frozen specification reach the same eligible units, denominator and strata.
- Synthetic records from differently structured organisations hold no participant's private data, so defects can be fixed before recruitment.
- Seed the awkward casesa late record, a duplicate campaign, a campaign in two language versions, a source-definition change, a suppressed cell and a withdrawal.
- Acceptance criteria belong to the protocol, never to a favourable benchmark value.
- If the test cannot be passed, publish "no usable benchmark" and keep the protocol.
Why test the protocol before recruitment
A protocol is only as good as its weakest undocumented judgement. Testing it on synthetic records finds those judgements while no participant's private data is involved, and while a fix costs a revised specification rather than a lost sample.
State the decision first, for example whether an internal process result is unusual among genuinely comparable campaigns. Then define the target population as the strategy and planning guide describes: organisations, campaigns or assets with an England operating scope and the same service boundary.
Recruitment, sharing and linkage all raise privacy, PECR and research-ethics questions. Name those reviewers in advance for recruitment, sharing, linkage, retention, suppression and publication. Translation and accessibility reviewers should check participant materials and final tables separately.
Give the test records the fields the protocol relies on
Build the synthetic set from the frozen specification, using records from differently structured organisations. Each record should carry the fields the protocol will later depend on, so that a missing or ambiguous field shows up as a defect instead of a guess.
Define Population Before Sampling
- State the decision first
- Set England operating scope
- Fix same service boundary
- Specify sector and organisation size
- Record paid or organic route
- List coverage gaps and duplicates
Start with the sampling frame. An analyst should be able to tell from the specification which units are eligible, using sector, organisation size, paid or organic route, objective, campaign period, language-production method and eligible social channels, and to record coverage gaps, invitation denominator and non-response.
Next come the metric cards. Every submitted measure needs the event, eligible population, numerator and denominator or count unit, period, timezone, source system, query or export version, inclusions, exclusions, deduplication, late-data rule, owner and uncertainty. A common definition is frozen before collection.
Metric Card Fields to Freeze
- Event and eligible population
- Numerator and denominator
- Period and timezone
- Source system and query version
- Inclusions, exclusions, deduplication
- Late-data rule and owner
- Uncertainty statement
A card that cannot be filled in for one synthetic organisation shows where comparison would stop. The test should record that point rather than fill the gap with a plausible value.
Stratify, Then Decide to Combine
Report per stratum
- Eligible count
- Always reported
- Responding count
- Always reported
- Missingness
- Always reported
- Small or selective cell
- Suppress or no estimate
Combine only if
- Eligible count
- Pre-specified weighting
- Responding count
- Defensible target population
- Missingness
- Documented method
- Small or selective cell
- Not combined
Last, the strata. Language version is one of them in a multilingual study. The test should reproduce each stratum's eligible count, responding count and missingness, and combine strata only under a pre-specified weighting method. A language stratum with too few units to report is suppressed, never merged silently into another. Participant language or format need, where a study asks for it, is self-described under an appropriate research permission, as the audience research guide explains.
Run the test with two analysts
Two people reading the same specification expose what one person's habits hide. Keep them apart until both have finished, so neither can borrow the other's reading of an unclear rule.
- Freeze the specification and give it to two analysts who have not discussed it.
- Ask each to select the eligible units from the synthetic records.
- Ask each to calculate the same denominator and apply the same missing-data code.
- Ask each to reproduce every stratum from the frozen specification alone.
- Compare the results and log every difference as a defect.
Say one analyst counts a repost as a duplicate campaign and the other counts it as a new one. If the specification is silent, the difference is a defect in the protocol, not a mistake by either analyst. Write the rule, version it and run the test again.
Seed the awkward cases
Clean records prove little. Add the cases that real data will contain, and decide in advance what a pass looks like for each one.
| Seeded case | What it tests | The protocol fails if |
|---|---|---|
| A late record | The late-data rule on the metric card | The figure changes and no rule says how |
| A duplicate campaign | Deduplication | Two analysts count it differently |
| One campaign in two language versions | Unit of analysis and per-language denominators | One analyst counts one campaign, the other two, and the specification does not say which |
| A source-definition change | Comparability with an earlier edition | The change is absorbed without a note |
| A suppressed cell | The suppression rule for a small cell | A small or selective cell is reported as an estimate |
| A withdrawal | Handling of a participant who leaves | Results cannot be recreated without that participant's private data |
Duplicate campaigns are one of the risk scenarios worth rehearsing before recruitment. The protocol fails if results depend on an undocumented judgement, whichever row exposes it.
Set acceptance criteria for the protocol, not the result
Set acceptance criteria for the protocol, not for a favourable benchmark value. A protocol passes when both analysts reproduce the same units, denominators and strata, and every seeded case is handled by a written rule. A test judged by how good the benchmark looks measures the wrong thing.
Record defects, revisions and the point at which a changed definition breaks comparability with an earlier edition. Document any exclusion, imputation or weighting, then test how each choice changes the result. The ONS describes survey estimates as carrying uncertainty made up of both sampling and non-sampling error.
The common ways a measurement record goes wrong are catalogued in six measurement mistakes, and each makes a useful seeded case for the next round of testing.
When the test fails or the study stops
If definitions, source access or sample quality prevent comparison, publish "no usable benchmark" and preserve the protocol. A stopped study still leaves a tested protocol and an internal baseline, which is a result worth keeping.
A disclosure names the publisher, funding and conflicts, research dates, geography, population, sample frame, invitation and response denominators, eligibility, exclusions, metric cards, strata, weights, missing-data treatment, uncertainty, source versions and correction policy.
The Office for Statistics Regulation's Code of Practice says producers must communicate the uncertainty in estimates so that the public can use the statistics with confidence. A private benchmark is not official statistics, but the test is directly useful. Separate operational, delivery, reach, response, conversion and commercial measures, and do not label observed differences as causal.
Before you act
- Build synthetic records from differently structured organisations.
- Give the frozen specification to two analysts who work apart.
- Seed a late record, a duplicate campaign, a two-language campaign, a definition change, a suppressed cell and a withdrawal.
- Log every difference between the analysts as a defect.
- Set acceptance criteria for the protocol, not the benchmark value.
- Decide in advance who may declare "no usable benchmark".
Common questions
Why test a benchmark protocol on synthetic records?
Synthetic records hold no participant's private data, so defects can be found and fixed before recruitment. Test records from differently structured organisations show whether the specification stands on its own.
What makes a protocol fail the test?
It fails if results depend on an undocumented judgement or cannot be recreated without one participant's private data. Two analysts reaching different units, denominators or strata points to a gap in the specification.
What happens if the protocol cannot pass?
If definitions, source access or sample quality prevent comparison, publish "no usable benchmark". Preserve the protocol and collect an internal baseline instead of presenting a UK or global number as England evidence.
