Ten stores can be an operationally manageable holdout. Ten is not a statistically privileged number, and it is not a guarantee that storms, store differences or ordinary variation will average out.
Before selecting a holdout, establish whether the proposed design can resolve the improvement that matters. If it cannot, a small test may still answer execution questions. Describe that narrower purpose explicitly.
I use five planning questions for this conversation: clarity, coverage, comparability, contamination and communication. This is my planning framework, not a retailer-mandated method.
Clarity: what decision will the comparison inform?
Name the change, the primary outcome and the smallest effect worth acting on. Separate observed business performance from the causal effect you want to estimate.
Also distinguish two very different proposals:
- A small treatment group compared with a small control group limits exposure to the new intervention.
- A broad rollout with ten locations held out exposes most of the business to the intervention. The small control group does not make that rollout a low-risk pilot.
Write the cost of the test and the possible operational downside for both groups. Do not count hypothetical forgone sales in the holdout as established lost revenue; whether those sales are incremental is part of the question.
Coverage: what business does the sample represent?
Define eligible stores or clubs, items, channels and periods before assignment. Record any exclusions, such as scheduled remodels or incompatible assortment, and explain how they limit the result's relevance elsewhere.
Use pre-period factors to form groups or matched pairs where useful. Then randomize treatment within the chosen structure. “Label one store test and the other control” is not equivalent to random assignment. Blocking can account for selected known differences; it cannot make a convenience sample representative of every location. NIST describes the role of randomized blocks.
If political or operational constraints fix a store's assignment, document them. You may have a nonrandomized comparison rather than the randomized experiment originally proposed.
Comparability: is there enough independent information?
Ten holdout stores and ten matched treatment stores provide ten pairs, not hundreds of independent experiments because they generate daily rows. Sample-size work must reflect the assignment, matching, outcome variability and correlation over time. Cluster-trial guidance makes the distinction between participants and independent clusters explicit.
Have the analysis owner assess likely precision using relevant history and the proposed design. With few stores, ordinary large-sample calculations can be misleading; choose inference appropriate to the actual randomization and small sample. If useful effects remain indistinguishable from noise, change the design or narrow the claim before launch.
Lock the calendar, primary analysis, treatment of missing records and stopping rule in advance. If a nonrandomized comparison is necessary, explain its assumptions and limitations. A prior period alone is vulnerable to seasonality and other concurrent changes.
Contamination: did each location receive its assigned condition?
Give the retailer and participating field teams an agreed treatment and control list. Define what business as usual means, which changes are part of the intervention, and how execution will be checked.
Check physical and digital exposure separately. A store may lack new signage while its shoppers still encounter the campaign online or visit nearby treated stores. Geography, customer movement, inventory transfers and shared teams can create spillover.
Coordinate any stock, fixture, order or price changes through the people and systems authorized to make them. Do not suppress orders or redirect stock simply to protect the experiment. Such changes can affect availability and alter the intervention being measured.
Log deviations and corrective action. Do not quietly discard poorly executed treatment stores after seeing their results. Preserve the original assignment for the planned analysis and identify any additional execution-based analysis as a separate, potentially biased comparison.
Communication: what will each team receive?
My recommended brief gives each stakeholder a practical answer:
| Stakeholder | What to make explicit |
|---|---|
| Merchant or account lead | The decision, relevant assortment, requested authorization and limits on rollout conclusions. |
| Operations and field teams | The exact change, timing, workload, exception route and stop authority. |
| Finance | Test cost, outcome definition, economic decision threshold and uncertainty. |
| Analyst or data owner | Assignment, source definitions, eligible population, missing-data plan and reproducible calculation. |
Read the result at the strength it supports
Report the estimated difference with uncertainty, execution coverage and the applicable business threshold. “Inconclusive” is a valid result. A positive point estimate alone is not a rollout recommendation, and a nonsignificant result does not demonstrate no effect.
If the pilot establishes only that a display can be installed consistently, say so. If it supports a sales effect in one season and format, identify the additional evidence or controlled expansion needed before applying it elsewhere.
The retail test design guide covers the broader planning sequence. Retail Reason Intelligence can help work through the retailer-specific context around a proposed test. Discuss your test with me for help shaping its decision, operating boundaries and readout.