Empty Shelves and Excess Stock

Ordering the right quantity of each product for each store, so the shelf is not empty when customers arrive and the backroom is not full of stock that ends in a markdown.

AI is suitable with conditions

Forecasting demand across tens of thousands of products and hundreds of stores is a genuine fit for statistical forecasting and machine learning, and pooled models that learn across many products won the M5 competition, the largest public retail forecasting benchmark. It is conditional rather than strong because the forecast is often not the only thing failing. An audit of one retailer's stores found nearly two thirds of inventory records inaccurate, and studies attribute a large share of empty shelves to store-level ordering and shelf replenishment rather than to upstream supply. A better model fed a stockout-suppressed sales history and a perpetual inventory that believes the shelf is full will place the same wrong orders with more confidence. The value of AI therefore depends on first correcting the demand history for stockouts and catching inventory records that no longer match the store.

Problem Statement

Every week a retailer has to decide how much of each product to send to each store.

Send too little and the shelf is empty when a customer comes looking, and send too much and the excess sits in the backroom until it is marked down or thrown away.

The decision looks like a forecasting problem, and the market for forecasting tools treats it as one.

The sources behind this playbook point at the data as often as at the model: a sales history that hides stockouts, and an inventory record that does not match the shelf.

Business Context

The scale is what makes this function distinctive.

A grocery or general merchandise chain can have tens of thousands of products across hundreds of stores, and that means millions of product and store combinations, each of which needs an order quantity on every delivery cycle.

Nobody can review those by hand, so the replenishment system places most orders automatically and planners intervene on the exceptions.

That makes the parameters and the inputs of the system, not the individual orders, the real place where planning decisions are made.

It also means a small systematic error, such as one that slightly under-orders every product that ran out last month, compounds across the whole chain.

Why This Is Hard

The first difficulty is that sales are not demand.

When a product is out of stock, sales for that day drop to zero or near zero, and the history records low demand rather than missing supply.

A forecast trained on that history orders less, which makes the next stockout more likely, which teaches the forecast that demand is even lower.

Practitioners correct for this by flagging stockout periods and either replacing their sales with an estimate of what would have sold, or treating them as censored observations that show demand was at least the recorded sales, as one independent practitioner demonstrates with a censored likelihood model.

The estimate only ever replaces a lower recorded figure, because a stockout can hide demand but cannot add to it.

The second difficulty is that the inventory record is not the inventory.

An audit of nearly 370,000 inventory records across 37 stores of one retailer found nearly two thirds inaccurate, through causes such as theft, damage, miscounted deliveries, and misplaced stock.

When the record says stock is present and the shelf is empty, the system never reorders, and the product can sit at zero sales for weeks while the system believes it is simply not selling.

The third difficulty is sparsity.

Most products in most stores sell a handful of units a week or less, so a single product and store history is mostly zeros.

Forecasting: Principles and Practice notes that percentage error metrics are undefined when sales are zero and that standard methods assume continuous data, which is why forecasters aggregate to a grouped level and spread the result down, use methods built for intermittent demand such as Croston’s, or use pooled models that learn across many series at once.

The last difficulty is that the forecast is the wrong output.

A replenishment order has to cover uncertain demand over the lead time, so what it needs is a range, such as the quantity that covers demand nineteen times out of twenty, and an average does not provide one.

How The Solutions Fit Together

Reorder-point rules are the floor.

They turn a forecast and a service level target into an order quantity in a way every planner and store manager can check, and they remain the mechanism that places orders even when a model supplies the inputs.

A stockout-corrected statistical forecast supplies those inputs for the whole range, after the sales history has been cleaned of stockout periods.

It is fast, it is explainable component by component, and it is the benchmark that any more complex model has to beat.

Phantom inventory screening runs beside both, looking for products whose sales have stopped in a way their demand history does not explain, and sending a store worker to check the shelf.

It addresses a cause the other solutions cannot see, the empty shelf whose record says it is full, and research has shown products missing from the shelf can be detected from sales data with useful accuracy.

A pooled demand model with quantile output supports the statistical forecast where it is measurably weak, typically on promotions, new products, and the slowest sellers, by learning from many similar products and stores at once.

None of these decides on its own.

Each changes the numbers a planner sees and the orders the system proposes, and the planner remains responsible for service levels, promotions, and overrides.

Implementation Considerations

Measure on-shelf availability physically before changing anything, with shelf audits or gap scans on a sample of stores, because the system’s own in-stock figure will flatter the starting point.

Build the stockout flag and the corrected demand history first, since every later step depends on it.

Benchmark every forecast against a seasonal naive forecast and exponential smoothing, on corrected demand, with a scaled error metric rather than MAPE.

Run the comparison as a backtest over several past periods, never a single holdout, because seasonal retail series can look good or bad on one period by chance.

Set order quantities from a demand quantile chosen by the planner’s service level target, and let the planner change that target by category rather than editing orders one at a time.

Log every planner override with the reason, so their effect can be measured instead of debated.

ROI Considerations

The return comes from three places, and they have to be counted together.

Recovered sales from shelves that are no longer empty, much of which comes from correcting phantom inventory and stockout-suppressed history rather than from a better model.

Lower markdowns, waste, and working capital from stock that no longer needs to be held as a cushion against an untrusted forecast.

Planner time recovered from routine overrides and redirected to promotions, launches, and the exceptions that actually need judgement.

A project that raises availability by inflating stock everywhere has moved cost from lost sales to markdowns and has not produced a return.

Security, Privacy, and Regulation

Sales, stock, and inventory data are not personal data, but cost prices, supplier terms, and promotion plans are commercially sensitive and should be limited to the functions that need them.

Loyalty and basket data are personal data, so use them only in aggregated form for substitution analysis, and only within the purposes customers were told about.

Phantom inventory flags can point towards theft in a particular store, so keep them operational and route any loss prevention investigation through the retailer’s existing process rather than treating a flag as evidence about an individual.

Where The Evidence Disagrees

The sources do not settle which forecasting model to start with.

Forecasting: Principles and Practice presents exponential smoothing as producing reliable forecasts quickly for a wide range of series, which makes it the usual fast default per product, while pooled models that learn across products won the M5 competition on several years of daily sales from a single large chain.

No source here says exactly where one approach stops beating the other on a given retailer’s data.

This playbook’s choice, which is a judgement rather than a sourced finding, is to make the statistical forecast the default because it is cheap and explainable, and to make the pooled model earn its place category by category in a backtest.

Published estimates of how much of the out-of-stock problem starts in the store, rather than further up the supply chain, also vary widely between studies and summaries of them.

All of them put a substantial share in the store, so this playbook states it qualitatively rather than quoting a single figure.

Symptoms

Customers find gaps on the shelf for products the replenishment system says the store has on hand.
The same store carries too much of one product and none of another in the same category, often in the same week.
Promotions sell out on the first day in some stores while other stores mark down the leftover promotional stock afterwards.
Planners override a large share of system orders by hand because they do not trust the forecast, and nobody measures whether the overrides help.
Perishable waste and end-of-season markdowns grow at the same time as lost sales from out-of-stocks.

Root causes

Sales history records what was sold, not what customers wanted, so every day a product was out of stock is recorded as low demand and the next forecast orders less, which causes the next stockout.
Perpetual inventory records drift away from what is physically in the store through theft, damage, receiving errors, and miscounted returns, so the system believes stock is present when the shelf is empty and never reorders.
Stock can be in the backroom while the shelf is empty, because shelf replenishment on the store floor did not happen in time.
Most products in most stores sell slowly and irregularly, so a store-level history is mostly zeros and too thin to forecast on its own.
Promotions, price changes, holidays, and new products are not recorded consistently enough for any forecast to learn their effect.
Order quantities are set from a point forecast, while the real decision is how much buffer to hold against uncertainty, which a single number cannot express.

Business impact

Lost sales when the shelf is empty, part of which is lost for good because the customer buys elsewhere rather than substituting or waiting.
Customer trust in the store erodes when a regularly bought product is repeatedly missing, which costs more than any single lost sale.
Working capital tied up in excess stock, and backroom space that the store has to manage.
Markdowns on seasonal and promotional overstock, and write-offs on perishable stock that expires before it sells.
Planner time spent overriding system orders by hand instead of fixing the parameters and the data that cause the bad orders.

Traditional approaches

Reorder points and min/max levels per product and store, with a safety stock set from rules of thumb or a service level target.
Moving averages or exponential smoothing on weekly sales, run inside the replenishment or ERP system.
Planner judgement, applied as manual overrides for promotions, events, and local knowledge.
Periodic cycle counts and full stocktakes to correct inventory records.
Gap scans or shelf walks by store staff to find empty shelves and refill them from the backroom.

Solution approaches

Each of these is written for this problem specifically, not borrowed from a generic catalogue.

Recommendation

Start with explicit reorder-point rules and a stockout-corrected statistical forecast for every product and store, because these are cheap, explainable, and the benchmark anything more complex must beat. Run phantom inventory screening alongside them from the start, since it finds the empty shelves that no forecast will ever reorder. Add a pooled demand model with quantile output only where the statistical forecast is measurably weak, typically on promotions, new products, and very slow sellers, and set order quantities from the quantiles rather than a point forecast. Planners keep control of promotions, new product launches, and the service level targets, and store staff confirm a phantom inventory flag on the shelf before any record is corrected.

Data requirements

Point of sale history medium sensitivity: Units sold per product, store, and day or week, with price paid, used as the demand signal once corrected for stockouts.
Perpetual inventory and stockout flags low sensitivity: Daily on-hand quantity per product and store, plus any recorded out-of-stock events, needed to tell low demand apart from no stock.
Receipts, transfers, and adjustments low sensitivity: Deliveries received, inter-store transfers, write-offs, and count adjustments, used to explain why records and physical stock diverge.
Promotion and price calendar medium sensitivity: Planned and past promotions, price changes, display placements, and flyer features per product and store.
Product and store master data low sensitivity: Product hierarchy, case pack, shelf capacity, shelf life, store cluster, and delivery schedule, used to group sparse series and to turn a forecast into a feasible order.
Loyalty or basket data high sensitivity: Optional customer-level purchases that show substitution when a product is missing. Personal data, so use only aggregated where possible.

Evaluation metrics

On-shelf availability measured by shelf audits or gap scans, not the in-stock rate the system reports from its own records.
Lost sales estimated from stockout-corrected demand, alongside markdown and waste value, judged together because cutting one by inflating the other is not a win.
Inventory days of supply or turns per category, against the service level target the planner set.
Forecast error on stockout-corrected demand, measured with a scaled error and with bias tracked separately, compared with a seasonal naive and an exponential smoothing benchmark.
Share of system orders overridden by planners, and whether the overridden orders turned out better or worse than the system's.

Human oversight

Planners own service level targets, promotion and launch forecasts, and the decision to override an order, and every override is logged so its effect can be measured. Store staff confirm a phantom inventory flag on the shelf before the record is corrected. No model changes a service level target, cancels a supplier order, or writes off stock on its own.

Common pitfalls

Training any forecast on raw sales history, so stockout days teach it that demand is low and it orders even less next time.
Reporting availability from the system's own inventory records, which counts phantom stock as on the shelf and hides the problem the project was meant to fix.
Judging a forecast on MAPE at product and store level, where most periods are zero or near zero and the metric becomes undefined or meaningless.
Replacing a planner-trusted process with a black box score and no comparison against a simple seasonal benchmark, so nobody can show the new model is better.
Setting order quantities from the point forecast alone, which guarantees stockouts on roughly half of periods for any product whose demand is symmetric around it.
Buying an AI forecasting tool to fix empty shelves whose real cause is backroom stock not reaching the shelf.

When not to use AI

Inventory record accuracy is unknown or known to be poor, so record correction will move availability more than any forecast change.
Most empty shelves have stock in the backroom, which is a store execution problem that a forecast cannot solve.
The product range is small and stable enough that reorder points and a seasonal average already hit the service level target.
Promotions and price changes are not recorded, so the drivers a model would need to learn from do not exist in the data.
The product is new or one-off with no comparable history, where planner judgement and analogues from similar products beat any trained model.

Sources

Practitioners

Demand Forecasting with Censored Likelihood, Juan Camilo Orduz, 2024
Forecasting: Principles and Practice, 3rd edition, Rob J Hyndman and George Athanasopoulos, OTexts

Research

Inventory Record Inaccuracy: An Empirical Analysis, Management Science, 2008
Desperately seeking shelf availability: an examination of the extent, the causes, and the efforts to address retail out-of-stocks, International Journal of Retail & Distribution Management, 2003
M5 accuracy competition: Results, findings, and conclusions, International Journal of Forecasting, 2022
A decision support system for detecting products missing from the shelf based on heuristic rules, Decision Support Systems, 2009

Agent context

A distilled brief for an AI agent or an engineer scoping this work. It carries the constraints and pitfalls that change an implementation decision, not the whole page.

Get the agent context as Markdown

Objective

Produce stockout-corrected demand forecasts and order quantities per product and store that raise measured on-shelf availability without increasing markdowns, waste, or days of supply, and surface inventory records that no longer match the shelf.

Constraints

Never train or evaluate on raw sales. Flag stockout periods and replace their sales with a demand estimate first, and only ever replace a lower recorded figure, since a stockout can hide demand but never add to it.
Do not trust the perpetual inventory record as ground truth. A product with stock on record and no sales for an implausible run of days is a candidate phantom stockout, not a non-selling product.
Never correct an inventory record, cancel a supplier order, or write off stock automatically. A store worker or planner confirms first.
Benchmark every forecast against a seasonal naive forecast and exponential smoothing on the same backtest periods, per category.
Use a scaled error metric and report bias separately. Do not use MAPE at product and store level.
Set order quantities from a demand quantile tied to the planner's service level target, not from a point forecast.
Respect case pack, minimum presentation stock, and shelf capacity when converting a forecast into an order.
Use loyalty or basket data only in aggregated form.

Inputs

Daily or weekly units sold and price paid per product and store.
Daily on-hand quantity, on-order quantity, and recorded out-of-stock events per product and store.
Receipts, transfers, write-offs, and count adjustments.
Promotion and price calendar, with discount depth and display placement.
Product hierarchy, case pack, shelf capacity, shelf life, store cluster, lead time, and delivery schedule.
Service level target per category, set by the planner.

Outputs

A stockout-corrected demand history per product and store, with the corrected periods marked.
Demand quantiles per product and store over the lead time and review period.
A proposed order quantity per product and store, with the on-hand, reorder point, and quantile that produced it.
A daily phantom inventory list per store, ranked by how implausible the sales gap is and truncated to the time store staff have to check it.
A per-category backtest comparing each forecast with the benchmarks, visible to planners.

Success criteria

On-shelf availability measured by shelf audit rises in the target categories against matched control stores.
Markdown, waste, and days of supply do not rise to pay for the availability gain.
Forecast bias on corrected demand is near zero per category, and the forecast beats the seasonal naive benchmark wherever it is used.
A meaningful share of phantom inventory flags are confirmed empty or found in the backroom when checked.
Planner override rate falls, and logged overrides can be compared against the system order they replaced.

Last updated 2026-10-01