Quantification of Health Commodities

May-June 2017: A collection of experts led the discussion on the quantification of health commodities. The following topics were moderated by different groups of experts:

1. Country Experiences. Moderated by Wambui Waithaka, regional technical advisor with JSI and Gilbert Mateshi, program manager with Clinton Health Access Initiative.

2. Country Processes and Tools. Moderated by Irene Agyemang, team lead for public health programs under the recently ended USAID | DELIVER PROJECT (Ghana) and Sami Tewfik Edris, deputy country director for JSI.

3. Commercial Sector Approaches – moving toward “big data.” Moderated by Alfons Van Woerkom from the Global Fund and Attila Dobi, a data scientist with Zenysis Technologies.

4. Performance Management. Moderated by Noel Watson PhD, founder of OPS MEND and Stephanie Buscher, analytics consultant at General Mills.

Summary of the discussions:

Ways various countries have approached quantification:

• Generally there are 2 methods used in quantification: forecasts based on population/demographic data and/or consumption/service data.
• Often more than one quantification workshops are held a year with multiple partners involved.
• Some countries have quantifications performed at the county levels while others are at the national level in a centralized manner.
• Some products, like LLINs, were done in a decentralized fashion, which is based on malaria commodity micro-planning exercises done at district level.
• Some countries will also compare logistics data: those from quarterly reports and purchase orders and then readjust the data as needed.
• Sometimes commodities are not nationally forecasted, but separately by each project/program who fund them.

Good quantification is important because:

• Effective quantification results in adequate stocks availability, improved services, and enhanced quality of life for patients.
• Resources for health commodities are finite and should be used prudently to avoid wastages and stock outs that grossly affect health outcomes.

Roles of the Ministry of Health (MOH) in quantification for countries include:

• Coordinating the quantification exercise with the support from the involved partners.
• Estimating and quantifying the annual need
• Presenting the needs and asking if partners can contribute to purchasing the health inputs during committee meetings
• Providing future program targets and anticipating programmatic changes influencing future demand for commodities.

Some quantification challenges faced in different countries and how to overcome them:

• Over-forecasted needs with the population data.
  • Combine the consumption approach where data exists for products.
• The supply plan doesn’t exist.
  • Support the organization/agency to create the supply plan as a result of quantification exercise.
• Products are not forecasted nationally, but each stakeholder does separately; the MOH doesn’t coordinate the process.
  • Support the MOH in coordinating the quantification efforts nationally and to the provinces—first at provincial level, then the national level to synthetize.
• Quantification has not been institutionalised at the council/facility levels. Often it is guided by the available budgets and follows a top down approach—it is not demand driven.
  • The MOH and donor partners should instituionalise the processes and it should be changed to follow a bottom up approach so that it can reflect the actual demands.
• Lack of appropriate systems, such as the LMIS, at the peripherals to adequately generate and process quantification data for the national level. This results in stockouts and shortages.
• Availability of data and lack of adequate bookkeeping records at the service delivery points (SDPs) making it difficult to get accurate data.
  • Adoption of quasi-LMIS and scientific methods of data collections—e.g. requesting SDPs to flip through records to collect data with strict monitoring; using mobile technology to send the data.
• Poor quality of source data at the facility level. Average monthly consumption (AMCs) are inaccurately calculated in many facilities.
  • Electronic dispensing systems with modules to assist in AMC calculations.
  • Support facilities through training and supervision to ensure they get AMC data right.
• Frequent industrial actions has affected staff morale and in turn affected quantification.
• Seasonal products (e.g. malaria products); overstocks in the dry season and stockouts in the rainy.
  • Develop a seasonality index to obtain consumption that accounts for seasonal variation.

What members hope to learn from others experienced in quantification:

• To learn from others experiences with quantification and their good practices to inform ways to improve within my own country.
• To learn more from the commercial sector: more advanced forecasting techniques and tools.

What strategies have you pursued to improve the availability and quality of data used for forecasting and supply planning? How can we work with governments to invest in reliable master data processes and data collection/storage infrastructure?

• Data issues are often associated with promptness and completeness; after field visits, follow up on the recommendations are often lacking. A solution was to recruit some focal points at the rate of one per district; they facilitate the follow-up work by supporting and accompanying the executive teams of the district, but do not do this in the executive team’s place.
• Ensure the staff at different levels of the pyramid are trained and equipped to facilitate the work.
• The more that governments utilize data in making rapid decisions, the more they are willing to invest in the data acquisition pipeline. It has been helpful working directly with the ministry on the ground with the analysis and the directors. You must explain why data should be collected in the first place; value may not immediately obvious.

How do you put the health consumer at the center of your planning process – i.e., mapping the expectations from the consumer in your country/region, and aligning your plans?

• To do this, we must connect care in the community. The community agents need to take ownership of what we do as well as the CBOs and CGSH.

What challenges remain even if data quality and visibility improve? What are some potential solutions?

• The biggest challenge will be the sustainability of the achievements. As long as the actors of ministries do not themselves take ownership of the activities and their motivation depends on our contribution, we will always return to the beginning.

Other notes:

• Regarding pediatric ARV forecasting in particular – a challenge often is obtaining a national dataset of weight-band distribution data that is representative of the country’s current pediatric HIV treatment cohort. Previously used WHO reference data on weight band distribution data no longer seems to adequately represent the national dosing and formulation needs in many African countries in particular. This is likely because globally, the pediatric HIV cohort is aging as we move closer to elimination – which means that kids requiring ARV treatment are getting heavier.

• A potential support to data collection can be to look at baselines from the last few years, assuming that if all things stayed equal, you should have seen a repetitive demand curve; if your demand increased or decreased, certain events have happened. By understanding if these events happened nationally or if these events happened, this split can help you understand the causes of your ‘events’ without actually needing all the local patient data.
  Once you understand the drivers of your events, you can then determine that with low data availability your best resolution.

• Many times errors are made when transcribing notes from hand to spreadsheets/databases. For an example of a simple algorithm that uses to flag outliers based on historical data:
  https://docs.google.com/presentation/d/1w13rmeqAAGl8i4iXgSF_Qu2SE2-yYubU1UXt1z5Ck38/edit?usp=sharingWe have worked on malaria forecasting based on infection data from the MoH, other programatic indicators, economic indicators and satellite weather information. We’d like to deploy generalized models that can adapt to any outcome variable and deal with the hierarchical structure in the data. An example of what we have been working on:
  https://docs.google.com/presentation/d/1r2u0TVdvdRfVBzwrzyrYQjUSF1Mi_vjJwK_C_zadgXs/edit?usp=sharing
  An article comparing contrasting time series forecasting methods:
  https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-15-276
  Article on forecasting malaria rates in Uganda:
  https://malariajournal.biomedcentral.com/articles/10.1186/s12936-015-0758-4Steps for model building:
  1. Identify important predictors (anything at all)
  2. Gather ‘good’ historical data for training/testing/validation
  3. Deploy and validate continuously as new data become available.
  4. Continually learn from the incoming data, retrain and validate as new information becomes available.

  Some models to consider for your forecasting given multiple time series.
  1. ARIMA
  2. Random forest regressors.
  3. Linear regression with ridge/lasso regularization (It is very important to consider regularize the models to prevent overfitting to the data).
  4. LLSTM (if you have a lot of data, deep learning may be useful)

How often do you monitor your forecasts and supply plans? How and why? How often do you make changes? Who leads this?

• At General Mills, they measure forecast accuracy with a monthly process where the sales, finance, marketing, and supply chain teams meet to align on a single forecast and to triangulate the right demand; blended forecasting is a proven method for increasing forecast accuracy. Forecast changes are most often made during the monthly meeting to drive location-specific changes. This process was similar to the process for the Nigerian Federal Ministry of Health Family Health Department and the USAID | DELIVER PROJECT for reviewing consumption, updating forecasts and the pipeline estimates, and coordinating donor procurement plans and new shipments.

Why is it important to take steps to improve forecasting?

• Any forecast will inherently be wrong and forecast figures need a statement of variability. Because of the inherent flaws in forecasting, the use of flexible buffers as defined in Demand Driven Material Requirements Planning (DDMRP) can be used.

What are some forecast accuracy methods you have attempted and what results did you achieve? What are other metrics?

• People often talk about forecast “accuracy,” but it would be helpful to have more specific metrics, working definitions, and data requirements to enable quality improvement in forecasting and supply planning, etc.
• One of many measures of forecast accuracy is MAPE (mean absolute percentage error). It is calculated by taking the absolute difference between the forecast and the actual (absolute means without regard to direction) and dividing it by the actual demand or consumption.
• In calculating MAPE, a challenge is ensuring the completeness and quality of consumption data. Fortunately, countries have progressively become more diligent in monitoring their consumption data once they started looking at forecast accuracy on a regular basis.
• A question was raised about whether product characteristics have influence on forecast accuracy, assuming there are no biases or blind spots and that a strengthened LMIS is in place.

What are the sources of bias that have been identified in your forecasts? Has over-forecasting ever cost you valuable warehouse space or under forecasting hindered your ability to serve? How could you make up for “forecast inaccuracy”? (Inventory, faster transit times, etc.)

• One thing that MAPE does not cover is bias or trend. In the health industry, this may mean not being treated or not having the product where and when it’s needed.
• A challenge in many places will continue to be obtaining complete, timely, and accurate consumption data in order to generate the forecast indicator.
• A challenge is often that new consumption data does not come in monthly or regularly and the budget and procurement control are in HQ offices instead of the field offices. Because of this, there is more concern that funding is made available instead of whether it is used effectively. Additionally, because of constraints on funding, there is often overstock of some products and shortages in others.

Once you’ve analyzed your data and settled on a forecast, how do you best use that data to make decisions (e.g., procurement, inventory, supply, warehousing, transportation, budget decisions)?

• General Mills uses both the forecast and historical accuracy to negotiate contracts with external manufacturing facilities, transportation, and warehousing. For inventory, standard inventory calculation techniques are used for safety stock: demand, forecast accuracy, lead time, and lead time variance to calculate the full variability over our lead times. Then based on our promised service level to customers, we determine what percent to cover. Our service level is dictated by both our customers and our leadership.

 

What do you think is the best way to calculate service for health companies? Would you set a standard level (97%) or would you try to balance costs of shortage vs excess? Could what is acceptable change depending on what kind of product it is?

• In a health setting, not having a particular commodity could mean a question of life or death, so universally, program managers most often strive for 100% service levels; however, this may ignore funding realities and physical constraints. A reasonable approach would be to segment the product list into life-saving, essential, and non-essential commodities and strive for different levels depending on the necessity of the product. The challenge would be that you now have to manage multiple inventory rules, and many would prefer to manage to a single rule because managing multiple rules requires LOE and supporting systems.
• Not many of the settings we work in use the total cost formula, or have accurate estimates of the components for determining SC parameters (total cost = purchase costs+ ordering costs + holding costs + shortage costs). This formula helps determine the cost of providing various customer service levels (CSLs). Without knowing each portion, determining your inventory policies and therefore the CSL, becomes difficult. It is especially difficult to calculate the cost of a shortage. Should we as a community look to using health economic terms like quality-adjusted life years (QALYs) or other similar terms along with inventory costs to determining our CSL?