June-July 2017: With this discussion topic, we wanted to explore options for triangulating data to better inform immunization service delivery planning from a program perspective and how to link it to coverage rates.
This discussion sparked a lot of ideas about how to get more accurate target population numbers, data accuracy and use for better forecasting, key performance indicators, vaccine wastage rate, and even options for a better performing supply chain such as changing vial size to reduce wastage.
A few key themes that emerged are summarized here:
1. There was general agreement that demographic data used to set target populations for the immunization program are oftentimes inaccurate, particularly at the sub-national level, due to mobile populations, inconsistent growth across a country, out-of-date information, or the information is shared too late or doesn’t have the level of detail needed for catchment populations. These data may be generally acceptable when forecasting national level vaccine need; however, for sub-national vaccine distribution and resupply, inaccurate target populations can lead to stockouts or oversupply.
2. WHO describes three ways of forecasting vaccine need: population based, consumption based, and session method. Population based is most often used because of the lack of accurate data from health facilities on consumption and/or session sizes. It’s also important to recognize that the population base used for forecasting is also used for coverage estimates, and coverage rates are how a MOH and partners define success.
3. Many respondents to this discussion noted the problem of getting accurate data from health facilities. Because of this, there is an over-reliance on demographic data for forecasting. There seemed to be group consensus that improving data accuracy should be a priority. This can pave the way to use different forecasting methodology for more accuracy.
4. It was noted that as more countries begin using logistics management information systems (LMIS), data accuracy should improve. That should also go hand-in-hand with the need to adapt norms and standards for EPI. The Visibility and Analytics Network (VAN) was mentioned as a new standard for continuous improvement for supply chain management.
5. A general agreement from the group was that a good practice is to use a mixed method for forecasting based on different data sets and then to reconcile for best accuracy. It was duly noted that any forecasting method needs to be adapted to the local context and, by extension, to sub-national levels as well. Contributors also agreed that regular review and updates of forecasts are helpful; one example from a Family Planning program in Nigeria described systemic review of data that helped improve forecast accuracy.
6. Other ideas for getting more accurate population estimates included using a birth registry in the catchment area, local administrative data, other project data (such as RED/REC), and even high resolution satellite imagery that would feed back into the government system to then link to the target population for coverage estimates.
7. Wastage rate sparked a lot of interest as well from this group. It seems there is not enough data being collected to completely understand open vial wastage. Recommendations for improvement included more detailed data collection that would shine light on true wastage. Triangulation of data sources could also be a methodology to use. A mathematical model to optimally plan sessions is also available, although no one would want to plan sessions at the expense of vaccinating children.
8. There was a lively discussion on KPIs and the DISC Guidelines (http://www.technet-21.org/iscstrengthening/index.php/en/data-for-management-documents-and-downloads/indicator-reference-sheets). Not all respondents were satisfied with these KPIs. A couple of the KPIs tend to overlap with each other, and none of the KPIs would address the issue of maintaining stock availability at the expense of vaccinating children.
9. As was noted, there is a fundamental trade-off between cost and availability. It was suggested that cost should be included as a KPI in order to get a balanced scorecard. This should also be linked to coverage, not as an indicator for supply chain specifically but as an indicator of the success of EPI.
10. One suggestion from our colleague from Conakry to reduce costs is to look at the dose per container. In certain situations, smaller vial size for some vaccines could reduce wastage. This could also have an impact on coverage as more health workers are willing to open a vial for all children who present.
11. The final key point from this discussion is related to leadership and governance and the importance of leaders and decision makers being willing to change. If different forecast methodologies or triangulation of data need to be applied at sub-national level in order to have more accurate forecasts, that needs to be recognized by national level leaders in order to feedback into planning, national forecasts, and coverage estimates.
And finally, the answer to a fun question: What is the difference between an accurate forecast and the Loch Ness monster? The Loch Ness monster has actually been seen!