The 5 steps to survey design
How to create SMART indicators
Megan Borole and Tyryn Carnegie
The SMART framework is a useful way to identify quality indicators. It stands for Specific, Measurable, Achievable, Relevant and Time-bound. The first criterion, Specific, means that the indicator needs to be narrow and accurately describe what needs to be measured. Measurable means that regardless of who uses the indicator it would be measured in the same way. Achievable (or attainable) means that collecting the data should be straightforward and cost-effective. Relevant requires that the indicator be closely linked to the relevant outcome. Finally, Time-bound means that there should be a timeframe linked to the indicator (such as the frequency with which it is collected or measured).
Indicators are an essential part of quantifying your impact and conducting effective monitoring and evaluation. We are focusing today on quantitative indicators and will soon share a post on qualitative indicators.
To identify our SMART indicators, we start with the Theory of Change which is our essential roadmap to impact. Check out our blog post for 5 easy steps to creating your Theory of Change. Our next step is to map indicators to the Theory of Change and use these indicators to track progress and evaluate impact. Let's check out an example of how to create SMART indicators, and the pitfalls to watch out for.
Checking if an indicator is SMART
We will use a Theory of Change from an initiative to improve community health through new cooking stoves. A shortened Theory of Change and 2 examples of potential indicators are shown in the table.
Are these good indicators? Remember, our indicators need to be Specific, Measurable, Achievable, Relevant and Time-bound.
Both indicators score fairly well on the following:
-
Specific: The location of the households is fairly well specified ('in the community') however we need to be careful as to what constitutes a part of 'the community', and whether all researchers would delineate this consistently. A 'smokeless stove' is specific as it indicates zero smoke can be emitted. If the indicator had said 'low-smoke' this would have been more ambiguous as it leaves it open for interpretation as to what constitutes low or high smoke emission. The air pollution indicator is very specific as it quantifies the exact number of 'particle micrograms' that are deemed too high.
-
Measurable: The indicators provide clear instructions on how to be measured. A percentage is easy to measure (count the total households using smokeless stoves in the community, divided by total households, multiplied by 100). Similarly, 'current households' defines which households are to be included (only households that exist at the time of the survey). Finally, the item to be measured is clearly defined ('smokeless stoves', and number of 'particle micrograms' in the air).
-
Relevant: Both indicators are closely related to the Outcome/Output they need to measure. 'Smokeless stoves' are an improvement on smoky ones with regards to health, and the number of 'particle micrograms' are a direct measure of 'air pollution'.
-
Time-bound: We are looking at 'current households' that existed at the time of the survey, which is a time-bound measure. However, it would be useful to specify regularity with which we will measure the indicators (e.g. annually, bi-annually).
The major difference between the two indicators is their Achievability. Measuring the number of particle micrograms in the air may be expensive, requiring household visits and costly measuring equipment, whilst ascertaining which households have smoke-free stoves should require only the household visits. The first indicator is likely to be more achievable therefore.
Creating useful indicators often requires compromises (particularly because many non-profits face significant cost constraints) and this may mean that we cannot collect data on all indicators in all time periods for our monitoring and evaluation.
Indicators are not targets
Note that indicators should not specify a particular level of achievement. We shouldn’t be using the words like ‘improved’, ‘increased’ or ‘decreased’ when setting indicators. As an example, 'Increased % of households using smokeless stoves' would not be considered an indicator, rather an Outcome. Get in touch if you are confused between indicators and targets, it can be confusing!
Global examples of SMART indicators
It's useful checking out some global indicators to see how they have used the SMART framework. Let's look at one of the Millennium Development Goals - 'Eradicate extreme poverty and hunger', and the associated outcome is to 'Halve poverty' within a specified time period.
Measuring the proportion of people with income less than a dollar a day requires a clearly defined indicator that will guide people all over the world to measure the outcome in the same way. This will ensure that all relevant income sources are included, that the measure is assigned to all relevant individuals, and measured accurately in each time period. The indicator above provides a clear definition of how to measure this, and scores well on the SMART framework. Check it yourself to see if it is Specific, Measurable, Achievable, Relevant and Time-bound.
Other well-known examples of indicators include business Key Performance Indicators, country GDP growth rates and unemployment levels.
The importance of indicators
Indicators are an important M&E tool across the lifespan of a programme. At the beginning of a programme, indicators are important to define how the programme will be measured. Through indicators, managers are able to pre-determine how effectiveness will be evaluated in a precise and clear manner. Indicators also inform what data should be collected over the course of the programme and how best to do so, allowing managers to continually assess progress and highlight areas for possible improvement. Lastly when the programme is being evaluated, indicators allow evaluators to assess programme impact.
Now that we know what good indicators are and why they are important, check out this comprehensive guide:
Feel free to get in touch if you are defining indicators in your organisation and want some help!