This site is an early prototype of an indicator registry for human services that is being developed by LogicalOutcomes, a Canadian nonprofit.
Most of the initial content is from a collection of indicators that were developed for a two-year project led by Prosper Canada and funded by the Canadian Bankers Association (CBA) to strengthen the evaluation of financial literacy programs in Canada. The project is a collaboration between Prosper Canada, the Government of Canada (led by the Financial Consumer Agency of Canada), and CBA. LogicalOutcomes provided background research and analysis.
The first phase of Prosper Canada's project was to identify good indicators for financial literacy programs. The second phase, which is ongoing, is to develop an online evaluation toolkit that enables organizations to select outcome indicators that best match their goals and activities. Prosper Canada's evaluation toolkit can be found at www.outcomeeval.org. It addresses only financial literacy outcomes.
For more information about how we selected and rated the indicators, read about our criteria in the next section below.
The registry is built on the Aristotle Metadata Registry. Aristotle is an open-source metadata registry that follows international metadata standards as defined by the ISO/IEC 11179 specification.
Aristotle represents a new way to manage and federate content built on and extending the principles of leading metadata registries. The Aristotle Metadata Registry is a free open-source system, built upon the Django web framework and the mature model of the ISO/IEC 11179 standard.
Criteria for indicator selection
For the purposes of the Prosper Canada project on financial literacy indicators, we focused on outcome measures that could be administered in surveys or in face-to-face counselling sessions. We did not collect measures of satisfaction, organizational change, numbers served, service quality and so on. We hope to include those in the future.
In addition, we have not (yet) included comparative indicators that should be built on simple survey measures. For example, a fully defined indicator would include comparisons between pre- and post-measures, or between different populations (treated and non-treated), and would involve mathematical calculations of percentages or change scores. We started simple - with individual questions or sets of questions.
We went through three stages in selecting financial literacy indicators:
We identified potential instruments and measures through a review of the literature and interviews with leaders in the field. We considered only instruments that had credibility in terms of publication in policy or academic research, or that were being used by recognized experts in financial education programs in Canada.
We briefly screened the instruments and measures on their relevance for evaluating financial education programs serving individual Canadian adults living on low or middle incomes. In other words, we did not consider children or youth, nor population-wide interventions, nor indicators that were only relevant to high-income Canadians (defined here as those who could reasonably be expected to hire their own professional financial advisers). We asked, (a) Were they designed specifically for the evaluation of adult financial education programs or, if not (b) could they be used to evaluate financial education programs in their current form, without significantly revising the wording?
We then used six criteria to rate indicators according to their ‘goodness’ for measuring outcomes of financial education programs.
The criteria were adapted from the harmonized indicators initiative of the World Health Organization (see eRegistries: indicators for the WHO Essential Interventions for reproductive, maternal, newborn and child health) and quotes are from that document.
- Action focused
“It is clear what needs to be done to improve outcomes associated with this indicator". For example, if the indicator measures level of debt, it implies that the program should help participants reduce their debt levels. This is where a well-articulated theory of change becomes essential. If your program’s program theory includes a feeling of hopefulness and a sense that life is meaningful, you would select indicators that address mental health, and would use the resulting data to improve your program activities.
For the purposes of this project, we rejected indicators that could not meaningfully be used in pre-post designs in community agencies.
“The indicator and the data generated will make a relevant and significant contribution to determining how to effectively respond to the problem.” It is not enough that an indicator is relevant to the issue – it must be important, and there must be evidence that the data collected by the indicator will indeed help the agency or funder provide more effective services.
"The indicator is quantifiable; definitions are precise and reference standards are developed and tested or it is feasible to do so.” We did not include indicators that we found confusing or too vague to be compared across organizations. Most qualitative questions did not meet this criterion – they are often extremely useful at an agency level, but are difficult to use across organizations and thus do not address the objectives of this project. We did, however, include a qualitative question for ‘Most Significant Change’ to encourage the collection of participant perspectives about what is most important to them.
“It is feasible to collect data required for indicator in the relevant setting.” This criterion addresses the issue of usability, and includes linguistic comprehension, reading literacy, the difficulties of collecting survey data months after the program has completed, and the cost of analyzing complex indicators.
- Simple and valued
“The people involved in the service can understand and value indicator.” This criterion is often called ‘face validity’, and refers to the extent that decision-makers and stakeholders think the indicator matters. If decision-makers do not value the results of an indicator, it probably isn’t worth collecting the data.
- Open access (this criterion was added by LogicalOutcomes, and is implicit in the WHO initiative)
The indicators should be freely available to use and share across organizations without cost and without going through a process of obtaining written permission from the authors. At the very least, the indicators should be free to use for non-commercial use as long as attribution is given. Preferably there should be no restrictions for use and revisions, since many small service providers and social enterprises are not incorporated as nonprofits and would not be covered under ‘non-commercial use’. Preferably the licensing should allow derivative works (i.e., revising the wording of the survey questions) to allow for language translation.
Criteria for rating indicators
After going through the process described above, we had dozens of survey questions and measures that met the screening criteria to some extent. Our next question was how to recommend a subset that would be most helpful to users. Our advisors suggested a rating scale based on four questions:
- Has the indicator actually been used in evaluating programs?
- Has the indicator undergone cognitive and usability testing with users from the targeted respondent groups?
- Is the indicator tied to a desired high level outcome through a feasible program theory?
- Is there evidence of construct validity through an acceptable research methodology?
Prosper Canada and LogicalOutcomes team members scored each item and gave them a combined rating of 1-4, and you can see the rating on the left menu.
Next steps and future plansBased on funding from funders or NGOs, we plan to:
- Import hundreds of high quality indicators from existing collections like World Health Organization, USAID, PhenX, LOINC, the Association of Community Health Centres, Trillium Foundation and others. We will categorize them according to the 18 Sustainable Development Goals developed by the United Nations.
- Develop complex indicators like comparisons against targets, populations, pre-post measures and so on.
- Export indicators into DHIS2 and/or XLSForms (e.g., KoboToolbox) for instant implementation within a data collection tool.