AICAFMHA:
promoting mental health for young Australians
Australian Infant, Child, Adolescent and Family Mental Health Association Ltd
ABN 87 093 479 022

Working Locally With National Outcomes And Indicators For Mental Health: Making Evaluation Realistic And Reliable
Robert Bush, Maria Donald and Jo Dower, Centre for Primary Health Care, School of Population Health, University of Queensland

Background
Overview of Action Plan
Definitions of Terms
Criteria for Development
Linking the Local with the National
Conclusion
References

Background

Progressing national indicators for mental health promotion, prevention and early intervention became essential when the Second National Mental Health Plan (Australian Health Ministers, 1998) provided for developing a Mental Health Promotion and Prevention National Action Plan (Commonwealth Department of Health and Aged Care, 1998).   The 1999 Action Plan was a working document which has been revised and refined in a more recent document entitled the National Action Plan for Promotion, Prevention and Early Intervention for Mental Health 2000 (Commonwealth Department of Health and Aged Care, 2000).  The current paper is concerned with the development of the outcomes and indicators for the Action Plan (2000) and is based on a consultancy conducted for the Commonwealth Department of Health and Aged Care. The consultancy involved a national consultative process, which engaged numerous key stakeholders and built consensus around both the national level evaluation and local level evaluation of this multisectorial National Action Plan. 

Following an overview of the Action Plan (2000) the paper reviews the definitions and criteria used to develop the outcomes and indicators for the Action Plan (2000). These criteria can be applied to the development of national, strategic sector or local level outcomes and indicators. The six key strategic outcome indicators from the Action Plan (2000) are then used as an example of applying the criteria for developing valid indicators. Finally, potential mechanisms for improving the relevance and reliability of evaluation at a national level, strategic sector level and local level are reviewed.
Back to top

Overview of the Action Plan (2000)

The Action Plan (2000) is a joint Commonwealth, State and Territory initiative and the document outlines a strategic framework to meet promotion, prevention and early intervention priorities in mental health. The primary objectives of the action plan remained the same between 1998 and 2000 and are to:

Enhance mental health and social functioning among populations and individuals;

Reduce the incidence, prevalence and sequelae of mental health problems and disorders, and;

Improve the range, quality and effectiveness of public health strategies to promote mental health and prevent mental health problems and disorders among the Australian population.

Mental health promotion is defined in the Action Plan (2000) as ‘any action taken to maximize mental health and well-being among populations and individuals.’ Prevention is referred to as ‘interventions that occur before the onset of a disorder’ to reduce the incidence and prevalence.  Three types of prevention are discussed. Universal prevention is to be targeted at the general public, selective prevention is targeted at those identified as at risk of developing a mental disorder and indicated prevention at those showing signs and symptoms of a disorder. Early intervention “comprises interventions that are appropriate for and specifically target people displaying the early signs and symptoms of a mental health problem or mental disorder and people developing or experiencing a first episode of mental disorder”.   Figure 1 below shows the adapted version of the Mrazek and Haggerty (1994) framework for mental health interventions, which is used in the Action Plan (2000).

Back to top

The Action Plan (2000) defines 15 priority groups, these are:

Whole of community

Groups across the lifespan

Perinatal and infants 0-2 years

Toddlers and pre-schoolers 2-4 years

Children 5-11 years

Young people 12-17 years

Young adults 18-25 years

Adults

Older adults

Other priority populations

Individuals, families and communities experiencing adverse life events

Rural and remote communities

Aboriginal people and Torres Strait Islanders

People from diverse cultural and linguistic backgrounds

Key strategic priority groups

Consumers and carers

Media

Health professionals and clinicians

The broad spectrum of interventions and the wide range of priority groups included in the Action Plan (2000) proved to be particularly challenging in the development of outcomes and indicators.
Back to top

Definitions of Terms

In order to develop useful outcomes and indicators, consensus about the meanings of these terms is essential.  There is considerable confusion about the meaning of the terms ‘outcomes’ and ‘indicators’ among policy makers, consumers, researchers and practitioners across Australia. The confusion to some degree reflects differences in the use of these terms in research and policy. Depending on the immediate purpose, authors and planners attach these terms to a family of associated concepts and processes, sometimes making sharp distinctions, sometimes eliding them or using them interchangeably.

In a research setting, an outcome usually refers to a result, whereas in a policy context an outcome most commonly refers to what an action plan/program/project expects to achieve through implementation. The definition of an outcome provided in the Action Plan (2000) is “the anticipated benefits of promotion, prevention and early intervention activities for the identified priority group.”

Indicator is a term used in many sectors, including health, education, welfare, communications, employment, transport, environment, information technology, and local government (to name just a few). Generally, an indicator is regarded as the information used to monitor progress towards an end point. The most common definition of an indicator is information collected to monitor progress towards achievement of an outcome.  The Action Plan (2000) defines outcome indicators and process indicators. This distinction is useful because it considers not only the ends, but also the important context and process for achieving the ends.  Outcome indicators are defined as “indicators of changes in health status specific to each priority group”. This definition can also include indicators of the changes in risk and protective factors for individuals, populations, communities, organizations and environments. Process indicators are defined in the Action Plan (2000) as “measures of progress that are specific to each priority group in order to attain desired outcomes.”   This definition refers to the expected achievement in building capacity of individuals, populations, communities, organizations and environments to achieve the outcomes.

In addition to consensus about the definition of terms, it is also essential to develop validity around outcomes and indicators.  To achieve this a set of criteria have been developed which provide a quality assurance process for maximizing the validity of outcomes and indicators. 
Back to top

Criteria for Development of Outcomes and Indicators

These criteria should be considered the “gold standard” in developing outcomes and indicators and it should be recognised that while this standard may not always be achievable every effort should be made to come as close as possible to meeting these criteria. For example, the “gold standard” should not be enforced at the cost of compromising innovation.

Criteria for the development of outcomes

Criterion 1:Outcomes should be closely related to and congruent with the evidence. This criterion makes explicit the need for congruence through the available evidence to the development of outcomes, that is, actions should be evidence-based. If there is no congruence through the evidence to the outcome then successful achievement of the outcome is unlikely. This criterion relates to the evidence based for the pathways to the outcome.

Criterion 2:The level of precision of the outcomes should be considered. For outcomes to be useful, it is necessary to consider at what level the outcome is relevant. For example, the national level, the strategic sector level or the local level. It is not always possible to develop outcomes that are relevant at both national and local levels. However, it is useful if national and local level outcomes overlap at some point, so that information collected at the local level can inform both local and national initiatives, and vice versa.

Criterion 3: Outcomes should be stated clearly and concisely and have face validity to funders, project implementers, evaluators and consumers.  It is vital that outcomes are stated clearly and concisely, because they guide action at national, strategic sector and local levels. It is therefore best to reduce ambiguity by stating clearly what is to be achieved.
Back to top

Criteria for the development of indicators

The first three criteria for the development of indicators are the same as those for the development of outcomes.

Criterion 1: Indicators should be closely related to and congruent with the evidence. This criterion makes explicit the need for congruence not only between outcomes and indicators but also for the need for congruence from the available evidence through to the indicators. As such, indicators should be clearly aligned to the available evidence and the specified outcomes. Clearly, such congruence contributes to ensuring an evidence-based approach.

Criterion 2: The level of precision of the indicators should be considered. For indicators to be applied easily, it is useful to specify the level where the indicator is relevant. For example, the national level, the strategic sector level or the local level.  It is not always possible to develop indicators that are relevant at both national and local levels. However, it is useful if national and local level indicators overlap at some point, so that information collected at the local level can inform both local and national initiatives, and vice versa.

The Board on Health Promotion and Disease Prevention: Institute of Medicine, USA, (1996) (http://www.nap.edu/readingroom/books/concept/#disc) states: “…there is tension between the need for standardized indicators and the need for community flexibility in defining indicators. Standardized indicators are advantageous for making comparisons within and between communities, for simplifying the synthesis of data from different sources, and for developing data systems. However, in designing and monitoring interventions in individual communities, the development of more specific indices may be helpful and standardization may be less important.”

Criterion 3. Indicators should be stated clearly and concisely and have face validity to funders, project implementers, evaluators and consumers.  It is vital that indicators are stated clearly and concisely because essentially they guide the development and use of specific measurement tools and databases. Furthermore, these elements enhance the application of indicators by reducing ambiguity.

Criterion 4: Indicators must be sufficiently sensitive to be able to measure changes over time.  Ideally, baseline data for an indicator should exist, against which to compare change over time.  Many indicators may not be feasible because no baseline information is available.  This criterion is difficult to achieve in instances where there is low prevalence or incidence of the outcome. Take, for example, evaluation of a program designed to prevent eating disorders among young people. Such disorders are relatively rare (though of considerable significance), and it would be inappropriate to take the prevalence of eating disorders within a particular school as an indicator of the efficacy of a school-based program to prevent the onset of eating disorders. The relatively rare occurrence of the disorder means it is unlikely a considerable change in prevalence would be found in a single school. A more appropriate indicator may be a proxy measure such as self-reported body image or dieting behaviour among the high school students. Body image and restricted eating behaviours are thus regarded as precursors to the onset of eating disorders, and indicators of the efficacy of the program. This approach accepts the concept of pathways to the development of social and health outcomes, in which changes in known risk or protective factors may be regarded as indicators of effectiveness of the intervention.

Criterion 5. Indicators must be measurable and accessible. Indicators are most useful when they can use existing measurement tools and databases. If measurement tools and databases do not exist, it is necessary to balance the cost and effort involved in developing these instruments with the utility of the information the measurement tool and databases would provide. If databases are difficult to access, or the information is difficult to interpret, it is unlikely indicators drawing on such databases will be used.

Criterion 6. The cost of collecting information for the indicator must be affordable. The level of investment in resources and people that is required to apply the indicator at either the national level, strategic sector levels or the local level should be considered. This application requires some judgement about the return for effort in terms of providing information, it is important to direct the funds for evaluation towards indicators that will provide information to add to our knowledge. For example, if the evidence is strong concerning the effects of a particular program, the level of investment in evaluation may be less than when the level of evidence is limited. On the other hand, the application of psychosocial interventions across different communities and settings requires the ongoing review and adaptation to these different environments which may maintain research and evaluation costs.

Criterion 7. Prioritisation, uniqueness and comprehensiveness should be considered in the development of indicators.  Specifically, attention needs to be given to both the uniqueness of each indicator and the comprehensiveness of the set of indicators for a particular outcome. That is, it is important that each indicator provides information that is not available in any other way and that, in combination, the indicators cover the entirety of the outcome.
Back to top

Linking the Local with the National and Applying the Criteria

Table 1 shows an example of how the national, strategic sector (e.g. education, crime prevention, drug and alcohol, media) and local level outcomes and indicators can be linked.  It is important that the quality assurance criteria are considered across these three levels. While different outcomes and indicators will be required across these levels, the criteria for developing valid outcomes and indicators remain the same, regardless of the level being considered.    The National and Strategic Sector outcomes and indicators are examples taken from the Action Plan 2000.

Table 1: Making the link between National, Strategic Sector and Local level outcomes and indicators

Back to top
Barriers and Solutions in the Application of the Criteria

There are several issues that need to be addressed in order to improve the processes involved in developing and utilising valid outcomes and indicators for realistic and reliable evaluations within the Australian context.  

Accountability and Learning

In the first instance, the role of the funder could be extended from managing an accountability mechanism to responsibility for increasing the capacity to conduct valid evaluations. For example, a strong emphasis needs to be placed on learning from evaluations at the national and local level, as well as the current emphasis on accountability. It will aid the learning process if we create an environment where to report failure as well as success is acceptable and will not necessarily jeopardise future funding.  The requirement on local practitioners to evaluate their programs as a condition of grant fails to aid learning from evaluations subsequently limiting the outcomes of current evaluation practices to crude forms of accountability that are hard to validate.   As well as addressing the question, ‘Does our intervention produce the desired outcome?’, there should be more emphasis on the question, ‘What are we learning from our practice?’. A broader range of questions will lead to a deeper understanding of the purposes of evaluation.

Evaluation Infrastructure and Information Dissemination

There is a need to shift from ad-hoc funding of evaluations to a national structure that fosters, coordinates, and advances an evaluation culture for the longer term. Such a structure may broker many of the processes of evaluation (such as tendering, business planning, training, dissemination of knowledge and quality assurance). If programs are to improve their performance, and if we are to learn nationally from local evaluations, a national mechanism to collate, synthesise, and disseminate the results of evaluations would be appropriate.  These alliances are likely to insist that evaluation include mechanisms for disseminating knowledge so that a continuous learning environment is created nationally. The drivers for linking evaluation with knowledge dissemination will come from an increasing emphasis on public policy outcomes rather than specific sector outcomes. In addition, linking evaluation knowledge to information technology will be an important mechanism for achieving this strategic direction.
Back to top

Improving the Skill Base

The skill base for evaluation is patchy and locked inside particular organisations rather than widespread and at a local level, therefore there are few structures to support development of an evaluation culture.  The perception of research and evaluation as exclusive also inhibits the development of an evaluation culture. An approach to building this culture may involve working directly with tertiary institutions to encourage instruction in evaluation methods across disciplines. This also involves developing an understanding of the need to support a stronger alliance between tertiary institutions and local communities. The tertiary sector faces the issue of valuing wider sources of knowledge, including knowledge from communities. Communities face the challenge of embracing the values of tertiary education in everyday community life. Community members and academics will need to recognise the benefits of sharing their skills and knowledge. Such a partnership between tertiary eduction and communities in evaluation is likely to be encouraged through greater community involvement in curriculum development, teaching, and field practice.

Partnerships and Consideration of Local Context

A related barrier considers the growing recognition that it is difficult to develop and implement evaluations in communities without strong local support. This is especially the case in indigenous communities. Ethical demands are now placed on evaluators to consider the local context and achieve consensus from local communities about their activities.  These partnerships will mean the findings of evaluation are more easily built into local practice and will change the nature of evaluation, making it more accountable and responsive to the people and settings where interventions take place. The issue of considering local contexts in evaluation is not to challenge the importance of established methodologies supported by existing funding bodies, but how to fund broader approaches to major social and population health problems with the same determination for excellence. In some other countries, this need is addressed by establishing social and economic development research councils.

Conclusion

The relationship between national and local level evaluations is not always apparent, yet if the links can be made explicit it is likely to assist in the development of realistic and relevant evaluations both nationally and locally.  For example, if local programs can envisage how a local evaluation can contribute to the National evaluation agenda and how this in turn can contribute to the evidence-base, the process will aid in the development of commitment to evaluation and the development of an evaluation culture. Evaluation frameworks should not be such that they inhibit the effective daily operations of a program and furthermore should not be too rigid that they inhibit innovation.  A balance must be struck between innovation, scientific rigour and flexibility.
Back to top

References

Australian Health Ministers (1998). Second National Mental Health Plan, Canberra: Mental Health Branch, Commonwealth Department of Health and Family Services

Board on Health Promotion and Disease Prevention: Institute of Medicine (1996).  Using Performance Monitoring to Improve Community Health: Conceptual Framework and Community Experience. http://www.nap.edu/readingroom/books/concept/#disc

Commonwealth Department of Health and Aged Care (2000). National Action Plan for Promotion, Prevention and Early Intervention for Mental Health 2000. Mental Health and Special Programs Branch, Canberra.

Commonwealth Department of Health and Aged Care (1998). Mental Health Promotion and Prevention National Action Plan. Promotion and Prevention Section. Mental Health Branch, Canberra.

Mrazek, PJ & Haggerty, RJ. (1994). Reducing Risks for Mental Disorders: Frontiers for Preventive Intervention Research. National Academy Press, Washington DC

Back to top

Back to papers index

Last Modified: 27-11-2002 10:11:02