The Health Research Economics (HRE) team is leading research impact assessment (RIA) in health and medical research (HMR) in Australia. What sets our approach to RIA apart is FAIT (Framework to Assess the Impact of Translational health research).
Research impact in health involves changing knowledge, health policy, health systems and clinical practice with the aim of improving the health and wellbeing of society and the economy.
Despite major investments in research and development, relevant research findings are not always being adopted by healthcare systems or being used by end users like policy makers, health services or the public. Suboptimal translation means return on research investments is lower than it could be.
There is growing recognition that translation of research into policy and practice needs to increase and the pathways to impact need to be more transparent. [McKeon, 2013] In Australia developments in this space have been led by the National Health and Medical Research Council, the Australian Research Council, and the Medical Research Future Fund but a national RIA framework has not yet been adopted. However, as funders seek to maximise returns on their research investments, measurement of research impact is likely to become mandatory rather than optional.
The Framework to Assess the Impact from Translational health research (FAIT)
FAIT is a novel framework specifically developed by the team to encourage research translation and measure and report on research impact in a multi-dimensional way.
FAIT was developed using a mixed methods approach involving:
- a scoping review of existing research impact frameworks and techniques to inform the development;
- a development stage to design the prototype of FAIT;
- a feedback stage where versions of FAIT were presented to a committee of researchers to gain views and suggestions on how FAIT could be improved
- a pilot and evaluation phase where HRE staff are implementing FAIT within different settings to
- validate its ability to encourage translational activities and behaviours
- assess its effectiveness in reporting research impact
- determine the resources required for its implementation
FAIT is a hybrid of three proven methodologies for measuring research impact, namely a modified Payback method, an economic analysis, and a narrative of the process by which research translates and generates impact. The main strength of FAIT and its point of difference from other frameworks is its combination of three methods and impact perspectives allowing for triangulation of the findings. The below figure shows the three key components of FAIT.
Modified payback
The modified payback framework is based on domains of benefit such as knowledge impacts, health service impacts, and political and policy impacts and quantifying impacts for each domain. Data is collected via semi-structured interviews with researcher, administrative data, bibliometric analysis and verification studies.
Economic analysis
Out of the suite of potential economic evaluation techniques, cost-benefit analysis (CBA) is the most appropriate tool for impact assessment. The reportable metric of CBA is a ratio of benefit per dollar of cost, or a ‘return on investment’. Where a CBA is not viable, a cost-consequence analysis may be used.
Narratives
The third method is based on narratives of how translation occurred and how research impact was generated. These are supported with evidence from the modified payback and economic analysis and explain variations in research costs, outputs and impacts.
Modified program logic model
The assessment methods are underpinned by a modified program logic model.
The logic model identifies
- the demand being addressed by the research program;
- the aims of the research program;
- the research activities being supplied to meet the ‘demand’;
- the expected research outputs;
- the end-users of those research outputs;
- the anticipated intermediate and final impacts when end users use the research outputs.
The logic model also emphasises the involvement of end users from the start.
FAIT was deliberately designed to address the following limitations of other research measurement tools and frameworks:
Limitation: Research impact assessment on its own is insufficient to increase impact
Research translation is a prerequisite for research impact. FAIT was designed to incentivise and assist researchers understand, plan for, and implement processes to increase the likelihood of research translation.
Limitation: Research impact assessment does not enable the implementation of improvement processes when research translation fails
Embedded within FAIT is a module of process metrics based on monitoring and feedback principles that are separate to the measures of impact. Feedback allows research leaders/ managers to evidence translational activities on the pathway to impact and assess whether the implementation of these activities is appropriate or whether they require attention.
Limitation: Research impact assessment can be costly and difficult
FAIT is designed to be implemented prospectively so cost-effective data collection techniques can be implemented. Collecting data retrospectively is often not feasible or affordable.
Limitation: Research impact assessment findings can be difficult to communicate
FAIT brings different perspectives to understanding and measuring research impact. It combines quantitative and qualitative measurement techniques that facilitate communication on research impact by expressing impact using quantified metrics, economic measures and narratives.
The following publications by members of the HRE team are useful for potential FAIT users who require more information about the development of FAIT, the different methods used in FAIT and how it is being applied in the real world.
Seminal FAIT paper – everything you need to know about the Framework
An Approach to Measuring and Encouraging Research Translation and Research Impact
Scoping literature review of the objectives for impact assessment
Measuring research impact in Australia’s medical research institutes: a scoping literature review of the objectives for and an assessment of the capabilities of research impact assessment frameworks
Attitudes towards research impact assessment frameworks
Measuring research impact in medical research institutes: a qualitative study of the attitudes and opinions of Australian medical research institutes towards research impact assessment frameworks.
Protocol for the implementation of FAIT to five Indigenous primary health research projects
Encouraging translation and assessing impact of the Centre for Research Excellence in Integrated Quality Improvement: rationale and protocol for a research impact assessment.
Protocol for the implementation of FAIT to a childhood obesity prevention program
Measurement of the translation and impact from a childhood obesity trial programme: rationale and protocol for a research impact assessment.
Protocol for the implementation of FAIT to streams of stroke rehabilitation research
Implementing a protocol for a research impact assessment of the Centre for Research Excellence in Stroke Rehabilitation and Brain Recovery
Researcher opinions of research translation – it’s scope, importance, responsibility and skills and knowledge required
This publication presents results from a mixed-methods study to explore the opinions of research translation held by researchers working in a Centre of Research Excellence in Australia.
Strengthening and measuring research impact in global health: lessons from applying the FAIT framework
This paper presents the results from applying FAIT to two public health projects in low to middle income countries and the learnings and limitations of applying FAIT retrospectively.
The following resources are available for those interested in applying FAIT. Both are designed to be applied upfront, at the planning stages of a project.
The Program Logic Template is designed to assist users to map out the intended pathway to impact. It includes a step-by-step process for developing a program logic model for a research project from demand all the way through aims, activities, outputs, potential users and impacts.
Click here to download the Program Logic template
The Research Translation Planning Template is designed to assist users to develop a research translation plan. This activity builds on the development of the program logic model by expanding on the types of translational activities that will be required to achieve the intended impacts. It also helps to identify the skills and resources required to implement the research translation aspects upfront
Click here to download the Research Translation Planning template
This section contains pdf versions of powerpoint slides from a selection of poster and presentations about FAIT and Research Impact Assessment at a range of conferences:
- NHMRC Research Translation Symposium 2015
- AAMRI Conference 2016
- NHMRC Research Translation Symposium 2016
- Primary Health Care Conference 2017
- Measuring Research Outcomes Conference 2017
- Primary Healthcare Conference 2018
- Stroke 2018/Smart Strokes Australia Conference 2018
- NHMRC Research Translation Symposium 2018 - presentation 1
- NHMRC Research Translation Symposium 2018 - presentation 2
- The Lowitja Institute International Health and Wellbeing Conference 2019
This section provides links to resources on research impact assessment and research translation that could potentially be useful to projects that are implementing FAIT including two NHMRC Centres of Research Excellence.
Research Impact
Dr Shanthi Ramanathan
Melanie Rao
Dr Kurtis Budden
