2009). For example, the development of a spin out can take place in a very short period, whereas it took around 30 years from the discovery of DNA before technology was developed to enable DNA fingerprinting. Overview of the types of information that systems need to capture and link. evaluation of these different kinds of evaluands. Metrics in themselves cannot convey the full impact; however, they are often viewed as powerful and unequivocal forms of evidence. Where narratives are used in conjunction with metrics, a complete picture of impact can be developed, again from a particular perspective but with the evidence available to corroborate the claims made. There are a couple of types of authorship to be aware of. The justification for a university is that it preserves the connection between knowledge and the zest of life, by uniting the young and the old in the imaginative consideration of learning. To understand the method and routes by which research leads to impacts to maximize on the findings that come out of research and develop better ways of delivering impact. This atmosphere of excitement, arising from imaginative consideration transforms knowledge.. The quality and reliability of impact indicators will vary according to the impact we are trying to describe and link to research. Despite the concerns raised, the broader socio-economic impacts of research will be included and count for 20% of the overall research assessment, as part of the REF in 2014. When considering the impact that is generated as a result of research, a number of authors and government recommendations have advised that a clear definition of impact is required (Duryea, Hochman, and Parfitt 2007; Grant et al. Definitions of Evaluation ( by different authors) According to Hanna- "The process of gathering and interpreted evidence changes in the behavior of all students as they progress through school is called evaluation". Given that the type of impact we might expect varies according to research discipline, impact-specific challenges present us with the problem that an evaluation mechanism may not fairly compare impact between research disciplines. (2007) surveyed researchers in the US top research institutions during 2005; the survey of more than 6000 researchers found that, on average, more than 40% of their time was spent doing administrative tasks. 0000007559 00000 n Understanding what impact looks like across the various strands of research and the variety of indicators and proxies used to evidence impact will be important to developing a meaningful assessment. The aim of this study was to assess the accuracy of 3D rendering of the mandibular condylar region obtained from different semi-automatic segmentation methodology. However, there has been recognition that this time window may be insufficient in some instances, with architecture being granted an additional 5-year period (REF2014 2012); why only architecture has been granted this dispensation is not clear, when similar cases could be made for medicine, physics, or even English literature. Attempts have been made to categorize impact evidence and data, for example, the aim of the MICE Project was to develop a set of impact indicators to enable impact to be fed into a based system. To enable research organizations including HEIs to monitor and manage their performance and understand and disseminate the contribution that they are making to local, national, and international communities. The Social Return on Investment (SROI) guide (The SROI Network 2012) suggests that The language varies impact, returns, benefits, value but the questions around what sort of difference and how much of a difference we are making are the same. In endeavouring to assess or evaluate impact, a number of difficulties emerge and these may be specific to certain types of impact. Collecting this type of evidence is time-consuming, and again, it can be difficult to gather the required evidence retrospectively when, for example, the appropriate user group might have dispersed. Husbands-Fealing suggests that to assist identification of causality for impact assessment, it is useful to develop a theoretical framework to map the actors, activities, linkages, outputs, and impacts within the system under evaluation, which shows how later phases result from earlier ones. In the educational context, the . Downloadable! In terms of research impact, organizations and stakeholders may be interested in specific aspects of impact, dependent on their focus. Without measuring and evaluating their performance, teachers will not be able to determine how much the students have learned. 0000011585 00000 n Research findings including outputs (e.g., presentations and publications), Communications and interactions with stakeholders and the wider public (emails, visits, workshops, media publicity, etc), Feedback from stakeholders and communication summaries (e.g., testimonials and altmetrics), Research developments (based on stakeholder input and discussions), Outcomes (e.g., commercial and cultural, citations), Impacts (changes, e.g., behavioural and economic). Evidence of academic impact may be derived through various bibliometric methods, one example of which is the H index, which has incorporated factors such as the number of publications and citations. The risk of relying on narratives to assess impact is that they often lack the evidence required to judge whether the research and impact are linked appropriately. What is The Concept of Evaluation With its Importance? 2007). 0000009507 00000 n What emerged on testing the MICE taxonomy (Cooke and Nadim 2011), by mapping impacts from case studies, was that detailed categorization of impact was found to be too prescriptive. The case study does present evidence from a particular perspective and may need to be adapted for use with different stakeholders. What is the Difference between Formative and Summative Evaluation through Example? Citations (outside of academia) and documentation can be used as evidence to demonstrate the use research findings in developing new ideas and products for example. Oxford University Press is a department of the University of Oxford. While the case study is a useful way of showcasing impact, its limitations must be understood if we are to use this for evaluation purposes. Teresa Penfield, Matthew J. Baker, Rosa Scoble, Michael C. Wykes, Assessment, evaluations, and definitions of research impact: A review, Research Evaluation, Volume 23, Issue 1, January 2014, Pages 2132, https://doi.org/10.1093/reseval/rvt021. A comprehensive assessment of impact itself is not undertaken with SIAMPI, which make it a less-suitable method where showcasing the benefits of research is desirable or where this justification of funding based on impact is required. In viewing impact evaluations it is important to consider not only who has evaluated the work but the purpose of the evaluation to determine the limits and relevance of an assessment exercise. Differences between these two assessments include the removal of indicators of esteem and the addition of assessment of socio-economic research impact. 2007; Nason et al. RAND Europe, Capturing Research Impacts. 0000001178 00000 n Cb)5. From the outset, we note that the understanding of the term impact differs between users and audiences. , . Scriven (2007:2) synthesised the definition of evaluation which appears in most dictionaries and the professional literature, and defined evaluation as "the process of determining merit, worth, or significance; an evaluation is a product of that process." . For full access to this pdf, sign in to an existing account, or purchase an annual subscription. 8. It has been acknowledged that outstanding leaps forward in knowledge and understanding come from immersing in a background of intellectual thinking that one is able to see further by standing on the shoulders of giants. Frameworks for assessing impact have been designed and are employed at an organizational level addressing the specific requirements of the organization and stakeholders. Throughout history, the activities of a university have been to provide both education and research, but the fundamental purpose of a university was perhaps described in the writings of mathematician and philosopher Alfred North Whitehead (1929). The development of tools and systems for assisting with impact evaluation would be very valuable. Definition of evaluation. The book also explores how different aspects of citizenship, such as attitudes towards diverse population groups and concerns for social issues, relate to classical definitions of norm-based citizenship from the political sciences. One of the advantages of this method is that less input is required compared with capturing the full route from research to impact. Authors from Asia, Europe, and Latin America provide a series of in-depth investigations into how concepts of . Collating the evidence and indicators of impact is a significant task that is being undertaken within universities and institutions globally. To demonstrate to government, stakeholders, and the wider public the value of research. (2007) adapted the terminology of the Payback Framework, developed for the health and biomedical sciences from benefit to impact when modifying the framework for the social sciences, arguing that the positive or negative nature of a change was subjective and can also change with time, as has commonly been highlighted with the drug thalidomide, which was introduced in the 1950s to help with, among other things, morning sickness but due to teratogenic effects, which resulted in birth defects, was withdrawn in the early 1960s. The point at which assessment takes place will therefore influence the degree and significance of that impact. It is concerned with both the evaluation of achievement and its enhancement. What are the challenges associated with understanding and evaluating research impact? Muffat says - "Evaluation is a continuous process and is concerned with than the formal academic achievement of pupils. %PDF-1.4 % They are often written with a reader from a particular stakeholder group in mind and will present a view of impact from a particular perspective. This article aims to explore what is understood by the term research impact and to provide a comprehensive assimilation of available literature and information, drawing on global experiences to understand the potential for methods and frameworks of impact assessment being implemented for UK impact assessment. % Worth refers to extrinsic value to those outside the . To be considered for inclusion within the REF, impact must be underpinned by research that took place between 1 January 1993 and 31 December 2013, with impact occurring during an assessment window from 1 January 2008 to 31 July 2013. Many theorists, authors, research scholars, and practitioners have defined performance appraisal in a wide variety of ways. The exploitation of research to provide impact occurs through a complex variety of processes, individuals, and organizations, and therefore, attributing the contribution made by a specific individual, piece of research, funding, strategy, or organization to an impact is not straight forward. In the UK, the Russell Group Universities responded to the REF consultation by recommending that no time lag be put on the delivery of impact from a piece of research citing examples such as the development of cardiovascular disease treatments, which take between 10 and 25 years from research to impact (Russell Group 2009). Impact can be temporary or long-lasting. A Preferred Framework and Indicators to Measure Returns on Investment in Health Research, Measuring Impact Under CERIF at Goldsmiths, Anti-Impact Campaigns Poster Boy Sticks up for the Ivory Tower. Introduction, what is meant by impact? 0000010499 00000 n To understand the socio-economic value of research and subsequently inform funding decisions. (2005), Wooding et al. Indicators, evidence, and impact within systems, Department for Business, Innovation and Skills 2012, http://www.arc.gov.au/pdf/ERA_Indicator_Principles.pdf, http://www.charitystar.org/wp-content/uploads/2011/05/Return_on_donations_a_white_paper_on_charity_impact_measurement.pdf, http://www.oecd.org/science/innovationinsciencetechnologyandindustry/37450246.pdf, http://www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf, http://mice.cerch.kcl.ac.uk/wp-uploads/2011/07/MICE_report_Goldsmiths_final.pdf, http://www.timeshighereducation.co.uk/story.asp?storyCode=409614§ioncode=26, http://www.odi.org.uk/rapid/Events/ESRC/docs/background_paper.pdf, http://www.iscintelligence.com/archivos_subidos/usfacultyburden_5.pdf, http://blogs.lse.ac.uk/impactofsocialsciences/tag/claire-donovan/, http://www.atn.edu.au/docs/Research%20Global%20-%20Measuring%20the%20impact%20of%20research.pdf, http://www.hbs.edu/research/pdf/10-099.pdf, http://www.esf.org/index.php?eID=tx_ccdamdl_file&p[file]=25668&p[dl]=1&p[pid]=6767&p[site]=European%20Science%20Foundation&p[t]=1351858982&hash=93e987c5832f10aeee3911bac23b4e0f&l=en, http://www.rand.org/pubs/research_briefs/2007/RAND_RB9202.pdf, http://www.rand.org/pubs/documented_briefings/2010/RAND_DB578.pdf, http://ukirc.ac.uk/object/report/8025/doc/CIHE_0612ImpactReport_summary.pdf, http://www.timeshighereducation.co.uk/story.asp?storyCode=415340§ioncode=26, http://www.publicengagement.ac.uk/sites/default/files/80096%20NCCPE%20Social%20Value%20Report.pdf, http://www2.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf, http://www.artscouncil.org.uk/media/uploads/documents/publications/340.pdf, http://www.ref.ac.uk/media/ref/content/pub/researchexcellenceframeworkimpactpilotexercisefindingsoftheexpertpanels/re01_10.pdf, http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/02_11.pdf, http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf, http://www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12.pdf, http://www.russellgroup.ac.uk/uploads/REF-consultation-response-FINAL-Dec09.pdf, http://www.siampi.eu/Pages/SIA/12/625.bGFuZz1FTkc.html, http://www.siampi.eu/Content/SIAMPI/SIAMPI_Final%20report.pdf, http://www.thesroinetwork.org/publications/doc_details/241-a-guide-to-social-return-on-investment-2012, http://www.ucu.org.uk/media/pdf/n/q/ucu_REFstatement_finalsignatures.pdf, http://www.esrc.ac.uk/_images/Case_Study_of_the_Future_of_Work_Programme_Volume_2_tcm8-4563.pdf, Receive exclusive offers and updates from Oxford Academic, Automated collation of evidence is difficult, Allows evidence to be contextualized and a story told, Incorporating perspective can make it difficult to assess critically, Enables assessment in the absence of quantitative data, Preserves distinctive account or disciplinary perspective, Rewards those who can write well, and/or afford to pay for external input.