Summary

AGORRA (A Global Observatory of Responsible Research Assessment) is a collaboration between research funders, evaluation agencies and meta-researchers which aims to generate comparative data, evidence and analysis to support and accelerate responsible research assessment (RRA). With a specific focus on national-level assessment frameworks, it also aims to inform and accelerate the broader reform and transformation of research assessment systems, supporting and complementing global initiatives like CoARA (The Coalition for Advancing Research Assessment) and DORA (Declaration on Research Assessment). The project consists of three workstreams:

• Collaboratively assessing the possibilities for next-generation research assessment, including the use of novel indicators, dashboard and the feasibility of real-time research assessment to replace burdensome one-off national assessment exercises.
• Populating the Atlas of Assessment, our observatory of national research assessment systems aiming to provide the first global overview of such systems, enabling tracking and comparison of emerging practices, evolutions, convergence, divergence and effectiveness.
• Mapping and defining the indicators of research culture and environment at a global level.

---

Project team

• Alex Rushforth, Project lead, CWTS
• Peter Kolarz, Head of Programmes, RoRI
• James Wilsdon, Executive Director, RoRI
• Gunnar Sivertsen, NIFU, Norway & Senior Visiting Fellow, RoRI
• Cameron Neylon, Senior Research Fellow, RoRI
• Moumita Koley, Researcher, DST, Indian Institute of Science (ISC), India
• Laura Rovelli, Researcher, CONICET, Argentina
• Marta Wróblewska, Assistant Professor, SWPS University, Poland

Partner organisations

• Research England/UKRI
• Australian Research Council 
• ANVUR (Agenzia Nazionale Di Valutazione Del Sistema Universitario E Della Ricerca)
• Evaluation Research Centre, Chinese Academy of Sciences (CAS), Beijing
• Dutch Research Council (NWO)
• The National Research Foundation of South Africa (NRF-SA)
• Canadian Institutes of Health Research (CIHR)
• Social Sciences and Humanities Research Council (SSHRC)
• Swedish Research Council (SRC)
• Research Council of Norway (RCN)
• Ministry of Education and Science, Ukraine
• Volkswagen Foundation

The project in detail

Strand 1: Next-generation research assessment

Much interest – and in some instances hype – currently surrounds the ‘next generation’ of research metrics. Examples include new approaches for measuring novelty and disruptiveness, open science practices, societal impact, and methods that harness AI and emerging open data sources. In light of such developments, there is a strong case to revisit and update our understanding of metrics use for research assessment.

The fundamental research question underpinning this work is: Can research indicators and dashboards be developed for national assessment systems that harness promising advances in indicator methodologies and data, while also upholding principles of responsible metrics and responsible research assessment more broadly? This includes ensuring the diversity of research cultures is respected, voices of a diverse range of stakeholders are consulted, and unintended consequences are anticipated.

What are the possibilities and limitations for metric use in different kinds of assessment systems, with varying purposes and characteristics? For instance, how do considerations differ between assessments where funding and accountability are key purposes, versus assessments that produce strategic advice?

The Next Generation Research Assessment workstream tackles these timely questions through a mixed-methods, co-produced partnerships between research funders and meta-researchers. Crucially, we are placing our focus on the socio-technical, not just the technical dimensions of new indicators and data sources.

Strand 2: New geographies of research assessment

National systems for research assessment have gained increasing prominence in recent years. These systems are variously used to allocate institutional research funding, to gather intelligence about a country’s research strengths, and to induce behaviour change in the sector. These may pertain to enhancing research productivity or focusing research on particular thematic areas, but also often focus on tenets of RRA as expressed most notably through CoARA (Coalition for Advancing Research Assessment).

Much of the literature on national research assessment focuses only on a small number of countries, typically from the global north. But to fully understand trends, convergences, divergences, and how different systems and contexts have different needs, a far more comprehensive perspective is required. 

The New Geographies of Research Assessment work strand addresses this need. It is a continuation of the AGORRA project, using the Atlas of Assessment – a free, open-access tool that maps research assessment systems worldwide. Launched in May 2025, the Atlas initially included 13 tester countries and continues to expand.

The New Geographies of Research Assessment work strand aims to populate the Atlas as far as possible. The ambition is for the Atlas to become a permanent tool with the fullest possible country coverage and regular updates to all countries included, contributing to a more complete and inclusive picture of research assessment systems.

Strand 3: Indicators of research culture and environment (IRCE)

There is an increasing interest in monitoring and evaluating research cultures and environments as part of national, programme and institutional assessment processes. The RRA agenda aims to expand concepts of quality and excellence with the goal of creating incentives for improving research cultures and environments. This has led to a demand for relevant, trusted and reliable indicators that can provide evidence for monitoring and evaluation systems (the UK being just one recent example).

This strand of AGORRA provides a global assessment of the prospects for research culture indicators. This includes three components:

• Providing a landscape view of qualities for which indicators would be valuable
• A survey of indicators and forms of evidence in use
• A practical review of implementation possibilities including a longitudinal evaluation of candidate indicators.

The workstream is led by the Curtin Open Knowledge Initiative in collaboration with CWTS and the core RORI team.

The workstream has identified the diversity of issues that are tackled globally under the heading of culture and will provide a critical analysis of the qualities of possible indicators. This will include advice on which indicators are appropriate for incorporation into responsible research assessment practices. It will explicitly identify indicators that are inappropriate for application within responsible research assessment systems as well as identifying if and how such indicators might be made fit for purpose over time.

---