Cooperating on quality of higher education – the measuring and decision-making practices of Dutch accreditation panels
Aim of the project
This project investigates how stakeholders of Dutch higher education cooperate in assessing the quality of higher education programmes. In the Dutch accreditation system, such assessments are undertaken by panels of independent peers. These ‘accreditation panels’ are expected to measure and decide about the quality of programmes in accordance with the rules, values and goals of the accreditation system’s assessment framework. This project examines the alignment of the goals and values of the accreditation system, its assessment framework, and the measuring and decision-making practices of accreditation panels.
Theoretical background
Evaluating the quality of higher education involves the application of social scientific methods (Wanzer, 2021). In the case of Dutch accreditation reviews, these measurements are intersubjective, involving panels of independent peers; and they are multi-stake, aimed at weighing and reconciling the different perspectives and interests of higher education’s primary stakeholders. To this end, a specific measurement practice has been developed since the 1980s, culminating in a national assessment framework that was launched in 2003 (current version: NVAO Nederland, 2018).
The Dutch assessment framework makes explicit the accreditation system’s values and goals, which are both epistemic and non-epistemic in nature. For instance, assessments must be reliable, do equal justice to the panellists’ perspectives, but also place minimal administrative burden on the programme. This evaluation framework and practice can be shown to meet three important requirements of social-scientific measurements (Cartwright & Runhardt, 2014). That is: the concept of quality of higher education is formally characterised (into ‘NVAO standards’), represented (e.g. a programme ‘meets or does not meet generic quality’) and detailed into measurement procedures (e.g. ‘the panel must reach a unanimous or majority collective opinion’).
To what extent should the resulting measurement practice be expected to meet the epistemic and non-epistemic goals of the Dutch accreditation system? This project combines a normative analysis of the assessment framework’s design, with a qualitative study into the work of accreditation panels. Within the framework’s limits –or possibly transgressing them– how do accreditation panels work together and reach their collective verdict? To what extent can and do such collective (intersubjective, multi-stake) verdicts align with the accreditation system’s values and goals? And ultimately, to what extent could this measurement practice promote the sustained cooperation of all stakeholders of Dutch higher education involved in its quality assurance?
This study will build upon studies of peer review in academia (Forsberg et al., 2022; Lamont, 2009; Lee et al., 2013). Conceptualising evaluation of higher education as a measurement practice, the study will refer to philosophical studies into the construction of measurement instruments, especially intersubjective measurement practices, and with a special interest for their value- and theory-ladenness (Chang, 2008; Galison, 1987; Tal, 2016). Finally, building on judgment aggregation theory, the study will look at how this framework’s design and ensuing practice, impacts the panel’s opinion formation and judgment aggregation process (Dietrich & List, 2005; List, 2006).
Research design
The project combines qualitative research with philosophical enquiry. The philosophical enquiry is informed by –and developed in tandem with– a qualitative study into the collective measurement and decision-making processes by Dutch accreditation panels. The qualitative study is divided into two partial studies. The first partial study will involve in-depth interviews with accreditation panel chairpersons and secretaries, to explore how accreditation panels collectively assess and decide about a programme’s quality. The second partial study will involve (participatory) observations of accreditation reviews during site visits, to study in detail the interactions between panellists as they deliberate towards a collective judgment.
In addition, there will be limited (historical/archival) research into the development of the Dutch system of external quality assurance since the 1980s. This research is intended to contextualize the past and present involvement of different stakeholders in the Dutch accreditation system, and the resulting consensus regarding the system’s (epistemic and non-epistemic) purposes.
Valorization
This project’s practical and societal relevance is periodically monitored by an advisory group comprised of representatives of the following stakeholders of the Dutch accreditation system:
- Dutch-Flemish accreditation organisation NVAO.
- The Dutch Inspectorate of Education.
- Assessment agencies Academion, AeQui, Hobéon and NQA.
- Several Dutch research universities and universities of applied sciences.
- Student organizations ISO and LSVb.
In consultation with this advisory group, one of the ultimate aims of this project is to provide recommendations towards improving the current accreditation system. By analyzing the alignment of the accreditation system’s values and goals, the assessment framework’s design, and the resulting evaluation and decision-making processes by accreditation panels, this project hopes to contribute to the sustainable cooperation of stakeholders of the quality assurance of Dutch higher education.
-
Project Initiators
-
Prof. dr. J.W. (Jan-Willem) Romeijn
-
Prof. dr. R. P. M. (Rafael) Wittek, Coordinating researcher
- Discipline
Philosophy, Sociology - Location
University of Groningen, Faculty of Philosophy, Department of Ethics, Social and Political Philosophy - Period
August 21, 2021 – November 20, 2026
Literature
Cartwright, N., & Runhardt, R. (2014). Measurement. In N. Cartwright & E. Montuschi (Eds.), Philosophy of Social Science: A New Introduction (pp. 265–287). Oxford University Press. Chang, H. (2008). Inventing Temperature: Measurement and Scientific Progress. Oxford University Press. Dietrich, F., & List, C. (2005). Strategy-Proof Judgment Aggregation. LSE STICERD Research Paper No. PEPP09. https://ssrn.com/abstract=1158331 Forsberg, E., Geschwind, L., Levander, S., & Wermke, W. (2022). Peer Review in Academia. In E. Forsberg, L. Geschwind, S. Levander, & W. Wermke (Eds.), Peer review in an Era of Evaluation: Understanding the Practice of Gatekeeping in Academia (pp. 3–36). Springer International Publishing. Galison, P. (1987). How Experiments End. University of Chicago Press. Lamont, M. (2009). How professors think: Inside the curious world of academic judgment. Harvard University Press. Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in Peer Review. Journal of the American Society for Information Science and Technology, 64(1), 2–17. List, C. (2006). The Discursive Dilemma and Public Reason. Ethics, 116(2), 362–402. NVAO Nederland. (2018). Beoordelingskader accreditatiestelsel hoger onderwijs Nederland. Nederlands-Vlaamse Accreditatieorganisatie. https://www.nvao.net/files/attachments/.89/Beoordelingskader_accreditatiestelsel_hoger_onderwijs_Nederland_2018.pdf Tal, E. (2016). How does measuring generate evidence? The problem of observational grounding. Journal of Physics: Conference Series, 772(1), 012001. https://doi.org/10.1088/1742-6596/772/1/012001 Wanzer, D. L. (2021). What Is Evaluation? Perspectives of How Evaluation Differs (or Not) From Research. American Journal of Evaluation, 42(1), 28–46.