Bibliometric measures of epistemic change – theoretical foundations and validation
Workshop series of three workshops
Jochen Gläser, TU Berlin, Germany and Kevin Boyack, Scitech Strategies, USA
Dates: 11 – 12 April 2024, 16 – 18 September 2024, 10-11 April 2025
Location/Conference Venue: TU Berlin
1. Background
The aim of this workshop series is to draw together sociologists of science and bibliometricians in a discussion of a problem both fields share, namely the empirical identification of epistemic properties of research and the reconstruction of processes of epistemic change. The two fields show complementary weaknesses and strengths in dealing with this topic. Conceptual foundations are weak in bibliometrics but strong in sociology, while the ability to analyse macro-level processes of change are weak in sociology and strong in bibliometrics.
– Qualitative sociological studies of science have a long history of including epistemic properties of research in their investigations. Comparative analyses of fields of research (Knorr-Cetina 1999, Whitley 2000 [1984]) were rarely linked to methods for empirically determining the proposed properties of fields. Empirical studies of specific epistemic properties of research processes or fields have shown that such properties can be derived from interviews and observations ex post (Zuckerman and Merton 1973 [1972], Laudel and Gläser 2014) but reliable methods of empirical measurement are still missing. Studies of processes of epistemic change through the emergence of new fields (see Edge and Mulkay 1976: 351-402 for an overview) or scientific innovations within fields (e.g. Fujimura 1988, Laudel et al. 2014). These studies offer detailed accounts of epistemic change through emerging or changing topics and methods of research but were not concerned with comparatively measuring these changes. Neither studies including epistemic properties of research nor studies of processes of epistemic change have developed protocols for empirically identifying epistemic properties of research or their change. Few of these studies produced detailed accounts of epistemic change from empirical data.
– In bibliometrics, ’science of science’, management of innovation studies and the economics of science, many attempts have been made in the last two decades to measure the change of epistemic properties of research. Commonly used concepts include novelty, originality, innovativeness, disruptiveness, interdisciplinarity and epistemic distance between researchers (e.g. Ayoubi et al. 2021, Park et al. 2023). These measures are rarely defined by reference to a theory; most definitions are operational in the sense that a concept is defined by what its indicator measures. This has led to the same indicator being used to measure different concepts and the same concept being measured by different indicators, with no theoretical justifications being provided.
The few attempts to validate these measures are not convincing. The main challenge for validation attempts is the missing theoretical background. Since bibliometric measures do not operationalise theoretical concepts, they cannot be validated by using a measure that operationalises the same concept and whose validity had already been established. Validation studies are by and large limited to comparing measurements to the opinions of researchers about the novelty, originality, innovativeness and so on of selected publications. Not surprisingly, these comparisons return mixed results (Bornmann et al. 2019).
Thus, the two fields face complementary problems: Bibliometric studies of science invest considerable work in the development of sophisticated indicators for epistemic properties of research but are reluctant to define the underlying concepts and (thus) have difficulties with the validation of their measures. In contrast, the sociology of science appears to have a better conceptual understanding of epistemic properties of research and can empirically identify them but has difficulties going beyond single cases and idiosyncratic approaches.
2. Approach
The separation of the two fields has limited both the opportunities of bibliometrics to contribute to advancing our theories of scientific knowledge production and the opportunities of the sociology of science to trace epistemic change in scientific communities. We will use a series of three workshops to achieve a combination of sociological and bibliometric perspectives that makes theoretical progress possible and leads to new opportunities for validating bibliometric indicators of epistemic change. The two ideas of the workshop series are a) to start from theoretical considerations to discuss bibliometric indicators as partial indicators of theoretical concepts and b) to use qualitative studies as sources for “ground truths” of epistemic change that can be used for the validation of bibliometric methods and indicators.
The workshops will build on each other and contribute to developing a shared understanding of participants by bibliometricians and sociologists of the theoretical and methodological problems involved in bibliometrically measuring epistemic change. We invite workshop participants to work on
agreed-upon goals between the workshops and will use workshops to present results of that work in addition to conventionally invited presentations.
Workshop 1 State of the art of conceptualizing and measuring epistemic change
The first workshop will bring together researchers to establish the state of the art in the three areas relevant to the aim of the workshop series, namely bibliometric studies of epistemic change, qualitative studies of epistemic change and theories of epistemic change. In addition, we will develop a joint work programme of studying known cases of epistemic change and invite workshop participants to participate in it. Possible cases include the molecular-biological approach to cancer research (Fujimura 1988), High-temperature superconductivity (Felt and Nowotny 1992) or the experimental realisation of Bose-Einstein Condensation (Laudel et al. 2014).
Workshop 2: Discussing bibliometric measures of epistemic change
Workshop participants will discuss the outcomes of the joint exercise initiated in the first workshop. In addition, researchers will be invited to present their work on a) qualitative reconstructions of epistemic change, with a particular focus on how it is represented in publications, and b) bibliometric reconstructions of epistemic change (e.g. emerging fields or the diffusion of scientific innovations), also with a focus on how these change processes are represented in publications.
Workshop 3: Synthesis and validation strategies
Workshop participants will discuss a synthesis of results obtained since the first workshop. They will also be invited to present their work on the validation of bibliometric indicators and approaches to represent epistemic properties of research.
3. Call for participation
A core group of researchers from bibliometrics and the sociology of science has already committed to participate in the workshop series:
Thomas Franssen; CWTS Leiden, Netherlands
Wolfgang Glänzel; KU Leuven, Belgium
Grit Laudel; TU Berlin, Germany
Ismael Rafols; CWTS Leiden, Netherlands
Sotaro Shibayama; Lund University, Sweden
Cassidy Sugimoto, School of Public Policy at Georgia Institute of Technology, U.S.
Theresa Velden; Centre for Higher Education Research and Science Studies, Germany;
Lingfei Wu; University of Pittsburgh, U.S.
We invite researchers from science studies and bibliometrics to participate in our interdisciplinary workshop series. We are particularly interested in contributions that
– provide case studies of epistemic change including methodological innovations, the turn to new objects of study or emerging fields that can be used as testbeds for bibliometric indicators,
– propose sociological questions that can be better answered by including bibliometric studies, or
– critically analyse bibliometric indicators that are currently in use or propose new methods for the validation of bibliometric indicators.
We have reserved five places for PhD students. PhD students who are interested in participating are invited to submit a statement of interest that explains their interest in the topic and its links to their PhD project.
Please send proposals and statements of interest to Jochen.Glaeser@tu-berlin.de and kboyack@mapofscience.com. Deadline for submissions is 31 January 2024.
Thanks to a generous grant from the Volkswagen Foundation, we will be able to fully fund the participation in the workshops in Berlin. Since we consider the synergies that come from participants being in the same room as critical, we don’t expect to be offering a remote option.
Bibliography
Ayoubi, C., M. Pezzoni and F. Visentin (2021). “Does It Pay to Do Novel Science? The Selectivity Patterns in Science Funding.”
Science and Public Policy 48(5): 635-648.
Bornmann, L., A. Tekles, H. H. Zhang and F. Y. Ye (2019). “Do we measure novelty when we analyze unusual combinations of
cited references? A validation study of bibliometric novelty indicators based on F1000Prime data.” Journal of
Informetrics 13(4): 100979.
Edge, D. and M. J. Mulkay (1976). Astronomy Transformed: The Emergence of Radio Astronomy in Britain. New York, John
Wiley & Sons, Inc.
Felt, U. and H. Nowotny (1992). “Striking Gold in the 1990s: The Discovery of High-Temperature Superconductivity and Its
Impact on the Science System.” Science, Technology, & Human Values 17(4): 506-531.
Fujimura, J. H. (1988). “The Molecular Biological Bandwagon in Cancer Research: Where Social Worlds Meet.” Social Problems
35(3): 261-283.
Knorr-Cetina, K. (1999). Epistemic Cultures: How the Sciences Make Knowledge. Cambridge, Harvard University Press.
Laudel, G. and J. Gläser (2014). “Beyond breakthrough research: Epistemic properties of research and their consequences for
research funding.” Research Policy 43(7): 1204–1216.
Laudel, G., E. Lettkemann, R. Ramuz, L. Wedlin and R. Woolley (2014). Cold Atoms – Hot Research: High Risks, High Rewards
in Five Different Authority Structures. Organizational Transformation And Scientific Change: The Impact Of Institutional
Restructuring On Universities And Intellectual Innovation. R. Whitley and J. Gläser. Bingley, Emerald Group: 203-234.
Park, M., E. Leahey and R. J. Funk (2023). “Papers and patents are becoming less disruptive over time.” Nature 613(7942):
138-144.
Whitley, R. (2000 [1984]). The Intellectual and Social Organization of the Sciences. Oxford, Oxford University Press.
Zuckerman, H. and R. K. Merton (1973 [1972]). Age, Aging, and Age Structure in Science. The Sociology of Science. R. K.
Merton. Chicago, The University of Chicago Press: 497-559