Bibliometrics

ARTICLE

Bibliometrics is the use of statistical methods to analyse books, articles and other publications, especially in scientific contents. Bibliometric methods are frequently used in the field of library and information science. Bibliometrics is closely associated with scientometrics, the analysis of scientific metrics and indicators, to the point that both fields largely overlap.
Bibliometrics studies first appeared in the late 19th century. They have known a significant development after the Second World War in a context of “periodical crisis” and new technical opportunities offered by computing tools. In the early 1960s, the Science Citation Index of Eugene Garfield and the citation network analysis of Derek John de Solla Price laid the fundamental basis of a structured research program on bibliometrics.
Citation analysis is a commonly used bibliometric method which is based on constructing the citation graph,[footnote “Hutchins et al. 2019.”] a network or graph representation of the citations shared by documents. Many research fields use bibliometric methods to explore the impact of their field, the impact of a set of researchers, the impact of a particular paper, or to identify particularly impactful papers within a specific field of research. Bibliometrics tools have been commonly integrated in descriptive linguistics, the development of thesauri, and evaluation of reader usage. Beyond specialized scientific use, popular web search engines, such as the pagerank algorithm implemented by Google have been largely shaped by bibliometrics methods and concepts.
The emergence of the Web and the open science movement has gradually transformed the definition and the purpose of “bibliometrics.” In the 2010s historical proprietary infrastructures for citation data such as the Web of Science or Scopus have been challenged by new initiatives in favor of open citation data. The Leiden Manifesto for Research Metrics (2015) opened a wide debate on the use and transparency of metrics.

Definition

The term bibliométrie was first used by Paul Otlet in 1934,[footnote “Otlet 1934.”] and defined as “the measurement of all aspects related to the publication and reading of books and documents.”[footnote “Rousseau 2014.”] The anglicized version bibliometrics was first used by Alan Pritchard in a paper published in 1969, titled “Statistical Bibliography or Bibliometrics?”[footnote “Pritchard 1969.”] He defined the term as “the application of mathematics and statistical methods to books and other media of communication.” Bibliometrics was conceived as a replacement for statistical bibliography, the main label used by publications in the field until then: for Pritchard, statistical bibliography was too “clumsy” and did not make it very clear what was the main object of study.[footnote “Hertzel 2003, p. 288.”]
The concept of bibliometrics “stresses the material aspect of the undertaking: counting books, articles, publications, citations”[footnote “Bellis 2009, p. 3.”] In theory, bibliometrics is a distinct field from scientometrics (from the Russian naukometriya)[footnote “Bellis 2009, p. 12.”] which relies on the analysis of non-bibliographic indicators of scientific activity. In practice, bibliometrics and scientometrics studies tend to use similar data sources and methods, as citation data has become the leading standard of quantitative scientific evaluation during the mid-20th century: “insofar as bibliometric techniques are applied to scientific and technical literature, the two areas of scientometrics and bibliometrics overlap to a considerable degree.”[footnote “Bellis 2009, p. 3.”] The development of the web and the expansion of bibliometrics approach to non-scientific production has entailed the introduction of broader labels in the 1990s and the 2000s: infometrics, webometrics or cybermetrics.[footnote “Bellis 2009, p. 4.”] These terms have not been extensively adopted, as they partly overlap with pre-existing research practices, such as information retrieval.

History

Scientific works, studies and researches that have a bibliometric character can be identified, depending on the definition, already for the 12th century in the form of Jewish indexes.[footnote “Jovanovic 2012.”]

Early experiments (1880–1914)

Bibliometric analysis appeared at the turn of the 19th and the 20th century.[footnote “Bellis 2009, p. 2.”][footnote “Godin 2006.”][footnote “Danesh Mardani-Nejad 2020.”][footnote “Hertzel 2003, p. 292.”] These developments predate the first occurrence of the concept of bibiometrics by several decades. Alternative label were commonly used: bibliography statistics became especially prevalent after 1920 and continued to remain in use until the end of the 1960s.[footnote “Hertzel 2003, p. 292.”] Early statistical studies of scientific metadata were motivated by the significant expansion of scientific output and the parallel development of indexing services of databases that made this information more accessible in the first place.[footnote “Bellis 2009, p. 6.”] Citation index were first applied to case law in the 1860s and their most famous example, Shepard’s Citations (first published in 1873) will serve as a direct inspiration for the Science Citation Index one century later.[footnote “Bellis 2009, p. 23.”]
The emergence of social sciences inspired new speculative research on the science of science and the possibility of studying science itself as a scientific object: “The belief that social activities, including science, could be reduced to quantitative laws, just as the trajectory of a cannonball and the revolutions of the heavenly bodies, traces back to the positivist sociology of Auguste Comte, William Ogburn, and Herbert Spencer.”[footnote “Bellis 2009, p. 1.”] Bibliometric analysis was not conceived as a separate body studies but one of the available methods for the quantitative analysis of scientific activity in different fields of research: science history (Histoire des sciences et des savants depuis deux siècles of Alphonse de Candolle in 1885, The history of comparative anatomy, a statistical analysis of the literature by Francis Joseph Cole and Nellie B. Eales in 1917), bibliography (The Theory of National and International Bibliography of Francis Burburry Campbell in 1896) or sociology of science (Statistics of American Psychologists of James McKeen Cattell in 1903).
Early bibliometrics and scientometrics work were not simply descriptive but expressed normative views of what science should be and how it could progress. The measurement of the performance of individual researchers, scientific institutions or entire countries was a major objective.[footnote “Bellis 2009, p. 6.”] The statistical analysis of James McKeen Cattell acted as a preparatory work for a large scale evaluation of American researchers with eugenicists undertones: American Men of Science (1906), “with its astoundingly simplistic rating system of asterisks attached to individual entries in proportion to the estimated eminence of the starred scholar.”[footnote “Bellis 2009, p. 2.”]

Development of bibliography statistics (1910–1945)

After 1910, bibliometrics approach increasingly became the main focus in several study of scientific performance rather than one quantitative method among others.[footnote “Bellis 2009, p. 7.”] In 1917, Francis Joseph Cole and Nellie B. Eales argued in favor of the primary statistical value of publications as a publication “is an isolated and definite piece of work, it is permanent, accessible, and may be judged, and in most cases it is not difficult to ascertain when, where, and by whom it was done, and to plot the results on squared paper.”[footnote “Cole Eales 1917, p. 578.”] Five years later, Edward Wyndham Hulme expanded this argument to the point that publications could be considered as the standard measure of an entire civilization: “If civilization is but the product of the human mind operating upon a shifting platform of its environment, we may claim for bibliography that it is not only a pillar in the structure of the edifice, but that it can function as a measure of the varying forces to which this structure is continuously subjected.”[footnote “Hulme 1923, p. 43.”] This shift toward publication had a limited impact: well until the 1970s, national and international evaluation of scientific activities “disdained bibliometric indicators” which were deemed too simplistic, in favor of socological and economic measures.[footnote “Bellis 2009, p. 14.”]
Both the enhanced value attached to scientific publications as a measure of knowledge and the difficulties met by libraries to manage the growing flow of academic periodicals entailed the development of the first citation indexes.[footnote “Bellis 2009, p. 9.”] In 1927, P. Gross and E. M. Gross compiled the 3,633 references quoted by the Journal of the American Chemical Society during the year 1926 and ranked journals depending on their level of citation. The two authors created a set of tools and methods still commonly used by academic search engines, including attributing a bonus to recent citations since “the present trend rather than the past performance of a journal should be considered first.”[footnote “Gross Gross 1927, p. 387.”] Yet the academic environment measured was markedly different: German rather than English ranked by far the main language of science of chemistry with more than 50% of all references.[footnote “Gross Gross 1927, p. 388.”]
In the same period, fundamental algorithms, metrics and methods of bibliometrics were first identified in several unrelated projects,[footnote “Bellis 2009, p. 75.”] most of them being related to the structural inequalities of scientific production. In Alfred Lotka introduced its law of productivity from an analysis of the authored publications in the Chemical Abstracts and the Geschichtstafeln der Physik: the number of authors producing an n number of contributions is equal to the 1/n^2 number of authors that only produced one publication.[footnote “Bellis 2009, p. 92.”] In, the chief librarian of the London Science Museum, Samuel Bradford derived a law of scattering from his experience in bibliographic indexing: there are exponentially diminishing returns of searching for references in science journals, as more and more work need to be consulted to find relevant work. Both the Lotka and Bradford law have been criticized as they are far from universal and rather uncovers a rough power law relationship rendered by deceivingly precise equations.[footnote “Bellis 2009, p. 99.”]

Periodical crisis, digitization and citation index (1945–1960)

After the Second World War, the growing challenge in managing and accessing scientific publications turned into a full-fledged “periodical crisis”: existing journals could not keep up with the rapidly increasing scientific output spurred by the big science projects.[footnote “Wouters 1999, p. 61.”][footnote “Bellis 2009, p. 12.”] The issue became politically relevant after the successful launch of Sputnik in 1957: “The Sputnik crisis turned the librarians’ problem of bibliographic control into a national information crisis..”[footnote “Wouters 1999, p. 62.”] In a context of rapid and dramatic change, the emerging field of bibliometrics was linked to large scale reforms of academic publishing and nearly utopian visions of the future of science. In 1934, Paul Otlet introduced under the concept of bibliométrie or bibliology an ambitious project of measuring the impact of texts on society. In contrast with the bounded definition of bibliometrics that will become prevalent after the 1960s, the vision of Otlet was not limited to scientific publication nor in fact to publication as a fundamental unit: it aimed for “by the resolution of texts into atomic elements, or ideas, which he located in the single paragraphs (alinéa, verset, articulet) composing a book.”[footnote “Bellis 2009, p. 10.”] In 1939 John Desmond Bernal envisioned a network of scientific archives, which was briefly considered by the Royal Society in 1948: “The scientific paper sent to the central publication office, upon approval by an editorial board of referees, would be microfilmed, and a sort of print-on-demand system set in action thereafter.”[footnote “Bellis 2009, p. 52.”] While not using the concept of bibliometrics, Bernal had a formative influence of leading figures of the field such as Derek John de Solla Price.
The emerging computing technologies were immediately considered as a potential solution to make a larger amount of scientific output readable and searchable. During the 1950s and 1960s, an uncoordinated wave of experiments in indexing technologies resulted in the rapid development of key concepts of computing research retrieval.[footnote “Bellis 2009, p. 27.”] In 1957, IBM engineer Hans Peter Luhn introduced an influential paradigm of statistical-based analysis of word frequencies, as “communication of ideas by means of words is carried out on the basis of statistical probability.”[footnote “Luhn 1957.”] Automated translation of non-English scientific work has also significantly contributed to fundamental research on natural language processing of bibliographic references, as in this period a significant amount of scientific publications were not still available in English, especially the one coming from the Soviet block. Influent members of the National Science Foundation like Joshua Ledeberg advocated for the creation of a “centralized information system”, SCITEL, partly influenced by the ideas of John Desmond Bernal. This system would at first coexist with printed journals and gradually replace them altogether on account of its efficiency.[footnote “Wouters 1999, p. 60.”] In the plan laid out by Ledeberg to Eugen Garfield in November 1961, a centralized deposit would index as much as 1,000,000 scientific articles per year. Beyond full-text searching, the infrastructure would also ensure the indexation of citation and other metadata, as well as the automated translation of foreign language articles.[footnote “Wouters 1999, p. 64.”]
The first working prototype on an online retrieval system developed in 1963 by Doug Engelbart and Charles Bourne at the Stanford Research Institute proved the feasibility of these theoretical assumptions, although it was heavily constrained by memory issues: no more than 10,000 words of a few documents could be indexed.[footnote “Bourne Hahn 2003, p. 16.”] The early scientific computing infrastructures were focused on more specific research areas, such as MEDLINE for medicine, NASA/RECON for space engineering or OCLC Worldcat for library search: “most of the earliest online retrieval system provided access to a bibliographic database and the rest used a file containing another sort of information—encyclopedia articles, inventory data, or chemical compounds.”[footnote “Bourne Hahn 2003, p. 12.”] Exclusive focus on text analysis proved limitative as the digitized collections expanded: a query could yield a large number results and it was difficult to evaluate the relevancy and the accuracy of the results.[footnote “Bellis 2009, p. 30.”]
The periodical crisis and the limitations of index retrieval technologies motivated the development of bibliometric tools and large citation index like the Science Citation Index of Eugene Garfield. Garfield’s work was initially primarily concerned with the automated analysis of text work. In contrast with ongoing work largely focused on internal semantic relationship, Garfield highlighted “the importance of metatext in discourse analysis”, such as introductory sentences and bibliographic references.[footnote “Bellis 2009, p. 34.”] Secondary forms of scientific production like literature reviews and bibliographic notes became central to Garfield’s vision as they have already been to John Desmond Bernal’s vision of scientific archives.[footnote “Bellis 2009, p. 53.”] By 1953, Garfield’s attention was permanently shifted to citation analysis: in a private letter to William C. Adair, the vice-president of the publisher of the Shepard’s Citation index, “he suggested a well tried solution to the problem of automatic indexing, namely to “shepardize” biomedical literature, to untangle the skein of its content by following the thread of citation links in the same way the legal citator did with court sentences.”[footnote “Bellis 2009, p. 35.”] In 1955, Garfield published his seminal article “Citation Indexes for Science”, that both laid out the outline of the Science Citation Index and had a large influence on the future development of bibliometrics.[footnote “Bellis 2009, p. 35.”] The general citation index envisioned by Garfield was originally one of the building block of the ambitious plan of Joshua Lederberg to computerize scientific literature.[footnote “Bellis 2009, p. 36.”] Due to lack of funding, the plan was never realized.[footnote “Bellis 2009, p. 37.”] In 1963, Eugene Garfield created the Institute for Scientific Information that aimed to transform the projects initially envisioned with Lederberg into a profitable business.

Bibliometric reductionism, metrics, and structuration of a research field (1960–1990)

The field of bibliometrics coalesced in parallel to the development of the Science Citation Index, that was to become its fundamental infrastructure and data resource:[footnote “Bellis 2009, p. 49.”] “while the early twentieth century contributed methods that were necessary for measuring research, the mid-twentieth century was characterized by the development of institutions that motivated and facilitated research measurement.”[footnote “Sugimoto Larivière 2018, p. 8.”] Significant influences of the nascent field included along with John Desmond Bernal, Paul Otlet the sociology of science of Robert K. Merton, that was re-interpreted in a non-ethic manner: the Matthew Effect, that is the increasing concentration of attention given to researchers that were already notable, was no longer considered as a derive(?) but a feature of normal science.[footnote “Bellis 2009, p. 57.”]
A follower of Bernal, the British historian of science Derek John de Solla Price has had a major impact on the disciplinary formation of bibliometrics: with “the publication of Science Since Babylon (1961), Little Science, Big Science (1963), and Networks of Scientific Papers (1965) by Derek Price, scientometrics already had a sound empirical and conceptual toolkit available.”[footnote “Bellis 2009, p. 49.”] Price was a proponent of bibliometric reductionism.[footnote “Bellis 2009, p. 62.”] As Francis Joseph Cole and Nellie B. Eales in 1917, he argued that a publication is the best possible standard to lay out a quantitative study of science: they “resemble a pile of bricks (…) to remain in perpetuity as an intellectual edifice built by skill and artifice, resting on primitive foundation.”[footnote “Price 1975, p. 162.”] Price doubled down on this reductionist approach by limiting in turn the large set of existing bibliographic data to citation data.
Price’s framework, like Garfield’s, takes for granted the structural inequality of science production, as a minority of researchers creates a large share of publication and an even smaller share have a real measurable impact on subsequent research (with as few as 2% of papers having 4 citations or more at the time).[footnote “Bellis 2009, p. 65.”] Despite the unprecedented growth of post-war science, Price claimed for the continued existence of an invisible college of elite scientists that, as in the time of Robert Boyle undertook the most valuable work.[footnote “Bellis 2009, p. 67.”] While Price was aware of the power relationships that ensured the domination of such an elite, there was a fundamental ambiguity in the bibliometrics studies, that highlighted the concentration of academic publishing and prestige but also created tools, models and metrics that normalized pre-existing inequalities.[footnote “Bellis 2009, p. 67.”] The central position of the Scientific Citation Index amplified this performative effect. In the end of the 1960s Eugene Garfield formulated a law of concentration that was formally a reinterpretation of the Samuel Bradford’s law of scattering, with a major difference: while Bradford talked for the perspective of a specific research project, Garfield drew a generalization of the law to the entire set of scientific publishing: “the core literature for all scientific disciplines involves a group of no more than 1000 journals, and may involve as few as 500.” Such law was also a justification of the practical limitation of the citation index to a limited subset of core journals, with the underlying assumption that any expansion into second-tier journals would yield diminishing returns.[footnote “Bellis 2009, p. 103.”] Rather than simply observing structural trends and patterns, bibliometrics tend to amplify and stratify them even further: “Garfield’s citation indexes would have brought to a logical completion, the story of a stratified scientific literature produced by (…) a few, high-quality, “must-buy” international journals owned by a decreasing number of multinational corporations ruling the roost in the global information market.”[footnote “Bellis 2009, p. 104.”]
Under the impulsion of Garfield and Price, bibliometrics became both a research field and a testing ground for quantitative policy evaluation of research. This second aspect was not a major focus of the Science Citation Index has been a progressive development: the famous Impact Factor was originally devised in the 1960s by Garfield and Irving Sher to select the core group of journals that were to be featured in Current Contents and the Science Citation Index and was only regularly published after 1975.[footnote “Bellis 2009, p. 187.”] The metric itself is a very simple ratio between the total count of citation received by the journal on the past year and its productivity on the past two years, to ponderate the prolificity of some publications.[footnote “Bellis 2009, p. 186.”] For example, Nature had an impact factor of 41.577 in 2017:[footnote “Nature. 2017 Journal Citation Reports (PDF) (Report). Web of Science (Science ed.). Thomson Reuters. 2018.(verification needed)”]

(table to include)

The simplicity of the impact factor has likely been a major factor in its wide adoption by scientific institutions, journals, funders or evaluators: “none of the revised versions or substitutes of ISI IF has gained general acceptance beyond its proponents, probably because the alleged alternatives lack the degree of interpretability of the original measure.”[footnote “Bellis 2009, p. 194.”]
Alongside these simplified measurements, Garfield continued to support and fund fundamental research in science history and sociology of science. First published 1964, The Use of Citation Data in Writing the History of Science compiles several experimental case studies relying on the citation network of the Science Citation Index, including a quantitative reconstruction of the discovery of the DNA.[footnote “Bellis 2009, p. 153.”] Interest in this area persisted well after the sell of the Index to Thomson Reuters: as late as 2001, Garfield unveiled HistCite, a software for “algorithmic historiography” created in collaboration with Alexander Pudovkin, and Vladimir S. Istomin.[footnote “Bellis 2009, p. 173.”]

The Web turn (1990–…)

The development of the World Wide Web and the Digital Revolution had a complex impact on bibliometrics.
The web itself and some of its key components (such as search engines) were partly a product of bibliometrics theory. In its original form, it was derived from a bibliographic scientific infrastructure commissioned to Tim Berners-Lee by the CERN for the specific needs of high energy physics, ENQUIRE. The structure of ENQUIRE was closer to an internal web of data: it connected “nodes” that “could refer to a person, a software module, etc. and that could be interlined with various relations such as made, include, describes and so forth.”[footnote “Hogan 2014, p. 20.”] Sharing of data and data documentation was a major focus in the initial communication of the World Wide Web when the project was first unveiled in August 1991 : “The WWW project was started to allow high energy physicists to share data, news, and documentation. We are very interested in spreading the web to other areas, and having gateway servers for other data.”[footnote “Tim Berners-Lee, Qualifiers on Hypertext Links, mail sent on 6 August 1991 to the alt.hypertext”] The web rapidly superseded pre-existing online infrastructure, even when they included more advanced computing features.[footnote “Star Ruhleder 1996, p. 131.”] The core value attached to hyperlinking in the design of the web seem to validate the intuitions of the funding figures of bibliometrics: “The onset of the World Wide Web in the mid-1990s made Garfield’s citationist dream more likely to come true. In the world network of hypertexts, not only is the bibliographic reference one of the possible forms taken by a hyperlink inside the electronic version of a scientific article, but the Web itself also exhibits a citation structure, links between web pages being formally similar to bibliographic citations.”[footnote “Bellis 2009, p. 285.”] Consequently, bibliometrics concepts have been incorporated in major communication technologies the search algorithm of Google: “the citation-driven concept of relevance applied to the network of hyperlinks between web pages would revolutionize the way Web search engines let users quickly pick useful materials out of the anarchical universe of digital information.”[footnote “Bellis 2009, pp. 31–32.”]
While the web expanded the intellectual influence of bibliometrics way beyond specialized scientific research, it also shattered the core tenets of the field. In contrast with the wide utopian visions of Bernal and Otlet that partly inspired it, the Science Citation Index was always conceived as a closed infrastructure, not only from the perspective of their users but also from the perspective of the collection index: the logical conclusion of Price’s theory of invisible college and Garfield’s law of concentration was to focus exclusively on a limited set of core scientific journals. With the rapid expansion of the Web, numerous forms of publications (notably preprints), scientific activities and communities suddenly became visible and highlighted by contrast the limitations of applied bibliometrics.[footnote “Bellis 2009, p. 289.”] The other fundamental aspect of bibliometric reductionism, the exclusive focus on citation, has also been increasingly fragilized by the multiplication of alternative data sources and the unprecedented access to full text corpus that made it possible to revive the large scale semantic analysis first envisioned by Garfield in the early 1950s: “Links alone, then, just like bibliographic citations alone, do not seem sufficient to pin down critical communication patterns on the Web, and their statistical analysis will probably follow, in the years to come, the same path of citation analysis, establishing fruitful alliances with other emerging qualitative and quantitative outlooks over the web landscape.”[footnote “Bellis 2009, p. 322.”]
The close relationship between bibliometrics and commercial vendors of citation data and indicators has become more strained since the 1990s. Leading scientific publishers have diversified their activities beyond publishing and moved “from a content-provision to a data analytics business.”[footnote “Aspesi et al. 2019, p. 5.”] By 2019, Elsevier has either acquired or built a large portofolio platforms, tools, databases and indicators covering all aspects and stages of scientific research: “the largest supplier of academic journals is also in charge of evaluating and validating research quality and impact (e.g., Pure, Plum Analytics, Sci Val), identifying academic experts for potential employers (e.g., Expert Lookup5), managing the research networking platforms through which to collaborate (e.g., SSRN, Hivebench, Mendeley), managing the tools through which to find funding (e.g., Plum X, Mendeley, Sci Val), and controlling the platforms through which to analyze and store researchers’ data (e.g., Hivebench, Mendeley).”[footnote “Chen et al. 2019, par. 25.”] Metrics and indicators are key components of this vertical integration: “Elsevier’s further move to offering metrics-based decision making is simultaneously a move to gain further influence in the entirety of the knowledge production process, as well as to further monetize its disproportionate ownership of content.”[footnote “Chen et al. 2019, par. 29.”] The new market for scientific publication and scientific data has been compared with the business models of social networks, search engines and other forms of platform capitalism[footnote “Moore 2019, p. 156.”][footnote “Chen et al. 2019.”][footnote “Wainwright Bervejillo 2021.”] While content access is free, it is indirectly paid through data extraction and surveillance.[footnote “Wainwright Bervejillo 2021, p. 211.”] In 2020, Rafael Ball envisioned a bleak future for bibliometricians where their research contribute to the emerge of a highly invasive form of “surveillance capitalism”:scientists “be given a whole series of scores which not only provide a more comprehensive picture of the academic performance, but also the perception, behaviour, demeanour, appearance and (subjective) credibility (…) In China, this kind of personal data analysis is already being implemented and used simultaneously as an incentive and penalty system.”[footnote “Ball 2020, p. 504.”]
The Leiden manifesto for research metrics (2015) highlighted the growing rift between the commercial providers of scientific metrics and bibliometric communities. The signatories stressed the potential social damage of uncontrolled metric-based evaluation and surveillance: “as scientometricians, social scientists and research administrators, we have watched with increasing alarm the pervasive misapplication of indicators to the evaluation of scientific performance.”[footnote “Hicks et al. 2015, p. 430.”] Several structural reforms of bibliometric research and research evaluation are proposed, including a stronger reliance on qualitative assessment and the reliance on “open, transparent and simple” data collection.[footnote “Hicks et al. 2015, p. 430.”] The Leiden Manifesto has stirred an important debate in bibliometrics/scientometrics/infometrics with some critics arguing that the elaboration of quantitative metrics bears no responsibility on their misuse in commercial platforms and research evaluation.[footnote “David Frangopol 2015.”]

Bibliometrics and open science

The open science movement has been acknowledged as the most important transformation faced by bibliometrics since the emergence of the field in the 1960s.[footnote “Bellis 2009, p. 288 sq..”][footnote “Heck 2020.”] The free sharing of a wide variety of scientific outputs on the web affected the practice of bibliometrics at all levels: the definition and the collection of the data, infrastructure, and metrics.
Before the crystallization of the field around the Science Citation Index and the reductionist theories of Derek de Solla Price, bibliometrics has been largely influenced by utopian projects of enhanced knowledge sharing beyond specialized academic communities. The scientific networks envisioned by Paul Otlet or John Desmond Bernal have gained a new relevancy with the development of the Web: “The philosophical inspiration of the pioneers in pursuing the above lines of inquiry, however, faded gradually into the background (…) Whereas Bernal’s input would eventually find an ideal continuation in the open access movement, the citation machine set into motion by Garfield and Small led to the proliferation of sectorial studies of a fundamentally empirical nature.”[footnote “Bellis 2009, p. 336.”]

From altmetrics to open metrics

Further information: Altmetrics
In the early developments, the open science movement partly co-opted the standard tools of bibliometrics and quantitative evaluation: “the fact that no reference was made to metadata in the main OA declarations (Budapest, Berlin, Bethesda) has led to a paradoxical situation (…) it was through the use of the Web of Science that OA advocates were eager to show how much accessibility led to a citation advantage compared to paywalled articles.”[footnote “Torny, Capelli Danjean 2019, p. 1.”] After 2000, an important bibliometric literature was devoted to the citation advantage of open access publications.[footnote “Sugimoto Larivière 2018, p. 70.”]
By the end of the 2000s, the impact factor and other metrics have increasingly held responsible a systemic locked-in of prestigious non-accessible sources. Key figures of the open science movement like Stevan Harnad called for the creation of “open access scientometrics” that would take “advantage of the wealth of usage and impact metrics enabled by the multiplication of online, full-text, open access digital archives.”[footnote “Bellis 2009, p. 300.”] As the public of open science expanded beyond academic circles, new metrics should aim for “measuring the broader societal impacts of scientific research.”[footnote “Wilsdon et al. 2017, p. 9.”]
The concept of alt-metrics was introduced in 2009 by Cameron Neylon and Shirly Wu as article-level metrics.[footnote “Neylon Wu 2009.”] In contrast with the focus of leading metrics on journals (impact factor) or, more recently, on individual researchers (h-index), the article-level metrics makes it possible to track the circulation of individual publications: “(an) article that used to live on a shelf now lives in Mendeley, CiteULike, or Zotero – where we can see and count it”[footnote “Priem et al. 2011, p. 3.”] As such they are more compatible with the diversity of publication strategies that has characterized open science: preprints, reports or even non-textual outputs like dataset or software may also have associated metrics.[footnote “Wilsdon et al. 2017, p. 9.”] In their original research proposition, Neylon and Wu favored the use of data from reference management software like Zotero or Mendeley.[footnote “Neylon Wu 2009.”] The concept of altmetrics evolved and came to encover data extracted “from social media applications, like blogs, Twitter, ResearchGate and Mendeley.”.[footnote “Wilsdon et al. 2017, p. 9.”] Social media sources proved especially to be more reliable on a long-term basis, as specialized academic tools like Mendeley came to be integrated into a proprietary ecosystem developed by leading scientific publishers. Major altmetrics indicators that emerged in the 2010s include Altmetric.com, PLUMx and ImpactStory.
As the meaning of altmetrics shifted, the debate over the positive impact of the metrics evolved toward their redefinition in an open science ecosystem: “Discussions on the misuse of metrics and their interpretation put metrics themselves in the center of open science practices.”[footnote “Heck 2020, p. 513.”] While altmetrics were initially conceived for open science publications and their expanded circulation beyond academic circles, their compatibility with the emerging requirements for open metrics has been brought into question: social network data, in particular, is far from transparent and readily accessible.[footnote “Bornmann Haunschild 2016.”][footnote “Tunger Meier 2020.”] In 2016, Ulrich Herb published a systematic assessment of the leading publications’ metrics in regard to open science principles and concluded that “neither citation-based impact metrics nor alternative metrics can be labeled open metrics. They all lack scientific foundation, transparency and verifiability.”[footnote “Herb 2016, p. 60.”]

Assessment of leading publication metrics and altmetrics in regard to open science principles[footnote “Herb 2016.”]

(table to include)

Herb laid an alternative program for open metrics that have yet to be developed.[footnote “Herb 2012, p. 29.”][footnote “Herb 2016, p. 70.”] The main criteria included:

• A large selection of publication items (journal articles, books, dataset, software) that agree with the writing and reading practices of scientific communities.[footnote “Herb 2016, p. 70.”]
• Fully documented data sources.[footnote “Herb 2016, p. 70.”]
• Transparent and reproducible process for the calculation of the metrics and other indices.[footnote “Herb 2016, p. 70.”]
• Open software [footnote “Herb 2016, p. 70.”]
• Promotion of reflexive and interpretive uses of the metrics, to prevent their misuse in quantitative assessments.[footnote “Herb 2016, p. 70.”]

This definition has been implemented in research programs, like ROSI (Reference implementation for open scientometric indicators).[footnote “Hauschke et al. 2018.”] In 2017, the European Commission Expert Group on Altmetrics expanded the open metrics program of Ulrich Herb under a new concept, the Next-generation metrics. These metrics should be managed by “open, transparent and linked data infrastructure”.[footnote “Wilsdon et al. 2017, p. 15.”] The expert group underline that not everything should be measured and not all metrics are relevants: “Measure what matters: the next generation of metrics should begin with those qualities and impacts that European societies most value and need indices for, rather than those which are most easily collected and measure”.[footnote “Wilsdon et al. 2017, p. 15.”]

Infrastructure for open citation data

Further information: Initiative for Open Citations
Until the 2010s, the impact of open science movement was largely limited to scientific publications: it “has tended to overlook the importance of social structures and systemic constraints in the design of new forms of knowledge infrastructures.”[footnote “Okune et al. 2018, p. 13.”] In 1997, Robert D. Cameron called for the development of an open databases of citation that would completely alter the condition of science communication: “Imagine a universal bibliographic and citation database linking every scholarly work ever written — no matter how published — to every work that cites and every work that cites it. Imagine that such a citation database was freely available over the Internet and was updated every day with all the new works published that day, including papers in traditional and electronic journals, conference papers, theses, technical reports, working papers, and preprints.”[footnote “Cameron 1997.”] Despite the development of specific indexes focused on open access works like CiteSeer, a large open alternative to the Science Citation Index failed to materialize. The collection of citation data, remained dominated by large commercial structure such as the direct descendant of the Scientific Citation Index, the Web of Science. This had the effect of maintaining the emerging ecosystem of open resources at the periphery of academic networks: “common pool of resources is not governed or managed by the current scholarly commons initiative. There is no dedicated hard infrastructure and though there may be a nascent community, there is no formal membership.”[footnote “Bosman et al. 2018, p. 19.”]
Since 2015, open science infrastructures, platforms and journals have converged to the creation of digital academic commons, increasingly structured around a shared ecosystem of services and standards has emerged through the network of dependencies from one infrastructure to another. This movement stem from an increasingly critical stance toward leading proprietary databases. In 2012, the San Francisco Declaration on Research Assessment (DORA) called for “ending the use of journal impact factors in funding, hiring and promotion decisions.”[footnote “Wilsdon et al. 2017, p. 7.”] The Leiden Manifesto for research metrics (2015) encouraged the development of “open, transparent and simple” data collection.[footnote “Hicks et al. 2015, p. 430.”]
Collaborations between academic and non-academic actors collectively committed in the creation and maintenance of knowledge commons has been a determining factor in the creation of new infrastructure for open citation data. Since 2010, a dataset of open citation data, the Open Citation Corpus, has been collected by several researchers from a variety of open access sources (including PLOS and Pubmed).[footnote “Peroni et al. 2015.”] This collection was the initial kernel of the Initiative for OpenCitations, incepted in 2017 in response to issues of data accessibility faced by a Wikimedia project, Wikidata. A conference, given by Dario Taraborelli, head of research at the Wikimedia Foundation showed that only 1% of papers in Crossref had citations metadata that were freely available and references stored on Wikidata were unable to include the very large segment of non-free data. This coverage expanded to more than half of the recorded papers, when Elsevier finally joined the initiative in January 2021.[footnote “Waltman, Ludo (22 December 2020). QA about Elsevier’s decision to open its citations. Leiden Madtrics. Universiteit Leiden. Retrieved 11 June 2021.”]
Since 2021, OpenAlex has become a major open infrastructure for scientific metadata. Initially created as a replacement for the discontinued Microsoft Academic Graph, OpenAlex indexed in 2022 209 millions of scholarly works from 213 millions authors as well as their associated institutions, venues and concepts in a knowledge graph integrated into the semantic web (and Wikidata).[footnote “Priem, Piwowar Orr 2022, p. 1-2.”]. Due to its large coverage and large amount of data properly migrated from the Microsoft Academic Graph (MAG), OpenAlex “seems to be at least as suited for bibliometric analyses as MAG for publication years before 2021.”[footnote “Scheidsteger Haunschild 2022, p. 10.”] In 2023, a study on the coverage of data journals in scientific indexes found that OpenAlex, along with Dimensions, “enjoy a strong advantage over the two more traditional databases, WoS and Scopus” [footnote “Jiao, Li Fang 2023, p. 14.”] and is overall especially suited for the indexation of non-journal publications like books[footnote “Laakso 2023, p. 166.”] or from researchers in non-western countries[footnote “Akbaritabar, Theile Zagheni 2023.”]
The opening of science data has been a major topic of debate in the bibliometrics and scientometrics community and had wide range social and intellectual consequences. In 2019, the entire scientific board of the Journal of Infometrics resigned and created a new open access journals, Quantitative Science Studies. The journal was published by Elsevier since 2007 and the members of the board were increasingly critical of the lack of progress in the open sharing of open citation data: “Our field depends on high-quality scientific metadata. To make our science more robust and reproducible, these data must be as open as possible. Therefore, our editorial board was deeply concerned with the refusal of Elsevier to participate in the Initiative for Open Citations (I4OC).”[footnote “Waltman et al. 2020, p. 1.”]

Bibliometrics without evaluation: the shift to quantitative science studies

The unprecedented availability of a wide range of scientific productions (publications, data, software, conference, reviews…) has entailed a more dramatic redefinition of the bibliometrics project. For new alternative works anchored in the open science landscape, the principles of bibliometrics as defined by Garfield and Price in the 1960s need to be rethought. The pre-selection of a limited corpus of important journals seem neither necessary nor appropriate. In 2019, the proponents of the Matilda project, “do not want to just “open” the existing closed information, but wish to give back a fair place to the whole academic content that has been excluded from such tools, in a “all texts are born equal” fashion.”[footnote “Torny, Capelli Danjean 2019, p. 2.”] They aim to “redefine bibliometrics tools as a technology” by focusing on the exploration and mapping of scientific corpus.[footnote “Torny, Capelli Danjean 2019, p. 7.”]
Issues of inclusivity and more critical approach of structural inequalities in science have become more prevalent in scientometrics and bibliometrics, especially in relation to gender imbalance.[footnote “Larivière et al. 2013.”][footnote “Torny, Capelli Danjean 2019.”][footnote “Chary et al. 2021.”] After 2020, one of the most heated debate in the field[footnote “Gingras 2022.”] revolved around the reception of a study on the gender imbalance in fundamental physics.[footnote “Strumia 2021.”]
The structural shift in the definition of bibliometrics, scientometrics or infometrics has entailed the need for alternative labels. The concept of Quantitative Science Studies was originally introduced in the late 2000s in the context of a renewed critical assessment of classic bibliometric findings.[footnote “Glänzel 2008.”] It has become more prevalent in the late 2010s. After leaving Elsevier, the editors of the Journal of Infometrics opted for this new label and created a journal for Quantitative Science Studies. The first editorial removed all references to metric and aimed for a wider inclusion of quantitative and qualitative research on the science of science:

We hope that those who identify under labels such as scientometrics, science of science, and metascience will all find a home in QSS. We also recognize the diverse range of disciplines for whom science is an object of study: We welcome historians of science, philosophers of science, and sociologists of science to our journal. While we bear the moniker of quantitative, we are inclusive of a breadth of epistemological perspectives. Quantitative science studies cannot operate in isolation: Robust empirical work requires the integration of theories and insights from all metasciences.[footnote “Waltman et al. 2020.”]