I would add to this a metric for how often people click the doi to copy the link. When I find something interesting, if I’m moving quickly, I’ll copy DOIs and paste them into paperpile to download and return to. Not sure these metrics would capture that.

When considering impact, I often think about how research and discovery seeds future projects, particularly through what gets funded. It would be fascinating to explore how grant funding patterns could be tied to publications, shedding light on their influence on subsequent funded research.

Tracking how many citations and shared keywords from Arcadia’s research appear in successful grant applications could provide valuable insights into its influence on funding trends. Publicly available funding data could be leveraged to draw connections between your work and funded projects across different fields. For this aim as well, incorporation of AI tools to scan and tag publications that cite or adapt your research could help uncover unexpected intersections of reuse, highlighting the multidimensional impact of your outputs.

Another thought is the idea of creating an interactive map that visualizes the lineage of research outputs. This map could trace how work is adopted and expanded upon—not just in grant-funded projects but also in broader contexts such as policy changes, industry innovations, or public health interventions.

These ideas are tied to providing a more comprehensive picture of how research contributes to tangible progress across domains. Moreover, making these insights easily accessible to stakeholders could inspire greater trust and engagement with open science models.

When it comes to relevance, for me, I’ld simply say it is topic based, so a very clear honest summary, keywords, and limitations would be most helpful. If we take traditional abstracts and summaries they are often a bit vague, and advertising rather then stating limitations. For tools, something that outlines possible uses, citations to how it was used (maybe some kind of summary of uses), and setting it apart from other similar tools or even quick comparison with other tools and again specific strength/limitation summary could help.

I think it would be really helpful to communicate the scope and impact of publications. This could help policymakers and the general public understand the focus and utility of different publications. These are some possible impact and scope questions with associated scale:

‘What is the scope of study?’ [peer reviewers would link related topics covered in this paper (e.g., electrochemistry, protein engineering, etc.)]

‘What is the scale of impact for this paper?’ [a 1-5 rating scale from niche to universal]

‘How accessible is this paper?’ [a 1-5 scale from very niche to easily understood]

In addition, a publication’s impact on various sectors (such as policy, technology, education, medicine, etc) could be assessed.

These benchmarks could be determined by a weighted mixed voting system (similar to rotten tomatoes’ rating system). Peer reviewers and approved readers would give each publication an initial score, and then every reader would have the chance to score the paper's impact and scope. The scores of approved users and peer reviewers would carry more weight than those of the general reader, but the general readers would get to report on how they found the article.

Assessing a paper's scope and impact via weighted crowdsourcing would help assess the subjective response to publications. This impact and scope rating system could easily be added to the existing peer-review process and journal/publication platforms.

Before attempting to answer any of these questions, I think it is necessary to look at (and probably measure, if possible) the reach and penetration of Arcadia Science in the research community. Only if a significant fraction (used here as a loose term) of the researchers are aware of the company and its work, it will make sense for the impact to be measured. Please note that this reach is different from the reach mentioned in this pub, which is a measure of the views, downloads, comments, etc.

Active Outreach:

The reach of the company could, in many ways, depend on the kind of outreach strategies the company employs. I think what the company currently needs is what I call an “active outreach” strategy. This essentially involves directly reaching out to researchers in a given field through email, in person at conferences, or other relevant means. For example, if the company works on a project on Actin structure, the company needs to actively reach out to the researchers (PIs, postdocs) working in the same field and let them know about the work that it has recently published. This would allow them to know about the work and engage with it as per their preference.

After an initial round of active outreach in a given field, this circle would be expected to grow and the work of the company could be expected to penetrate within the community with time. This would potentially help the pubs to receive more visibility, feedback, and probably get cited in other publications. These would then give the company to have something on the slate to start with upon which it can build useful metrics and approaches to track utility, rigour, and reuse.

Such an active approach may ultimately pave the way for either other researchers to directly engage with the research group at Arcadia Science (so that reuse could be directly tracked through personal communication and ultimately as citations in published works) or the company to develop much robust measures for the reuse of its work.

It may be interesting to display who and how the work is being used in other contexts. Similar to the “works for me” button found on protocols.io, researchers could indicate if the tool or result is “in use” and perhaps elaborate on their use case. This could lead to a dropdown list or other feature that shows all the places the work is currently in use and what it is being used for. Listing the examples of reuse could also drive traffic to the work of other scientists in the community so it may incentivize others to indicate that they’re using a tool, result, etc. Just an idea! :)

Before we can evaluate success, we need a clear definition of "success".
I believe that there is no single definition of success, however, when it comes to measuring the impact of science because that definition evolves depending on the desired outcome. The desired outcomes differ depending on the role of the person or the organization.

For instance, a government official tasked with creating a new public health program might want to learn about the outcomes across multiple interventions from an existing case study, or potential outcomes from a health economics study. Downloads and views are not sufficient to reflect the impact of a publication - because what follows from reading the publication might be more nebulous - e.g., the government official then contacts the lead authors of the study for engagement/consultation. The publication may facilitate the initial point of contact, but the impact goes beyond a single metric we can measure - I imagine one of the ways to measure such impact (if we narrow the scope to comparing publishing models) is to ask how effectively X publishing platform/model itself resonates with non-scientists? If we are only measuring metrics like downloads, it implies a single direction of information flow. A successful model, I reckon, will be bidirectional, and we need to find ways to perhaps quickly prototype a model for publishing/sharing science and see how it resonates with people who are not directly involved in scientific knowledge production.

That may be beyond the scope of the current discussion. Going back to measuring success across publishing models, I think we need to consider the ease of adoption of a new publication/research-sharing model. This might be a chicken and egg problem where we don't have enough people trying out a new model and hence we can't evaluate it. But I suspect there might be subgroups of people who face similar obstacles to adopting a new publishing model. 

My thoughts on what to measure now are quite nebulous and I will expand them in the coming weeks. But to close it off, for now, here's one question I'd like to pose for discussion: Who uses science/research findings, and for what purpose? Answering this question will help us identify different metrics to measure other than metrics like number of downloads.

Graphical abstracts are always really valuable to me–I can quickly scan to see if the subject and findings of the paper are something that applies to my interests.

It seems like this is already being implemented here, but having a comments or discussion where people can compare notes seems really key, especially for those of us who work in really niche fields. Negative results may not be publishable, but knowing about them may save your colleagues or future researchers a ton of work.

I look at figures and methods and to see if results match conclusions.

Methods section… is the protocol detailed enough to suggest how to replicate an experiment?

I love commentaries because they often effectively communicate technically niche or complex data to a broader audience, provide more unbiased discussion of the significance of the findings, and occasionally provide criticism of the design and implementation of the study. I would imagine that finding a third party, uninvolved in the production of the publication in question, to write a commentary would provide a valuable tool in interpreting the use cases and limitations of the findings and determining how relevant the findings are to my interests.

Generally, the tracking of citations of your publication is used to determine the extent to which your work is being verified or built on. This, however, isn’t sufficiently comprehensive (as not all pertinent research may find your publication, especially if there are barriers to finding or accessing your work). I would imagine that a supplemental measure would be to track publications that cite the same publications that you cited in your publication. Other publications that use a certain threshold of shared citations are more likely to be pertinent to your research, even in the event that they are not citing your work specifically.

As a general issue in scientific publishing, I believe that literature search friction and duration can be improved if publications are tagged both by subject and by common, predictable use cases. I’m often looking for particular perspectives/angles on a given subject, and often have to infer the kind of perspective/angle that a paper provides only after reading it in some depth. Here are some examples of what I might personally seek during a literature search:

• New results/data that corroborate/support a given theory or investigation

• New theories/models with better explanatory power

• New results/data that challenge a given theory or investigation

• Strategic and/or historical commentary

• Summaries and expository work

• Reproduced/replicated studies

• Methodological innovations

• Reviews or criticism

I was happy to see the “Negative Data” and “Open Question” tags on Arcadia publications; they’re a step in the right direction. I personally favor more tags, though I imagine that avoidance of over-tagging is an important curation issue.

When I see “Negative Data”, I immediately wonder – what is negated? Is it a hypothesis previously formed within Arcadia? Is it a major assumption that the entire community holds? (That is, what is its “metascientific scope”?) Does the negative result present a quandary, or is a new explanation offered instead? (What is the “deliverable scope”?) In summary, it would be helpful to know, when browsing, what kind of scientific proposition the data is negating, and what I’ll stand to gain by reading the publication (e.g. a new explanation, a scientific dilemma to mull over, a study replication prospect, a new methodology to consider, etc.)

I recognize that tags won’t be able to capture much of the above information, but I can imagine 1-3 tags (e.g. subject, use-case, scope) being quite informative.

Well, I think the best solution to this problem is a somewhat “longer-term” prospect, but I’ll raise it here nonetheless. I think the success of a publishing model pertains to its ability to directly facilitate advancement of science itself, as opposed to, say, “network-based” indicators that only approximate social adoption/discussion/sharing. Fundamentally, scientific theories/hypotheses/models are “algorithms” (defined as loosely as you like) that eat data and produce explanations/predictions, and any publishing model that helps gather better input and/or deliver better output should garner prestige for doing so.

As a highly simplified toy model, consider “research” that is quite directly related to algorithms/data, such as quantitative finance: CrunchDAO (which is a kind of decentralized hedge fund…) actually has a leaderboard in place that ranks participants by the performance of their models. Of course, theories and models in many scientific fields are often not actual algorithms, and assessing their ability to best explain data would require a qualitative (though, nevertheless, standardizable) metascientific criterion. Fortunately for humankind, the realm of biology is more sophisticated than that of hedge funds :D And metascience will require criteria more sophisticated than “market returns”. Nevertheless, I wonder to what extent publishing platforms could ultimately be “metascientifically ranked” by their ability to gather the best data and develop the best explanations.

It might be interesting to have different types of citations — some citations are “this is a finding I’m referencing” and others are “this is a method/datatset from this work I’m using”

My question is: Can we apply the same metrics to all publications, given that they can belong to different categories? Publications may introduce significant new biological concepts or bring novelty and clarity to a particular topic. They can also report important new technological developments and open new avenues for investigation.

An important question to include in the survey is: Has this publication changed the way you think about biology or the specific problem in question? Alternatively, has this publication introduced you to a new method for performing an experiment?

### How can we measure and communicate the impact of science?

The discussion elaborates on measuring and communicating the impact of science through different classes of publications, focusing on Reviews, Research Papers, and Methods Papers.

1. **Classes of Publications**

- Reviews

- Research Papers

- Methods Papers

2. **Evaluating Reviews**

- Comprehensive coverage

- Recency and up-to-date information

- Use of extensive research publications

- Risk of "Review of Reviews" and error propagation

- Can we create a metric for evaluating reviews based on the ratio of original research publications cited versus the number of reviews cited?

- \*\*Summary for Reviews\*\*: Their impact is gauged by comprehensiveness, recency, and the extent of original research citations, avoiding the pitfalls of relying too much on other reviews, which can propagate errors.

3. **Evaluating Methods Papers**

- Wet-Lab Methods Papers

  - Accessibility and availability of reagents and materials

  - Reproducibility

  - Challenges in replicating experiments

    - Using the same reagents and conditions as the original method

    - Getting the same results

    - Importance of internal controls

- Computational Methods Papers

  - Computational Environment and Software Availability

    - Can we use the same computational environment?

    - Can we use the same software?

    - Is the software used accessible and available?

  - Challenges in replicating experiments

    - Importance of internal controls

    - Issues with computational reproducibility

- Researchers are more likely to test a protocol or a pipeline using their own data. Researchers are unlikely to use the data described/used in the original publication. Therefore, it is very hard to evaluate to what extent we were able to reproduce and replicate a given described experiment protocol or computational pipeline.

- In addition, in the case of computational pipelines, researchers are likely to use the latest version of software or scripts and/or genome files, thus making a clean and unbiased judgment regarding the reproducibility of a given computational pipeline very suspect and potentially biased.

- \*\*Summary for Methods Papers\*\*: Judged by reproducibility, including the accessibility of reagents and materials and the ability to replicate procedures in both wet lab and computational contexts. Challenges include not using original conditions and lacking proper internal controls.

4. **Evaluating Research Papers**

- The evaluation of research papers is inherently complex.

- Rarely do reviewers have expertise in all experimental aspects of the paper.

- Research papers can potentially be evaluated at different levels.

- Deconvolution of reproducibility:

    - Figures

        - Is the primary data used to generate the figures readily available?

        - Has a Python/R/Shell script that can regenerate the figure been provided? Scripts in R or Python (pandas, seaborn, matplotlib) can be generated to reproduce most figures.

        - If the answer to the above questions is yes, then the figure should be eligible to be assigned a DOI.

    - Tables

        - Is the primary data used to generate the tables readily available?

        - Has a Python/R/Shell script that can regenerate the figure been provided? Scripts in R or Python (pandas, seaborn, matplotlib) can be generated to reproduce most tables.

        - If the answer to the above questions is yes, then the table should be eligible to be assigned a DOI.

    - Materials

        - Have the drugs, reagents, and other materials been reported in detail?

            - Can these drugs and reagents be obtained? Are they available?

            - How likely are these drugs and reagents to be affected by batch production effects?

            - Are the animals used in the research available?

            - How robust is the phenotype of the animals used in the experiment? What is the probability that a variable genetic background will affect a given phenotype used in the experiments?

    - Methods

        - How detailed is the description of the methods?

        - Are the reagents, antibodies, oligonucleotides, and dyes used readily available?

        - Can the experiment be reproduced using the same materials?

        - How dependent on specific hardware and/or specific laboratory equipment are the methods?

    - Data availability and integrity

        - Is the primary data used in the manuscript readily available?

        - Primary data from repositories like NCBI SRA

            - Is the data processing affected by issues related to metadata stripping in sequencing files during submission?

    - Scripts and commands documentation

        - Availability and documentation of processing scripts

        - Documentation of pipeline and virtual environments

        - Availability of scripts in R or Python (pandas, seaborn, matplotlib)

- \*\*Summary for Research Papers\*\*: Unique due to their combination of new observations and methodologies. Evaluating their reproducibility involves ensuring primary data availability, processing it according to the original study, and producing equivalent figures. Challenges include data integrity from repositories like NCBI SRA and proper documentation of processing methods. Reproducibility also hinges on the availability of experimental materials (reagents, antibodies) and proper documentation of data processing methods. Sharing primary data and scripts (preferably in Python) for figure generation enhances understanding and reproducibility, potentially leading to granular micro-publications with unique identifiers.

5. **Evaluating Supplementary Data**

- Problems with supplementary data storage and accessibility

    - Issues with broken links and deleted data

- Solutions for better data preservation

    - Using repositories like Zenodo or Figshare

    - Dissociating supplementary data from main publications

    - Exploring different publication models (GigaDB, GigaScience)

- \*\*Summary for Supplementary Data\*\*: The storage and accessibility of supplementary data present significant challenges. Issues with broken links and deleted data necessitate better preservation solutions, such as using repositories like Zenodo or Figshare and dissociating supplementary data from main publications. Different publication models (e.g., GigaNDB, GigaScience) offer various approaches to addressing these challenges.

6. **Evaluating Data Integrity After Publication - The FASTQ File Headers Issue**

- To save disk space, NCBI-SRA started removing metadata present in headers of FASTQ files containing link number, tile number, X and Y coordinates.

- How important are FASTQ file headers for quality control?

- What information is being lost by NCBI-SRA redefining the FASTQ headers?

- Is it worth preserving this information in another format to be able to reattach to the downloaded files if necessary?

- See \[GitHub discussion for details\](https://github.com/ncbi/sra-tools/issues/130)

- \*\*Summary for FASTQ Files\*\*: A specific issue is raised about FASTQ file headers, especially in RNA-seq datasets. A recent GitHub discussion highlighted the importance of headers containing metadata like link number, tile number, and X and Y coordinates, which are stripped upon submission to NCBI SRA. This loss of information hampers the identification of PCR artifacts and read quality assessment. The need for alternative ways to preserve this metadata before submission is emphasized to ensure the replicability of original results. The removal of header information from 10x Genomics files at some point further complicated data reanalysis, underscoring the necessity of methods to preserve and reattach this metadata if necessary (although this might not be a current issue for 10x Genomics data).\

\

**Note** The content of this comment was developed by first recording the ideas. ChatGPT-4 then transcribed the recordings and used these transcripts to generate an outline. The outline was subsequently manually edited and refined.

This could take the form of badging as proposed here (and I’m sure other sources, too): https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002234

Astera or other nonprofit orgs like eLife could organize badging principles/practices and deploy technical experts for rigor, or others for open science audits. ASAP (Aligning Science Against Parkinson’s) has strict open-science requirements and a template for materials sharing audits that could be helpful.

Also relates to “is verifiable” in the discussion, here: https://research.arcadiascience.com/pub/open-question-measuring-reuse#nfd6bck2cia

Engagement in the new publishing models is key. (1) Importantly, how many **collaborative** studies have been published in these new forums? It’s much easier to make radical publishing decisions by oneself than inspire colleagues to take risks. How do we incentivize groups to build-in public release of their study through these models at the design phase of a project? The open science community is large but spread-out, so it might be important to get folks together that are experts in different disciplines/methods to foster collaboration aimed explicitly at publication experimentation.

(2) Are people reviewing, and getting reviews? What are the quality of those reviews? Are they leading to meaningful changes in the research studies? While traditional peer review has its downsides, the upsides are a level of deep reading/scrutiny that is unusual when reading literature daily (I’ve spent upwards of 10-15 hours reviewing papers!). How do we get a mix of the things readers can catch quickly—but also incentivize deep review? Is there a way to build-in incentives for peer review, such as the ability to request verified reviewers in exchange for needing to review (e.g., a token system where you must review 2-3X more than request? The tokens could somehow scale with the depth of review?)

(3) Ultimately, are papers being read and informing new studies? Citation and other quantitative metrics give some hint at this but take a long time to measure and can be biased.

There is always a moment (or years) when research would be logically applied within it’s own microcosm and I think that may be more true in a world where there are less structural barriers to publication/writing.

1. A Metric/visualization mechanism that allows for the velocity + depth of the research with differentiation (as others have suggested) between validation vs. expansion would appear extremely useful. If we’ve moved past the construct of “only experts” being able to validate an idea before publication then the overall impact needs to be more than sheer ‘number of comments’ as this would appear easily manipulatable.

   The nature of “expanding” on the research has value both within the field of intended research and outside of it. Separating those two is often necessary in my own experience as research goals and expectations of outcomes are often very different. This could enable separate scoring mechanisms on “impact” and separate the noise of the discussion of progress in very different arenas.

2. Language/specificity: If you are looking for the broadest possible audience, then some mechanism for allowing translation of the research for different fields of research and languages would likely be useful.

   From a pure language perspective, translation would seem simple and easy to incentivize given how widespread geographically the open science community is.

3. Purely commenting on how this paper can be improved or it’s faults without response/validation of the logic in those comments can be an equally slippery slope of manipulation. If commentary on research will be public, then to an extent, the validation and citation of those ideas could be incorporated into future research to help substantiate the thought process. That way users have an incentive to give great feedback that is intelligible, constructive, and usable in future research. This is then usable both from a search mechanism but also user curated “this was useful for my research in [field X].”

I think an important need in science publishing to make it more transparent and accessible is publication of “negative” results. Often research is bogged down by people repeating things that take extensive time and resources that do not end up working out in the end. Open science community needs to be more forgiving and transparent in sharing not only important discoveries but also things that do not work out, so people can build from it and not start from scratch.

Have you considered partnering with a vendor to track how its sales change following a publication? I’m thinking that if someone is trying to replicate the same experiment, they may opt to use your exact vendor for reagents.

Have you considered adding an “endorsement” feature? Once a scientist has used and validated your catalyst/method/technique/code, they can “endorse” the publication.

As a visual learner, one thing I enjoy is when publications have is a graphical abstract. These can increase initial engagement to an article for inter-disciplinary individuals that might typically see a wall of text and be drawn away.

I admire deeply how you’re setting up a new system for sharing science - I hope the community can help you craft a model that works. I’m worried about the incentives: Why would other scientists give you the time it takes to comment, review, offer feedback? Cynically I think willingness drops off with seniority.

Consider moving to a licensing model for your information? A no-money “paywall” that gives access to pubs only to registered users. Access to the next pub requires feedback about the last or by spending ‘community credit’, which is earned through comments on other pubs, feedback on how pubs were used, etc. This could even be filling out a quick survey.

This is still basically free and accessible to anyone, but the data you might gather through (now somewhat mandatory) feedback will tell you what is impactful. Willingness to spend community credit to, say, read past the abstract, will give you information about the interest level in a pub. Later, ratio of citations to how many accessed tells you something about a pub’s impact.

Also cynically, I believe people treat something proportionally to what it cost to get it. Asking for just a little bit of community engagement in exchange for your pubs would both increase users’ perception of their value and get you much more of the feedback you’re seeking.

One idea that could enhance the visibility and accessibility of research is to create a dynamic, flowchart-like visualization—a simplified network graph—that connects a current result, data, or publication to the relevant prior works it builds upon (via incoming lines or nodes). Additionally, it could illustrate the potential impact of the research on various fields, labs, proteins, or disease areas (through outgoing lines or nodes). This network map should be placed prominently, near or alongside the abstract, offering a clear and concise overview of the relevant research landscape. It would allow the scientific community to quickly grasp the context and significance of the work.

Such a map would be especially useful for those who may not be experts in the field but have a general understanding. Establishing these connections can otherwise be time-consuming and complex. By providing this network, tracking key publications and labs working in the field would become much easier for newcomers.

Furthermore, this model could serve as a metric to gauge the impact of the publication. By monitoring the number of clicks on specific parts of the map, researchers could gain insights into which works and applications are being discovered and how the research is influencing other fields.

Given the wide scope of techniques and even wider scope of applications of these techniques, it could be useful for readers to easily access some basic contextual information about usage. My first intuition of relevant context usage is 1) what field is the user coming from? 2) In what industry is the user applying the technique or effect? Given that growth and usage of open access appears to be disparate across scientific fields (Severin et al., 2020), this information could be beneficial both for readers and Arcadia’s tracking of reach and impact. Further, this could provide demographic information in where open access science is being used the most and which disciplines are viable targets for improving reach. This information could be easily accessed by asking PubPub users to provide their basic field and industry when they create their account, then creating an interactive feature for each Pub (something like “I used and endorse this”).

I envision a sliding-scale graphic with relative proportions of field and industry usage, respectively, with a composite number indicating exact frequency (i.e. if 20 people have used the technique, the number 20 appears on the end, with a sliding scale indicating what proportion of those users come from what disciplines and in what industries). This information not only could be used by readers to determine if those in their field use the technique and find it helpful, but also identify barriers to usage on a discipline level that could encourage cross-industry and interdisciplinary conversation and collaboration.

In science (and in most contexts, really), I believe in the importance of balance. Overemphasizing benchmarks like citations and impact factors can foster a "publish or perish" mentality, leading to competitive environments that prioritize metrics over genuine scientific progress and healthy collaboration. However, benchmarks and metrics, when used judiciously, are valuable tools for evaluating rigorous and impactful research. I think it would be helpful to develop metrics in molecular biology that encourage personal improvement and progress rather than overall standing in a scientific field. For instance, tracking the number of citations or interactions with an article, relative to the number of readers who engage with an entire article versus just the abstract could provide insights into its accessibility and impact, pre-publication. This would be particularly useful for in-progress work such as articles posted on BioRxiv or similar platforms.

A similar balance is essential when considering modularity in scientific contributions. Smaller, incremental outputs, particularly within teams, can enhance collaboration and progress. But overly fragmented publications can hinder public engagement and lead to misinterpretation, as the broader context or significance of the work may be lost. The challenge lies in striking the right balance: fostering meaningful, modular contributions while maintaining coherence and accessibility for a broader audience.

Interactive formats, like discussion-based platforms or preprint systems with editable components (e.g., Google Docs-style commenting on bioRxiv), could help address these challenges. For example, modular in-progress publications could allow casual readers to focus on short, digestible articles while enabling those with deeper interest to explore detailed discussions or linked resources – cool website format, by the way!

A significant challenge in molecular biology is the lack of long-term infrastructure for organizing and distributing datasets. In my undergraduate work, I worked in Oceanography. I think this field is actually a great example of a field that maintains long-term publicly accessible datasets. Dedicated organizations like NOAA and Copernicus have established models for managing and sharing large datasets with scientists and the public. Molecular biology, I think, could benefit from additional entities to handle, for example genomic and cellular atlas data. A centralized, long-term data repository with integrative pipelines, managed by dedicated organizations rather than transient student researchers (we all have to graduate eventually!), would greatly improve accessibility and sustainability in the field.

I think this actually undersells what is described below - there is an important distinction between “easy to find, access, use, and repurpose” and whether or not people actually find and use something. It’s possible to meet these criteria by merely posting in publicly-accessible spaces and granting open access, but those alone do not guarantee actual engagement (as you discuss further in this paper).

When the research includes code, having the code not only publicly available but open to collaboration on a platform such as GitHub is a great way to track this. Looking at the number of forks, PRs, and comments can provide real-time data on reuse. Ideally, authors will encourage not only reuse but contributions and new adaptations of the code, and these should all be linked and visible through GH.