The experiment begins: Arcadia publishing 1.0

Prachee Avasthi; Megan L. Hochstrasser

doi:10.57844/arcadia-050a-q254

Perspective Feedback requested Reimagining scientific publishing

Published on May 31, 2022 by Arcadia Science

The experiment begins: Arcadia publishing 1.0

Building on the open-source platform PubPub, we’re sharing the first iteration of our publishing website. In addition to posting our first set of research pubs, we’re documenting our progress in developing this new system for sharing science and hope you’ll provide feedback.

The experiment begins: Arcadia publishing 1.0

Purpose

In thinking about how to share Arcadia’s research, we wanted to keep features of traditional publishing that have been honed over centuries, but improve upon what hasn’t quite adapted to the nature of modern science and technology. We have a unique opportunity to use our own research to develop mechanisms of sharing and quality control that can be more agile and adaptable. Our initial attempt is outlined here and we will continue to iterate upon it, always keeping the advancement of knowledge as our guiding principle when making decisions on what to try next.

This pub is intended to help you understand our thinking thus far, to provide a sense of what we’ve done and how the platform works, and to serve as a place to provide feedback on our strategy and the platform itself.

This pub is part of the project, “Reimagining scientific publishing.” Visit the project narrative for more background and context.

Share your thoughts!

Feel free to provide feedback by commenting in the box at the bottom of this page or by posting about this work on social media. Please make all feedback public so other readers can benefit from the discussion.

Introduction

We are reimagining scientific publishing — sharing our work early and often, maximizing utility and reusability, and improving our science on the basis of public feedback.

This is our first draft. We have ambitious goals and we’re committed to replicable long-term solutions, but we also know that “perfection is the enemy of good.” We’re using this platform to release findings now rather than hiding them until we’ve gotten everything exactly how we want it. Readers can think of the pubs on this platform as drafts that will evolve over time, shaped by public feedback. The same goes for the platform itself! We’re treating our publishing project like an experiment — we’re not sure where we will land, but we can only learn if we try. In this pub, we’re sharing our strategy and the reasoning behind some of our key decisions, highlighting features we’re excited about and areas for improvement.

We don’t want to clutter our scientific pubs with comments about website functionality, aesthetics, or our model, but we invite you to share those thoughts here! All first drafts need feedback. What do you think of this site? Are our first pubs understandable? What important features are we missing? What are we doing right? What can we improve? Please don’t hesitate to share constructive commentary and ideas.

The platform

When starting a new experiment, it’s tempting to build every piece from scratch. We could try to build an ideal publishing mechanism where form seamlessly fits function. However, we are a research organization with the goal of advancing our science with the maximal impact on the rest of the scientific community. If we held back our science from the world so we could build a platform from scratch, we would be fundamentally obstructing our experiment from its main purpose.

Enter Knowledge Futures Group (KFG) and PubPub. While there are features we intend to expand and customize, PubPub meets our main requirements for functionality and, importantly, is run by a team that is deeply aligned with our open science mission. KFG is already developing functionality to make scientific research more mineable and its usage quantifiable. Their forward-thinking plans were not based exclusively on meeting today’s publishing needs, but preparing for a world in which there is demand for exactly what Arcadia wants: an author-driven, maximally reusable, and community feedback-strengthened research-sharing platform. We hope that others can try our approach. Ultimately, the true measure of our success will be other scientists adopting, and adapting, publication paradigms that better serve science.

Initial framework

This section is a summary of our publishing strategy and what we’ve decided to try initially. Subsequent sections expand upon some of the more complex points. Check out the table of contents (click “CONTENTS” on the right side of the screen!) to get an overall sense of how this pub is organized, and check out whichever topics are most interesting to you.

Where can our research be found?

Raw data will go in FAIR (Findable, Accessible, Interoperable, Reusable) repositories.
Detailed protocols will be deposited in protocols.io.
Contextual information for the raw data will live on the PubPub platform.

How can the scientific community use our work?

Each pub is citable and will ultimately have a permanent identifier (like a digital object identifier, or DOI, through DataCite).
Each pub will follow rules for Google Scholar indexing and be discoverable.
Each pub will follow whatever structure and include whatever media can best convey the desired information to the intended audience. This may mean including executable code blocks, interactive content, and other resources embedded within pubs.

The nature of each pub will depend largely on the kind of data being reported and be geared towards facilitating use/reuse.
We will include information about planned experiments/approaches, informative failures, potential uses by others, explicit questions soliciting feedback, and other components not typically included in scientific literature.
An integral part of every research product will be public feedback that will be visible alongside the original pub. We will use this to iterate and improve on the work.
We will provide context through “project narratives,” running logs that link individual pubs and keep the big picture in focus.
While versions of the work will have permanent identifiers so they can be cited, work is not static — evaluation will happen after the pub is publicly released. The products will be improved/built upon over time, incorporating feedback throughout.

Who will contribute to our pubs?

Each person who contributed to different aspects of the science will be listed as a contributor — including those commenters whose critical feedback shaped the work.
Each contributor’s role(s) will be stated explicitly and in a standardized format, making the nature of their involvement clear.
A subset of contributors responsible for the work will be “byline” contributors. All contributors will be listed in the metadata for the DOI, but only byline contributors will technically be listed as “authors,” and have their names used in citations.
Arcadia’s byline contributors will be empowered to release their research with the support of our publishing team as needed. Wherever possible, we want to facilitate sharing rather than restrict or delay it. Anyone who makes a PubPub account can add comments on our pubs. We will also capture as much of the Twitter conversation as possible by embedding a feed collecting mentions of each pub.

We will post publicly at multiple stages of a project. Pubs are modular and we will share them when we’re ready for community feedback or think we have information that could be useful to others rather than stuffing more and more information into a single publication [1].
We will improve pubs over time as we receive feedback and do more work. For transparency, each version of the pub will remain available.
There will be synthesis works posted as well, which integrates modular data already released. This will be accomplished partially through project narratives, which track the running progress of a project as whole by stringing together individual pubs. In cases where we want to elaborate, we will share separate review/perspective pubs.

Site organization

As we create more content, the structure will likely shift and expand. For now, there are three layers to this site:

The landing page provides background on Arcadia and our goals, and, most importantly, will tell you what’s new. [View landing page]
Project narratives are running logs that maintain a high-level view of our goals and connect modular pubs together into a more cohesive story. [View example]
Pubs are detailed research products. These are akin to traditional papers, but tend to have a narrower focus and looser structure. This is where we will collect community feedback through comments. [View example]

Project narratives

Our research is generally broken up into projects, concerted efforts to answer specific biological questions, solve particular problems with technology, or sometimes simply to explore a curiosity.

These projects are described on “project narratives.” These are intended to serve as a running narratives of progress, almost like a journal or collection of field notes. The project narrative will reflect the current status of the research, where we’re heading next, and, sometimes, how you can influence our direction. Project narratives are always evolving, and will not have listed authors or DOIs.

We think they’ll be particularly important because we’re going to publish much smaller pieces of research than what is traditionally shared in journal articles. With so many modular publications, or “pubs,” it could become tough to follow the thread. Project narratives will string the individual pubs together into a coherent narrative, summarizing emergent takeaways and laying out planned next steps. Though pubs will work as individual pieces, with all essential context included, the project narratives provide space for more in-depth background.

We’re interested in sharing these narratives, especially the failures and dead-ends, because although these stories are fascinating and informative, they are rarely revealed in published papers. A chronological presentation is more reflective of how scientific work is actually done, and could be useful for students and trainees to study as they set expectations for their own research.

Pub structure and style

Pubs are our standalone “units” of research output.

In structuring individual pubs, we’ve tried to strike a balance between improving usability and what scientists expect to see in a research publication. Together, the pub’s title, subtitle/description, and the “Purpose” section provide a high-level summary of key takeaways, features of the pub, and quick links to connected content like the related project narrative, data, or protocols. Section headings will be similar for each “pub type” (e.g. Resource, Observation, Negative Data) so regular readers can get used to their organization, but we will always prioritize making an individual pub as clear as possible. Some sections are similar to those in traditional papers, like “The results” and “The method,” while others, like “What’s next?” feel more like headings you’d see in a news article or blog post.

This informal style is intentional. We’ll use scientific terms as necessary to be accurate and to effectively communicate to other experts, but generally aim to strike a more conversational tone to enhance readability. Long sentences and jargon make papers difficult to parse, and we’re hopeful that we can improve the reading experience for all audiences, including technical experts.

The table of contents makes pubs navigable, providing a bird’s-eye view of what readers will find in the piece and quick links to jump to the desired information. Right now, users can open the table of contents by clicking the “CONTENTS” button on the right side of each pub.

This is a feature that we’d like to improve upon in subsequent iterations of the site. We’ve made the table of contents “sticky,” so it’s always available no matter where within the pub a reader is looking, but we suspect that having it pre-expanded will be much better, especially for those new to the platform. We welcome other suggestions on how to navigate our pubs, especially any examples of sites that do this well.

Non-text elements like figures, code, or other media will appear as close to where they’re discussed as possible to minimize scrolling. Whenever possible, we will directly embed content so readers can consider it in context and don’t have to juggle multiple open windows. Every reference to a figure also contains a link that allows jumping directly to that figure.

This is another area where we’re already imagining future experiments in readability and utility. What if figures always followed you when reading the section in which they’re described? What if you could view figures and text in two separate columns? What if individual elements within figures were clickable and could change to display data differently or show more/less information? We’d again appreciate thoughts on features you’d appreciate, lessons learned from previous attempts at improving figures and other media elements, and examples of people or sites making progress in this area.

Finally, pubs are not static — we will revise them iteratively based on feedback. We’ve decided to start with a three-stage tag system to indicate where we stand in the iteration cycle. At the top of each pub, you’ll see either “Feedback requested” when we’ve just posted it and are awaiting constructive criticism, “Revised after community feedback” when we’ve updated the pub based on comments, or “Revised, no longer actively updating” once the feedback tapers off and any additional comments are better addressed through a new pub.

You can see all release versions of a pub by clicking the rewind/clock icon in the header. When changes more substantial than adjusting formatting or correcting typos are posted, we will include a release note that describes the nature of the changes.

To make our work as useful to others as possible, we are planning to share as much practical, usable material as we can. This means sharing full protocols, raw data, executable code, and more. We’ll strive to always keep our audience and goals in mind when sharing a pub — if we want our readers to be able to replicate our work or apply a method in their own research, we need to include sufficient detail, especially for those whose expertise differs from our own.

When adapting or developing new methods, we aim to post detailed, step-by-step protocols on protocols.io. This pub contains an example. We’ve embedded the protocol directly so it’s easy for readers to skim and determine if it’s relevant, and they can of course also view and use it in a separate window as well.

Our data will be deposited in stable repositories that are findable, accessible, interoperable, and reusable (FAIR) [2]. The goal is for data to meet standards that will maximally facilitate interaction by both humans and machines, further increasing the reach and impact of the open data. In some cases, we may not have yet identified an appropriate repository, or deposition may be in progress. To avoid delays in sharing, we may first share our datasets directly and update links once repository deposition is complete.

We want to share code in ways that make it straightforward to run and make outputs reproducible. This goal is more achievable than ever as cloud computing becomes widespread and browsers support embeddable, in-line programming environments. For example, we are taking advantage of tools like Google Colab notebooks or Github gists, which allow users to easily edit and run code on their own or view code directly within our pubs. We’d eventually like to find a reliable way to share executable code within pubs as well. This pub is a great example of this theory in action, and we appreciate any feedback on its utility.

Citation style

Our chosen citation style is minimal, dropping issue/page numbers and journal titles. While non-experts on a specific topic need to know whether experts have weighed in on scientific research, science is often inappropriately gauged using journal names as a proxy for scientific quality. This is typically based on their presumed level of selectivity or due to inappropriate application of journal-based metrics [3] to individual research articles, as is frequently done with journal impact factors. To reduce the dependence on poor proxies of scientific quality, we have removed journal names from our citations. For simplicity, we have also removed issue and page details, which are less relevant in a digital context, but retained DOI links to make our cited references easily findable. Readers can easily see a citation by hovering over the bracketed citation number (desktop) or tapping it (mobile), and can then click the URL to see the paper.

Other PubPub users can adopt this approach by selecting the “Arcadia Science” style in PubPub’s dropdown list of citation styles. Outside of PubPub, you can access this simplified citation style by downloading the Citation Style Language (CSL) file on our public Github. It will eventually be available in the official CSL repo, pending review, which will make it a style option in many citation management tools.

Authorship, credit, and responsibility

To facilitate transparency and collaboration, we are capturing the substance of individual contributions using a modified version of the contributor roles taxonomy (CRediT). The roles include areas like conceptualization, editing, investigation, methodology, supervision, and more. These roles are indicated next to each alphabetically ordered contributor at the bottom of our pubs. We are extending the taxonomy to include key players not typically included in author lists, such as public commenters who may play an instrumental role in shaping the direction of our work.

The subset of these contributors who are responsible for the content of a pub will have signed off on the released versions and will be “byline” contributors. Whether a contributor is considered byline is determined by the set of roles that they played. We are deciding (and plan to share in upcoming communications) which roles convert a contributor to byline by default, so the system can be applied to each new pub with minimal conflict or confusion. Byline contributors will technically be included as the “authors” in the metadata for the DOI provided through DataCite. These are the names that will appear in citations of our work.

Why make this distinction? What does it mean to be “responsible” for scientific research? When it comes to responsibility and accountability for pubs, we are aligned with the recommendations from the International Committee of Medical Journal Editors (ICMJE), in which byline contributors are accountable for their own work as well as aware/can vouch for the integrity of the contributions of other byline contributors. The byline contributors are similarly the first point of interaction for responding to public feedback and they may engage other contributors as appropriate.

Community feedback as a means of peer review

How will we and others be able to gauge the quality and rigor of our work? Quality control is typically attempted through pre-publication peer review, a process in which manuscripts are sent to journals and experts provide comments that must be addressed or rebutted before an editorial decision is made. While currently considered the gold standard, there’s room for improvement. We’re trying three approaches, outlined below.

First, we want to solicit feedback from a much broader range of people than the two or three included in typical journal-led peer review. If a large number of scientists provide feedback, their comments can be more modular, focusing on the specific area of the commenter’s expertise and requiring less time to contribute. The science of today is also highly interdisciplinary, and with community-level feedback, we hope to achieve scrutiny across a greater portion of a publication, from big picture to nitty gritty methods. For example, a cell biologist might comment on the microscopy, a statistician on the study design, and a bioinformatician on the data analysis, with organismal biologists providing appropriate context. Collectively, the feedback can be more relevant, efficient, and useful.

Next, we want to make all feedback public. While we may not engage with every comment, the contributions of those commenters will still be visible to all and it will be in the interest of our scientists to attend to critical questions. All readers can then learn from and consider the opinions shared, especially those by experts outside of their domain. Beyond PubPub comments, we will also try to capture public commentary on our work from social media.

Finally, this open evaluation will happen over time rather than at a single point, better matching the reality that science is a living, breathing body of work. A rigorous finding is one that stands the test of time, is built upon, and is challenged by orthogonal experimentation. We will solicit feedback, address it, and solicit more feedback. Eventually, we will no longer actively check or update pubs, but comments will always remain open.

Iterating on public review and learning along the way

The scientific literature is growing exponentially [4] and mechanisms for quality control and improvement must adapt with it. Crafting sustainable and scalable mechanisms of peer review may be the trickiest piece of the publishing puzzle, and we’ll need to test multiple solutions. Here are two major issues we anticipate and initial ideas for how to deal with them:

Will the scientific community participate? Will experts provide sufficient comments to yield confidence in our work without traditional editorial motivators? The heavily managed process in which journal editors with stature directly request reviews cannot be replaced by shouting new science into the void and hoping people will notice. We, too, will need to actively solicit feedback. Now that our initial content is available, we’re beginning to explore ideas to cultivate engagement with our work, and hope to relay our learnings in future pubs. We welcome ideas too!

Without the pre-existing audience of a journal, how will we attract interest at all? We hope that, in addition to actively soliciting reviewers, doing the kind of work that matters to people will help us build an invested community. We will have to produce work that can easily be used and improved by others. We must demonstrate true impact over time, and welcome feedback from all who might benefit from our contributions. We must also be responsive to the community’s bigger-picture direction, being willing to re-chart our scientific path to better serve the greater need.

Our own contribution to public review

Why should anyone donate their time to give public feedback to a company? We aim to be good community members in return.

First, we aspire to generate the kind of data, insights, and technologies that can dramatically advance what scientists in the community are able to do in their own work. Feedback will just make the quality of our contributions and responsiveness to the needs of the community that much better.

Second, we are asking our scientists to actively participate in commenting on preprints and other public research products at a level commensurate with their own research output. Arcadia will appear as a reviewer group on Sciety and our comments on preprints will be displayed there and on bioRxiv. We hope our comments help the authors of those products improve their own work such that we can contribute to the collective advancement of knowledge and earn their valuable feedback.

Standing on the shoulders of the open science community

While we have the unique opportunity to run agile publishing experiments through a growing research organization, this first version of this model was inspired by the work of many prior and forthcoming open science efforts. These include the preprint servers arXiv, biorRxiv, OSFpreprints, and so many others. Our efforts are also inspired by the F1000 research platform for post-publication review, the “publish, then review” efforts at eLife based on a publish-review-curate model, crowd review efforts at ASAPbio.org, community preprint review efforts such as PreReview and Peer Community In, and activities of many other organizations represented at Incentivizing Collaborative Open Research (ICOR), who are innovating on open and reproducible efforts throughout every stage of the research cycle.

Acknowledgements
- We are grateful to everyone who has contributed to this first iteration of our publishing site. Thank you to Mert Celebi for weighing in on countless technical questions and developing solutions to key challenges. Thank you to Seemay Chou for all your thoughtful insights and for making the time to help pull everything together in the past few weeks. Thank you to Gabe Stein and everyone on the PubPub team for being great thought partners and making so many site adjustments for us. Thank you to the whole And—Now team for creating and implementing fantastic designs incredibly quickly. Finally, thanks to all the scientists at Arcadia for your amazing ideas and patience over the past few months! Special shout-out to our publishing pioneers, Peter Thuy-Boun and Jase Gehring, for being brave enough to be the first of our scientists to release their work in this new way.
Competing interests
- P.A. is the president of ASAPbio and serves on the board of directors at eLife, groups that are also experimenting with scientific publishing.

Share your thoughts!

Pub details

Content 2 contributors

4 references

Activity 117 discussions

39 social posts

This work is licensed under CC BY 4.0

Purpose
Introduction
The platform
Initial framework
Where can our research be found?
How can the scientific community use our work?
What do we intend to share?
Who will contribute to our pubs?
When will we share our work?
Site organization
Project narratives
Pub structure and style
Sharing protocols, data, and code
Citation style
Authorship, credit, and responsibility
Community feedback as a means of peer review
Iterating on public review and learning along the way
Our own contribution to public review
Standing on the shoulders of the open science community

Prachee Avasthi

Conceptualization, Editing, Supervision, Writing

Megan L. Hochstrasser

Conceptualization, Editing, Writing

Michael Hoffman on May 31, 2022

Selection

gramming environments. For example, we are taking advantage of tools like Google Colab notebooks or Github gists, which allow users to easily edit and run code on their own or view code directly within our pubs.

Why gists? Gists can be created instantly but are relatively anonymous and unstructured. I would think having a repo for each Pub would be a better match, and would be more easily discoverable through GitHub rather than mainly through the Pub itself.

Leron Perez on Jun 03, 2022

Megan L. Hochstrasser on Jun 07, 2022

Thank you for the feedback! Most of our code will actually be shared via Github repos. Gists are just easy to visually embed into PubPub at the moment. We thought there might be value in letting readers quickly see the code without leaving the page and then being able to click out to Github, Colab, or elsewhere if they decide to investigate further or try using it themselves. We’ve applied the same approach with protocols, visually embedding them largely as a preview of what to expect and letting users click to see the full protocol in a format that’s easier to navigate.

Michael Hoffman on May 31, 2022

Selection

t example of this theory in action, and we appreciate any feedback on its utility.Citation styleOur chosen citation style is minimal, dropping issue/page numbers and journal titles. While non-experts on a specific topic need to know whether experts have weighed in on scientific re

The data I get when I click the “Cite” link uses a “journal article” entry type with “Arcadia Science” in the journal field. Personally, I have had endless issues dealing with bibliography management software and journal production systems with citing electronic journals that don’t use traditional volume and page/article number systems. So I wish you would either use some other entry type or add article numbers.

Gabriel Stein on Jun 02, 2022

thanks for the note, Michael. Which bibtex type do you think should be used for content like this? It does feel like @misc kind of undersells it, but I agree that the journal article isn’t quite the right fit, either.

Michael Hoffman on Jun 03, 2022

This seems to fit best into the technical report style. @report or @techreport in BibTeX

Megan L. Hochstrasser on Jun 07, 2022

Thank you for the feedback! We’re likely going to use “techreport” going forward to minimize clashes with citation management software.

Kat katherinebaney@gmail.com on May 31, 2022

Selection

el more like headings you’d see in a news article or blog post.This informal style is intentional. We’ll use scientific terms as necessary to be accurate and to effectively communicate to other experts, but generally aim to strike a more conversational tone to enhance readability. Long sentences and jargon make papers difficult to parse, and we’re hopeful that we can improve the

When I started reading I immediately felt the value of a narrative story in the plural first person. The mention of specific hurdles and confusions that you overcame as scientists adds soooo much implicit value to understanding the research story. This stuff is missing from traditionally published papers.

Seemay Chou on May 31, 2022

@Kat - I totally agree. So much of science is hurdles/confusion - we are doing ourselves and the process a disservice if we write it out of the narrative. I also worry about how we can properly teach students about the process without this

Kat katherinebaney@gmail.com on May 31, 2022

Selection

s and trainees to study as they set expectations for their own research. Pub structure and stylePubs are our standalone “units” of research output.In structuring individual pubs, we’ve tried to strike a balance between improving usability and wh

It is valuable for each unit (Method, Data, etc…) to exist in its respective Context — this offers a massive amount of approachability to the concepts in the pub and allows them to stand alone, however the redundancy might be high effort for the author and annoying to navigate for a reader familiar to the field. I think there must be another strategy that enables the best of both worlds: approachability without redundancy.

Aditi Sengupta on Jun 01, 2022

Had similar thoughts. When dealing with multiple units of research output (e.g. a new set of results from the same study), how would one add the information and how will it be connected to a previous unit of the same research? Does one leave out the big picture intro/background, methods, and instead provide a link to the older unit?

Leron Perez on Jun 03, 2022

I strongly agree with this separation of narrative data and methods.

git provides a model for how to represent these dependencies:

data, methods, and narrative can each be their own independent version-controlled products (ie via git).

A new experiment may copy/reference (simple citation) or fork (copy + edit) previous research artifacts to arrive at a remixed research product with dependencies specified by its citations.

A particular experiment depends on snapshots of the referenced works from a particular moment in time. As the referenced works are updated (ie more data is added to a database), the citing work can be re-compiled (ie re-run the existing code on the expanded dataset).

A particular dataset/method/narrative can have a branching history as it is adapted for various purposes.

This all already exists in Github/Gitlab and could be adapted for scientific works.

Here is an example of one such effort: https://github.com/gchure/reproducible_research

Megan L. Hochstrasser on Jun 06, 2022

Thanks for these comments! I’m very interested in iterating on this. Initially, we’re hoping the “Project Narratives” (new name for “Project Pages” that will appear in our next releases) will help — they’ll provide somewhat evergreen background and sprinkle in context between pubs so that authors only need to add a few framing sentences or paragraphs to each pub to allow it to stand alone. Like you said though, we need to balance approachability and redundancy.

For now, project narratives are linked in the intro to each pub, but I’d be curious to explore ways to automatically pull background in from the main project in a way that is expandable so that more knowledgeable readers can skip it, and those who want background can access it. We could potentially autopopulate an expandable “Big Picture” section/box with key connections to previous/subsequent pubs within a project, project-level goals, and other content.

I think we’ll learn a lot about what we need to do here and what will/won’t work as our projects begin to have larger #s of pubs, projects are abandoned or rethought, etc. Right now, we’re not planning on versioning/forking the project narratives but instead treating them as fluid pages that readers should expect to change over time. They won’t get DOIs but will still be citable via their URL. Again, this is our v1 thinking, it may evolve as we encounter real-world use cases.

julien colomb on Jun 01, 2022

super interesting, have not read all of it yet, but it would really nice to discuss:\

extension of CREDIT: how to make it usable by all, should this go to the CRO ontology. Would you have some funding to revive CRO development?\
author metadata in pubpub, and its reuse: you may want to look/participate in the jams initiative: see https://github.com/jam-schema/jams/issues/7\
As a general remark, the link to twitter and google scholar is not aligned with the open science objective, it should at least be also schema.org (also used by google), and fediverse (open source alternative to twitter) compatible. Maybe a link to Wikidata would be feasible ?

Megan L. Hochstrasser on Jun 06, 2022

Thank you for all these ideas! Right now, we’re not planning to fund external efforts, but we will look into the resources you’ve pointed out.

Regarding your third point, as we test other models of sharing and engagement, we’re doing our best to balance pure open science principles with the need to interact with scientists where they are in the existing ecosystem. For example, we’d love for our work and discussion to be maximally replicable via open mechanisms, but we’re also cognizant that Twitter is a major hub for scientific discussion, and we don’t want to miss that. We’re definitely interested in capturing conversations wherever they are, and will need to think about how to track comments on different sites, especially as social media channels change over time.

Aditi Sengupta on Jun 01, 2022

Selection

ow to navigate our pubs, especially any examples of sites that do this well.Non-text elements like figures, code, or other media will appear as close to where they’re discussed as possible to minimize scroll

I was looking forward to a graphic to understand the process. There is a lot of information and “docking points” that talk about specific topics. I was getting lost on how each of those docking points are connected. A figure might help readers here.

Megan L. Hochstrasser on Jun 06, 2022

Thanks for this feedback! We will consider making a graphic or figure.

Aditi Sengupta on Jun 01, 2022

Selection

ial CSL repo, pending review, which will make it a style option in many citation management tools.Authorship, credit, and responsibilityTo facilitate transparency and collaboration, we are capturing the substance o

As an early career researcher, will my publication in this format count towards hiring, appointment, rank and tenure? That is my biggest concern. I am all for this format but I do not know what the thoughts of those sitting on the other side of the table deciding if my scholarship efforts were enough and/or meritorious when published on this platform. I do not have a solution at this point. Just thinkng through things. I am sure this is not the first time such a question has been brought up.

Prachee Avasthi on Jun 06, 2022

Hi there. This is a great question and one we are running the experiment to answer. All evaluators are likely to make their own decisions on what “counts,” but it’s our feeling that the materials with the most transparency (public materials with public feedback) give evaluators the most information possible upon which to make their decisions. It is also our hope that by virtue of the existence of other avenues of research sharing that are attractive to scientists, evaluators might expand what they consider, as has happened in some circumstances with preprints. This shift won’t happen overnight, but needs to start somewhere, so we hope to be able to chart the path that others might feel comfortable trying once they see it in action.

Leron Perez on Jun 03, 2022

Selection

the raw data will live on the PubPub platform.How can the scientific community use our work?Each pub is citable and will ultimately have a permanent identifier (like a digital object identifier, or DOI, through DataCite).Each pub will follow rules for Goo

will major revisions have a new DOI? Or will there be a way for citing publications to refer to a specific snapshot of the publication?

what happens if there is a revision that contradicts or significantly changes the originally interpretation of the pub?

Megan L. Hochstrasser on Jun 06, 2022

Great questions! We are currently planning to keep a single DOI for each overall pub. For all revisions, users can cite specific versions by including the URL to the specific release. Note that PubPub saves each version (aka “release”) and provides a unique URL for each.

Regarding contradictory or otherwise major changes to the interpretation, we definitely do not want to hide the original data/interpretation, nor do we want people to stumble upon it and miss the critical update. Thus, at a minimum, we will add a hard-to-miss note to the top of the pub and make edits within, and may add an entirely new pub with its own DOI to relay the new data, discuss how the interpretation changed, why this has happened, etc. Any major (non-contradictory) additions to our data or understanding will be added as new pubs.

So far, we’ve learned a ton each time we’ve encountered a new scenario and ended up rethinking our plans, but this is our initial outlook.

Matthew Akamatsu on Jun 09, 2022

I wonder if ISCC codes could be more flexible than DOIs in trying to track different versions of one pub: https://iscc.codes/ One part of the identifier stays the same for a given pub, and another part of the code changes with each version

5A64 (Pseudonym) on Feb 02, 2023

A note that DOI versioning is possible. Zenodo implements this feature.

Megan L. Hochstrasser on Oct 09, 2024

Interesting ideas — thanks!

Leron Perez on Jun 03, 2022

Selection

to share as much practical, usable material as we can. This means sharing full protocols, raw data, executable code, and more. We’ll strive to always keep our audience and goals in mind when sharing a pub — if we wa

will the code be documented or just shared?

Megan L. Hochstrasser on Jun 07, 2022

As much as possible, code will be documented. We want to balance reproducibility and utility with how much time it takes our scientists to add documentation, so the extent of documentation may differ from case to case.

Leron Perez on Jun 03, 2022

Selection

dapting or developing new methods, we aim to post detailed, step-by-step protocols on protocols.io. This pub contains an example. We’ve embedded the protocol directly so it’s easy for readers to skim and deter

this link is broken

Megan L. Hochstrasser on Jun 06, 2022

Thank you! It will be fixed in the next release. Here is the intended link: https://research.arcadiascience.com/pub/method-mass-spec-proteomics-transcriptomics

Leron Perez on Jun 03, 2022

Selection

about and areas for improvement. We don’t want to clutter our scientific pubs with comments about website functionality, aesthetics, or our model, but we invite you to share those thoughts here! All first drafts need fe

minor: the banner at the top of the page with the title does not appear to be adjusting to my window size on Firefox

Leron Perez on Jun 03, 2022

actually not so minor, makes it difficult or not possible to interact with the comments as well

Megan L. Hochstrasser on Jun 03, 2022

Noted, thank you! Replicated on my end as well, we’ll address this.

Matt Clancy on Jun 03, 2022

Selection

ience into the void and hoping people will notice. We, too, will need to actively solicit feedback. Now that our initial content is available, we’re beginning to explore ideas to cultivate engagement with our work, and hope to relay our learnings in future pubs. We welcome ideas too!Without the pre-existing audience of a journal, how will we attract interest at all? We hope that,

This is the key challenge and I look forward to seeing your ideas for incentivizing people to engage or creating new norms around it. In my field, it is normal to circulate drafts for comments among people in your network; if similar norms exist in your field, you might be able to co-opt them and simply encourage authors of new pubs to circulate to interested parties, noting that any feedback is welcome, especially on the platform.

Megan L. Hochstrasser on Jun 06, 2022

Agreed! We have been trying this with our initial pubs, jury’s still out on how well it’s working since we’re still in the middle of the process. We’re also hoping to hire someone to think about this full-time!

Matt Clancy on Jun 03, 2022

Selection

an invested community. We will have to produce work that can easily be used and improved by others. We must demonstrate true impact over time, and welcome feedback from all who might benefit from our contributions. We must also be responsive to the community’s bigger-picture direction, being willing to re-chart

I have a similarly structured pub pub website built on modular updatable articles with a synthesis overlay, and have had a lot of luck with an email newsletter that functions more or less like a changelog of the website. Subscribers receive emails for each new pub, and once a month I also send out an email bundling up all the updates I’ve made to the site. Each email has most of the new text in it, so people do not have to leave their inbox to read it, but I include a disclaimer that the underlying linked article is always updated, and that people should follow the link to see the latest version.

The tradeoff is people mostly engage with the email newsletter, rather than the main site. But it is a relatively frictionless way to tell people what’s new, and substack has built a very shareable product that attracts interested subscribers quite well.

Megan L. Hochstrasser on Jun 06, 2022

Oh this is an interesting approach! We’re just setting up our subscription options now, and it never would have occurred to me to include the new text directly within the email. Currently using Mailchimp but might look into Substack!

Austin H. Patton on Jun 03, 2022

Selection

ucture and include whatever media can best convey the desired information to the intended audience. This may mean including executable code blocks, interactive content, and other resources embedded within pubs.What do we intend to share?The nature of each pub

I don’t know if there is any existing infrastructure for such a thing, but I wonder if there might be value in having not only an executable code block, but one that’s interactive and modifiable - modifications could be shared by the community.

I think this sort of interactivity could facilitate data exploration, and provide a framework for scientific engagement by the community that’s currently lacking in traditional scientific communication. But again, this is pretty “pie in the sky” since that infrastructure might not even exist yet!

Leron Perez on Jun 03, 2022

Interactivity exists to varying degrees in Javascript (Observable), Python (Jupyter), and Julia (Pluto).

See also:

https://nextjournal.com/

https://explorabl.es/

http://worrydream.com/LadderOfAbstraction/

Austin H. Patton on Jun 03, 2022

Related, it seems like Colab, which they use in the spatial genomics/MERFISH pub could facilitate this sort of interactivity and engagement/communication between the public and Arcadia?

Feridun Mert Celebi on Jun 08, 2022

Hi Austin and Leron,

First off great questions! And thanks for sharing these resources, Leron. Nextjournal especially looks interesting.

To answer the original question: a lot of this is possible and has gotten easier over the last couple of years. The solutions are not perfect yet but there are many tools out there that combine the code and the compute environment. We’ve experimented with Colab and Binder and are excited about what these tools enable. They’re both hosted Jupyter notebook solutions. They both work nicely with Github (though Binder is much better at this given the tight integration with repo2Docker). And they both support different forms of collaboration and “forking”, so it’s easy to modify the code and share your modified version with others.

However, there are some limitations and they require some thoughtfulness to make them easily usable by others. The biggest limitation is the amount of compute that is available freely. So we have to be thoughtful about what we can host and how much compute it requires. Also, the data that’s used in the notebooks needs to be public and should be part of the setup step, though this is not a terrible problem.

Regardless, we are excited about the possibilities here and will be experimenting more around our tooling so that the code we share is interactive and modifiable by others.

Austin H. Patton on Jun 12, 2022

Hi Feridun, thanks for the response! Happy to keep the conversation going.

This all is super interesting - and I’d not yet thought about the challenge/issue of the balance between providing executable or interactive code, while navigating the limitations of computing resources. I can see how non-interactive code blocks keeps the demand on computing resources “reigned in” - something that enabling interactivity could break. I’m sure this is not an easy problem to solve, but an exciting direction for sure!

Nate Jacobs on Aug 25, 2022

Also checkout papers with code for ideas!

Austin H. Patton on Jun 03, 2022

Selection

rs can open the table of contents by clicking the “CONTENTS” button on the right side of each pub.This is a feature that we’d like to improve upon in subsequent iterations of the site. We’ve made the table of contents “sticky,” so it’s always available no matter where within the pub a reader is looking, but we suspect that having it pre-expanded will be much better, especially for those new to the platform. We welcome other suggestions on how to navigate our pubs, especially any examples of sites that do

I like this idea of having it pre-expanded. It also seems like if could be useful if in this table of contents you could directly link to other related pubs under the same umbrella. For instance in the table of contents for the project page, including links to subsections of not only that document, but also linking to the “downstream” pubs - data, methods/protocol, results, etc. Could help to more seamlessly integrate the different pubs?

Megan L. Hochstrasser on Jun 06, 2022

I like this idea! When you’re viewing a pub, you can currently click the project title tag at the top of the header to see a dropdown of all the pubs included within that project. This functionality could be useful on the project narratives themselves as well. As the project narratives grow, this sort of organization/navigability will become more important.

Austin H. Patton on Jun 12, 2022

Yeah, that could be a perfect place to implement that. I wonder if that could also track down with the page header so it’s always easy to access without scrolling back to the top of the page?

Megan L. Hochstrasser on Jun 13, 2022

Love that idea!

Austin H. Patton on Jun 03, 2022

Selection

h “project pages,” narrative pieces that link individual pubs and keep the big picture in focus.While versions of the work will have permanent identifiers so they can be cited, work is not static — evaluation will happen after the pub is publicly released. The products will be improved/built upon over time, incorporating feedback throughout.Being responsive to community needs?Who will contribute to our pubs?Each person who cont

Will there be a sort of “version control” for citation? Given that these pubs are continually being revised, it seems that it would be important that when citing these works, the citation should link to a specific version, since findings may change/be modified in later iterations.

Seems like the three-stage tag system could be used/integrated to facilitate this?

Megan L. Hochstrasser on Jun 06, 2022

Thank you for these questions and ideas! PubPub supports versioning and lets users view all previous “releases” of a pub, each with its own URL and with the option to add a “release note” explaining what has changed. We are currently planning to keep a single DOI for each overall pub. For all revisions, users can cite specific versions by including the URL to the specific release.

For contradictory or otherwise major changes to the interpretation, we definitely do not want to hide the original data/interpretation, nor do we want people to stumble upon it and miss the critical update. Thus, at a minimum, we will add a hard-to-miss note to the top of the pub and make edits within, and may add an entirely new pub with its own DOI to relay the new data, discuss how the interpretation changed, why this has happened, etc. Any major (non-contradictory) additions to our data or understanding will be added as new pubs

Applying the tagging system here is an interesting idea as well. We could consider adding a “Major Revision” or “Critical Update” tag, or otherwise noting this somewhere.

Austin H. Patton on Jun 12, 2022

Of course! That’s great there will be a URL for each version - that should lend to fairly unambiguous linking of individual citations to the version in question.

I think this combination of including those types of notes, along with different tags (i.e. minor/major revision or critical updates) at the top of updated versions would work well to clearly track/communicate each pubs development. Perhaps this could be incorporated into the citation format as well to allow for citing the static DOI, including the url for the cited version, and the tag indicating what “stage” the citation is referencing?

Megan L. Hochstrasser on Jun 13, 2022

Oh good catch! I think at a minimum, the “Cite” button should load the URL for the specific version you’re viewing rather than the base URL.

Austin H. Patton on Jun 03, 2022

Selection

hese roles are indicated next to each alphabetically ordered contributor at the bottom of our pubs. We are extending the taxonomy to include key players not typically included in author lists, such as public commenters who may play an instrumental role in shaping the direction of our work.The subset of these contributors who are responsible for the content of a pub will have signed off

I love this “open arms” approach - this is a fantastic incentive for encouraging involvement from the public!

Austin H. Patton on Jun 03, 2022

Selection

This is great - directly gets at the frustation of bouncing around from text and figure in traditional journal publications! To improve legibility while “following” throughout the section, figures could expand/contract based on reader selection? That way you can always get to the figure you’re looking for, but it’s impact on reading the text is minimal?

Megan L. Hochstrasser on Jun 06, 2022

Great suggestion!

Leron Perez on Jun 03, 2022

Selection

atus of the research, where we’re heading next, and, sometimes, how you can influence our direction. Project pages are always evolving, and will not have listed authors or DOIs.We think they’ll be particularly important because we’re going to publish much smaller pieces of r

in terms of measuring scientific progress, it seems like projects are a very important piece to track and that they should be considered together with pubs.

if projects remain unquantified, while pubs are counted and cited, I am concerned that incentives may continue supporting maximizing/focusing on pubs rather than projects.

The best solution might not be including a DOI, but I am concerned that in the current system, projects will remain in the periphery of scientific documentation and communication because they are not associated with measures of progress.

A last thought: perhaps including a DOI for projects could be accompanied with norms that the project DOI is cited for motivations/context building, while specific results, numbers, or methods could be cited from the individual pubs.

Megan L. Hochstrasser on Jun 06, 2022

Good points, and we’ll keep these considerations in mind. Given that project narratives may change substantially, we’ve chosen not to use DOIs for now, but they will be citable via their URL. I do think some sort of versioning could be useful in the future, but I think we’ll have to see what happens to these over time and decide on a strategy based on the real examples that come up.

Matthew Akamatsu on Jun 09, 2022

Related to a comment I made above: maybe ISCC codes could also be used to track which project each pub is part of? Part 1 of the code would be Arcadia publications, part 2 a particular project, part 3 a pub, and part 4 a particular version of the pub? https://iscc.codes/ I think that might let you track how much a particular project or pub gets cited

Megan L. Hochstrasser on Jun 13, 2022

Will look into this, thank you!

Leron Perez on Jun 03, 2022

Selection

g to publish much smaller pieces of research than what is traditionally shared in journal articles. With so many modular publications, or “pubs,” it could become tough to follow the thread. Project pages will string the individual pubs together into a coherent narrative, summarizing emergent takeaways and laying out planned next steps. Though pubs will work as individu

Will this introduce a tendency to editorialize and revise the biology to fit a story even when it may not?

Narrative should live separately from data and methods (in pubs and projects)

will dependencies between projects be tracked? ie one project motivated experiments that were inconclusive in the former context, but that led to the initiation of a new project. Will that new project cite the old project somehow. Is the experiment part of the new project or the old project?

Perhaps projects and pubs could benefit from a model like git: ie projects can be revised and changed in time but forks (citations) are specific to an particular revision

Megan L. Hochstrasser on Jun 06, 2022

Partially addressed this in a previous reply, but will summarize here! I agree that we may end up wanting some sort of versioning/forking with project narratives in the future.

Right now, we’re hoping to avoid editing the intro/goals substantially and to verbally describe shifts in the project’s direction within the “Progress” section. If we do end up drastically changing direction/goals, we’ll likely create a new project narrative page or otherwise attempt to clearly explain and document the shift.

We can easily hyperlink between projects, but it could also be cool to eventually develop a way to embed projects like we do with pubs or add a higher-level page type that tracks links between projects themselves as they evolve, inspire new projects, etc.

Thanks for all this food for thought!

5A64 (Pseudonym) on Feb 02, 2023

Re: hyperlinking between projects. It might be helpful to visualise this graphically as tree trunks and branches. I’m thinking something similar to Obsidian’s graph view. May be a better solution to tracking projects/pubs as Arcadia scales.

Leron Perez on Jun 03, 2022

Selection

ally any examples of sites that do this well.Non-text elements like figures, code, or other media will appear as close to where they’re discussed as possible to minimize scrolling. Whenever possible, we will directly embed content so readers can consider it in context and don’t

what if the reader could chose which figures to simultaneously view? what if they could view more than one?

Megan L. Hochstrasser on Jun 06, 2022

Love these ideas! 💡

Leron Perez on Jun 03, 2022

Selection

ppreciate, lessons learned from previous attempts at improving figures and other media elements, and examples of people or sites making progress in this area.Finally, pubs are not static — we will revise them iteratively based on feedback. We’ve decided to

see https://cremerlab.github.io/flux_parity/

Megan L. Hochstrasser on Jun 06, 2022

Wow! So cool!

5A64 (Pseudonym) on Feb 02, 2023

(Potential glitch: clicking on links within comments leads to the same pub, it doesn’t direct to the actual link. I had to manually copy and paste into a new tab. Using Chrome on MacOS)

Megan L. Hochstrasser on Oct 09, 2024

This is because pasting a URL in a comment would turn it into a hyperlink to the pub — PubPub has since changed this!

https://example.com/

Matthew Akamatsu on Jun 09, 2022

Selection

oration, we are capturing the substance of individual contributions using a modified version of the contributor roles taxonomy (CRediT). The roles include areas like conceptualization, editing, investigation, methodology, supervision,

It would be nice to be able to query based on these roles. For example, I could imagine having a contributor page that describes the nature of my contributions to various projects. It should be possible to start with an autofill of my contributions; i.e. “conceptualization role in pubs x and y for project z.” A systematic treatment like this might make it more likely for these taxonomic roles to be considered, both for contributors and evaluators.

Megan L. Hochstrasser on Jun 13, 2022

This is a cool idea! For reasons I won’t get into, the “Contributors” section you see right now is added manually, with the same information separately added to the metadata behind the scenes. This should change in the future, and once it does, you should be able to click on a contributor’s name to see their profile, which lists all pubs to which they’ve contributed. I like the idea of being able to additionally sort by contribution type!

Matthew Akamatsu on Jun 09, 2022

Selection

much broader range of people than the two or three included in typical journal-led peer review. If a large number of scientists provide feedback, their comments can be more modular, focusing on the specific area of the commenter’s expertise and

Just like in using hypothes.is, it might become overwhelming to readers to parse which comments they find useful. I wonder if it will become necessary to filter or prioritize comments? Having upvotes might also help to gauge how much engagement a particular article is getting, and how pressing a comment might be.

I also like the idea of subjective filtering, where you can rate comments as well as other raters. If I highly rate e.g. Prachee’s opinion, then comments that she rates highly will appear first for me. This “subjective review” is implemented at https://braid.news/ .

Jacob Bumgarner on Jun 28, 2022

+1 to comment rating.

Along this vein of thought, will there be comment moderation?

5A64 (Pseudonym) on Feb 02, 2023

+1 to comment rating.

Matthew Akamatsu on Jun 09, 2022

Selection

ted content like the related project page, data, or protocols. Section headings will be similar for each “pub type” (e.g. Resource, Observation, Negative Data) so regular readers can get used to their organization,

I am really curious about whether pubs will naturally sort into a specific set of “pub types.” Like, will each novel finding be most easily communicated as an individual Observation, and later on an Integration cites a few Observations to draw a new conclusion? Or will the pub types become really heterogenous? If the former, then I think it will be really powerful to be able to query based on pub type; e.g. find Observations that are referenced by a particular Integration. Example: https://roamjs.com/extensions/discourse-graph/synthesis-query

Megan L. Hochstrasser on Jun 13, 2022

I’m curious about this too. So far, I’ve outlined a set of about a dozen pub types that I think can work for most use cases and I’m trying to have us stick to these (I’ll share the list in a future version of this pub, or perhaps as a page that explains key features of each type). We’ll see how straightforward it is to use this standard set as we try to categorize future findings.

I like that feature of Roam! I’ll add it to my list of things to think about as we develop our framework and think about how best to connect and query for individual pubs. PubPub has a “Connections” feature that lets us connect pubs to each other and to external sites (e.g. deposited data) that may be the right structure for this — I’ll have to look into how searchable it is.

Matthew Akamatsu on Jun 09, 2022

Selection

l reflect the current status of the research, where we’re heading next, and, sometimes, how you can influence our direction. Project pages are always evolving, and will not have listed authors or DOIs.We think they’ll be

It would be fun for there to be space for community contributions on the project page. In an open-source model, other researchers could help address a given research question, provide hypotheses, or help out with a particular analysis. And then get credit if the contribution becomes useful.

Megan L. Hochstrasser on Jun 13, 2022

Yes! We definitely intend to add community members as “contributors“ to pubs if they provide critical feedback. I think it’ll be important to show how we’ve incorporated comments into later versions of the pubs to 1) show people that we value their feedback, 2) be fair about how people are credited/rewarded, and 3) demonstrate that our post-publication review approach is working.

Major contributions can be highlighted on the project narratives too, via written shout-outs. We’ll see how this evolves if we eventually end up designing those pages to function closer to pubs (which I suspect we may end up doing later on).

Jacob Bumgarner on Jun 28, 2022

Selection

released. The products will be improved/built upon over time, incorporating feedback throughout.Being responsive to community needs?Who will contribute to our pubs?Each person who contributed to different aspects of the scie

A general proofing comment - it seems like this question was meant to be addressed/filled out in the drafting stage.

Megan L. Hochstrasser on Jun 28, 2022

Good catch! Not sure how that made it through.

Jacob Bumgarner on Jun 28, 2022

Selection

Each person who contributed to different aspects of the science will be listed as a contributor — including those commenters whose critical feedback shaped the work.Each contributor’s role(s) will be stated explicitly and in a standardized format, making the na

Will there be stated guidelines that indicate the level of commenter contribution needed to be listed as a contributor?

E.g.,\

Those that proof the publications for grammatical issues won’t be included as contributors\
Those that provide ideas for new critical experiments/alternative analytical approaches will be listed as contributors.

Megan L. Hochstrasser on Jun 28, 2022

This is a great question. We’re developing a list of definitions for all contributor roles, including “Critical Feedback,” which can apply to public commenters or colleagues who weigh in early in the research process. Since we’re building on the casrai.org “CRediT” system but diverging in a few key ways, we plan to share this list once it’s built out. It could potentially be a standalone pub to support versioning, since we may make changes over time, otherwise we’ll add it as a page.

Based on your feedback here, I think eventually we may want to add a page just intended for would-be commenters that explains how to comment, how comments may be used, in what circumstances someone may be credited as a contributor, etc.

Jacob Bumgarner on Jun 28, 2022

Selection

n access this simplified citation style by downloading the Citation Style Language (CSL) file on our public Github. It will eventually be available in the official CSL repo, pending review, which will make it a sty

This is a general comment that applies to all of the links in this pub, but it would be nice if links automatically opened in new tabs so that the readers aren’t directed away from the publication page.

Megan L. Hochstrasser on Jun 28, 2022

Agreed! We can’t control this right now, but it’s on our list of requests.

Jacob Bumgarner on Jun 28, 2022

Selection

o jump to the desired information. Right now, users can open the table of contents by clicking the “CONTENTS” button on the right side of each pub.This is a feature that we’d like to improve upon in subsequ

A general website aesthetic comment - the table of contents dropdown covers the text.

I’m viewing the publications on Safari 15.4.

Megan L. Hochstrasser on Jun 28, 2022

Oh, thank you! I actually don’t see it that way on Safari on my end, so this is great to know. I’m definitely seeing issues with the ToC in general across browsers too. We’ll work on this!

Bruce R. Caron on Jul 02, 2022

Selection

hieve scrutiny across a greater portion of a publication, from big picture to nitty gritty methods. For example, a cell biologist might comment on the microscopy, a statistician on the study design, and a bioinformatician on the data analysis, with organismal biologists providing appropriate context. Collectively, the feedback can be more relevant, efficient, and useful.Next, we want to make all

This type of broad, but focused public feedback inside the article is of much greater use than any current closed review system.

Bruce R. Caron on Jul 02, 2022

Selection

y of our contributions and responsiveness to the needs of the community that much better. Second, we are asking our scientists to actively participate in commenting on preprints and other public research products at a level commensurate with their own research output. Arcadia will appear as a reviewer group on Sciety and our comments on preprints will be displayed th

What about doing overlay journals?

Prachee Avasthi on Jul 11, 2022

Overlay journals are fantastic and a welcome addition to the post-publication review and curation ecosystem. As our primary focus is as a research organization rather than a review entity, we aren’t formally organizing review or rendering decisions the way a journal might but rather want to make sure that the comments for work we read/review are captured publicly so that readers can benefit from a richer set of available information.

Bruce R. Caron on Jul 02, 2022

Selection

ganization, this first version of this model was inspired by the work of many prior and forthcoming open science efforts. These include the preprint servers arXiv, biorRxiv, OSFpreprints, and so many others. Our efforts

An invitation to take a look at the Open Scientist Handbook, here on PubPub: https://openscientist.pubpub.org/

Steven Massa on Jul 13, 2022

A small thing: when I click on a comment which is tied to a hyperlink in the text, it takes me to the linked page instead of opening the comment. I’m encountering this issue with Jacob Bumgarner’s comment in the Citation Style section. If I try to add a comment near there, it also automatically directs me to the linked page. My browser is Chrome Version 103.0.5060.11.

In relation to this, should comments about technical issues with the webpage be separated somehow from comments about the content of the pub itself?

Megan L. Hochstrasser on Jul 13, 2022

Thank you for pointing this out! We’ve relayed this bug to the PubPub team and it’s in their queue. Please do point out technical issues on this pub—I’d like to be aware of them myself and can make sure they get attention.

Nate Jacobs on Aug 25, 2022

Selection

uch of the Twitter conversation as possible by embedding a feed collecting mentions of each pub.When will we share our work?We will post publicly at multiple stages of a project. Pubs are modular and we will share them w

This is a great testing ground for new workflows and finding what will work for the community. Really excited to see how things shake out and what patterns and trends emerge in user behavior, and what that could mean for workflow refinements.

I have found product design tension between building up workflow specifically designed for small modular atomized content (micropubs) and exactly how and when this relates to longer form content. I don’t have any strong answers or intuitions on this yet, but I often think about what twitter would be like if it allowed blog posts. would that be a good thing or bad thing? how would that change user behavior and product focus?

One observation to note is that if you don’t implement specific creative constraints like length and types of content, user behavior will probably drift towards longer and more entropy. which may be a good thing! or it could make it difficult for certain types of content to have the correct incentive and cultural structure. Could twitter have ever evolved from an email or blog platform, or would it only have emerged and gotten traction if it only allowed certain type and format of content?

One more observation is I think it’s important to know how many products and workflows you are trying to support. By not choosing, sometimes that means you are choosing to support a wide variety of products and workflows (ie all the things the users choose to do on the platform). For example, it’s possible this approach could lead to competing user needs for 1) modular content, 2) preprint / blog, 3) more formal publishing and visibility needs for career advancement.

With my work on micropubs, I made a deliberate decision to NOT support longer form content, and am working towards integrations with external platforms that do that better (preprints, journals). I haven’t gotten very far with this, so maybe wait until I have actual data / experience before taking this advice, but it’s something I’ve found useful to at least think through.

I could be completely wrong about all of this so will be curious to see what actually happens in this case!!

I have a strong bias / idiosyncrasy towards constrained design serving narrowly focused workflows, which works well sometimes and other times is a terrible decision haha.

Perhaps an intermediate approach, informed by an initial round of loose experimentation, could be to further constrain (or loosen) format and workflow depending on what authors are doing and finding most helpful!

Megan L. Hochstrasser on Oct 09, 2024

Thanks for these thoughts! We’ve noticed this tension as well, and have seen pubs stretch to become longer and longer. Sometimes it’s useful, sometimes less so. That said, we ultimately want to support whatever length/format our scientists think is best to convey their work, so we try to keep it flexible. We’ll have a lot more options for this when we switch over to the new version of PubPub, called “Platform”

Nate Jacobs on Aug 25, 2022

Selection

d on Arcadia and our goals, and, most importantly, will tell you what’s new. [View landing page]Project narratives are running logs that maintain a high-level view of our goals and connect modular pubs together into a more cohesive story. [View example]Pubs are detailed research products. These are akin to traditional papers, but te

I love this approach and have thought about this a lot (mostly in circles). Curious to see how this goes and happy to share more detailed thoughts if helpful.

For me a few questions I always had with this is:\

what is the best governance structure? Should this be org / journal level - like special features that editors are updating contextual content about? Or a content type that anyone can create?\
If coming from authors / users, how does this compete with (or not) other workflows like preprints, blogs, etc?\
do you treat this as a scholarly object or as a stepping stone / facilitator between micropubs (modular pubs) and longer form content / meta-analyses?

Nate Jacobs on Aug 25, 2022

Selection

l view of our goals and connect modular pubs together into a more cohesive story. [View example]Pubs are detailed research products. These are akin to traditional papers, but tend to have a narrower f

are pubs different than modular pubs?

Megan L. Hochstrasser on Jul 03, 2024

We don’t make a distinction. We aim to make all pubs modular, meaning we try not to jam too much into a single pub but instead split things up based on topic, audience, engagement goals, or other factors. For example, these three pubs (1, 2, 3) started out as a single pub, but it was really tough to read them all together, and the audience for each was distinct, so we went more modular. There are certainly pubs we’ve released that became long and should have gone out earlier as small pubs, though — it’s not always easy to make this happen in practice.

Nate Jacobs on Aug 25, 2022

Selection

er specific biological questions, solve particular problems with technology, or sometimes simply to explore a curiosity.These projects are described on “project narratives.” These are intended to serve as a running nar

all open science endeavors should be empowering researchers to lean into their curiosity again.

Nate Jacobs on Aug 25, 2022

Selection

ng figures and other media elements, and examples of people or sites making progress in this area.Finally, pubs are not static — we will revise them iteratively based on feedback. We’ve decided to start with a three-stage tag system to indicate where we stand in the iteration cycle. At the top of each pub, you’ll see either “Feedback requested” when we’ve just posted it and are awaiting constructive criticism, “Revised after community feedback” when we’ve updated the pub based on comments, or “Revised, no longer actively updating” once the feedback tapers off and any additional comments are better addressed through a new pub.You can see all release versions of a pub by clicking the rewind/clock icon in the header. When ch

what about retraction? or, in the positive direction, replication? Lots of really cool ideas to experiment with here.

I think I saw comment about this somewhere else, but visibility and organization of comments / feedback at different states (in review, final, etc) is worth thinking about. No right answers but can have an effect on how users engage with content.

Nate Jacobs on Aug 25, 2022

Selection

, we need to include sufficient detail, especially for those whose expertise differs from our own.When adapting or developing new methods, we aim to post detailed, step-by-step protocols on protocols.io. This pub contains an example. We’ve embedded the protocol directly so it’s easy for readers to skim and determine if it’s relevant, and they can of course also view and use it in a separate window as well.Our data will be deposited in stable repositories that are findable, accessible, interoperable, an

<3. I think this can be the new methods section. just don’t even try to summarize it, post all the protocols and then reference in context.

Nate Jacobs on Aug 25, 2022

Selection

accessible, interoperable, and reusable (FAIR) . The goal is for data to meet standards that will maximally facilitate interaction by both humans and machines, further increasing the reach and impact of the open data. In some cases, we may not have yet ident

Similar arguments can be made for the structure of the article itself. There’s a lot of really cool meta-analyses etc that could be done if papers were structured a bit better. For me this is more important than for underlying data sets, which may be hard to automate machine use of given the high degree of variability between data sets.

Nate Jacobs on Aug 25, 2022

Selection

y done with journal impact factors. To reduce the dependence on poor proxies of scientific quality, we have removed journal names from our citations. For simplicity, we have also removed issue and page details, which are less relevant in a digital context, but retained DOI links to make our cited references

:) sweet!

Nate Jacobs on Aug 25, 2022

Selection

s, we will also try to capture public commentary on our work from social media.Finally, this open evaluation will happen over time rather than at a single point, better matching the reality that science is a living, breathing body of work. A rigorous finding i

I really love this. Its an aspiration not a decision, but I hope that we can all move peer review towards this new type of workflow.

one thought that I keep coming back to after taking this first step away from deriving “truth” from single article, is does this open up a new opportunity for more robust mechanisms for curation and drawing attention to ground truth?

one example would be content state changes if findings are replicated. another example could be meta-analyses that then establish higher levels of confidence (or lack thereof) in a finding.

for me this is a really important first step towards:\

decoupling peer review / journal brand as proxy for “truth” or “consensus”\
deriving consensus from subsequent research outputs (vs commentary and endorsements), ie data driven peer review with replications, contradictions, and extensions.

Jessica Carstens-Kass on Nov 17, 2022

Selection

All first drafts need feedback. What do you think of this site? Are our first pubs understandable? What important features are we missing? What are we doing right? What can we improve? Please don’t hesitate to share constructive commentar

Hi! For citing your pubs, I see you provide the bibtex code which is great; it would be extra nice to be able to export a .ris file for citation managers like EndNote!

Megan L. Hochstrasser on Nov 21, 2022

Added to my list! Thanks so much for the feedback!

5A64 (Pseudonym) on Feb 02, 2023

Selection

d utility. What if figures always followed you when reading the section in which they’re described? What if you could view figures and text in two separate columns? What if individual elements within figures were clickable and could change to display data differen

+1 on this idea, it’ll make reading a lot of more easier compared to scrolling up and down OR jumping up/down using hyperlinks. For reading papers via PDF on a computer, I would open 2 copies - 1 for the text and then the other for the figure. Even more necessary when journals decide to place the figure legend on a separate page to the figure (e.g. Cell).

PDF of journals have their own fixed structure. The advantage of PubPub should be the flexibility i.e. you can choose two column if you prefer, can change how references are displayed (numbered vs last author / year format etc.)

5A64 (Pseudonym) on Feb 02, 2023

Selection

ection in which they’re described? What if you could view figures and text in two separate columns? What if individual elements within figures were clickable and could change to display data differently or show more/less information? We’d again appreciate thoughts on features you’d appreciate, lessons learned from previous attempts

+1. This would require uploading source data (not just the figure). Would be esp. helpful for barplots as many plots will not show individual biological replicates - which can reveal a lot of info about the spread of the data that would otherwise be hidden if you just presented the mean value.

Easy to implement for bar plots, may be more difficult for non-quantitative data e.g. flow cytometry plots (in immunology, some will choose to use contour plots over dot plots to obscure the fact that there may only be <10 events in a flow cytometry acquisition)

5A64 (Pseudonym) on Feb 02, 2023

Selection

b.You can see all release versions of a pub by clicking the rewind/clock icon in the header. When changes more substantial than adjusting formatting or correcting typos are posted, we will include a release note that descr

Since this is online (not PDF), it could benefit by adopting something similar to Microsoft Word’s track changes or minimally an automatic highlighting feature to track areas where changes occured. (Similar to sending back a manuscript for revision so reviewers can easily identify the changes)

Megan L. Hochstrasser on Oct 09, 2024

We’d love to have this feature!

5A64 (Pseudonym) on Feb 02, 2023

Selection

mplified citation style by downloading the Citation Style Language (CSL) file on our public Github. It will eventually be available in the official CSL repo, pending review, which will make it a style option

(Potential bug: Clicking on a comment that is linked to a text with hyperlink, seems to go straight to the hyperlink, not the comment. On Chrome MacOS)

5A64 (Pseudonym) on Feb 02, 2023

Selection

te engagement with our work, and hope to relay our learnings in future pubs. We welcome ideas too!Without the pre-existing audience of a journal, how will we attract interest at all? We hope that, in addition to actively soliciting reviewers, doing the kind of work that matters to

In regards to a reading audience, I don’t represent everyone, but I don’t go to a journal for papers of interest. I use pubcrawler to give me a weekly feed of articles that match my criterias of interest. If the Pubs are in PubMed (just like recently now with pre-prints), this shouldn’t be a major issue.

I only occasionally go to a journal to read broad (e.g. I’m in immunology but I’ll read synthetic biology or physics)

Megan L. Hochstrasser on Oct 09, 2024

Absolutely — we’re in Google/Google Scholar and working to get indexed in additional places like OpenAlex, Dimensions, Europe PMC, etc.

5A64 (Pseudonym) on Feb 02, 2023

Selection

team as needed. Wherever possible, we want to facilitate sharing rather than restrict or delay it. Anyone who makes a PubPub account can add comments on our pubs. We will also capture as much of the Twitter conversation as possible by embedding a feed collectin

Combined with open-access, this means permissionless contribution to science! It could come from anyone (no degree requirement), perhaps even those that are outside of science. If they understand the science, then it could provide a new creative insights as they are likely to have a ‘beginner mind’ or see parallels with their own discipline.

5A64 (Pseudonym) on Feb 02, 2023

I wonder if this will also mean a new career path in academia - a ‘consultant’ for those that enjoy scientific discussion but not lab experiments, and are hired to assess the integrity of the science, contribute to new directions etc.

Reviewers of course exist. But it’s not exactly clear what their contribution was (somewhat now better with transparent peer review). Since Arcadia is crediting public feedback, this also means a potential metric to assess the value that a ‘consultant’ provides.

Megan L. Hochstrasser on Oct 09, 2024

Absolutely — and in future versions of PubPub, comments can potentially have their own DOIs to help facilitate this

5A64 (Pseudonym) on Feb 06, 2023

Selection

hat we can contribute to the collective advancement of knowledge and earn their valuable feedback.Standing on the shoulders of the open science communityWhile we have the unique opportunity to run agile p

Some long-term bigger questions:

If the future of publishing is no longer PDFs and journals (akin to New York Times —> Substack) how do you curate your own library if the native config of PubPub is an online document? Will PubPub allow you to have a private copy? Some may want to have a copy in their private library so that they can add highlights/comments not for the public. Although you can download the PDF copy of the PubPub, it doesn’t make sense to revert back to the PDF and EndNote format.
Will all this info be indexed? Can it be easily retrieved by new AI tools like Elicit.org that can draw insights? This could even help extract insights within Arcadia as the institute scales with more projects.

5A64 (Pseudonym) on Feb 08, 2023

Selection

pen and reproducible efforts throughout every stage of the research cycle.Share your thoughts!Watch a video tutorial on making a PubPub account and commenting. Please feel free to add line-by-line comments anywhere wi

As I read the pub alongside the in-line comments, I assumed that I had covered all the comments so I didn’t bother looking at the bottom ‘Comments’ section. Didn’t realise there were also general comments in this section that applied to the pub as a whole. But these are buried under the in-line comments.
If many others also don’t realise this, it may be worthwhile improving the vid to mention how to easily view general (not in-line) comments via the filter option (disable ‘Show anchored comments’).

Megan L. Hochstrasser on Oct 09, 2024

We’d love to have a way to show # of comments at the top of a pub so readers can immediately see that there’s discussion (or consider starting one) — the upcoming new version of PubPub should accommodate this!

Contributors (A-Z)

Purpose

Share your thoughts!

Introduction

The platform

Initial framework

Where can our research be found?

How can the scientific community use our work?

Who will contribute to our pubs?

Site organization

Project narratives

Pub structure and style

Citation style

Authorship, credit, and responsibility

Community feedback as a means of peer review

Iterating on public review and learning along the way

Our own contribution to public review

Standing on the shoulders of the open science community

References

Share your thoughts!

Pub details

Table of contents

Contributors (A-Z)

Purpose

Share your thoughts!

Introduction

The platform

Initial framework

Where can our research be found?

How can the scientific community use our work?

What do we intend to share?

Who will contribute to our pubs?

When will we share our work?

Site organization

Project narratives

Pub structure and style

Sharing protocols, data, and code

Citation style

Authorship, credit, and responsibility

Community feedback as a means of peer review

Iterating on public review and learning along the way

Our own contribution to public review

Standing on the shoulders of the open science community

References

Share your thoughts!

Pub details

Table of contents