Skip to main content
SearchLoginLogin or Signup

The experiment begins: Arcadia publishing 1.0

Building on the open-source platform PubPub, we’re sharing the first iteration of our publishing website. In addition to posting our first set of research pubs, we’re documenting our progress in developing this new system for sharing science and hope you’ll provide feedback.
Published onMay 31, 2022
The experiment begins: Arcadia publishing 1.0
·

Purpose

In thinking about how to share Arcadia’s research, we wanted to keep features of traditional publishing that have been honed over centuries, but improve upon what hasn’t quite adapted to the nature of modern science and technology. We have a unique opportunity to use our own research to develop mechanisms of sharing and quality control that can be more agile and adaptable. Our initial attempt is outlined here and we will continue to iterate upon it, always keeping the advancement of knowledge as our guiding principle when making decisions on what to try next.

This pub is intended to help you understand our thinking thus far, to provide a sense of what we’ve done and how the platform works, and to serve as a place to provide feedback on our strategy and the platform itself.

Share your thoughts!

Watch a video tutorial on making a PubPub account and commenting. Please feel free to add line-by-line comments anywhere within this text, provide overall feedback by commenting in the box at the bottom of the page, or use the URL for this page in a tweet about this work. Please make all feedback public so other readers can benefit from the discussion.

Introduction

We are reimagining scientific publishing — sharing our work early and often, maximizing utility and reusability, and improving our science on the basis of public feedback.

This is our first draft. We have ambitious goals and we’re committed to replicable long-term solutions, but we also know that “perfection is the enemy of good.” We’re using this platform to release findings now rather than hiding them until we’ve gotten everything exactly how we want it. Readers can think of the pubs on this platform as drafts that will evolve over time, shaped by public feedback. The same goes for the platform itself! We’re treating our publishing project like an experiment — we’re not sure where we will land, but we can only learn if we try. In this pub, we’re sharing our strategy and the reasoning behind some of our key decisions, highlighting features we’re excited about and areas for improvement. 

We don’t want to clutter our scientific pubs with comments about website functionality, aesthetics, or our model, but we invite you to share those thoughts here! All first drafts need feedback. What do you think of this site? Are our first pubs understandable? What important features are we missing? What are we doing right? What can we improve? Please don’t hesitate to share constructive commentary and ideas.

The platform

When starting a new experiment, it’s tempting to build every piece from scratch. We could try to build an ideal publishing mechanism where form seamlessly fits function. However, we are a research organization with the goal of advancing our science with the maximal impact on the rest of the scientific community. If we held back our science from the world so we could build a platform from scratch, we would be fundamentally obstructing our experiment from its main purpose.

Enter Knowledge Futures Group (KFG) and PubPub. While there are features we intend to expand and customize, PubPub meets our main requirements for functionality and, importantly, is run by a team that is deeply aligned with our open science mission. KFG is already developing functionality to make scientific research more mineable and its usage quantifiable. Their forward-thinking plans were not based exclusively on meeting today’s publishing needs, but preparing for a world in which there is demand for exactly what Arcadia wants: an author-driven, maximally reusable, and community feedback-strengthened research-sharing platform. We hope that others can try our approach. Ultimately, the true measure of our success will be other scientists adopting, and adapting, publication paradigms that better serve science.

Initial framework

This section is a summary of our publishing strategy and what we’ve decided to try initially. Subsequent sections expand upon some of the more complex points. Check out the table of contents (click “CONTENTS” on the right side of the screen!) to get an overall sense of how this pub is organized, and check out whichever topics are most interesting to you.

Where can our research be found?

  • Raw data will go in FAIR (Findable, Accessible, Interoperable, Reusable) repositories.

  • Detailed protocols will be deposited in protocols.io.

  • Contextual information for the raw data will live on the PubPub platform.

How can the scientific community use our work?

  • Each pub is citable and will ultimately have a permanent identifier (like a digital object identifier, or DOI, through DataCite).

  • Each pub will follow rules for Google Scholar indexing and be discoverable.

  • Each pub will follow whatever structure and include whatever media can best convey the desired information to the intended audience. This may mean including executable code blocks, interactive content, and other resources embedded within pubs.

What do we intend to share?

  • The nature of each pub will depend largely on the kind of data being reported and be geared towards facilitating use/reuse. 

  • We will include information about planned experiments/approaches, informative failures, potential uses by others, explicit questions soliciting feedback, and other components not typically included in scientific literature.

  • An integral part of every research product will be public feedback that will be visible alongside the original pub. We will use this to iterate and improve on the work.

  • We will provide context through “project narratives,” running logs that link individual pubs and keep the big picture in focus.

  • While versions of the work will have permanent identifiers so they can be cited, work is not static — evaluation will happen after the pub is publicly released. The products will be improved/built upon over time, incorporating feedback throughout.

Who will contribute to our pubs?

  • Each person who contributed to different aspects of the science will be listed as a contributor — including those commenters whose critical feedback shaped the work.

  • Each contributor’s role(s) will be stated explicitly and in a standardized format, making the nature of their involvement clear.

  • A subset of contributors responsible for the work will be “byline” contributors. All contributors will be listed in the metadata for the DOI, but only byline contributors will technically be listed as “authors,” and have their names used in citations.

  • Arcadia’s byline contributors will be empowered to release their research with the support of our publishing team as needed. Wherever possible, we want to facilitate sharing rather than restrict or delay it. Anyone who makes a PubPub account can add comments on our pubs. We will also capture as much of the Twitter conversation as possible by embedding a feed collecting mentions of each pub.

When will we share our work?

  • We will post publicly at multiple stages of a project. Pubs are modular and we will share them when we’re ready for community feedback or think we have information that could be useful to others rather than stuffing more and more information into a single publication [1].

  • We will improve pubs over time as we receive feedback and do more work. For transparency, each version of the pub will remain available.

  • There will be synthesis works posted as well, which integrates modular data already released. This will be accomplished partially through project narratives, which track the running progress of a project as whole by stringing together individual pubs. In cases where we want to elaborate, we will share separate review/perspective pubs.

Site organization

As we create more content, the structure will likely shift and expand. For now, there are three layers to this site:

  1. The landing page provides background on Arcadia and our goals, and, most importantly, will tell you what’s new. [View landing page]

  2. Project narratives are running logs that maintain a high-level view of our goals and connect modular pubs together into a more cohesive story. [View example]

  3. Pubs are detailed research products. These are akin to traditional papers, but tend to have a narrower focus and looser structure. This is where we will collect community feedback through comments. [View example]

Project narratives

Our research is generally broken up into projects, concerted efforts to answer specific biological questions, solve particular problems with technology, or sometimes simply to explore a curiosity.

These projects are described on “project narratives.” These are intended to serve as a running narratives of progress, almost like a journal or collection of field notes. The project narrative will reflect the current status of the research, where we’re heading next, and, sometimes, how you can influence our direction. Project narratives are always evolving, and will not have listed authors or DOIs.

We think they’ll be particularly important because we’re going to publish much smaller pieces of research than what is traditionally shared in journal articles. With so many modular publications, or “pubs,” it could become tough to follow the thread. Project narratives will string the individual pubs together into a coherent narrative, summarizing emergent takeaways and laying out planned next steps. Though pubs will work as individual pieces, with all essential context included, the project narratives provide space for more in-depth background.

We’re interested in sharing these narratives, especially the failures and dead-ends, because although these stories are fascinating and informative, they are rarely revealed in published papers. A chronological presentation is more reflective of how scientific work is actually done, and could be useful for students and trainees to study as they set expectations for their own research. 

Pub structure and style

Pubs are our standalone “units” of research output.

In structuring individual pubs, we’ve tried to strike a balance between improving usability and what scientists expect to see in a research publication. Together, the pub’s title, subtitle/description, and the “Purpose” section provide a high-level summary of key takeaways, features of the pub, and quick links to connected content like the related project narrative, data, or protocols. Section headings will be similar for each “pub type” (e.g. Resource, Observation, Negative Data) so regular readers can get used to their organization, but we will always prioritize making an individual pub as clear as possible. Some sections are similar to those in traditional papers, like “The results” and “The method,” while others, like “What’s next?” feel more like headings you’d see in a news article or blog post.

This informal style is intentional. We’ll use scientific terms as necessary to be accurate and to effectively communicate to other experts, but generally aim to strike a more conversational tone to enhance readability. Long sentences and jargon make papers difficult to parse, and we’re hopeful that we can improve the reading experience for all audiences, including technical experts.

The table of contents makes pubs navigable, providing a bird’s-eye view of what readers will find in the piece and quick links to jump to the desired information. Right now, users can open the table of contents by clicking the “CONTENTS” button on the right side of each pub.

This is a feature that we’d like to improve upon in subsequent iterations of the site. We’ve made the table of contents “sticky,” so it’s always available no matter where within the pub a reader is looking, but we suspect that having it pre-expanded will be much better, especially for those new to the platform. We welcome other suggestions on how to navigate our pubs, especially any examples of sites that do this well.

Non-text elements like figures, code, or other media will appear as close to where they’re discussed as possible to minimize scrolling. Whenever possible, we will directly embed content so readers can consider it in context and don’t have to juggle multiple open windows. Every reference to a figure also contains a link that allows jumping directly to that figure.

This is another area where we’re already imagining future experiments in readability and utility. What if figures always followed you when reading the section in which they’re described? What if you could view figures and text in two separate columns? What if individual elements within figures were clickable and could change to display data differently or show more/less information? We’d again appreciate thoughts on features you’d appreciate, lessons learned from previous attempts at improving figures and other media elements, and examples of people or sites making progress in this area.

Finally, pubs are not static — we will revise them iteratively based on feedback. We’ve decided to start with a three-stage tag system to indicate where we stand in the iteration cycle. At the top of each pub, you’ll see either “Feedback requested” when we’ve just posted it and are awaiting constructive criticism, “Revised after community feedback” when we’ve updated the pub based on comments, or “Revised, no longer actively updating” once the feedback tapers off and any additional comments are better addressed through a new pub.

You can see all release versions of a pub by clicking the rewind/clock icon in the header. When changes more substantial than adjusting formatting or correcting typos are posted, we will include a release note that describes the nature of the changes.

Sharing protocols, data, and code

To make our work as useful to others as possible, we are planning to share as much practical, usable material as we can. This means sharing full protocols, raw data, executable code, and more. We’ll strive to always keep our audience and goals in mind when sharing a pub — if we want our readers to be able to replicate our work or apply a method in their own research, we need to include sufficient detail, especially for those whose expertise differs from our own.

When adapting or developing new methods, we aim to post detailed, step-by-step protocols on protocols.io. This pub contains an example. We’ve embedded the protocol directly so it’s easy for readers to skim and determine if it’s relevant, and they can of course also view and use it in a separate window as well.

Our data will be deposited in stable repositories that are findable, accessible, interoperable, and reusable (FAIR) [2]. The goal is for data to meet standards that will maximally facilitate interaction by both humans and machines, further increasing the reach and impact of the open data. In some cases, we may not have yet identified an appropriate repository, or deposition may be in progress. To avoid delays in sharing, we may first share our datasets directly and update links once repository deposition is complete.

We want to share code in ways that make it straightforward to run and make outputs reproducible. This goal is more achievable than ever as cloud computing becomes widespread and browsers support embeddable, in-line programming environments. For example, we are taking advantage of tools like Google Colab notebooks or Github gists, which allow users to easily edit and run code on their own or view code directly within our pubs. We’d eventually like to find a reliable way to share executable code within pubs as well. This pub is a great example of this theory in action, and we appreciate any feedback on its utility.

Citation style

Our chosen citation style is minimal, dropping issue/page numbers and journal titles. While non-experts on a specific topic need to know whether experts have weighed in on scientific research, science is often inappropriately gauged using journal names as a proxy for scientific quality. This is typically based on their presumed level of selectivity or due to inappropriate application of journal-based metrics [3] to individual research articles, as is frequently done with journal impact factors. To reduce the dependence on poor proxies of scientific quality, we have removed journal names from our citations. For simplicity, we have also removed issue and page details, which are less relevant in a digital context, but retained DOI links to make our cited references easily findable. Readers can easily see a citation by hovering over the bracketed citation number (desktop) or tapping it (mobile), and can then click the URL to see the paper.

Other PubPub users can adopt this approach by selecting the “Arcadia Science” style in PubPub’s dropdown list of citation styles. Outside of PubPub, you can access this simplified citation style by downloading the Citation Style Language (CSL) file on our public Github. It will eventually be available in the official CSL repo, pending review, which will make it a style option in many citation management tools.

Authorship, credit, and responsibility

To facilitate transparency and collaboration, we are capturing the substance of individual contributions using a modified version of the contributor roles taxonomy (CRediT). The roles include areas like conceptualization, editing, investigation, methodology, supervision, and more. These roles are indicated next to each alphabetically ordered contributor at the bottom of our pubs. We are extending the taxonomy to include key players not typically included in author lists, such as public commenters who may play an instrumental role in shaping the direction of our work.

The subset of these contributors who are responsible for the content of a pub will have signed off on the released versions and will be “byline” contributors. Whether a contributor is considered byline is determined by the set of roles that they played. We are deciding (and plan to share in upcoming communications) which roles convert a contributor to byline by default, so the system can be applied to each new pub with minimal conflict or confusion. Byline contributors will technically be included as the “authors” in the metadata for the DOI provided through DataCite. These are the names that will appear in citations of our work.

Why make this distinction? What does it mean to be “responsible” for scientific research? When it comes to responsibility and accountability for pubs, we are aligned with the recommendations from the International Committee of Medical Journal Editors (ICMJE), in which byline contributors are accountable for their own work as well as aware/can vouch for the integrity of the contributions of other byline contributors. The byline contributors are similarly the first point of interaction for responding to public feedback and they may engage other contributors as appropriate.

Community feedback as a means of peer review

How will we and others be able to gauge the quality and rigor of our work? Quality control is typically attempted through pre-publication peer review, a process in which manuscripts are sent to journals and experts provide comments that must be addressed or rebutted before an editorial decision is made. While currently considered the gold standard, there’s room for improvement. We’re trying three approaches, outlined below.

First, we want to solicit feedback from a much broader range of people than the two or three included in typical journal-led peer review. If a large number of scientists provide feedback, their comments can be more modular, focusing on the specific area of the commenter’s expertise and requiring less time to contribute. The science of today is also highly interdisciplinary, and with community-level feedback, we hope to achieve scrutiny across a greater portion of a publication, from big picture to nitty gritty methods. For example, a cell biologist might comment on the microscopy, a statistician on the study design, and a bioinformatician on the data analysis, with organismal biologists providing appropriate context. Collectively, the feedback can be more relevant, efficient, and useful.

Next, we want to make all feedback public. While we may not engage with every comment, the contributions of those commenters will still be visible to all and it will be in the interest of our scientists to attend to critical questions. All readers can then learn from and consider the opinions shared, especially those by experts outside of their domain. Beyond PubPub comments, we will also try to capture public commentary on our work from social media.

Finally, this open evaluation will happen over time rather than at a single point, better matching the reality that science is a living, breathing body of work. A rigorous finding is one that stands the test of time, is built upon, and is challenged by orthogonal experimentation. We will solicit feedback, address it, and solicit more feedback. Eventually, we will no longer actively check or update pubs, but comments will always remain open.

Iterating on public review and learning along the way

The scientific literature is growing exponentially [4] and mechanisms for quality control and improvement must adapt with it. Crafting sustainable and scalable mechanisms of peer review may be the trickiest piece of the publishing puzzle, and we’ll need to test multiple solutions. Here are two major issues we anticipate and initial ideas for how to deal with them:

Will the scientific community participate? Will experts provide sufficient comments to yield confidence in our work without traditional editorial motivators? The heavily managed process in which journal editors with stature directly request reviews cannot be replaced by shouting new science into the void and hoping people will notice. We, too, will need to actively solicit feedback. Now that our initial content is available, we’re beginning to explore ideas to cultivate engagement with our work, and hope to relay our learnings in future pubs. We welcome ideas too!

Without the pre-existing audience of a journal, how will we attract interest at all? We hope that, in addition to actively soliciting reviewers, doing the kind of work that matters to people will help us build an invested community. We will have to produce work that can easily be used and improved by others. We must demonstrate true impact over time, and welcome feedback from all who might benefit from our contributions. We must also be responsive to the community’s bigger-picture direction, being willing to re-chart our scientific path to better serve the greater need. 

Our own contribution to public review

Why should anyone donate their time to give public feedback to a company? We aim to be good community members in return.

First, we aspire to generate the kind of data, insights, and technologies that can dramatically advance what scientists in the community are able to do in their own work. Feedback will just make the quality of our contributions and responsiveness to the needs of the community that much better. 

Second, we are asking our scientists to actively participate in commenting on preprints and other public research products at a level commensurate with their own research output. Arcadia will appear as a reviewer group on Sciety and our comments on preprints will be displayed there and on bioRxiv. We hope our comments help the authors of those products improve their own work such that we can contribute to the collective advancement of knowledge and earn their valuable feedback.

Standing on the shoulders of the open science community

While we have the unique opportunity to run agile publishing experiments through a growing research organization, this first version of this model was inspired by the work of many prior and forthcoming open science efforts. These include the preprint servers arXiv, biorRxiv, OSFpreprints, and so many others. Our efforts are also inspired by the F1000 research platform for post-publication review, the “publish, then review” efforts at eLife based on a publish-review-curate model, crowd review efforts at ASAPbio.org, community preprint review efforts such as PreReview and Peer Community In, and activities of many other organizations represented at Incentivizing Collaborative Open Research (ICOR), who are innovating on open and reproducible efforts throughout every stage of the research cycle.


Share your thoughts!

Watch a video tutorial on making a PubPub account and commenting. Please feel free to add line-by-line comments anywhere within this text, provide overall feedback by commenting in the box at the bottom of the page, or use the URL for this page in a tweet about this work. Please make all feedback public so other readers can benefit from the discussion.


  • Acknowledgements

    • We are grateful to everyone who has contributed to this first iteration of our publishing site. Thank you to Mert Celebi for weighing in on countless technical questions and developing solutions to key challenges. Thank you to Seemay Chou for all your thoughtful insights and for making the time to help pull everything together in the past few weeks. Thank you to Gabe Stein and everyone on the PubPub team for being great thought partners and making so many site adjustments for us. Thank you to the whole And—Now team for creating and implementing fantastic designs incredibly quickly. Finally, thanks to all the scientists at Arcadia for your amazing ideas and patience over the past few months! Special shout-out to our publishing pioneers, Peter Thuy-Boun and Jase Gehring, for being brave enough to be the first of our scientists to release their work in this new way.

  • Contributors
    (A–Z)

    • Prachee Avasthi

      • Conceptualization, Editing, Supervision, Writing

    • Megan L. Hochstrasser

      • Conceptualization, Editing, Writing

  • Competing interests

    • P.A. is the president of ASAPbio and serves on the board of directors at eLife, groups that are also experimenting with scientific publishing.

Comments
52
5A64 (Pseudonym):

As I read the pub alongside the in-line comments, I assumed that I had covered all the comments so I didn’t bother looking at the bottom ‘Comments’ section. Didn’t realise there were also general comments in this section that applied to the pub as a whole. But these are buried under the in-line comments.
If many others also don’t realise this, it may be worthwhile improving the vid to mention how to easily view general (not in-line) comments via the filter option (disable ‘Show anchored comments’).

5A64 (Pseudonym):

Some long-term bigger questions:

  1. If the future of publishing is no longer PDFs and journals (akin to New York Times —> Substack) how do you curate your own library if the native config of PubPub is an online document? Will PubPub allow you to have a private copy? Some may want to have a copy in their private library so that they can add highlights/comments not for the public. Although you can download the PDF copy of the PubPub, it doesn’t make sense to revert back to the PDF and EndNote format.

  2. Will all this info be indexed? Can it be easily retrieved by new AI tools like Elicit.org that can draw insights? This could even help extract insights within Arcadia as the institute scales with more projects.

5A64 (Pseudonym):

Combined with open-access, this means permissionless contribution to science! It could come from anyone (no degree requirement), perhaps even those that are outside of science. If they understand the science, then it could provide a new creative insights as they are likely to have a ‘beginner mind’ or see parallels with their own discipline.

5A64 (Pseudonym):

I wonder if this will also mean a new career path in academia - a ‘consultant’ for those that enjoy scientific discussion but not lab experiments, and are hired to assess the integrity of the science, contribute to new directions etc.

Reviewers of course exist. But it’s not exactly clear what their contribution was (somewhat now better with transparent peer review). Since Arcadia is crediting public feedback, this also means a potential metric to assess the value that a ‘consultant’ provides.

5A64 (Pseudonym):

In regards to a reading audience, I don’t represent everyone, but I don’t go to a journal for papers of interest. I use pubcrawler to give me a weekly feed of articles that match my criterias of interest. If the Pubs are in PubMed (just like recently now with pre-prints), this shouldn’t be a major issue.

I only occasionally go to a journal to read broad (e.g. I’m in immunology but I’ll read synthetic biology or physics)

5A64 (Pseudonym):

(Potential bug: Clicking on a comment that is linked to a text with hyperlink, seems to go straight to the hyperlink, not the comment. On Chrome MacOS)

5A64 (Pseudonym):

Since this is online (not PDF), it could benefit by adopting something similar to Microsoft Word’s track changes or minimally an automatic highlighting feature to track areas where changes occured. (Similar to sending back a manuscript for revision so reviewers can easily identify the changes)

5A64 (Pseudonym):

+1. This would require uploading source data (not just the figure). Would be esp. helpful for barplots as many plots will not show individual biological replicates - which can reveal a lot of info about the spread of the data that would otherwise be hidden if you just presented the mean value.

Easy to implement for bar plots, may be more difficult for non-quantitative data e.g. flow cytometry plots (in immunology, some will choose to use contour plots over dot plots to obscure the fact that there may only be <10 events in a flow cytometry acquisition)

5A64 (Pseudonym):

+1 on this idea, it’ll make reading a lot of more easier compared to scrolling up and down OR jumping up/down using hyperlinks. For reading papers via PDF on a computer, I would open 2 copies - 1 for the text and then the other for the figure. Even more necessary when journals decide to place the figure legend on a separate page to the figure (e.g. Cell).

PDF of journals have their own fixed structure. The advantage of PubPub should be the flexibility i.e. you can choose two column if you prefer, can change how references are displayed (numbered vs last author / year format etc.)

Jessica Carstens-Kass:

Hi! For citing your pubs, I see you provide the bibtex code which is great; it would be extra nice to be able to export a .ris file for citation managers like EndNote!

Megan L. Hochstrasser:

Added to my list! Thanks so much for the feedback!

Nate Jacobs:

I really love this. Its an aspiration not a decision, but I hope that we can all move peer review towards this new type of workflow.

one thought that I keep coming back to after taking this first step away from deriving “truth” from single article, is does this open up a new opportunity for more robust mechanisms for curation and drawing attention to ground truth?

one example would be content state changes if findings are replicated. another example could be meta-analyses that then establish higher levels of confidence (or lack thereof) in a finding.

for me this is a really important first step towards:
1) decoupling peer review / journal brand as proxy for “truth” or “consensus”
2) deriving consensus from subsequent research outputs (vs commentary and endorsements), ie data driven peer review with replications, contradictions, and extensions.

Nate Jacobs:

:) sweet!

Nate Jacobs:

Similar arguments can be made for the structure of the article itself. There’s a lot of really cool meta-analyses etc that could be done if papers were structured a bit better. For me this is more important than for underlying data sets, which may be hard to automate machine use of given the high degree of variability between data sets.

Nate Jacobs:

<3. I think this can be the new methods section. just don’t even try to summarize it, post all the protocols and then reference in context.

Nate Jacobs:

what about retraction? or, in the positive direction, replication? Lots of really cool ideas to experiment with here.

I think I saw comment about this somewhere else, but visibility and organization of comments / feedback at different states (in review, final, etc) is worth thinking about. No right answers but can have an effect on how users engage with content.

Nate Jacobs:

:)

all open science endeavors should be empowering researchers to lean into their curiosity again.

Nate Jacobs:

are pubs different than modular pubs?

Nate Jacobs:

I love this approach and have thought about this a lot (mostly in circles). Curious to see how this goes and happy to share more detailed thoughts if helpful.

For me a few questions I always had with this is:
1) what is the best governance structure? Should this be org / journal level - like special features that editors are updating contextual content about? Or a content type that anyone can create?
2) If coming from authors / users, how does this compete with (or not) other workflows like preprints, blogs, etc?
3) do you treat this as a scholarly object or as a stepping stone / facilitator between micropubs (modular pubs) and longer form content / meta-analyses?

Nate Jacobs:

This is a great testing ground for new workflows and finding what will work for the community. Really excited to see how things shake out and what patterns and trends emerge in user behavior, and what that could mean for workflow refinements.

I have found product design tension between building up workflow specifically designed for small modular atomized content (micropubs) and exactly how and when this relates to longer form content. I don’t have any strong answers or intuitions on this yet, but I often think about what twitter would be like if it allowed blog posts. would that be a good thing or bad thing? how would that change user behavior and product focus?

One observation to note is that if you don’t implement specific creative constraints like length and types of content, user behavior will probably drift towards longer and more entropy. which may be a good thing! or it could make it difficult for certain types of content to have the correct incentive and cultural structure. Could twitter have ever evolved from an email or blog platform, or would it only have emerged and gotten traction if it only allowed certain type and format of content?

One more observation is I think it’s important to know how many products and workflows you are trying to support. By not choosing, sometimes that means you are choosing to support a wide variety of products and workflows (ie all the things the users choose to do on the platform). For example, it’s possible this approach could lead to competing user needs for 1) modular content, 2) preprint / blog, 3) more formal publishing and visibility needs for career advancement.

With my work on micropubs, I made a deliberate decision to NOT support longer form content, and am working towards integrations with external platforms that do that better (preprints, journals). I haven’t gotten very far with this, so maybe wait until I have actual data / experience before taking this advice, but it’s something I’ve found useful to at least think through.

I could be completely wrong about all of this so will be curious to see what actually happens in this case!!

I have a strong bias / idiosyncrasy towards constrained design serving narrowly focused workflows, which works well sometimes and other times is a terrible decision haha.

Perhaps an intermediate approach, informed by an initial round of loose experimentation, could be to further constrain (or loosen) format and workflow depending on what authors are doing and finding most helpful!

?
Steven Massa:

A small thing: when I click on a comment which is tied to a hyperlink in the text, it takes me to the linked page instead of opening the comment. I’m encountering this issue with Jacob Bumgarner’s comment in the Citation Style section. If I try to add a comment near there, it also automatically directs me to the linked page. My browser is Chrome Version 103.0.5060.11.

In relation to this, should comments about technical issues with the webpage be separated somehow from comments about the content of the pub itself?

Megan L. Hochstrasser:

Thank you for pointing this out! We’ve relayed this bug to the PubPub team and it’s in their queue. Please do point out technical issues on this pub—I’d like to be aware of them myself and can make sure they get attention.

Bruce R. Caron:

An invitation to take a look at the Open Scientist Handbook, here on PubPub: https://openscientist.pubpub.org/

Bruce R. Caron:

What about doing overlay journals?

?
Prachee Avasthi:

Overlay journals are fantastic and a welcome addition to the post-publication review and curation ecosystem. As our primary focus is as a research organization rather than a review entity, we aren’t formally organizing review or rendering decisions the way a journal might but rather want to make sure that the comments for work we read/review are captured publicly so that readers can benefit from a richer set of available information.

Bruce R. Caron:

This type of broad, but focused public feedback inside the article is of much greater use than any current closed review system.

?
Jacob Bumgarner:

A general website aesthetic comment - the table of contents dropdown covers the text.

I’m viewing the publications on Safari 15.4.

Megan L. Hochstrasser:

Oh, thank you! I actually don’t see it that way on Safari on my end, so this is great to know. I’m definitely seeing issues with the ToC in general across browsers too. We’ll work on this!

?
Jacob Bumgarner:

This is a general comment that applies to all of the links in this pub, but it would be nice if links automatically opened in new tabs so that the readers aren’t directed away from the publication page.

Megan L. Hochstrasser:

Agreed! We can’t control this right now, but it’s on our list of requests.

?
Jacob Bumgarner:

Will there be stated guidelines that indicate the level of commenter contribution needed to be listed as a contributor?

E.g.,
- Those that proof the publications for grammatical issues won’t be included as contributors
- Those that provide ideas for new critical experiments/alternative analytical approaches will be listed as contributors.

Megan L. Hochstrasser:

This is a great question. We’re developing a list of definitions for all contributor roles, including “Critical Feedback,” which can apply to public commenters or colleagues who weigh in early in the research process. Since we’re building on the casrai.org “CRediT” system but diverging in a few key ways, we plan to share this list once it’s built out. It could potentially be a standalone pub to support versioning, since we may make changes over time, otherwise we’ll add it as a page.

Based on your feedback here, I think eventually we may want to add a page just intended for would-be commenters that explains how to comment, how comments may be used, in what circumstances someone may be credited as a contributor, etc.

?
Jacob Bumgarner:

A general proofing comment - it seems like this question was meant to be addressed/filled out in the drafting stage.

Megan L. Hochstrasser:

Good catch! Not sure how that made it through.

?
Matthew Akamatsu:

It would be fun for there to be space for community contributions on the project page. In an open-source model, other researchers could help address a given research question, provide hypotheses, or help out with a particular analysis. And then get credit if the contribution becomes useful.

Megan L. Hochstrasser:

Yes! We definitely intend to add community members as “contributors“ to pubs if they provide critical feedback. I think it’ll be important to show how we’ve incorporated comments into later versions of the pubs to 1) show people that we value their feedback, 2) be fair about how people are credited/rewarded, and 3) demonstrate that our post-publication review approach is working.

Major contributions can be highlighted on the project narratives too, via written shout-outs. We’ll see how this evolves if we eventually end up designing those pages to function closer to pubs (which I suspect we may end up doing later on).

?
Matthew Akamatsu:

I am really curious about whether pubs will naturally sort into a specific set of “pub types.” Like, will each novel finding be most easily communicated as an individual Observation, and later on an Integration cites a few Observations to draw a new conclusion? Or will the pub types become really heterogenous? If the former, then I think it will be really powerful to be able to query based on pub type; e.g. find Observations that are referenced by a particular Integration. Example: https://roamjs.com/extensions/discourse-graph/synthesis-query

Megan L. Hochstrasser:

I’m curious about this too. So far, I’ve outlined a set of about a dozen pub types that I think can work for most use cases and I’m trying to have us stick to these (I’ll share the list in a future version of this pub, or perhaps as a page that explains key features of each type). We’ll see how straightforward it is to use this standard set as we try to categorize future findings.

I like that feature of Roam! I’ll add it to my list of things to think about as we develop our framework and think about how best to connect and query for individual pubs. PubPub has a “Connections” feature that lets us connect pubs to each other and to external sites (e.g. deposited data) that may be the right structure for this — I’ll have to look into how searchable it is.

?
Matthew Akamatsu:

Just like in using hypothes.is, it might become overwhelming to readers to parse which comments they find useful. I wonder if it will become necessary to filter or prioritize comments? Having upvotes might also help to gauge how much engagement a particular article is getting, and how pressing a comment might be.

I also like the idea of subjective filtering, where you can rate comments as well as other raters. If I highly rate e.g. Prachee’s opinion, then comments that she rates highly will appear first for me. This “subjective review” is implemented at https://braid.news/ .

?
Jacob Bumgarner:

+1 to comment rating.

Along this vein of thought, will there be comment moderation?

+ 1 more...
?
Matthew Akamatsu:

It would be nice to be able to query based on these roles. For example, I could imagine having a contributor page that describes the nature of my contributions to various projects. It should be possible to start with an autofill of my contributions; i.e. “conceptualization role in pubs x and y for project z.” A systematic treatment like this might make it more likely for these taxonomic roles to be considered, both for contributors and evaluators.

Megan L. Hochstrasser:

This is a cool idea! For reasons I won’t get into, the “Contributors” section you see right now is added manually, with the same information separately added to the metadata behind the scenes. This should change in the future, and once it does, you should be able to click on a contributor’s name to see their profile, which lists all pubs to which they’ve contributed. I like the idea of being able to additionally sort by contribution type!

?
Leron Perez:

this link is broken

Megan L. Hochstrasser:

Thank you! It will be fixed in the next release. Here is the intended link: https://research.arcadiascience.com/pub/method-mass-spec-proteomics-transcriptomics

?
Leron Perez:

will the code be documented or just shared?

Megan L. Hochstrasser:

As much as possible, code will be documented. We want to balance reproducibility and utility with how much time it takes our scientists to add documentation, so the extent of documentation may differ from case to case.

Megan L. Hochstrasser:

Wow! So cool!

+ 1 more...
?
Leron Perez:

what if the reader could chose which figures to simultaneously view? what if they could view more than one?

Megan L. Hochstrasser:

Love these ideas! 💡

?
Leron Perez:

Will this introduce a tendency to editorialize and revise the biology to fit a story even when it may not?

Narrative should live separately from data and methods (in pubs and projects)

will dependencies between projects be tracked? ie one project motivated experiments that were inconclusive in the former context, but that led to the initiation of a new project. Will that new project cite the old project somehow. Is the experiment part of the new project or the old project?

Perhaps projects and pubs could benefit from a model like git: ie projects can be revised and changed in time but forks (citations) are specific to an particular revision

Megan L. Hochstrasser:

Partially addressed this in a previous reply, but will summarize here! I agree that we may end up wanting some sort of versioning/forking with project narratives in the future.

Right now, we’re hoping to avoid editing the intro/goals substantially and to verbally describe shifts in the project’s direction within the “Progress” section. If we do end up drastically changing direction/goals, we’ll likely create a new project narrative page or otherwise attempt to clearly explain and document the shift.

We can easily hyperlink between projects, but it could also be cool to eventually develop a way to embed projects like we do with pubs or add a higher-level page type that tracks links between projects themselves as they evolve, inspire new projects, etc.

Thanks for all this food for thought!

+ 1 more...
?
Leron Perez:

in terms of measuring scientific progress, it seems like projects are a very important piece to track and that they should be considered together with pubs.

if projects remain unquantified, while pubs are counted and cited, I am concerned that incentives may continue supporting maximizing/focusing on pubs rather than projects.

The best solution might not be including a DOI, but I am concerned that in the current system, projects will remain in the periphery of scientific documentation and communication because they are not associated with measures of progress.

A last thought: perhaps including a DOI for projects could be accompanied with norms that the project DOI is cited for motivations/context building, while specific results, numbers, or methods could be cited from the individual pubs.

Megan L. Hochstrasser:

Good points, and we’ll keep these considerations in mind. Given that project narratives may change substantially, we’ve chosen not to use DOIs for now, but they will be citable via their URL. I do think some sort of versioning could be useful in the future, but I think we’ll have to see what happens to these over time and decide on a strategy based on the real examples that come up.

+ 2 more...
Austin H. Patton:

I love this “open arms” approach - this is a fantastic incentive for encouraging involvement from the public!

Austin H. Patton:

This is great - directly gets at the frustation of bouncing around from text and figure in traditional journal publications! To improve legibility while “following” throughout the section, figures could expand/contract based on reader selection? That way you can always get to the figure you’re looking for, but it’s impact on reading the text is minimal?

Megan L. Hochstrasser:

Great suggestion!

?
Leron Perez:

will major revisions have a new DOI? Or will there be a way for citing publications to refer to a specific snapshot of the publication?

what happens if there is a revision that contradicts or significantly changes the originally interpretation of the pub?

Megan L. Hochstrasser:

Great questions! We are currently planning to keep a single DOI for each overall pub. For all revisions, users can cite specific versions by including the URL to the specific release. Note that PubPub saves each version (aka “release”) and provides a unique URL for each.

Regarding contradictory or otherwise major changes to the interpretation, we definitely do not want to hide the original data/interpretation, nor do we want people to stumble upon it and miss the critical update. Thus, at a minimum, we will add a hard-to-miss note to the top of the pub and make edits within, and may add an entirely new pub with its own DOI to relay the new data, discuss how the interpretation changed, why this has happened, etc. Any major (non-contradictory) additions to our data or understanding will be added as new pubs.

So far, we’ve learned a ton each time we’ve encountered a new scenario and ended up rethinking our plans, but this is our initial outlook.

+ 2 more...
Austin H. Patton:

I like this idea of having it pre-expanded. It also seems like if could be useful if in this table of contents you could directly link to other related pubs under the same umbrella. For instance in the table of contents for the project page, including links to subsections of not only that document, but also linking to the “downstream” pubs - data, methods/protocol, results, etc. Could help to more seamlessly integrate the different pubs?

Megan L. Hochstrasser:

I like this idea! When you’re viewing a pub, you can currently click the project title tag at the top of the header to see a dropdown of all the pubs included within that project. This functionality could be useful on the project narratives themselves as well. As the project narratives grow, this sort of organization/navigability will become more important.

+ 2 more...
Megan L. Hochstrasser:

Noted, thank you! Replicated on my end as well, we’ll address this.

?
Leron Perez:

minor: the banner at the top of the page with the title does not appear to be adjusting to my window size on Firefox

+ 1 more...
Austin H. Patton:

Will there be a sort of “version control” for citation? Given that these pubs are continually being revised, it seems that it would be important that when citing these works, the citation should link to a specific version, since findings may change/be modified in later iterations.

Seems like the three-stage tag system could be used/integrated to facilitate this?

Megan L. Hochstrasser:

Thank you for these questions and ideas! PubPub supports versioning and lets users view all previous “releases” of a pub, each with its own URL and with the option to add a “release note” explaining what has changed. We are currently planning to keep a single DOI for each overall pub. For all revisions, users can cite specific versions by including the URL to the specific release.

For contradictory or otherwise major changes to the interpretation, we definitely do not want to hide the original data/interpretation, nor do we want people to stumble upon it and miss the critical update. Thus, at a minimum, we will add a hard-to-miss note to the top of the pub and make edits within, and may add an entirely new pub with its own DOI to relay the new data, discuss how the interpretation changed, why this has happened, etc. Any major (non-contradictory) additions to our data or understanding will be added as new pubs

Applying the tagging system here is an interesting idea as well. We could consider adding a “Major Revision” or “Critical Update” tag, or otherwise noting this somewhere.

+ 2 more...
Austin H. Patton:

I don’t know if there is any existing infrastructure for such a thing, but I wonder if there might be value in having not only an executable code block, but one that’s interactive and modifiable - modifications could be shared by the community.

I think this sort of interactivity could facilitate data exploration, and provide a framework for scientific engagement by the community that’s currently lacking in traditional scientific communication. But again, this is pretty “pie in the sky” since that infrastructure might not even exist yet!

?
Leron Perez:

Interactivity exists to varying degrees in Javascript (Observable), Python (Jupyter), and Julia (Pluto).

See also:

https://nextjournal.com/

https://explorabl.es/

http://worrydream.com/LadderOfAbstraction/

+ 4 more...
Matt Clancy:

I have a similarly structured pub pub website built on modular updatable articles with a synthesis overlay, and have had a lot of luck with an email newsletter that functions more or less like a changelog of the website. Subscribers receive emails for each new pub, and once a month I also send out an email bundling up all the updates I’ve made to the site. Each email has most of the new text in it, so people do not have to leave their inbox to read it, but I include a disclaimer that the underlying linked article is always updated, and that people should follow the link to see the latest version.

The tradeoff is people mostly engage with the email newsletter, rather than the main site. But it is a relatively frictionless way to tell people what’s new, and substack has built a very shareable product that attracts interested subscribers quite well.

Megan L. Hochstrasser:

Oh this is an interesting approach! We’re just setting up our subscription options now, and it never would have occurred to me to include the new text directly within the email. Currently using Mailchimp but might look into Substack!

Matt Clancy:

This is the key challenge and I look forward to seeing your ideas for incentivizing people to engage or creating new norms around it. In my field, it is normal to circulate drafts for comments among people in your network; if similar norms exist in your field, you might be able to co-opt them and simply encourage authors of new pubs to circulate to interested parties, noting that any feedback is welcome, especially on the platform.

Megan L. Hochstrasser:

Agreed! We have been trying this with our initial pubs, jury’s still out on how well it’s working since we’re still in the middle of the process. We’re also hoping to hire someone to think about this full-time!

?
Aditi Sengupta:

As an early career researcher, will my publication in this format count towards hiring, appointment, rank and tenure? That is my biggest concern. I am all for this format but I do not know what the thoughts of those sitting on the other side of the table deciding if my scholarship efforts were enough and/or meritorious when published on this platform. I do not have a solution at this point. Just thinkng through things. I am sure this is not the first time such a question has been brought up.

?
Prachee Avasthi:

Hi there. This is a great question and one we are running the experiment to answer. All evaluators are likely to make their own decisions on what “counts,” but it’s our feeling that the materials with the most transparency (public materials with public feedback) give evaluators the most information possible upon which to make their decisions. It is also our hope that by virtue of the existence of other avenues of research sharing that are attractive to scientists, evaluators might expand what they consider, as has happened in some circumstances with preprints. This shift won’t happen overnight, but needs to start somewhere, so we hope to be able to chart the path that others might feel comfortable trying once they see it in action.

?
Aditi Sengupta:

I was looking forward to a graphic to understand the process. There is a lot of information and “docking points” that talk about specific topics. I was getting lost on how each of those docking points are connected. A figure might help readers here.

Megan L. Hochstrasser:

Thanks for this feedback! We will consider making a graphic or figure.

?
julien colomb:

super interesting, have not read all of it yet, but it would really nice to discuss:
- extension of CREDIT: how to make it usable by all, should this go to the CRO ontology. Would you have some funding to revive CRO development?
- author metadata in pubpub, and its reuse: you may want to look/participate in the jams initiative: see https://github.com/jam-schema/jams/issues/7

As a general remark, the link to twitter and google scholar is not aligned with the open science objective, it should at least be also schema.org (also used by google), and fediverse (open source alternative to twitter) compatible. Maybe a link to Wikidata would be feasible ?

Megan L. Hochstrasser:

Thank you for all these ideas! Right now, we’re not planning to fund external efforts, but we will look into the resources you’ve pointed out.

Regarding your third point, as we test other models of sharing and engagement, we’re doing our best to balance pure open science principles with the need to interact with scientists where they are in the existing ecosystem. For example, we’d love for our work and discussion to be maximally replicable via open mechanisms, but we’re also cognizant that Twitter is a major hub for scientific discussion, and we don’t want to miss that. We’re definitely interested in capturing conversations wherever they are, and will need to think about how to track comments on different sites, especially as social media channels change over time.

Kat [email protected]:

It is valuable for each unit (Method, Data, etc…) to exist in its respective Context — this offers a massive amount of approachability to the concepts in the pub and allows them to stand alone, however the redundancy might be high effort for the author and annoying to navigate for a reader familiar to the field. I think there must be another strategy that enables the best of both worlds: approachability without redundancy.

?
Aditi Sengupta:

Had similar thoughts. When dealing with multiple units of research output (e.g. a new set of results from the same study), how would one add the information and how will it be connected to a previous unit of the same research? Does one leave out the big picture intro/background, methods, and instead provide a link to the older unit?

+ 2 more...
Kat [email protected]:

When I started reading I immediately felt the value of a narrative story in the plural first person. The mention of specific hurdles and confusions that you overcame as scientists adds soooo much implicit value to understanding the research story. This stuff is missing from traditionally published papers.

?
Seemay Chou:

@Kat - I totally agree. So much of science is hurdles/confusion - we are doing ourselves and the process a disservice if we write it out of the narrative. I also worry about how we can properly teach students about the process without this

Michael Hoffman:

The data I get when I click the “Cite” link uses a “journal article” entry type with “Arcadia Science” in the journal field. Personally, I have had endless issues dealing with bibliography management software and journal production systems with citing electronic journals that don’t use traditional volume and page/article number systems. So I wish you would either use some other entry type or add article numbers.

Gabriel Stein:

thanks for the note, Michael. Which bibtex type do you think should be used for content like this? It does feel like @misc kind of undersells it, but I agree that the journal article isn’t quite the right fit, either.

+ 2 more...
Michael Hoffman:

Why gists? Gists can be created instantly but are relatively anonymous and unstructured. I would think having a repo for each Pub would be a better match, and would be more easily discoverable through GitHub rather than mainly through the Pub itself.

?
Leron Perez:

+1

+ 1 more...