Skip to main content
SearchLoginLogin or Signup

The experiment continues: Arcadia publishing 2.0

Two years into our publishing experiment, we’ve learned a lot. We built internal processes that worked but inadvertently decreased scientists' agency and creativity. Now, we’re minimizing process in an effort to empower our scientists to share their work how they see fit.
Published onJun 13, 2024
The experiment continues: Arcadia publishing 2.0
·

Purpose

We’ve been developing our open publishing model for the past two years and recently took a step back to reflect on how things were going. We realized that we’d become far too process-intensive, accidentally adding friction and sapping our scientists of ownership. What we observed in practice was that scientists simply weren’t sharing their work as freely and creatively as we’d originally hoped. And ultimately, enabling scientists to act with agency was one of our main goals.

But the company has now established a solid foundation of understanding and culture around our open science mission. Therefore, we’re now poised to try a drastically different and more ambitious approach.

In this pub, we introduce the next evolutionary stage of our open science experiment: a fully scientist-led publishing system. We want our scientists to decide when and how they share their work, putting the onus on them to grapple with issues surrounding these decisions. Our publishing and commercialization teams will continue to be available for strategic advice or support as needed. All scientists will, of course, need to consider how their decisions contribute to Arcadia’s company goals, such as doing useful science, ensuring high-quality research, and enabling financial sustainability. But there are only two non-negotiable publishing rules. First, all byline contributors must sign off that the work is ready for release. Second, all data, code, or other material essential for reproducing the work must be available. With our scientists in the driver’s seat, we hope to share more quickly and creatively, paving the way to more radical experimentation around publishing.

We’re sharing this as an update for those following along with our publishing experiment or just stumbling upon it now. We hope other groups and individuals working to re-envision scientific publishing will find it useful in thinking about their own approaches. We’re happy to discuss more — feel free to leave a public comment to start the conversation!

Share your thoughts!

Watch a video tutorial on making a PubPub account and commenting. Feel free to add line-by-line comments anywhere within this text, provide overall feedback by commenting in the box at the bottom of the page, or use post about this work on social media. Please make all feedback public so other readers can benefit from the discussion.

Lessons from our first 50 pubs

Two years ago, we embarked on an ambitious experiment to reimagine scientific publishing. Using PubPub, a free and open-source platform, we began sharing our research openly, iteratively, and collaboratively to accelerate scientific discovery and maximize the impact of our work. We're proud of what we've achieved in this first phase. Fifty publications — each accompanied by the data, code, protocols, and other materials necessary to replicate their findings — are now openly available. Reaching this milestone was a collaborative effort, with scientists, software engineers, and our publishing team working together to build a new model of scientific publishing.

As with any experiment, we've learned a great deal along the way. Our initial framework, while a positive step away from the shortfalls we identified in “The experiment begins: Arcadia publishing 1.0,” [1] highlighted areas where we could improve. We encountered challenges in creating a streamlined and efficient internal publishing process that still maintained the high standards we set for our research. 

Our multifaceted process — which included review from contributors, in addition to internal scientific, legal, software, and editorial review — seemed crucial to upholding our high sharing standards (Figure 1). In hindsight, we recognize that this approach, while thorough and helpful, hampered our ability to quickly disseminate our work. While avoiding pre-publication review through journals, we’d recapitulated a form of it internally. Ultimately, we replicated a common gatekeeping mechanism in scholarly dissemination that slows down progress, reduces iterative feedback, and gives the illusion that scientific work is ever “done.” This is an unacceptable outcome. Like others experimenting in open science, we’d fallen into the trap of backsliding toward the system from which we tried to break free.

Further, our emphasis on collaboration, designed to promote team science and facilitate interdisciplinary work, sometimes made it difficult to balance collective effort with ownership over the pub. Having our publishing team lead engagement efforts, which we’d intended to relieve the burden of identifying and engaging interested readers, inadvertently limited individual scientists from taking a more proactive role in discussing their work with external audiences. Additionally, when our scientists did speak publicly about Arcadia’s research, they sometimes struggled to do so with clarity and excitement because of their confusion around what might stand in the way of future commercialization.

Figure 1

Our most extensive publishing process.

While we had lighter tracks for pubs and contributors with different needs, this had become our default approach. The complexity and number of hand-offs make coordinating a process like this time-intensive, even if the individual tasks are straightforward. Not included but formerly part of our default process were code review, figure development, IP check, and engagement strategy.

Arrows represent rounds of revision, where two arrows indicate that multiple rounds are typical.

We also realized that our standardized publishing process could benefit from increased flexibility. This one-size-fits-all approach didn’t fully accommodate the diverse needs and working styles of our scientists. It hindered efficiency and creativity for some, while stretching the publishing team too thin to provide timely and deeply thoughtful help to others.

Finally, relying heavily on our publishing team for shepherding all stages made the process operationally expensive. It’s important to us that other scientists external to Arcadia can apply any solutions we find, and our prior system would be difficult to replicate without multiple staff members dedicated to publishing. While this may seem acceptable for well-funded research groups, it’s more difficult to extend such a practice to individuals, companies, or leaner organizations that want to share their science. We want our model to be accessible to everyone.

In general, our scientists are biased towards being comprehensive and careful not to make mistakes. But this conscientiousness can sometimes backfire when their well-intended reliance on a manager, the publishing team, or others for added input diminishes their sense of ownership over a pub. Some of our more collaborative pubs have suffered when individual contributors only reviewed their own sections — they’ve lacked cohesion and we’ve had cases where we nearly missed obvious errors. While we’re okay with making mistakes (and would be delighted for readers to point them out to us!), we want to help our scientists continue to take pride in and responsibility for their work.

What's not going to change

The speed and collaborative nature of modern science demands a more open and agile approach to sharing research. We thus launched our publishing experiment hoping to maximize three key qualities that benefit our science as a company: speed, utility, and rigor.

As we transition to publishing version 2.0, these fundamental features will remain unchanged. We’re as committed as ever to:

  • Openly sharing a broad range of research outputs: We share most of our work. Not just exciting results and tools, but negative data (check out our icebox), pre-experimentation ideas, open questions to the community, and more. The norm for biotech companies is not to share early-stage research, and we’re proud to have released so many pubs already. We’ll continue releasing science we think might benefit other scientists or that we’d like feedback on.

  • Embracing community feedback and public review: The traditional model of closed peer review is too slow and doesn’t scale with the growth of modern science. Soliciting feedback from a broad range of experts and making those comments public to benefit readers, not just authors, leads to more rigorous research and scientific discussion. Our publications will continue to be living documents, openly evolving and incorporating insights from the wider scientific community.

  • Prioritizing team science and collaborative authorship: The interdisciplinary nature of science, especially our work at Arcadia, demands a collaborative approach. We’ll continue to champion team science, accelerating our work and ensuring that all contributors receive clear and appropriate credit for their diverse expertise and contributions.

  • Focusing on utility and impact: Publishing 1.0 mandated the release of all data, code, protocols, and other materials necessary to reproduce and build upon our findings. We’ll continue doing this, striving to make our work as accessible and usable as we can, maximizing the chance of making a meaningful difference for the broader community.

We’re doubling down on these core principles while providing our scientists with additional operational freedom to truly experiment with and push the boundaries of these ideals.

Our new approach

To address the shortcomings articulated above, we’ve developed a vision for the next iteration of our experiment. Version 2.0 of our publishing strategy focuses on empowering scientists and streamlining our workflows. The central change is to try and catalyze a shift toward fully scientist-driven publishing, where researchers will take a more active role in managing their publications. The overriding assumption is that science will indeed be released. This is a big shift for us. And it will feel uncomfortable at times. But we’re excited to see what comes of it.

Our hypothesis is that by allowing scientists to choose the process, timeline, and format that best suit their individual projects and working styles, we will see greater ownership and a wider range of formats and approaches instigated by scientists themselves. We also hope this increases utility and engagement with others.

While scientists will still be expected to contribute to Arcadia’s scientific and commercial goals, the choice of publishing will sit squarely in their hands. One bottleneck we foresee is our scientists worrying about how their choices may impact Arcadia’s commercialization opportunities. We’re still thinking through the various solutions that can address this — it’s certainly a challenging and interesting part of our experiment and we’ll provide more updates on this soon.

This shift will take deliberate effort, as it’s not just operational but also cultural. To facilitate, we’re streamlining our workflows and introducing new support mechanisms. Standing meetings and previously required steps, such as a kickoff meeting with the publishing team, will be replaced with on-demand consultations, offering greater flexibility and efficiency for both scientists and the publishing team. Ultimately, a successful shift is in the hands of the scientists themselves.

We’ve whittled our process down to just two core publishing requirements. A pub can go out if:

All byline contributors approve release.

All code, data, or other materials necessary to replicate or reuse the work are available in FAIR repositories.

These requirements ensure that we maintain our commitment to open science and collaborative accountability while allowing for greater freedom and ownership at the individual level.

As we make these changes, the publishing team will focus on developing more sophisticated resources and providing enhanced, targeted support to our scientists. This includes exploring the development of tools like AI-assisted text generation and editing, further streamlining the publication process. This pub itself benefited from a more creative approach — we generated a first draft using AI to transcribe and summarize a live, in-person, internal company presentation given just last week. Having this starting point helped us get activated quickly, editing and filling in the gaps over just a few days. 

Next steps

This updated vision is not a final product nor a condemnation of our original approach, but rather a continued evolution of our experiment that we feel lucky to iterate on. As we move toward a more agile and scientist-driven model, we expect to encounter new challenges and learn from them. This approach lets us refine our process and make sure our publishing model stays adaptable, effective, and replicable. As before, we’ll assess this new path to see how it plays out and continue to adapt.

Increased flexibility might lead to some variation in publication quality, longer release times for some pubs, or accidental release of potentially commercializable insights as we adjust to the new system. However, we believe that scientists having more agency to publish as they see fit will expand our experimentation, foster a stronger commitment to quality in the long term, and allow them to realize the promise of openness to accelerate and improve their science. 

In version 2.0 of our publishing approach, our scientists will take the lead with the ultimate goal of creating a more replicable and impactful model for sharing science.

Methods

We used Google’s Gemini 1.5 Pro model (via Google AI Studio) to analyze a video recording of an internal presentation where Prachee Avasthi described this new version of Arcadia’s publishing experiment. We included this video and the original text of “The experiment begins: Arcadia publishing 1.0,” then prompted the model to turn the presentation into a pub that could serve as a follow-up to the original, asking it to write in a similar style and voice. We then wrote additional text to expand its output and edited the text to better reflect our thoughts.


Share your thoughts!

Watch a video tutorial on making a PubPub account and commenting. Feel free to add line-by-line comments anywhere within this text, provide overall feedback by commenting in the box at the bottom of the page, or use post about this work on social media. Please make all feedback public so other readers can benefit from the discussion.


  • Acknowledgements

    • Thank you to Amar Ghodasara, Katie Kelleher, Becky Pferdehirt, Jessica Polka, and Greg Wright for participating in the “deep dive” discussion on author agency that helped shape this new model.

  • Competing interests

    • Prachee is also the head of open science at Astera Institute.


Contributors
(A–Z)
Conceptualization, Supervision, Writing
Visualization
Editing
Supervision, Visualization, Writing
Resources, Writing
Comments
5
?
Nanticha Lutt:

Thank you for sharing this exciting evolution in your publishing model. Given the shift toward empowering scientists to have more agency over when and how they share their work, how do you envision balancing open science principles with the protection of potentially commercializable discoveries? What strategies or tools are you considering to ensure that open publishing enhances Arcadia’s ability to pursue commercialization opportunities without inadvertently disclosing valuable intellectual property prematurely?

Prachee Avasthi:

Thanks for this question! The strategy we’re employing here is to ask our scientists to think more carefully about commercialization and not act out of fear or conservatism. We’re saying a vague notion of future value is insufficient to justify protecting information. This has the dual benefit of scientists educating themselves on a path to commercialization for their ideas and also allows us to leverage all the benefits of open science to improve and speed our science. Further we think (and have heard feedback to this effect) that sharing evidence of our discoveries increases faith on the part of investors that they can vet if the advancements we are making are solid and useful rather than banking on unsupported promises. We believe the vast majority of early stage discovery is not what makes sense to protect and is better leveraged by sharing. Only trying this will yield confidence to this effect and we have the unique opportunity and privilege to do this experiment.

Stephanie Melchor:

I’m curious about the relationship between rigor and timeliness, especially given how quickly a scientific story can change with more data.

No scientific work IS ever done, but I think some of it is more done than others. It seems like from a structural standpoint, pubs are replacing traditional peer-reviewed journal articles, and if I were a scientist operating at that assumption at Arcadia, I’d probably be pretty slow to share in-progress results too!

But then I was thinking about how sharing research at all stages of “doneness” is important and still a pretty standard part of academic research (at least for trainees): you get an interesting result or run into a weird technical problem and you bring it up at lab meeting, where you get feedback from your peers and colleagues.

You plug that feedback in, get more robust results, then present your research-in-progress to your department. Their comments improve how you design your experiments or do your statistical analysis or the conclusions you draw.

Rinse and repeat for poster presentations/conference talks until you get to a peer-reviewed publication.

Is the idea that at least some of the pubs might be representing a story at these earlier stages, making it more of a global lab meeting presentation than a publication? (Sorry this is so long!)

Megan L. Hochstrasser:

Thanks for your thoughts! Yes, the idea is to share pubs across all stages of the research cycle, even early ideas we haven’t even tested yet. Some may evolve over time as we add new data and release new versions, but some may stay as-is and only ever reflect the early stages of a project. When we share early or less-developed work, we try to be abundantly clear about what stage it’s in, pointing out any caveats so as not to mislead readers. For example, in this pub, we couldn’t replicate some initial findings, but since we weren’t going to pursue the project further, we decided to share what we’d found in case it’s useful to others, just with multiple flags dropped in about reproducibility.

?
Femi Alakija:

I like the publishing process. I was just wondering how other groups and individuals working on open science or re-envisioning publishing contribute to or learn from Arcadia’s experiment and what lessons from the first phase of Arcadia’s publishing experiment can be applied to help other organizations or individuals aiming to reimagine their own scientific publishing models.

Prachee Avasthi:

As we have learned from many other open science experiments done by others, we hope that others innovating on their publishing models note the aspects of our model that suit their needs and which we have derisked. Aspects of our model I think were informative:

  • People can and do comment on our publications even though they live outside the ecosystem (something many were skeptical of).

  • CRediT taxonomy really helped solidify a culture of collaboration that I think is difficult to do with a hierarchical/winner-take-all author lists.

  • Author agency is essential to maintain otherwise we risk gatekeeping in new ways.

  • Scientists are rigorous and have perfectionist tendencies that make them reluctant to share early even if all the incentives and motivations are aligned for them to do so.

  • Measuring impact and reuse in the absence of proxies is difficult but essential work.

There are many others and we hope to release another pub soon that tries to capture learnings from our latest iteration. As far as contributing, please reach out if you or anyone else has run experiments that we might find informative! Thanks for the question!

?
Maxine Levesque:

Buried the lede a little bit here, I think! How hands-on did you find you had to be with this workflow? The content itself is awesome, but I am extra amazed if these methods generalize; the implications are cool to think about!

Robert Roth:

Hi Maxine — thank you for commenting! Agreed that the implications are very cool to think about, and we want to explore trying this with scientific publications to see if it’s able to capture what we’d need it to. With this pub in particular, I’d say that the workflow was fairly hands-on. But, it significantly sped up the release and provided a solid base to build on. The main elements that required human intervention were:

  1. editing the video down to fit into the 1,048,576 token context window (the limit at the time of writing) alongside detailed style instructions, a pub template, and the full text of ‘The experiment begins.’ We did put the style guide and pub template in the system instructions, so we’ve been able to test this a few times without needing to transfer that information over each time.

  2. expanding on points the model had no way of knowing or connecting. Naturally, the model couldn't intuit certain details or connections that we didn’t include in the talk. This was an internal presentation where we omitted some information because most Arcadians are already well-versed in our publishing process. This created gaps for the model, which was a helpful indication that over-explaining is an effective strategy when asking an LLM to create a publication from a talk. We also expanded upon some of our points that the model summarized too briefly. Something I’ve found useful is to explicitly encourage the model to generate long outputs, as they tend toward brevity (and these long outputs are still fairly summarized). Being succinct is something we strive for, but these tools aren’t always the best at deciding which pieces to cut, and it tends to be faster for us to remove text rather than expand text.

  3. editing the voice and style slightly. We found that Gemini provided text that was closest in style to our voice in pubs compared to other LLMs, but there were still areas where it came across slightly robotic. This was not a huge lift though and it came surprisingly close to our style.

Hopefully that gives you some more context! Happy to answer any questions about it too.

+ 2 more...
?
L. Robert Hollingsworth:

As a template for publishing at a for-profit company, how does this interplay with Arcadia IP? For example, public disclosure regulations behind filing patents or trade secret for spinoff companies? Do scientists still go through legal prior to publication?

Prachee Avasthi:

Hi Bobby, thanks for your question. Our scientists are explicitly not required to go through a mandatory check through legal for their pubs anymore. Like all other aspects of publishing, we aim to make educational resources/expertise available to them so they can learn more about what might have commercial value. We are still figuring out all the mechanisms for this but the certain thing is that there is no mandatory gate through legal. I personally believe that thinking more carefully about commercialization themselves will make our scientists more equipped to pursue the more promising revenue streams AND share things that will accelerate their science upstream. We expect to learn a lot more by trying out this shift in a way that allows more experimentation across the team than via a single mandatory model.

+ 1 more...