Purpose

“Icebox” is our list of projects that we’ve “iced,” or decided not to pursue. We share these efforts and our decision-making openly, so that others can build off of our most ambitious swings and misses. This pub is a data-driven look at how the icebox approach has affected our science and our culture. Overall, our employees feel very positive about icebox and we’ve directly shared knowledge and materials with several external research groups.

We hope sharing this pub will be useful to others building science organizations, and that it can provide a window into our company’s values and processes. If you try implementing your own icebox, or you’ve experimented with other approaches for sharing “iced” projects, we’d love to hear about it in the comments!

• This pub is part of the model creation effort, “Building a science company.” Visit the model narrative for more background and context.

• We previously published a blog post about icebox and why we decided to try it.

• Check out all the iced pubs and projects in our icebox!

Share your thoughts!

Watch a video tutorial on making a PubPub account and commenting. Please feel free to add line-by-line comments anywhere within this text, provide overall feedback by commenting in the box at the bottom of the page, or use the URL for this page in a tweet about this work. Please make all feedback public so other readers can benefit from the discussion.

Background

At Arcadia, our goal is to explore the full diversity of life so we can find and translate biology’s most useful inventions. We chose this ambitious goal knowing that it would not be an easy task. If we are meeting our desired level of scientific risk (high!) and being appropriately discerning about what we follow up on, our default expectation is that we should constantly and quickly be churning through ideas and projects.

While one might think of the projects that we don’t end up pursuing as “failures” in the traditional sense of the word, we actually view them as huge successes. Our “failed” projects teach us valuable lessons, and are essential ingredients for strategic iteration and process improvement. With this in mind, we aim to fail fast and fail often. We created the “icebox” — an open collection of projects that we have paused, or “put on ice” — as a way to encourage, celebrate, and share these incredibly valuable outcomes.

Failure is common in science, and lots of labs and companies discontinue work over time. Since our research portfolio is optimized for risk and breadth, we know that we’ll have to ice a lot of projects to find the ones most likely to succeed at Arcadia. However, it can be incredibly uncomfortable to do high-risk science when it doesn’t work, and it can be hard to walk away from projects that we care about. Our hypothesis is that by creating visibility around our high failure rate, and highlighting the valuable lessons found in our iced projects, we can push ourselves further scientifically and feel happier doing it.

We’ve chosen to share our icebox publicly because of our open publishing model and because we believe sharing failures is particularly valuable. Every iced project contains a lesson, and we hope that these insights will accelerate scientific progress for others. You may notice that we frequently choose to walk away from projects that technically “worked” but didn’t make sense for us to pursue strategically at Arcadia. In such cases, we really hope that we can help others pick up where we left off scientifically. We use material transfer agreements (MTAs) to share physical samples and we publicly share other resources in open repositories. We’re strongly committed to helping others use data, insights, and materials associated with our iced projects and are thrilled when we have the opportunity to help research move forward beyond our walls.

Icing projects and sharing them openly is a major part of our process and culture at Arcadia. In this pub, we take a data-driven look at the impacts of this practice on Arcadia as well as the broader scientific community. To understand how this process impacts our company, we surveyed our employees, who we call “Arcadians,” to ask for their feedback and opinions about icebox. Our survey had a 72% response rate among scientists at Arcadia, including almost all scientists who had led an iced project. To measure the impact on the broader community, we’ve tracked the number of iced projects and knowledge/material transfers of projects to others outside Arcadia. If you’ve followed up on any of the science we’ve iced, we’d love to hear about your experience in a comment!

How’s it going?

We’ve iced a lot of projects

We began our practice of icing projects in earnest at the beginning of this year, and announced our icebox effort in June. So far, we’ve iced and shared a ton of stuff! We’ve iced four larger projects (“Spatial genomics: Tracking biomolecules across space and time,” “Dissecting mechanisms of environmental adaptation in protists,” “Understanding the evolution of actin-binding proteins across diverse species,” and “Exploring bacteriophage nucleic acid chemistries”) and multiple smaller pilot projects. In total, 11 out of the 34 pubs released before this one are part of our icebox, which is 32% of the pubs we have so far. You can browse them all here.

When we surveyed scientists to ask how many had either led or contributed to iced projects, we found that the majority (two-thirds) of scientists had been involved in a now-iced project in some capacity (Figure 1). Many scientists had been part of multiple iced projects, sometimes in a lead role and sometimes in a contributor role.

Arcadians are very positive about icebox 

In our survey, we asked Arcadians to rate how they felt about the concept of icebox, both initially and currently. While we sensed that icebox was overall well-received at Arcadia, it was still surprising to see the degree to which icebox perception is positive (Figure 2). Over time, the number of people who were leaning negative about icebox has decreased, and the number of people who feel very positive has increased, though the changes are not statistically significant.

We also wondered if people in scientist roles, especially those who have had to ice a project, might have different opinions than the broader group, which includes people who work in operations. However, there was no difference in initial or current opinion between scientists who have iced something vs. scientists who haven’t (unpaired T test; initial: p = 0.83, current: p = 0.45) or between people in scientist roles vs. non-scientist roles (unpaired T test; initial: p = 0.06, current: p = 0.80).

To better understand what is driving the company-wide perception of icebox, we also asked people to explain how icebox has affected them in a free-response question. We saw that the celebratory nature of icebox helped people embrace failure, instead of fearing it. We also saw that it was motivating for people to be able to publicly release work that would normally be forgotten about or lost.

We’ve included a few direct quotes that capture these sentiments: 

“I love that icing is a fun celebration of moving on from the ideas that didn't work.”

“I think that the icebox has personally inspired me to work with less of a fear of failure. I think its an excellent method for keeping ourselves honest about progress and usefulness while still celebrating people's efforts in getting something off the ground.”

“I've seen a pretty significant shift in the attitudes of people regarding their tolerance for failure and releasing science that they may not have otherwise released.”

“It was very encouraging to have someone reach out with an interest in continuing our iced work.”

People are reading and building on iced pubs 

A huge goal of icebox is to share our work so others can make use of it. As of October 2023, our icebox pubs have a total of ~8,000 views, which tells us that there is awareness of our iced projects in the broader community. So far, two of our scientists have received comments on their pubs with requests to make use of their work. This has resulted in two MTAs to move our research materials outside of Arcadia. We consider this an incredibly successful outcome of sharing our iced work, and are excited to enable more knowledge and materials sharing to the broader scientific community.

Here is a direct quote from one scientist who shared their iced work with another group:

“Because we had iced the project, we were able to publicly share a rich dataset and to provide relevant experimental samples to someone outside of Arcadia. Icing the project facilitated quick sharing of our findings that enabled new external projects to build upon our work.”

Who decides to ice a project? 

We suspect that Arcadians also view icebox positively because the decision to ice a given project came from the lead scientist, and not a manager or executive, the majority of the time (seven out of 12 responses, ~58%). This is our preferred outcome, both for cultural and scientific reasons. While it is inevitable and normal for company leadership to make top-down decisions about scientific strategy, in most cases, the scientist actually doing the work will be in the best position to evaluate if the project is going well or not. We want to actively promote a high-agency culture where the people doing the work are the decision-makers, and we hope to increase the proportion of scientist-led icing even further in the future.

Anecdotal benefits

Beyond the quantifiable aspects of icebox, we also have observed some useful benefits. Below, we describe some of the benefits we’ve observed, with supporting quotes from Arcadians.

Recruitment and career development

Icebox allows us to openly share a very honest slice of our science. This gives potential candidates valuable insight into our company’s values, science, and decision-making process. It also lets us openly discuss scientific details and company strategy with job candidates during interviews. This is also true for Arcadians who will have a public record of their work when they one day want to seek new job opportunities.

"Job candidates I've spoken to and new employees at Arcadia have both relayed that they've seen icebox statements on pubs and think that's a really great thing to do/share.”

Project creation process

To build our projects, we now run short project “pilot” cycles where we rapidly evaluate the utility of a given scientific direction for a 2–3-month period. This enables nimbleness and rapid exploration, maximizing the types of scientific risks we take while minimizing opportunity cost. Pilots that show promise may grow into full-fledged, longer-term projects. Rapid and frequent failure lets us quickly run as many pilots as possible. We want to optimize for risk here, which means our rate of moving successful pilots forward should be low and our icebox should grow proportionally to the risks we’re taking. Check out “A capsid-based search recovers viral sequences from human brain sequencing data” as an example of an iced pilot!

“I can see how the ‘icebox’ plays an important role in the pace of the pilot process, where there is a ‘fail fast and often’ culture. It helps to not get attached to the project itself, but rather to keep exploring new projects.”

Alignment with translational goals

The translational arm of Arcadia functions as a startup company creation studio. Our process begins by searching across the tree of life for the most useful biological inventions to translate into the therapeutics space. We think that to find the best and most translationally valuable innovations, we need our translational project scientists to churn through a lot of different pilots. More translational pilots ultimately means more chances at spinning out successful startups.

“Overall, I think that the icebox and our positive mindset around the icebox has been and will be beneficial in helping us achieve our overall company goals. Getting a lot of shots on goal is important, and that means that iterating quickly is important. The icebox and learning that icing a project doesn't mean it ‘failed’, hopefully makes that process better and encourages people to keep iterating.”

Maintaining focus

We can’t do everything, so it's important that the things we do, we do well. What we choose to stop working on is just as important for our future productivity as what we do choose to move forward with. 

“My personal experience has been that when doing research and approaching new problems it's vital to have a regular system in place to stop work. Otherwise you end up with an ever growing list of things being worked on, which leads to less and less getting finished.”

What’s next for us?

1. Keep pilot projects failing fast and often. Right now, our pilot projects run on roughly three-month cycles, where we do rapid (generally computational) de-risking of project ideas. The tools we’re building in-house are designed to enable a rapid exploration across biological scales [1][2][3]. As our toolbox grows and we continue to hone our pilot process, we hope to greatly increase the throughput of pilots. And we need to be designing FOR risk, not against it. We’ll be sharing more about this soon.
   

2. More scientist-led icing. We want scientists to be empowered to make decisions about whether they continue a project or not. We think that this is vitally important for the success of our company. We ask our scientists to dream big and propose risky and innovative projects. It’s critical that the process of winding projects down feels positive enough to inspire them to try again.
   

3. More materials/knowledge transfer outside of Arcadia. We want our science to benefit the broader ecosystem. We hope that in the future, we do more material transfers for both active and iced projects, and for our scientists to be able to more actively share their learnings with other external scientists. We’re committed to helping others make use of our iced projects — please take a look at our icebox to see if there’s something useful there for you!

Ideas for implementing your own “icebox”

We’re still figuring out our scientific strategies and practices, and icebox has been a nice tool to facilitate and learn from this iteration. It’s not always easy to grapple with failure and share it publicly, but it’s been incredibly valuable for us as a company. If you want to do something similar in your own lab or science organization, we recommend it! To help, here are our top learnings about how to create an icebox:

• Put a name on it. The single highest-value action of making icebox part of our culture was putting a name on our practice. Creating our own words gave us a way of talking about project discontinuation together, which was our first step in doing the psychological and cultural reframing around what “failure” means at Arcadia.

• Build a shared understanding. We’ve learned that it’s very important to ensure a solid, shared understanding of why we let things go at Arcadia. This helps scientists be maximally strategic in how they approach their work and how they make decisions about icing or continuing projects.

• Align all contributors. When sharing the reasons for icing a project publicly, it’s an opportunity to make sure everyone on the project is aligned. Different contributors might have different takes, so it’s key to get everyone on the same page.

• Make sure it’s fun! The point is to celebrate our iced projects and for it to genuinely be a positive and enriching part of the culture. We are actively working on how to get the most fun out of icebox, and we look forward to seeing what other approaches convergently evolve outside Arcadia.

You might share discontinued work or negative data as a preprint, a blog post on your lab/institutional website, or you can follow our structure and make a free PubPub community. If you decide to try it, we’d love to hear how it goes.

Share your thoughts!

Watch a video tutorial on making a PubPub account and commenting. Please feel free to add line-by-line comments anywhere within this text, provide overall feedback by commenting in the box at the bottom of the page, or use the URL for this page in a tweet about this work. Please make all feedback public so other readers can benefit from the discussion.

• Acknowledgements

  • Thank you to all Arcadians who filled out the survey and who have been experimenting with icebox.