Continuous Delivery at Facebook

Continuous Deployment is the process of taking software and deploying it to production quickly, safely, and repeatedly without impacting the development lifecycle. Here are a few features to have in a continuous deployment process:

  1. Fully automated deployments,
  2. Self-service deployments for developers 
  3. An automated testing suite for generated software artifacts 
  4. Frequent software artifact deployments with incremental development improvements

If you’d like to learn more about these features, I highly recommend these key concepts covered in the following Deliver Better posts on Continuous Delivery and Testing Methodologies

It’s rare to find developers who can handle the entire cycle end to end, and very few organizations practice continuous deployment. And as a result these organizations continue to innovate and enhance their product lines quickly, efficiently, and in a scalable manner. This post shares how to leverage continuous deployment practices to enable developers to deploy numerous times a day to various environments, while incrementally improving their production code! 

Case Study @ Facebook

In the research paper titled “Continuous Deployment at Facebook and OANDA,” Facebook engineers and researchers published a study identifying the key elements needed by organizations to adopt and make viable continuous deployments processes. 

There is no official formula for continuous deployment, and so the paper offers insights derived from the practices of continuous deployment at Facebook and OANDA. 

The Key Learnings 


Desirability of CD & Empowered Culture


Developers want to deploy more frequently, and they prefer faster deployments in comparison to slower ones. As a result, they can focus on newer features, leverage newer technologies, and increase their productivity. Developers want to focus on building new things rather than focusing on the release around the software. If there are bugs, since these changes are incremental, the team can quickly and safely address these bugs while developing new features. While releasing new software, the developer's motivation can increase because they can see their contribution to products sooner and can be more proactive in owning the functionality and its behavior. 


Therefore development teams should prioritize the quality and performance of their features rather than the release process to get the feature out to an end customer. Developers should leverage deployment automation to increase the productivity and quality of their software releases.

The figure above “depicts the release rate over time for daily and weekly releases. Initially, they are fairly similar. However, as time progresses, the number of deployments in the weekly release decreases, even in the presence of rapid engineering growth rates… [therefore] we conclude the preference is towards shorter release cycles.”

Considerable And Continuous Investment

In order to achieve Continuous Deployment within an organization, there needs to be significant investment in resources, tooling, and education. 

Resources

Consider the costs of roles across your organization.

It may be difficult to find developers with the skill set needed to fully own the end to end lifecycle of your software application (This includes the development, testing, deployment, and maintenance of the software). These individuals are rare to find and it’s hard to attract them to non-technology focused companies, consider ways to grow your existing talent at your organization. 

Technical Managers and Leaders are also another cost for an organization. To accelerate Continuous Deployment within an organization, look for leaders who understand Continuous Deployment, it’s benefits and its risks. Leaders who understand and promote the practice impact the development and engineering culture of the organization. It encourages developers to deploy more frequently and not sit on their work. 

Tooling

Both Facebook and OANDA invested in building a Deployment Management System(DMS) to automate, monitor, and track their deployments into a production environment. 

For Startups and smaller companies the justification for building a DMS may be more difficult to make as these organizations may not have the developer capacity to build said tool. There isn’t a repeatable and uniform process for building a DMS. Some organizations look to their cloud providers to offer a solution before evaluating the need to build their own deployment system. 

Similarly, for larger Enterprises, where software products or services are not the core business, expertise around building a DMS  may not be there. Developers require deployment and domain knowledge to avoid dedicating 20% - 100% of their time building an internal tool that doesn’t tie to their core business or align to their engineering organization’s needs. This investment can become extremely expensive and can have a greater impact on R&D later on. 

Education

Proper developer onboarding, enablement, and training are all common practices within DevOps that should also be applied to organizations adopting continuous deployment.  It’s a behavior that was started by High Profile Internet Companies, eventually spreading to the rest of the IT industry. Every organization does education enablement differently. Companies like Facebook and OANDA invest in a “ramp-up” period of 6-12 weeks where developers join a boot camp to learn the organization’s internal development practices. 

Enablement sessions should focus on what common feature improvements look like versus higher impact changes like global configuration changes which may impact multiple teams. As a developer, they need to understand what they built and how it gets released to a production environment. Aim to answer questions like:

What work needs to be done to maintain a codebase?

How do I fix my feature bugs? 

When can I mark my feature for release? 

Where can I monitor my feature’s performance? 

How can I improve my feature? 

These are common considerations developers need to have some sort of playbook for so they don’t waste cycles coming up with their process. 

  1. Managing the risk-reward trade-offs
  2. Need for objective retrospective when failures occur
  3. Absence of separate testing team
  4. Dependencies Considered Harmful
  5. Versatile and Skilled Developers Required
  6. Considerable and continuous investment required 
  7. Technical Management Required

These considerations ensure the process is uniform, safe, and repeatable to ensure developer efficiency. 

Conclusion

If your organization is considering continuous deployment, please check out Facebook’s research paper linked here. It provides some really valuable insight on what it can look like in an organization that may not be an “internet company”. Here at Deliver Better, we also have a bunch of other neat articles to provide some more technical and in-depth insight into delivering better software! 

Whitepaper

Continuous Deployment at Facebook and OANDA

Related Posts