Continuous Delivery Industry Best Practice



    While the DevOps movement inspired large, enterprise organization with agile practices to allow developers to make quick changes, many of these organization have faced with the difficulty in getting full benefit as their legacy development process were not designed for supporting short software development delivery cycles and frequent production release.

    In order to address the needs to quickly, repeatably, and reliably release software into production, the DevOps movement has developed Continuous Delivery (a.k.a CD).

    This document helps you have comprehensive understanding for CD with principles and practices to attain successful CD strategy and guides practical delivery pipeline as the industry best practice.

What and Why is Continuous Delivery?

    According to Martin Fowler,

    Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.

    The word “discipline” in above definition means that CD is to allows developers to deliver incomplete software during their development iterations and get users to give valuable early feedback, essentially developers want to have as quick feedback with users as possible(a.k.a Fail Fast). It is all about feedback loops that provides not only the quality of code, but the quality of functionality, and the quality of software deployment process.

    In CD, Repeated testing process before production deployment takes advantage like that most of the errors are discovered early on and have minimal impact and fixing such the errors become much easier with fewer changes per release. Consequently software can be released much faster, more frequently and reliably into production due to heavy focus on visibility, instant feedback and incremental changes. The goal of CD is to make deployments much predictable and routine affairs that can be performed on demand.

    As opposed to single tool or product you can purchase, because achieving successful CD is difficult and requires holistic approach, it is worthy referring that CD is a collection of tools, processes, culture and philosophy used to deliver incremental value to and have a positive effect on business as a whole.

What are differences from Continuous Integration and Continuous Deployment ?

Continuous Integration vs Continuous Delivery

    Continuous Integration(a.k.a CI) is a development practice that requires developers to integrate code into a shared central repository several times a day. Every single commit, it is then verified by an automated build with unit test to detect integration errors as quickly as possible.

    In CI, the majority of the work is performed by automated build with unit test. It makes use of a unit test framework along with a build server, which performs the actual tests. but it doesn’t take care of business logic since unit test typically is unable to narrow down onto business logic problems, especially design issues.

    CD is basically the logical extension of CI, it is a more holistic solution than CI as it encapsulates a lot more than just the development of software. Even though CI take care of the build with unit test part, it don’t allow testing of the business logic since unit tests typically are unable to narrow down onto business logic problems, especially the design issues. In CD, small sets of the developed works done in CI pipeline are continually supplied for user acceptance testing (a.k.a UAT) or to the staging environment for review or evaluation. This enables business issues to be discovered early in the development cycle, ensuring that fixing takes place before moving ahead to other aspects of the project. The feedback loop in CD is a good example of how CD is more holistic than CI.

    CD focuses a lot more on requirements than CI and involves many people on the delivery pipeline than CI. It can be worthy mentioning that CD embodies the practices prescribed by CI and the Agile principal of satisfying customers through early.

Continuous Deployment vs Continuous Delivery

     Continuous Deployment goes one step further than continuous delivery. Every changes in the code of software that verified with all of the tests, unit test, UAT etc, are automatically deployed to the production.

    Software with CD is always ready to release, since every single change is delivered to a staging environment using complete automation and iterates the feedback loop, it assures that all developers become confident the application can be deployed to production with a push of a button when the business is ready.

     Even though Continuous Deployment is an excellent way to accelerate the feedback loop with users and should ideally be the goal for every organization, if an organization have the step to wait for a business decision to go a feature live, it will be impractical, so the final deployment to production will be a manual step. The point is to decide whether Continuous Deployment is right for your organization based on business needs (Source : Continuous Delivery vs. Continuous Deployment: An Overview ).


    The goal of CD is to establish an optimized end-to-end process, enhance software development cycle, eliminate the risk of problems in production release as much as possible and provide quicker time to market. In order to achieve the goal, there are 8 principles of CD as follows (Source : Jez Humble and David Farley co-authored, Continuous Delivery).

1.  Repeatable and Reliable Process

    Repeatably using same process in all environments is a way to discover most of issues early and fixing such the errors become much easier with fewer changes. Lastly, the process comes to reliable one through all environments.

2.  Automate Everything

    Unlike manual processes, which are more like inherently less repeatable, more prone to error and less efficient, somehow automated processes allow you get consistent results, which tends to lead to reliability.

3.  Bring the Pain Forward

    Because It is natural to have difficulties or pain in breaking through delivering pipeline where you build software, it meas in every single time when you face with such a difficulty, it should be dealt with as soon as and as often as you can. In the end, it will lead you to improve it, probably automate it, and make it easy or painless.

4.  Version Control Everything

    To provide reliability, everything getting through delivering pipeline, like source code, configuration, script, static files, documentations, even database related stuffs should be under version control with sophisticated SCM, it is all about one source of truth.

5.  “Done” means “Released”

    This implies ownership of a project right up until it’s in the hands of the user, and working properly, a feature is done only when it is in production. Having a clear definition of “done” right from the start will help everyone communicate better, and realize the value in each feature.

6.  Build Quality In

    Investing in your quality metrics, which could be unit test coverage, code convention, high cohesion&low coupling, complexity etc, will invariably be better than one without, and easier to maintain in the long run

7.  Everyone is Responsible

    Similar to “Done” means “Released”, until a feature is in the hands of the user, it doesn’t value anything. Developers should develop projects with a mind for how to deploy them. Project managers should plan projects with attention to deployment. Testers should test for deployment defects with as much care and attention as they do for code defects and this should be automated and built into the deployment task itself.

8.  Continuous Improvement

    This means your system will always be capable of get evolved and therefore easier to change when needs be. This principle is the most important for effective automation.


    To go with the 8 principles there are also 7 practices that are supported by a set of complementary task-specific tools, in what’s known as the “DevOps toolchain” (Source : Jez Humble and David Farley co-authored, Continuous Delivery).

1.  Build Binaries Once

    Building one single binary as the built artifact only once eliminates the risk of untracked differences due to various environments, third party libraries, different compilation contexts, and configuration differences that will result in an unstable or unpredictable release.

    The binary should be stored centralized repository from where deployment process can retrieve the relevant artifacts.

2.  Deploy the Same Way to All Environments

    For consistent or reliable deployment process in all environments, the same set of automated deployment must be repeated. This ensures the same results in lower environments with many times, such as integration, QA, etc, as in higher environments like pre-production and production , with fewer deployments. it can’t be afford to fail deployment to production because of the least tested deployment process.

3.  Smoke Test

    Including Smoke Test in the deployment process make sure that most crucial functions of a program work and will pass basic diagnostic. Smoke Test is non-exhaustive software testing (of all elements, such as services, database, messaging bus, external services, etc.) and doesn’t provide the same fine-grain comprehensiveness as full test suites. However, it can be run frequently and quickly, often in a matter minutes (rather than hours or days). This allows for a much quicker turnaround on what can become very time-consuming and basic issues.

4.  Stop the Line If Fails

    If a problem on a stage in the pipeline arises, discarding deployment, you should fix the issue properly and start the process again. Don’t patch, don’t hack.

5.  Deploy into a Production-like Environment

    Production-like environment validating changes before pushing to production allow you to have the opportunity to effectively eliminate mismatches and last-minute surprises.

    The copy of production should be as close to production as possible with the actual final production environment regarding to infrastructure, operating system, databases, patches, network topology, firewall and configuration.

6.  Instant Propagation

    CD must be an ongoing process of building, testing and releasing. Every check-in to a source control repository should trigger the next one immediately upon successful completion.

    CI tools, like Jenkins / CloudBees, Bamboo, TeamCity, ensure that cascade of actions along the delivery pipeline is consistent, automatic and instant. Thus, development teams merge developed code and get feedback from automated test tools many times a day, for a more efficient deployment and update process.

7.  Include the Database

    Unlike other software components or compiled code that are easily copied from development to testing to production, having database participated in the delivery pipeline is neglected in many times.

    Because the database requires dealing with different challenges than application code, specialized database tools such as enforced database source control , a database build automation tool and database release and verification processes can ensure a stable resource in your DevOps chain.

Mature Continuous Delivery Pipeline and Tools Landscape

    CD pipeline is like sophisticated workflow helps establish release mechanism that allow you to practice production release with the quick feedback loop repeatedly before release production. Also it makes sure a change committed by developers is processing in flow under the control.


    As described as the above, in order to achieve successful CD, it is much more needed to take into consideration how to let the thought, culture, even philosophy of CD (including CI) sink into developers then technical approach like thinking what tool or product would be better and going one step further, it will be nice to provide CD pipeline as the best practice.

    To be realistic thought, it would be obvious true that it is pretty much tough or highly costly that let more than thousands developers across more than hundred development teams align to the same software development cycle based on single idea without authorized suppressing, especially for large organization like Rakuten.

    Because unlike CI which includes a lot of developer’s issues, like build, unit test, source code quality management, or a way of implementation in relation to specific framework or library, actually may people from producer, QA, business or even infrastructure getting involved in CD with the feeback loop may not care about those issues, but what they care about is more holistic issue whether it satisfy the customers, lastly it brings value to company or not, I think CD would have more room to let the developers align to the same delivering or deployment cycle than CI.

    The recommended practices for those who are seriously intended to build and extend CD as Service over the organization, is to do small start with the deployment in the feedback loop related stages in CD pipeline, which somehow works to many developers to get them on a same compliance. As for CI, for the time being, it wouldn’t be big deal let developers do on their own even it is unique way until the thought of CD sink into developers, since most of the issues in CI should be developer’s matter.