What is Release procedure?

Release procedures are often done ad-hoc with the project development. Getting it right from the start can benefit overall project health in the long term and take a toll away from the team.

There are a lot of tools to choose from for the job but what matters most is to clearly define requirements, procedures, and environments.

All the code is almost always stored in some kind of version control system. Let's say, Git (Which is the choice in most cases).

The first thing that comes to your mind should be the branching strategy.

So be it, we can choose a top-to-bottom or bottom-to-top approach when designing any process.

1. Branching strategy > Environments (ttb)
2. Environments > Branching strategy (btt) - Preferred

Branching strategy

The first step is designing a branching strategy. You can choose from any existing branching strategies:

• Gitflow strategy
• Trunk-based strategy
• Github strategy
• Gitlab strategy

Gitflow

Gitflow is the most popular strategy out here but it's the most complex.

It has two main branches:

• develop (Development code)
• master (Production code)

Develop branch is used for the development code. The master branch should contain, at any time, production-ready code.

Also, it has three help branches:

• release (Created from the develop when creating the release)
• hotfix (Created from the master when creating hotfix)
• feature (Created from the develop when working on the feature)

The release branch is created from the develop and merged into the master.

Hotfix branch is created from the master and merged to develop and master.

A feature branch is created from the develop and merged back to develop.

This is it in a nutshell. Pretty simple idea but when work is done and not properly used it can make a hell of your life.

It's used in bigger teams mostly and in long-term projects.

Trunk-based strategy

Trunk-based strategy is pretty simple. You have one main branch and off that branch, all the work is done in other short-lived branches.

Trunk-based doesn't use any other long-lived branches like develop, feature, release, etc.

Developers commit work in smaller pieces directly to the main branch as often as possible.

Github strategy

Github strategy is the soft version of Gitflow. It's the same as Gitflow but it doesn't have a development branch. All the feature branches are created from the main/master branch.

Gitlab strategy

Gitlab strategy is somewhat similar to the GitHub strategy but it takes into account environments. For every environment, you should have a branch that represents the state of that environment at any time.

What branching strategy should I choose?

Well, it depends on the project.

Gitflow

If your project is a long-term project that will have a bigger team - Gitflow is the way to go. Yes, it does introduce the complexity but it also gives the possibility to be very flexible in the releases. It's clearly defined how to approach all situations possible.

Gitflow is a choice when you have long release cycles. It's not very usable with the CI/CD concepts but can be managed to implement automation very well using GitFlow strategy.

Trunk-based

Trunk-based is neat when you have a smaller experienced team that can track work done and when you want to implement CI/CD properly. The work is committed as often as possible to the main branch and it's deployed directly to the production hopping through the process.

GitHub flow

GitHub flow is a choice when you don't want the complexity of the GitFlow but want to separate the work using feature branches and want to go faster to the production.

It also enables CI/CD in a way where the feature branches can go to production as soon as merged in the main branch.

GitLab flow

GitLab flow is a choice when you have requirements the same as the one in the GitHub flow and also you want to separate the code per environment.

In a nutshell:

• GitFlow works well with long release cycles, multiple versioning, flexibility, and bigger teams. Doesn't work well with CI/CD processes but automation can be implemented.
• Trunk-based works well with experienced smaller teams, CI/CD processes, and go fast and break things.
• GitHub works well with CI/CD processes and goes to production faster than GitFlow (Shorter release cycles)
• GitLab works well for the scenarios same as in the GitHub flow but also when multiple environments are present.

Environments

When working on larger projects there are usually multiple environments before the code gets to the production.

The release is hoping from the lower environments to the upper environments. The coined term is promotion.

Eg.

develop > staging > production

This is one case scenario. There can be an arbitrary number of environments.

In the case above:

• Develop can be used for the developers to test their code and see how it is behaving in the environment different than the local environment
• Staging can be used for the QA engineers to hunt for the bugs and to see how the release is behaving in the "near prod" environment
• Production is the holy grail and it's used for the real testing. The test of the usage by end users.

Also, you can have only one environment, two, three, and so on. It's up to you really.

Rule of thumb: The number of the environment doesn't decrease the number of bugs in the production code but it does increase the complexity to maintain the environments.

So when choosing a number of environments have in mind next things:

• The more environments the more complex would be to maintain the infrastructure
• The more environments the more complex would be to track the release and what is deployed where (Full time job for the Release manager)
• The lesser the environments the work done is colliding within the team and/or external users (Third party company, External consultant, client, etc.)

So the optimal range should be between 2-4. Going lower is not advised but going upper is also not advised.

Branching and environments

Let's go through the next scenario:

1. Start with environments and come up with a branching strategy

1. Environments > Branching strategy

The first and most important step is to define the requirements. Should be done in consultation with the client, team, stakeholders, and all other entities affected by the decision.

Environments cost money. In terms of the infrastructure, engineers maintaining it, SREs, Release managers, etc.

Also, environments are technical debt by themselves. They need constant operators and this can create a toll on the team.

External factors are also there sometimes. I've seen a lot of times incompetent managers increase the number of environments for the testing of the one/two-member team to spend the budget or for some other political reason. This decision can make teams burn out faster and people leave the team.

All these things should be taken into consideration before choosing to add one more environment, as you can see it has a great impact on a lot of things.

So let's say next:

• We need three environments: develop, stage, prod.
• Develop is used for the developers work
• Stage for QA testing and pre-prod checks
• Prod is used to the real thing - used by the end users
• The release cycle is longer: 4 times per year (3mo/cycle)

These requirements point out that we shall use the Gitflow procedure because:

• 3mo/cycle features are developed but the previous release may need hotfixes
• The develop will represent the state of the development environment
• The master will represent the state of the stage environment
• All the developer can work on their branches without interrupting other work

Designing it this way we solved the downsides we encountered:

• 2 branches - 3 environments
• Keeping up the master as much as close to the develop so it doesn't drift a lot which makes our lives easier with merging/rebasing

Implementing automation can be designed like this:

• PR to the develop deploys the code to the develop environment
• PR from develop to the master deploys the code to the stage environment
• When the proper release candidate from the master is tested on the stage it can be promoted to the prod for the end users

Work can be done on the develop/master branches as much as possible.

Hotfixes initiated by the bugs from the stage are created from the master and it's merged in the develop and master. Releases are created from the develop and are treated like the release candidates. The feature is created from the develop branch and merged to develop.

Using the ratio of 2 branches - 3 environments is ideal for this setup for these kinds of project requirements.

The production environment is isolated and is only used for promotion from the stage.

The downside of this design is that the infrastructure and configuration part of the releases can drift off between the stage and prod.

The solution is to be very strict when applying configuration/infrastructure changes to the stage and prod environments.

If you have any comments/remarks on this issue just hit me up in the comments. I am eager to learn other experiences in release procedures.