One of the most important considerations within large projects is a clearly defined and structured merge and integration process. Imagine different teams working on the same code base - these teams could be split over different countries. Problems usually arise when these teams need to implement features affecting more than one subsystem and embed them within the existing solution. Often different teams work on features that affect other subsystems and they try to merge them together within one central point. To coordinate the integration and prevent problems, we need a defined gateway to the common code base - a defined merge and integration process. In this Techletter I will describe how you can define such a merge and integration process using the SCRUM methodology. We will assume that we have different teams implementing features that impact more than one product module or subsystem.
Definitions and Abbreviations
Feature which affects several subsystems
Feature which impacts only one subsystem
Integration of all subsystems in SW solution
Regular Product owner meeting where features are analyzed, for example implementation dependencies and cross-subsystem impact
Planning meeting where the sprint scope is assessed and the projected outputs agreed.
Regular feature team meeting where implementation across subsystems is coordinated.
Integration of subsystems involved in dependent feature implementation for validation/testing purposes.
Scrum of Scrum meeting
Meeting between scrum masters of different scrum teams.
Integration problems in large projects
When observing large projects, which include the implementation of features affecting the functionality of several modules, certain conclusions can be drawn. These include application of insufficient focus on work collaboration and coordination among subsystem teams, including functionality validation on affected integrated subsystems. Great attention has been paid to feature development of individual subsystems. However, issues affecting individual subsystems have generally been reviewed at the end of the sprint within the final merger and integration of the whole application, resulting in a higher risk that the entire solution will fail. Observed gaps summary
- Individual planning of feature implementation on subsystems
- Individual implementation on subsystems
- Individual verification of implemented dependent features on subsystems
- One central integration at the end of the sprint
- Tool and environment availability constrains
A predefined merge and integration process is used to manage the implementation of features affecting several modules. This process is adapted within the SCRUM process.
How to solve these problems
We need a process which can manage an implementation of dependent features and as a result its focus goes beyond the boundaries of one subsystem. Features that impact only one subsystem without interfering with other subsystems are not subject to this process. The process has two main goals, one is to improve collaboration and cooperation among subsystem teams working on a feature affecting these subsystems. The second is to ensure the stability of the whole SW solution during the SCRUM sprint period. The process covers planning, implementation and testing activities in development. It starts with defining the sprint scope and finishes with the solution demonstration at the end of the sprint.
The planning phase is coved by a series of PO Sync meetings. The PO Sync meeting shall identify dependent features and define individual priorities, dependencies, affected subsystems and acceptance criteria for these features. The feature analysis must be discussed in collaboration with architects to ensure there is a good understanding of the dependencies. The dependent feature definition and feature impact analysis shall be finalized for all planned features in preparation for the start of the upcoming sprint. As with independent features, before the sprint starts, the dependent features are assessed during the sprint planning meeting for each subsystem. During this meeting the role of feature coordinator shall be assigned to individual developers allocated to specific features. At the same time, when the plan is set up, the integration plan must be defined for the whole system per sprint. The entire team must agree upon the feasibility of the integration plan.
During the daily SCRUM meetings the dependent features are selected for implementation. As soon as the implementation of a dependent feature begins the Feature Sync meeting takes place. During this meeting the feature coordinator receives feedback from the feature team regarding the implementation status of each subsystem. The appropriate measures are then applied based on this feedback. Before the meeting takes place, the feature assigned tester is informed when and which test environment shall be prepared for subsystem integration testing. If the feature team reports a critical issue, constrain or subsystem integration test failure, the feature coordinator must immediately report this issue to the appropriate person within the SCRUM of SCRUM team or to PO. When the whole feature team is finished with implementing the feature, the feature coordinator will start to organize the subsystem integration testing.
Subsystem integration Testing
The objective of subsystem integration testing is to ensure that the correct functionality of the implemented feature affects the correct integrated subsystems. Subsystem integration requires a collaborative tester and a feature team. The collaborative tester prepares the required test environment. The feature team prepares subsystem builds for integration and prepares and performs the functionality tests. Functionality tests are a type of black box testing as there is usually a given input and the output is measured. With these tests we check what the system does. As soon as the feature passes functionality testing, it is approved and can be passed back to development. This must be done by development according to the integration plan. From this moment the dependent feature becomes a part of the current sprint output. When the dependent feature fails in the subsystem integration testing, the feature team needs to analyze and assess the reasons for this failure during the Feature sync meeting. The feature coordinator also needs to report this as an issue to the relevant people. In this case the dependent feature cannot be merged into the development branch until the problem is fixed and the functionality is approved with subsystem integration testing. In SCRUM of SCRUM meetings the integration plan shall be negotiated and communicated to the testing team in order to plan and assign resources for full integration testing.
What is needed
To ensure success, the number of dependent features defined for the sprint should be kept to a reasonable amount. Only careful feature assignment to the dependent features can guarantee that all steps defined in this process do not cause a significant increase of additional effort needed to apply the process. However, the process objectives defined above represent some additional effort in comparison to an undefined merge process. This increase of effort is required within work collaboration and coordination among the subsystem teams on one side, and on integration, testing and validation activities extension on the other side. This additional effort is required to achieve the desired improvements.
Solution state rule
- Main branch is in a consistent state throughout the entire sprint
- Development branch is in a consistent state during the majority of the sprint time
Dependent features are:
- Features which impact the whole system
- Features with impact on multiple subsystems
- Features which influence or interface with another subsystem(s)
- Fixing feature which impacts multiple subsystems
How to deal with dependent features:
- Implementation in mutual collaboration of all impacted subsystems
- For each subsystem a responsible contact person for each feature is defined
- Dependent feature tested with merged subsystems on instrumentation or in a simulator when appropriate
- Incomplete or failed features are not merged with any development branch
Possible branching concept
A possible branching concept could look like the following:
Fig. 2: A possible branching concept
Merge Bug fixes to current Baseline
Bug fixes are always done on a Release branch. Bringing back the amended code to the current baseline requires some special attention. In large projects with a complex setup and architecture, a merge of bug fixes with the baseline should always be applied with a separate bug fix feature branch. This additional bug fix feature branch is required in order to avoid breaking the integration flow of the changed code. The Configuration of Development team must be able to control the code change on the development branch. Bringing code from the Release to this newly created feature can be done with a baseless merge or by copy/paste.
Fig. 3: Bug fix merge
Steps for the Bug fix merge:
- Identify if Fix needs to be applied on current source baseline
- Create a feature branch from Development branch for fixes affecting more than one subsystem
- Base-less merge (or Copy-Paste) of required code changes
- Conduct test as if Bug fix is a new feature
- Merge (with FI if required) to Development
- Fix will be included in next Release
- Any implementation of features which effect several subsystems require a FEATURE branch
- NO development on and MAIN (always green)
- Sub-System Integration on DEVELOPMENT, only tested features can be integrated to DEVELOPMENT
- Nightly builds from DEVELOPMENT
- Weekly Build, Release Candidates and Releases only from MAIN
- Merge from DEVELOPMENT to MAIN executed by Configuration Manager and controlled by Integration Team
- Bug fixes and Release stabilization ONLY on Release branches
A clear and defined merging and integration process is an essential part of a successful project. It greatly reduces lost time, while merging and providing integration.
- Visual Studio TFS Branching and Merging Guidance: http://msdn.microsoft.com/en-us/magazine/gg598921.as