We’ve all seen it. The application breaks on a developer’s laptop. It’s fixed by a configuration change such as adding a data source or changing a threshold or toggling something. A day later, the issue is breaking across the earlier test environments and the fix is communicated. A week later, QA files a bug. It is rejected as “mis-configured”and testers are told how to fix the configuration. Later, the application fails in the pre-prod environment. Operations moan about poor deployment documentation, and will (hopefully) add a line-item to their release plan for production.
What a colossal waste of everyone’s time.
- Classic Facepalm
This is a DevOps problem. We see communication issues, exacerbated by silos, which lead to changes being more expensive and riskier than they should be. There are at least two solid approaches to addressing this pattern.
Approach 1: Configuration as Code
Cool-kid DevOps suggests that infrastructure and configuration should be code. The developer who makes the code change that requires a configuration change is made responsible for encoding that change in deployment automation, scripts or configuration files pushed with each change. The code change is not “done” until the configuration automation change is done.
“Code” does not necessarily mean actual scripts. It means that the rules about what configuration should be applied exist in a versioned and executable state. This could be a script in a Git repo or as CI in a CMDB that some software reads from.
Strengths: The responsibility for the configuration is clearly owned by the developer. It limits unpredictable new tasks for other people based on your ‘done’ work. There is little room for miscommunication or errors in manual steps.
Weaknesses: It can take considerable engineering shifts to learn how to automate configuration changes, and some tools are scriptable than others. Also, there may be required knowledge the developer is not allowed to have. For example, if a new datasource is required the developer likely won’t know production passwords, only how to access the development copy.
Approach 2: Easy Capture & Communication of Config Changes
If fully automating this approach is not feasible, it is important to lower the barriers to communicating the change. This is where DevOps stops being about cool automation and really focuses in on tearing down the barriers between silos. In many organizations, the Developer is asked to fill out a ticket with dozens of required fields for each change made to each environment. That is a pretty strong incentive to “forget” about it.
Instead there needs to be a low-ceremony, easy strategy for capturing changes that need to be made, and recording which environments the change is captured on and which will need it when they get the code change. Options vary from release planning tools; to cards on middleware team’s white-board; to wikis. The key is for operations and release management to find something that meets their needs to protect production while being friendly enough to development that it is actually used.
Strengths: This approach can accommodate a wider variety of changes with less pressure on the development organization to take on new tasks. Instead, it should reduce the pressure to write long deployment docs that developers hate writing and often lack the quality release engineering needs.
Weaknesses: This approach is still prone to misunderstanding instructions and manual errors when executed.
Hybrid approaches work
Hybrid approaches are also doable. An approach my customers may follow would be to update uDeploy with the new configuration task and have it run with each deployment. The environment specific information which the deployment automation guys might not have access to would need to be configured by the environment owners. uRelease makes it easy to capture this kind of task and track which environments it has been completed in. When all the environments are configured properly (along with an environment provisioning automation to cover future environments) the task drops off the release plans.
Notice that in any of these approaches, the QA team is 100% correct that there is an unacceptable bug from Development if using the standard deployment processes does not result in successfully running applications. The configuration required by the application is part of the application. Various stakeholders in the organization can nudge their colleagues towards these techniques by standing up for these types of bugs and not closing them until the configuration change is incorporated in the deployment process as either an automated step or clearly documented step in a release plan.
