The situation
Our customer is a large enterprise with a portfolio spanning financial and insurance products and services. At the centre is a key piece of infrastructure, supporting more than 30 internal and customer-facing applications. This platform is a complex array of business logic, policies, and multiple interfaces, creating a fragile environment.
As a consequence, attempts to upgrade the platform generally caused catastrophic failures, with unpredictable and severe business and customer impacts continuing for weeks following the upgrade.
The hardware support for this application was reaching its end of life, forcing the organisation to undertake a risky migration piece to move the platform to newer hardware. Migrating to the cloud was not an option, and low internal and external confidence had created a paralysing effect on the team’s attitude towards the complex migration.
The challenge
Throughout the migration, dependency, and stakeholder management were critical, as some of the connecting apps were core business applications. Being critical infrastructure, minimising downtime and maintaining data integrity was paramount during migration. What’s more, the small support team were stretched and worked with lengthy manual processes which were complicated and prone to human error.
Typical upgrades required the platform to be shut down for a period of time. To minimise customer impact, maintenance windows were set up outside of business hours (generally starting at midnight on a Saturday). This process took anywhere from 2 to 4 hours, with more than 30 people required to manually validate that their connected application was operational before a go/no-go decision.
Cprime Elabor8 was engaged to build a suite of automated tests to provide rapid feedback throughout development work, eliminating the need for a large on-call team. At the same, help was needed to build stakeholder confidence throughout the migration.
The process
Due to the scale and complexity of the work, coupled with low confidence, it was clear that misaligned priorities and coordination between teams was slowing down development. This resulted in costly extended support fees ($x00,000), which were a growing concern for the organisation.
We built interactive architecture diagrams highlighting operational flows through the system. This broke down the work into ways the team understood, creating a virtual ‘war room’ that focused on a single flow, driving a unified approach to delivery.
While creating the automation framework, we identified a need to understand the ‘customer’ and how their applications interface with the platform. We introduced Behaviour Driven Development (BDD) to describe the behaviour of these connecting applications. This gave the team a solid understanding of the needs of the customer and their connecting applications, enabling them to quickly test theories and validate changes.
Using these scenarios, we built automated tests against the original platform and repointed these to the new platform. This rapidly demonstrated to the team that platform deployments were successful, driving a boost in confidence that led to the introduction of further automation.
The results
- Increased productivity and engagement – Teams worked together to deliver specific components of functionality and build a roadmap to success.
- Reduced maintenance windows – Maintenance became a smaller, simpler operation with no business or customer impacts.
- Confident, high-performing teams – By focusing on small pieces of functionality and real customer cases, teams are now able to provide quality to vast numbers of end users.
- Convenient maintenance times – Maintenance work is now seen as a business hours’ activity, without teams needing to log in for hours during late night maintenance windows.
- Early problem identification – Continuous product testing allows teams to identify customer problems sooner and improve incident management.
