Making Continuous Deployment a Reality

An IT solutions innovator delivering an exceptional customer experience

Electric is a leading IT solutions company on a mission to simplify IT by providing easy-to-use IT technology and solutions to SMBs. The company has experienced rapid growth over the past several years and now counts over 1,000 small and mid-sized businesses across every major industry as customers.

Key to Electric’s growth is delivering exceptional customer service. “To win in our business, it’s essential that we deliver an outstanding customer experience”, explained Greg Pellegrino, VP Engineering. Staying ahead of the competition by rapidly innovating on Electric’s customer facing applications has been key to staying on top.

The challenge: increase release velocity while minimizing escaped defects

With all this growth, the engineering team has been actively looking for ways to ensure a fast release cadence that’s consistently high quality. “One of our biggest challenges has been preventing escaped defects. You never want a customer to find a bug that a release process misses,” noted Tom McGurl, Senior Software Engineering Manager.

Despite employing modern DevOps practices and testing tools, the Electric team found it challenging to continuously test new features against the entire Electric codebase.

“We’re good at smoke testing new features but the continuing problem is regressions between new and existing code.”

--- Tom McGurl, Senior Software Engineering Manager

This led to an unsatisfactory number of bugs identified in post-release firefighting. “Fixing big issues can kill an entire week”, added Devin Viss, Sr.Software Engineer. While the team used a session replay tool to find UX bugs, “the issue was fixed in the moment but not future releases”. So, Electric, like most companies, was caught in an endless loop of shipping new code that couldn’t be tested end-to-end against old code, users finding bugs in production that manual tests didn’t cover, and the development team spending countless hours fixing issues for the moment only to have them pop up again in a later release.

Solution: continuously test real user flows in Electric’s CI/CD Pipeline

To break the cycle, Electric was excited about Relicx’ approach of starting with real-world user flows. “Automating user feedback into the CI/CD process allows us to generate tests from the actual usage of our product in production and then run these tests against new code in our staging environment,” explained McGurl. By starting with the customer experience,“Relicx makes continuous deployment possible.” Electric experienced a renewed sense of confidence in the effectiveness of their DevOps program.

Once Relicx was deployed, the team was able to innovate faster and make changes to their code in ways that hadn’t before been possible. Relicx improved Electric’s operations in four key areas:

• Continuously test in CI/CD pipeline. The tight integration between Relicx and Electric’s CI/CD tools makes continuous deployment a reality. McGurl walked through their new deployment workflow: “Relicx sets up a suite of user-defined tests, automatically runs these tests after every merge, triggers a webhook after successful validation, which then automates deployment to production. Our release process is automated end-to-end.”
• Fix a bug once, fix it forever. When an Electric developer finds a customer issue, he can save that user session as a test. “Now I know that the bug that was fixed will stay fixed”, noted McGurl.
• Easy to reproduce and root-cause front-end issues. “When our team or the support team can’t reproduce an issue internally, we can rewatch the session from the user’s point of view and reproduce the exact steps a user took,” explained Viss. Then the team uses the developer console to root cause the issue by pinpointing the error in the relevant stack traces and logs. “It’s a big time saver to have the replay, user steps, logs, and network waterfall all in sync,” concluded Viss.
• No more “what will happen when we deploy?” McGurl and Viss agreed that one of the most powerful impacts of deploying Relicx has been their confidence to make bigger changes pre-production than before. As an example, during their evaluation of Relicx, the Electric team moved one of their backend services from Heroku to AWS. But with Relicx’ real-world test coverage and risk scoring, the team knew they could fully test this platform change in staging to ensure a smooth migration. “Our users never knew,” noted McGurl.

CX intelligence for the whole company

In summary, McGurl and Viss explained how Relicx has changed the way they communicate with management and partner with other teams in their organization. “We have a data-driven way to communicate to management our confidence in the quality of our next release,” concluded McGurl. “We spend less time building test scripts and fixing bugs and more time developing new features that improve our customer’s experience. That’s the best outcome for our users and our company.”