Rolling out software products and major patches smoothly is super challenging but you can reduce the stress on eht whole team and your customers by taking some proactive, reactive and deliberative actions.
Proactive —
The whole team needs to conduct regular pre-mortems in which the whole team imagines that the release has gone horribly wrong and lists out the causes. The meeting needs to include product manager, QA, support, engineers, documentation team to get a diverse perspective. It would be great to include customers when possible.
The software engineers on the team need to give everyone a headstart by following good coding standards key among them would be exception handling, logging diagnostic information and test-driven-development. This will be tremendously useful in diagnosing and regression testing in the rare case that errors are found after deployment (laughs sarcastically). The engineers also need to work with QA and senior management to set release criteria for the software and patch families. QA needs to work with documentation team to publish FAQs, installation readme and user guides.
Whenever possible, rollout cautiously to customers who have a smaller user base and fewer mission critical transactions. Select complementary customers to cover all of the functionality.
Reactive — There are no perfect rollouts. When issues occur, the first, second and third level support teams can prepare themselves with a runbook that helps them intake, diagnose and resolve problems. IF they do have to escalate issues to engineering then they need to provide as much relevant diagnostic informaiton as possible.
Deliberative — The team needs to have a post-mortem after resolving major issues. These meetings should include representatives from all levels of support, engineering and QA. The notes should be fed into the FAQ, user guides and any other repository which can help prevent recurrence of the issue. Always celebrate small vicstories. If you’ve gone 2 weeks without at least half of the team celebrating then its too long. Sustaining energy, motivation and joy is the leader’s job.
Metrics —
- Customer satisfaction.
- Number of issues handled by support.
- Number of escalations to engineering.
- Number of recurrent issues.
Rolling out new software products, major patches and functionality is challenging as well as rewarding. I hope some of these ideas can help you and your team.
Good luck.
