Balancing Speed and Safety: Portnox CEO on Lessons from the CrowdStrike Incident – VMblog QA

This article was originally posted on WMBlog.

In the wake of the recent CrowdStrike incident that sent shockwaves through the cybersecurity industry, VMblog sat down with Denny LeCompte, CEO of Portnox, to discuss the delicate balance between rapid security updates and potential risks.

LeCompte shares valuable insights on how cybersecurity companies can maintain customer trust while ensuring robust protection. From implementing “escape hatches” in software deployments to adapting quality assurance procedures, this interview delves into the critical strategies that can help prevent and mitigate the impact of security software failures. As the industry grapples with the fallout from high-profile incidents, LeCompte’s perspective offers a roadmap for rebuilding customer confidence and improving transparency in the ever-evolving landscape of cybersecurity.

VMblog: The CrowdStrike incident has highlighted the potential for widespread disruption from a single software update. How can companies like Portnox balance the need for rapid security updates with the risks of pushing untested changes to all customers simultaneously?

Denny LeCompte:  The CrowdStrike incident underscores a critical challenge in the cybersecurity industry: the delicate balance between the urgency of deploying security updates and the potential risks that come with them. At Portnox, we recognize that the stakes are high when it comes to protecting our customers’ networks and data. That’s why our approach is twofold: robust testing protocols and customer-focused flexibility.

While speed is essential, it’s equally important to ensure that updates undergo rigorous testing before they are released. We combine automated and manual testing of all Portnox releases.  Furthermore, for risky changes, we update a subset of customers to gauge impact in the real world before releasing to everyone.

The key to balancing rapid security updates with the risk of disruption lies in a disciplined, customer-centric approach. It’s about being fast, but also being thorough, and always keeping our customers’ needs at the forefront.

VMblog: You’ve mentioned the concept of an ‘escape hatch’ in software deployments. Can you elaborate on how Portnox implements this strategy, and what advice would you give to other cybersecurity firms looking to minimize the impact of potential bugs?

LeCompte:  The idea of an “escape hatch” is to have a plan in case things go wrong even after you’ve thoroughly tested your code.  Can you escape a catastrophic outcome by quickly reverting to a prior stable state while minimizing disruptions to security or operations?  If you only think about the “happy path,” you won’t be ready if things go wrong.  Software is built by humans, so mistakes will happen.

We work towards perfection at Portnox, but we also plan for disasters.  Whenever possible, we retain the option to roll back a bad change.  When that’s not possible, we update in phases and keep open the possibility of failing over to a part of the system that has not yet updated.

VMblog: The CrowdStrike incident has likely shaken customer confidence in automated security updates. How do you think this incident will impact the cybersecurity industry’s approach to software deployment, and what changes might we see in the near future?

LeCompte:  The CrowdStrike incident has certainly put a spotlight on the inherent risks of automated security updates, and it’s likely to prompt some significant introspection. Trust is the cornerstone of our business, and when an incident like this occurs, it can understandably shake customer confidence. Moving forward, we’llwe’ll see several key shifts in how the industry approaches software deployment.

First, customers will demand more visibility into what’s happening behind the scenes during updates. We can expect cybersecurity firms to be more transparent about their deployment processes, including what safeguards are in place and how issues will be addressed if they arise.   Furthermore, the industry will also likely double down on testing protocols, with an emphasis on real-world scenarios. There’s always been a push to get updates out quickly, but this incident will reinforce the need for exhaustive testing to catch potential problems before they reach customers.

In response to concerns about automated updates, cybersecurity companies may offer more customization and control to their customers. This could include the ability to schedule or defer updates, opt out of certain types of updates, or deploy updates in stages. Many vendors, including Portnox, already do this.  Giving customers more control over how and when updates are applied can help rebuild trust and ensure they feel secure in the process.

I believe the industry will shift towards a more cautious and customer-centric approach to software deployment. The emphasis will be on building and maintaining trust through better transparency, control, and reliability.

VMblog: Drawing from your past experiences with consequential bugs, what processes do you believe are crucial for cybersecurity companies to implement to prevent similar incidents? How has Portnox adapted its quality assurance procedures in light of these lessons?

LeCompte:  Having been in the industry long enough to have seen my share of consequential bugs, I can say with certainty that the key to preventing such incidents lies in a comprehensive, multi-layered approach to quality assurance (QA). No single process or tool will ever be enough on its own-effective QA requires a combination of best practices, rigorous testing, and a culture that prioritizes quality and security at every stage of development.

By deploying updates to a small group of users first, you can monitor for problems in a controlled environment and quickly address any problems before a broader release. This approach not only reduces the risk of widespread disruption but also allows for faster feedback loops and more responsive updates.

When a bug does slip through-because, let’s face it, no system is perfect-conducting a thorough, blameless post-mortem analysis is essential. Understanding what went wrong, why it happened, and how it can be prevented in the future is critical to continuous improvement. We make it a point to involve cross-functional teams in these reviews, ensuring that lessons learned are shared across Portnox and integrated into our processes moving forward. Keeping post-mortem reviews focused on preventing future problems and away from assigning blame for past mistakes is crucial to fostering a culture of transparency and continual improvement.

Never underestimate the value of direct customer feedback. Engaging with customers to understand their pain points and their experiences with your software can provide invaluable insights that might not be apparent through internal testing alone. We’ve made it a priority to maintain an open line of communication with our Portnox customers, allowing us to be more proactive in addressing potential issues before they become critical.

VMblog: The CrowdStrike incident has demonstrated the far-reaching consequences of a security software failure. How can the cybersecurity industry work to rebuild trust with customers and improve transparency after such events?

LeCompte:  The CrowdStrike incident has reminded us of the profound impact a security software failure can have-not just on the affected company, but on the entire cybersecurity industry. Trust is the foundation of our relationship with customers, and when that trust is shaken, it’s our responsibility to rebuild it with transparency, accountability, and a renewed commitment to excellence.

The first step in rebuilding trust is acknowledging the issue and taking full responsibility for it. Customers need to know that their concerns are being taken seriously and that the company is committed to rectifying the situation. This means being transparent about what went wrong, how it happened, and what is being done to prevent it from happening again.

Ultimately, actions speak louder than words, and one of the most effective ways to rebuild trust is by demonstrating a commitment to strengthening security practices. This includes conducting comprehensive security audits, enhancing testing protocols, and possibly engaging third-party experts to review and certify security measures.

Rebuilding trust also involves collaboration with customers to ensure their voices are heard in the process of improvement. Engaging with customers through forums, surveys, and direct feedback channels allows us to understand their concerns and priorities. We place a high value on customer feedback at Portnox, using it to guide our development and security practices. By involving customers in the conversation, we can work together to create a more secure and trustworthy product.