Never Waste A Good Crisis

While listening to a recent Podcast featuring one of my colleagues, a former Enterprise IT executive, she mentioned the phrase “never waste a good crisis”.  The last time that I had heard that phase was during the 2008 financial crisis.

“You never want a serious crisis to go to waste. And what I mean by that is an opportunity to do things that you think you could not do before.”

― Rahm Emanuel

The Crisis

Over the last 16 months I’ve listened to a similar story from IT leaders all over the globe across almost every industry vertical who sent employees home on a Friday only to have to figure out on Monday how to get them access to the Enterprise applications they needed to do their job. 

The seemingly ‘easy’ route was to lean on traditional remote access VPN. However, that VPN hardware was originally scoped to support ~35-40% of an Enterprise’s employees working remotely at any given time. Those VPN concentrators became immediately saturated and it was nearly impossible to quickly instantiate additional capacity in a traditional appliance based model. In order to attempt to alleviate some of the capacity strain and end user performance impact of being backhauled to the corporate WAN via VPN, Enterprises were forced to implement split-tunneling policies allowing for higher bandwidth and performance impacted applications like O365, Zoom, WebEx and Teams to be split off from the VPN and go direct to Internet creating risk via a security and visibility gap

The Opportunity

While some forward thinking IT leaders had envisioned and already shifted employee remote access away from a traditional VPN paradigm to an elastically scalable cloud delivered zero trust architecture, others had been considering for some time how they might embrace more of a zero trust paradigm and kick their users off the corporate network. Well, here is where the opportunity emerged amid crisis….COVID-19 sent almost everyone home and kicked the users off the corporate network. The conundrum facing IT leaders was now no longer how do I go about kicking the users off the corporate network, but how do I go about securely providing them just enough access to a limited set of things that they need in order to do their jobs.

Tackling Legacy Technical Debt

Some legacy enterprise IT applications that were designed around a fundamental principal of having direct network access to user endpoints.

In response to this challenge many Enterprises went about investigating how to either adapt the way these legacy network application communication flows worked from a push model (server-to-client) to a pull model (client-to-server) or abandon on-prem hosted entirely in favor of adopting cloud delivered instances of the same function that would address providing these functions to off-network users without having to bring a user back onto the internal private network. In my many discussions with global Enterprises I had the benefit of learning what their IT teams had done to adjust to this crisis and below I list some typical examples of migrations away from traditional on-prem hosted server to client communication models to client to server and remote work friendly cloud hosted models.

It is very important to note that this is not a recommendation or endorsement on my part of any of the specific products below, simply just a recap of what I have seen come up in conversations with prospects and existing customers.

Patch Management – Pulling down software updates over traditional VPNs leads to several problems. First, if there is a large number of off-net users attempting to pull patches that can put significant strain on your existing VPN concentrators and internal network bandwidth. Secondly, if the patch management system is only available to remote users over VPN then they need to know to turn on the VPN in order for the patch update to even happen which can lead to lag time where some systems remain unpatched and vulnerable. Instead of forcing users to come back onto the corporate network over VPN for patch management, some enterprises moved to completely cloud hosted (SaaS) implementations of patch management (cloud management gateways) that allowed users access to updates directly over the public internet without care or concern around internal private access or whether their VPN was turned on. Others who already had replaced their traditional VPN solution with a Zero Trust no network access offering simply flipped their patch management to leverage a pull model whereby the client device would ‘check-in’ at regular intervals looking for updates. In either event both approaches result in preventing any significant patch update lag that would otherwise lead to unpatched systems and increased risk.

Remote Desktop Support Management – When users need support personnel to access their machine in order to help with troubleshooting an issue it was not uncommon to leverage things like Remote Desktop from the support engineer’s PC to the end user’s endpoint. This type of connectivity model clearly only works when there is direct network connectivity from IT support to the afflicted end users machine. Several implementations exist which allow for “meet in the middle’ type of remote support access where a cloud delivered solution can securely enable IT support personnel to remote access an end user device regardless of what network it resides on.  While clearly not an exhaustive list, some popular examples I have heard mentioned are Microsoft Quick Assist, BeyondTrust Remote Support (formerly Bomgar), ConnectWise and TeamViewer.

Legacy VOIP – For VOIP Softphone usage and for Call Center VOIP implementations it was common to see offerings like Avaya and Cisco Call Manager that leveraged traditional direct network access in order to function properly. What was pretty common across all enterprises was that the end user base dependent on these systems represented a small percentage of the total overall end user count.  Some enterprises simply maintained a much smaller deployment of traditional remote access VPN technology to address this user base while adjusting their IT planning and budgets towards future deployments of UCaaS (Unified Communications As A Service) implementations of these types of systems with the eventual goal being to retire traditional remote access VPN entirely. Others expedited existing plans and completely migrated users toward UCaaS implementations like Teams, WebEx and Zoom. For Call Center applications some enterprises are looking to adopt CCaaS (Contact Center As a Service) solutions like Genesys Cloud, Amazon Connect Contact Center, Five9 or Nice’s InContact.

Vulnerability Scanning – Having to have a user’s device brought onto the corporate network in order to scan it for vulnerabilities runs counter to the goal of a zero trust model. If we do vulnerability scanning because we don’t trust that the end user’s device isn’t compromised then why would we want to risk bringing a potentially compromised endpoint onto the network where that compromise can potentially spread laterally? If we look at what happened with the SolarWinds supply chain compromise as an example, then its probably not the best of ideas to give a tool complete unfettered access to everything on the entire internal private network. Having a good modern EDR tool like CrowdStrike, CarbonBlack, SentinelOne or Windows Defender combined with an agent providing asset vulnerability data that can phone home to the cloud from an off-network remote user device runs more inline with the goals of a zero trust model.

In summary, those Enterprises that had seized the opportunity to accelerate shifting away from legacy network-centric applications like the ones covered above towards cloud delivered models didn’t need to bring their end users back onto the corporate network. They were able to migrate away from traditional remote access VPN model to a more identity and context based least privilege access model where users only get access to the applications the need to do their job, not access to the internal private network. By ‘keeping the users kicked off the network’ not only are these organizations inherently more secure, but they can now start to evaluate further security and network transformation starting with “if my users can do their jobs off of the internal corporate network, then do we even need to operate a traditional internal private WAN anymore?”.

Curious to hear thoughts and experiences from others on the challenges they faced in addressing the shift to remote work. Feel free to leave a comment !

Disclaimer: The views expressed here are my own and do not necessarily reflect the views of my employer Zscaler, Inc.