As businesses shuttered and others reinvented, those operating at the centre of critical services supporting New Zealand’s response to the global coronavirus pandemic scaled up at lightning speed.
Whakarongorau Aotearoa (formerly known as Homecare Medical) is a central cog in New Zealand’s public health information and support engine – perhaps best known for its team of registered nurses, paramedics, and health advisors behind the country’s critical telehealth service Healthline.
In pre-Covid-19 times, the organisation’s call centres handled around 7,000 calls a day. In March 2020, when the World Health Organisation (WHO) declared a global pandemic, that number exploded over the following weeks to 166,000 – an increase of 2,000 plus percent.
In the preceding weeks, when the tidal wave of demand for services became evident, Whakarongorau Aotearoa scaled up operations, increasing headcount from approximately 450 to 1,100 employees – many new to the health industry – spread over multiple contact centres and at-home locations.
Spotlight turns to rapidly scalable workplace applications
The expected flood of new employees requiring immediate access to the provider’s customer management system demanded a swift response. Public cloud was an obvious answer, though still left questions about the right mechanism to securely deliver applications to Whakarongorau Aotearoa’s staff everywhere they worked.
With partner CCL, the healthcare provider settled on Azure Virtual Desktop (AVD, formerly known as Windows Virtual Desktop, or WVD) – a virtual desktop infrastructure service, delivered from Microsoft Azure, for remotely accessing client desktops and applications.
However, while running a virtual desktop solution from Microsoft Azure was expected to lighten the admin load, a mountain of work was required to design and deliver a finished product ready to take life as a “gold stamped” desktop image available to employees.
With the clock ticking, a small team of CCL engineers and support staff swung into action, delivering what was needed in just 24 hours.
Anatomy of 24-hour delivery
New Zealand was on edge. The country’s first case of Covid-19 was reported on 28 February, followed two weeks later by the WHO’s declaration of an official pandemic.
The following is a blow-by-blow account of CCL’s response to Whakarongorau Aotearoa’s urgent requirement for a virtual desktop solution, starting with a lunchtime meeting at CCL’s Auckland office, on Thursday 19 March 2020 – two days out from the government’s introduction of a four-level alert system to combat the pandemic.
12 noon, Thursday 19 March
CCL’s client team – a blend of suits and a project manager – convened a virtual meeting at CCL’s office in College Hill, Auckland. An earlier discussion with client Whakarongorau Aotearoa directed play to Microsoft Azure. The healthcare provider was bedding in approximately 200 new contact centre agents to deliver non-clinical assessments. Just how quickly could CCL deliver a virtual desktop solution primed for a “golden” desktop image and rapid deployment to Whakarongorau Aotearoa’s ballooning workforce?
Over the next three hours a one-page plan took shape, covering key tasks, technical resources, and central elements of the solution. Key among these included a VPN running between Microsoft Azure and the provider’s core applications hosted in CCL’s Albany data centre, domain controllers, Active Directory Servers, security, and a ‘blank’ virtual desktop image ready for population and deployment.
CCL project manager Jamie Thomas briefs the delivery team – a collection of cloud, Windows, and network engineers, and a solution architect. Work starts on building a VPN gateway – a necessary precursor to establish access to Active Directory and domain controllers. The gateway fails and is rebuilt over the early evening.
VPN endpoints between Microsoft Azure and CCL’s data centre are up and running.
Work starts on the domain controller – the server controlling access to user accounts and group policies within Active Directory services. The VPN, now in place, smooths the way for the replication of Whakarongorau Aotearoa user accounts to its Microsoft Azure tenancy.
Domain controller and active directory server setup is completed. Attention turns to configuring the server controlling virtual desktop deployment. Automation is the name of the game, so when Whakarongorau Aotearoa employees log in, the system automatically spins up new servers to deliver a desktop image, rather than an admin having to do the job manually. Project manager Thomas heads home.
3:00 am, Friday 20 March
Firewalls and security layers for outbound network connections, traffic filtering, and access, are set and deployed. Thomas catches some sleep.
Deployment server configuration is completed. However, a network problem is detected, and additional work is required to map multiple IP address ranges in the provider’s local environment to its virtual environment in Microsoft Azure.
CCL delivers blank desktop image ready for loading applications to create a master image.
Whakarongorau Aotearoa infrastructure engineers create “gold stamped” desktop image and conduct final testing.
The first five virtual desktops are up and running, with the number increasing as new employees are onboarded and training commences.
By Thursday 26 March – the first official day of level four lockdown – 350 Whakarongorau Aotearoa staff and partners are working on a virtual desktop.
“CCL and Spark just got on with it. When the situation is dire, the only way to help is to make something happen.”Mike Mulvaney, IT Manager, Whakarongorau Aotearoa
In the hot seat: Mike Mulvaney, IT Manager, Whakarongorau Aotearoa
“The result was pleasingly successful,” Mulvaney said. “The effort from our internal engineers and CCL that went into standing up the new platform was massive – 24-36 hours solid. It saved our ability to take calls from the public.”
He said that under normal conditions a project of this nature would follow a six-month schedule to manage solution evaluation, architecture design, building, testing, and phased rollout.
While his CEO is excited by what IT can deliver within a massively condensed timeframe, Mulvaney is careful to temper expectations about the speed of future deployments. “I’m an architecture guy,
so I’m careful to do things properly.
There are still a few things to tidy up post implementation – and that was always going to be the case. I think you have to find a balance,” he said.
Looking back, Mulvaney remembers the stress and fatigue wrought by working at such a frenetic pace. “We were burnt out by the end, but we did what we had to do. Our partners made it possible. CCL and Spark just got on with it, realising what the reality was and got it done. When the situation is dire, the only way to help is to make
Designed and delivered by CCL, Whakarongorau Aotearoa’s AVD solution addressed the headache of rapidly scaling its customer management platform to support a ballooning workforce. The virtual desktop
infrastructure service, delivered from Microsoft Azure, presents the health provider’s application within a virtual
- Whakarongorau Aotearoa infrastructure engineers manage desktop images, profiles, and VMs in a single pane of glass admin portal in three clicks or less
- Server automation fires up new VMs to support additional users, as they log in
- The health provider’s AVD environment is protected by the same security blanketing as Microsoft Azure. Corporate data is transferred via Microsoft Azure ExpressRoute, which replaced a VPN to provide a faster, more reliable connection to reduce latency between AVD and other services, including CIC softphone, hosted in a local data centre. Data is rendered on screen for employees rather than arriving at the endpoint, reducing requirements for endpoint protection
- ‘Idle’ servers provide great business continuity, sparking into life as required
- CCL monitors the application and underlying Azure infrastructure to deliver a slick user experience.
Turbulent times, unprecedented response
Under normal circumstances, the design and deployment of an Azure Virtual Desktop solution can take weeks, and sometimes even months as the project progresses through staged development, reviews, and approvals.
But like vaccine development in the heat of a global pandemic, the process can be massively accelerated with extra resourcing and a work programme variously described as “building the plane as you fly”. The process isn’t perfect, but it’s acceptable – if the plane remains airborne.
“We’ve improved our security posture and driven a mind-shift to embed security as a normal part of development work and cycles – not as an afterthought,” said Anderson. “The business is now much more confident about our technology. We hear a lot in the media about security attacks and their consequences – it’s a monstrous evil world out there – so our approaches to the public cloud are always under the spotlight. DevSecOps has changed our IT landscape and ensures the business trusts the technology and remains confident about its ability to function securely.”