Customers expect the services they use to work, even during a hurricane, pandemic, or war. The last nine months have dramatically changed the approach to ensuring business operations in Ukraine. Companies have been passing through experiences never known before. The disaster plan and its implementation became the real thing.

Vira Tkachenko, CTO at MacPaw, spoke to Apple admins at Jamf’s JNUC event to explain how the company ensured security and stability during the russian invasion of Ukraine.

Any business leader would benefit from this insightful crisis management and planning lesson. So here are some of the points Vira raised throughout her presentation.

Disaster planning

MacPaw started disaster planning around two months before the war. Local and western media were full of news about russia’s possible open invasion of Ukraine. We’ve seen satellite images of military vehicles and troops aggregating along the borders. Of course, we hoped it would be impossible to start a war in a European country in 2022, but we decided to start preparation.

The stages of handling the emergency are the same as the actual stages of war from the business perspective we witness now. So we’ve started with the zero stage – planning and assessing the risks.

Our priority #0 was the physical security of the team. Priority #1 was the stability and security of our services and products. We’ve started assessing possible risks in various areas to compose a plan.

Possible risks

We have prepared the following list of possible risks:

  • Loss of internet connection
  • Occupation of office by invaders, making access impossible and data insecure
  • Cyberattacks against the company and its services because of the public position of MacPaw
  • An increase in the frequency of phishing attacks against employees
  • Attacks on corporate social media
  • Unauthorized access (using captured or lost devices, for example)
  • Hardware supply chain disruptions, including interruptions in equipment supply and the provision of logistics, including power and transportation
  • Potential disruption due to sanctions and war zone company status.

Each risk had a procedure to mitigate it. But planning is crucial because people in stressful conditions can’t do something complicated. So the Executive Team developed a simple Google spreadsheet with risks. The most critical columns included an asset, primary and backup procedures, and a responsible person.

Actions planned

Emergency team. We formed a dedicated team of representatives from each product and service team. Their main job was to ensure their stability and security. We asked them to move out of Ukraine or to the western part of the country to stay in a safe area.

Code freeze. We prepared a code freeze regime for our products and infrastructure. Only people from the emergency team were allowed to make or approve changes and assess if they were safe.

Remote work. “Thanks” to COVID times, we were ready for remote work and didn’t have to plan any activities.

Alternative communication channel. We use Slack as the primary work communicator. Telegram messenger is popular among our employees as a personal communicator. However, we’ve also added Signal to have more redundancy and privacy. We asked all employees to install the app and connect with their communication lead.

Completely move office infrastructure to the cloud. We don’t have lots of it in our office. We were using the Mac mini as a build agent for the Bamboo CI/CD system and as a test device for QA people. So the war sped us up in this movement. We’ve switched to MacStadium + Azure Pipelines and Anka. Switch to cloud VPN - we’ve also hosted a VPN in the office and decided to move to the Pritunl solution. Our engineers were happy as it’s more configurable.

Buy laptops in advance and set up Satellite internet to cover the risk of internet loss.

Stage 1: The war outbreak

MacPaw experienced the low performance of a team for the first month because of safety risks and shocking news we’ve heard. Day one was special for us; we had to activate the risk mitigation plan: code freeze, notify the team, and hold executive and emergency team meetings.

As many team members moved worldwide, it wasn’t easy to understand if everyone was safe. So our engineers created a Slack app called ‘Together.’ It has one simple function – asking people their location daily and if they can work or volunteer.

We were surprised to find no internet access disruption, except in occupied territories and war zones. Moreover, Elon Musk provided Starlink – a cheap, fast, low latency solution for internet connection.

Hardware supply chains were distracted, but pre-bought laptop storage helped us immensely. Our system administrators sometimes even had to deliver laptops by their personal cars. We’ve also gained new experience in buying laptops not in Ukraine but in Apple Stores in other countries.

Stage 2: Used to the state of war

The final stage is the new normal, a new reality. We got used to the state of war and our IT and security processes adapted.

MacPaw continues to deliver from Kyiv. The emergency team is dissolved. We’ve also reduced the frequency of check-ins to once weekly because most people are in a safe place.

However, around ten of our teammates serve in the military. Some as soldiers of cyber war, some – at the frontline.

MacPaw prepared a procedure to handle mobilized employees according to three groups by risk level:

  • Safe (there are no active military activities in the region)
  • Potentially not safe
  • Not safe (frontline and defense lines; active military activities are in the team member’s location).

The new twist – blackout

Today, there are new challenges for business after russia’s massive attack on Ukrainian energetical infrastructure. Every day people in Ukraine stay without electricity for up to 10 hours per day. So, MacPaw has to adapt to new conditions.

As more teammates now come to the office because they don’t have an electricity supply at home, a few steps we’ve taken to ensure the company’s work during blackouts:

  • ensured two separate power supply lines
  • bought a generator for a backup power supply
  • prepared two separate internet access lines and uninterrupted internet access point (Starlink) with additional batteries
  • turned off all electrical appliances not required for use in the office
  • replaced the power supply in the security systems so that the office’s security is not affected even in a long-term blackout.

Lessons learned

Lesson 1: Do not neglect planning

When you plan, you’re not only more ready with an action plan, but you are more ready emotionally.

Lesson 2: You can’t understand an emergency till it happens

Some risks we anticipated didn’t happen, and some - were unexpected. Our expectations were based on movies, articles, and analytics. But the reality is different, so you should accept it and move forward.

Lesson 3: Write your emergency procedures in a way to follow instructions very easily

You will not be in a stable emotional condition to do complicated things.

Lesson 4: Changes are swift

Be ready to change plans and make quick decisions.

We urge you to watch Vira's entire speech for more insights, inspiration, and practical advice: