The coronavirus pandemic has been a very difficult time for everyone. Many businesses have struggled immensely or had to close altogether. We’ve been incredibly fortunate: because of what we do, we’ve seen huge company growth, but we weren’t able to scale up quickly enough to maintain reliability and customer service, which caused a lot of problems for merchants who needed to quickly react to what was happening.
And for that, I’m deeply sorry.
I’m writing this article now because we’re in a drastically better place and have been able to reflect on the past few months. I want to talk about what happened, what we’ve done about it, what we will do, and of course, to thank you for sticking with us.
What happened
As lockdowns everywhere kicked in, demand for our apps exploded. We hit our year’s projected growth in 10 days and it just didn’t stop. Normally that would be cause for celebration but given why it was happening, we just wanted to do a good job and help as many people as we could.
But keeping up with demand wasn’t easy: we weren’t able to scale up our infrastructure and team at the same rate as stores were installing the apps.
On top of that, adding pickup and delivery isn’t something you’d normally rush into. Getting the online/offline interaction working smoothly takes planning and thought, but that wasn’t an option: businesses needed to get going immediately. Many merchants were even using Shopify itself for the first time.
Very quickly, our customer support became overwhelmed. Ticket numbers were growing - literally - exponentially. Every ticket we solved was replaced by many more, and the backlog grew into the thousands. Our team were working crazy hours, we stopped offering calls and live chat to focus on email, and we brought in as many people as we could.
At the same time, the load on our servers grew dramatically. Not just from all the new stores, but all the new business online stores were doing as shopping moved away from the high street. We began to see outages at the busiest times of the week - including regular server crashes on Thursdays and Fridays after Masterchef aired.
Crucially, we didn’t have enough redundancy built into our systems. When we started having repeated problems - such as third-party services going down - there wasn’t an alternative system to pick up the slack. So things just broke and stayed broken, until we fixed it.
That led to a nightmare situation: our apps were breaking, and merchants couldn’t get the help they needed. I feel terrible about that.
As I said, pickup and delivery is full of complexities. There are thousands of Shopify themes, endless types of products, and no two bricks-and-mortar businesses that are identical. As a result, part of our ‘thing’ has always been a very personal and flexible approach - we gladly do manual installations, customise the app for new use cases, and spend a lot of time with stores that need extra help to get things configured in a way that suits their set up.
But in the face of the pandemic, that model and approach stopped working, and the results were predictable.
What we did
Since the crisis began, we’ve worked incredibly hard to solve the problems we experienced and set ourselves up for the future. I believe we’re now in the strongest position we’ve ever been. Here’s some of the most important things we’ve done:
Customer support:
- Since the start of the crisis, we’ve scaled up our support team by 300%. Of course there’s a lag - new team members needed time to get trained and become familiar with the work. But I’m really pleased with the people who’ve come on board. We’ve managed to keep the same amazing passionate culture and ethos.
Redundancies and reliability:
- We’ve rebuilt large parts of the apps to include multiple redundancies so that failures, server crashes and third-party outages can be dealt with quickly and automatically.
Orders:
- If tags fail to add or notifications fail to send, the system will now try again multiple times.
- We now have multiple order processing servers and duplicated databases. As well as dealing with outages, that also means we can easily carry out maintenance on the primary systems without any disruption.
Rates:
- The Rates engine has been completely rebuilt, and is now hosted on multiple high-availability servers with managed support.
- Redeveloped the ZapietId to encrypt customer checkout preferences, eliminating the need for additional requests to our rates servers. This improves stability and speeds up checkout. It also improves compatibility with third party apps, making future integrations easier.
Locations and Calendar APIs:
- The Locations and Calendar APIs have been fully rebuilt, with greatly improved efficiency thanks to better caching.
- The APIs have been decoupled from our legacy hosting provided and moved to high availability serverless architecture.
Tests:
- Test, tests, and more tests
- Manual testing has been complemented with vastly improved automated testing suites to drastically reduce the risk of code errors during deployment.
Status page:
- An improved status page gives merchants more ways to stay up to date with performance and issues, and act more quickly in response.
Help centre updates:
- We’ve continued to add new articles and update existing ones.
- We created 31 new video tutorials that are helping to make articles more accessible for those who prefer a visual approach.
- Our video tutorials have been viewed more than 8,000 times in just a few months.
Languages:
- We’ve invested in support for French and Spanish.
What's next
With the worst behind us, we’re going to continue to invest in improving our infrastructure, reliability and team. To complement the already much-enlarged support team, we’re going to expand the engineering team as well. That will give us the capacity we need to develop better infrastructure and solve bugs, at the same time as improving the feature-set.
We’re also planning to introduce free professional onboarding for new stores. That will greatly improve the experience of getting started and help merchants make sense of our apps’ extensive customisability. To help our global user base, we’re also planning to invest in translations of our help centre, website and app interface.
Thank you for sticking with us
Once again, I’m deeply sorry that at a critical time, we didn’t manage to deliver our usual standards to our merchants. I’m confident that everything we’ve done in the past few months has brought us to where we need to be and has created a strong platform for the future. I’m confident enough that from now on, we’ll gladly grant immediate, no-questions-asked refunds to any store reporting performance or stability issues.
With all the development work in recent months, I haven’t spent as much time as I normally do engaging directly with our merchants, and I want to do more of that. Our fantastic support team is our first port of call, but you can feel free to reach me directly at andy@zapiet.com.
Thanks again, and I hope you, your team, and families are staying safe.
Andy,
Founder of Zapiet
