Reflections on May 3rd Give Local America Day
- May 10, 2016
- giving platform
For those of us who have been fortunate to have worked on a successful Gives Day, nothing beats the excitement and enthusiasm generated by the overwhelming generosity of donors supporting nonprofit organizations in their community. The spirit of philanthropy is contagious and the resulting funding for nonprofits heartwarming. A culture of giving is nurtured and communities are strengthened.
All of us work toward this goal and when all does not go as planned the disappointment is as great as the excitement when things do go well. Most of us have experienced some type of technology failure in our life; it is inevitable. In all sectors of our economy we see technology failures – in banking, ticketing, news, space, and commerce. The biggest of players and the smallest of players have all fallen victim to technology failure. However, some recover better than others.
In building systems to support Gives Days, we collectively need to continue to test our systems to avoid failures, but just as importantly, we need to build architectures that support recovery from failures. Over the course of the past eight years, hosting some of the country’s largest Gives Days and processing over $200 million dollars in donations, CiviCore has experienced our fair share of technology issues. We have learned that despite all we do to avoid these issues, they are bound to happen. By accepting the inevitability of technology failures, we have focused our thinking toward mitigating the impact of potential failures in addition to avoiding them. If designed accordingly, in most cases, technology issues can be addressed by seamlessly rolling over to redundant hardware in a way that doesn’t impact users of the system. In the worst case, the issues require several minutes of performance degradation before one of the backup options can be implemented in order to solve the issue.
While preparing for a Gives Day, in addition to spending time on preventing the failures through pre-Gives Day testing, it is also important that significant time be spent building out options to mitigate the impact of potential technology failures. Although it isn’t possible to guarantee that there won’t be any technology failures, it is possible to guarantee several layers of backup options in response to a failure.
Specifically, we have learned the importance of the following:
- Designing an architecture whereby servers can be gracefully decommissioned so that if a server fails it can be seamlessly replaced as part of a cluster of servers,
- Building an architecture that is scalable so that new servers can be added to the cluster in order to quickly address performance issues or spikes in demand,
- Segmenting Gives Days on multiple technical architectures so that an issue with one Gives Day can be isolated and not impact other Gives Days,
- Mirroring our architecture at a data center in another part of the country so that if there is a problem with one data center we can redirect traffic to another data center, and
- Maintaining a completely independent code base to address worse case scenarios with a code base issue that cannot be quickly remedied.
The Give Local America event held on May 3rd was a difficult day to observe as the technology failures experienced by one of the providers of Gives Day technology impacted Give Local events around the country as described here and here. Our hearts went out to all of the people working at foundations, nonprofits and technology companies who had dedicated so much to make the day a success.
Collectively, we know we can continue to build on the great momentum for philanthropy that Gives Days have created and continue to develop better systems and approaches to not only preventing failures but also being able to respond to them when they do occur.