System Outage Feb 28th, 2023

[United Resource Connection]

Tuesday, February 28th, we experienced a system outage as a result of an automatic upgrade that had not been fully vetted for implementation on our system. While the main site located at unitedresourceconnection.org was not effected, all branches of united resource connection, including our large Cincinnati branch, were down for several hours.

All of our branches have a similar code structure, and the coding language used to implement that structure was updated, just prior to the crash. The updated language abandoned several code statements, which need to be now stated in different ways. Because we had not changed the way that we coded our structure prior to the language upgrade, those structures were not understood after the update, resulting in the structural failure experienced. This is the simplest way to describe it. For interested IT professionals, our server auto-upgraded to php8, while and our highly customized branches of URC had some depreciated code incompatible with the upgrade. We were able to quickly identify which code needed to be removed for the branches to be able to come back online.

All future upgrades of the coding language will be vetted in staging environment prior to implementation.Our team is currently defining a “lessons learned” outline to use in development of a best practice standard for future updates. If you have any questions about our best practice standards, let us know, we are happy to share our knowledge.

[Author]

Posts