Bottom Line Up Front

  • The issue with our University Events (Localist) plugin has been resolved. 
  • A short-term fix has been implemented and is in production. Your site should now be fully functional. 
  • A long-term fix has been identified for development in January. 
  • You can view the full incident report here or stay up-to-date with our latest updates here.

More Information Than You Need:
On Friday afternoon, Localist encountered a Denial of Service attack where increased levels of bot traffic were hitting their servers. They noticed a WFU IP (34.192.207.124) requesting https://events.wfu.edu/api/2/events/filters?sort=name multiple times a second and blocked our servers as part of our immediate response. Once the culprit was found, our servers were unblocked.

During the time that our servers were blocked, however, Localist returned an HTTP exception to our WordPress instances which caused a fatal error. These errors began to pile up and eventually crashed the Apache instances as the requests to Localist began to hang. 


Resolution
Hotfix
Remove the Localist plugin from the WordPress installations.

Short Term Resolution
Change the way we are making calls in the Localist plugin to account for external services returning fatal errors.

Long Term Resolution
Rewrite wfu/wfu Guzzle package to account for a failure of any external service. 

Categories: Incident Report

Archives