tdp_org,
@tdp_org@mastodon.social avatar

Hands up if you caused a global outage today...
Just me?
Sorry!

I was making a change to our "outside the UK" CDN config today for www.bbc.co.uk & www.bbc.com & the change included 2 bugs which pre-testing didn't spot:

  • A regex typo which caused 404s on www.bbc.co.uk
  • An incorrect TLS cert on the CDN origin which caused 503s on www.bbc.com

These caused ~7 minutes of significant global outage.

I spent most of the afternoon writing tests to catch this for next time.

Cache status graph for www.bbc.com which shows a significant spike of "error" for ~7 minutes

  • All
  • Subscribed
  • Moderated
  • Favorites
  • devops
  • GTA5RPClips
  • DreamBathrooms
  • InstantRegret
  • ethstaker
  • magazineikmin
  • Youngstown
  • thenastyranch
  • mdbf
  • slotface
  • rosin
  • modclub
  • kavyap
  • cisconetworking
  • osvaldo12
  • JUstTest
  • khanakhh
  • cubers
  • Durango
  • everett
  • ngwrru68w68
  • tester
  • normalnudes
  • tacticalgear
  • anitta
  • megavids
  • Leos
  • provamag3
  • lostlight
  • All magazines