tdp_org,
@tdp_org@mastodon.social avatar

Hands up if you caused a global outage today...
Just me?
Sorry!

I was making a change to our "outside the UK" CDN config today for www.bbc.co.uk & www.bbc.com & the change included 2 bugs which pre-testing didn't spot:

  • A regex typo which caused 404s on www.bbc.co.uk
  • An incorrect TLS cert on the CDN origin which caused 503s on www.bbc.com

These caused ~7 minutes of significant global outage.

I spent most of the afternoon writing tests to catch this for next time.

Cache status graph for www.bbc.com which shows a significant spike of "error" for ~7 minutes

Whiskeyomega,
@Whiskeyomega@cupoftea.social avatar

@tdp_org Reminds me of the time Comedy Dave took Radio 1 off the air for 45mins

amyfou,
@amyfou@lingo.lol avatar

deleted_by_author

    •
    tdp_org,
    @tdp_org@mastodon.social avatar

    @amyfou 🤣
    Yep, basiclally that - was all set for a smooth day...notsomuch

    mbpaz,
    @mbpaz@mas.to avatar

    @tdp_org thank you for being open and honest. Errors do happen.

    tdp_org,
    @tdp_org@mastodon.social avatar

    @mbpaz They do indeed, cheers. I always believe in being honest and sharing mistakes, even the bigger mistakes.🙌🏻

    GossiTheDog,
    @GossiTheDog@cyberplace.social avatar

    deleted_by_author

    •
    tdp_org,
    @tdp_org@mastodon.social avatar

    @GossiTheDog In theory it wouldn't have affected you in the UK but sorry if it did...one of those "why's everything on the main monitoring dashboard failing?...ohhhhh" moments 🤣
    Hopefully the new test suite means it won't happen again.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • devops
  • InstantRegret
  • DreamBathrooms
  • ngwrru68w68
  • osvaldo12
  • cubers
  • magazineikmin
  • ethstaker
  • Youngstown
  • rosin
  • slotface
  • everett
  • kavyap
  • Durango
  • khanakhh
  • megavids
  • thenastyranch
  • anitta
  • modclub
  • GTA5RPClips
  • mdbf
  • cisconetworking
  • tester
  • tacticalgear
  • provamag3
  • Leos
  • normalnudes
  • JUstTest
  • lostlight
  • All magazines
    •