(Topic ID: 291292)

Back after a major outage

By robin

3 years ago

Topic Heartbeat

Topic Stats

  • 331 posts
  • 210 Pinsiders participating
  • Latest reply 3 years ago by boagman
  • Topic is favorited by 6 Pinsiders


Linked Games

No games have been linked to this topic.

    Topic Gallery

    View topic image gallery

    image (resized).jpg
    pasted_image (resized).png
    69DFCFDC-E12D-4C2D-8036-6C15CDB4799D (resized).png
    waterdatacenter (resized).png
    uOcsyp (resized).png
    B4EC046D-B13A-4FFE-8F23-3C245F7AE44F (resized).png
    umm-humm-when-keeping-it-real-goes-wrong (resized).jpg
    10b05741d4aa7afec9de5412a443548c (resized).jpeg
    49E5334B-EE51-4A76-B09B-8E99668037D7 (resized).jpeg
    pasted_image (resized).png
    6DB69A6E-3813-42D2-91AE-969F14A01757 (resized).jpeg
    pasted_image (resized).png
    There are 331 posts in this topic. You are on page 1 of 7.
    #1 3 years ago

    We're back!

    What a crazy week it has been for Pinside (and a stressful week for me, I might add) with our first major outage in the 20 years or so that Pinside exists. Sure, we've had hiccups but nothing like this. The cause: a power surge, followed by a fire in a generator, followed by sprinklers and our server going down. I wasn't there to see it, but I don't think it was pretty.

    Anyways, we're running on a temporary server now, hosted all the way over in Germany (well, that's close for me). Please be nice to it, it's not as powerful as our previous one and there may be problems. Also, not everything functions yet: We have no e-mail notifications yet. And no maps. And probably a bunch of other stuff that doesn't work right now. But at least we're back. I'll be moving to a brand new server (and a different hosting company) later on but I don't want to rush that choice.

    As for our data recovery procedure. Well there really wasn't one in place as you may have noticed. Sure I had dual power supplies, dual network uplinks, dual (raid) drives. But I would not have expected a datacenter fire. I should have. I should have had a backup server somewhere, ready to spin up and continue service after a few hours. Truth is, I never got around to setting this up. But I will obviously be working on this, so a downtime like this cannot happen again in the future.

    My last database backup was dated 10:02pm, while the server went down 11:39pm. So we lost 1h37m of data. That means any posts, PMs or Marketplace ads placed during that time are gone. And even if they restore our server (they're still working on it but are now giving me an ETA of "weeks") I will not be bringing that data back as that would be a technical nightmare.

    If you made a donation or marketplace payment, please contact me. I will also be proactively look into any payments made during that 1h37m and take action or contact you.

    Speaking of donations. Feel free to make a donation, if you're happy that we're back!

    Let's get the pinball talk going again and please bear with me while I try to keep this temporary server running nicely.

    -- Robin

    PS: Seeing any issues? Let me know!

    #2 3 years ago

    Thanks for all the hard work getting Pinside back up. I wasn't sure what to do with my free time the last few days. I tried moderating my wife, but we all know how that went.

    #3 3 years ago

    Thanks Robin!
    I was starting to suffer from withdrawal syndrome... Good to be back!

    #4 3 years ago

    AH-ha!!! I’ve been hammering the beta site refresh hoping it would come online first.

    God bless. I was having massive withdrawals. My almost 4 year streak (1318 days) on this website, crushed!!

    #5 3 years ago

    Well, you all didn't take long to crash our temporary server. After 30-40 minutes it went down smoking. So I went back to the drawing board and ordered a brand new server. This should work, but please bear with me as I will be tweaking and tuning some more over the next week(s). The upside is that the new server is twice as powerful as the previous one, so with the right approach this could all end better than it started.

    Anyways, if you're able to read this: welcome back!

    #6 3 years ago


    pasted_image (resized).pngpasted_image (resized).png
    #9 3 years ago

    Yeah! Vitamine-P

    #10 3 years ago

    I didnt realize how much I need this site. Scary 4 days.

    #12 3 years ago

    Man, good to be able to visit pinside again. Goed werk Robin!

    #13 3 years ago

    Thanks Robin!

    #14 3 years ago

    time for containers, paas, and scaling

    Glad to see you back.

    #15 3 years ago

    Glad to be back. Thanks!

    #16 3 years ago

    Thank God pinside is back , I had to speak to my wife

    #17 3 years ago

    I just donated a dollar for every time I hit the refresh button this week.

    #18 3 years ago

    Thank you as well!

    I am sure this last week was a tonne of work.

    #19 3 years ago


    I visit Pinside multiple times a day, so it was a very tough and lonely time.

    Hope this donation helps bring you back to 100%.

    #20 3 years ago

    I saw a comment elsewhere on line being very critical of Robin for running Pinside from a personal server at home, instead of from a professional hosting service that could never possibly crash.

    Glad you had backup, since it looks like lots of folks on webnx are screwed!

    #21 3 years ago

    All is right in the world again.

    Welcome back, Robin!

    #22 3 years ago

    Withdrawal is tough.....I am so glad you are back!!!

    #23 3 years ago

    My first question on learning what happened was "What kind of datacenter uses WATER as a fire suppression choice instead nitrogen or another non-damaging non-electrically conducting gas?"

    Crazy. We all learned something here.


    #24 3 years ago

    Nice work Robin! We're back!

    #25 3 years ago

    So glad Pinside is back. Pinball just wasn’t the same without it.

    #26 3 years ago

    Here's a link to a board where people with a clue are discussing the server outage.
    Still sound BAD.

    #27 3 years ago

    Thanks Robin for keeping us updated on social media and working I’m sure countless hours to get this back up and functioning. Hope things continue to go in the right direction with this while you decide on a hosting service. This community has been fantastic for so many of us. Looking forward to many more years of Pinside as we continue to grow and develop along with it.

    #28 3 years ago

    You've done well to lose less than 2hrs worth of data even if the downtime was more than you'd wanted in a perfect world. Glad the site is back.

    #29 3 years ago


    #30 3 years ago

    #31 3 years ago
    6DB69A6E-3813-42D2-91AE-969F14A01757 (resized).jpeg6DB69A6E-3813-42D2-91AE-969F14A01757 (resized).jpeg
    #32 3 years ago

    Wow, this was a major outage. Congrats on getting it back.

    #34 3 years ago

    I can sleep easy. My streak lives!

    pasted_image (resized).pngpasted_image (resized).png
    #35 3 years ago

    Great job, Robin !

    Next time when I'm @ Pinside HQ, you get a big hug and will bring you cold beers.

    Now get some rest

    #36 3 years ago

    Congrats on the recovery! I'm responsible for a few critical datacenters and this scenario is my nightmare... and simply put, no matter how well prepared you think you are, when it hits there are ALWAYS things that slip and don't work as expected (or tested). So you'll get no gripes from me! And now that people are back to usual, you should take a well-earned break for a bit!

    #37 3 years ago

    Whoop whoop!

    #38 3 years ago

    Ahhhhhh i missed you pinside!!!
    Thanks for all your hard work Robin.

    We need some kind of TShirts!
    Fuck Corona......, i was survived the great pinside blackout 2021!

    #39 3 years ago

    Crazy stuff, glad Pinside is back.

    #40 3 years ago

    Thank you Robin !

    LTG : )

    #41 3 years ago

    Glad it's back. Hanging out on Facebook for any kind of talk was awful.

    #42 3 years ago
    Quoted from frolic:

    I can sleep easy. My streak lives!
    [quoted image]

    sheesh, I need to logout once in a while

    #43 3 years ago

    I seriously got so much done in my life.

    Thank god that's over!

    #44 3 years ago

    Great to be back! Didnt realize how often I was on the site till it was gone ...

    #45 3 years ago

    May want to consider hosting on AWS in the future....

    #46 3 years ago

    Happy to see it back online!

    #47 3 years ago

    It was worse than a pandemic

    #48 3 years ago

    Even with limited capability, I am happy Pinside is back!!

    #49 3 years ago

    I thought it was just a fuse....

    Glad to see ya back!

    #50 3 years ago

    Nice to have Pinside back! Four code updates since it’s been down (aiq, LZ and 2 for GnR). Was kind of an eery silence in the pinball world for 6 days. Makes you realize what a valuable resource it is and how we all should contribute to it.

    Quoted from toyotaboy:

    sheesh, I need to logout once in a while

    Yeah, me too, but it has to be for 24 hours, not sure how I’ll manage that.

    There are 331 posts in this topic. You are on page 1 of 7.


    Wanna join the discussion? Please sign in to reply to this topic.

    Hey there! Welcome to Pinside!

    Donate to Pinside

    Great to see you're enjoying Pinside! Did you know Pinside is able to run without any 3rd-party banners or ads, thanks to the support from our visitors? Please consider a donation to Pinside and get anext to your username to show for it! Or better yet, subscribe to Pinside+!

    This page was printed from and we tried optimising it for printing. Some page elements may have been deliberately hidden.

    Scan the QR code on the left to jump to the URL this document was printed from.