We're back!

After a disaster in our datacenter took down our server, we are now running Pinside on a brand new server. However, things may not work as expected as we are still polishing the playfield and adjusting some switches! We might be switching to maintenance mode a few times. Please see this forum topic for more info.

(Topic ID: 291292)

Back after a major outage


By robin

3 days ago

Topic Heartbeat


Topic Stats

  • 220 posts
  • 176 Pinsiders participating
  • Latest reply 8 minutes ago by SonicZone
  • Topic is favorited by 7 Pinsiders
  • Topic is a super sticky!

You

Linked Games

No games have been linked to this topic.

    There are 220 posts in this topic. You are on page 1 of 5.
    199
    #1 3 days ago

    We're back!

    What a crazy week it has been for Pinside (and a stressful week for me, I might add) with our first major outage in the 20 years or so that Pinside exists. Sure, we've had hiccups but nothing like this. The cause: a power surge, followed by a fire in a generator, followed by sprinklers and our server going down. I wasn't there to see it, but I don't think it was pretty.

    Anyways, we're running on a temporary server now, hosted all the way over in Germany (well, that's close for me). Please be nice to it, it's not as powerful as our previous one and there may be problems. Also, not everything functions yet: We have no e-mail notifications yet. And no maps. And probably a bunch of other stuff that doesn't work right now. But at least we're back. I'll be moving to a brand new server (and a different hosting company) later on but I don't want to rush that choice.

    As for our data recovery procedure. Well there really wasn't one in place as you may have noticed. Sure I had dual power supplies, dual network uplinks, dual (raid) drives. But I would not have expected a datacenter fire. I should have. I should have had a backup server somewhere, ready to spin up and continue service after a few hours. Truth is, I never got around to setting this up. But I will obviously be working on this, so a downtime like this cannot happen again in the future.

    My last database backup was dated 10:02pm, while the server went down 11:39pm. So we lost 1h37m of data. That means any posts, PMs or Marketplace ads placed during that time are gone. And even if they restore our server (they're still working on it but are now giving me an ETA of "weeks") I will not be bringing that data back as that would be a technical nightmare.

    If you made a donation or marketplace payment, please contact me. I will also be proactively look into any payments made during that 1h37m and take action or contact you.

    Speaking of donations. Feel free to make a donation, if you're happy that we're back!

    Let's get the pinball talk going again and please bear with me while I try to keep this temporary server running nicely.

    -- Robin

    PS: Seeing any issues? Let me know!

    11
    #2 3 days ago

    Thanks for all the hard work getting Pinside back up. I wasn't sure what to do with my free time the last few days. I tried moderating my wife, but we all know how that went.

    #3 3 days ago

    Thanks Robin!
    I was starting to suffer from withdrawal syndrome... Good to be back!

    #4 3 days ago

    AH-ha!!! I’ve been hammering the beta site refresh hoping it would come online first.

    God bless. I was having massive withdrawals. My almost 4 year streak (1318 days) on this website, crushed!!

    58
    #5 1 day ago

    Well, you all didn't take long to crash our temporary server. After 30-40 minutes it went down smoking. So I went back to the drawing board and ordered a brand new server. This should work, but please bear with me as I will be tweaking and tuning some more over the next week(s). The upside is that the new server is twice as powerful as the previous one, so with the right approach this could all end better than it started.

    Anyways, if you're able to read this: welcome back!

    18
    #6 1 day ago

    Woo-hoo!

    pasted_image (resized).png
    #9 1 day ago

    Yeah! Vitamine-P

    #10 1 day ago

    I didnt realize how much I need this site. Scary 4 days.

    #12 1 day ago

    Man, good to be able to visit pinside again. Goed werk Robin!

    #13 1 day ago

    Thanks Robin!

    #14 1 day ago

    time for containers, paas, and scaling

    Glad to see you back.

    #15 1 day ago

    Glad to be back. Thanks!

    #16 1 day ago

    Thank God pinside is back , I had to speak to my wife

    10
    #17 1 day ago

    I just donated a dollar for every time I hit the refresh button this week.

    #18 1 day ago

    Thank you as well!

    I am sure this last week was a tonne of work.

    #19 1 day ago

    Donated.

    I visit Pinside multiple times a day, so it was a very tough and lonely time.

    Hope this donation helps bring you back to 100%.

    #20 1 day ago

    I saw a comment elsewhere on line being very critical of Robin for running Pinside from a personal server at home, instead of from a professional hosting service that could never possibly crash.

    Glad you had backup, since it looks like lots of folks on webnx are screwed!

    #21 1 day ago

    All is right in the world again.

    Welcome back, Robin!

    #22 1 day ago

    Withdrawal is tough.....I am so glad you are back!!!

    #23 1 day ago

    My first question on learning what happened was "What kind of datacenter uses WATER as a fire suppression choice instead nitrogen or another non-damaging non-electrically conducting gas?"

    Crazy. We all learned something here.

    https://datacenterfrontier.com/generator-catches-fire-causes-lengthy-data-center-outage-at-webnx/

    #24 1 day ago

    Nice work Robin! We're back!

    #25 1 day ago

    So glad Pinside is back. Pinball just wasn’t the same without it.

    #26 1 day ago

    Here's a link to a board where people with a clue are discussing the server outage.
    Still sound BAD.
    https://www.webhostingtalk.com/showthread.php?t=1842301

    #27 1 day ago

    Thanks Robin for keeping us updated on social media and working I’m sure countless hours to get this back up and functioning. Hope things continue to go in the right direction with this while you decide on a hosting service. This community has been fantastic for so many of us. Looking forward to many more years of Pinside as we continue to grow and develop along with it.

    #28 1 day ago

    You've done well to lose less than 2hrs worth of data even if the downtime was more than you'd wanted in a perfect world. Glad the site is back.

    #29 1 day ago

    Woohoo!

    #31 1 day ago
    6DB69A6E-3813-42D2-91AE-969F14A01757 (resized).jpeg
    #32 1 day ago

    Wow, this was a major outage. Congrats on getting it back.

    #33 1 day ago

    Good job and thanks

    11
    #34 1 day ago

    I can sleep easy. My streak lives!

    pasted_image (resized).png
    #35 1 day ago

    Great job, Robin !

    Next time when I'm @ Pinside HQ, you get a big hug and will bring you cold beers.

    Now get some rest

    #36 1 day ago

    Congrats on the recovery! I'm responsible for a few critical datacenters and this scenario is my nightmare... and simply put, no matter how well prepared you think you are, when it hits there are ALWAYS things that slip and don't work as expected (or tested). So you'll get no gripes from me! And now that people are back to usual, you should take a well-earned break for a bit!

    #37 1 day ago

    Whoop whoop!

    10
    #38 1 day ago

    Ahhhhhh i missed you pinside!!!
    Thanks for all your hard work Robin.

    We need some kind of TShirts!
    Fuck Corona......, i was survived the great pinside blackout 2021!

    #39 1 day ago

    Crazy stuff, glad Pinside is back.

    #41 1 day ago

    Glad it's back. Hanging out on Facebook for any kind of talk was awful.

    #42 1 day ago
    Quoted from frolic:

    I can sleep easy. My streak lives!
    [quoted image]

    sheesh, I need to logout once in a while

    #43 1 day ago

    I seriously got so much done in my life.

    Thank god that's over!

    #44 1 day ago

    Great to be back! Didnt realize how often I was on the site till it was gone ...

    #45 1 day ago

    May want to consider hosting on AWS in the future....

    #46 1 day ago

    Happy to see it back online!

    #47 1 day ago

    It was worse than a pandemic

    #48 1 day ago

    Even with limited capability, I am happy Pinside is back!!

    #49 1 day ago

    I thought it was just a fuse....

    Glad to see ya back!

    #50 1 day ago

    Nice to have Pinside back! Four code updates since it’s been down (aiq, LZ and 2 for GnR). Was kind of an eery silence in the pinball world for 6 days. Makes you realize what a valuable resource it is and how we all should contribute to it.

    Quoted from toyotaboy:

    sheesh, I need to logout once in a while

    Yeah, me too, but it has to be for 24 hours, not sure how I’ll manage that.

    There are 220 posts in this topic. You are on page 1 of 5.

    Hey there! Got a moment?

    Great to see you're enjoying Pinside! Did you know Pinside is able to run thanks to donations from our visitors? Please donate to Pinside, support the site and get anext to your username to show for it! Donate to Pinside