(Topic ID: 291292)

Back after a major outage

By robin

1 year ago

Topic Heartbeat

Topic Stats

  • 331 posts
  • 210 Pinsiders participating
  • Latest reply 1 year ago by boagman
  • Topic is favorited by 6 Pinsiders


Linked Games

No games have been linked to this topic.

    Topic Gallery

    View topic image gallery

    image (resized).jpg
    pasted_image (resized).png
    69DFCFDC-E12D-4C2D-8036-6C15CDB4799D (resized).png
    waterdatacenter (resized).png
    uOcsyp (resized).png
    B4EC046D-B13A-4FFE-8F23-3C245F7AE44F (resized).png
    umm-humm-when-keeping-it-real-goes-wrong (resized).jpg
    10b05741d4aa7afec9de5412a443548c (resized).jpeg
    49E5334B-EE51-4A76-B09B-8E99668037D7 (resized).jpeg
    pasted_image (resized).png
    6DB69A6E-3813-42D2-91AE-969F14A01757 (resized).jpeg
    pasted_image (resized).png

    There are 331 posts in this topic. You are on page 2 of 7.
    #51 1 year ago

    Thank you for your hard work robin! Were having a skål here in the bunker for you.

    #52 1 year ago

    Woo Hoo! I was starting to convulse..

    #53 1 year ago

    I'm so glad we're back in business here! Although I have steeled myself for a rash of typos until everyone's shakes abate.

    #54 1 year ago

    I didn't realize how much I read pinside daily until it was taken away, I was going through withdrawls!

    #55 1 year ago

    Great that you are back. Thanks for all your hard work, robin . I just donated some more. This website is vital, and I at least was taking access to it for granted a bit.

    #56 1 year ago

    Phew ... that was a tough week.

    I believe the topic galleries are missing ?
    Hope they come back, I can't figure out my T2 playfield without it ...

    #57 1 year ago
    #58 1 year ago

    So good to be back in my habitat. That was a productive week for me around the house in the evenings. Not anymore haha. Thanks robin for the hard work to get everything back on track so far.

    #59 1 year ago

    Thank goodness! Longest 5 days EVER!!!

    #60 1 year ago
    Quoted from RCA1:

    I saw a comment elsewhere on line being very critical of Robin for running Pinside from a personal server at home, instead of from a professional hosting service that could never possibly crash.
    Glad you had backup, since it looks like lots of folks on webnx are screwed!

    Those people are clueless, they all kept saying it’s a personal server when WebNX is a professional outfit who goofed up hard lol.
    The Facebook peanut gallery proved they are toxic af.

    Quoted from alexanr1:

    May want to consider hosting on AWS in the future....

    No thanks, Amazon doesn’t need to own the last free half % of the Internet.

    #61 1 year ago

    Merci beaucoup Robin !
    'It's when you lose things that you realize how much you value them'

    #62 1 year ago

    New server seems way faster than the crappy old one!

    #63 1 year ago

    Some blame fire suppression, some blame Robin, I blame all those price bubble threads.

    #64 1 year ago
    Quoted from Lermods:

    Was kind of an eery silence in the pinball world for 6 days.

    For all the haters, it really showed what a force Pinside is. I'm willing to bet BUSINESS was down across the board.

    #65 1 year ago
    Quoted from Friengineer:

    Some blame fire suppression, some blame Robin, I blame all those price bubble threads.

    I could see Robin being like...nope turns out those threads didn’t survive the backup. Darn corrupted sector

    #66 1 year ago

    Good job Robin !

    But look out for a better provider .... I read in the internet that they tried to fight a fire in a data center with ... water ....
    That seems very stupid.

    #67 1 year ago

    Great job on getting us up and running again so soon! I can't figure out how to attach a pic yet, but maybe its just me!

    #68 1 year ago
    Quoted from robin:

    My last database backup was dated 10:02pm, while the server went down 11:39pm. So we lost 1h37m of data. That means any posts, PMs or Marketplace ads placed during that time are gone.

    Oh man, that's exactly when I made some of my best posts!!!

    Seriously though, thanks for getting it back up! I'm sure it's quite stressful.

    #69 1 year ago

    Glad to see you’re back up and running finally. I hope you can stay that way for a while. Can’t wait to see the new servers in the future! Take care and best wishes.

    #70 1 year ago

    Nice to be back after the blip

    49E5334B-EE51-4A76-B09B-8E99668037D7 (resized).jpeg
    #71 1 year ago

    Robin -

    Thanks for your hard work. I really appreciate it.

    Put your next server someplace where they use halon as a fire suppressant.

    #72 1 year ago

    Thanks so much for bringing Pinside back!

    I guess it’s true what they say about not knowing how good something is until it’s gone

    I sure did miss it.

    Thanks again

    #73 1 year ago

    Welcome back Robin!

    Thanks for everything you do, I can't image what you have had to go through to get things back up so far.

    If you need any help please let me know.

    I have my own servers out here in the Bay Area:

    All virtual on multiple hosts / load balanced. Redundant: uplinks, firewalls, 1g Cisco WAN switches, 10g Cisco SAN/LAN switches. Redundant Flash SAN (2 active/active nodes)

    If you need any server admin help from a fellow pinhead just let me know! (I'm sure others would be happy to pitch in as well!)



    #74 1 year ago
    Quoted from PinMonk:

    My first question on learning what happened was "What kind of datacenter uses WATER as a fire suppression choice instead nitrogen or another non-damaging non-electrically conducting gas?"
    Crazy. We all learned something here.

    The real question is, what kind of datacenter puts the generators INSIDE with the servers?

    WebNX = clowntown.

    #75 1 year ago

    Welcome back all!

    #76 1 year ago

    Do not yet coils lock on and have it lead to needing the FD to clean it up.

    #77 1 year ago
    Quoted from Billc479:

    Robin -
    Thanks for your hard work. I really appreciate it.
    Put your next server someplace where they use halon as a fire suppressant.

    fire codes in an lot of places make you have wet with an dry system.

    #78 1 year ago

    Welcome back Robin! Look at it this way- the worst has happened and now you are back and better than ever before .
    Thank you and the support staff for all your hard work!

    #79 1 year ago

    I don't know if it's the new server or just reduced load but pinside is a lot faster than usual! Thanks robin !

    #80 1 year ago

    Finally it was a nightmare !

    #81 1 year ago

    Thanks Robin

    #82 1 year ago

    This was worse than losing out on the GNR CE. Glad I have something to do now.

    #83 1 year ago

    Welcome back everyone!

    #85 1 year ago

    I’m surprised the site isn’t hosted on a cloud provider. New “server” in <1 minute, multiple availability zones per region, DB replication across AZs, etc., etc.

    #86 1 year ago

    Great job Robin!! I'm sure things will be even more secure and backed up than ever now!

    Talking to a friend the other day I pointed out that the root cause of all of this was most likely those two useless pinside gif threads. All of the bandwidth and storage space they use probably caused the server rebuild to take twice as long even if they didn't cause the initial power failure and fire.

    #87 1 year ago

    Thanks Robin! Not a bad outcome all things considered.

    #88 1 year ago

    With the recent announcement that Pinball Rescue was closing for good and then Pinside going down... I was going full pinball panic mode. Still kind of am, but relieved to be back.

    Pinside is an awesome and vital resource for the community. In fact, I wouldn't be where I am today without it. <3

    #89 1 year ago

    Thanks Robin for working hard to bring the site back. There was a gaping hole in my life the last week and Im glad were back!

    #90 1 year ago

    I work for a company that does nothing else but assess data centers for their ability to withstand stress. That's what we do and after looking at literally tens of thousands of data centers, all I can say is you have no idea how silly some site owners are. The crazy choices they make. It's a combination of cost, business model and expertise.

    That said, the hosting/co-location world is very Darwinian... with so many to choose from, it really is survival of the fittest!

    #91 1 year ago

    Thanks robin !! Donation sent. Great job on the recovery. Please take some time for a well deserved rest....

    #92 1 year ago

    I sent over a donation to help with the new server and your time robin
    We all appreciate your efforts to get things up and running.

    #93 1 year ago
    Quoted from Joe_Blasi:

    fire codes in an lot of places make you have wet with an dry system.

    I'm dating myself - should have said FM200. Also, the purpose was to thank Robin for his work.

    Fire suppressant discussions should follow NFPA 12 and NFPA 2001 as amended. Hard to imagine why the section with servers only would fall under class 2 water suppression unless they were mixed with other combustibles.

    #94 1 year ago
    Quoted from jorro:

    Ahhhhhh i missed you pinside!!!
    Thanks for all your hard work Robin.
    We need some kind of TShirts!
    Fuck Corona......, i was survived the great pinside blackout 2021!

    Robin you should sell t-shirts that say this. I'll buy one!

    "I survived the Great Pinside Blackout of 2021"

    #95 1 year ago

    Thanks for getting everything back and running, robin. Its been a weird time in pinball lately. This site was down for a week, the virtual pinball community has been going through some major drama and site closings. Nothing too serious on the grand scale, but it's nice to have some normalcy back. I miss the pinside machine database as much as or more than the forum. I've always enjoyed reading through the reviews and watching the linked youtubes to kill time here and there.

    #96 1 year ago


    glad to be back

    #97 1 year ago

    I "donated" about an hour before the site went down with my pinball for sale post. I immediately had people interested but couldn't msg them for 5-6 days due to the outage. Maybe pinside could donate to ME & my buyers for our inconvenience?

    #98 1 year ago

    Damn glad to see ya...

    #99 1 year ago

    I am addicted to pinside !!!
    Got very stressed out every time I checked and seen that it was still down....
    Had nothing to do at work...... when it was down....LOL

    #100 1 year ago
    Quoted from kevmad:

    "I survived the Great Pinside Blackout of 2021"

    And in years to come. "My parents made me in the Great Pinside Blackout of 2021". Baby boom in 9 months.

    LTG : )

    There are 331 posts in this topic. You are on page 2 of 7.


    Wanna join the discussion? Please sign in to reply to this topic.

    Hey there! Welcome to Pinside!

    Donate to Pinside

    Great to see you're enjoying Pinside! Did you know Pinside is able to run without any 3rd-party banners or ads, thanks to the support from our visitors? Please consider a donation to Pinside and get anext to your username to show for it! Or better yet, subscribe to Pinside+!

    This page was printed from https://pinside.com/pinball/forum/topic/back-after-a-major-outage/page/2?hl=catvilledoorman and we tried optimising it for printing. Some page elements may have been deliberately hidden.

    Scan the QR code on the left to jump to the URL this document was printed from.