(Topic ID: 291292)

Back after a major outage

By robin

3 years ago


Topic Heartbeat

Topic Stats

  • 331 posts
  • 210 Pinsiders participating
  • Latest reply 2 years ago by boagman
  • Topic is favorited by 6 Pinsiders

You

Linked Games

No games have been linked to this topic.

    Topic Gallery

    View topic image gallery

    Pimball.GIF
    image (resized).jpg
    pasted_image (resized).png
    69DFCFDC-E12D-4C2D-8036-6C15CDB4799D (resized).png
    waterdatacenter (resized).png
    uOcsyp (resized).png
    B4EC046D-B13A-4FFE-8F23-3C245F7AE44F (resized).png
    umm-humm-when-keeping-it-real-goes-wrong (resized).jpg
    10b05741d4aa7afec9de5412a443548c (resized).jpeg
    49E5334B-EE51-4A76-B09B-8E99668037D7 (resized).jpeg
    giphy.gif
    pasted_image (resized).png
    6DB69A6E-3813-42D2-91AE-969F14A01757 (resized).jpeg
    pasted_image (resized).png
    There are 331 posts in this topic. You are on page 2 of 7.
    #51 3 years ago

    Thank you for your hard work @robin! Were having a skål here in the bunker for you.

    #52 3 years ago

    Woo Hoo! I was starting to convulse..

    #53 3 years ago

    I'm so glad we're back in business here! Although I have steeled myself for a rash of typos until everyone's shakes abate.

    #54 3 years ago

    I didn't realize how much I read pinside daily until it was taken away, I was going through withdrawls!

    #55 3 years ago

    Great that you are back. Thanks for all your hard work, robin . I just donated some more. This website is vital, and I at least was taking access to it for granted a bit.

    #56 3 years ago

    Phew ... that was a tough week.

    I believe the topic galleries are missing ?
    Hope they come back, I can't figure out my T2 playfield without it ...

    #57 3 years ago
    giphy.gifgiphy.gif
    #58 3 years ago

    So good to be back in my habitat. That was a productive week for me around the house in the evenings. Not anymore haha. Thanks robin for the hard work to get everything back on track so far.

    #59 3 years ago

    Thank goodness! Longest 5 days EVER!!!

    #60 3 years ago
    Quoted from RCA1:

    I saw a comment elsewhere on line being very critical of Robin for running Pinside from a personal server at home, instead of from a professional hosting service that could never possibly crash.
    Glad you had backup, since it looks like lots of folks on webnx are screwed!

    Those people are clueless, they all kept saying it’s a personal server when WebNX is a professional outfit who goofed up hard lol.
    The Facebook peanut gallery proved they are toxic af.

    Quoted from alexanr1:

    May want to consider hosting on AWS in the future....

    No thanks, Amazon doesn’t need to own the last free half % of the Internet.

    #61 3 years ago

    Merci beaucoup Robin !
    'It's when you lose things that you realize how much you value them'

    #62 3 years ago

    New server seems way faster than the crappy old one!

    #63 3 years ago

    Some blame fire suppression, some blame Robin, I blame all those price bubble threads.

    #64 3 years ago
    Quoted from Lermods:

    Was kind of an eery silence in the pinball world for 6 days.

    For all the haters, it really showed what a force Pinside is. I'm willing to bet BUSINESS was down across the board.

    #65 3 years ago
    Quoted from Friengineer:

    Some blame fire suppression, some blame Robin, I blame all those price bubble threads.

    I could see Robin being like...nope turns out those threads didn’t survive the backup. Darn corrupted sector

    #66 3 years ago

    Good job Robin !

    But look out for a better provider .... I read in the internet that they tried to fight a fire in a data center with ... water ....
    That seems very stupid.

    #67 3 years ago

    Great job on getting us up and running again so soon! I can't figure out how to attach a pic yet, but maybe its just me!

    #68 3 years ago
    Quoted from robin:

    My last database backup was dated 10:02pm, while the server went down 11:39pm. So we lost 1h37m of data. That means any posts, PMs or Marketplace ads placed during that time are gone.

    Oh man, that's exactly when I made some of my best posts!!!

    Seriously though, thanks for getting it back up! I'm sure it's quite stressful.

    #69 3 years ago

    Glad to see you’re back up and running finally. I hope you can stay that way for a while. Can’t wait to see the new servers in the future! Take care and best wishes.

    #70 3 years ago

    Nice to be back after the blip

    49E5334B-EE51-4A76-B09B-8E99668037D7 (resized).jpeg49E5334B-EE51-4A76-B09B-8E99668037D7 (resized).jpeg
    #71 3 years ago

    Robin -

    Thanks for your hard work. I really appreciate it.

    Put your next server someplace where they use halon as a fire suppressant.

    #72 3 years ago

    Thanks so much for bringing Pinside back!

    I guess it’s true what they say about not knowing how good something is until it’s gone

    I sure did miss it.

    Thanks again

    #73 3 years ago

    Welcome back Robin!

    Thanks for everything you do, I can't image what you have had to go through to get things back up so far.

    If you need any help please let me know.

    I have my own servers out here in the Bay Area:

    All virtual on multiple hosts / load balanced. Redundant: uplinks, firewalls, 1g Cisco WAN switches, 10g Cisco SAN/LAN switches. Redundant Flash SAN (2 active/active nodes)

    If you need any server admin help from a fellow pinhead just let me know! (I'm sure others would be happy to pitch in as well!)

    Cheers

    ~Steve

    #74 3 years ago
    Quoted from PinMonk:

    My first question on learning what happened was "What kind of datacenter uses WATER as a fire suppression choice instead nitrogen or another non-damaging non-electrically conducting gas?"
    Crazy. We all learned something here.
    https://datacenterfrontier.com/generator-catches-fire-causes-lengthy-data-center-outage-at-webnx/

    The real question is, what kind of datacenter puts the generators INSIDE with the servers?

    WebNX = clowntown.

    #75 3 years ago

    Welcome back all!

    #76 3 years ago

    Do not yet coils lock on and have it lead to needing the FD to clean it up.

    #77 3 years ago
    Quoted from Billc479:

    Robin -
    Thanks for your hard work. I really appreciate it.
    Put your next server someplace where they use halon as a fire suppressant.

    fire codes in an lot of places make you have wet with an dry system.

    #78 3 years ago

    Welcome back Robin! Look at it this way- the worst has happened and now you are back and better than ever before .
    Thank you and the support staff for all your hard work!

    #79 3 years ago

    I don't know if it's the new server or just reduced load but pinside is a lot faster than usual! Thanks robin !

    #80 3 years ago

    Finally it was a nightmare !

    #81 3 years ago

    Thanks Robin

    #82 3 years ago

    This was worse than losing out on the GNR CE. Glad I have something to do now.

    #83 3 years ago

    Welcome back everyone!

    #85 3 years ago

    I’m surprised the site isn’t hosted on a cloud provider. New “server” in <1 minute, multiple availability zones per region, DB replication across AZs, etc., etc.

    #86 3 years ago

    Great job Robin!! I'm sure things will be even more secure and backed up than ever now!

    Talking to a friend the other day I pointed out that the root cause of all of this was most likely those two useless pinside gif threads. All of the bandwidth and storage space they use probably caused the server rebuild to take twice as long even if they didn't cause the initial power failure and fire.

    #87 3 years ago

    Thanks Robin! Not a bad outcome all things considered.

    #88 3 years ago

    With the recent announcement that Pinball Rescue was closing for good and then Pinside going down... I was going full pinball panic mode. Still kind of am, but relieved to be back.

    Pinside is an awesome and vital resource for the community. In fact, I wouldn't be where I am today without it. <3

    #89 3 years ago

    Thanks Robin for working hard to bring the site back. There was a gaping hole in my life the last week and Im glad were back!

    #90 3 years ago

    I work for a company that does nothing else but assess data centers for their ability to withstand stress. That's what we do and after looking at literally tens of thousands of data centers, all I can say is you have no idea how silly some site owners are. The crazy choices they make. It's a combination of cost, business model and expertise.

    That said, the hosting/co-location world is very Darwinian... with so many to choose from, it really is survival of the fittest!

    #91 3 years ago

    Thanks robin !! Donation sent. Great job on the recovery. Please take some time for a well deserved rest....

    #92 3 years ago

    I sent over a donation to help with the new server and your time robin
    We all appreciate your efforts to get things up and running.
    -Mike

    #93 3 years ago
    Quoted from Joe_Blasi:

    fire codes in an lot of places make you have wet with an dry system.

    I'm dating myself - should have said FM200. Also, the purpose was to thank Robin for his work.

    Fire suppressant discussions should follow NFPA 12 and NFPA 2001 as amended. Hard to imagine why the section with servers only would fall under class 2 water suppression unless they were mixed with other combustibles.

    #94 3 years ago
    Quoted from jorro:

    Ahhhhhh i missed you pinside!!!
    Thanks for all your hard work Robin.
    We need some kind of TShirts!
    Fuck Corona......, i was survived the great pinside blackout 2021!

    Robin you should sell t-shirts that say this. I'll buy one!

    "I survived the Great Pinside Blackout of 2021"

    #95 3 years ago

    Thanks for getting everything back and running, @robin. Its been a weird time in pinball lately. This site was down for a week, the virtual pinball community has been going through some major drama and site closings. Nothing too serious on the grand scale, but it's nice to have some normalcy back. I miss the pinside machine database as much as or more than the forum. I've always enjoyed reading through the reviews and watching the linked youtubes to kill time here and there.

    #96 3 years ago

    THANK YOU ROBIN !!

    glad to be back

    -67
    #97 3 years ago

    I "donated" about an hour before the site went down with my pinball for sale post. I immediately had people interested but couldn't msg them for 5-6 days due to the outage. Maybe pinside could donate to ME & my buyers for our inconvenience?

    #98 3 years ago

    Damn glad to see ya...

    #99 3 years ago

    I am addicted to pinside !!!
    Got very stressed out every time I checked and seen that it was still down....
    Had nothing to do at work...... when it was down....LOL

    19
    #100 3 years ago
    Quoted from kevmad:

    "I survived the Great Pinside Blackout of 2021"

    And in years to come. "My parents made me in the Great Pinside Blackout of 2021". Baby boom in 9 months.

    LTG : )

    There are 331 posts in this topic. You are on page 2 of 7.

    Reply

    Wanna join the discussion? Please sign in to reply to this topic.

    Hey there! Welcome to Pinside!

    Donate to Pinside

    Great to see you're enjoying Pinside! Did you know Pinside is able to run without any 3rd-party banners or ads, thanks to the support from our visitors? Please consider a donation to Pinside and get anext to your username to show for it! Or better yet, subscribe to Pinside+!


    This page was printed from https://pinside.com/pinball/forum/topic/back-after-a-major-outage/page/2?hl=manny65 and we tried optimising it for printing. Some page elements may have been deliberately hidden.

    Scan the QR code on the left to jump to the URL this document was printed from.