(Topic ID: 291292)

Back after a major outage

By robin

1 year ago


Topic Heartbeat

Topic Stats

  • 331 posts
  • 210 Pinsiders participating
  • Latest reply 1 year ago by boagman
  • Topic is favorited by 6 Pinsiders

You

Linked Games

No games have been linked to this topic.

    Topic Gallery

    View topic image gallery

    Pimball.GIF
    image (resized).jpg
    pasted_image (resized).png
    69DFCFDC-E12D-4C2D-8036-6C15CDB4799D (resized).png
    waterdatacenter (resized).png
    uOcsyp (resized).png
    B4EC046D-B13A-4FFE-8F23-3C245F7AE44F (resized).png
    umm-humm-when-keeping-it-real-goes-wrong (resized).jpg
    10b05741d4aa7afec9de5412a443548c (resized).jpeg
    49E5334B-EE51-4A76-B09B-8E99668037D7 (resized).jpeg
    giphy.gif
    pasted_image (resized).png
    6DB69A6E-3813-42D2-91AE-969F14A01757 (resized).jpeg
    pasted_image (resized).png

    There are 331 posts in this topic. You are on page 5 of 7.
    #201 1 year ago

    Thanks for everything you do Robin !
    Can’t live without Pinside in my life.
    Wow... what a long week it was!

    #202 1 year ago
    Quoted from swampfire:

    Azure is normally pretty solid, but man is it expensive. I can understand going with a lower-cost provider.

    As someone who develops and hosts realtime enterprise grade services that must live in azure... i wish i could say they are more reliable. They roll maintenance disruptions with short notice all the effing time. Percentage wise they are fine... but i have at least one or two customer impacts a month due to a host unexpectedly dropping out.

    The azure AD/login outtages lately were big ones too.

    #204 1 year ago
    Quoted from Compy:

    I don't think this calls for a downvote. You're asking some very very good questions. I have businesses that are centered around pinside for sales, and as such, we donate or pay marketplace fees to them for the audience. We will absolutely continue to do so. It just reinforces the fact that we need diversity of both a web frontend and the pinside marketplace to sell our product.
    Some things, like "ordering a new server" this day and age for a site that is one (or many) people's primary source of income is still one hell of a head scratcher for me. Especially when cloud offerings exist that solve (lack of) redundancy issues. But hey, everyone's a site reliability engineer in hindsight The good news is we all learn and move on!
    -- Jimmy

    Hush and get back to working on the CFTBL hologram replacement.

    Just kidding ; ).

    #205 1 year ago
    Quoted from d0n:

    I "donated" about an hour before the site went down with my pinball for sale post. I immediately had people interested but couldn't msg them for 5-6 days due to the outage. Maybe pinside could donate to ME & my buyers for our inconvenience?

    This post alone makes me wish advertising that putting some on ignore is not against the rules.

    Like you could not sell the same game 5 to 6 days later.

    If anything, your apparent inability to sell your games without Pinside demonstrates why WE should donate.

    Sheer audacity, man.....

    #206 1 year ago

    Kind of confusing. Seller couldn't PM his potential customers. But they couldn't PM him either. They must have figured out what the issue was. Neither party was ignoring each other on purpose.

    Hard to lose a sale or buy a game when contact isn't possible.

    Really interested parties would try again once Pinside was back up.

    LTG : )

    #207 1 year ago

    I didn't lose a sale during the outage...I completed one. Anybody that hadn't traded contact info by the time of the outage wasn't a serious buyer (yet), so you're right back where you started when the site came back up.

    #208 1 year ago
    Quoted from PinMonk:

    My first question on learning what happened was "What kind of datacenter uses WATER as a fire suppression choice instead nitrogen or another non-damaging non-electrically conducting gas?"
    Crazy. We all learned something here.
    https://datacenterfrontier.com/generator-catches-fire-causes-lengthy-data-center-outage-at-webnx/

    At a former job, the data center had nitrogen and water. They tested the plumbing one day and we found a giant puddle about 1 ft. away from a very critical server. A pipe leaked. Shit happens...

    #209 1 year ago

    Good to see the only website that matters is back. Ok, there are a couple others too.

    Keep up the great work!

    #210 1 year ago

    Just want to say thanks to Robin for all your hard work. Glad to have Pinside back.

    #211 1 year ago

    Robin, nice work. Take a well deserved break, at least a small one! Glad to be back!

    #212 1 year ago

    Ah I see Achievements are down right now, and the streaks are paused! Now I’m slowly noticing what’s working and what’s not.

    Robin, while you’re rebuilding the site, are you keeping some things offline or are you slowly finding out what’s broken and what’s not? I know this a monumental task, so I know patch notes don’t take priority, just figured I might ask!

    #213 1 year ago
    Quoted from Isochronic_Frost:

    Ah I see Achievements are down right now

    That was intentional. It was a resource intensive site feature that wasn't essential, so it was temporarily disabled.

    Since it looks like things have been going well so far, I'm sure it will return soon.

    #214 1 year ago

    Robin - Thank you for all your hard work on getting this site back up and running again! I've used various guides and advice from so many on this site, I'm kinda lost without it. Also thank you to everyone who's helped me learn throughout the process... I've missed all you as well.

    #215 1 year ago

    Appreciate all the hard work and efforts getting the site back online. I am amazed at the resources the site provides, and see how important it is to so many. Thank you.

    #216 1 year ago
    Quoted from PinMonk:

    My first question on learning what happened was "What kind of datacenter uses WATER as a fire suppression choice instead nitrogen or another non-damaging non-electrically conducting gas?"
    Crazy. We all learned something here.
    https://datacenterfrontier.com/generator-catches-fire-causes-lengthy-data-center-outage-at-webnx/

    This is my job (Critical datacenters)...There is a reason why water suppression is used rather than gas. It's all about client risk, DC risk and cost.

    Most of the DC's I've managed had a water based suppression system. Running a Tier III or Tier IV DC is extremely expensive, as well as gaseous fire suppression systems, and when customers want to pay the cheapest price possible w/o redundant sites, then, when an incident like this happens (it is a "rare" occurrence like a plane crash, but they do happen), there are some consequences.

    Just last month, Europe's biggest cloud service DC company (OVH) has a catastrophic fire, which many clients have lost their data (anyone plays Rust? LOL).

    Not putting any blame on the pinside crew in any way, shape or form. Redundant client sites/DC's/servers "doubles" the hosting costs. Again, it's all about cost, risk and impact. You would have had a better chance of winning the lottery than face a catastrophic DC failure.

    Congrats on the admin crew to bounce back so quickly and bringing back pinside in a few days.... major kudos!!!

    #217 1 year ago

    I had a couple racks in the old Enron data center in New Orleans during hurricane Katrina. We were up for a couple of days until the US Coast Guard confiscated our diesel supply on the roof for their recovery efforts! I was pissed.

    Unless you’ve got Google bucks, things can and will happen.

    #218 1 year ago
    Quoted from radium:

    Unless you’ve got Google bucks, things can and will happen.

    Lot's of big companies out there with DR solutions they hope they never have to use.

    20
    #219 1 year ago
    Quoted from d0n:

    I "donated" about an hour before the site went down with my pinball for sale post. I immediately had people interested but couldn't msg them for 5-6 days due to the outage. Maybe pinside could donate to ME & my buyers for our inconvenience?

    If you feel like that I think you need to leave. Let me make that easy for you, I will refund your donation if you leave and never come back. I myself don't need your messed up attitude.

    #220 1 year ago
    Quoted from Isochronic_Frost:

    Ah I see Achievements are down right now, and the streaks are paused!

    Quoted from ForceFlow:

    That was intentional. It was a resource intensive site feature that wasn't essential, so it was temporarily disabled.
    Since it looks like things have been going well so far, I'm sure it will return soon.

    I happened to notice that late yesterday and PM'ed Robin about it. I had a hunch something like that might be what was going on with the achievements.

    #221 1 year ago

    Thank you for this great resource and to all of those who have worked hard to get it back up and running.

    #222 1 year ago

    Thanks Robin! Was a tough week without you.

    #223 1 year ago

    Thank you Robin for all your hard work and dedication. I love Pinside!!

    #224 1 year ago

    Is it "safe" to contribute at this point or should I wait until everything is completely back to normal? Thanks!

    Jeff

    #225 1 year ago

    Anyone else have issues with Hotmail? I’m not getting emails from my pm’s, and last time I think it was because Pinside was blacklisted or something.

    #226 1 year ago
    Quoted from jeffro01:

    Is it "safe" to contribute at this point or should I wait until everything is completely back to normal? Thanks!
    Jeff

    The first thing I did when I noticed pinside was back up was update my donation. Easy peasy and no issues, donate with confidence!

    #227 1 year ago

    Donated, well my yearly subscription was almost up but yeah upped it as well. Thanks robin for all the hard work going on In The background. Appreciate it mate.

    #228 1 year ago
    Quoted from Mudflaps:

    Anyone else have issues with Hotmail? I’m not getting emails from my pm’s, and last time I think it was because Pinside was blacklisted or something.

    I'm not sure if the email notification system is back up and running yet.

    #229 1 year ago
    Quoted from ForceFlow:

    I'm not sure if the email notification system is back up and running yet.

    I’m getting emails.

    #230 1 year ago

    Ok, I just heard from Robin about the emails. It sounds like if you have a Microsoft email address (outlook, hotmail, etc), the notification emails aren't getting through right now, but the issue should hopefully be resolved soon.

    #231 1 year ago

    I just got one, its all fixed. Thanks!

    52
    #232 1 year ago

    Hey everyone. Thanks a million for all the nice words in this thread. Believe me, it's much appreciated!

    Last week was really a nightmare scenario. Having moved over to a smaller Web Hosting company last year with the idea of getting more direct attention in case of network or hardware issues didn't work out quite like I had hoped.

    So what exactly took me so damn long to get back online? I had backups right? Well, for anyone who's interested in that story, I'll try to recall at some time - but let's just say I wasn't prepared for this kind of disaster. Redundancy is cool, until your datacenter catches fire.

    But of course I should have prepared for this, in hindsight. And I will be in the future as I'm working on a good DR strategy as we speak.

    ... waterdatacenter (resized).png

    So what's cooking now?

    I'm still working on getting everything back up. I got email fully working again today. Of course, being a new server and IP, Outlook immediately threw us on their blacklist. A few emails with their team and several hours later they mitigated this.

    Got email replies working again!

    Just got the Pinside maps server back online!

    Currently working on Marketwatch emails.

    And of course working on backup strategies.

    Coming back soon: topic image gallery, Pinside achievements and anything else that is temporarily disabled or that I forgot. Not much else, I think? Let me know what you are missing besides the stuff I just mentioned.

    Finally, a huge thanks to people who (offered) help/support. And my wife for being extremely patient with me as I was away most of the week and probably terrible to be around during the times I was at home. My brother Martijn of course for moral support. Forceflow. [email protected], [email protected] for relaying messages that we didn't disappear to Timbuktu with all your gif uploads. And Ed R for being funny.

    ** And all of you who donated in the past 48 hours. Amazing to see all that support **

    To be continued. good to be back!

    #233 1 year ago

    Thanks for the news Robin!

    In the list of little bugs, the Top 100 isn't refreshed on monday, unlike usual.

    #234 1 year ago
    Quoted from robin:

    Hey everyone. Thanks a million for all the nice words in this thread......

    You have done wonders getting the site back up! I was just going to post that PDF uploads in one of the forum posts wasn't displaying but I see that it has been fixed. Will keep my eyes open for other issues.

    #235 1 year ago

    Although I'm relatively new to this space, I was surprised how much I missed the Pinside world! So sorry for the catastrophe robin!! Your work and commitment is unbelievable and really truly appreciated!

    #236 1 year ago

    Many thanks Robin. The hero we need and deserve!

    #237 1 year ago

    Thank you Robin for all your hard work!

    17
    #238 1 year ago
    Quoted from robin:

    So what exactly took me so damn long to get back online? I had backups right?

    Robin, don't even worry about being down for a few days. It's a minor disturbance among our first world problems and everything appears to be fine.

    I am apart of another forum that focuses on 1960's Shelby cars (SAAC Forum). Well, that site crashed 18 months ago and everything was lost. 15 years of online forum history, discussions and significant information regarding everything Shelby...... gone. There were no backups and nothing was recoverable so we lost it all. It was a damn shame.

    You've done an amazing job on this site and loosing a couple of hours of Pinside history is insignificant. Consider what happened to my other beloved forum and just think how lucky we all are to have everything back up and running again with almost zero loss of content. Besides, a couple days without Pinside made me realize how much I appreciate this site. You know what they say, "absence makes the heart grow fonder". So true.

    Great job Robin, great job.

    QSS

    #239 1 year ago

    Would like to speak out the appreciation to the team ! Of course robin ,it must have been stressful.. but on the bright side, you survived!! Big time..
    As many others , I was amazed how much I checked the site.
    But as always , as Johan Cruijf says.. elk nadeel heb zn voordeel ...
    Donation will be made..

    11
    #240 1 year ago

    As of now, Pinside MarketWatch™ notifications should be working again!

    #241 1 year ago
    Quoted from QuickSilverShelby:

    Robin, don't even worry about being down for a few days. It's a minor disturbance among our first world problems and everything appears to be fine.
    I am apart of another forum that focuses on 1960's Shelby cars (SAAC Forum). Well, that site crashed 18 months ago and everything was lost. 15 years of online forum history, discussions and significant information regarding everything Shelby...... gone. There were no backups and nothing was recoverable so we lost it all. It was a damn shame.
    You've done an amazing job on this site and loosing a couple of hours of Pinside history is insignificant. Consider what happened to my other beloved forum and just think how lucky we all are to have everything back up and running again with almost zero loss of content. Besides, a couple days without Pinside made me realize how much I appreciate this site. You know what they say, "absence makes the heart grow fonder". So true.
    Great job Robin, great job.
    QSS

    Wow that’s depressing. Pinside is 18, I shudder to think about losing all the precious resources and stories, vid’s guides, and especially all the archives of pinheads who have coined up that great pinball hall in the sky.

    Their stories are maintained here and nowhere else, they’d never come back if we lost them. It’s sad to hear about all that Shelby data lost, hopefully some of it has been archived?

    Thank YOU Robin for being prepared for the worst, Pinside has survived to see another day!

    #242 1 year ago
    Quoted from Isochronic_Frost:

    Thank YOU Robin for being prepared for the worst, Pinside has survived to see another day!

    I agree here, the story here is not that Pinside was down, but that it was really, really, really well backed up. And going forward it is going to be even more resilient. robin , thanks for creating such an enduring and important resource for our community!

    #243 1 year ago
    Quoted from Azmodeus:

    Good to see the only website that matters is back. Ok, there are a couple others too.
    Keep up the great work!

    Cough cough youporn ciugh cough

    #244 1 year ago

    For robin. This buds for you.

    22
    #245 1 year ago
    Quoted from Isochronic_Frost:

    I shudder to think about losing all the precious resources and stories, vid’s guides, and especially all the archives of pinheads who have coined up that great pinball hall in the sky.
    Their stories are maintained here and nowhere else, they’d never come back if we lost them.

    You're absolutely right. You'll be pleased to hear that, going forward, Pinside is now running *hourly* data backups. These get stored in 3 separate datacenters by 3 different companies in 3 different geographic locations. Encrypted and safe, also from ransomware due to object locks.

    And to give you another Shudder story: As I was restoring backups to my temporary server, I swear I could not remember where I had stored my data encryption key. Maybe it was the stress or what not, but it took me an hour of frantic searching. After having a coffee, it dawned on me where it was. I literally shouted out a relieved "Whoohoo!" when I found it. Another learning moment, ain't gonna happen to me again.

    #246 1 year ago
    Quoted from robin:

    And to give you another Shudder story: As I was restoring backups to my temporary server, I swear I could not remember where I had stored my data encryption key. Maybe it was the stress or what not, but it took me an hour of frantic searching. After having a coffee, it dawned on me where it was. I literally shouted out a relieved "Whoohoo!" when I found it. Another learning moment, ain't gonna happen to me again.

    Kind of like that guy who threw out his hard drive containing the encryption key to all his bitcoins.

    https://www.cnn.com/2021/01/15/uk/bitcoin-trash-landfill-gbr-scli-intl/index.html

    #247 1 year ago
    Quoted from Isochronic_Frost:

    Wow that’s depressing. Pinside is 18, I shudder to think about losing all the precious resources and stories, vid’s guides, and especially all the archives of pinheads who have coined up that great pinball hall in the sky.
    Their stories are maintained here and nowhere else, they’d never come back if we lost them. It’s sad to hear about all that Shelby data lost, hopefully some of it has been archived?
    Thank YOU Robin for being prepared for the worst, Pinside has survived to see another day!

    Yes, it was and is depressing. So much history, resources and stories were lost on the SAAC site. This temporary shut down of Pinside was trivial and nothing of concern. Robin saved 99.9999% of everything on Pinside. We are so damn lucky to have this computer savvy pinball nerd running the show.

    When I say "computer savvy pinball nerd" I mean that in the highest form of compliment. After all, that's what I am and most of the rest of us are. I consider myself, and most of all the forum members here, absolute "computer savvy pinball nerd, shit!!" ......... figuratively speaking of course.

    QSS

    #248 1 year ago
    Quoted from robin:

    ...Coming back soon: topic image gallery, Pinside achievements and anything else that is temporarily disabled or that I forgot. Not much else, I think? Let me know what you are missing besides the stuff I just mentioned...

    Seems like my "All posts made by you" activity is sort of 'hit or miss' w/respect to being current.

    And thanks for all the hard work!

    #249 1 year ago

    nice work Robin. Good decent backup time too - this is valuable resource so we appreciate it indeed.

    EEVBLOG Dave in Aus, his forum went down and he made a video on it

    #250 1 year ago

    Don't forget to update your Continuity of Operations Plan (COOP) and Contingency Plans for lessons learned

    I'm going to make my yearly donation earlier today and play some pinball. Maybe post some gifs.

    Thanks Robin.

    There are 331 posts in this topic. You are on page 5 of 7.

    Reply

    Wanna join the discussion? Please sign in to reply to this topic.

    Hey there! Welcome to Pinside!

    Donate to Pinside

    Great to see you're enjoying Pinside! Did you know Pinside is able to run without any 3rd-party banners or ads, thanks to the support from our visitors? Please consider a donation to Pinside and get anext to your username to show for it! Or better yet, become a Pinside+ member!