Thanks for everything you do Robin !
Can’t live without Pinside in my life.
Wow... what a long week it was!
Thanks for everything you do Robin !
Can’t live without Pinside in my life.
Wow... what a long week it was!
Quoted from swampfire:Azure is normally pretty solid, but man is it expensive. I can understand going with a lower-cost provider.
As someone who develops and hosts realtime enterprise grade services that must live in azure... i wish i could say they are more reliable. They roll maintenance disruptions with short notice all the effing time. Percentage wise they are fine... but i have at least one or two customer impacts a month due to a host unexpectedly dropping out.
The azure AD/login outtages lately were big ones too.
Quoted from Compy:I don't think this calls for a downvote. You're asking some very very good questions. I have businesses that are centered around pinside for sales, and as such, we donate or pay marketplace fees to them for the audience. We will absolutely continue to do so. It just reinforces the fact that we need diversity of both a web frontend and the pinside marketplace to sell our product.
Some things, like "ordering a new server" this day and age for a site that is one (or many) people's primary source of income is still one hell of a head scratcher for me. Especially when cloud offerings exist that solve (lack of) redundancy issues. But hey, everyone's a site reliability engineer in hindsight The good news is we all learn and move on!
-- Jimmy
Hush and get back to working on the CFTBL hologram replacement.
Just kidding ; ).
Quoted from d0n:I "donated" about an hour before the site went down with my pinball for sale post. I immediately had people interested but couldn't msg them for 5-6 days due to the outage. Maybe pinside could donate to ME & my buyers for our inconvenience?
This post alone makes me wish advertising that putting some on ignore is not against the rules.
Like you could not sell the same game 5 to 6 days later.
If anything, your apparent inability to sell your games without Pinside demonstrates why WE should donate.
Sheer audacity, man.....
Quoted from NPO:This post
Kind of confusing. Seller couldn't PM his potential customers. But they couldn't PM him either. They must have figured out what the issue was. Neither party was ignoring each other on purpose.
Hard to lose a sale or buy a game when contact isn't possible.
Really interested parties would try again once Pinside was back up.
LTG : )
I didn't lose a sale during the outage...I completed one. Anybody that hadn't traded contact info by the time of the outage wasn't a serious buyer (yet), so you're right back where you started when the site came back up.
Quoted from PinMonk:My first question on learning what happened was "What kind of datacenter uses WATER as a fire suppression choice instead nitrogen or another non-damaging non-electrically conducting gas?"
Crazy. We all learned something here.
https://datacenterfrontier.com/generator-catches-fire-causes-lengthy-data-center-outage-at-webnx/
At a former job, the data center had nitrogen and water. They tested the plumbing one day and we found a giant puddle about 1 ft. away from a very critical server. A pipe leaked. Shit happens...
Good to see the only website that matters is back. Ok, there are a couple others too.
Keep up the great work!
Ah I see Achievements are down right now, and the streaks are paused! Now I’m slowly noticing what’s working and what’s not.
Robin, while you’re rebuilding the site, are you keeping some things offline or are you slowly finding out what’s broken and what’s not? I know this a monumental task, so I know patch notes don’t take priority, just figured I might ask!
Quoted from Isochronic_Frost:Ah I see Achievements are down right now
That was intentional. It was a resource intensive site feature that wasn't essential, so it was temporarily disabled.
Since it looks like things have been going well so far, I'm sure it will return soon.
Robin - Thank you for all your hard work on getting this site back up and running again! I've used various guides and advice from so many on this site, I'm kinda lost without it. Also thank you to everyone who's helped me learn throughout the process... I've missed all you as well.
Appreciate all the hard work and efforts getting the site back online. I am amazed at the resources the site provides, and see how important it is to so many. Thank you.
Quoted from PinMonk:My first question on learning what happened was "What kind of datacenter uses WATER as a fire suppression choice instead nitrogen or another non-damaging non-electrically conducting gas?"
Crazy. We all learned something here.
https://datacenterfrontier.com/generator-catches-fire-causes-lengthy-data-center-outage-at-webnx/
This is my job (Critical datacenters)...There is a reason why water suppression is used rather than gas. It's all about client risk, DC risk and cost.
Most of the DC's I've managed had a water based suppression system. Running a Tier III or Tier IV DC is extremely expensive, as well as gaseous fire suppression systems, and when customers want to pay the cheapest price possible w/o redundant sites, then, when an incident like this happens (it is a "rare" occurrence like a plane crash, but they do happen), there are some consequences.
Just last month, Europe's biggest cloud service DC company (OVH) has a catastrophic fire, which many clients have lost their data (anyone plays Rust? LOL).
Not putting any blame on the pinside crew in any way, shape or form. Redundant client sites/DC's/servers "doubles" the hosting costs. Again, it's all about cost, risk and impact. You would have had a better chance of winning the lottery than face a catastrophic DC failure.
Congrats on the admin crew to bounce back so quickly and bringing back pinside in a few days.... major kudos!!!
I had a couple racks in the old Enron data center in New Orleans during hurricane Katrina. We were up for a couple of days until the US Coast Guard confiscated our diesel supply on the roof for their recovery efforts! I was pissed.
Unless you’ve got Google bucks, things can and will happen.
Quoted from radium:Unless you’ve got Google bucks, things can and will happen.
Lot's of big companies out there with DR solutions they hope they never have to use.
Quoted from d0n:I "donated" about an hour before the site went down with my pinball for sale post. I immediately had people interested but couldn't msg them for 5-6 days due to the outage. Maybe pinside could donate to ME & my buyers for our inconvenience?
If you feel like that I think you need to leave. Let me make that easy for you, I will refund your donation if you leave and never come back. I myself don't need your messed up attitude.
Quoted from Isochronic_Frost:Ah I see Achievements are down right now, and the streaks are paused!
Quoted from ForceFlow:That was intentional. It was a resource intensive site feature that wasn't essential, so it was temporarily disabled.
Since it looks like things have been going well so far, I'm sure it will return soon.
I happened to notice that late yesterday and PM'ed Robin about it. I had a hunch something like that might be what was going on with the achievements.
Thank you for this great resource and to all of those who have worked hard to get it back up and running.
Is it "safe" to contribute at this point or should I wait until everything is completely back to normal? Thanks!
Jeff
Anyone else have issues with Hotmail? I’m not getting emails from my pm’s, and last time I think it was because Pinside was blacklisted or something.
Quoted from jeffro01:Is it "safe" to contribute at this point or should I wait until everything is completely back to normal? Thanks!
Jeff
The first thing I did when I noticed pinside was back up was update my donation. Easy peasy and no issues, donate with confidence!
Quoted from Mudflaps:Anyone else have issues with Hotmail? I’m not getting emails from my pm’s, and last time I think it was because Pinside was blacklisted or something.
I'm not sure if the email notification system is back up and running yet.
Quoted from ForceFlow:I'm not sure if the email notification system is back up and running yet.
I’m getting emails.
Ok, I just heard from Robin about the emails. It sounds like if you have a Microsoft email address (outlook, hotmail, etc), the notification emails aren't getting through right now, but the issue should hopefully be resolved soon.
Hey everyone. Thanks a million for all the nice words in this thread. Believe me, it's much appreciated!
Last week was really a nightmare scenario. Having moved over to a smaller Web Hosting company last year with the idea of getting more direct attention in case of network or hardware issues didn't work out quite like I had hoped.
So what exactly took me so damn long to get back online? I had backups right? Well, for anyone who's interested in that story, I'll try to recall at some time - but let's just say I wasn't prepared for this kind of disaster. Redundancy is cool, until your datacenter catches fire.
But of course I should have prepared for this, in hindsight. And I will be in the future as I'm working on a good DR strategy as we speak.
... waterdatacenter (resized).png
So what's cooking now?
I'm still working on getting everything back up. I got email fully working again today. Of course, being a new server and IP, Outlook immediately threw us on their blacklist. A few emails with their team and several hours later they mitigated this.
Got email replies working again!
Just got the Pinside maps server back online!
Currently working on Marketwatch emails.
And of course working on backup strategies.
Coming back soon: topic image gallery, Pinside achievements and anything else that is temporarily disabled or that I forgot. Not much else, I think? Let me know what you are missing besides the stuff I just mentioned.
Finally, a huge thanks to people who (offered) help/support. And my wife for being extremely patient with me as I was away most of the week and probably terrible to be around during the times I was at home. My brother Martijn of course for moral support. Forceflow. Jay@Scorbit, Jeff@TWIP for relaying messages that we didn't disappear to Timbuktu with all your gif uploads. And Ed R for being funny.
** And all of you who donated in the past 48 hours. Amazing to see all that support **
To be continued. good to be back!
Thanks for the news Robin!
In the list of little bugs, the Top 100 isn't refreshed on monday, unlike usual.
Quoted from robin:Hey everyone. Thanks a million for all the nice words in this thread......
You have done wonders getting the site back up! I was just going to post that PDF uploads in one of the forum posts wasn't displaying but I see that it has been fixed. Will keep my eyes open for other issues.
Although I'm relatively new to this space, I was surprised how much I missed the Pinside world! So sorry for the catastrophe @robin!! Your work and commitment is unbelievable and really truly appreciated!
Quoted from robin:So what exactly took me so damn long to get back online? I had backups right?
Robin, don't even worry about being down for a few days. It's a minor disturbance among our first world problems and everything appears to be fine.
I am apart of another forum that focuses on 1960's Shelby cars (SAAC Forum). Well, that site crashed 18 months ago and everything was lost. 15 years of online forum history, discussions and significant information regarding everything Shelby...... gone. There were no backups and nothing was recoverable so we lost it all. It was a damn shame.
You've done an amazing job on this site and loosing a couple of hours of Pinside history is insignificant. Consider what happened to my other beloved forum and just think how lucky we all are to have everything back up and running again with almost zero loss of content. Besides, a couple days without Pinside made me realize how much I appreciate this site. You know what they say, "absence makes the heart grow fonder". So true.
Great job Robin, great job.
QSS
Would like to speak out the appreciation to the team ! Of course robin ,it must have been stressful.. but on the bright side, you survived!! Big time..
As many others , I was amazed how much I checked the site.
But as always , as Johan Cruijf says.. elk nadeel heb zn voordeel ...
Donation will be made..
Quoted from QuickSilverShelby:Robin, don't even worry about being down for a few days. It's a minor disturbance among our first world problems and everything appears to be fine.
I am apart of another forum that focuses on 1960's Shelby cars (SAAC Forum). Well, that site crashed 18 months ago and everything was lost. 15 years of online forum history, discussions and significant information regarding everything Shelby...... gone. There were no backups and nothing was recoverable so we lost it all. It was a damn shame.
You've done an amazing job on this site and loosing a couple of hours of Pinside history is insignificant. Consider what happened to my other beloved forum and just think how lucky we all are to have everything back up and running again with almost zero loss of content. Besides, a couple days without Pinside made me realize how much I appreciate this site. You know what they say, "absence makes the heart grow fonder". So true.
Great job Robin, great job.
QSS
Wow that’s depressing. Pinside is 18, I shudder to think about losing all the precious resources and stories, vid’s guides, and especially all the archives of pinheads who have coined up that great pinball hall in the sky.
Their stories are maintained here and nowhere else, they’d never come back if we lost them. It’s sad to hear about all that Shelby data lost, hopefully some of it has been archived?
Thank YOU Robin for being prepared for the worst, Pinside has survived to see another day!
Quoted from Isochronic_Frost:Thank YOU Robin for being prepared for the worst, Pinside has survived to see another day!
I agree here, the story here is not that Pinside was down, but that it was really, really, really well backed up. And going forward it is going to be even more resilient. robin , thanks for creating such an enduring and important resource for our community!
Quoted from Azmodeus:Good to see the only website that matters is back. Ok, there are a couple others too.
Keep up the great work!
Cough cough youporn ciugh cough
Quoted from Isochronic_Frost:I shudder to think about losing all the precious resources and stories, vid’s guides, and especially all the archives of pinheads who have coined up that great pinball hall in the sky.
Their stories are maintained here and nowhere else, they’d never come back if we lost them.
You're absolutely right. You'll be pleased to hear that, going forward, Pinside is now running *hourly* data backups. These get stored in 3 separate datacenters by 3 different companies in 3 different geographic locations. Encrypted and safe, also from ransomware due to object locks.
And to give you another Shudder story: As I was restoring backups to my temporary server, I swear I could not remember where I had stored my data encryption key. Maybe it was the stress or what not, but it took me an hour of frantic searching. After having a coffee, it dawned on me where it was. I literally shouted out a relieved "Whoohoo!" when I found it. Another learning moment, ain't gonna happen to me again.
Quoted from robin:And to give you another Shudder story: As I was restoring backups to my temporary server, I swear I could not remember where I had stored my data encryption key. Maybe it was the stress or what not, but it took me an hour of frantic searching. After having a coffee, it dawned on me where it was. I literally shouted out a relieved "Whoohoo!" when I found it. Another learning moment, ain't gonna happen to me again.
Kind of like that guy who threw out his hard drive containing the encryption key to all his bitcoins.
https://www.cnn.com/2021/01/15/uk/bitcoin-trash-landfill-gbr-scli-intl/index.html
Quoted from Isochronic_Frost:Wow that’s depressing. Pinside is 18, I shudder to think about losing all the precious resources and stories, vid’s guides, and especially all the archives of pinheads who have coined up that great pinball hall in the sky.
Their stories are maintained here and nowhere else, they’d never come back if we lost them. It’s sad to hear about all that Shelby data lost, hopefully some of it has been archived?
Thank YOU Robin for being prepared for the worst, Pinside has survived to see another day!
Yes, it was and is depressing. So much history, resources and stories were lost on the SAAC site. This temporary shut down of Pinside was trivial and nothing of concern. Robin saved 99.9999% of everything on Pinside. We are so damn lucky to have this computer savvy pinball nerd running the show.
When I say "computer savvy pinball nerd" I mean that in the highest form of compliment. After all, that's what I am and most of the rest of us are. I consider myself, and most of all the forum members here, absolute "computer savvy pinball nerd, shit!!" ......... figuratively speaking of course.
QSS
Quoted from robin:...Coming back soon: topic image gallery, Pinside achievements and anything else that is temporarily disabled or that I forgot. Not much else, I think? Let me know what you are missing besides the stuff I just mentioned...
Seems like my "All posts made by you" activity is sort of 'hit or miss' w/respect to being current.
And thanks for all the hard work!
nice work Robin. Good decent backup time too - this is valuable resource so we appreciate it indeed.
EEVBLOG Dave in Aus, his forum went down and he made a video on it
Wanna join the discussion? Please sign in to reply to this topic.
Great to see you're enjoying Pinside! Did you know Pinside is able to run without any 3rd-party banners or ads, thanks to the support from our visitors? Please consider a donation to Pinside and get anext to your username to show for it! Or better yet, subscribe to Pinside+!
This page was printed from https://pinside.com/pinball/forum/topic/back-after-a-major-outage/page/5?hl=freddykaboodle and we tried optimising it for printing. Some page elements may have been deliberately hidden.
Scan the QR code on the left to jump to the URL this document was printed from.