We're back!

What a crazy week it has been for Pinside (and a stressful week for me, I might add) with our first major outage in the 20 years or so that Pinside exists. Sure, we've had hiccups but nothing like this. The cause: a power surge, followed by a fire in a generator, followed by sprinklers and our server going down. I wasn't there to see it, but I don't think it was pretty.

Anyways, we're running on a temporary server now, hosted all the way over in Germany (well, that's close for me). Please be nice to it, it's not as powerful as our previous one and there may be problems. Also, not everything functions yet: We have no e-mail notifications yet. And no maps. And probably a bunch of other stuff that doesn't work right now. But at least we're back. I'll be moving to a brand new server (and a different hosting company) later on but I don't want to rush that choice.

As for our data recovery procedure. Well there really wasn't one in place as you may have noticed. Sure I had dual power supplies, dual network uplinks, dual (raid) drives. But I would not have expected a datacenter fire. I should have. I should have had a backup server somewhere, ready to spin up and continue service after a few hours. Truth is, I never got around to setting this up. But I will obviously be working on this, so a downtime like this cannot happen again in the future.

My last database backup was dated 10:02pm, while the server went down 11:39pm. So we lost 1h37m of data. That means any posts, PMs or Marketplace ads placed during that time are gone. And even if they restore our server (they're still working on it but are now giving me an ETA of "weeks") I will not be bringing that data back as that would be a technical nightmare.

If you made a donation or marketplace payment, please contact me. I will also be proactively look into any payments made during that 1h37m and take action or contact you.

Speaking of donations. Feel free to make a donation, if you're happy that we're back!

Let's get the pinball talk going again and please bear with me while I try to keep this temporary server running nicely.

-- Robin

PS: Seeing any issues? Let me know!

Well, you all didn't take long to crash our temporary server. After 30-40 minutes it went down smoking. So I went back to the drawing board and ordered a brand new server. This should work, but please bear with me as I will be tweaking and tuning some more over the next week(s). The upside is that the new server is twice as powerful as the previous one, so with the right approach this could all end better than it started.

Anyways, if you're able to read this: welcome back!

As of now, Pinside MarketWatch™ notifications should be working again!

You're absolutely right. You'll be pleased to hear that, going forward, Pinside is now running *hourly* data backups. These get stored in 3 separate datacenters by 3 different companies in 3 different geographic locations. Encrypted and safe, also from ransomware due to object locks.

And to give you another Shudder story: As I was restoring backups to my temporary server, I swear I could not remember where I had stored my data encryption key. Maybe it was the stress or what not, but it took me an hour of frantic searching. After having a coffee, it dawned on me where it was. I literally shouted out a relieved "Whoohoo!" when I found it. Another learning moment, ain't gonna happen to me again.

Super quick update. All is going great with the new server!

The Pinside Top 100 is back to weekly updates.

I’m still working on the image server.

And don’t worry: Everything that was on Pinside before the outage will be back. I mean achievements and topic image gallery. I just need some more time while I first move to a new image system. More on that later!

Good to be back!

This issue should be fixed, Jasper.

“Posts by you” is a known issue for which I need some more time to get it optimized and working properly again.

Collection page bug is news to me, I’ll see if I can replicate it here.

Looking into it. The files are still present in backups so something weird's going on there...

Found the cause of some images not working: the resolution was higher that I would allow. This has been fixed but you may need to clear your cache to see it.

Could you contact me via PinsideMail and possibly include screenshots of what you are doing (and of the exact error message)? Thanks!

Seeing the same thing here, I will investigate why this is happening. Meanwhile, I found that tapping the score box will show the keyboard. So the workaround is to enter the score first, then tap the machine name field to enter the machine.

The 'not actually possible at the moment' is from when editing scores was not yet possible. I forgot to remove that but will do so

Whoah!! What happened to your fonts?

What do you mean by date markers? Do you mean the green "unread posts" boxes? Maybe log out and back in again.

This should be fixed now!

Interesting findings. I always figured Linux would be fine falling back to Open Sans font.

Maybe Pinside should use font embedding for our main font? Currently our font stack is font-family:"Verdana","Open Sans",sans-serif;. Although I don't think I can embed Verdana without paying a licensing fee.

Another idea could be to install the MS Core Fonts TTF package in your linux distro. It adds some basic fonts, including Verdana. This way you'll see Pinside as was intended.

But that's all font talk. Your issue is with the colors and I wonder if you are doing some kind of override causing this?

The "All posts made by you" problem should be fixed now. the page also loads really fast now, compared to the 3 sec loading time it took before (database query rewritten). Please take a look and let me see if returned results are as expected.

Yeah, this is related to achievements and I hope to bring it back soon.

Cool, thanks. I will fix that today.

Thanks, I'm aware and it's on my to-do list!

The upload button is there, but it's tiny on some devices. Fix coming!

Doing some research into this. Our new image server apparently increases file size on some gifs. I'm implementing some smart code today to always serve the smallest file (either processed or raw). Please also note that the first load of an animated gif (or any image resize for that matter) will require some (more) processing time. After that initial processing, the image is cached in our CDN and subsequent loads will be mucho faster for everyone else.

Quick update:

Image rotate should be functioning now. It isn't well tested, but should work. Please report any bugs!

The blue "Add Image(s)" button should now always be visible, also on Android phones

Gif images will now be automatically converted to WebP, if your browser accepts that format (Chrome, Firefox and the latest Safari on Big Sur). WebP gives a huge size reduction over gif! If your browser does not support WebP, I'm now leaving the gif image alone and serving it as uploaded (which could mean large file sizes!)

Example:

This animated image of my buddy Pim is 2,5 MB as a .gif file. But only 508 kB when served as WebP.

Pimball.GIF

They are, just not high priority right now. Sorry!