Server down?

[Pouncy]

I got in on Torch and finished testing my mission in AE.  not gona to do work on it as not expecting any changes to be saved.  THey might, but not expecting.

[Yoru-hime]

Just posted:

 

Telephone — Today at 8:56 PM

@here All shards are available for play. We are still operating on backup networking capability and expect some degraded performance. Global services (such as the AH) may also be slightly less reliable than normal. We'll monitor everything closely and update you when we know more about the return of our primary networking.

[Qwerty MacDoogal]

And ty to all the devs and GMs and techie types who are working on this. Tyvm! ☺️

[0tis]

SO how do we buy the Devs that took care of this and kept us updated a beer or pizza or a weeks worth of coffee? I know several linux peeps let you buy them a beer or pizza etc. They deserve at least an att a boy for doing a great job!

[GraspingVileTerror]

@Telephone, what do you like on your pizzas?

[Bionic_Flea]

[Luminara]

16 minutes ago, GraspingVileTerror said:

@Telephone, what do you like on your pizzas?

 

[nancároFan]

When do donations open?

The last Saturday of each month at around 13:00 Eastern (click to view in your timezone), but sometimes it'll open a little later.

 

 

 

[JayboH]

Great work gang, hope you can get it back to normal once things are sorted.  For now, on to doing some respecs after offering my respects!

[Telephone]

2 hours ago, GraspingVileTerror said:

@Telephone, what do you like on your pizzas?

 

I like mine Detroit-style, with those delicious cupping pepperoni and lots of garlic.

[GraspingVileTerror]

You are a true savant and scholar, @Telephone.

[Nostromo21]

EXCELSIOR! 🤖

[Oklahoman]

Edited June 23, 2021 by Oklahoman

[Player2]

[RageusQuitus2]

7 hours ago, Telephone said:

 

I like mine Detroit-style, with those delicious cupping pepperoni and lots of garlic.

Damn right.  Detroit style is the best.

[Dahkness]

Brooklyn pie anytime anywhere. 

[laudwic]

19 hours ago, laudwic said:

Ok folks, I'm going to be home in around 4 hours.  The wife and the kids will be home in about 7 hours.  My basic understanding of math tells me that gives me around 3 hours to play.

 

No pressure. . . .

 

Well, it wasn't up in time.  I cleaned the kitchen, got started on the laundry.  My Wife was really happy with me.

 

Now, I need to know.  Is my wife conspiring with Nemesis?  This was a Nemesis Plot to get me to clean the kitchen, right?

 

Edited June 23, 2021 by laudwic

[Pouncy]

This speaks volumes on how well they protect our characters.  The crash happened while I was in my AE testing it. When , to use their terms, the"Ducktape Fix" was implemented and I got on just to check what kind of roll back I would be facing. I was right at the AE pillar facing as one would if they had exited. My contact was the one on the Holo. 

 

 

[UberGuy]

3 hours ago, Pouncy said:

This speaks volumes on how well they protect our characters.  The crash happened while I was in my AE testing it. When , to use their terms, the"Ducktape Fix" was implemented and I got on just to check what kind of roll back I would be facing. I was right at the AE pillar facing as one would if they had exited. My contact was the one on the Holo. 

 

In no way to diminish the efforts made to protect our characters and other server-side data, it makes sense that there was zero data loss with this issue. By all reports, this is a networking issue only, along with an unplanned reboot of the auth server. Basically everyone who was connected timed out or was booted, much like for the regular server restarts. It stands to reason that everyone would reappear in the same places (at least in the open world zones) that we were when things went south.

 

Backups (off-site ones in particular) would really only come into relevance if our servers were physically damaged or wiped clean. And I'm certain there would be data loss in such a case. Even the most aggressive backups are a snapshot in time, so there would be "lost" time between the event an the date/time of the last backup. But we would not lose everything, and most importantly, we would not lose things that had been around a long time.

[starro]

thank you homecoming.

 

Starro felt alone with only millions of his starfish minions to keep him company tuesday 

Edited June 24, 2021 by starro

[beerdetective]

REunion is down again.

[Pengy]

The launcher just claims it's down again.

[Telephone]

6 hours ago, UberGuy said:

Backups (off-site ones in particular) would really only come into relevance if our servers were physically damaged or wiped clean. And I'm certain there would be data loss in such a case. Even the most aggressive backups are a snapshot in time, so there would be "lost" time between the event an the date/time of the last backup. But we would not lose everything, and most importantly, we would not lose things that had been around a long time.

 

This is very true! For some more information about our database redundancy policy:

• We continuously synchronize all databases to a warm spare server (log shipping, for those of you who are familiar with SQL Server).
  • The log shipping cycle is 15 minutes. In a failure scenario we'd expect the warm spare to be no more than 30 minutes out of date.
• We do a full on-site backup every day to a separate machine.
• We do a full off-site backup every week.
  • 'Off-site' above means that we copy the backup to the EU cluster.
  • This backup is also copied to entirely separate storage not on OVH.
  • This backup includes AE, which can only be backed up when the cluster is down, so it's done during the weekly restart.

 

So in the event of a single database server failure, we'd expect to lose no more than 30 minutes of data. In the event of both the primary and warm spare failing, we'd expect to lose no more than a day's worth of data. If the entire cluster were to be destroyed, we'd expect to lose no more than a week.

 

Probably the next improvement we will make is to start synchronizing backups on a daily basis to EU from NA. Given that they are on multiple machines located in different physical locations in the NA data center, this wasn't considered an extremely high risk before, but that was before Strasbourg happened.

[fancy ketchup]

So I assume the servers are currently up as their status says.. however, I have had a black screen when opening Island Rum on my mac since they went down.

running  OSX 10.13.6 

Time to reinstall?