Jump to content
Name Release Activates November 1st at 9:30 PM Eastern ×

Server down?

jaywashere

By jaywashere
in Help & Support

 Share

Recommended Posts

Recommended

Posted by Cipher,

Hey all,   Unfortunately this one is a bit out of our control. At around 1 AM UTC, we started experiencing severe packet loss and fluctuating latency across the private network (vRack) that our servers use to communicate. We've since put a support ticket in with OVH and have attempted to call anyone we could get ahold of at OVH to have this issue investigated. Bad news is, though, their automated response system is currently rejecting calls to all departments (incl. incident & outage
Recommended by Cipher
  • Thumbs Up
  • Sad
  • Thanks
  • Like

26 reactions

Go to this post
Pouncy

Pouncy

Posted
Posted

I got in on Torch and finished testing my mission in AE.  not gona to do work on it as not expecting any changes to be saved.  THey might, but not expecting.

An Ounce of Pounce is worth a Pound of Bounce.

Link to comment
Share on other sites
Yoru-hime

Yoru-hime

Posted
Posted

Just posted:

 

Telephone — Today at 8:56 PM

@here All shards are available for play. We are still operating on backup networking capability and expect some degraded performance. Global services (such as the AH) may also be slightly less reliable than normal. We'll monitor everything closely and update you when we know more about the return of our primary networking.
  • Like 1
Link to comment
Share on other sites
0tis

0tis

Posted
Posted

SO how do we buy the Devs that took care of this and kept us updated a beer or pizza or a weeks worth of coffee? I know several linux peeps let you buy them a beer or pizza etc. They deserve at least an att a boy for doing a great job!

  • Like 5
  • Thanks 1
  • Thumbs Up 4
Link to comment
Share on other sites
Oklahoman

Oklahoman

Posted
Posted (edited)

47439673-15044761596224358_origin.png.d22ce4f23a3e955da5507edd9e3e7e54.png.d68dd988d8c2d3607bc701c1ca86afad.png

Edited by Oklahoman

Oklahoman, Okie, Vayne Glorious, Sooner Magic, Treehugging Wacko, Boy Band, etc

Farming Incarnate Salvage - 1 salvage roll every 15 minutes!  ||  Why NO TELLS to join your little MSR thing?

Using DEMORECORD To Find Who Is Sabotaging Lambda Badge Runs  ||  https://www.twitch.tv/oklahomancoh

Excelsior Bases: The Sooner State (OK-8602), Atlas Records (ROCK-29730), Generic Heroes (G-16581), Sooner Nation (SOONER-8490)

Link to comment
Share on other sites
laudwic

laudwic

Posted
Posted (edited)
19 hours ago, laudwic said:

Ok folks, I'm going to be home in around 4 hours.  The wife and the kids will be home in about 7 hours.  My basic understanding of math tells me that gives me around 3 hours to play.

 

No pressure. . . .

 

Well, it wasn't up in time.  I cleaned the kitchen, got started on the laundry.  My Wife was really happy with me.

 

Now, I need to know.  Is my wife conspiring with Nemesis?  This was a Nemesis Plot to get me to clean the kitchen, right?

 

Edited by laudwic
  • Haha 1
Link to comment
Share on other sites
Pouncy

Pouncy

Posted
Posted

This speaks volumes on how well they protect our characters.  The crash happened while I was in my AE testing it. When , to use their terms, the"Ducktape Fix" was implemented and I got on just to check what kind of roll back I would be facing. I was right at the AE pillar facing as one would if they had exited. My contact was the one on the Holo. 

 

 

  • Like 2

An Ounce of Pounce is worth a Pound of Bounce.

Link to comment
Share on other sites
UberGuy

UberGuy

Posted
Posted
3 hours ago, Pouncy said:

This speaks volumes on how well they protect our characters.  The crash happened while I was in my AE testing it. When , to use their terms, the"Ducktape Fix" was implemented and I got on just to check what kind of roll back I would be facing. I was right at the AE pillar facing as one would if they had exited. My contact was the one on the Holo. 

 

In no way to diminish the efforts made to protect our characters and other server-side data, it makes sense that there was zero data loss with this issue. By all reports, this is a networking issue only, along with an unplanned reboot of the auth server. Basically everyone who was connected timed out or was booted, much like for the regular server restarts. It stands to reason that everyone would reappear in the same places (at least in the open world zones) that we were when things went south.

 

Backups (off-site ones in particular) would really only come into relevance if our servers were physically damaged or wiped clean. And I'm certain there would be data loss in such a case. Even the most aggressive backups are a snapshot in time, so there would be "lost" time between the event an the date/time of the last backup. But we would not lose everything, and most importantly, we would not lose things that had been around a long time.

Link to comment
Share on other sites
starro

starro

Posted
Posted (edited)

thank you homecoming.

 

Starro felt alone with only millions of his starfish minions to keep him company tuesday 

Edited by starro

 

 

"She who lives by the cybernetic monstrosity powered by living coral, all too often dies by the cybernetic monstrosity powered by living coral."  -Doc Buzzsaw

Pineapple 🍍 Pizza 🍕 is my thumbs up. 

Link to comment
Share on other sites
  • City Council
Telephone

Telephone

Posted
  • City Council
Posted
6 hours ago, UberGuy said:

Backups (off-site ones in particular) would really only come into relevance if our servers were physically damaged or wiped clean. And I'm certain there would be data loss in such a case. Even the most aggressive backups are a snapshot in time, so there would be "lost" time between the event an the date/time of the last backup. But we would not lose everything, and most importantly, we would not lose things that had been around a long time.

 

This is very true! For some more information about our database redundancy policy:

  • We continuously synchronize all databases to a warm spare server (log shipping, for those of you who are familiar with SQL Server).
    • The log shipping cycle is 15 minutes. In a failure scenario we'd expect the warm spare to be no more than 30 minutes out of date.
  • We do a full on-site backup every day to a separate machine.
  • We do a full off-site backup every week.
    • 'Off-site' above means that we copy the backup to the EU cluster.
    • This backup is also copied to entirely separate storage not on OVH.
    • This backup includes AE, which can only be backed up when the cluster is down, so it's done during the weekly restart.

 

So in the event of a single database server failure, we'd expect to lose no more than 30 minutes of data. In the event of both the primary and warm spare failing, we'd expect to lose no more than a day's worth of data. If the entire cluster were to be destroyed, we'd expect to lose no more than a week.

 

Probably the next improvement we will make is to start synchronizing backups on a daily basis to EU from NA. Given that they are on multiple machines located in different physical locations in the NA data center, this wasn't considered an extremely high risk before, but that was before Strasbourg happened.

  • Like 1
  • Thanks 6
  • Thumbs Up 4
Link to comment
Share on other sites
  • 2 years later
Cadmus

Cadmus

Posted
Posted
On 6/22/2021 at 8:13 PM, Captain Valiant said:

All five servers show green, but we're unable to log in as yet; which is progress.....btrururr'

 

  • Confused 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...