Jump to content
Arcanum's Spellbinding Holiday Giveaway! (12/14/24 - 12/25/24)
Holiday Support Closure: Dec. 22 - Dec. 29

Server down?

jaywashere

By jaywashere
in Help & Support

 Share

Recommended Posts

Recommended

Posted by Cipher,

Hey all,   Unfortunately this one is a bit out of our control. At around 1 AM UTC, we started experiencing severe packet loss and fluctuating latency across the private network (vRack) that our servers use to communicate. We've since put a support ticket in with OVH and have attempted to call anyone we could get ahold of at OVH to have this issue investigated. Bad news is, though, their automated response system is currently rejecting calls to all departments (incl. incident & outage
Recommended by Cipher
  • Thumbs Up
  • Sad
  • Thanks
  • Like

26 reactions

Go to this post
Pouncy

Pouncy

Posted
Posted

I got in on Torch and finished testing my mission in AE.  not gona to do work on it as not expecting any changes to be saved.  THey might, but not expecting.

An Ounce of Pounce is worth a Pound of Bounce.

Yoru-hime

Yoru-hime

Posted
Posted

Just posted:

 

Telephone — Today at 8:56 PM

@here All shards are available for play. We are still operating on backup networking capability and expect some degraded performance. Global services (such as the AH) may also be slightly less reliable than normal. We'll monitor everything closely and update you when we know more about the return of our primary networking.
  • Like 1
Qwerty MacDoogal

Qwerty MacDoogal

Posted
Posted

And ty to all the devs and GMs and techie types who are working on this. Tyvm! ☺️

  • Like 1
  • Thumbs Up 2

Qwerty MacDoogal 50+3 Fire/Emp Controller 1600 Badges

RadE8 50+3 Rad/Rad Tank 1599 Badges

V Idiot V 50+3 Dark/Staff Tank 1599 Badges

Peenutz 50+3 Kin/Rad Defender 1503 Badges

 

0tis

0tis

Posted
Posted

SO how do we buy the Devs that took care of this and kept us updated a beer or pizza or a weeks worth of coffee? I know several linux peeps let you buy them a beer or pizza etc. They deserve at least an att a boy for doing a great job!

  • Like 5
  • Thanks 1
  • Thumbs Up 4
JayboH

JayboH

Posted
Posted

Great work gang, hope you can get it back to normal once things are sorted.  For now, on to doing some respecs after offering my respects!

image.png.440bd3ba66421192ca1fb954c5d313c2.pngspacer.pngFlint Eastwood

  • City Council
Telephone

Telephone

Posted
  • City Council
Posted
2 hours ago, GraspingVileTerror said:

@Telephone, what do you like on your pizzas?

 

I like mine Detroit-style, with those delicious cupping pepperoni and lots of garlic.

  • Like 2
  • Thanks 3
  • Thumbs Up 5
laudwic

laudwic

Posted
Posted (edited)
19 hours ago, laudwic said:

Ok folks, I'm going to be home in around 4 hours.  The wife and the kids will be home in about 7 hours.  My basic understanding of math tells me that gives me around 3 hours to play.

 

No pressure. . . .

 

Well, it wasn't up in time.  I cleaned the kitchen, got started on the laundry.  My Wife was really happy with me.

 

Now, I need to know.  Is my wife conspiring with Nemesis?  This was a Nemesis Plot to get me to clean the kitchen, right?

 

Edited by laudwic
  • Haha 1
Pouncy

Pouncy

Posted
Posted

This speaks volumes on how well they protect our characters.  The crash happened while I was in my AE testing it. When , to use their terms, the"Ducktape Fix" was implemented and I got on just to check what kind of roll back I would be facing. I was right at the AE pillar facing as one would if they had exited. My contact was the one on the Holo. 

 

 

  • Like 2

An Ounce of Pounce is worth a Pound of Bounce.

UberGuy

UberGuy

Posted
Posted
3 hours ago, Pouncy said:

This speaks volumes on how well they protect our characters.  The crash happened while I was in my AE testing it. When , to use their terms, the"Ducktape Fix" was implemented and I got on just to check what kind of roll back I would be facing. I was right at the AE pillar facing as one would if they had exited. My contact was the one on the Holo. 

 

In no way to diminish the efforts made to protect our characters and other server-side data, it makes sense that there was zero data loss with this issue. By all reports, this is a networking issue only, along with an unplanned reboot of the auth server. Basically everyone who was connected timed out or was booted, much like for the regular server restarts. It stands to reason that everyone would reappear in the same places (at least in the open world zones) that we were when things went south.

 

Backups (off-site ones in particular) would really only come into relevance if our servers were physically damaged or wiped clean. And I'm certain there would be data loss in such a case. Even the most aggressive backups are a snapshot in time, so there would be "lost" time between the event an the date/time of the last backup. But we would not lose everything, and most importantly, we would not lose things that had been around a long time.

starro

starro

Posted
Posted (edited)

thank you homecoming.

 

Starro felt alone with only millions of his starfish minions to keep him company tuesday 

Edited by starro

Pineapple 🍍 Pizza 🍕 is my thumbs up. 

  • City Council
Telephone

Telephone

Posted
  • City Council
Posted
6 hours ago, UberGuy said:

Backups (off-site ones in particular) would really only come into relevance if our servers were physically damaged or wiped clean. And I'm certain there would be data loss in such a case. Even the most aggressive backups are a snapshot in time, so there would be "lost" time between the event an the date/time of the last backup. But we would not lose everything, and most importantly, we would not lose things that had been around a long time.

 

This is very true! For some more information about our database redundancy policy:

  • We continuously synchronize all databases to a warm spare server (log shipping, for those of you who are familiar with SQL Server).
    • The log shipping cycle is 15 minutes. In a failure scenario we'd expect the warm spare to be no more than 30 minutes out of date.
  • We do a full on-site backup every day to a separate machine.
  • We do a full off-site backup every week.
    • 'Off-site' above means that we copy the backup to the EU cluster.
    • This backup is also copied to entirely separate storage not on OVH.
    • This backup includes AE, which can only be backed up when the cluster is down, so it's done during the weekly restart.

 

So in the event of a single database server failure, we'd expect to lose no more than 30 minutes of data. In the event of both the primary and warm spare failing, we'd expect to lose no more than a day's worth of data. If the entire cluster were to be destroyed, we'd expect to lose no more than a week.

 

Probably the next improvement we will make is to start synchronizing backups on a daily basis to EU from NA. Given that they are on multiple machines located in different physical locations in the NA data center, this wasn't considered an extremely high risk before, but that was before Strasbourg happened.

  • Like 1
  • Thanks 6
  • Thumbs Up 4
fancy ketchup

fancy ketchup

Posted
Posted

So I assume the servers are currently up as their status says.. however, I have had a black screen when opening Island Rum on my mac since they went down.

running  OSX 10.13.6 

Time to reinstall?

  • 2 years later
Cadmus

Cadmus

Posted
Posted
On 6/22/2021 at 8:13 PM, Captain Valiant said:

All five servers show green, but we're unable to log in as yet; which is progress.....btrururr'

 

  • Confused 1

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...