Sanctuary shut down

[Rishidian]

Unfortunately, if you have a Internet connected network, it's more WHEN than IF you are going to be hit.  And they may not be targeting you for gain.  It may be just practice.  A lot of the threat actors also hack for bragging rights.

 

As for Sanctuary, They may have been targeted, or it could have been street traffic.  If you look at the logs, those connecting IP's are from all over.  And what's more, they are probably all spoofed, so where they originated is most likely shrouded.  It's sad that we have to go to such extreme lengths to protect our systems.

 

I was trying to get a position with a local hospital group, and ended up giving up.  I found the hospital administration believed they were a health care provider, and not a technology company, and treated their CISSP like crap until he left.  They didn't even have an Executive Security officer. It's only a matter of time (watching other local hospital groups get hacked and having patient records stolen) before this way of thinking will be forcefully, painfully, and expensively, proven wrong.  Network security has to be a priority in any network worth looking at.

 

Something to think about:  The bad guys are working every day at their trade, and they have to succeed only once.  The good guys are working 8 to 5, and they have to succeed every time.

 

I feel bad for the Sanctuary folks, and I hope they recover.

[Snarky]

Someone needs to add a Task Force in game.  We have saved dams.  We have saved Nuclear power plants. We need to save gaming servers!!!!

 

The main culprit can be a "network" of Bosses and minions that all have to be wiped with X amount of time or they start recreating themselves.  

 

I start to see the issue.  I had heard that crypto currency mining is INSANELY lucrative and the requirements for energy to do it is actually increasing global warming.  I was like WTF?  Comps doing that?  But from what I understand they have to crank a truly stupid number of patterns per second to get rich.  And they will burn the planet (or some unfortunate group of gamers) or anything else that stands in their way to do it.  Villainous Capitalism at its purest.

[Clave Dark 5]

6 hours ago, MTeague said:

I am once again reminded that dogs are better people than a sizeable chunk of my own species.

I mean, dogs aren't exactly great conversationalists, and I'm sure if I was squirrel, I'd have a very dim view of dogs, too.

But if you think of servers as fire hydrants... well, dogs cause problems too and I don't care to follow this simile any further.

[Apparition]

FWIW, it sounds like Sanctuary was a complete loss.  However, the owner of Sanctuary is starting a new CoH server in its place, Ascension, and promises to take security more seriously this time around.

[Ironblade]

On 3/15/2021 at 6:48 PM, Faultline said:

If you run a server and you don't have a very robust backup strategy, you are risking losing everything out of nowhere every single day.

I would go further and say that, if you use a computer for anything more significant than social media, and you don't have a back-up system in place, you're an idiot.

 

I recall an audio recording I heard over 30 years ago.  It was a guy whose laptop had suddenly died calling the tech company he had sent it to for repairs.  They informed him that the hard drive had failed completely and the data was unrecoverable.  The guy lost it, screaming that his doctoral thesis was on that computer and that he had no back-ups.  "That's a year of my life", he screamed.  You could just tell that the tech guy wanted to reply, "Dude, you were using an antique computer and had zero back-ups.  What did you expect?"

 

People make an occasional back-up and have some spare hardware and think they're in good shape.  No.  Assume that your computer is going to catch fire one day, and possibly also burn the house down.  Prepare contingency plans accordingly.

[MTeague]

12 hours ago, Ironblade said:

I recall an audio recording I heard over 30 years ago.  It was a guy whose laptop had suddenly died calling the tech company he had sent it to for repairs.  They informed him that the hard drive had failed completely and the data was unrecoverable.  The guy lost it, screaming that his doctoral thesis was on that computer and that he had no back-ups.  "That's a year of my life", he screamed.  You could just tell that the tech guy wanted to reply, "Dude, you were using an antique computer and had zero back-ups.  What did you expect?"

 

I feel like the tech support guy should be offering up a haiku of some kind. 

https://forums.techguy.org/threads/haiku-windows-error-messages.81492/

 

I particularly like: 

 

Your file was so big.

It might be very useful.

But now it is gone.
 

[Doc_Scorpion]

1 hour ago, MTeague said:

I feel like the tech support guy should be offering up a haiku of some kind. 

https://forums.techguy.org/threads/haiku-windows-error-messages.81492/

As an occasional haiku poet...  *shudder*  Those are to haiku as microwave dinners are to cooking.

[srmalloy]

8 hours ago, MTeague said:

 

I feel like the tech support guy should be offering up a haiku of some kind. 

https://forums.techguy.org/threads/haiku-windows-error-messages.81492/

They're in the game:

[Herotu]

On 3/14/2021 at 1:19 AM, Snarky said:

Beyond the great sadness at fellow gamers being griefed I think it is important for us all to be aware.  To do what we can, even if it is just to stay informed.

 

First they came for someone in the neighborhood.  i did nothing.  Then for someone on my street.  i did nothing.  Then my next door neighbor.  I didnt know what to do.  Now they are here for me....

Sometimes it's best to turn a blind eye.

[Apparition]

Ascension (the server that was started to replace Sanctuary), has now shut down as well.  It sounds like there may have been data loss as well as the owner indicates that the host deleted all of the data.  You can read more about it here if you wish.  I hope that the Sanctuary/Ascension players find a new home quickly.

[TalonBlue]

@Faultline

 

Wow...that information is enlightening and scary. Especially, given the recent hacks in the U.S. Thank you for sharing this.

[GraspingVileTerror]

Hold on . . . "OVH" sounds familiar . . . 

https://forums.homecomingservers.com/topic/29090-mapserver-host-hardware-changes/

um . . . 

 

 . . . insufficient information.  Error!  Assumption of worst case scenario probability rising.

Please help!

[Faultline]

36 minutes ago, GraspingVileTerror said:

Hold on . . . "OVH" sounds familiar . . . 

https://forums.homecomingservers.com/topic/29090-mapserver-host-hardware-changes/

um . . . 

 

 . . . insufficient information.  Error!  Assumption of worst case scenario probability rising.

Please help!

 

Don't worry. OVH doesn't generally suspend accounts for no reason. I don't want to speculate on what the reasons are because the could be construed as an attack on Ascension's sysadmin, but even if OVH were to actually suspend HC out of nowhere for no reason and delete everything in our servers, it'd only take us a couple of days to find a new host and restore from off-site backups. The worst case scenario is not a server death for us, because we have contingency plans in place.

 

This does, once again, highlight the importance of backups. It's been two months since all the data on Sanctuary was lost, that should have been a big incentive to back up Ascension off-site every single day, from the start. Why they were operating without off-site backups after the giant wake-up call two months ago is beyond me.

[KauaiJim]

I just read through their posts and honestly it doesn’t make much sense. Three things stand out as odd to me:

1). OVH shutting down their server with no actual beforehand notice.

2)  OVH failing to give them an actual explanation for the shutdown. 
3)  OVH immediately deleting data with no backup or restore offered. 
 

First and foremost I am not blaming or calling out OVH. I’m simply trying to understand their business practices. Second, obviously the DDOSer’s are to blame for this nonsense, not OVH or the Ascension folks. This kind of stuff is very unfortunate but not uncommon. 
 

However the part I am ‘concerned’ about is could this occur with HC?  I’m not asking ‘theoretically’, rather is there something about targeted DDOS attacks and the way OVH handles them that maybe has HC exposed to this potentially happening as well. I would HOPE not. But reading that other thread did not give me much comfort. 
 

Dang @Faultline you are fast!  Thank you!  I feel a bit better. (For us here on HC, not for what happened to them)

 

Edited June 2, 2021 by KauaiJim

[Faultline]

26 minutes ago, KauaiJim said:

 I’m not asking ‘theoretically’, rather is there something about targeted DDOS attacks and the way OVH handles them that maybe has HC exposed to this potentially happening as well.

 

I'm tiptoing here because, again, I don't want to appear to attack the Ascension sysadmin, but citing a DDoS attack as a reason doesn't make sense. All OVH services include their Anti-DDoS solution. This is one of the things they explicitly, actively work to mitigate.

[DoctorDitko]

2 hours ago, Faultline said:

This does, once again, highlight the importance of backups.

 

Faultline, as I'm sure you know but didn't bother mentioning: Backups are great, A for effort. F for results unless: said backups are offline ('cause ransomware) and tested ('cause real life).

 

There is an ancient story (Aw, Doc! Not again?) about a software engineer who was hired to write a backup system for a major corporation. He did this, and they promptly fired him. (This, unfortunately, happens a lot.) One day, the corporation suffered a catastrophic hardware failure (all the hamsters died at once). "No worries!" cried the management, "We planned for this!" They went into the engineer's cobweb-laden cubicle and opened his files. And found the notes for his never-written restore system...

 

My sympathies lie with the Sanctuary admin, but it sounds like they are out of their depth. Next time, I suggest they publish their security setup in detail. If it's proper, the bad guys learn nothing useful from knowing it. If not, the good guys can suggest fixes. (Ask a cryptographer, they publish their source code. It's designed to be robust even if the attacker knows the entire source code.)

 

Disclaimer: I know little of Windows servers, I only run the desktop version, and only for games. (And with nothing but TCP/IP 4 enabled on the NIC!) My idea of securing a Windows server is to put it behind a locked-down Linux server!

 

But, still, sorry to hear of this, and I can only hope there was a script-kiddie with a grudge behind the attack. Good luck, admins!

[Hyperstrike]

5 hours ago, DoctorDitko said:

 

Faultline, as I'm sure you know but didn't bother mentioning: Backups are great, A for effort. F for results unless: said backups are offline ('cause ransomware) and tested ('cause real life).

 

There is an ancient story (Aw, Doc! Not again?) about a software engineer who was hired to write a backup system for a major corporation. He did this, and they promptly fired him. (This, unfortunately, happens a lot.) One day, the corporation suffered a catastrophic hardware failure (all the hamsters died at once). "No worries!" cried the management, "We planned for this!" They went into the engineer's cobweb-laden cubicle and opened his files. And found the notes for his never-written restore system...

 

My sympathies lie with the Sanctuary admin, but it sounds like they are out of their depth. Next time, I suggest they publish their security setup in detail. If it's proper, the bad guys learn nothing useful from knowing it. If not, the good guys can suggest fixes. (Ask a cryptographer, they publish their source code. It's designed to be robust even if the attacker knows the entire source code.)

 

Disclaimer: I know little of Windows servers, I only run the desktop version, and only for games. (And with nothing but TCP/IP 4 enabled on the NIC!) My idea of securing a Windows server is to put it behind a locked-down Linux server!

 

But, still, sorry to hear of this, and I can only hope there was a script-kiddie with a grudge behind the attack. Good luck, admins!

Translation.  Unless you've been able to SUCCESSFULLY restore from an archived file, YOU DO NOT HAVE A BACKUP!

 

As someone working in IT as a "Hail Mary" level of disaster support, I've seen horrendous data disasters at even carefully managed clients.

I've seen backup vendors simply stop backing up directories because there were live files in play and they didn't like "messy backup logs with "errors" in them".

Had a client come close to losing a year of data because of that.

When we found THAT out, the company was shown the door and they were sued into oblivion.

Currently I have two backups off my local machine.  This is the larger of the two backup devices.

The reason it's not fuller is because I'm currently still moving files and doing testing.  Rather than just plopping EVERYTHING over there and HOPING it works.

And, on top of that, using an internet backup tool.  With versioning.
Even so, this is the "cheapskate" option.

[Snarky]

13 hours ago, Apparition said:

Ascension (the server that was started to replace Sanctuary), has now shut down as well.  It sounds like there may have been data loss as well as the owner indicates that the host deleted all of the data.  You can read more about it here if you wish.  I hope that the Sanctuary/Ascension players find a new home quickly.

The only word I got here started with a F, and rhymes with luck.

 

The peeps that run this server are hardcore City of Heroes fans, of course.  As are all the people playing.  I am so terribly sad over this.

 

I am also grateful that Homecoming is here and has not been brought down by these types of situations.  I want to thank the hard working behind the scenes crew who give us Villains, Rogues, ...and the other guys I suppose, a place to call home.

[Marine X]

This is exactly when it becomes crystal clear that all the time and effort the Homecoming Team spends on extremely Unsexy Infrastructure updates is SO Important.

I have a Windows Server, ( no hard drive or OS ), sitting at my house that I got from work after our recent upgrade. I really plan to build it up and use it as a COH Local Client on my home Network for posterity, but have never got moving on it. HC IS SPOILING ME, LOL.

 

Sorry to hear about the other servers demise, but thanks to the Homecoming Team for all that you do to keep us running and oblivious.

[WanderingAries]

23 hours ago, Apparition said:

Ascension (the server that was started to replace Sanctuary), has now shut down as well.  It sounds like there may have been data loss as well as the owner indicates that the host deleted all of the data.  You can read more about it here if you wish.  I hope that the Sanctuary/Ascension players find a new home quickly.

 

Ouch

 

13 hours ago, Hyperstrike said:

Translation.  Unless you've been able to SUCCESSFULLY restore from an archived file, YOU DO NOT HAVE A BACKUP!

 

And this is why I always separate data backups as well.

[Novacat]

The attacks were relentless, both on Sanctuary, and once it returned within a few days of Ascension starting up.

[zenblack]

Any server you put online is going to be attacked almost immediately. Don't go online until you're ready to jump in those waters. I've seen servers have intrusions in less than an hour.

Edited June 4, 2021 by zenblack

[Snarky]

bitcoin farmers trying to create zombie mining nodes.  

[JnEricsonx]

Can I just say anyone who attacked people trying to run a beloved game for people's enjoyment can kiss my fat Irish ass.  And people wonder why my personal quote is "Eventually, cynicism becomes observation."    What does crapping on a server running COH do, other than increase some hackers e-peen?  Seriously.

[KauaiJim]

When I was an adolescent / younger man I did my share of stupid thoughtless stuff.  I had a good heart even then, but youth and foolishness (along with peer pressure) still had it's grip on me.  I'm talking about nonsense like prank (or crank if you prefer) phone calls, TP'ing houses, throwing eggs at cars, etc.  I have done some inconsiderate and even mean things myself.  When computers were first becoming popular when the 'internet' was still young and bulletin boards were still a thing, I was into trying to get into sites (primitive hacking by today's standards) just to see if I could.  I did no harm - it was mainly curiosity and ego but still, I had no business doing it OR bragging about it for attention.  I also used to crack games just to see if I could.  I did not distribute anything and I always owned a copy, but again it was more about just seeing if I could do it and gave little thought to whether I should do it.  Where am I going with this?  

 

My HOPE is that this is just some youngsters being stupid and inconsiderate and that some day they will come to realize their shenanigans were dumb and hurtful.  I turn 57 tomorrow.  I grew out of that stuff a long time ago.  But the last few years I have to say @JnEricsonx your personal quote is something I am beginning to struggle with myself.  "Eventually, cynicism becomes observation."  I sometimes feel this way.  I still believe people are mostly good however when I see this kind of thing happening so much - hacking, ddos'ing, encrypting critical software demanding ransom, bit coin mining bots, scam emails trying to steal folks retirement, etc. - it really is hard to not acknowledge that some folks are purposefully malevolent, evil and just plain cold hearted.

 

It's hard not to become more cynical as I get older.  But I still resist it.  I still try to find the good in things.  If not life begins to lose its color.  Its beauty.  Its joy.

 

So, for example, maybe what happened to Sanctuary/Ascension will result in something good - like everyone running these servers (those folks included) perhaps will take every precaution going forward (i.e. regular offsite backups that have been tested to be sure they work).  Ascension could have been up for longer and accumulated more players who invested more time building characters and bases, etc. only to have this happen on perhaps a larger scale some months from now.  I feel very bad for those folks.  I truly hope they try to put it back together and build it like a fortress this time, with a backup plan for their backup plan.  Bad people are going to keep on being bad people (at least for now).  I can't stop that.  But I can encourage good people to try to keep on being good people.

 

And as for those who are willingly, intentionally and gleefully being evil just for some kind of sick pleasure, well, I truly believe those folks are living on borrowed time and will eventually get what's coming to them if they don't make a hard 180.  That doesn't make me feel any better either (I don't wish bad on anyone) but it does allow me to forgive them so my heart doesn't fill up with poison from their deeds.  That doesn't mean though that I don't have the strong urge to kick these punks asses.  I just find that letting that feeling go (as best I can) is better than adding it to the LAKE of life's frustrations, because frankly, that dam is already under enough pressure with things that really do matter.

 

Sorry to go all 'confessiony' and 'moralizy' on you guys...  

Edited June 5, 2021 by KauaiJim