SQL Host Hardware Upgrades

[Number Six]

1 minute ago, Doc_Scorpion said:

Just out of curiosity, do you recall the cap in the 2019 time frame?  When I finally came back in Aug/Sept of that year, three red dots on Excelsior were pretty common.  IIRC, a combination of natural erosion of the playerbase and upgrading the hardware brought an end to that.  I don't recall seeing heavy loads after mid/late 2020 or so.

 

The red dots have nothing to do with the cap.

 

The red dots are hardcoded to appear once there are more than 1,100 players concurrent on the shard. That applies regardless of what the cap or queue threshold is set to. As far as I can tell that's always been the case and has never changed. Yellow dots appear above 250.

[wjrasmussen]

Things have been feeling more databasey today.

[Cipher]

28 minutes ago, wjrasmussen said:

Things have been feeling more databasey today.

 

The impending upgrade must have the old server shaking in its rack, we haven't migrated to the new host just yet 🙂

 

[Ironblade]

1 hour ago, BurtHutt said:

I don't have facts but will throw common sense at you (which is not common, clearly). If Live had 1500 players per shard and had 12 shards or so, they wouldn't survive. The sub fee from 1500 people per shard would not come close to covering their costs etc.

 

The HC capacity is far different than Live.

Stop and re-read what you wrote.  No one is even close to saying that live had 1500 players per shard.  You're confusing numbers of players online simultaneously with total number of paying customers.

[skoryy]

As a sysadmin quite fluent in the horrors of database administration, you have my sympathies.

 

Especially if it's Interbase.  Good Gawd Awmighty, you'd have my prayers.

[Skyhawke]

If watching the IT Crowd taught me anything, it's that this could be fixed by simply turning it off and on again. 

[starro]

Call me an old IT guy, but powersets, story arcs and new zones are nice, but infrastructure and stability is sexy as hell.  

[Tenkay Volts]

Thanks for all the insights! I love being able to hear what's going on behind the curtains, it's all fascinating to hear how it all ticks.

[AlpineTsar]

Tried Excelsior in 2019, was too laggy (prime times) so moved to indom, never had a lag until the "official announcement" news wave hit.

So good move HC!

[Wispur]

I clearly remember there being multiple instances of Atlas park on busy nights shortly after launch on Live.
2 instances were pretty common, and I feel like I remember a few times it even went as high as 4.
(but they may have had smaller thresholds early on, or I'm misremembering)

So Homecoming, as it is now, seems pretty inline with what I remember from my time playing early CoH in terms of population.

I definitely remember Rubberbanding was a HUGE problem, comparable with Asheron's Call's rubberbanding.
So all of these problems feel like nostalgia to me, lol.

[ZacKing]

Thanks for the efforts here.  Appreciate the work you've all put into this. 

[High_Beam]

On 1/24/2024 at 1:37 PM, Number Six said:

The red dots have nothing to do with the cap.

What about the three seashells.

[High_Beam]

On 1/24/2024 at 7:00 PM, starro said:

Call me an old IT guy, but powersets, story arcs and new zones are nice, but infrastructure and stability is sexy as hell.  

I know right.  I may have to set aside some carnival time with myself.

[Hyperstrike]

Well, considering you're hitting all three of the hardware "gotchas" (sufficient CPU (speed and total processing power), memory and solid state drives, here's hoping this upgrade will fix (or at least ameliorate) the issue.

And yes, I'm preaching to the choir.  But look at how much memory gets held onto (Properties/Memory - Minimum Server Memory) before clearing the DB.

If you're set to default 0 MB, this can cause slowdowns as the system tries to dump memory and can actually cause processes to hang.
Worse, with zero, it can sometimes cause the service to choke itself out (memory starvation),

Try 256/512/1024.

I've run into this on several clients and have gotten beneficial results.

If this doesn't appear to be your case, feel free to file my suggestion in the bit bucket.
 

[Skyhawke]

It's in progress! Here's to more stability! (Gotta have it somewhere in life, amiright?)

[BlackHorde22]

What's the largest factor to memory / data need?

Could it be the Architect with soooooo many created maps taking up all the memory and space?

[Lockely]

2 minutes ago, BlackHorde22 said:

What's the largest factor to memory / data need?

Could it be the Architect with soooooo many created maps taking up all the memory and space?

AE maps are small, even the very long ones. My very long story with nothing but custom critters and villain groups totals out at 357kb for the whole file.

[Jordisk]

I understand that this requires a shutdown of the system used for those two shards, but why does it shut down ALL the shards? Two of the shards are on one machine, the other two NA shards are on another, and then the EU shard is on a separate machine in (IRRC) Germany. Shouldn't you just be able to shut down one machine, upgrade/replace it, then run a reset to make sure it is connected properly and running?

[Story Archer]

On 1/24/2024 at 10:07 AM, Doc_Scorpion said:

Yeah, me too.  I've seen nothing on Excelsior that even remotely resembles what I regularly saw on Freedumb back in the day in terms of players and activity.  Not in invasions, not in GM hunts, not in the LFG channel, not anywhere.

The OP is correct that compared to the rest of the market CoX was a small population, niche game.  But I'm not buying the claim that Homecoming even approaches anything resembling the numbers back in the OG days.

The game is also quite different today, with SO many people doing nothing but power-levelling, farming or doing post-50 content... the servers have always seemed sparse (until recently) compared to back in the day. I can't remember the last time I saw or was able to participate in 2-4 man teams just running around doing missions when that used to be the order of the day.

[Story Archer]

On 1/30/2024 at 12:03 PM, Hyperstrike said:

Well, considering you're hitting all three of the hardware "gotchas" (sufficient CPU (speed and total processing power), memory and solid state drives, here's hoping this upgrade will fix (or at least ameliorate) the issue.
 

 

Gotta listen to a man who'll use 'ameliorate' in casual conversation. 🙂

[Lunar Ronin]

11 minutes ago, Jordisk said:

I understand that this requires a shutdown of the system used for those two shards, but why does it shut down ALL the shards? Two of the shards are on one machine, the other two NA shards are on another, and then the EU shard is on a separate machine in (IRRC) Germany. Shouldn't you just be able to shut down one machine, upgrade/replace it, then run a reset to make sure it is connected properly and running?

 

Because it's not just the two shards for one thing.  They're also performing weekly maintenance on the other three shards.

[Number Six]

16 minutes ago, Jordisk said:

I understand that this requires a shutdown of the system used for those two shards, but why does it shut down ALL the shards? Two of the shards are on one machine, the other two NA shards are on another, and then the EU shard is on a separate machine in (IRRC) Germany. Shouldn't you just be able to shut down one machine, upgrade/replace it, then run a reset to make sure it is connected properly and running?

 

Global services like the auction house, chat server, etc. work by the service itself connecting to each shard, and if one of the shards are down it tries to reconnect every few seconds. But it does this synchronously, freezing the entire process, and causes lag on the other shards if they are waiting for a reply from the service.

 

Also accountserver is not the most well written piece of software and instead tends to get really crashy if shards are down.

[Ridiculous Girl]

can someone explain this to me like i am a five year old? 😄

[Number Six]

29 minutes ago, BlackHorde22 said:

What's the largest factor to memory / data need?

Could it be the Architect with soooooo many created maps taking up all the memory and space?

 

No, on SQL it's simply having more buffer cache. It's about getting as much of the character database in-memory as possible to minimize I/O wait, which gets amplified in high load scenarios when the game's dbserver has a large amount of database transactions pending.

 

AE missions are on a completely different VM and don't even go into SQL.

[Number Six]

1 minute ago, Ridiculous Girl said:

can someone explain this to me like i am a five year old? 😄

 

Bigger server go faster.