December 2022 Donations & Finances

[Cipher]

Hello everyone!

 

As always with these posts, I’d like to drop a quick reminder that there are more ways to support Homecoming than just through donations. For a general overview of the different ways you can help Homecoming, check out this thread.

 

Our next donation window will be moved up to Saturday, December 17th, 2022.

 

Notice for upcoming changes to our monthly operating costs:

Unfortunately, OVH is increasing prices across most of their products and services. Because of this, starting December 1st, our infrastructure costs will increase by $547.55 (~%14) per month. While it may take some time, we are currently evaluating our options to reduce our monthly costs. We’ll share more as soon as we’ve had time to review everything, and we appreciate everyone's patience and understanding.

 

For more information about OVH’s pricing changes, please see their blog posts:

• OVHcloud price adjustments in 2022 and 2023
• Additional IPv4: new pricing

 

Here is a breakdown of our monthly operating costs effective December 1st:

Spoiler

Some information has been redacted such as license or internal names, a service we utilize for administration / security ($9.73/mo), and phone numbers.

 

 

For more information about our infrastructure, please see Telephone’s excellent posts:

• Mapserver Host Hardware Changes (May 2021)

• Infrastructure breakdown (June 2020)

 

---

Last Month: November 2022

 

Last month we accepted $4,446.18 in donations, $15.88 over our target for the month. As always, any additional funding remaining at the end of each month is put towards our operating expenses for the following month or additional expenses for the current month.

 

Here is the breakdown of our expenses for November:

• $4,031.87 for our hardware infrastructure from OVH
• $120.25 for our Google G-Suite services
• $100.00 for our certificate of good standing
• $25.00 for our Air instance from Paperspace
• $19.25 for the AWS EC2 instance serving our email
• $9.73 for security / administration software

 

---

This Month: December 2022

 

The only addition to our target this month outside of the aforementioned OVH pricing changes is a $105 6-month renewal for our forum software.

 

The donation target for December 2022 is $4,843.66. Here is a breakdown of how the funds are allocated:

• $4,579.42 for our hardware infrastructure from OVH
• $121.57 for our Google G-Suite services
• $105.00 to renew our forum software for 6 months
• $25.00 for our Air instance from Paperspace
• $19.25 for the AWS EC2 instance serving our email
• $9.73 for security / administration software

 

---

 

Our donation system will be open as of the time of this post with a donation target of $4,843.66 and will automatically close once that target is met.

 

- The Homecoming Tea

[Anker Steadfast]

Dropped $10 in the bucket, and I'm pleased to see the bar is already at 50% after only one ½ hour !  😁

[drdread]

Always happy to contribute.

[Glacier Peak]

Lights still on, donations continue! 

[Asacledhae]

Here I am, once again.

You will hopefully keep seeing my ugly emu face on these threads, as our City lives on.

Huge thanks to everyone who donated, and to our team, for keeping our City alive.

 

See you guys in-game!

[TorontoSunshine]

In like Flynn and donated. 

 

Villainy shall continue. 🙂

Edited November 27, 2022 by TorontoSunshine

[jblee]

Yay!! We finally got to donate once! Normally it's already closed when I get to look.

[ArtoriosRex]

A lot of folks might be afk because of Thanksgiving weekend... Hopefully there's no problem wrapping it up tomorrow. 

[MistressOhm]

Passed you $25, every little bit helps... 🙂

[BrandX]

Donated, now at 74%

[Hardship]

Happy to be able to donate this time! Woot!!

[MoggieCanada]

Donated $10 US (which is probably $13-something Cdn). Not too bad in terms of progress. I expected it to be not as far ahead for the following reasons:

 

* Thanksgiving (US)

* The Black Friday/Cyber Monday weekend.

* Increased cost of living that we all are grappling with.

 

 

 

Edited November 27, 2022 by MoggieCanada

[agentx5]

16 hours ago, Cipher said:

Notice for upcoming changes to our monthly operating costs:

Unfortunately, OVH is increasing prices across most of their products and services. Because of this, starting December 1st, our infrastructure costs will increase by $547.55 (~%14) per month. While it may take some time, we are currently evaluating our options to reduce our monthly costs. We’ll share more as soon as we’ve had time to review everything, and we appreciate everyone's patience and understanding.

Good grief... I know there's inflation to consider, but that coverall pricing seems to be grossly overcharging our community.  I have donated to help, but I strongly recommend taking time between now and early 2023, or maybe even merging some of the least populated shards if necessary (poor Torchbearer, I started there in 2019 but it's been looking pretty empty)


Some possible alternatives to maybe consider and get customized price quote on?

24 racks of m5a.xlarge Ohio-based servers costs about $2354 per month from Amazon AWS, but that's only if all of those are used which to be fair they won't be and the pricing is charged based on actual usage needs updated every 5 minutes.  So it it's a low server population, it then you don't pay for it -- which could be a game changer in cost savings while not compromising on a low latency experience.
https://aws.amazon.com/gamelift/pricing/

HostHavoc has been one I've been keeping an eye on as they have some of the best reviews on the 3rd party TrustPilot website with people frequently saying their price was fair and the customer support was above and beyond (especially for complicated game server setups).  One review that caught my eye was a that one of the HostHavoc admins communicated with a customer via Discord to basically re-code and fix them up with a working VPN remote access JSON, like at no additional charge.  There's multiple servers, especially on the East and West Coast US.  They do have scalable dedicated servers.
https://hosthavoc.com/dedicated-servers

The Atlanta location Fragnet server location has a whooping 7 ISPs (means many of the players would actually have less hops), comparable hardware to what your are using, VPN remote access for SQL databases, and has been used by Bohemia Interactive (for Day-Z and Altis Life) and the ESL for competition servers.  They do custom requests from what I have read on other 3rd party review sites, as well as in-person LAN setups for eSports (seems to be what they are known for).
https://fragnet.net/page/about#

Other popular hosts like https://scalacube.com/ are very Minecraft focused and don't seem to have the VPN feature we would want and need for CoH HomeComing.  In fact that seemed to be the majority of ones I looked into which I personally would NOT recommend for our community.  Reliability is super important, obviously.  And changing providers is like moving houses -- It can be a hassle, but could be worth it in the long-term.  And just like real estate, sometimes it's just good to shop around even if you decide not to move, because it's good to know what's out there options wise.

Regardless, I'm happy to donate my part to continue that bar past the 88.2% it's currently at as of writing this.
And I appreciate all the HC team and GMs do to keep the show rolling on for all of us!
Thank y'all!

Edited November 27, 2022 by agentx5

[MoggieCanada]

3 hours ago, agentx5 said:

(a lot of good points!)

I agree with the idea to merge shards. It was a given that eventually the population would start to decline once the novelty of having City of Heroes/Villains back wore off. There are only two very active shards and the other shards are an unnecessary expense. I'm sure the RPing nature of Everlasting will not interfere the ability of players who do not RP their ability to play the game.

[Glacier Peak]

Folks may have missed the discussion about shard merging not being relevant to cost saving in last month's donation thread, so I'm going to link it below:

 

https://forums.homecomingservers.com/topic/38673-november-2022-donations-finances/?do=findComment&comment=516629

[Astralock]

3 hours ago, agentx5 said:

Good grief... I know there's inflation to consider, but that coverall pricing seems to be grossly overcharging our community.  I have donated to help, but I strongly recommend taking time between now and early 2023, or maybe even merging some of the least populated shards if necessary (poor Torchbearer, I started there in 2019 but it's been looking pretty empty)

 

 

2 minutes ago, MoggieCanada said:

I agree with the idea to merge shards. It was a given that eventually the population would start to decline once the novelty of having City of Heroes/Villains back wore off. There are only two very active shards and the other shards are an unnecessary expense. I'm sure the RPing nature of Everlasting will not interfere the ability of players who do not RP their ability to play the game.

 

 

All four North American shards in Canada utilize the exact same hardware.  Having one North American shard costs the exact same amount as having four North American shards.  There would be no cost savings merging shards.  Per Number Six a couple of months ago, merging shards will not happen since there are no cost savings to be had, it would just create headaches with SG bases lost and name conflicts, and some people prefer playing on quiet shards.

[MoggieCanada]

2 minutes ago, Astralock said:

 

 

 

 

All four North American shards in Canada utilize the exact same hardware.  Having one North American shard costs the exact same amount as having four North American shards.  There would be no cost savings merging shards.  Per Number Six a couple of months ago, merging shards will not happen since there are no cost savings to be had, it would just create headaches with SG bases lost and name conflicts, and some people prefer playing on quiet shards.

I stand corrected on the expense.

[Monty Haull]

Made my donation.  Thanks for all that you do.

[Abraxus]

Once again, as always, my pleasure to contribute.

[Web]

Closed it out for the month, thank you for everything!

[TorontoSunshine]

3 hours ago, Glacier Peak said:

Folks may have missed the discussion about shard merging not being relevant to cost saving in last month's donation thread, so I'm going to link it below:

 

https://forums.homecomingservers.com/topic/38673-november-2022-donations-finances/?do=findComment&comment=516629

It's an honest mistake to make and mistakes are the best way to learn.

[starro]

it does not surprise me inflation creeping into COH too.  While insulated from salaries and wages I was curious why we did not see rising costs more in 2020 and 2021.  At work our utility bills have just exploded (water, electrical, gas, you name it)  while consumption has flat or declined in our buildings.  I cannot imagine what the bills would be like with 1/3 of the work force in an offices or how we ran bills pre 2020.  

 

Any-hoo thank you players, volunteers, game masters and all the people that put their heart, souls and free time into giving us a virtal city to play in each month.

[Jetpack]

wow closed already.

[Telephone]

On 11/27/2022 at 4:43 AM, agentx5 said:

Good grief... I know there's inflation to consider, but that coverall pricing seems to be grossly overcharging our community.  I have donated to help, but I strongly recommend taking time between now and early 2023, or maybe even merging some of the least populated shards if necessary (poor Torchbearer, I started there in 2019 but it's been looking pretty empty)


Some possible alternatives to maybe consider and get customized price quote on?

24 racks of m5a.xlarge Ohio-based servers costs about $2354 per month from Amazon AWS, but that's only if all of those are used which to be fair they won't be and the pricing is charged based on actual usage needs updated every 5 minutes.  So it it's a low server population, it then you don't pay for it -- which could be a game changer in cost savings while not compromising on a low latency experience.
https://aws.amazon.com/gamelift/pricing/

HostHavoc has been one I've been keeping an eye on as they have some of the best reviews on the 3rd party TrustPilot website with people frequently saying their price was fair and the customer support was above and beyond (especially for complicated game server setups).  One review that caught my eye was a that one of the HostHavoc admins communicated with a customer via Discord to basically re-code and fix them up with a working VPN remote access JSON, like at no additional charge.  There's multiple servers, especially on the East and West Coast US.  They do have scalable dedicated servers.
https://hosthavoc.com/dedicated-servers

The Atlanta location Fragnet server location has a whooping 7 ISPs (means many of the players would actually have less hops), comparable hardware to what your are using, VPN remote access for SQL databases, and has been used by Bohemia Interactive (for Day-Z and Altis Life) and the ESL for competition servers.  They do custom requests from what I have read on other 3rd party review sites, as well as in-person LAN setups for eSports (seems to be what they are known for).
https://fragnet.net/page/about#

Other popular hosts like https://scalacube.com/ are very Minecraft focused and don't seem to have the VPN feature we would want and need for CoH HomeComing.  In fact that seemed to be the majority of ones I looked into which I personally would NOT recommend for our community.  Reliability is super important, obviously.  And changing providers is like moving houses -- It can be a hassle, but could be worth it in the long-term.  And just like real estate, sometimes it's just good to shop around even if you decide not to move, because it's good to know what's out there options wise.

Regardless, I'm happy to donate my part to continue that bar past the 88.2% it's currently at as of writing this.
And I appreciate all the HC team and GMs do to keep the show rolling on for all of us!
Thank y'all!

 

 

Thank you very much for the suggestion! Unfortunately, the way City of Heroes scales is not suitable for use with AWS GameLift. We can't simply start and stop mapserver hosts to scale with load, because scaling down would boot players off active map instances on those hosts (we can scale up relatively easily, however, though existing mapserver instances won't move to new hosts). In the very early days of Homecoming, we hosted on Digital Ocean and did some rudimentary scaling to help with load, but it was intensely manual and also resulted in booting players every time we scaled down.

 

Other important factors include:

• The mapserver hosts make extensive use of shared memory and hence fewer, larger hosts are much more efficient than many small hosts. The absolute minimum memory to run a Homecoming mapserver host is 24GB. Our beta and prerelease mapserver hosts have 32GB each; the NA mapserver hosts have 192GB each (but could run with less); the EU mapserver host has 96GB.
• The mapserver hosts do need a significant amount of CPU. They need to run all the game calculations, enemy AI, pathing, and so on, as well as some of the physics simulation.
• The mapserver hosts run Windows. We actually use an older version of Windows on the mapserver hosts to save on per-core licensing; this would be less feasible on AWS. It is possible to run the mapserver hosts with WINE, but we haven't done this on anything like Homecoming's scale.

 

AWS Dedicated Hosting

 

Our three existing NA mapserver hosts are much more comparable to a c6a.12xlarge each ($1,340/mo Linux / $2,952/mo Windows). They're actually even more powerful, but assuming we were to make some compromises, we would probably end up going with the c6a.8xlarge ($893/mo Linux / $1,973/mo Windows). EBS would add another $10/mo. Each mapserver host also transfers approximately 5TB/mo to the Internet, which is another $450/mo.

 

So not even including the rest of our cluster, the three NA mapserver hosts are already at between $4,059/mo (3 Linux 8x, and we'd have to run nested virtualization to use our existing Windows licenses, or swap to WINE-based mapservers) and $10,236/mo (3 Windows 12x). On top of this we'd need the rest of our cluster and then the SQL servers themselves.

 

Even if we were to really grit our teeth and try to get a minimal AWS NA cluster, here's how it would probably pan out:

- 3x Linux m6a.4xlarge mapserver hosts: $960/mo each including storage/bandwidth - $2,880/mo

- 2x Linux m6a.2xlarge SQL Server Web hosts: $375/mo each including storage - $750/mo

- 4x Linux m6a.large shard hosts: $80/mo including storage/bandwidth - $320/mo

- 1x Linux m6a.2xlarge services host: $140/mo including storage - $140/mo

- 1x Linux m6a.large auth host: $80/mo including storage/bandwidth - $80/mo

 

The above comes out to $4,170/mo and doesn't include beta, GM support, developer support, forums, backups, CDN, security services, and a host of other things. It could possibly be pared down a bit more, but AWS (and most VPS providers) are just not geared towards our usage profile.

 

 

HostHavoc

 

I looked over HostHavoc's offerings, and it looks like their dedicated servers are almost all out of stock. I also didn't see a way to acquire SQL Server licensing through them (though they might offer that as a custom option). They also don't seem to have very much variety in available hardware (though we could certainly work with the offered Ryzen hosts).

 

 

FragNet

 

FragNet doesn't seem to offer dedicated servers yet, though they do say 'Coming Soon'.

 

 

Other Options

 

Just for completeness, there are other options out there. Hetzner is a competitor in OVH's space, but doesn't have quite the right mix of servers for our needs. OVH also offers some lower tiers of servers (Rise), but those don't have the vRack functionality or flexibility that the Advance/Scale lines offer. OVH has their own resellers as well (Kimsufi and SoYouStart), but they are even more limited than the Rise line and whatever hardware is available is limited to servers OVH has decommissioned from their main lines.

 

Other VPS providers we could consider include Digital Ocean (as mentioned above, Homecoming was actually originally hosted there for a short while), Linode, and others. Most of them end up slightly to somewhat cheaper than AWS, but for our usage profile, we end up getting much less bang for our buck and still spending more than OVH.

 

 

Thanks again!

[MistressOhm]

Just to put a thought into brains about AWS - while OVH currently charges for hardware (run as many hosts on it as you like as long as you don't max out what the hardware can support), AWS often goes for usage, which means that the more people play, the more CPU cycles are consumed, the more network packets are transmitted, and the more RAM memory is occupied... the price goes up.  This WOULD force HC to scale down operations and consolidate servers, because a lot of people on a few hosts would be a cheaper overall cost than a few people on a lot of hosts and therefore being charged a larger base cost.  Neither one, however, would be a better deal than the OVH "use this hardware" cost.

 

Something else to consider: While Amazon does provide "close hop" connection to Content Distribution Networks (fat-pipe network providers like Level3, Convergys, and AT&T), they would charge a LOT of money for CDN edge connections in the multiple-terabit range.  And if HC had to branch out into another AWS superfarm (say, on the Eastern seaboard instead of just Oregon or San Francisco), any intra-network traffic between the servers would be crossing across the country, possibly internationally, and likely charged for it, whereas if they're in the same datacenter, no one's going to meter that nearly as closely - communicating via fiber to another server in the next rack is NOT an expensive thing at all, and any company that did charge by the gigabit for internal dataflow would soon be lacking in customers.