Finances Deep Dive

[Cipher]

Hey all,

 

It’s been a while since we last did a deep dive into Homecoming’s finances, and with our operating costs settling into a fairly stable amount each month again, I thought it would be a good time to go over our monthly invoices to show exactly where the money for each service is going.

 

Our latest invoices for each server are included at the bottom of each section. If you would like to see the invoices and skip the explanations, an album containing all of them is available here.

OVH (Main Infrastructure)

• Total Monthly Cost: $4,645.44

Most of our hardware infrastructure is rented from OVH. As of the time of this post, we are currently renting the following on a monthly basis:

• 16 dedicated servers
• 4 SQL server licenses
• 7 failover IP address blocks

Below is a detailed breakdown of the various servers, their purpose / role, and their cost. A full copy of our latest invoice from OVH with sensitive information redacted has been provided below as well.

 

All of our servers are from OVH’s Advance Dedicated Server lineup. Please note that their Advanced server offerings are occasionally updated, and some of the specific configurations that we are currently using are no longer available.

 

Every server we rent has a 1Gbps vRack upgrade at a cost of $23.00 per month, per server. This is an important upgrade that allows all of our servers to communicate over a VLAN with a full gigabit of bandwidth, which is necessary because of the amount of traffic that goes between our servers.

 

Our hardware infrastructure might seem expensive compared to some other setups, but this is largely because we’re supporting many more concurrent users and mapservers (missions, AE maps, zones, etc.), and each of our services are built with redundancy and stability as a core factor. This means that we generally have warm spares for each core part of our infrastructure that we are ready to migrate to with very short notice in the event of an outage or other issue.

 

See also: this amazing post by Telephone back in May.

Map Hosts

• Total Monthly Cost: $2,484.93
• Hosts (7): HC-MS-NA1, HC-MS-NA2, HC-MS-NA3, HC-MS-NA4, HC-MS-NA5, HC-MS-NA6, HC-MS-EU1

Our shared mapserver pool consists of 7 Advance-5 servers (6 servers in NA and one in EU). These host all the maps (every zone, mission, and AE map) running in the game.

 

Each server costs $354.99 per month and has two Xeon Gold 5118 CPUs (24c/48t total @ 2.3GHz), and 96GB of memory.

Database & Storage Servers

• Total Monthly Cost: $724.96
• Hosts (4): HC-CentralDB0, HC-CentralDB1, HC-CentralDB2, HC-CentralDBEU

We currently run 4 SQL Server 2017 database servers in a high-availability configuration (3 hot, 1 warm spare for redundancy). All of these servers run pretty high on their memory usage (typically at anywhere from 85-95% physical memory usage from the SQL servers).

 

HC-CentralDB0 is an Advance-3 with a single Xeon D-2141I CPU, 64GB of memory, and 2 2TB SSDs at a cost of $213.99 per month. This server acts as a warm spare for each of the three other database servers in addition to providing read-only access to the data from each shard for various parts of our tooling and backend services.

 

HC-CentralDB1 is an Advance-3 with a single Xeon D-2141I CPU and 64GB of memory at a cost of $160.99 per month. This server hosts the databases for the CoH global services (Auction House, Chat, Account, AE, etc.) in addition to the databases for Torchbearer and Indomitable.

 

HC-CentralDB2 is an Advance-3 with a single Xeon D-2141I CPU and 64GB of memory at a cost of $160.99 per month. This server hosts the databases for Excelsior and Everlasting, in addition to the databases for all beta, prelease, and development shards.

 

HC-CentralDBEU is an Advance STOR-2 in the European cluster with a single Xeon D-1541 (8c/16t @ 2.1GHz), 32GB of memory, 2 240GB SSDs, and 4 12TB HDDs (RAID, 220GB + 22TB of storage accessible) at a cost of $188.99 per month. This server hosts the database for Reunion in addition to acting as one of our backup targets and a general-purpose storage server.

Virtual Machine Hosts

• Total Monthly Cost: $971.97
• Hosts (3): HC-VM1, HC-VM2, HC-VM3

We currently use 3 Advance-4 servers running Windows Server Datacenter and Hyper-V to host all of our virtual machines. Each of these servers  has a single AMD EPYC 7351P CPU (16c/32t), 256GB of memory, and 2 TB SSDs at a cost of $323.99 per month each.

 

The virtual machines running on these servers are used for a variety of purposes, including:

• The DBServers for each NA shard (1 hot per shard plus warm spares on each other host for redundancy)
• The CoH auth servers (1 live, 2 warm spares for redundancy)
• 3 domain controllers (in a high-availability configuration)
• A Kubernetes cluster
• Our version control and build infrastructure
• The website and forums
• The content delivery network (CDN)
• Our development infrastructure
  • Including our beta and pre-release shards
• Various other backend systems and tools

The Bastion Host (AKA “Citadel”)

• Total Monthly Cost: $102.99
• Hosts (1): HC-Citadel

We currently use an Advance-1 server with a single Xeon D-2123I (4c/8ct) and 32GB of memory as a bastion host. This server costs $102.99 per month.

Reunion’s DBServer

• Total Monthly Cost: $102.99
• Hosts (1): HC-DB-Reunion

We currently use an Advance-1 server in our European cluster as Reunion’s DBServer. This server has a single Xeon D-2123I (4c/8ct) and 32GB of memory, and costs $102.99 per month.

Licenses & Failover IP Addresses

• Total Monthly Cost: $257.60

We currently have 4 SQL Server 2017 Web licenses (one for each of our SQL DB servers) at a cost of $64.40 each per month.

 

We also have 7 failover IP blocks allocated to various servers and virtual machines. OVH no longer bills monthly for active IP addresses, so we are no longer paying for these on a monthly basis.

 

See our latest invoice from OVH here: https://imgur.com/a/725kB5L

Paperspace

• Total Monthly Cost: $25.00
• Hosts (1): HC-Imagineer1

We currently rent a single Paperspace Air instance which is used as a rendering node for some of our tooling and future projects which will rely on the service. This instance is paid monthly at a cost of $22.00 per month for the base instance, and an additional $3.00 per month for a static IP address.

 

See our latest invoice from Paperspace here: https://imgur.com/a/3w9IiRL

External Services

• Total Monthly Cost: $140.75

Google G-Suite

• Total Monthly Cost: $121.02

We currently use Google G-Suite Business as our primary user email service, as one of our backup targets, and for collaboration using Google Drive and it’s suite of tools (Docs, Sheets, etc.). We have 9 users (some of which are “server” accounts) costing $108.00 per month and 1 active Google Voice number costing $13.02 per month.

 

See our latest invoices from Google here: https://imgur.com/a/Qwz44dI

Jira Cloud (Atlassian)

• Total Monthly Cost: $10.00

We currently have a 10-user Atlassian Jira Cloud account which we previously used for project management and issue tracking. We have moved away from this platform in favor of other options and plan to eliminate this charge very soon.

 

See our latest invoice from Atlassian here: https://imgur.com/a/oneIiRr

Twilio

• Total Monthly Cost: variable

We use Twilio’s programmable SMS service to setup shared Homecoming accounts (such as social media accounts, etc.) with 2-factor authentication. The cost of this service is $1.00 per month plus $0.0075 per SMS message received. Billing on Twilio is balance based and the minimum balance charge is $20.00. As of the time of this post, we still have $15.90 remaining balance.

 

See our latest invoice from Twilio here: https://imgur.com/a/FL5mf2d

“Security / Admin Software”

• Total Monthly Cost: $9.73

If you’ve been paying attention to the monthly PayPal history we show in each Donations & Finances post you may have noticed there’s a monthly charge for which the name is redacted. As this charge is related to administration and security, we will not be disclosing the name of the product or service, or show the associated invoice.

[Abraxus]

Thanks for the breakdown Cipher.  It added some detail to the monthly statements that I peruse when I donate, but it all tracks.  Still happy to support the effort!  Especially one with this much transparency.

 

[EmperorSteele]

So, maybe this was answered once already, but, I noticed that we're not paying for a lawyer anymore (and haven't been for a while). As I understand, this individual was a key player in the talks with NCSoft. Is there anything you can disclose about what's going on there?

[Jimmy]

2 minutes ago, EmperorSteele said:

So, maybe this was answered once already, but, I noticed that we're not paying for a lawyer anymore (and haven't been for a while). As I understand, this individual was a key player in the talks with NCSoft. Is there anything you can disclose about what's going on there?

We've got a fairly flexible arrangement with them and they tend to not bill us for smaller bits of work, and due to the covid-related slowdown in recent months there's only been smaller bits of work.

 

They are handling something a bit more substantial at the moment which likely will get billed - so expect to see that in the next donation round (or possibly the one after, they can be a bit slow with their billing!).

[skoryy]

1 hour ago, Jimmy said:

They are handling something a bit more substantial at the moment which likely will get billed

 

[agentx5]

YES!  This breakdown is wonderful, more like this!

 

 

PS:  Any idea what's been causing the jitter between Canadian OVH servers and Spectrum's Columbus OH and Greenville SC headends somewhere between 9PM and 10PM EDT daily lately?  It's usually a steady 43ms but lately it goes up to upwards of >500ms (you can definitely feel that in-game) for about 15 minutes solid and the goes away.  This seems to effect everyone (with Spectrum) on East Coast US when it happens, and at same time (we're usually on Discord, even though in different states). I'm thinking either something wonky with the IXP that might have to be escalated, ot may there some kind of scheduled thing y'all got going on with OVH.  I don't know many people with AT&T or Comcast to compare against, that I know of anyways.

Edited September 29, 2020 by agentx5

[CamdenJaxx]

The transparency of the Homecoming team is nothing short of upstanding! Major kudos to you all!

[CrazzieRedd]

14 hours ago, Cipher said:

“Security / Admin Software”

• Total Monthly Cost: $9.73

If you’ve been paying attention to the monthly PayPal history we show in each Donations & Finances post you may have noticed there’s a monthly charge for which the name is redacted. As this charge is related to administration and security, we will not be disclosing the name of the product or service, or show the associated invoice.

LIKELY STORY - SOMEONES GETTING RICH OFF THIS! _^/s

[myksmith]

This is fantastic, not only to understand how our money is used, but as a shining example of how a cloud-hosted company should be run. 
Major kudos to the team.

[Buckets]

If there happen to be any questions regarding some of the technical stuff, Ill happily answer what I can.

[FoulVileTerror]

Thanks, @Buckets.

When you say "technical stuff," do you mean specifically regarding the server hardware et cetera, or technical stuff in a broader context?  Which technical fields are your area of expertise here, so we can ensure the questions are appropriate?

[Buckets]

On 9/30/2020 at 4:59 PM, FoulVileTerror said:

Thanks, @Buckets.

When you say "technical stuff," do you mean specifically regarding the server hardware et cetera, or technical stuff in a broader context?  Which technical fields are your area of expertise here, so we can ensure the questions are appropriate?

The technical team, we handle all of the infrastructure, from networking, servers, and security. Telephones post goes over a good amount of what we are involved in. A lot of what we do is to ensure stability and security for our players.

[RogerWilco]

On 9/29/2020 at 4:39 AM, CamdenJaxx said:

The transparency of the Homecoming team is nothing short of upstanding! Major kudos to you all!

Yes.

 

And the setup looks very nice and professional. Especially little things like the SMS account for 2 factor auth.

You really seem to have given this some thought.

[JayboH]

On 9/28/2020 at 8:56 PM, skoryy said:

 

 

Apparently not.  Is there any updates to these talks to NCSoft after all these years?

[Glacier Peak]

18 hours ago, JayboH said:

Apparently not.  Is there any updates to these talks to NCSoft after all these years?

 

[kelika2]

Man there really needs to be some kinda auto-lock feature for old threads.  Came here in a panic thinking the worst like 10k usd a month to keep things going due to the current time of this post financial climate.  happy to be disappointed but what the fuck 2 year old thread

[CienFuegos]

On 9/29/2020 at 6:17 AM, CrazzieRedd said:

LIKELY STORY - SOMEONES GETTING RICH OFF THIS! _^/s

i blame Nemisis

[GM Crumpet]

On 6/27/2022 at 4:47 PM, CienFuegos said:

i blame Nemisis

It's always Nemesis.

 

A bit of a necro thread this one, but the situation hasn't changed. Things are going as they have been going, finances are detailed every month as to where they go, and contrary to some peoples opinions, nobody is getting rich off this. We truly appreciate the people who donate every month, we couldn't do it without you. 

 

When we get any news on anything that is actually concrete then we will of course share it as soon as possible. Until then just enjoy the game, have fun and stop worrying we will be shut down tomorrow 🙂 

[JayboH]

Donations were detailed at some points partially going toward supposed lawyers to negotiate with NCSoft and being curious about how many more years people should donate money that partially goes to these negotiations is fairly natural, no?

[Ironblade]

4 hours ago, JayboH said:

Donations were detailed at some points partially going toward supposed lawyers to negotiate with NCSoft and being curious about how many more years people should donate money that partially goes to these negotiations is fairly natural, no?

No.  Well, perhaps the curiosity is natural but don't expect an answer.  And you certainly have zero right to demand an answer.  Any negotiations are confidential.

 

If you don't like how the server is being run, you don't have to donate.  It is 100% voluntary.  Me?  I donate on a regular basis.

[JayboH]

13 hours ago, Ironblade said:

No.  Well, perhaps the curiosity is natural but don't expect an answer.  And you certainly have zero right to demand an answer.  Any negotiations are confidential.

 

If you don't like how the server is being run, you don't have to donate.  It is 100% voluntary.  Me?  I donate on a regular basis.

I didn't demand anything, I don't even know where that idea came from.  Look at my last two posts in this thread, which you can tell are unedited. 

 

Asking is harmless.  It makes sense to ask and I am surprised more haven't asked before.  Are they on retainer?  Do they have any kind of estimate?  Are they working toward an agreement that benefits servers outside of HC, making it universal?  Is it a licensing deal?  etc.

[Safehouse]

16 hours ago, JayboH said:

I didn't demand anything, I don't even know where that idea came from.  Look at my last two posts in this thread, which you can tell are unedited. 

 

Asking is harmless.  It makes sense to ask and I am surprised more haven't asked before.  Are they on retainer?  Do they have any kind of estimate?  Are they working toward an agreement that benefits servers outside of HC, making it universal?  Is it a licensing deal?  etc.

I say this stressing I have no idea how this stuff works, but is it possible that they’ve reached a “gentleman’s agreement” to not have an agreement? Maybe “the talks” carry on in perpetuity without anything actually being set in stone between HC and NCSoft. That way NC holds onto their property while “allowing” people to carry on with the understanding that, should they “cross a line,” the hammer comes down? I don’t even know if that’s a possibility but I’m sometimes curious if it is.