Jump to content

Finances Deep Dive


Recommended Posts

  • Homecoming Team

Hey all,

 

It’s been a while since we last did a deep dive into Homecoming’s finances, and with our operating costs settling into a fairly stable amount each month again, I thought it would be a good time to go over our monthly invoices to show exactly where the money for each service is going.

 

Our latest invoices for each server are included at the bottom of each section. If you would like to see the invoices and skip the explanations, an album containing all of them is available here.

OVH (Main Infrastructure)

  • Total Monthly Cost: $4,645.44

Most of our hardware infrastructure is rented from OVH. As of the time of this post, we are currently renting the following on a monthly basis:

  • 16 dedicated servers
  • 4 SQL server licenses
  • 7 failover IP address blocks

Below is a detailed breakdown of the various servers, their purpose / role, and their cost. A full copy of our latest invoice from OVH with sensitive information redacted has been provided below as well.

 

All of our servers are from OVH’s Advance Dedicated Server lineup. Please note that their Advanced server offerings are occasionally updated, and some of the specific configurations that we are currently using are no longer available.

 

Every server we rent has a 1Gbps vRack upgrade at a cost of $23.00 per month, per server. This is an important upgrade that allows all of our servers to communicate over a VLAN with a full gigabit of bandwidth, which is necessary because of the amount of traffic that goes between our servers.

 

Our hardware infrastructure might seem expensive compared to some other setups, but this is largely because we’re supporting many more concurrent users and mapservers (missions, AE maps, zones, etc.), and each of our services are built with redundancy and stability as a core factor. This means that we generally have warm spares for each core part of our infrastructure that we are ready to migrate to with very short notice in the event of an outage or other issue.

 

See also: this amazing post by Telephone back in May.

Map Hosts

  • Total Monthly Cost: $2,484.93
  • Hosts (7): HC-MS-NA1, HC-MS-NA2, HC-MS-NA3, HC-MS-NA4, HC-MS-NA5, HC-MS-NA6, HC-MS-EU1

Our shared mapserver pool consists of 7 Advance-5 servers (6 servers in NA and one in EU). These host all the maps (every zone, mission, and AE map) running in the game.

 

Each server costs $354.99 per month and has two Xeon Gold 5118 CPUs (24c/48t total @ 2.3GHz), and 96GB of memory.

Database & Storage Servers

  • Total Monthly Cost: $724.96
  • Hosts (4): HC-CentralDB0, HC-CentralDB1, HC-CentralDB2, HC-CentralDBEU

We currently run 4 SQL Server 2017 database servers in a high-availability configuration (3 hot, 1 warm spare for redundancy). All of these servers run pretty high on their memory usage (typically at anywhere from 85-95% physical memory usage from the SQL servers).

 

HC-CentralDB0 is an Advance-3 with a single Xeon D-2141I CPU, 64GB of memory, and 2 2TB SSDs at a cost of $213.99 per month. This server acts as a warm spare for each of the three other database servers in addition to providing read-only access to the data from each shard for various parts of our tooling and backend services.

 

HC-CentralDB1 is an Advance-3 with a single Xeon D-2141I CPU and 64GB of memory at a cost of $160.99 per month. This server hosts the databases for the CoH global services (Auction House, Chat, Account, AE, etc.) in addition to the databases for Torchbearer and Indomitable.

 

HC-CentralDB2 is an Advance-3 with a single Xeon D-2141I CPU and 64GB of memory at a cost of $160.99 per month. This server hosts the databases for Excelsior and Everlasting, in addition to the databases for all beta, prelease, and development shards.

 

HC-CentralDBEU is an Advance STOR-2 in the European cluster with a single Xeon D-1541 (8c/16t @ 2.1GHz), 32GB of memory, 2 240GB SSDs, and 4 12TB HDDs (RAID, 220GB + 22TB of storage accessible) at a cost of $188.99 per month. This server hosts the database for Reunion in addition to acting as one of our backup targets and a general-purpose storage server.

Virtual Machine Hosts

  • Total Monthly Cost: $971.97
  • Hosts (3): HC-VM1, HC-VM2, HC-VM3

We currently use 3 Advance-4 servers running Windows Server Datacenter and Hyper-V to host all of our virtual machines. Each of these servers  has a single AMD EPYC 7351P CPU (16c/32t), 256GB of memory, and 2 TB SSDs at a cost of $323.99 per month each.

 

The virtual machines running on these servers are used for a variety of purposes, including:

  • The DBServers for each NA shard (1 hot per shard plus warm spares on each other host for redundancy)
  • The CoH auth servers (1 live, 2 warm spares for redundancy)
  • 3 domain controllers (in a high-availability configuration)
  • A Kubernetes cluster
  • Our version control and build infrastructure
  • The website and forums
  • The content delivery network (CDN)
  • Our development infrastructure
    • Including our beta and pre-release shards
  • Various other backend systems and tools

The Bastion Host (AKA “Citadel”)

  • Total Monthly Cost: $102.99
  • Hosts (1): HC-Citadel

We currently use an Advance-1 server with a single Xeon D-2123I (4c/8ct) and 32GB of memory as a bastion host. This server costs $102.99 per month.

Reunion’s DBServer

  • Total Monthly Cost: $102.99
  • Hosts (1): HC-DB-Reunion

We currently use an Advance-1 server in our European cluster as Reunion’s DBServer. This server has a single Xeon D-2123I (4c/8ct) and 32GB of memory, and costs $102.99 per month.

Licenses & Failover IP Addresses

  • Total Monthly Cost: $257.60

We currently have 4 SQL Server 2017 Web licenses (one for each of our SQL DB servers) at a cost of $64.40 each per month.

 

We also have 7 failover IP blocks allocated to various servers and virtual machines. OVH no longer bills monthly for active IP addresses, so we are no longer paying for these on a monthly basis.

 

See our latest invoice from OVH here: https://imgur.com/a/725kB5L

Paperspace

  • Total Monthly Cost: $25.00
  • Hosts (1): HC-Imagineer1

We currently rent a single Paperspace Air instance which is used as a rendering node for some of our tooling and future projects which will rely on the service. This instance is paid monthly at a cost of $22.00 per month for the base instance, and an additional $3.00 per month for a static IP address.

 

See our latest invoice from Paperspace here: https://imgur.com/a/3w9IiRL

External Services

  • Total Monthly Cost: $140.75

Google G-Suite

  • Total Monthly Cost: $121.02

We currently use Google G-Suite Business as our primary user email service, as one of our backup targets, and for collaboration using Google Drive and it’s suite of tools (Docs, Sheets, etc.). We have 9 users (some of which are “server” accounts) costing $108.00 per month and 1 active Google Voice number costing $13.02 per month.

 

See our latest invoices from Google here: https://imgur.com/a/Qwz44dI

Jira Cloud (Atlassian)

  • Total Monthly Cost: $10.00

We currently have a 10-user Atlassian Jira Cloud account which we previously used for project management and issue tracking. We have moved away from this platform in favor of other options and plan to eliminate this charge very soon.

 

See our latest invoice from Atlassian here: https://imgur.com/a/oneIiRr

Twilio

  • Total Monthly Cost: variable

We use Twilio’s programmable SMS service to setup shared Homecoming accounts (such as social media accounts, etc.) with 2-factor authentication. The cost of this service is $1.00 per month plus $0.0075 per SMS message received. Billing on Twilio is balance based and the minimum balance charge is $20.00. As of the time of this post, we still have $15.90 remaining balance.

 

See our latest invoice from Twilio here: https://imgur.com/a/FL5mf2d

“Security / Admin Software”

  • Total Monthly Cost: $9.73

If you’ve been paying attention to the monthly PayPal history we show in each Donations & Finances post you may have noticed there’s a monthly charge for which the name is redacted. As this charge is related to administration and security, we will not be disclosing the name of the product or service, or show the associated invoice.

  • Like 25
  • Thanks 17

Cipher

Homecoming Team

 

If you need help, please submit a support request here or use /petition in-game.

 

Got time to spare? Want to see Homecoming thrive? Consider volunteering as a Game Master!

Link to post
Share on other sites

Thanks for the breakdown Cipher.  It added some detail to the monthly statements that I peruse when I donate, but it all tracks.  Still happy to support the effort!  Especially one with this much transparency.

 

  • Like 4

What was no more, is REBORN!

Link to post
Share on other sites
  • Homecoming Team
2 minutes ago, EmperorSteele said:

So, maybe this was answered once already, but, I noticed that we're not paying for a lawyer anymore (and haven't been for a while). As I understand, this individual was a key player in the talks with NCSoft. Is there anything you can disclose about what's going on there?

We've got a fairly flexible arrangement with them and they tend to not bill us for smaller bits of work, and due to the covid-related slowdown in recent months there's only been smaller bits of work.

 

They are handling something a bit more substantial at the moment which likely will get billed - so expect to see that in the next donation round (or possibly the one after, they can be a bit slow with their billing!).

  • Like 5
  • Thanks 7

Got time to spare? Want to see Homecoming thrive? Consider volunteering as a Game Master!

Link to post
Share on other sites

YES!  This breakdown is wonderful, more like this!

 

:classic_cool:

 

PS:  Any idea what's been causing the jitter between Canadian OVH servers and Spectrum's Columbus OH and Greenville SC headends somewhere between 9PM and 10PM EDT daily lately?  It's usually a steady 43ms but lately it goes up to upwards of >500ms (you can definitely feel that in-game) for about 15 minutes solid and the goes away.  This seems to effect everyone (with Spectrum) on East Coast US when it happens, and at same time (we're usually on Discord, even though in different states). I'm thinking either something wonky with the IXP that might have to be escalated, ot may there some kind of scheduled thing y'all got going on with OVH.  I don't know many people with AT&T or Comcast to compare against, that I know of anyways.

Edited by agentx5
Link to post
Share on other sites
14 hours ago, Cipher said:

“Security / Admin Software”

  • Total Monthly Cost: $9.73

If you’ve been paying attention to the monthly PayPal history we show in each Donations & Finances post you may have noticed there’s a monthly charge for which the name is redacted. As this charge is related to administration and security, we will not be disclosing the name of the product or service, or show the associated invoice.

LIKELY STORY - SOMEONES GETTING RICH OFF THIS! /s

  • Like 1
  • Haha 7
Link to post
Share on other sites
  • Homecoming Team
On 9/30/2020 at 4:59 PM, FoulVileTerror said:

Thanks, @Buckets.

When you say "technical stuff," do you mean specifically regarding the server hardware et cetera, or technical stuff in a broader context?  Which technical fields are your area of expertise here, so we can ensure the questions are appropriate?

The technical team, we handle all of the infrastructure, from networking, servers, and security. Telephones post goes over a good amount of what we are involved in. A lot of what we do is to ensure stability and security for our players.

  • Thanks 3
Link to post
Share on other sites
  • 3 months later
On 9/29/2020 at 4:39 AM, CamdenJaxx said:

The transparency of the Homecoming team is nothing short of upstanding! Major kudos to you all!

Yes.

 

And the setup looks very nice and professional. Especially little things like the SMS account for 2 factor auth.

You really seem to have given this some thought.

The adventurous Space Janitor reporting for duty. Cleaning the universe since 1992 and Paragon City, the Rogue Isles and Praetoria since 2011.

BlueYellowRed.png.cffb9b692dd0484133ca1d9ee2c8c4ce.png

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...