City Council Cipher Posted September 28, 2020 City Council Share Posted September 28, 2020 Hey all, It’s been a while since we last did a deep dive into Homecoming’s finances, and with our operating costs settling into a fairly stable amount each month again, I thought it would be a good time to go over our monthly invoices to show exactly where the money for each service is going. Our latest invoices for each server are included at the bottom of each section. If you would like to see the invoices and skip the explanations, an album containing all of them is available here. OVH (Main Infrastructure) Total Monthly Cost: $4,645.44 Most of our hardware infrastructure is rented from OVH. As of the time of this post, we are currently renting the following on a monthly basis: 16 dedicated servers 4 SQL server licenses 7 failover IP address blocks Below is a detailed breakdown of the various servers, their purpose / role, and their cost. A full copy of our latest invoice from OVH with sensitive information redacted has been provided below as well. All of our servers are from OVH’s Advance Dedicated Server lineup. Please note that their Advanced server offerings are occasionally updated, and some of the specific configurations that we are currently using are no longer available. Every server we rent has a 1Gbps vRack upgrade at a cost of $23.00 per month, per server. This is an important upgrade that allows all of our servers to communicate over a VLAN with a full gigabit of bandwidth, which is necessary because of the amount of traffic that goes between our servers. Our hardware infrastructure might seem expensive compared to some other setups, but this is largely because we’re supporting many more concurrent users and mapservers (missions, AE maps, zones, etc.), and each of our services are built with redundancy and stability as a core factor. This means that we generally have warm spares for each core part of our infrastructure that we are ready to migrate to with very short notice in the event of an outage or other issue. See also: this amazing post by Telephone back in May. Map Hosts Total Monthly Cost: $2,484.93 Hosts (7): HC-MS-NA1, HC-MS-NA2, HC-MS-NA3, HC-MS-NA4, HC-MS-NA5, HC-MS-NA6, HC-MS-EU1 Our shared mapserver pool consists of 7 Advance-5 servers (6 servers in NA and one in EU). These host all the maps (every zone, mission, and AE map) running in the game. Each server costs $354.99 per month and has two Xeon Gold 5118 CPUs (24c/48t total @ 2.3GHz), and 96GB of memory. Database & Storage Servers Total Monthly Cost: $724.96 Hosts (4): HC-CentralDB0, HC-CentralDB1, HC-CentralDB2, HC-CentralDBEU We currently run 4 SQL Server 2017 database servers in a high-availability configuration (3 hot, 1 warm spare for redundancy). All of these servers run pretty high on their memory usage (typically at anywhere from 85-95% physical memory usage from the SQL servers). HC-CentralDB0 is an Advance-3 with a single Xeon D-2141I CPU, 64GB of memory, and 2 2TB SSDs at a cost of $213.99 per month. This server acts as a warm spare for each of the three other database servers in addition to providing read-only access to the data from each shard for various parts of our tooling and backend services. HC-CentralDB1 is an Advance-3 with a single Xeon D-2141I CPU and 64GB of memory at a cost of $160.99 per month. This server hosts the databases for the CoH global services (Auction House, Chat, Account, AE, etc.) in addition to the databases for Torchbearer and Indomitable. HC-CentralDB2 is an Advance-3 with a single Xeon D-2141I CPU and 64GB of memory at a cost of $160.99 per month. This server hosts the databases for Excelsior and Everlasting, in addition to the databases for all beta, prelease, and development shards. HC-CentralDBEU is an Advance STOR-2 in the European cluster with a single Xeon D-1541 (8c/16t @ 2.1GHz), 32GB of memory, 2 240GB SSDs, and 4 12TB HDDs (RAID, 220GB + 22TB of storage accessible) at a cost of $188.99 per month. This server hosts the database for Reunion in addition to acting as one of our backup targets and a general-purpose storage server. Virtual Machine Hosts Total Monthly Cost: $971.97 Hosts (3): HC-VM1, HC-VM2, HC-VM3 We currently use 3 Advance-4 servers running Windows Server Datacenter and Hyper-V to host all of our virtual machines. Each of these servers has a single AMD EPYC 7351P CPU (16c/32t), 256GB of memory, and 2 TB SSDs at a cost of $323.99 per month each. The virtual machines running on these servers are used for a variety of purposes, including: The DBServers for each NA shard (1 hot per shard plus warm spares on each other host for redundancy) The CoH auth servers (1 live, 2 warm spares for redundancy) 3 domain controllers (in a high-availability configuration) A Kubernetes cluster Our version control and build infrastructure The website and forums The content delivery network (CDN) Our development infrastructure Including our beta and pre-release shards Various other backend systems and tools The Bastion Host (AKA “Citadel”) Total Monthly Cost: $102.99 Hosts (1): HC-Citadel We currently use an Advance-1 server with a single Xeon D-2123I (4c/8ct) and 32GB of memory as a bastion host. This server costs $102.99 per month. Reunion’s DBServer Total Monthly Cost: $102.99 Hosts (1): HC-DB-Reunion We currently use an Advance-1 server in our European cluster as Reunion’s DBServer. This server has a single Xeon D-2123I (4c/8ct) and 32GB of memory, and costs $102.99 per month. Licenses & Failover IP Addresses Total Monthly Cost: $257.60 We currently have 4 SQL Server 2017 Web licenses (one for each of our SQL DB servers) at a cost of $64.40 each per month. We also have 7 failover IP blocks allocated to various servers and virtual machines. OVH no longer bills monthly for active IP addresses, so we are no longer paying for these on a monthly basis. See our latest invoice from OVH here: https://imgur.com/a/725kB5L Paperspace Total Monthly Cost: $25.00 Hosts (1): HC-Imagineer1 We currently rent a single Paperspace Air instance which is used as a rendering node for some of our tooling and future projects which will rely on the service. This instance is paid monthly at a cost of $22.00 per month for the base instance, and an additional $3.00 per month for a static IP address. See our latest invoice from Paperspace here: https://imgur.com/a/3w9IiRL External Services Total Monthly Cost: $140.75 Google G-Suite Total Monthly Cost: $121.02 We currently use Google G-Suite Business as our primary user email service, as one of our backup targets, and for collaboration using Google Drive and it’s suite of tools (Docs, Sheets, etc.). We have 9 users (some of which are “server” accounts) costing $108.00 per month and 1 active Google Voice number costing $13.02 per month. See our latest invoices from Google here: https://imgur.com/a/Qwz44dI Jira Cloud (Atlassian) Total Monthly Cost: $10.00 We currently have a 10-user Atlassian Jira Cloud account which we previously used for project management and issue tracking. We have moved away from this platform in favor of other options and plan to eliminate this charge very soon. See our latest invoice from Atlassian here: https://imgur.com/a/oneIiRr Twilio Total Monthly Cost: variable We use Twilio’s programmable SMS service to setup shared Homecoming accounts (such as social media accounts, etc.) with 2-factor authentication. The cost of this service is $1.00 per month plus $0.0075 per SMS message received. Billing on Twilio is balance based and the minimum balance charge is $20.00. As of the time of this post, we still have $15.90 remaining balance. See our latest invoice from Twilio here: https://imgur.com/a/FL5mf2d “Security / Admin Software” Total Monthly Cost: $9.73 If you’ve been paying attention to the monthly PayPal history we show in each Donations & Finances post you may have noticed there’s a monthly charge for which the name is redacted. As this charge is related to administration and security, we will not be disclosing the name of the product or service, or show the associated invoice. 28 23 If you need help, please submit a support request here or use /petition in-game. Got time to spare? Want to see Homecoming thrive? Consider volunteering as a Game Master! Link to comment Share on other sites More sharing options...
Abraxus Posted September 28, 2020 Share Posted September 28, 2020 Thanks for the breakdown Cipher. It added some detail to the monthly statements that I peruse when I donate, but it all tracks. Still happy to support the effort! Especially one with this much transparency. 5 What was no more, is REBORN! Link to comment Share on other sites More sharing options...
EmperorSteele Posted September 28, 2020 Share Posted September 28, 2020 So, maybe this was answered once already, but, I noticed that we're not paying for a lawyer anymore (and haven't been for a while). As I understand, this individual was a key player in the talks with NCSoft. Is there anything you can disclose about what's going on there? 2 Link to comment Share on other sites More sharing options...
City Council Jimmy Posted September 28, 2020 City Council Share Posted September 28, 2020 2 minutes ago, EmperorSteele said: So, maybe this was answered once already, but, I noticed that we're not paying for a lawyer anymore (and haven't been for a while). As I understand, this individual was a key player in the talks with NCSoft. Is there anything you can disclose about what's going on there? We've got a fairly flexible arrangement with them and they tend to not bill us for smaller bits of work, and due to the covid-related slowdown in recent months there's only been smaller bits of work. They are handling something a bit more substantial at the moment which likely will get billed - so expect to see that in the next donation round (or possibly the one after, they can be a bit slow with their billing!). 5 8 Got time to spare? Want to see Homecoming thrive? Consider volunteering as a Game Master! Link to comment Share on other sites More sharing options...
skoryy Posted September 29, 2020 Share Posted September 29, 2020 1 hour ago, Jimmy said: They are handling something a bit more substantial at the moment which likely will get billed 2 2 1 Everlasting's Actionette Also Wolfhound, Starwave, Blue Gale, Relativity Rabbit, and many more! Link to comment Share on other sites More sharing options...
agentx5 Posted September 29, 2020 Share Posted September 29, 2020 (edited) YES! This breakdown is wonderful, more like this! PS: Any idea what's been causing the jitter between Canadian OVH servers and Spectrum's Columbus OH and Greenville SC headends somewhere between 9PM and 10PM EDT daily lately? It's usually a steady 43ms but lately it goes up to upwards of >500ms (you can definitely feel that in-game) for about 15 minutes solid and the goes away. This seems to effect everyone (with Spectrum) on East Coast US when it happens, and at same time (we're usually on Discord, even though in different states). I'm thinking either something wonky with the IXP that might have to be escalated, ot may there some kind of scheduled thing y'all got going on with OVH. I don't know many people with AT&T or Comcast to compare against, that I know of anyways. Edited September 29, 2020 by agentx5 Link to comment Share on other sites More sharing options...
CamdenJaxx Posted September 29, 2020 Share Posted September 29, 2020 The transparency of the Homecoming team is nothing short of upstanding! Major kudos to you all! 2 Link to comment Share on other sites More sharing options...
CrazzieRedd Posted September 29, 2020 Share Posted September 29, 2020 14 hours ago, Cipher said: “Security / Admin Software” Total Monthly Cost: $9.73 If you’ve been paying attention to the monthly PayPal history we show in each Donations & Finances post you may have noticed there’s a monthly charge for which the name is redacted. As this charge is related to administration and security, we will not be disclosing the name of the product or service, or show the associated invoice. LIKELY STORY - SOMEONES GETTING RICH OFF THIS! /s 1 7 Link to comment Share on other sites More sharing options...
myksmith Posted September 29, 2020 Share Posted September 29, 2020 This is fantastic, not only to understand how our money is used, but as a shining example of how a cloud-hosted company should be run. Major kudos to the team. Link to comment Share on other sites More sharing options...
City Council Buckets Posted September 30, 2020 City Council Share Posted September 30, 2020 If there happen to be any questions regarding some of the technical stuff, Ill happily answer what I can. 1 2 Link to comment Share on other sites More sharing options...
FoulVileTerror Posted September 30, 2020 Share Posted September 30, 2020 Thanks, @Buckets. When you say "technical stuff," do you mean specifically regarding the server hardware et cetera, or technical stuff in a broader context? Which technical fields are your area of expertise here, so we can ensure the questions are appropriate? Link to comment Share on other sites More sharing options...
City Council Buckets Posted October 3, 2020 City Council Share Posted October 3, 2020 On 9/30/2020 at 4:59 PM, FoulVileTerror said: Thanks, @Buckets. When you say "technical stuff," do you mean specifically regarding the server hardware et cetera, or technical stuff in a broader context? Which technical fields are your area of expertise here, so we can ensure the questions are appropriate? The technical team, we handle all of the infrastructure, from networking, servers, and security. Telephones post goes over a good amount of what we are involved in. A lot of what we do is to ensure stability and security for our players. 3 Link to comment Share on other sites More sharing options...
RogerWilco Posted January 19, 2021 Share Posted January 19, 2021 On 9/29/2020 at 4:39 AM, CamdenJaxx said: The transparency of the Homecoming team is nothing short of upstanding! Major kudos to you all! Yes. And the setup looks very nice and professional. Especially little things like the SMS account for 2 factor auth. You really seem to have given this some thought. The adventurous Space Janitor reporting for duty. Cleaning the universe since 1992 and Paragon City, the Rogue Isles and Praetoria since 2011. Link to comment Share on other sites More sharing options...
JayboH Posted June 25, 2022 Share Posted June 25, 2022 On 9/28/2020 at 8:56 PM, skoryy said: Apparently not. Is there any updates to these talks to NCSoft after all these years? Flint Eastwood Link to comment Share on other sites More sharing options...
Glacier Peak Posted June 26, 2022 Share Posted June 26, 2022 18 hours ago, JayboH said: Apparently not. Is there any updates to these talks to NCSoft after all these years? I lead weekly Indom Badge Runs / A newer giant monster guide by Glacier Peak / A tour of Pocket D easter eggs! / Arena All-Star Accolade Guide! Best Post Ever.... Link to comment Share on other sites More sharing options...
kelika2 Posted June 26, 2022 Share Posted June 26, 2022 Man there really needs to be some kinda auto-lock feature for old threads. Came here in a panic thinking the worst like 10k usd a month to keep things going due to the current time of this post financial climate. happy to be disappointed but what the fuck 2 year old thread 1 Link to comment Share on other sites More sharing options...
CienFuegos Posted June 27, 2022 Share Posted June 27, 2022 On 9/29/2020 at 6:17 AM, CrazzieRedd said: LIKELY STORY - SOMEONES GETTING RICH OFF THIS! /s i blame Nemisis 1 Link to comment Share on other sites More sharing options...
Game Master GM Crumpet Posted June 30, 2022 Game Master Share Posted June 30, 2022 On 6/27/2022 at 4:47 PM, CienFuegos said: i blame Nemisis It's always Nemesis. A bit of a necro thread this one, but the situation hasn't changed. Things are going as they have been going, finances are detailed every month as to where they go, and contrary to some peoples opinions, nobody is getting rich off this. We truly appreciate the people who donate every month, we couldn't do it without you. When we get any news on anything that is actually concrete then we will of course share it as soon as possible. Until then just enjoy the game, have fun and stop worrying we will be shut down tomorrow 🙂 1 2 Link to comment Share on other sites More sharing options...
JayboH Posted June 30, 2022 Share Posted June 30, 2022 Donations were detailed at some points partially going toward supposed lawyers to negotiate with NCSoft and being curious about how many more years people should donate money that partially goes to these negotiations is fairly natural, no? Flint Eastwood Link to comment Share on other sites More sharing options...
Ironblade Posted June 30, 2022 Share Posted June 30, 2022 4 hours ago, JayboH said: Donations were detailed at some points partially going toward supposed lawyers to negotiate with NCSoft and being curious about how many more years people should donate money that partially goes to these negotiations is fairly natural, no? No. Well, perhaps the curiosity is natural but don't expect an answer. And you certainly have zero right to demand an answer. Any negotiations are confidential. If you don't like how the server is being run, you don't have to donate. It is 100% voluntary. Me? I donate on a regular basis. Originally on Infinity. I have Ironblade on every shard. - My only AE arc: The Origin of Mark IV (ID 48002) Link to the story of Toggle Man, since I keep having to track down my original post. Link to comment Share on other sites More sharing options...
JayboH Posted July 1, 2022 Share Posted July 1, 2022 13 hours ago, Ironblade said: No. Well, perhaps the curiosity is natural but don't expect an answer. And you certainly have zero right to demand an answer. Any negotiations are confidential. If you don't like how the server is being run, you don't have to donate. It is 100% voluntary. Me? I donate on a regular basis. I didn't demand anything, I don't even know where that idea came from. Look at my last two posts in this thread, which you can tell are unedited. Asking is harmless. It makes sense to ask and I am surprised more haven't asked before. Are they on retainer? Do they have any kind of estimate? Are they working toward an agreement that benefits servers outside of HC, making it universal? Is it a licensing deal? etc. Flint Eastwood Link to comment Share on other sites More sharing options...
Safehouse Posted July 1, 2022 Share Posted July 1, 2022 16 hours ago, JayboH said: I didn't demand anything, I don't even know where that idea came from. Look at my last two posts in this thread, which you can tell are unedited. Asking is harmless. It makes sense to ask and I am surprised more haven't asked before. Are they on retainer? Do they have any kind of estimate? Are they working toward an agreement that benefits servers outside of HC, making it universal? Is it a licensing deal? etc. I say this stressing I have no idea how this stuff works, but is it possible that they’ve reached a “gentleman’s agreement” to not have an agreement? Maybe “the talks” carry on in perpetuity without anything actually being set in stone between HC and NCSoft. That way NC holds onto their property while “allowing” people to carry on with the understanding that, should they “cross a line,” the hammer comes down? I don’t even know if that’s a possibility but I’m sometimes curious if it is. Liberty and Virtue server refugee. Everlasting resident. Main/Planned Characters: Astellus - Kinetic/Energy/Mu Scrapper (Magic) Plasmitar - Radiation/Energy/Flame Blaster (Science) Scionic - Psychic/Atomic/Soul Blaster (Mutation) Safehouse - Street Justice/Energy Aura Scrapper (Magic) Starshear - Energy/Atomic/Force Blaster (Science) Neonstar - Luminous/Luminous Peacebringer (Natural) Faerwald - Gravity/Energy/Psionic Dominator (Science) Fomalhaut - Rad/Rad Sentinel (Science) Link to comment Share on other sites More sharing options...
Lorgar Posted June 21 Share Posted June 21 Really cool you all do this! Thanks for the info and insight! Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now