City Council Cipher Posted May 30, 2020 City Council Posted May 30, 2020 Hello everyone! In the upcoming month, we will be making some changes to our hardware infrastructure in order to reduce our long-term operating costs, help us utilize our existing hardware more efficiently, and aid with future development. The up-front cost for these changes will be $1,316.94 broken up as: $971.97/month - purchase of 3x Advance-4 servers configured with: 256GB of memory 2x1920GB NVMe SSDs 1Gbps Private (vRack) bandwidth $344.97 one time setup fee These changes will allow us to remove the following hardware from our current infrastructure: 5x Advance-1 servers ($102.99 each) 2x Advance-2 servers ($108.99 each) 1x Advance-4 server ($377.99) Once the changes have been made and we have removed the older hardware from our infrastructure, the reduction to monthly operating costs will be $138.95. May 2020 Breakdown Last month we accepted $5,297.72 in donations - $123.40 over our target for the month. Here is the breakdown of what May’s funds went towards: $4,820.39 to OVH for our hosting $199.99 to GoDaddy for the 1-year renewal of the code signing certificate This is the $199.99 withdraw to bank account - this was not charged directly from our PayPal $132.00 for our UPS business address (6-month renewal) $120.69 for our Google G-Suite Business account $75.00 for our 6-month Invision Community software renewal $25.00 for the Paperspace Air instance $10.00 for our Atlassian Jira Cloud instance (10-user) $9.73 for the security / administration software $5.52 for a droplet on DigitalOcean which was used for testing network changes June 2020 We were left with a $49.53 deficit at the end of May due to changing invoice amounts; this amount will be included in the donation target for June to ensure that we maintain our $1,250 contingency fund. Additionally, we have received the latest invoice from our primary law firm totaling $131.59. In total, the donation target for June 2020 is $6,513.87. Here is a breakdown of how the funds are allocated: $4,820.39 to OVH for our hosting + $1,316.94 to make the aforementioned changes to our infrastructure $131.59 to our primary law firm for services rendered $120.69 for our Google G-Suite Business account $49.53 to recoup the deficit from May $30.00 for our 1&1 IONOS domain & outbound email service $25.00 for the Paperspace Air instance $10.00 for our Atlassian Jira Cloud instance (10-user) $9.73 for the security / administration software The donation target has been met and donations have been closed. Thank you for the support! Our donation system will be open as of the time of this post with a donation target of $6,513.87 and will automatically close once that target is met. As always, thank you for all of the support! - The Homecoming Team 3 4 If you need help, please submit a support request here or use /petition in-game. Got time to spare? Want to see Homecoming thrive? Consider volunteering as a Game Master!
YourOlPalVoodoo Posted May 30, 2020 Posted May 30, 2020 The honor is to give! Thank you for your efforts. 1 Love, Your Ol' Pal Voodoo
Glacier Peak Posted May 30, 2020 Posted May 30, 2020 Keep up that fire HC Team! I lead weekly Indom Badge Runs / A newer giant monster guide by Glacier Peak / A tour of Pocket D easter eggs! / Arena All-Star Accolade Guide! Best Post Ever....
Asacledhae Posted May 30, 2020 Posted May 30, 2020 I'm here.. Again... And again... And again............ Made it, this month! Have a nice day.
archgemini24 Posted May 30, 2020 Posted May 30, 2020 Yay! Made it this time! Mostly on Torchbearer, but if you ever see me on, feel free to say hello! Astral.Kai - Peacebringer; Dark.Enforcer - Dark/Shield Scrapper; Spark.Enforcer - Electrical/Shield Scrapper; Shadow.Reign - Dark/Regen Brute; Glitter - Warshade; And others to be added as I get them up to snuff, lol!
Peerless Girl Posted May 30, 2020 Posted May 30, 2020 Awesome work as always guys, donating again for the month.
Kinematics Posted May 30, 2020 Posted May 30, 2020 Done - and thank you for all you do! Depicted in profile photo: Gamma Raze on Everlasting
nowherewoman Posted May 30, 2020 Posted May 30, 2020 Glad to finally be able to kick in- before the flood of 'Take Mah Moneh!' filled your coffers 😛
incantation Posted May 30, 2020 Posted May 30, 2020 I'm in for the month. Usually not up this early. lol
Destruktor Posted May 30, 2020 Posted May 30, 2020 How will the new hardware be divided up? Quite a few of us play on Reunion which as far as I know is still hosted in Europe. Is one of these upgrades relevant for this instance?
theDarkeOne Posted May 30, 2020 Posted May 30, 2020 Once I return to work I am happy to donate more but was able to make a small contribution this month because without you guys this lockdown would have driven me completely insane. 1
LaRowen Posted May 30, 2020 Posted May 30, 2020 Glad to give, I'm often at work on the last Saturday of the month (even now, in Zoom), and can't donate, but I made it in time!
Peerless Girl Posted May 30, 2020 Posted May 30, 2020 Going to presume one of those will be in Europe, and the other 2 in the US area, one probably for backend/dev stuff or beta servers. Just my guess. But the systems have so many available threads and cores they can divide them up a lot of ways.
Destruktor Posted May 30, 2020 Posted May 30, 2020 That was also my guess Peerless, but it would be nice to know.
Abraxus Posted May 30, 2020 Posted May 30, 2020 I missed the window for the first time in months. But, I'm happy that the enthusiasm for CoH, and Homecoming are still so high that the windows close this quickly! I'll get ya next month! 1 What was no more, is REBORN!
MornaInPink Posted May 30, 2020 Posted May 30, 2020 Donated! Thanks for all you do to keep us playing!!
Shadowmere Posted May 30, 2020 Posted May 30, 2020 Wow. I missed the window again lol. You guys are amazing! I'll get in on the donating one of these times I'm sure. :)
starro Posted May 30, 2020 Posted May 30, 2020 wow! was it done in about an hour? That was done while I was cutting the grass! "She who lives by the cybernetic monstrosity powered by living coral, all too often dies by the cybernetic monstrosity powered by living coral." -Doc Buzzsaw Pineapple 🍍 Pizza 🍕 is my thumbs up.
City Council Telephone Posted May 30, 2020 City Council Posted May 30, 2020 1 hour ago, Peerless Girl said: Going to presume one of those will be in Europe, and the other 2 in the US area, one probably for backend/dev stuff or beta servers. Just my guess. But the systems have so many available threads and cores they can divide them up a lot of ways. All three of these will actually be placed in our NA data center. Read on for more details! Reunion Reunion's already a bit of the odd one out - we've got a mini-cluster of three hosts in EU for it (the shard host itself, the SQL database host, and the mapserver host). The hardware there is a little bit overprovisioned for Reunion's actual load, but we've found a number of creative additional uses for it to use that spare capacity without affecting player experience (for example, the SQL host also has bulk storage attached and serves as one of our backup tiers and our WSUS update server in addition to its normal SQL duties). We don't have any plans to change the EU cluster at this time since doing so would probably be an increase in costs or decrease in flexibility without any gains. The A1s and the A4 When we originally built the Homecoming cluster a year ago, we allocated physical hardware for each shard's main host, as well as a separate host just for running authentication (the five A1s mentioned above). Since then, we've done a lot of optimization, fixing memory leaks, and profiling to determine how the cluster scales. In addition, all of our beta and prerelease shards (as well as our build servers) are running entirely on the single A4. So we're now a lot more comfortable running more of our game infrastructure on VMs. This will give us a lot more fault tolerance, since if one goes down we can temporarily run its VMs on the other two. Some of you may also recall the authentication issue a few months ago - that was an actual hardware failure on our physical authentication host, and we had to build a new one as a VM on the fly while that was being fixed. The A2s Not addressed yet are the two A2s - those are our general purpose VM hosts. While they're not struggling to handle the load, they're actually not as flexible as we had hoped: - They aren't very good at handling our build infrastructure (which wants lots of cores, but these only have six physical cores each). - We underestimated how much disk space they would need, so we occasionally have to juggle services to avoid a crunch. - Because two is no better than one in the world of high availability, we have to run backup instances of some of the services on these hosts on other hosts to ensure they are fully reliable. What's Not Changing Those of you who have kept up with our other technology posts may remember that we have a few other hosts in NA: - One more A1 which is used as a citadel host (most places would call this a bastion host, but this is CoH) - Three A3s which are used as SQL database hosts (two hot and one warm spare) - Six A5s which are used as mapserver hosts None of these are changing at this time. It's not that we're not confident of our ability to run the services of these hosts on VMs, but after analyzing the required capacity and the types of host we can purchase, we didn't feel there would be any significant cost savings while maintaining our desired reliability and performance criteria. The Future The net result is that the three new A4s will be running the following: - The authentication server (and warm spares) - The four NA live shard hosts (and we'll be able to quickly migrate if there is a failure) - All hosts for beta and prerelease (shard hosts, services hosts, and mapserver hosts) - Our forums - Our development infrastructure and build servers - Our other infrastructure (e.g. petitions/GM tools, system administration, patch and manifest management, CDN, our Kubernetes cluster). The hardware will be better than what we have now: - We'll have the same number of physical cores (48 - but note that the A4 cores are superior to the ones in the A1s/A2s, and also that many of our current cores are just not in places where it is convenient to use them) - We'll have more RAM (544GB -> 768GB). - We'll have more disk space in the right places (4.5TB -> 6TB). TL;DR - We'll reduce our footprint and long-term costs, increase our flexibility for future changes, and be more robust. - Nothing is changing in the EU cluster at this time. Finally, thank you all for donating! We wouldn't be here without you. 4 6
Destruktor Posted May 30, 2020 Posted May 30, 2020 Awesome 🙂 Thank you for the information, it is much appreciated.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now