Jump to content

Cipher

Homecoming Team
  • Content Count

    263
  • Joined

  • Last visited

  • Days Won

    7

Cipher last won the day on September 28

Cipher had the most liked content!

Community Reputation

721 Excellent
  1. Chris "Robotech Master" Meadows was a well-respected individual in so many communities, not the least of which was our own. Many players in our community knew of him as Robotech Master through his many top-notch guides and all of the help he has provided people here on the forums, on our Discord server, and back in live (if you didn't know, he was a very well-known player and guide author in live as well). To our team, he was also known as GM Cyclone and he was a valued member who will be greatly missed. Cyclone was always so proactive when it came to helping players, spending many nights roaming the city and finding ways to make the community a better place. He must have reset various broken events (especially Paladin!) hundreds of times. He cared so much about this community and really shined when he was brought onto the team; it's hard for me to imagine not having him around anymore. The world is truly a worse place without you, friend. May you rest in peace and forever be remembered through the great positive impact you had on this world, both real and virtual. I'll miss you.
  2. Hey all, It’s been a while since we last did a deep dive into Homecoming’s finances, and with our operating costs settling into a fairly stable amount each month again, I thought it would be a good time to go over our monthly invoices to show exactly where the money for each service is going. Our latest invoices for each server are included at the bottom of each section. If you would like to see the invoices and skip the explanations, an album containing all of them is available here. OVH (Main Infrastructure) Total Monthly Cost: $4,645.44 Most of our hardware infrastructure is rented from OVH. As of the time of this post, we are currently renting the following on a monthly basis: 16 dedicated servers 4 SQL server licenses 7 failover IP address blocks Below is a detailed breakdown of the various servers, their purpose / role, and their cost. A full copy of our latest invoice from OVH with sensitive information redacted has been provided below as well. All of our servers are from OVH’s Advance Dedicated Server lineup. Please note that their Advanced server offerings are occasionally updated, and some of the specific configurations that we are currently using are no longer available. Every server we rent has a 1Gbps vRack upgrade at a cost of $23.00 per month, per server. This is an important upgrade that allows all of our servers to communicate over a VLAN with a full gigabit of bandwidth, which is necessary because of the amount of traffic that goes between our servers. Our hardware infrastructure might seem expensive compared to some other setups, but this is largely because we’re supporting many more concurrent users and mapservers (missions, AE maps, zones, etc.), and each of our services are built with redundancy and stability as a core factor. This means that we generally have warm spares for each core part of our infrastructure that we are ready to migrate to with very short notice in the event of an outage or other issue. See also: this amazing post by Telephone back in May. Map Hosts Total Monthly Cost: $2,484.93 Hosts (7): HC-MS-NA1, HC-MS-NA2, HC-MS-NA3, HC-MS-NA4, HC-MS-NA5, HC-MS-NA6, HC-MS-EU1 Our shared mapserver pool consists of 7 Advance-5 servers (6 servers in NA and one in EU). These host all the maps (every zone, mission, and AE map) running in the game. Each server costs $354.99 per month and has two Xeon Gold 5118 CPUs (24c/48t total @ 2.3GHz), and 96GB of memory. Database & Storage Servers Total Monthly Cost: $724.96 Hosts (4): HC-CentralDB0, HC-CentralDB1, HC-CentralDB2, HC-CentralDBEU We currently run 4 SQL Server 2017 database servers in a high-availability configuration (3 hot, 1 warm spare for redundancy). All of these servers run pretty high on their memory usage (typically at anywhere from 85-95% physical memory usage from the SQL servers). HC-CentralDB0 is an Advance-3 with a single Xeon D-2141I CPU, 64GB of memory, and 2 2TB SSDs at a cost of $213.99 per month. This server acts as a warm spare for each of the three other database servers in addition to providing read-only access to the data from each shard for various parts of our tooling and backend services. HC-CentralDB1 is an Advance-3 with a single Xeon D-2141I CPU and 64GB of memory at a cost of $160.99 per month. This server hosts the databases for the CoH global services (Auction House, Chat, Account, AE, etc.) in addition to the databases for Torchbearer and Indomitable. HC-CentralDB2 is an Advance-3 with a single Xeon D-2141I CPU and 64GB of memory at a cost of $160.99 per month. This server hosts the databases for Excelsior and Everlasting, in addition to the databases for all beta, prelease, and development shards. HC-CentralDBEU is an Advance STOR-2 in the European cluster with a single Xeon D-1541 (8c/16t @ 2.1GHz), 32GB of memory, 2 240GB SSDs, and 4 12TB HDDs (RAID, 220GB + 22TB of storage accessible) at a cost of $188.99 per month. This server hosts the database for Reunion in addition to acting as one of our backup targets and a general-purpose storage server. Virtual Machine Hosts Total Monthly Cost: $971.97 Hosts (3): HC-VM1, HC-VM2, HC-VM3 We currently use 3 Advance-4 servers running Windows Server Datacenter and Hyper-V to host all of our virtual machines. Each of these servers has a single AMD EPYC 7351P CPU (16c/32t), 256GB of memory, and 2 TB SSDs at a cost of $323.99 per month each. The virtual machines running on these servers are used for a variety of purposes, including: The DBServers for each NA shard (1 hot per shard plus warm spares on each other host for redundancy) The CoH auth servers (1 live, 2 warm spares for redundancy) 3 domain controllers (in a high-availability configuration) A Kubernetes cluster Our version control and build infrastructure The website and forums The content delivery network (CDN) Our development infrastructure Including our beta and pre-release shards Various other backend systems and tools The Bastion Host (AKA “Citadel”) Total Monthly Cost: $102.99 Hosts (1): HC-Citadel We currently use an Advance-1 server with a single Xeon D-2123I (4c/8ct) and 32GB of memory as a bastion host. This server costs $102.99 per month. Reunion’s DBServer Total Monthly Cost: $102.99 Hosts (1): HC-DB-Reunion We currently use an Advance-1 server in our European cluster as Reunion’s DBServer. This server has a single Xeon D-2123I (4c/8ct) and 32GB of memory, and costs $102.99 per month. Licenses & Failover IP Addresses Total Monthly Cost: $257.60 We currently have 4 SQL Server 2017 Web licenses (one for each of our SQL DB servers) at a cost of $64.40 each per month. We also have 7 failover IP blocks allocated to various servers and virtual machines. OVH no longer bills monthly for active IP addresses, so we are no longer paying for these on a monthly basis. See our latest invoice from OVH here: https://imgur.com/a/725kB5L Paperspace Total Monthly Cost: $25.00 Hosts (1): HC-Imagineer1 We currently rent a single Paperspace Air instance which is used as a rendering node for some of our tooling and future projects which will rely on the service. This instance is paid monthly at a cost of $22.00 per month for the base instance, and an additional $3.00 per month for a static IP address. See our latest invoice from Paperspace here: https://imgur.com/a/3w9IiRL External Services Total Monthly Cost: $140.75 Google G-Suite Total Monthly Cost: $121.02 We currently use Google G-Suite Business as our primary user email service, as one of our backup targets, and for collaboration using Google Drive and it’s suite of tools (Docs, Sheets, etc.). We have 9 users (some of which are “server” accounts) costing $108.00 per month and 1 active Google Voice number costing $13.02 per month. See our latest invoices from Google here: https://imgur.com/a/Qwz44dI Jira Cloud (Atlassian) Total Monthly Cost: $10.00 We currently have a 10-user Atlassian Jira Cloud account which we previously used for project management and issue tracking. We have moved away from this platform in favor of other options and plan to eliminate this charge very soon. See our latest invoice from Atlassian here: https://imgur.com/a/oneIiRr Twilio Total Monthly Cost: variable We use Twilio’s programmable SMS service to setup shared Homecoming accounts (such as social media accounts, etc.) with 2-factor authentication. The cost of this service is $1.00 per month plus $0.0075 per SMS message received. Billing on Twilio is balance based and the minimum balance charge is $20.00. As of the time of this post, we still have $15.90 remaining balance. See our latest invoice from Twilio here: https://imgur.com/a/FL5mf2d “Security / Admin Software” Total Monthly Cost: $9.73 If you’ve been paying attention to the monthly PayPal history we show in each Donations & Finances post you may have noticed there’s a monthly charge for which the name is redacted. As this charge is related to administration and security, we will not be disclosing the name of the product or service, or show the associated invoice.
  3. Often times in CoH, and other games for that matter, bugs will present themselves in a way that seems fun or harmless ("hey look at this cool thing I figured out I can do; it doesn't seem to be affecting anything else!"). The reality to many of these bugs/reports is often a bit worse though, as many times what you're seeing is just the tip of a more deeply-rooted issue which could be affecting anything from performance and stability to a multitude of other systems. You would be surprised to see some of the things that we've found that seemed okay but were actually causing quite significant problems elsewhere. Informing us of the bug allows us the opportunity to investigate and hopefully fix the issue, ensuring that it's not going to become more of a problem down the road when other things are changed or implemented. If one result of the bug was something that the community has been wanting as a feature, then we should definitely look into making it so, just without being the result of an erroneous piece of code or data. That said, please do report any bugs that you find to us! It really does help us out tremendously in keeping the game stable. The Bug Hunter badge is even up for grabs (in some situations, depending on the bug) for players who help us identify these things!
  4. Donations are closed but there are still ways you can help! We've met our donation goal for the month, but if you're still looking for ways to help Homecoming out then check out our How You Can Help Homecoming thread! Thank you all for your wonderful support as always. Hello everyone! It’s the last Saturday of the month, and that means it’s time to go over our finances for the past and upcoming month. If you haven’t already, check out the Page 6 preview we posted a few days ago which also includes a link to Piecemeal’s Dev Diary. September 2020 Breakdown Last month we accepted $5,065.46 in donations, $378.10 over our target for the month. This overrun will be put towards the expenses for October. Overall, here is the breakdown of what September’s funds went towards: $4,645.45 to OVH for our hosting $121.15 for our Google G-Suite Business account $30.00 for our 1&1 IONOS domain & outbound email service This is the invoice we will be receiving for this service $25.00 for the Paperspace Air instance $10.00 for our Atlassian Jira Cloud instance (10-user) $9.73 for the security / administration software $1.62 to setup AWS SES as our new email provider October 2020 We do not have any additional charges or changes in October. The donation target for October 2020 is $4,422.52. Here is a breakdown of how the funds are allocated: $4,645.44 for our hardware infrastructure from OVH $121.02 for our Google G-Suite services $25.00 for our Air instance from Paperspace $10.00 for our Atlassian Jira (Cloud) service $9.73 for security / administration software Our donation system will be open as of the time of this post with a donation target of $4,422.52 and will automatically close once that target is met. As always, thank you for all of the support! - The Homecoming Team
  5. Hello everyone! It’s the last Saturday of the month, and that means it’s time to go over our finances for the past and upcoming month. If you haven’t already, check out the State of the Project post we made on August 12th for a peek at some updates we’ll be making in the near future. August 2020 Breakdown Last month we accepted $4,705.07 in donations, $202.20 over our target for the month. This overrun will be put towards the expenses for September. As mentioned last month, OVH has consolidated all of the new servers added in July into our invoice that charges on the first of each month. After a billing change to failover IPs, our monthly invoice was reduced by additional $45.00 per month, bringing our new total for OVH to $4,645.44 per month. We purchased some additional domains in August related to this news post which we will be putting into use in the near future. The total cost for the domains was $87.00, all of which was covered by overrun from the donation window. Overall, here is the breakdown of what August’s funds went towards: $4,645.44 to OVH for our hosting $121.02 for our Google G-Suite Business account $87.00 for domain purchases (NAME-CHEAP.com) $30.00 for our 1&1 IONOS domain & outbound email service $25.00 for the Paperspace Air instance $10.00 for our Atlassian Jira Cloud instance (10-user) $9.73 for the security / administration software September 2020 One of our domains is due for renewal early in September at a cost of $10.00. This will appear as an additional charge under 1&1 IONOS on our next invoice. As previously mentioned, our monthly invoice from OVH has been reduced by an additional $45.00 per month, bringing the new total down to $4,645.44. We do not have any other updates to our finances this month and we still have $163.83 in overrun from August which will be put towards reducing our donation target for September. The donation target for September 2020 is $4,687.36. Here is a breakdown of how the funds are allocated: $4,645.44 for our hardware infrastructure from OVH $121.02 for our Google G-Suite services $30.00 for outbound email and domain services from 1&1 IONOS We will be completing our migration from IONOS to AWS SES for outbound email this month. Once this migration is completed, this charge will be removed from our monthly expenses. $25.00 for our Air instance from Paperspace $10.00 for our domain renewal $10.00 for our Atlassian Jira (Cloud) service $9.73 for security / administration software The donation target has been met and donations are now closed. Thank you for the support! Our donation system will be open as of the time of this post with a donation target of $4,687.36 and will automatically close once that target is met. As always, thank you for all of the support! - The Homecoming Team
  6. Hello everyone! We do not have any changes or updates to our finances this month so things will be pretty straight-forward. July 2020 Breakdown Last month we accepted $4,857.11 in donations - $54.06 over our target for the month. Overall, here is the breakdown of what July’s funds went towards: $4,363.23 to OVH for our hosting $121.74 for our Google G-Suite Business account $30.00 for our 1&1 IONOS domain & outbound email service $25.00 for the Paperspace Air instance $10.00 for our Atlassian Jira Cloud instance (10-user) $9.73 for the security / administration software $4.74 for the DigitalOcean droplet we used to test experimental network changes August 2020 OVH will be consolidating the new servers we purchased in June into our primary invoice at the start of August bringing our monthly invoice to it’s new total. We still have $387.05 from July which will be put towards the expenses for August 2020. Otherwise, there are no changes or updates for the month of August. The donation target for August 2020 is $4,502.87. Here is a breakdown of how the funds are allocated: $4,693.44 for our hardware infrastructure from OVH $120.75 for our Google G-Suite services $30.00 for outbound email and domain services from 1&1 IONOS $25.00 for our Air instance from Paperspace $10.00 for our Atlassian Jira (Cloud) service $9.73 for security / administration software $1.00 for our Twilio programmable number Our donation system will be open as of the time of this post with a donation target of $4,502.87 and will automatically close once that target is met. As always, thank you for all of the support! - The Homecoming Team
  7. You should now have a list of the clubs you are currently in along the right sidebar when viewing the forum index. Allowing users to re-order the forum categories themselves is not something that Invision supports and unfortunately adding our own support for this would require quite a bit of development time and testing.
  8. All shards are now operational again. Thank you for your continued patience.
  9. Hello everyone! Over the past few weeks, we were able to configure, test, and put into production the hardware infrastructure changes outlined in last month’s finances post (and further detailed in this breakdown from Telephone). As a result, our monthly operating costs will be reduced by roughly $138.95 starting next month. June 2020 Breakdown Last month we accepted $6,692.37 in donations - $178.50 over our target for the month. The PayPal account statement for June (below) lists a number of $3.00 and $12.00 charges from OVH - these were small orders placed for additional IP addresses for the new hardware infrastructure and will be included with our invoices from OVH going forward. Additionally, we set up a Twilio account and purchased one programmable number to receive and route SMS messages for Homecoming accounts to the administration team to streamline usage of certain accounts. The cost to set up the account with an initial balance was $20.00 (as this is the minimum charge amount), from which the service will charge us based on usage. The cost for this will be $1.00 per month plus $0.0075 per message received, and the balance will automatically be recharged in $20.00 intervals each time it falls below $5.00. Finally, there is a $1.00 hold from Amazon Web Services. We are exploring alternatives (including AWS) for outbound emails as the forums have been having issues with the service that we currently use. Overall, here is the breakdown of what June’s funds went towards: $4,820.39 to OVH for our hosting + $1,364.94 in one-off expenses for the hardware infrastructure changes $131.59 to our primary law firm for services rendered $137.00 in total transferred to our Transferwise account to cover wiring fees $120.75 for our Google G-Suite Business account $40.00 for our 1&1 IONOS domain & outbound email service The $10.00 domain renewal got added to June’s invoice. This was charged to our bank account (rather than our PayPal account) in June and is shown in our PayPal history as the most recent withdrawal to the bank account. $25.00 for the Paperspace Air instance $20.00 to set up our Twilio account with an initial balance $10.00 for our Atlassian Jira Cloud instance (10-user) $9.73 for the security / administration software $5.00 for the DigitalOcean droplet we used to test experimental network changes July 2020 The expenses for July may be up to $485 lower than expected. This is due to the way that OVH bills new services in the month after they are added; and unfortunately, we aren’t informed of exactly where the cost will land until each invoice is produced. Any amount left over at the end of July will be put towards the donation target for August. We do not foresee any additional expenses in July and we still have $91.87 from June’s donation target which will be put towards the expenses for July. In total, the donation target for July 2020 is $4,803.05. Here is a breakdown of how the funds are allocated: $4,693.44 for our hardware infrastructure from OVH $120.75 for our Google G-Suite services $30.00 for outbound email and domain servers from 1&1 IONOS $25.00 for our Air instance from Paperspace $10.00 for our Atlassian Jira (Cloud) service $9.73 for security / administration software $5.00 for the minimal DigitalOcean droplet (to be removed in July) $1.00 for our Twilio programmable number Our donation goal has been met and donations have been closed. Thank you for your support! Our donation system will be open as of the time of this post with a donation target of $4,803.05 and will automatically close once that target is met. As always, thank you for all of the support! - The Homecoming Team
  10. We're aware of the outage affecting our manifest and our hosting provider (OVH) is currently working on the problem. Unfortunately, we do not have an ETA for resolution just yet but we'll let you all know when it's back online. Thank you for your patience 🙂
  11. Hello everyone! In the upcoming month, we will be making some changes to our hardware infrastructure in order to reduce our long-term operating costs, help us utilize our existing hardware more efficiently, and aid with future development. The up-front cost for these changes will be $1,316.94 broken up as: $971.97/month - purchase of 3x Advance-4 servers configured with: 256GB of memory 2x1920GB NVMe SSDs 1Gbps Private (vRack) bandwidth $344.97 one time setup fee These changes will allow us to remove the following hardware from our current infrastructure: 5x Advance-1 servers ($102.99 each) 2x Advance-2 servers ($108.99 each) 1x Advance-4 server ($377.99) Once the changes have been made and we have removed the older hardware from our infrastructure, the reduction to monthly operating costs will be $138.95. May 2020 Breakdown Last month we accepted $5,297.72 in donations - $123.40 over our target for the month. Here is the breakdown of what May’s funds went towards: $4,820.39 to OVH for our hosting $199.99 to GoDaddy for the 1-year renewal of the code signing certificate This is the $199.99 withdraw to bank account - this was not charged directly from our PayPal $132.00 for our UPS business address (6-month renewal) $120.69 for our Google G-Suite Business account $75.00 for our 6-month Invision Community software renewal $25.00 for the Paperspace Air instance $10.00 for our Atlassian Jira Cloud instance (10-user) $9.73 for the security / administration software $5.52 for a droplet on DigitalOcean which was used for testing network changes June 2020 We were left with a $49.53 deficit at the end of May due to changing invoice amounts; this amount will be included in the donation target for June to ensure that we maintain our $1,250 contingency fund. Additionally, we have received the latest invoice from our primary law firm totaling $131.59. In total, the donation target for June 2020 is $6,513.87. Here is a breakdown of how the funds are allocated: $4,820.39 to OVH for our hosting + $1,316.94 to make the aforementioned changes to our infrastructure $131.59 to our primary law firm for services rendered $120.69 for our Google G-Suite Business account $49.53 to recoup the deficit from May $30.00 for our 1&1 IONOS domain & outbound email service $25.00 for the Paperspace Air instance $10.00 for our Atlassian Jira Cloud instance (10-user) $9.73 for the security / administration software The donation target has been met and donations have been closed. Thank you for the support! Our donation system will be open as of the time of this post with a donation target of $6,513.87 and will automatically close once that target is met. As always, thank you for all of the support! - The Homecoming Team
  12. You can paste plaintext using Ctrl + Shift + V or by Ctrl + Right Clicking and clicking "Paste without formatting" 🙂
  13. Hello everyone! Homecoming is now officially 1 year old, and in 3 days City of Heroes will be 16 years old! In celebration of the anniversaries, @GM Miss is hosting two contests - a screenshot contest which is running until May 1st, and a costume contest on May 2nd. Additionally, players who log into the game during the month of May will receive the Resurgent badge, so don’t forget to log in on any characters that you want the badge on! For more information about the anniversaries and festivities, please see our latest Homecoming Server Update. April 2020 Breakdown Last month we accepted $6,583.43 in donations - $39.94 over our target for the month. Here is the breakdown of what the April funds went towards: $4,820.39 to OVH for our hosting $1,108.30 to our lawyers for their services through February 29th $300.00 for our Delaware LLC franchise tax “DELAWARE CORP &” in our PayPal history $125.00 for our Registered Agent service as required by Delaware state law “CORPORATE FILINGS LLC” in our PayPal history $101.79 for our Google G-Suite Business account + $12.59 for Google Voice $30.00 for our 1&1 IONOS domain & outbound email service $25.00 to Paperspace for an Air instance with a static IP address $10.00 for our Atlassian Jira Cloud instance (10-user) $9.73 for security / administration software $9.10 to Transferwise This is the wiring fee to pay our lawyers $5.00 for a license for the Plateia Bold font Used for some art assets “Payment to Steve Gardner” in our PayPal history - this is the author of the font May 2020 In April we received an unexpected $199.99 charge from GoDaddy for a 1-year renewal on a code signing certificate which was the result of an issue on GoDaddy’s side. If we are able to receive a refund for this charge then this amount will be put towards June’s operating expenses. Otherwise, our only additional expense for the month of May is $10.00 for our primary domain renewal. In total, the donation target for May 2020 is $5,174.32. Here is a breakdown of how the funds are allocated: $4,820.39 to OVH for our hosting $199.99 to GoDaddy for the 1-year renewal of the code signing certificate $114.38 for our Google G-Suite Business account This includes the $12.83 Google Voice line $30.00 to 1&1 IONOS for domain services & outgoing email $25.00 for the Paperspace Air instance $10.00 for our Atlassian Jira Cloud instance (10-user) $10.00 for our primary domain renewal $9.73 for the security / administration software Our donation goal has been met and donations are now closed. Thank you for your support! Our donation system will be open as of the time of this post with a donation target of $5,174.32 and will automatically close once that target is met. As always, thank you for the incredible support! We can’t wait to see everyone’s screenshots and costumes in the anniversary contests! - The Homecoming Team
  14. The issue started around 1200 UTC on April 12th but seems to have resolved itself now.
×
×
  • Create New...