Telephone

This is very true! For some more information about our database redundancy policy: We continuously synchronize all databases to a warm spare server (log shipping, for those of you who are familiar with SQL Server). The log shipping cycle is 15 minutes. In a failure scenario we'd expect the warm spare to be no more than 30 minutes out of date. We do a full on-site backup every day to a separate machine. We do a full off-site backup every week. 'Off-site' above means that we copy the backup to the EU cluster. This backup is also copied to entirely separate storage not on OVH. This backup includes AE, which can only be backed up when the cluster is down, so it's done during the weekly restart. So in the event of a single database server failure, we'd expect to lose no more than 30 minutes of data. In the event of both the primary and warm spare failing, we'd expect to lose no more than a day's worth of data. If the entire cluster were to be destroyed, we'd expect to lose no more than a week. Probably the next improvement we will make is to start synchronizing backups on a daily basis to EU from NA. Given that they are on multiple machines located in different physical locations in the NA data center, this wasn't considered an extremely high risk before, but that was before Strasbourg happened.

I like mine Detroit-style, with those delicious cupping pepperoni and lots of garlic.

Good artists create. Great artists steal. I'd have to have more details to answer this question, but I don't think this would be related to the CPU on the servers. It would most likely be related to network issues, issues with individual powers, and so on. If you mean true animation cancelling (where you can sneak a power in to break another power's animation, allowing you to execute more powers more quickly), this is generally considered a bug or balance issue and I believe @Captain Powerhouse has recently been extremely diligent in working to resolve these issues (but I am sure he will correct me if I am wrong). If you mean powers are failing to execute, that would most likely be a network or client issue. If the hosts are particularly loaded or (more likely) you are in a raid or otherwise CPU-saturated map, this would be a case that could be improved by the faster hardware. One thing I forgot to mention above is that mapserver processes are not heavily multi-threaded, so each map is generally limited to the power available from a single core (I believe some processing can run on other cores, but the bulk happens on the main thread). If you merely mean visual jankiness but things are actually executing correctly, City of Heroes is an older game with a complex animation system and a lot of lag tolerance (when it was released, many people played on dial-up!), and these things can be expected to happen. This doesn't preclude the possibility of actual animation bugs, which we do correct when reported and when we can. If you could provide more information (preferably in the Bug Reports forum, but please feel free to link your post in this thread), we might be able to provide more information.

Hello! My name is Telephone, and you may remember me from such technically-focused posts as: Server Architecture of City of Heroes To begin with, a brief primer on the server architecture of City of Heroes. Homecoming has several main types of hosts used to run the game. We call these the 'dbserver hosts' (which, with the exception of Reunion, were virtualized back in June 2020 - see the above-mentioned post), the 'mapserver hosts', the 'services hosts', and the 'authentication hosts' (also now virtualized). These are in addition to the various other hosts we have, which include our SQL servers, the forums (which have been virtualized since the beginning), our development infrastructure, backup and security infrastructure, and so on. And of course, the VM hosts themselves! Hosts, What Do? Services Hosts (and the Authentication Hosts) These hosts run what we call 'global services', which are used by every shard. For example, the auction server and Architect Entertainment run on the current active services host (only one services host is active at a given time; the others are warm spares). These services are connected to by every shard, and in turn the services manage database connections and so on. Similarly, the authentication hosts handle authentication for Homecoming, and work the same way (but are run in isolation for security reasons). DBServer Hosts What we called the 'dbserver hosts' above would be far better named the 'shard hosts'. Every shard has exactly one of these, and it runs a process called (not surprisingly), the 'dbserver'. This process serves as an interface between every other process in the shard and the actual SQL database. In addition, this host runs all the other shard-specific services (such as the arena, petitions, the login queue, LFG, and so on). Even given all the above, these hosts are quite lightweight. Homecoming actually runs no less than nine shards (5 Live, 2 Beta, and 2 Prerelease), and sometimes our developers spin up additional temporary development private shards for testing. The load these place on our infrastructure is quite minimal. Now, the primary load of City of Heroes comes from the final type of host ... Mapserver Hosts Everyone's favorite, the mapserver! All shards in a given region and realm (realms are things such as Live, Beta, Prerelease, etc) share the same pool of mapserver hosts. Every single active map is its own process running on one of these hosts, and these processes are the primary load driver for Homecoming. You may have heard mentions elsewhere of 'why is Homecoming running so many shards when they could lower their costs by running fewer?'. Well, the number of shards actually has little bearing on our costs because the mapserver hosts are shared by all shards (except Reunion, which has its own mapserver host). Homecoming currently has six physical mapserver hosts in North America, and one physical mapserver host for Europe (for Reunion). For Beta, Prerelease, and development, we use virtual mapserver hosts, as the performance requirements are far less. Each physical mapserver host (with the current hardware) can handle a maximum of a little over 1,500 players, which is why you may have heard us mention that Homecoming's maximum player capacity is approximately 10,000 in North America and 1,500 in Europe. Of course, when we are at maximum capacity, lag is quite noticeable (though we have not been near maximum capacity in quite some time). So, why keep so many mapserver hosts in North America if we aren't using that capacity at all times? For several reasons: These hosts need maintenance. We occasionally take one or two of the NA mapserver hosts out of service to perform maintenance on it (and then bring it back in the next week). Of course, this currently can't be done in the EU, so maintenance there needs more care. These hosts can fail. Some of you may remember the EU mapserver host failure a few months ago, where we had to perform an emergency redirection of Reunion to the NA mapserver pool. More about hardware failures below. Having spare capacity available helps reduce lag spikes - if there were too few cores or hosts available, players could see their performance impacted by people playing on other maps or even other shards. Imagine if your mission were to lag because of a Hamidon raid on another shard. We often use spare CPU capacity on these hosts to perform certain development tasks (configured to always yield to players, of course). When a map is created or modified by developers, an extremely CPU-intensive process called beaconizing may need to take place. Even with the CPU power available to us, beaconizing a Page release can take many hours. Lastly, it can take days to order a new physical mapserver host (OVH's lead time on this class of host is usually between 3 and 10 days), and the actual installation process is also quite time-consuming. What's Changing Now, after the entire primer above, here's what we're planning to change over the next few months. Our physical mapserver hosts are what OVH calls an 'Advance-5' (A5). These are the most powerful hosts available in their Advance line, and we pay about $360/mo for each one (our pricing was locked in when we set up these A5s back in 2019). Recently, we had a failure on one of our A5s in North America (a very bad drive failure). During the reinstallation, we were reviewing OVH's offerings and noticed that they have refreshed their A5s and that the new A5s are actually significantly better priced ($50-$100/mo cheaper, depending on various discounts). So, we began discussions to see how we could take advantage of that pricing, and just to be sure, we decided to review all of OVH's updated hosting offerings. Upon reviewing them, we found that OVH actually has a new offering which is even more cost-effective than the A5 - enter the SCALE-3. Each SCALE-3 is actually slightly better than two A5s! So, our plan going forward is that we will replace the six North American mapserver hosts with three SCALE-3 hosts. We've already cancelled the mapserver host with the drive failure, and ordered a SCALE-3. We'll replace the North American mapserver hosts two at a time (so we'll cancel two more next month when we order a second SCALE-3, and then the remaining three when we order the final SCALE-3). Although the immediate financial impact is somewhat painful ($1100 to spin up a single SCALE-3, due to a $600 setup fee and then paying $500/mo for the SCALE-3 itself), the savings are so great that we will see that paid back in just a few months. One minor negative is the lead time on SCALE-3s. Currently, we expect to wait 20 days for our first SCALE-3 to arrive. Of course, the monthly fee for the SCALE-3 will apply when it is actually installed, so even though we have paid for the first month up front, that month won't start until mid-June. Something else to note is that the SCALE-3s have ancillary storage on them, in addition to their normal storage. We've got plans to use this additional storage to improve our development and deployment architecture, further increasing our reliability and performance. Currently, the majority of our redundant bulk storage is hosted in the EU, so having access to it locally in NA will allow us to improve the efficiency of some of our operations. Can We Save More Money? OVH does allow us to order servers with a 12-month commitment, and doing so eliminates the setup cost and also gives a small discount. This would save quite a bit of extra money, but at this time we've decided not to pursue long commitments on hosting (especially since this is our first SCALE-3 - we need to make sure it has the performance we expect). We'll continue to evaluate this and may take advantage of it in future orders (including possibly the additional two SCALE-3s after this first one). OVH also has a cheaper line of servers, called the Rise servers. Unfortunately, these servers lack a very important feature which OVH calls vRack. We use an OVH vRack to link the entire Homecoming cluster into a single private network; not having vRack would be a complete nonstarter for us. Risks The biggest risk is that reducing the number of physical servers means a failure could be much worse for us (we'd lose a third of our capacity rather than just a sixth). Given the reliability we've had in the past on our servers and our average player load, we feel this is an acceptable risk. Even so, we've made backup plans to mitigate potential risks (including the ability to quickly spin up backup mapserver hosts using our VM infrastructure or even in the cloud). A smaller risk is that the SCALE-3 doesn't pan out to have the performance we expect. In this case, we would likely bite the bullet, accept the setup fee as a loss, and order new A5s to replace our existing A5s. This does have the possibility that the new A5s might also not have the performance we expect, but our existing VM hosts are very similar systems (though they are A4s rather than A5s) and we're comfortable based on their performance. Europe Of course, Europe doesn't need a SCALE-3. The existing hardware there is more than sufficient to handle the load, and doubling it would be a waste of money. But, we would very much like to refresh our hardware in Europe, and we'd like to increase our redundancy level there (after the unpleasant experience of the mapserver host there failing). We're still researching the best way to do this, but no matter what we do there will likely be a slight cost increase. Currently, we're paying approximately $750/mo for the EU cluster (but some of that cost also supports NA, because we use EU for offsite backup and other processing). The current possibility (subject to more research) is that we'll replace the existing EU cluster with an OVH INFRA-2 (which would run SQL, the Reunion dbserver, and the other EU services) and two OVH Advance-4s (as mapserver hosts for EU). This would increase our costs by approximately $100/mo, but would significantly increase our EU reliability. There are other possibilities as well which could be pursued without a cost increase, but we're not confident that they will have the player performance and experience we want to offer. Expect a follow-up post on this when we get closer to making a decision on how to handle Europe. TL;DR We're planning to spend a total of approximately $1,800 on hardware upgrade setup fees over the next few months. These will result in reducing our costs by several hundred dollars a month. Because we will have some overlap in time we are paying for the old and new servers, the actual payback time will be more than three months, but should be less than six months. Detailed Cost Calculations Current Cost: 6x Base A5's: $331.99 * 6 = $1,991.94 6x Upgraded 1Gb VRack: $23.00 * 6 = $138.00 Total = $2,129.94 New Expected Cost: 3x Base S3's: $513.99 * 3 = $1,541.97 Total = $1,541.97 Savings Per Month = $587.97 3x Initial Setup Cost: $598 * 3 = $1794 Months of Savings to pay off Setup Cost = 3.05 Months (but see above as to why it will actually take longer)

You're completely correct that I did open the door - the suggestion was general, however (not truly restricted to the travel power issue), so let's not discuss it too much further here in terms of specific powers. I agree with you completely that some convenience has been lost in the travel power change, and that should definitely be discussed in the travel powers thread and hopefully some consensus can be reached there. One other minor note - your use case can still be done with these commands with two buttons (of course, this still is not entirely optimal if you depend on looking at the spinning rings, and your argument that not everyone likes making macros is an important one): powexec_togglenext "Hover" "Fly" powexec_toggleoff "Hover"$$powexec_toggleoff "Fly"

While the structure of the missions is (necessarily) formulaic, each mission has its own text tied in to CoH lore and history. Thousands of lines of text were written and proofread by our volunteers (chief among them the previously-mentioned @Nayreia, who as mentioned wrote the text).

Even though this is intended to help with the Hover/Fly animation issues and also the potential issue of multiple travel powers being on and draining too much endurance, suggesting this here as it's more of a general improvement: Add a new command: /powexec_toggleswap "Toggle1" "Toggle2" If none of the specified toggles are active, toggle on Toggle1. If both of the specified toggles are active, toggle off Toggle2. If Toggle1 is active, toggle it off and toggle on Toggle2. If Toggle2 is active, toggle it off and toggle on Toggle1. Alternate suggestion: /powexec_togglenext "Toggle1" ... "ToggleN" If none of the specified toggles are active. toggle on Toggle1. If more than one of the specified toggles are active. toggle them all off, and toggle on Toggle1. If only one of the specified toggles is active, toggle it off and toggle on the next toggle in the list. If the active toggle is ToggleN, wrap around to Toggle1.

Thank you for the reports - Brainstorm hung on startup. We have an automated script which clicks the button to fix this, but apparently its clicking-finger was not correctly positioned. I corrected it manually and all should be good. Brainstorm appears to be operational now - please report if it is not.

This has been completed. As I sat in my comfy chair and issued the commands to destroy an entire universe, I thought to myself .. this is what being a supervillain must be like. It was a good feeling. Happy Thanksgiving to those who live in the US, and happy Thursday to those who do not!

My understanding is that this was strictly a bug fix - under some edge cases a given foe's defense stack would not properly replace itself and would instead stack additional times (so for Invincibility when the bug occurred you would receive multiple defense stacks from a single foe within range).

While I cannot confirm this as I do not own one, the Xbox Adaptive Controller is a standard XInput device and should be 100% supported. One of the drivers for the enhanced gamepad support is to improve accessibility options, so it would be great to have this confirmed by someone who does own one. Edited to add: the people who championed, designed, and built the Xbox Adaptive Controller are personal heroes of mine. Microsoft will probably never see a penny of profit from the device itself, but the fact that they went and and built it shows an amazing commitment to social good and realizing that doing the right thing can be more important than money. Gaming can be for everyone and their commitment to inclusion warms my cold villain heart.

This is a good point. As Troo points out above, there is a Joystick profile, but the defaults may be somewhat dated. I'll look into the possibility of adding an easy way to load defaults for modern gamepads into that profile.

Yes, it absolutely can still be bound as normal. For public education purposes: We spoke about this in Discord and Replacement bound RButton+S to Sprint and it worked as expected (after doing /extra_modifiers rbutton).

@Number Six actually resolved a number of these issues some time ago. There was a problem where the D-Pad on DS4s was reporting directions in a different manner than expected (shifted by 45 degrees, essentially). Please let us know if you're still seeing these!

The most important question: Where did Ms. Liberty get that awesome Ghost Widow costume?

Teleport Target's recharge should be the same as before depending on which target you use it on (teammate or foe). Please test it and if this is not the case, report the issue in the appropriate thread. The power of course cannot know which kind of target you intend to use it on next, so if you use it to bring a foe to you and then attempt to recall a friend, you will of course have to wait the longer recharge.

Here's a suggested set of controller binds which some might find useful. IMPORTANT NOTE: This works best with XInput gamepads (e.g. an Xbox controller) and less well with Playstation gamepads. You may find DS4Windows or other XInput wrappers helpful to use your Playstation gamepad reliably with Homecoming. You will also need to enable your gamepad in the Controls tab of Options: This is also assuming that you have set the triggers to be your controller modifiers (with /controller_modifiers 7 8) and the bumpers to be your extra modifiers (with /extra_modifiers joy5 joy6). Note that zooming in and out is not here because you can use virtual mouse mode to do so (hold both triggers and use the right analog stick), though you could easily bind it if you wish. Left Analog Stick: Forward/Back/Strafe Left/Strafe Right Right Analog Stick: Look Up/Look Down/Turn Left/Turn Right Left Thumb: Match Camera to Player Right Thumb: Face Target A-Button/Cross: Jump B-Button/Circle: Cancel Power X-Button/Square: Travel Power Y-Button/Triangle: Interact (you may need to /bind this, e.g. with /bind joy4 interact) Start/Options: Menu Back/Share: Map D-Pad: Targeting Binds (see below) Right Bumper + D-Pad: Inspirations (by column) Right Bumper + Face Buttons: Emotes or Self-Buffs Left Bumper + D-Pad/Face Buttons: Emotes, Quick Chats, or other Miscellaneous Left Trigger + Left Bumper/Right Bumper: Previous Tray/Next Tray Right Trigger + Left Bumper/Right Bumper: Previous Alt Tray/Next Alt Tray Left Trigger + D-Pad/Face Buttons: Tray Powers 1-8 Right Trigger + D-Pad/Face Buttons: Alt Tray Powers 1-8 /controller_modifiers 7 8 /extra_modifiers joy5 joy6 /bind dpadup target_enemy_next (or target_enemy_far) /bind dpaddown target_enemy_prev (or target_enemy_near) /bind dpadleft target_custom_prev teammate /bind dpadright target_custom_next teammate (I use a left-up-down-right pattern for inspirations and powers, so Left Trigger+D-Pad Left = Tray Slot 1, and Right Trigger + B-Button = Alt Tray Slot 8).

I'm unsure if I should be honored or frightened that my actions ring so loudly amongst our forum userbase. 😀

The Windows 7 issue was actually the same issue that affects Factorio seen here - https://forums.factorio.com/viewtopic.php?t=72946. Thanks to an incredibly helpful tester we discovered that the registry tweak from that post worked, and then we worked from there to solve it in the game itself. The Intel issue was a lot more work, and Number Six would know the technical details more than I.

This is the backup patch server that was deprecated some time ago. You should be able to click the dropdown to go back to the standard Homecoming manifest in Tequila, which will fix this.

All three of these will actually be placed in our NA data center. Read on for more details! Reunion Reunion's already a bit of the odd one out - we've got a mini-cluster of three hosts in EU for it (the shard host itself, the SQL database host, and the mapserver host). The hardware there is a little bit overprovisioned for Reunion's actual load, but we've found a number of creative additional uses for it to use that spare capacity without affecting player experience (for example, the SQL host also has bulk storage attached and serves as one of our backup tiers and our WSUS update server in addition to its normal SQL duties). We don't have any plans to change the EU cluster at this time since doing so would probably be an increase in costs or decrease in flexibility without any gains. The A1s and the A4 When we originally built the Homecoming cluster a year ago, we allocated physical hardware for each shard's main host, as well as a separate host just for running authentication (the five A1s mentioned above). Since then, we've done a lot of optimization, fixing memory leaks, and profiling to determine how the cluster scales. In addition, all of our beta and prerelease shards (as well as our build servers) are running entirely on the single A4. So we're now a lot more comfortable running more of our game infrastructure on VMs. This will give us a lot more fault tolerance, since if one goes down we can temporarily run its VMs on the other two. Some of you may also recall the authentication issue a few months ago - that was an actual hardware failure on our physical authentication host, and we had to build a new one as a VM on the fly while that was being fixed. The A2s Not addressed yet are the two A2s - those are our general purpose VM hosts. While they're not struggling to handle the load, they're actually not as flexible as we had hoped: - They aren't very good at handling our build infrastructure (which wants lots of cores, but these only have six physical cores each). - We underestimated how much disk space they would need, so we occasionally have to juggle services to avoid a crunch. - Because two is no better than one in the world of high availability, we have to run backup instances of some of the services on these hosts on other hosts to ensure they are fully reliable. What's Not Changing Those of you who have kept up with our other technology posts may remember that we have a few other hosts in NA: - One more A1 which is used as a citadel host (most places would call this a bastion host, but this is CoH) - Three A3s which are used as SQL database hosts (two hot and one warm spare) - Six A5s which are used as mapserver hosts None of these are changing at this time. It's not that we're not confident of our ability to run the services of these hosts on VMs, but after analyzing the required capacity and the types of host we can purchase, we didn't feel there would be any significant cost savings while maintaining our desired reliability and performance criteria. The Future The net result is that the three new A4s will be running the following: - The authentication server (and warm spares) - The four NA live shard hosts (and we'll be able to quickly migrate if there is a failure) - All hosts for beta and prerelease (shard hosts, services hosts, and mapserver hosts) - Our forums - Our development infrastructure and build servers - Our other infrastructure (e.g. petitions/GM tools, system administration, patch and manifest management, CDN, our Kubernetes cluster). The hardware will be better than what we have now: - We'll have the same number of physical cores (48 - but note that the A4 cores are superior to the ones in the A1s/A2s, and also that many of our current cores are just not in places where it is convenient to use them) - We'll have more RAM (544GB -> 768GB). - We'll have more disk space in the right places (4.5TB -> 6TB). TL;DR - We'll reduce our footprint and long-term costs, increase our flexibility for future changes, and be more robust. - Nothing is changing in the EU cluster at this time. Finally, thank you all for donating! We wouldn't be here without you.

Development work is always ongoing! I can't say when we will have something to announce, but be assured that the team has plenty of plans to move forward and an internal roadmap.

You're absolutely correct that this is a bit confusing and needs a little more explanation. On the live shards, there are three pre-defined 'modifiers' (Shift, Ctrl, and Alt). These can be used in any keybind, etc. The patch allows you to define up to six additional modifiers: two 'Controller Modifiers' and up to four 'Extra Modifiers'. The 'Controller Modifiers' are special, as pressing them together enables Virtual Mouse Mode. They do work as bindable modifiers just like the pre-existing three. The up-to-four 'Extra Modifiers' are just additional inputs (either keys or buttons) that can be treated as modifiers when no other modifier is pressed. So for example, you could do /extra_modifiers LBumper RBumper to make the bumpers modifiers on your XInput gamepad. You could then use (for example) LBumper+Joy1 as a /bind (but also still be able to use LTrigger+LBumper as a separate bind). The 'Extra Modifiers' aren't limited to just gamepad buttons, however! If you want, you can bind any key on your keyboard as one (but note that due to some things that still need refactoring, you can't bind a specific Shift, Ctrl, or Alt key on your keyboard yet). You also point out that you both controller- and extra-modifiered the triggers. That could definitely cause issues and we'll fix that. We also don't (currently) have way for you to clear _all_ your extra modifiers, so we'll have to fix that as well (you can just specify a single one with /extra_modifiers and it should clear the other three). Not noted in the patch notes is that we added a number of other convenience aliases which you may have already noticed (e.g. LTrigger, LeftTrigger, RightBumper, StartButton, AButton, and so on). Thanks for the feedback! Please let us know if you have other questions as well.

I'd just like to note that the above patch notes don't encompass the full extent of the controller changes (the fault is mine, as I have been tracking notes in a long-running merge request and the notes there are from an earlier revision in a commit message). New Slash Commands /extra_modifiers [mod1] [mod2] [mod3] [mod4] Allows setting up to four extra modifiers. These (currently) only work when no other modifier is pressed. Recommended for XInput users is /extra_modifiers LBumper RBumper /controller_modifiers <first> <second> Allows setting two controller buttons as modifiers. On non-XInput devices, when both of these are pressed together, the controller will enter virtual mouse mode. The arguments are the button numbers to use (from 1-24). Recommended for XInput users is /controller_modifiers 7 8 (the two triggers, since these are currently hard-coded as virtual mouse mode on XInput) /controller_vmouse <LMB> <RMB> [MMB] [Snap] Configures virtual mouse mode buttons. By default, button 1 is the LMB and button 2 is the RMB. The other two are unset. Recommended for XInput users is /controller_vmouse 1 2 4 3 (which will give you the configuration from the above patch notes). Virtual Mouse Mode The above patch notes are largely correct, except that the middle mouse button and snap button are not configured by default. Note that on XInput devices, the Back and Start buttons are also hard-coded to pass through (so you can configure the Start button to activate the menu and then open it in VMM). There is no current way to modify VMM passthrough configuration, but this is planned. Note that only XInput devices can currently snap diagonally. Due to various controllers reporting inputs differently, this is difficult to fix, but we're looking into it. Future Potential Enhancements A real GUI to configure all these things. Passthrough configuration for VMM. Properly stackable modifiers (so you can finally bind Ctrl+Alt+X to your nuke). Latching modifiers (so that you can set up a bind for Left Trigger, then Left Bumper, then the X button). The ability to independently bind LShift/RShift/LCtrl/RCtrl/LAlt/RAlt as actual modifiers if you wish. Analog movement and camera control when using analog sticks. A bindable 'interact' which will interact with the closest thing in front of you (e.g. opening a door or talking to a contact). Questions for Users How does Virtual Mouse Mode feel? Any specific issues? Does the pointer move at a good speed? Are the snap positions good? Do binds save and restore well? Can you load a bind file from live without issues? There's currently a very unusual behavior in City of Heroes where if you bind Ctrl+X and Alt+X, then press Ctrl+Alt+X, both keybinds will fire. Do you depend on this behavior? We're considering removing it as it will make many of the above features much easier to implement. One More Thing If you copy your launch line from Tequila and then launch the game from Steam, any controller can be an XInput controller. (I personally use this on my DS4).

Farming itself was and is not an exploit. The base use of the option was also not an exploit. However, there were specific things you could do in combination with the option to significantly increase your Influence gain. Those are the exploits to which we are referring.