The Future - An Open Letter from the Homecoming Team

[theta31]

I am sograteful to see continued development, and I can't wait to see how the cool down timers work.  Specifically, will the timers increase if you are hit with a -recharge debuff or do they freeze for a while?  It would be really cool if the numbers turn red when you are affected by -recharge.

[QuiJon]

The only part I really have an issue with is the explanation for the nerfing of XP for the AE going forward. I get exactly what you have said about the original reason being the drag on the server resources. But I think the nerfed xp should have  been a short term fix and then take the mechanics out of the AE that allow for it to be exploited for farming OR if you use them the xp is then nerfed. Allowing those that want to RP and use the AE for stories to do whatever they want knowing that the xp will be lower but those that want to level in the AE that are playing missions even farming them that don't exploit the mechanics to get full or even double xp doing so.

As far as the exploitive nature of the farming goes all that nerfing the xp did was to change the flood of LFT spam to DFB instead of AE. The farming is still happening so really continuing to nerf the xp from the AE is a direct response to how SOME people feel the game should or should not be played. Put the xp back to normal nerf the mechanics themselves and let people decide how they choose to play for themselves. I personally mostly farm up to mid 20s. Then I start to play. I can do that in the AE or doing DFBs I would rather do AEs cause I can solo those just as quickly as I have to team to do a dfb. And I find it hard to believe that my preffered farm map of 24 bosses on a small tunnel map with 1 bot helping me is to much of a drag on server resources.

[cforciea]

dbquery is the "brute force" method because it's the most direct method and the one that works now to produce the character in the text container format. But dbquery doesn't touch SQL directly; so indices and such don't matter. Instead it opens a connection to the dbserver just like a map starting up would do and asks for it from there.

 

 

In testing we found that dbquery also can cause the dbserver to hiccup when it's run for some as yet unknown reason. That's really bad, since every map in a shard depends on dbserver being responsive not only to load/save characters but also to relay chat, mission updates, teams, moving between zones, etc. The dbservers are already struggling to handle more than 2,000 players online due to some design issues unrelated to the backend database, and a hiccup at the wrong time is catastrophic for a lot of people.

 

I'll own up to misunderstanding what you meant by dbquery. I had assumed you either just meant generally a database query, or something like the bit of LINQ library that also generates database queries, not to an existing executable. I also got off my butt and looked at the source for the tool other servers are running, and yep that's what's going on. My bad.

 

So that leaves either writing a tool to extract the data from SQL directly and format it the same way, or coming up with some other method. For security reasons the web tier shouldn't be allowed to touch the game database directly, and an extra replica for that would incur more license costs - at least until the pgsql conversion is done.

 

I wouldn't particularly worry about "the same format". As long as the data is parseable, somebody can always write a tool later to munge it to however it needs to look, especially if you open source whatever model code you'd be using to serialize it.

 

If the concern is just security, it'd be pretty trivial to throw up a lightweight API server using a framework in your language of choice, toss nginx in front of it, and limit requests to a specific incoming IP. Then your public-facing web server can just query that. The amount of code difference there is pretty negligible.

 

Actually using a replica would indeed generate more costs, but as I alluded to, I'd think you'd want a replica anyway. The alternative is all sorts of potential hilarity when you lose a drive on your db server and need to do a raid rebuild, or a rollback if the db eats its fingers. If the goal is stability, that should be pretty high on your list regardless of any export functionality.

 

 

There are also logistical issues that we've discussed the best way to handle. The website is the most obvious route. However, it's based on old forum software and at the risk of spoiling something that shouldn't be public yet, there are plans to replace it with something more modern for account management and other functions. It doesn't make a lot of sense to write something for the current site that will just have to be rewritten in a month or two.

 

If you do your interesting logic in an API layer separate from the website to alleviate your existing security concern, then all you are rewriting is the ability to read and display the output from that backend API call.

 

Web access may not be the best way to go for other reasons as well. On such an alt-heavy game, it could become quite burdensome to manually download your characters whenever you want to make a backup copy.

 

One other approach we're considering (and the one I'm personally advocating for) is to modify the protocol so that the game client receives a complete copy of the character on logout and automatically saves it in a backup folder. This would be opt-in to avoid privacy issues if you're playing on somebody else's computer, but once enabled would allow players the peace of mind of knowing they always have a current backup. This also shifts the burden of producing the character dump to the mapserver, which already has a full copy anyway so there is zero additional load on the dbserver.

 

Finally there is talk of adding a cryptographic signature so that the character export can be verified as authentic. This would be useful both in a worst-case-scenario recovery contingency, as well as for server operators who want to allow automated imports of verified characters. This would be coupled with adding a character UUID to prevent duplication.

 

Burdensome is substantially better than nonexistent. Don't let perfect be the enemy of good enough. It's perfectly possible to throw together something that gives us safety now and then still build those other, cooler things later.

 

Besides, I bet if you create a crappy web interface, other people could throw together a tool to log in to your website, crawl through the character list, and scrape all of the characters without you having to build anything else.

 

Right now the server team's development priorities are focused almost entirely on stability. There is an extensive rewrite of parts of dbserver ongoing to address its design issues preventing smooth scaling above 2000 players per shard. There is also a plan for a potential PostgreSQL migration; as its design meshes well with the update-heavy workload and it could reduce licensing costs in the long term. The powers and frontend/UI people are working on different things in their areas of expertise of course.

 

TL;DR: Character exports are something we think is a good idea, and something we're planning to look at once we're comfortable with the stability of the servers. But we're not going to slap something together and throw thousands of players at it without being confident it will work - we'll do it right.

 

Given how the centralized control model worked out for me in 2012, I tend to think that doing it "right" in this case is at least throwing together something simple so your players can manage their own character data before worrying about whether you can get more than 2000 players on a shard, but you are, of course, entitled to your own priorities.

 

[Zep]

I hear that PhysX has been fixed elsewhere, any word when the fix will make it hear?

 

I'm going to address this because it keeps being brought up, and it is intentionally not being "fixed" right now.

 

Reason #1: if you bump your Particle Physics to "Very High", you get the same amount of particle physics as if you had a PhysX card. This happens on the CPU, not the GPU, but for modern computers this barely registers.

 

Reason #2: the "fix" involves upgrading the PhysX libraries (the DLLs in your game folder) to a newer version. So if we do that we are no longer compatible with the libraries shipped with Issue 24 Beta. This is a problem because it breaks the original client, as well as the client of any server that doesn't implement this "fix". It's a really bad idea to mess with the DLLs as long as multiple servers are sharing a COH install.

 

A similar problem would happen if we were to enable the error reporting DLL (crashrpt.dll) in order to collect crash logs; as soon as we put that DLL in there, we start getting crash reports from every server, not just Homecoming. Bad idea.

 

The solution is the Sunrise patcher, which will keep server-specific files on their own folders. At that point we can certainly start upgrading DLLs and adding our own features. But as long as all the servers are sharing a COH install folder, messing with the DLLs is irresponsible at best, malicious at worst.

 

Thank you for the information. It is very much appreciated.

 

I look forward to the update which, I fully understand, will happen when it is ready.

 

Thank you to everyone making Homecoming happen.

[golstat2003]

dbquery is the "brute force" method because it's the most direct method and the one that works now to produce the character in the text container format. But dbquery doesn't touch SQL directly; so indices and such don't matter. Instead it opens a connection to the dbserver just like a map starting up would do and asks for it from there.

 

 

In testing we found that dbquery also can cause the dbserver to hiccup when it's run for some as yet unknown reason. That's really bad, since every map in a shard depends on dbserver being responsive not only to load/save characters but also to relay chat, mission updates, teams, moving between zones, etc. The dbservers are already struggling to handle more than 2,000 players online due to some design issues unrelated to the backend database, and a hiccup at the wrong time is catastrophic for a lot of people.

 

I'll own up to misunderstanding what you meant by dbquery. I had assumed you either just meant generally a database query, or something like the bit of LINQ library that also generates database queries, not to an existing executable. I also got off my butt and looked at the source for the tool other servers are running, and yep that's what's going on. My bad.

 

So that leaves either writing a tool to extract the data from SQL directly and format it the same way, or coming up with some other method. For security reasons the web tier shouldn't be allowed to touch the game database directly, and an extra replica for that would incur more license costs - at least until the pgsql conversion is done.

 

I wouldn't particularly worry about "the same format". As long as the data is parseable, somebody can always write a tool later to munge it to however it needs to look, especially if you open source whatever model code you'd be using to serialize it.

 

If the concern is just security, it'd be pretty trivial to throw up a lightweight API server using a framework in your language of choice, toss nginx in front of it, and limit requests to a specific incoming IP. Then your public-facing web server can just query that. The amount of code difference there is pretty negligible.

 

Actually using a replica would indeed generate more costs, but as I alluded to, I'd think you'd want a replica anyway. The alternative is all sorts of potential hilarity when you lose a drive on your db server and need to do a raid rebuild, or a rollback if the db eats its fingers. If the goal is stability, that should be pretty high on your list regardless of any export functionality.

 

 

There are also logistical issues that we've discussed the best way to handle. The website is the most obvious route. However, it's based on old forum software and at the risk of spoiling something that shouldn't be public yet, there are plans to replace it with something more modern for account management and other functions. It doesn't make a lot of sense to write something for the current site that will just have to be rewritten in a month or two.

 

If you do your interesting logic in an API layer separate from the website to alleviate your existing security concern, then all you are rewriting is the ability to read and display the output from that backend API call.

 

Web access may not be the best way to go for other reasons as well. On such an alt-heavy game, it could become quite burdensome to manually download your characters whenever you want to make a backup copy.

 

One other approach we're considering (and the one I'm personally advocating for) is to modify the protocol so that the game client receives a complete copy of the character on logout and automatically saves it in a backup folder. This would be opt-in to avoid privacy issues if you're playing on somebody else's computer, but once enabled would allow players the peace of mind of knowing they always have a current backup. This also shifts the burden of producing the character dump to the mapserver, which already has a full copy anyway so there is zero additional load on the dbserver.

 

Finally there is talk of adding a cryptographic signature so that the character export can be verified as authentic. This would be useful both in a worst-case-scenario recovery contingency, as well as for server operators who want to allow automated imports of verified characters. This would be coupled with adding a character UUID to prevent duplication.

 

Burdensome is substantially better than nonexistent. Don't let perfect be the enemy of good enough. It's perfectly possible to throw together something that gives us safety now and then still build those other, cooler things later.

 

Besides, I bet if you create a crappy web interface, other people could throw together a tool to log in to your website, crawl through the character list, and scrape all of the characters without you having to build anything else.

 

Right now the server team's development priorities are focused almost entirely on stability. There is an extensive rewrite of parts of dbserver ongoing to address its design issues preventing smooth scaling above 2000 players per shard. There is also a plan for a potential PostgreSQL migration; as its design meshes well with the update-heavy workload and it could reduce licensing costs in the long term. The powers and frontend/UI people are working on different things in their areas of expertise of course.

 

TL;DR: Character exports are something we think is a good idea, and something we're planning to look at once we're comfortable with the stability of the servers. But we're not going to slap something together and throw thousands of players at it without being confident it will work - we'll do it right.

 

Given how the centralized control model worked out for me in 2012, I tend to think that doing it "right" in this case is at least throwing together something simple so your players can manage their own character data before worrying about whether you can get more than 2000 players on a shard, but you are, of course, entitled to your own priorities.

 

Add me to the list of people who would rather see them do it correctly, then rush something through that is problematic or causes a problem for the servers. I agree with the devs approach.

[cforciea]

Add me to the list of people who would rather see them do it correctly, then rush something through that is problematic or causes a problem for the servers. I agree with the devs approach.

 

There are lots of options here besides "provide a client-side system for autosaving characters with RSA signatures" and "make a tool that hoses the game servers." I'm in no way advocating for doing anything dangerous or poorly architected, regardless of any claims about what doing it "correctly" would look like.

[CY13ERPUNK]

thank you for everything Jimmy & everyone on the Homecoming team ; mad respekt

 

i25 is everything i hoped ; great job so far

 

of course the big dreams for the future are the continuation of the storyline [The Battalion, Tyrants redemption, the praetorian Hamidon, more signature story arcs, Nemesis transformation into Silos, etc] and to see the rest of the incarnate slots finished {what a shame to have such a system killed before it had even fully bloomed, what did the OG devs have in mind to challenge us at the top-end???} ; but i understand that this is basically the biggest/hardest/most complex stuff that we could ever hope for {ie a full dev crew of dozens of ppl basically working full-time}

 

smaller but still probably just as impactful in some ways , have u guys thought about more power animation proliferations/options ; much like how we got the bright/dark choices, but also how Super Strength and Martial Arts where given some alternate animation choices ; we all know that the later sets have vastly superior/smoother/etc looking animations [looking at you Street Justice/Kinetics/Dual Pistols], and we know the OG dev team did eventually want to overhaul the whole system, but of course thats a big pill ; but how about just trying to give more powers an animation 'choice' based on similar ingame timings? and what's up with Martial Assault getting the small melee cone upgrade for their 'Spinning Kick' but Martial Arts Storm Kick gets forgotten? =P for shame

 

lastly something ive noticed recently the character lighting seems to be uneven/off in some buildings/missions ; say when you are in the hospital, but your character is lit like it is dark/night even though the room is bright ; definitely looks weird, obviously a bug/unintended, but not sure how complex the current ingame lighting system is, knowing that its almost a decade old tech

[Shirokiel]

First off, thanks a lot for all the amazing work your team has put into making this dream become a reality.

 

My question is concerning powersets and having a further look at them. I will take Cold Domination as an example. If i take Cold Domination on a Defender I have the opportunity to select "Low Visual FX" for powers like Ice Shield and Glacial Shield. If I take Cold Domination on a Controller I don't have the "Low Visual FX" option for those powers available. Is this something you will be implementing at some point in the future or is there a reasoning behind this decision?

 

Thanks a lot and keep up the good work! :)

[irishhawk]

I curious about any development plans concerning the:

 

1) Striker arch-type

 

2) Guardian arch-type

 

3) populating Kalisti Wharf.  Maybe even with a kit-bashed version of its intended inhabitants, the Battalion.

 

4) remaining Incarnate abilities

 

5) origin power pools

 

and lastly 6) Wind Control powerset

 

Thanks for any feedback!  :D

[MrAxe]

This is all amazing. Thank you so much. Really does feel like home. :)

[TheCatMan]

Thanks for the info, was wondering about some of the whys and this filled me in.  Looking forward to a long stay at Home.

[Yikes]

Once development on the game begins to happen will there be a seperate shard for people who want the "vanilla" experience?

 

You all are the best of the best! I cannot thank you enough for all you have done! Please never stop!

 

I second this. Some of us would much prefer a shard with vanilla progression but can't really get that experience on the i24 servers whose populations are unplayably small.

[DJBlack71]

I once again want to thank all that is involved in bringing this back to us and keeping this going, The fact you guys have plans  on updating stuff is remarkable. I hope we can keep this going for as long as possible

[Lady Sadako]

If you have any questions for myself or the rest of the Homecoming team please feel free to post them in this thread.

 

Hi Jimmy. Thanks for everything. :)

 

Leandro posted a video with some new base building stuff in. Do you have any idea when that might be going live?

[Jestunhi]

Any thing looking to be done about the epidemic of people doing DFB (Death from below)? I know some people are looking for quick levels. but when LFG chat is nothing but DFB LFM call outs. its a little bit annoying. (seeing this consistently on Everlasting)

 

Its even a little concerning the few times I did it, the groups I was in had at least 1 - 3 level 50+veterancy ( Task forces are wonderfull, so are strikes. would love to see more people wanting to do those to work on merits sets...)

 

This is the problem with keeping AE nerfed, not because it is actually causing a problem but because you feel it's better that people play different content. It empowers others to endlessly complain that people aren't doing the content that they want them to do - and something must be done about it!

 

And I can assure you, nerfing DFB will not fix anything. People will find something else efficient to do and the same people will complain just as much about that.

 

 

I do not see "but I want them to do something else!" as valid justification for nerfing content. Not at all.

[Perfectionist]

Words are endless for the appreciation and support you guys have given, let alone providing us our home yet again.

 

Thank you!

[Heroyouhate]

Hello, first i would like to say Thank You for bringing this back. Everyone has that 1 game in their life and this is my 1 game. Now to the point, I like to streamto Twitch. I have resisted the urge to stream thinking it may get you guys in trouble. Would there ever be a time i could stream it?

[Lockpick]

Any thing looking to be done about the epidemic of people doing DFB (Death from below)? I know some people are looking for quick levels. but when LFG chat is nothing but DFB LFM call outs. its a little bit annoying. (seeing this consistently on Everlasting)

 

Its even a little concerning the few times I did it, the groups I was in had at least 1 - 3 level 50+veterancy ( Task forces are wonderfull, so are strikes. would love to see more people wanting to do those to work on merits sets...)

 

This is the problem with keeping AE nerfed, not because it is actually causing a problem but because you feel it's better that people play different content. It empowers others to endlessly complain that people aren't doing the content that they want them to do - and something must be done about it!

 

And I can assure you, nerfing DFB will not fix anything. People will find something else efficient to do and the same people will complain just as much about that.

 

 

I do not see "but I want them to do something else!" as valid justification for nerfing content. Not at all.

 

Completely agree.  My earlier post should have been a reply to aggerrath, who is complaining about the content people play. 

[Kelanor]

I just want to throw in my two cents, I quit official CoH back in the day because all everyone ever played was AE... No one played any other content, the streets were barren. When I first logged into these servers and saw people actually PLAYING the game, I was immediately impressed and enjoyed the experience. Please don't lift the AE nerf, its a real game changer and I appreciate it 110%. Thank you all for a very pleasant and enjoyable game experience. :)

And yes I do kind of agree DFB spamming is annoying, BUT it's 110% better than the alternative. At least only a small handful of upper level people seem to be participating in it, instead of 75%+ of the whole community. (although i don't completely understand it TBH, after level 15-20, why do it? You only have access to like two powers on your bar, so why put yourself through that after getting through that first low level slump of not having many powers?)

 

(Also as a sub-note, this is easily the friendliest and most chill community I've ever had the pleasure of being part of. Thank you all.)

[Chuckers]

Thank you so much for all the work you do! I think I am having MORE fun in the game now, then when it was originally live! You guys rock!!!

[DeadJester]

As someone who bought CoV in the shops, took it home and installed it only to find the servers had died this project is something which makes me feel like someone has brought back a balloon I lost as a kid. It also says a lot that you've stopped taking donations when they hit the goal you wanted, as allowing people to keep their money for the future shows you have longevity in mind. I'd talk more about the other stuff you said but I'm too excited to finally delve into the character creator!

[Number Six]

Actually using a replica would indeed generate more costs, but as I alluded to, I'd think you'd want a replica anyway. The alternative is all sorts of potential hilarity when you lose a drive on your db server and need to do a raid rebuild, or a rollback if the db eats its fingers. If the goal is stability, that should be pretty high on your list regardless of any export functionality.

 

We do have a replica, but it's using log shipping since we're running the cheap version of SQL server, so it's not online and available for queries. The replica is inside the secure part of the network in case we need to go active/active and run some shards there, which I believe was done this morning as the primary SQL box got maxed out a few times last night.

 

If/when we switch to pg that should open up some more options here. The port is almost complete (everything except accountserver & auth is done) and just needs testing under real load conditions.

 

Burdensome is substantially better than nonexistent. Don't let perfect be the enemy of good enough. It's perfectly possible to throw together something that gives us safety now and then still build those other, cooler things later.

 

The team is quite pragmatic about it and time-to-production is definitely a factor. None of the options being considered are ones that would be prohibitively difficult or time consuming to implement.

 

Given how the centralized control model worked out for me in 2012, I tend to think that doing it "right" in this case is at least throwing together something simple so your players can manage their own character data before worrying about whether you can get more than 2000 players on a shard, but you are, of course, entitled to your own priorities.

 

I appreciate that. FWIW, the super-high priority list contains issues that are affecting people daily. Excelsior frequently has more than 2,000 connected -- it was nearly 2,500 last night when it started queuing people and it was running really close to the instability line. It's the same issue that took down Torchbearer yesterday -- at around 2300 online with multiple MSRs going, dbserver couldn't keep up and the queue of deferred updates to send to the SQL server grew to over 60,000 before it became unresponsive. That caused many to get disconnected and stuck in the dreaded "character is still logging out" state. There's also a problem somewhere in the netlink code that can hard crash an entire shard if global services (chatserver / accountserver) go offline unexpected, but the bug only seems to show up when there are lots of people connected and getting a good crash dump is proving tricky.

 

All those are issues that we're expecting to take a week or two to solve at most - not months. The dbserver changes in particular are what the temporary test server is for. That build also contains the pgsql port, but since it replaces the entire sqlconn layer to make it modular, it needs testing even with MSSQL.

 

There are offsite backups "just in case", and while not just a whole lot of people have asked for individual character-level backups, it's something that's high up on my own priority list once the immediate issues are solved, as I know it would make people feel safer after what happened in 2012.

[Willowe]

I wasn't on facebook the past month and randomly saw something from CoH Survivors come up on my fed. I was just perusing the page when I read that this all happened. I was so excited I called my mother immediately (even though she didn't answer...womp womp). See, my mother and I used to play when I was a teen, and when I moved out of state, it was a way we could still play video games together. This place is home, and even playing it the last couple of days, it still seems unreal. When she was finally available to talk (hours later with me stewing in my excitement), all I said was, it's back, it's back online," to which she responded, '"you're lying, no way!?" She knew exactly what I was talking about without me even saying it. That is what that game meant to us, and we're beyond thankful it's here again. Thank you.

[vallerainth]

DFB LFM pst

 

50+3 x8 LFM 5MIL a run

 

 

One of these I'm perfectly fine with. Because it's easy, it's fast and it's rewarding. Even for people who have a little social anxiety, or are strapped for time and want to get something done quickly in between the chores of the day.

 

The other I don't like at all. But, I do understand that everyone should have to freedom to play the game how they want to play it so I will not infringe on their choice. As unsightly as that may be.

 

 

I trust you guys here in Homecoming will take the time to evaluate the matter and make the best choice for the servers as a whole. I know it's been said many times already but it bears repeating:

 

Thank you for all you've done. We love you wonderful strangers.

[victorvndoom]

is there an easier way to install the game instead using torrents ?