Feedback: Testing Melee Set Performance

[Haijinx]

Just now, Infinitum said:

but to simulate how someone may choose something in game i dont see how running a different secondary could do anything but add to the data thats being collected - if you really are wanting the most accurate data that simulates what you would see in game thats one suggestion

Most secondaries add something offensively 

 

Invul adds to hit

 

SR adds recharge (plus it levels unevenly)

 

Might make it hard to find the other secondary

 

[Hopeling]

8 minutes ago, Infinitum said:

To be fair though you should at least test with a non +end secondary also to see if the results are dramatically off.

We plan to, as I've said before. If you want to help, feel free; there are a lot of powersets and multiple testing conditions, each one taking several minutes per run for an intended 10 runs. If you feel that something is not being represented in the data, step up and make sure it gets represented.

[Galaxy Brain]

Invuln would be closest, but the variable +ToHit could potentially mess with results vs other sets.

 

Of the secondary effects, the +Recovery on WP when 3 slotted, alongside stamina 3 slotting, *alongside* having all the WP toggles + CJ run is 2.55 end / sec. For comparison, 3 slotted stamina with no toggles at all is 2.47 / sec. SO it is better, but while running the toggles to emulate normal play its not that huge.

[Infinitum]

2 minutes ago, Haijinx said:

Most secondaries add something offensively 

 

Invul adds to hit

 

SR adds recharge (plus it levels unevenly)

 

Might make it hard to find the other secondary

 

well pick 3 then average them, something this important shouldnt exclude the weaknesses and include only the ones that makes it stronger

[juncyard3]

Titan weapon is not viable WITHOUT incarnates or an endurance replenishing damage gimped primary for tank, secondary for brutes and scrappers.

 

Nerfing a powerset...ANY powerset is the wrong direction, any perceived weaker powerset should be buffed.

 

Very counter intuitive to say the least to call for a nerf of a powerset based on the criteria I have seen in this thread.

 

I've got a bajillion hours of playtime from the dawn of live until now and I've seen enough to know better than the silliness I've read above.

[Indystruck]

51 minutes ago, Vayek said:

What about poison/toxic? those are ticks too. Are we gunan be fair here and objectively nerf every tick of every damage type? At what point does anecdotal evidence stop carrying weight in this discussion?

 

Well, actually, according to the datasheets posted here, on average across the whole game Smash/Lethal is resisted to .95, where as Toxic is resisted to .96, so they're actually pretty close as far as commonly being resisted.

[Galaxy Brain]

4 minutes ago, Indystruck said:

 

Well, actually, according to the datasheets posted here, on average across the whole game Smash/Lethal is resisted to .95, where as Toxic is resisted to .96, so they're actually pretty close as far as commonly being resisted.

You could even make a case that Toxic is *more* resisted since there are no enemies with a Toxic weakness, while there are some with weaknesses to S/L.

 

I'm also redoing those charts to have more nuanced breakdowns, more info to come 🙂 

Edited September 18, 2019 by Galaxy Brain

[Haijinx]

Just now, juncyard3 said:

Titan weapon is not viable WITHOUT incarnates or an endurance replenishing damage gimped primary for tank, secondary for brutes and scrappers.

 

Nerfing a powerset...ANY powerset is the wrong direction, any perceived weaker powerset should be buffed.

 

Very counter intuitive to say the least to call for a nerf of a powerset based on the criteria I have seen in this thread.

 

I've got a bajillion hours of playtime from the dawn of live until now and I've seen enough to know better than the silliness I've read 

Hmm.

 

We are attempting to move past the feels stuff towards figuring out metrics.  

 

The thread keeps sort of devolving back into opinions. 

 

It could be the metered trials will hint at no serious overperforming sets, then the whole thing is moot after all.

[Hopeling]

1 hour ago, Vayek said:

okay, so people are now chiming in with times it took them to kill something (which i really hope isn't a non-nonsensical pylon). And I am see an average of 1 to 2 minute differences. Is everyone really so petty they are crying out for an entire power set nerf over 1 to 2 minutes of subjective play testing? I mean, wow.

See, this is why I wanted you to read the thread. You're confused, because you don't understand what we're talking about, because a lot has happened and the conversation is no longer where it was on page 1.

 

Starting on page 5, we had interest in the topic from GMs and devs, eventually asking us to gather data comparing the performance of melee sets. This led to some discussion and eventually settled on the tests that we are just now starting to perform. My comment above is an extremely preliminary attempt; I've only even looked at two primaries, paired with a single secondary, using a single AT, on a specific mission setting, with a specific kind of build. We plan to explore all of those variables and more.

 

I am aware that Broadsword underperforms; that's specifically why I chose it to test next. I already explained this in the very post you quoted.

1 hour ago, Vayek said:

Other power sets have much faster cast times, similar damage numbers, and far less edurance cost. Titan weapons was properly balanced from the start with high damage, long casting times, and enormous end cost.

If you're correct about this, the data will support it. I am not a dev. I do not have the power to change anything. All I can do is collect data and present it to the devs. If you want to make sure that data is reliable, you are welcome to participate in data collection.

Edited September 18, 2019 by Hopeling

[Haijinx]

9 minutes ago, Galaxy Brain said:

Invuln would be closest, but the variable +ToHit could potentially mess with results vs other sets.

 

Of the secondary effects, the +Recovery on WP when 3 slotted, alongside stamina 3 slotting, *alongside* having all the WP toggles + CJ run is 2.55 end / sec. For comparison, 3 slotted stamina with no toggles at all is 2.47 / sec. SO it is better, but while running the toggles to emulate normal play its not that huge.

Invul would be better than SR though since SR scrappers get no taunt aura at all.  Plus no AOE protection until 35.

 

 

 

 

[Eclipse.]

I'd argue that if it's going to be nerfed [I don't think it should]. Nerf it in a way that makes thematic sense.

Currently it does [arguably] excessive damage for the recharge+activation+cone size, I'd argue that this -- thematically --  that makes sense (IT IS A BIG ASS SWORD!) . I won't get into details why I feel it's not excessive, others have covered that quite well in this thread IMO, and I have nothing to add on that front.

Decreasing damage doesn't make much sense, and it'd feel pretty bad . IMO increase the range by 10-20% leaving target cap the same^† - effectively no change majority of the time - but decrease the BaseHitChance of the powers to 63-65% instead of 75%.This level of BaseToHit is halfway between hardcontrol (hold, stun) AoEs and softcontrol (immob) AoEs. Thematically it makes sense that the big slow moving sword is easy to dodge, make Build Momentum (the skill) give an additional 5% to-hit and remove 5% damage buff from it to line up that it's harder to dodge now.
 

Effectively that'd make it the same as giving the mobs you're fighting +1 level's worth of defense more-or-less at all times [and at some levels it's effectively +2 levels of defense]. This can be overcome by taking Tactics and putting 2 level 50 ToHit IOs in it, but that also increases the endurance drain that already plagues the set, as well as takes up an additional skill+slot (assuming you don't already have tactics).


†I say to increase the range because currently it shares range with other skills, and while I don't think it's necessary to increase range I think it will help given the decrease to accuracy, and the issue TW currently has with fleeing mobs due to long animation times...something others in this thread [and my personal belief] have stated as an argument for why it should not be nerfed.

[Indystruck]

The last thing TW needs is a range buff. It's already an outlier in that respect compared to every other melee set. It only won't be for tanks after they implement those AoE buff changes (if they do) that are currently on beta.

[Haijinx]

TW does get a range increase I think, its 9 feet in mids/pines. 

 

Same as staff.

 

That's a 28% increase, which isn't bad.  

 

[Hopeling]

7 minutes ago, Eclipse. said:

Decreasing damage doesn't make much sense, and it'd feel pretty bad . IMO increase the range by 10-20% leaving target cap the same^† - effectively no change majority of the time - but decrease the BaseHitChance of the powers to 63-65% instead of 75%.

Decreasing the base accuracy is an interesting idea, and I agree it's kind of thematic. If the issue really is that TW scales better than most sets with IOs and Incarnate powers, though, that seems like a change in the wrong direction: high-end builds are the ones that can easily have accuracy out the wazoo, while SO builds will feel the penalty hardest.

 

Right now though, I'd prefer to focus on collecting data and determining whether changes are necessary at all and to what degree, rather than brainstorming what those changes could be.

[Galaxy Brain]

3 minutes ago, Vayek said:

I feel like this whole thing needs objective tests on the components of titan weapons.

A test on endurance management of going through 1 complete attack chain - where is the specific endurance after a single complete attack chain?

This can be done without testing, using MIDs or real numbers. In fact this was already done by me earlier in the thread vs War Mace.

 

3 minutes ago, Vayek said:

 

A dual test on the damage per second with using "build momentum" and without using momentum (not using secondary powers)

This... technically is not possible, and incredibly difficult to do. You would hit, wait 5 seconds, then hit again and subtract the time from the DPS? What about enemy regen skewing results in that time?

 

3 minutes ago, Vayek said:

 

A test on power recharge rates during attack chains, and how that directly impacts a damage per second scale.

This is incredibly variable depending on the lvl of recharge and the set in question.

 

3 minutes ago, Vayek said:

 

And if it's possible, test the accuracy average. Because titan weapons misses A LOT.

It has no negative acc penalties, when slotted for 1 Acc SO the powers should all have the standard 95% ToHit chance.

 

3 minutes ago, Vayek said:

 

 

These tests will likely be best performed without the use of the secondary power set. So, if there is an auto power that assists with endurance, accuracy, tohit, or damage, that power shouldn't even be taken and used. The point is to get as clear and objective results as possible in each category.

May as well not run any powers? That is why we chose Willpower, it provides the least invasive effects while still giving toggles to account for.

 

3 minutes ago, Vayek said:

 

Now, to make this an even more viable and objective test, everyone has to test the exact same map, same power sets, and same archetypes, and compare times. Then use those differences to account for unique human error, play style, and reaction delays.

 

This completely objective test would require at least, but preferably more people to test it. At the very least, this test might show where another power set needs to be buffed up a bit. But don't nerf a power set because another set feels weaker. instead, buff the weaker set.

We agree on this, but as for the last part... what we fear is that there is a major outlier that performs much better.

 

3 minutes ago, Vayek said:

 

So, someone has already proposed some testing standards:

Arc ID 15832

+0/x3

SOs only

No inspirations

No pools besides (Combat Jumping)

No epic powers

No Incarnate powers

That someone would be Hopeling and I 🙂 

 

3 minutes ago, Vayek said:

I suggest:

Scrapper Testing, Brute Testing, Tank Testing

Staff Fighting Against Titan Weapons (they have similar damage numbers for two-handed weapons and mechanics)

War Mace Against Titan Weapons (similar damage, but more difference in AOE styles and mechanics)

Dark armor as a secondary (as it has a PBAoE heal that does minimal damage and doesn't effect endurance drain of the primary set)

 

We should ideally test every primary, not just a few. We want to see underperformers in perspective as well.

 

Also.... Dark Armor most certainly effects endurance effeciency.

[Haijinx]

TW just feels like it misses a lot because momentum ends and you have to slooooow swing to get it back.

[Hopeling]

@Vayek, we've already chosen a set of test protocols to start with, using /WP. But if you want to run another battery of tests using Dark Armor, you are welcome to do so.

[Infinitum]

1 minute ago, Hopeling said:

@Vayek, we've already chosen a set of test protocols to start with, using /WP. But if you want to run another battery of tests using Dark Armor, you are welcome to do so.

Thats probably not the best idea though, because everyone plays differently, if you want the most accurate result, the same person, group of people should do all the testing.

[Galaxy Brain]

If you're looking for just raw DPS chains, we can calculate that easily using the tool tips in game without need to have somebody wail on a stationary target. 

 

Unfortunately, that will not be accurate due to HC changes

Edited September 18, 2019 by Galaxy Brain

[Eclipse.]

34 minutes ago, Hopeling said:

 high-end builds are the ones that can easily have accuracy out the wazoo, while SO builds will feel the penalty hardest.

 

This is why I suggested a change to BaseToHit.

No amount of [reasonable] accuracy in the world will fix a change to BaseToHit, only tohit, which is basically limited to ONLY being available via a handful of skills. Tactics as mentioned, BuildMomentum, and Focused Accuracy.

Accuracy is calculated AFTER a clamp is already applied using BaseToHit.

Assuming the formula is:
HitChance = Clamp( AccMods × Clamp( BaseHitChance + ToHitMods – DefMods ) )

 

If you are fighting a mob with 40% defenses and go from go from 75% BaseToHit 65% with +200% accuracy (fully slotted power+100% from skills) that takes you from a 70% chance to hit to a 50% chance to hit. At 20% (what equal level mobs have, based on a quick look in an AE farm, well 18.5%) that same calculation takes you from 110% chance to hit -- clamped to 95% -- to 90%, so maybe it is still too generous, let's drop it down to 63.5% instead of 65%. That same mob you now have an 87% chance to hit, instead of 95% chance, you'll miss nearly 3x as often.


Yes, you can compensate for it, but you'd have to go really really heavy into accuracy OR [as I mentioned before] pickup tactics and/or focused accuracy. If you go with a -12.5% debuff to BaseToHit instead of -10% then even having tactics AND focused accuracy [unslotted] wouldn't get you back up to the same levels as prebuff. It's a substantial hit [hehe] to their chancetohit

 

Edited September 18, 2019 by Eclipse.

[Hopeling]

5 minutes ago, Infinitum said:

Thats probably not the best idea though, because everyone plays differently, if you want the most accurate result, the same person, group of people should do all the testing.

That's why I said we should check whether results are reproducible.

 

At ten runs per powerset, each taking ~6-8 minutes, we're looking at over a dozen hours just to test every powerset in a single test condition. It's going to become very unreasonable to have one person go through very many conditions; splitting up the workload seems appropriate. Besides, we're primarily interested in variation between powersets, but within a single test condition: we want to know if TW/WP beats WM/WP, and we want to know if TW/Inv beats WM/Inv, but it's less useful to know whether TW/WP beats TW/Inv. So if one person takes a test condition, the results should still be useful even if they don't align perfectly.

9 minutes ago, Vayek said:

I only suggested dark to discourage using toggles at all. Get just testing done on one the attavk set and the endurance consumption.

Not using toggles is equivalent to giving a large recovery buff, since then you're not dealing with their endurance costs.

 

Also, for practical reasons, I simply don't think testing without toggles will work: your character will die. I tried making enemies that don't have attacks, but enemies run away when they can't attack; I don't know how to fix that besides by giving them an attack.

15 minutes ago, Vayek said:

Find something with loads of hitpoints that doesnt attack back (like a test dummy in RWZ) and lay into it for a set time.

So... a pylon? I thought we already agreed that testing under pylon conditions was not useful.

Just now, Eclipse. said:

This is why I suggested a change to BaseToHit.

No amount of accuracy in the world will fix a change to BaseToHit

That is not a thing that's possible to do. Accuracy is an attribute of powers, tohit is an attribute of your character. The closest you could get would be to bake a -tohit debuff into Momentum, but then that will affect non-TW powers too.

 

Moreover, high-end builds certainly can get enough accuracy to overcome tohit penalties. For example, it's very common to build for a capped hit chance against +3 or +4 enemies, against which you have a tohit penalty.

5 minutes ago, Vayek said:

Shouldn't raw DPS chains at least be a part of the data since one of the primary questions is if the set does considerably more damage than others?

No. We are specifically trying to use in-game data under realistic conditions. If you want theoretical DPS calculations on a single target, things like Kaeladin's DPS spreadsheet already exist.

[TheAdjustor]

Just ran the test maps. It's late only had time for 2 runs.

 

Build

 

Didn't realize till after I was done I hadn't bothered to actually put SOs into build up. All the other powers are slotted per the terms or in the case of pulverize worse. Wasn't using it much so it was hardly worth a respec. Combat jumping was taken and has a single def SO slotted. The rest were filled out with powers form the leaping pool and the medicine pool.

 

2 runs timed using windows stopwatch  from the time I first leapt into a spawn till the time the mission complete message (times rounded to the nearest 10 seconds)

1st Run ~10:20

2nd run ~07:10

 

Primary difference in the runs: 1st run I was unfamiliar with the map and did not realize there was a mesa with spawns on it.  would recommend changing this to shiva shard as there is no reason to test anyone's ability to play can you find the one that got away.

 

2nd Run: Being more familiar with the map and the enemies I switched from Killing the enemies where they spawned to a jump in hit, let the taunt aura aggro and herd three groups together before beginning the test.

 

The herding strategy had to be done, because the specified settings produced groups of 6.  A somewhat less than optimal choice.

 

Seeing as WP can easily survive a full x8 spawn of these even with SOs would recommend testing at +0x8 in the future  so the test is not one of how well you heard using combat jumping and Rise To The Challenge.

 

This is a good example of the fact that variance in performance is much more influenced by how the user approaches using the set and their familiarity with the task at hand than the damage output of the  set.

 

Edit: Either way not bad for not having played war mace in over 7 years and back then on a tank.

Edited September 18, 2019 by TheAdjustor

[Hopeling]

11 minutes ago, TheAdjustor said:

2nd Run: Being more familiar with the map and the enemies I switched from Killing the enemies where they spawned to a jump in hit, let the taunt aura aggro and herd three groups together before beginning the test.

I've actually tried to avoid herding. It's not feasible in most indoor maps, and the point of running on x3 was to face x3 spawns rather than x8 spawns. In any case, we need to agree on whether herding is "kosher".

[TheAdjustor]

3 minutes ago, Hopeling said:

I've actually tried to avoid herding. It's not feasible in most indoor maps, and the point of running on x3 was to face x3 spawns rather than x8 spawns. In any case, we need to agree on whether herding is "kosher".

If it's not kosher it's at least treif.

[Hopeling]

27 minutes ago, Vayek said:

No, I didn't mean to attack pylons (this quote is obviously just being taken out context for the sake of arguing. My post very clearly says the point is to get a raw (non-theoretical) damage per second parse.

On what? Target dummies don't work, because they do not take damage. That's why people use pylons.

 

I'm not asking just to argue. I legitimately do not know of a way to do the thing you want to do, besides a pylon or something like it.

27 minutes ago, Vayek said:

It's becoming more unclear to me exactly what you're goal is with these tests then.

The goal is to quantify performance under something resembling normal gameplay conditions: a mix of enemy ranks, enough enemies that AoE is worthwhile but not so many that ST becomes unimportant, and comparing analogous builds. This is not a thing that anybody has done before in any depth, to my knowledge; meanwhile, things like sustained ST DPS are well-explored, such as in pylon tests and the spreadsheet I just linked.

 

If you want to look into raw ST DPS, you're welcome to, but it isn't what this project is about - because, bluntly, we tried arguing from raw power data for about 25 pages, and got nowhere with it. TW is insanely good on paper or under perfect conditions (see my first post on page 3 for the numbers), but it's not clear how well this carries over into actual gameplay.

Edited September 18, 2019 by Hopeling