City RNG appears to be slightly biased

[gchpaco]

So some discussion concerning the CoX RNG had me dig out my chat logs and look for everything that was "rolled a" and check the RNG seeding.  With 208,000 samples (as I didn't consistently log chat), I saw the attached results, when every number's decimal portion is truncated (so 19.2 -> 19, 0.02 -> 0, and 100.0 which happens sometimes -> 100; that's why the 100 bucket is so odd).  There appears to be a subtle but definitely present bias in the City RNG toward high values.

 

The graph of the 1% buckets:

And I've attached the excel worksheet with my data.  I generated it with

grep 'rolled a ' chatlog\ 20* | sed -E 's/.*rolled a ([0-9]+[.][0-9]*).*/\1/' | awk -F. '{ print $1 }' | sort | uniq -c

But this isn't especially difficult to do and I expect any similar piece of code would suffice.

 

RNG.xlsx

[Keen]

Ran the script above against all my logs and here's the result.

 

[gchpaco]

Keen notes that this command worked better for him for export to a Google spreadsheet.

 

 

grep 'rolled a ' chatlog\ 20* | sed -E 's/.*rolled a ([0-9]+[.][0-9]*).*/\1/' | awk -F. '{ print $1 }' | sort | uniq -c | awk '{print $2 "," $1 }'

 

[ROBOKiTTY]

It's so clean it has to be intended.

Edited May 11, 2020 by ROBOKiTTY

[Caulderone]

Streakbreaker?

[gchpaco]

Just now, Caulderone said:

Streakbreaker?

Streakbreaker lines show in the log as lines like

2020-05-10 02:17:17 HIT Centurion! Your Ball Lightning power was forced to hit by streakbreaker.

which doesn't cause an RNG value to be selected and so isn't picked out.

[tidge]

Pure anecdote: I've always felt that Streakbreaker was occurring too often for situations where my 'final chance to hit' was 95%. My understanding: If the wiki is to be believed, when the 'final chance to hit' is > 0.90,  Streakbreaker should prevent a "0.95" player from missing "three times". That is, the first miss is 'natural', the second miss is 'consecutive' and then the streak is automatically broken by 'streakbreaker' (on the next attack). I spent some time with my logs (small sample size) and it looked like streakbreaker was appearing (on average) after around 20 attacks (for 95% base)... which would imply that what should have been a 5% 'final chance to miss' was more like a 10% 'final chance to miss': Streakbreaker was fighting against the (assumed) weighting of hit rolls > 0.95 which was 'flooring' the effective 'final to hit chance' at around 90%.

 

I grew frustrated by my attempts to parse the logs, and I didn't want my amateurish attempt to study those be a lone voice of paranoia. I appreciate the folks willing to look at this more seriously.

[ArchVileTerror]

Sorry, Windows user here (one day I'll work on learning Linux).

Is there something I can do to grep on my inferior operating system?

[macskull]

51 minutes ago, tidge said:

Pure anecdote: I've always felt that Streakbreaker was occurring too often for situations where my 'final chance to hit' was 95%. My understanding: If the wiki is to be believed, when the 'final chance to hit' is > 0.90,  Streakbreaker should prevent a "0.95" player from missing "three times". That is, the first miss is 'natural', the second miss is 'consecutive' and then the streak is automatically broken by 'streakbreaker' (on the next attack). I spent some time with my logs (small sample size) and it looked like streakbreaker was appearing (on average) after around 20 attacks (for 95% base)... which would imply that what should have been a 5% 'final chance to miss' was more like a 10% 'final chance to miss': Streakbreaker was fighting against the (assumed) weighting of hit rolls > 0.95 which was 'flooring' the effective 'final to hit chance' at around 90%.

 

I grew frustrated by my attempts to parse the logs, and I didn't want my amateurish attempt to study those be a lone voice of paranoia. I appreciate the folks willing to look at this more seriously.

Streakbreaker will force a hit after a single miss with a hit roll of anything above 90. The list in the wiki can be read as "the number of misses you are allowed before streakbreaker activates."

Edited May 11, 2020 by macskull

[Rathulfr]

1 hour ago, ArchVileTerror said:

Sorry, Windows user here (one day I'll work on learning Linux).

Is there something I can do to grep on my inferior operating system?

Use the FIND command.

[Burk]

I don't have as much data as robokitty. I broke it down into hit rolls by the player to an enemy and hit rolls by enemies to the player.

 

 

It looks like the enemies do have a higher chance for higher numbers, but the jump in the 95-99 range is only for the players.

[ROBOKiTTY]

1 hour ago, ArchVileTerror said:

Sorry, Windows user here (one day I'll work on learning Linux).

Is there something I can do to grep on my inferior operating system?

Do you have Python? If so, add the following code to a file with the extension .py, put it in your log folder, and then run it. It'll take a minute.

import os, re

results = []
for filename in os.listdir(os.getcwd()):
	with open(os.path.join(os.getcwd(), filename), "r", encoding="latin-1") as f:
		for line in f:
			reg = re.search(r"rolled a ([0-9]+[.][0-9])", line)
			if reg:
				val = int(float(reg.group(1)))
				results.append(val)
with open("results.csv", "w") as file:
	for i in range (0, 101):
		count = results.count(i)
		file.write(str(i) + "," + str(count) + "\n")

Basically does the same thing as the one-liner and should write the results to a file called results.csv. Then you can graph it in your spreadsheet program.

Edited May 11, 2020 by ROBOKiTTY

[ArchVileTerror]

Thanks for letting me know.

Haven't got around to installing Python on this computer yet, but I do have it on my laptop.  I'll get around to either remedying that tomorrow, or transferring the logs to the laptop.

[Lisava]

One thing that I can think of that could be inflating the 95-100 numbers is the fact that there are some powers that do not display the to-hit rolls unless they're a miss.  Particularly aura powers, like Conductive Aura, Lightning Field, etc.  Most of these auras with proper enhancements/bonuses will have a 95% chance to hit, and they'll be pulsing every second, only ever showing up in the logs if they roll a 95+.

 

Of course this doesn't effect every build, but if you do have an aura in your logs then that's one possibility at least for seeing more rolls in the 95+ range than others.

[gchpaco]

1 hour ago, Lisava said:

One thing that I can think of that could be inflating the 95-100 numbers is the fact that there are some powers that do not display the to-hit rolls unless they're a miss.  Particularly aura powers, like Conductive Aura, Lightning Field, etc.  Most of these auras with proper enhancements/bonuses will have a 95% chance to hit, and they'll be pulsing every second, only ever showing up in the logs if they roll a 95+.

 

Of course this doesn't effect every build, but if you do have an aura in your logs then that's one possibility at least for seeing more rolls in the 95+ range than others.

I can't speak for KiTTY or Keen here, who have much larger peaks than I observe that I cannot explain.  But for me, that may be part of it.  I'm seeing lines like this for Conductive Aura (from elec control)

2020-05-10 02:19:56 MISSED Centurion!! Your Conductive Aura power had a 53.69% chance to hit, you rolled a 97.52.
2020-05-10 02:19:57 You hit Centurion with your Conductive Aura for 20.51 points of their endurance.
2020-05-10 22:58:25 You hit Centurion with your Conductive Aura for 72.61 points of their endurance.
2020-05-10 22:58:25 You hit Centurion with your Conductive Aura for 72.61 points of their endurance.
2020-05-10 22:58:25 You hit Immunes Surgeon with your Conductive Aura for 51.92 points of their endurance.

For Lightning Field on an elec/elec blaster:

2020-05-11 01:22:00 You hit Conscript with your Lightning Field for 2.94 points of their endurance.
2020-05-11 01:22:01 Lightning Field missed!
2020-05-11 01:22:01 MISSED Guardian!! Your Lightning Field power had a 95.00% chance to hit, you rolled a 99.98.
2020-05-11 01:22:02 You hit Conscript with your Lightning Field for 26.59 points of Energy damage.
2020-05-11 01:22:02 You hit Conscript with your Lightning Field for 2.94 points of their endurance.
2020-05-11 01:22:10 You hit Conscript with your Lightning Field for 25.32 points of Energy damage.
2020-05-11 01:22:10 You hit Conscript with your Lightning Field for 2.94 points of their endurance.
2020-05-11 01:22:12 You hit Conscript with your Lightning Field for 25.66 points of Energy damage.
2020-05-11 01:22:12 You hit Conscript with your Lightning Field for 2.94 points of their endurance.
2020-05-11 01:22:14 You hit Conscript with your Lightning Field for 25.66 points of Energy damage.
2020-05-11 01:22:14 You hit Conscript with your Lightning Field for 2.94 points of their endurance.
2020-05-11 01:22:17 Lightning Field missed!
2020-05-11 01:22:17 MISSED Conscript!! Your Lightning Field power had a 95.00% chance to hit, you rolled a 99.57.
2020-05-11 01:22:30 You hit Conscript with your Lightning Field for 31.18 points of Energy damage.

Aside: there's definitely something weird going on there, but it does seem to report both hits and misses, but only chance rolls on misses.  That may be a manifestation of the problem.  So we need to filter out aura powers.

[Bopper]

I love the analysis. There does seem to be a bias and it appears it could be simply the logic of not outputting hit rolls on hits. The graph by Burk is a great illustration of this. For him, clearly he has a 95% chance to hit on every attack, and the hit rolls before the bias are uniform (0-95%). I suspect he may have 1-2 powers that were only 92.5% but, let's not pick nits.

 

The other great thing about Burk's plot is he also showed results for just the enemies, where you could see a slight trend in its data. This is likely due to a similar issue (hit rolls not being displayed on hits) and how enemies of different levels and different ranks get different accuracy modifications. So he has a good distribution of enemies with varying chances to hit, and the skewing can be seen across the entire spread of 0-100%.

 

Nice work everyone, hope the root cause does get pinned down.

[Veracor]

I'm going to assume there's something like a damage aura that's inflating the positives for the outliers, since they don't display chance to hit unless they miss.  Regardless there is still a slope so I decided to look a bit at the source code.  I did find something that might explain it.

 

First, the combat handler uses this line to generate a number whenever it wants to make a hit roll, and never uses any other method:

 

fRandRoll = fRand = (float)rand()/(float)RAND_MAX;

 

Which is completely normal, if a bit inefficient (two type castings per roll).  CoH does have a library or two for generating floats but it appears the dev who wrote this didn't care and just used C's rand().  I'm going to assume C's rand() has no significant bias outside of what Google says with modulo operations.  And I'm not seeing any operations performed on the randomized number other than comparing the hit roll to it.

 

So I moved on and eventually looked at CalcToHit(), because I wasn't seeing anything odd elsewhere (there's some hackery with the streakbreaker but the data in this thread does not include the streakbreaker).  This in particular handles the final bit of the hit roll calculation:

 

	fAccuracyFactor = (ppow->ppowBase->fAccuracy * ppow->pattrStrength->fAccuracy * combatmod_Accuracy(pcm, i) * (1 - fElusivity) );
	fToHit *= fAccuracyFactor;
	fDefenseRel *= fAccuracyFactor;
	
	if(fToHit<0.05f) // Under clamp, penalize defense
	{
		fDefenseRel -= (0.05f - fToHit);
		fToHit = 0.05f;
	}
	else if(fToHit>0.95f) // Over clamp
	{
		fToHit = 0.95f;
	}

	*out_fDefense = CLAMP(fDefenseRel, 0.0f, (1.0f-fToHit)-0.05f);

 

This caught my eye: I'm not sure if fDefenseRel is supposed to be getting multiplied by fAccuracyFactor when fToHit already is.  I am not a programmer and I might be misunderstanding this block, but I see no mention on paragonwiki of accuracy doing more in PvE than just multiplying ToHit.  If this is the case, it would mean CoH is biased towards missing a bit more than it's supposed to, because accuracy would be multiplying target's defenses positively.  The chance to hit is generally a much larger value than defense though, so if correct this bias would end up being minor.

 

If true, this would explain why the numbers seem to be predisposed to generating higher than lower if it is passively causing more high positives in the combat logs with powers (maybe not just auras) that only display a chance to hit if they miss -- this would generate the slowly-ascending slope seen in some of the graphs, especially Burk's enemy-only graph.  These logs were also not all of powers confirmed to log both ways, nor of powers being used against the same enemy defense values/level difference, as well as the enemy hit rolls against players (who usually dodge) are being included with any number of non-player powers that might only show hit rolls if they miss, like auras do.  If someone were willing to autofire a fast attack like Flares on a dummy in RWZ overnight with non-clamped ToHit, that would eliminate all potential false positives (until some rando runs by and buffs you, anyways).

 

It would also explain why it feels that things miss a bit more often than they should in this game, even when accounting for selection bias.  (Titan Weapons >_>)

 

EDIT 1.5 YEARS LATER: In case anyone stumbles into this old thread via forum search or something and actually reads it all, Number Six happened to answer on the HC Discord what this particular portion of the code is doing (It isn't the culprit):

 

Edited October 8, 2021 by Veracor

[Burk]

By my understanding, the basic formula is supposed to be something like

Chance to hit = Accuracy * (ToHit - Defense)

So Defense should be multiplied by Accuracy somehow. But I'm not sure exactly what is being done in that code.

 

Edit: Dang it Veracor! Now you have me looking through the source code too! Curiosity killed Burk's time along with the cat.

Edited May 11, 2020 by Burk

[ROBOKiTTY]

I cobbled together a quick list of aura powers, but excluding them didn't make a meaningful difference. There are probably more powers with oddities like these , and it'd take too much time to filter out, so I went for the opposite approach, which is to look at rolls from one single power.

 

This is looking at only rolls for the power "Fire Ball".

 

So I'm guessing the issue is not with the RNG but how the game outputs combat logs.

[ArchVileTerror]

Ok, that's a relief.  

Still, merits further testing and research, as everything does.  Can never be -too- certain, after all.  And with a broad community posting their findings publicly, as well as their methods for peer oversight and review, things can only improve.

Thanks to all of you who are doing this investigation!

[Redlynne]

12 minutes ago, ArchVileTerror said:

And with a broad community posting their findings publicly, as well as their methods for peer oversight and review, things can only improve.

Just because that's how I approach dealing with all of my build posts is no reason to cross-pollinate it over to this effort.

 

😝

[Veracor]

7 hours ago, Burk said:

By my understanding, the basic formula is supposed to be something like

Chance to hit = Accuracy * (ToHit - Defense)

So Defense should be multiplied by Accuracy somehow. But I'm not sure exactly what is being done in that code.

 

Edit: Dang it Veracor! Now you have me looking through the source code too! Curiosity killed Burk's time along with the cat.

(ToHitTotal - DefenseTotal) was already done earlier in the block.  From my understanding from the wiki, it's supposed to be just (ToHitFinal * Accuracy) afterward.  Unless the step on the wiki was simply omitted because critter Defense is usually low or zero, I'm not really sure why Defense is being multiplied here.

 

Or maybe I'm just misunderstanding what the final line in the block does.

[Burk]

1 hour ago, Veracor said:

(ToHitTotal - DefenseTotal) was already done earlier in the block.  From my understanding from the wiki, it's supposed to be just (ToHitFinal * Accuracy) afterward.  Unless the step on the wiki was simply omitted because critter Defense is usually low or zero, I'm not really sure why Defense is being multiplied here.

 

Or maybe I'm just misunderstanding what the final line in the block does.

One thing that does have me confused is that it doesn't seem to use fDefenseRel to modify fToHit after this point. What exactly does CLAMP do and is it saving fDefenseRel's value with the "*out_fDefense=CLAMP(...)" statement for some reason? Otherwise, what's the point of changing its value, because it looks like fToHit is the only thing returned by the CalcToHit function at the end.

[Veracor]

10 minutes ago, Burk said:

One thing that does have me confused is that it doesn't seem to use fDefenseRel to modify fToHit after this point. What exactly does CLAMP do and is it saving fDefenseRel's value with the "*out_fDefense=CLAMP(...)" statement for some reason? Otherwise, what's the point of changing its value, because it looks like fToHit is the only thing returned by the CalcToHit function at the end.

CLAMP takes a value, a minimum, and a maximum, and then clamps that value to be within that minimum and maximum if it's out of range.  It's doing this with the defense value after it's been multiplied, when (it seems) it should either not mulitply or be using fDefenseAbs instead.

 

The function does return fToHit at the end, but that would be modified by fDefenseRel in one of the clamp cases as well.  It's hard to understand what's going on here.  It's possible there was originally something involving accuracy vs defense when at clamp values that was either abandoned or written out incorrectly.

[Burk]

1 minute ago, Veracor said:

CLAMP takes a value, a minimum, and a maximum, and then clamps that value to be within that minimum and maximum if it's out of range.  It's doing this with the defense value after it's been multiplied, when (it seems) it should either not mulitply or be using fDefenseAbs instead.

 

The function does return fToHit at the end, but that would be modified by fDefenseRel in one of the clamp cases as well.  It's hard to understand what's going on here.  It's possible there was originally something involving accuracy vs defense when at clamp values that was either abandoned or written out incorrectly.

Even in the clamp cases, fToHit is just being set to 0.05 or 0.95. The value of fDefenseRel is not relevant to the final value of fToHit there. The CLAMP, as I suspected and you seem to verify, is only modifying the value of fDefenseRel using the value of fToHit. I'm wondering why they are bothering at that point to change a value that is not returned by the function. Of course it could be leftover code from some change they made and was actually useful previously.