Oubliette_Red

GoFundMe? I'm currently seeking employment, but I could chip in a few inf for the duration.

I'm getting the server timeout error now, whether I parse a single file or multiples. Errors.log

I am not a search guru when it comes to servers, since I've never needed the use of one. The only thing I've been able find so far that came close is https://www.tier.net/infrastructure

I threw it into Suggestions thread that it'd be nice if we had a Fun & Games section.

Either in the Reply or Preview option of posting, you'll see: + Attachments and other option Click on it.

Is the file too big to attach to your post?

I grabbed the latest build and on one large file and I am see a new line popping up regularly in the console. "Adding User <######> in thread <######>

He PK. I see the changes you did to the parser. I'm am now getting an error text file and associated .warc file moved to a sub-directory. When I look in the error text files, I do not see a file mentioned in the error. You original popup error (while not copy-able, listed the associated file right at the top. I have appended the numerical portion of the .warc file that was in the error sub-directory. When it's only 1 file not a big issue, but one of my sub-folders had 3 files in it with 1 error file that has multiple entries and no indication as to which error belongs to which file. Errors-24942-20120906-170948.log Errors.log

I am on the road, so not really able to contribute further until I get home sometime tomorrow.

Finally finished the 3.4m file I started at 3:30... whew. I am now parsing the following: boards.cityofheroes.com-threads-range-27851-20120906-050401 boards.cityofheroes.com-threads-range-11881-20120905-014249 boards.cityofheroes.com-threads-range-12360-20120905-044715 boards.cityofheroes.com-threads-range-11936-20120904-182407 boards.cityofheroes.com-threads-range-16457-20120907-063526 boards.cityofheroes.com-threads-range-11231-20120911-085820 boards.cityofheroes.com-threads-range-12767-20120906-235325 boards.cityofheroes.com-threads-range-11242-20120904-020055

You can run multiple instances of the parser .exe. As mentioned above, once you get the files extracted, you can create multiple folders and drag multiple .warc files into each of them. Then each time you launch the parser, just point to a different folder. Once all of my folders have been parsed, I will usually run the parser on the main folder to auto-delete any files someone else has parsed in the meantime. I've been running chunks of similar sized files, but PK may call on folks to proceed by file name, that way people can call out a section of files they are parsing at any given time, to minimize overlap.

Parsing error

Siphon Speed + Inertial Reduction

I just got that timeout as well. Is this maybe a problem with larger files, or just bad luck? I was running 4 separate chunks of 20k files, could just be a hiccup anywhere along the internet. I am also currently parsing boards.cityofheroes.com-threads-range-12058-20120912-121713 which is a 3.4MB file so I believe it will take a while.

Maybe an error message that's a little more NCSoft on the eyes? I'm no Paragon of virtue when it comes to this though... Nice pun, although the error messages would have been generated by Paragon Studios and were not always helpful. I recalled talking to one of the Devs way back in the day and he relayed a story regarding their move to Jira for bug tracking and that it auto-generated tickets for player-generated bugs using the reporting system. Bug ticket #1 was by a player, who'd description of their issue was "Shit's broke, fix it." Funny as hell but not at all helpful.

Big parse error this time around.

Have all of the 1m+ files been parsed already? I actually tried to run a parse of the full folder to remove already parsed files but none of those were deleted.

I suppose we ( I ) could stick to a strictly numerical filename process to avoid stepping on each other's toes. My apologies if I started a poor trend. <.<

I just started running multiple instances to parse all of the strictly 20k files, so 20,202-20,999

You can get around this by placing chunks of files in different folders and pointing each instance of the parser to different folders.

Hmmm... parser is hung up on a file, not progressing and no error. "boards.cityofheroes.com-threads-range-15130-20120904-150600, line 29072 of 313737" Pulled the other files out and reran the same file. Got the following error:

PK, are the parsed files being reported properly? The last couple of runs I've done have said "Now processing file # 0 of 1." or "#0 of 3'". It seems to continue as expected and creates the .warcskip file(s). If it's removing the files anyway, probably don't need the .warcskip files. But I throw it/them back into the archive folder, use it kind of like a placeholder for what I've already parsed.

Caught in the act. :D I figured I grab some smaller ones since the second one I parsed was something like 250k.

PK, I got the following error parsing a group of 5 files.