_NOPE_

I've now updated the OP to make things a bit more clear, to consolidate all links in one place. Let me know if I need to do anything else to make things better!

I have no idea why I didn't think of these ideas. Thanks Zep! Okay, well I shouldn't save the directory, because for all I know users might want to run it multiple times in multiple different directories. But I can save the username. So I am now. I'm also making an "Errors" subdirectory underneath the chosen directory once the first error is generated, then making an Errors.log file under there, and if the error is related to a file, it'll move that file automatically to the errors subdirectory. Also, what I've found with those timeout errors is that my server has a limited number of total allowable connections, so that kind of limits the speed of this project. So when you get those timeout errors (which will now just be in the console instead of a popup), you'll just need to move those files back into the main directory when you're done with all the other files and try again. New version of the Parser is now available to download. User the link in the OP.

Also, you can run the app multiple times at the same time, but you'll want to wait until the first time it deletes all known already processed files and starts processing its own first file. Then it's safe to start a new instance.

At this rate, maybe a month? If more people join in, sooner? Yes, as I said earlier in the thread, every time the program is done with a file, it deletes it, then queries the database to check if any MORE files have also been processed in that timeframe, and deletes them too.

The Parser is just that, a parser. There's no UI because there's nothing for you to do in the program, just let it run and do it's thing, it works automatically. If you want to stop it at any time, just close the window, and then you can restart it again at any time. The VIEWER is probably what you want, if you want to actually browse what people have parsed so far, but it's kind of crap right now, until I have time to make the link between the UserIDs and the ThreadIDs, that's the major missing piece right now, and I'll get around to parsing that when I have more time... it's the only thing I never have enough of... time.

Yeah, my instructions were a bit of a hyperbole. I just don't know why people bother to archive already archived files... it does nothing for them! The errors are normal, you can disregard them.

I haven't been processing, because I've been letting my computer sit to use all of it's resources for: Downloading a fresh copy of the archive to work with Unzipping that fresh copy the two times necessary to get to the WARCs Rezipping those WARCs to one flat file at the highest possible compression Creating the torrent file from that 7z file Uploading that torrent file to my site Uploading the actual 7z file to my site Starting an upload of all the raw WARCs in case people want to download invididual ones to parse to my site uncompressed, I'm not going to zip each one one at a time! Bugfixing the parser So, that's what I've been up to, oh, and family stuff too... and marketeering for the first time in my spare time. After these WARCs are all uploaded, then I'll start processing again.

Final results after selling all of them? Average sale price was between 200K and 500K, not surprising, so for some of them I probably lost some money, and some I probably still made some money. If I'd let them sit instead of going nuts and dumping them all, I could probably be a very rich man in a couple of weeks, but it's fun to experiment, and maybe I just made a few people happy when the prices of their enhancements dropped considerably for that short window. BTW, the highest "sell it now" value as a Positron's Blast for 2 Mill... the lowest was a Force Feedback... for 1 Inf, LOL.

Inspired by your thread Shinobu, I decided to jump into marketeering this weekend, for fun and profit. Since I was out and about with my family all week with limited access to my PC at only periodic moments, it was a perfect time to test this stuff out. So, here's what I did. I had started running those story arcs for my character to build up reward merits, and then turn them in for converters and boosters. Once I got about a million, this was the processed that I finally settled on to start to make inf fairly quick with minimal interaction: [*]Find a popular enhancement set category that contains both uncommons and rares that go from 20 to 50. I settled on "Targeted AoE" [*]Buy up ALL of the uncommon recipes for that set that don't require rare salvage of any sort, since I was going to rely on the fungibility of the market to do the magic for me. In this case, Air Burst and Detonation, but not all of them. [*]Make all of those uncommon enhancements [*]Use my converters to convert Out of Set, but by Set Category of Targeted Aoe: [*]If the bugger converted to the most desired set (in this case Positron's Blast), it's all gravy. I've been throwing them in the market for about 1.5 mill so that they sell reasonably quickly. They cost me maybe 40K to craft, I bought the recipes for maybe under 10K to buy them now, and I used 2 converters, and 75K for listing fee at that price for a total of about.... 325K. That gives me a profit of roughly 1.175 million per enhancement: [*]If instead, I get another uncommon, well I just suck it up and keep converting until I get a rare, because that's still a HUGE profit margin, even doing like 3 conversions using 6 converters: [*]Now, on the quite likely possiblity that I get the LESS desireable rare enhancement set "Annihiliation", no big deal, it just means a slightly different strategy. Instead, I spend one more converter, another 100K, to convert it into some other random rare, and put it on the market for the same 1.5 mill, where it will sell.... eventually: By just doing this one single thing over and over again, over the course of the weekend I went from 45 merits to about 89 million influence in my pocket, and over 100 crafted rare enhancements sitting there waiting to sell: And I'm now about to DUMP all of them on the market for 1 Inf each, just to see what happens.... for fun, for profit, and for CHAOS! I'm curious to see what the results will be, but yeah, with the market fungibility, and the discrepancy between the cost of converters vs. the value of the enhancements on the market after conversion... this is just WAY too easy for WAY too little work. And I wasn't even waiting that long to see if my goods would go for a higher price! Thanks for the inspiration Shinobu! Happy marketeering to you!

EXACTLY! Totally nuts. I am quite committed to my sales.

And now http://www.cityofplayers.com/2012%20City%20Of%20Heroes%20Forums%20Archive.7z is available for direct HTTP download, if anyone wants to try that instead of torrent. I'll update the OP when I have some more time.

Yeah, I've got roughly 75ish rare enhancements for sale at about 1.5 million each. I want to let them sit there like that, but I'm getting kind of inpatient...

I've just updated the parser to account of all of these errors. Please download the latest version of the parser from the link in the OP. Thank you all! Frankly, since the parser is working "most of the time", all of the "bug fixing" I'm doing is really just throwing try...catches around my assumptions of good data, and if it's bad data, I'm throwing it into a generic "page" that I'll look at later to see why it failed to parse.

Yeah dude, just use imgur.

Torrents are only as good as the peers, and since I'm currently the only host of that torrent... it's not the best, so if you're downloading it, keep it after you download it, so other can use your bandwidth. If you'd like to try direct HTTP from MY server of the smaller file instead of the Internet Archive's version, well, I'm still in the process of uploading that... currently at 45%, ETA about 10 hours... and then I'll provide that link as well.

Yeah, but it also means it'll take that much longer to decompress, is the only drawback.

I just verified on my own PC remoted jn from my phone that that link should be working. The final downloaded 7z file should be about 93GB ;D

I won't be able to test this until I get home late tonight, because I remoted jn on my phone, but if anyone wants to try my own torrent and let me know if it works for them, it'd be much appreciated: http://www.cityofplayers.com/2012%20City%20Of%20Heroes%20Forums%20Archive.7z.torrent

You could use my BBCoder to make your life easier...

Post a screenshot of the exact error please?

That's perfect, thank you. As I have time, I'll make new versions to fix these.

LOL +1 Inf Empath/doesn't matter because you'll only take the compulsory blast? xD

I've got some bad news for you, that's just the first round of unzipping.

That's not a bad idea. I've got space. Someone send me a list and links of eveything useful, and I can get it all centralized.

Problem solved. It turns out, I wasn't getting any more tips because I had three missions already, and apparently you can't get tips when you have three missions. I had finished one of those missions, and right after that, the next mob gave me a tip. I guess I had assumed that after I got ten points, I'd automatically get the morality mission. But appare tly, that's wrong and you have to get an 11th tip. Did the mission, now I've got the countdown power. YAY!