_NOPE_

By the way, I just figured out that since I didn't make it a single instance application, you can have multiple instances running and process more faster: You don't HAVE to of course... just saying....

Relevant:

Aaaaaaand, we have a new king! Thanks Cipher!

I just published a new version. Let me know if that auto-stops your processing after the next file processes like I intend it to. What I'm doing now is, if I run across an error in parsing (like, for example, a User Page like Red's error), I'm just throwing it into the "Pages" bucket. Then, sometime later, we can look at the Pages bucket and see if there's anything more we can do to salvage that data. If anyone has any better ideas than that, let me know. I kind of expected this to just be the first round of parsing anyways, where we got the lump done, and then there'd be additional parsing as people request more and more specific queries, which would necessitate additional "cross-reference" tables, which would necessitate additional parsing for that data.

Yes, try that!

My 99.9% (and stalled at this point) downloaded 219Gb tar file is definitely not extracting properly :( No problem with disk space, 7zip only reports 123 files (which is ~1.5Gb). Tried bash tar as well, no change. But, I'm processing the files it will give me and that's working. Thanks for your efforts! You can try the direct download option if you want. I had both running, and the direct download completed faster, so I cancelled the torrent. Maybe the torrent is in fact missing a bit? Ugh.

Yeah, it looks like the Internet Archive sat on this for a while and then stuck it at the bottom of its resource pile, since it's been around so long (and probably nobody cared about it until now!). Thanks for your efforts!

Yeah, this is what I'm talking about when I talk about old, junk data. It's apparently just an HTML page with no content whatsoever, weird: I'm processing the rest of that file now, so don't worry about that one.

Yes, they should be. Let me know if your DateTimeStamps in the Viewer don't show up properly. :P Also, new Parser just pushed out, no more WARCSKIP files! Bug you found was fixed!

CHEATER CHEATER PUMPKIN EATER! Heh, that's fine, it all goes to the cause. And we're in this for the long haul. I just wonder what it would take to get others to help, because really, the program takes VERY little resources, and runs in the background, and doesn't really interfere with anything (unless they come across an error). Speaking of which, I'll get the patched up shortly. Also, with the Viewer now, you don't really have to report the files that you processed. Hell, I probably should do away with creating those WARCSKIP files too, since I'm also using the database check, it's kind of redundant...

Holy shniekies, how are you processing files so fast? Did you just pick a bunch of small files and stick them in a folder by themselves? Curious about your system specs now.

Sure you can! Just move a specific file to a folder by itself and then point the program to that directory.

And our first "non-me" Contributor is on the board! Thanks Oubliette_Red!

/JRanger

+1

I've also just update the Parser to automatically delete files that have already been parsed. Redownload the Parser if you want this new functionality.

I've uploaded the beta version of the Viewer. You can find the link in the second post of this thread, along with a basic breakdown of the program's functionality. So now you can start to see what you're working for!

Fair enough. It's true.

Sounds like an argument for expanding the team even further to me... As for the cease and desist... do you think a queue system where no money was exchanged, just information, would trigger that?

Other than "because it's against the Terms of Service"? No. Hmmm... I wonder if a Terms of Service is worth anything if it's unenforced... interesting legal question...

I can't speak for everyone else, but my reason for wanting a queue system is because I'm VERY appreciative of this chance to get my game back, my digital home. And I want to show my appreciation by supporting it, in any way I can. The inability to be here, on the site, at JUST the right moment to contribute... is frustrating. I haven't been able to give anything yet.

Here's what I'm working on right now: A modification to the parser, so that it fetches the global list of all files from my server, then fetches the list of all files that have been already processed on the server. Then, on the loop between files being processed, I'll check those values again, and automatically delete/remove any local files in the working directory that have already been processed. Should save some time and confusion. Working on cleaning up the Viewer interface a bit, adding a Help menu option and a Contributors window that'll hold all the data about who contributed what, and running totals. Re-downloading the archive myself to get a fresh unaltered copy to populate the "all files" list, since I'd already deleted the ones that I'd already processed! It'll be interesting to see which downloads first, the HTTP request or the torrent, as they've both going EXCRUCIATINGLY SLOWLY:

Have you all talked at all about the "queuing" suggestion that I offered previously? What I'm thinking is this. People can "pledge" a certain amount, dropping their payment information down at the same time to be withdrawn at a later time. Their pledge goes into a waiting queue, while they are informed of the ESTIMATED time/date when their pledge amount will be withdrawn from their account, based on their location in the queue and the amount that they pledged. If anything ever changes about the estimated date/time of the withdrawl, they are informed automatically via e-mail. At any point, the players can delete their pledge from the queue. The next time they want to pledge, that pledge goes at the end of the queue. If they remove their pledge from the queue, everyone else's pledges get moved up automatically. Each month, a process runs automatically that goes through the current pledges, and withdraws enough to meet the required costs. I could assist in setting this up on the back end, if there's any issues figuring it out logistically. The benefits of this are: [*]Everyone can get a "chance in line" and not have to wait for a certain date/time to show up to be on the site and post a donation. [*]People can pledge as early/late/often as they want to, and each of those pledges can count as a separate pledge instance. [*]Homecoming has a known pool of future pledges, feeling more comfortable in its security. [*]Homecoming doesn't have to worry about having tons of extra money on-hand that's in their coffers, all they have is "information" that COULD potentially be turned into money at some point in the future, but they can still run "lean", officially. The only drawback that I can see if the concerns about Homecoming holding onto that payment information, rather than another third party vendor such as PayPal, but that could be alleviated if there was another third party company that could just hold the information and provide it to Homecoming on an as-needed basis. I don't know if such a service exists or not, but I'm sure someone else will speak up if they know about such a service. What say you?