_NOPE_

The single greatest modern rock album of all time, created by a band, I'll be none of you have ever heard of. Every song on this album is a hit. EVERY. SONG. I don't know why they never got any airplay or made any traction. I could listen to this album on random repeat forever.... This song specifically sums up their "sound" pretty well.

Okay, version 1.5 is now up on my server, and successfully processed the file that failed for you @WanderingAries. Thank you for your support. Just to warn you, I may not be able to do any more coding the rest of this weekend.

@KauaiJim @WanderingAries Thanks for the processing power, and for reporting these issues. This does not in fact appear to be an issue of conccurrency, at least not at this time (unless you've got some different error messages to report to me Jim?). The "file not found" issue actually appears to be RELATED to the "file size" issue discovered in Ionic's Zip functionality. And this would happen AFTER the WARC file was done being downloaded, thus why it stops the works, because it's an issue with processing the file (specifically, trying to add one of the output files into the final zip file). Look at your stacktrace here Aries: Line 2 of the stack trace shows that they can't find the file "xxxxxx" because line 3 of the stack trace shows that when Ionic's code is trying to get the length of the filename... the filename is too long. Windows filenames + paths accept a maximum length of 260 characters, and that filename/path 'C:\Users\USERNAME\AppData\Local\Temp\CoH_Forums_Output\https_accounts.google__servicemailpassivetruermfalsecontinuehttps_mail.google.com_mail_ui3D226ik3Debd018fc9626view3Datt26th3D12b563ab717b036626attid3D0.126disp3Dinline26realattid3Df_gem5zqoe126zwscc1ltmpldefaultltmplcache2.com_ServiceLogin' is 294 characters long! So it can't "find the file", because the filename itself is an invalid and unrecognizable filename for the OS. I don't know why I didn't think of that issue before this. I'll write into my "SanitizeFileName" procedure code at the end to truncate any paths tbat long down to the required 260 length limit (probably go 250 to be "safe"). Now THIS one, I'm concerned about: Because it's reporting the same error as the other, but the filename, 'C:\Users\USERNAME\AppData\Local\Temp\CoH_Forums_Output\http_boards.cityofheroes.com_styles_CoH_misc_background_renamed_70.jpg' is a mere 70 characters long. However, looking at the filename... notice the "_renamed_70"? That is in fact a bit of code that I had put into my SanitizeFileName method to try to help prevent concurrency issues.... by making sure each time I wrote an output file, if such a file of that name already exists, my code loops through all possible integers from 1 to infinity until it finds a filename that DOESN'T yet exist. However, if two processors at the same time happened to perform that check for the same base filename while processing the same output file at the same time... damn, I hadn't considered that. So, what I think I'm going to do here is in fact add an additional bit to my "temporary output" code to create a temp folder within the output folder using the same trick so that each instance makes a unique folder for itself to work in. That should take care of that issue... I hope. My family is going to a newly bought vacation cabin this weekend, sometime this morning. So I'll try to push this through and troubleshoot and debug/test the solution this morning before we leave. Keep you version 1.4 processors, and after I release 1.5, extract the new processor in another directory... just in case I fail in my troubleshooting and release a WORSE processor. Then you can fall back on the old "slightly faulty" one.

Once you get an error, please copy and paste the entire text of the error to this thread, and I'll investigate. If you happen to have the filename of the file that you were processing as well, that would be good.

If it errors out because it can't download the file, it should just loop and try to pickup the next file. If it errors out for any other reason, then I need to look at the errors and troubleshoot them.

Nevermind, I figured it out. Silly me, not realizing that my PUBLISHED web.config actually OVERWRITES whatever settings I try to set in IIS Configuration Manager. Go figure, and something good to remember going forward. I'll have to keep in mind that to change any "IIS" settings, I'll have to go google how to change them in the web.config file rather than being able to use a nice GUI. Basic page is now up, but for now, there's nothing to see: I'm going to build a mini-app to start extracting the zip files into the directory that that link goes to, so that you all will be able to see some of the progress so far yourselves. The final site won't look anything like this, trust me. This is just the temporary holding place while I work on the other phases to get this thing up and operational while phase 1 is still processing. What I'm picturing here is a basic front page that contains a search bar, as well as a list of links to the various "index" pages, to allow people to browse the painful way file-by-file if they want to. That list of links is really mostly for Google though, so that they have something to start from for their spider to start crawling. Most people will want to use the Search Feature. My final goal, ultimately, is to have it look JUST as it does on the Wayback engine, with the main subdomain by default loading a version of the boards front page that contains a special search bar at the top, but from there, people should theoretically be able to click around like they used to be able to way back in the day, with the realization that some of the links will go nowhere because external sites no longer exist, and that obviously the former "live" functionality is no longer usable, because, it's an archival state. If I get REALLY smart later, I'll be able to figure out how to remove those elements altogether, but for now, I'll just be happy if I can modify the HTML pages to point to each other instead of their original link destinations that no longer exist!

Y'all are doing awesome, last check we already had over 20% done already! w00t! Now, I DID have to do a restart on the server this morning, to try to troublshoot some issues I've been having with setting up IIS all on my own, so you may want to check your processors for errors, and restart them if they have errors. That's due to my inexperience as a webmaster (I'm naturally a desktop app code monkey, just inching my way into the web world). I already have the appropriate subdomain setup and pointing to the right IP address, but when you try to go to the site, for some reason it's not picking up my default document, giving me a 404 error, which is really weird. Actively troubleshooting now:

In other news, I'm at about my wit's end with my current web host. Look at this crap... Webhost Me Webhost Me Webhost Me Webhost Me It's like talking to a wall... ARRRRRGGGGHHHH!!!!

So, um... SLIGHT bugfix. I know I just posted 1.3 just two hours ago, but I was checking out the output files so far, and the actual files are all there and great! Thank you! But I didn't realize that Ionic's Zip functionality by default stored the PATH information to the files as well. So.... I suggest downloading 1.4 right now and using that instead, because I fixed it. Unless you WANT me to know where your temp path folder and computer login username is: I have a little script that I just wrote that I've tested, that will go through all of the output files, and if the ZipEntry filename isn't in the root path, it'll snip off the rest of that path until it's in the root, so no problems going forward or with the existing data. The future files will all now look like this, as they are supposed to:

Version 1.3 is now up on my server, if you want to get that, it has only two changes from 1.2: Removed the "input" check, to speed things up, since everything is uploaded and where it needs to be, and Fixed a slight bug with the calculations of the number of files that need to be uploaded, to prevent it from being a negative number.

We are back up! New server without the space constraints, though time will tell if it'll be able to handle the strain of a large number processing at once. To connect, you must go back to the site and download version 1.2 that I just uploaded. It points to the new server, and also has a bugfix for that "can't download file" error that just makes it go to the next iteration instead of failing. Now, to ensure that we're not double-processing those files that have already processed, I've added a new check at the beginning of each loop, that checks for already processed zips and compares them to WARCs in the input directory. This adds a few seconds to every iteration, but it's necessary for now until I can get all of the files copied over into the input directory. Once that happens (I suspect sometime tomorrow morning), I'll release version 1.3 with that check taken out. But feel free to process as you wish now, it appears to be working - I tested it on my work PC and my home PC (PCs located at two separate locations).

In related news... hey, anyone got $4300 they can loan me for a few years? 😛

Okay, let's put this on hold for the moment, I just received word that the amount of data is overwhelming my shared host. 😮 So, yeah... I'm going to recode the program to point to a different server that can hold that capacity. In the meantime, let's just put a pin in this. I'm going to delete all of the "input", and "processed" files to get the host off my back, but I will download the "output" work that you all have done so far, so that that isn't lost. Please stand by.

I'll try to change that to start the next loop instead of throwing an error. It'll have to be tomorrow though, I have to put my kids to bed.

@KauaiJim you can just delete those files. This is how the process works: I have an "input" folder that contains all of the WARCS that need to be processed. The program pulls from there. When someone successfully downloads a WARC, the server copy is moved to a "processing" folder. When someone successfully fully processes and uploads back a zipped up version of the extracted files, only THEN does my program move the "processing" file over to the final "processed" folder. So, any jobs that get "stuck" due to one issue or another? The WARC file will just end up getting "stuck" in the "processing" folder and won't be picked up by anyone else. I will manually move those files back to the input folder when all of the other files are done processing, but I want to keep them there for now, just in case there's errors within the WARC files that I need to investigate. Make sense? And yes, some of those files are MASSIVE. But you should note that this is from the Internet Archive "leeching" the forums for ALL data. Not just the posts, but also all images, etc. Most of the large files that I found were actually animated gifs that people were using for their avatars... I think @Acemace holds the record so far.

I just pushed out a software update to the processor, please get the latest version for maximum efficiency. Here's the update notes: Version number label added Added the number of files that have been uploaded and are waiting to be processed. I don't know why I forgot about this number, but it's there now. Added a "Verbose Logging Enabled" checkbox, which is disabled by default. Previously, "verbose logging" was always enabled, and I noticed that refreshing the screen for EVERY single file action was making the program take FOREVER. By disabling this, the program processes each file at LEAST four times faster. But, you can still enable it and process slower, if you want to have a log of every single thing that the program is doing. Just know that it will slow things down considerably! You can see here that I'm no longer showing individual files being added to the zip file and being deleted from the file system. That's been pushed to the "verbose logging" option. I've moved the "popup progress bars" to 5 pixels to the right and down from the main window, and did a little "trick" where I forced the window to be "always on top" for a moment, then immediately turned "always on top" off, to force the progress window to appear "in front of" the main window. Previously, it often times ended up hidden behind the form. I forgot to add a numerical dot for this, but the "files processing" number now includes the current program's processing of a file. Previously, it was only accounting for OTHER people that were processing files. Now, it counts everyone including the current running program. Please let me know if you have any comments, questions, or concerns about these changes, or the program in general.

Ah, sorry, I hadn't thought about that. I almost always browse on a desktop PC, where you can easily click and drag the scroll bar. I hadn't considered mobile users. I pushed it to a file attachment.

You.... didn't want to know how the process ended?

I'm still in the slow process of uploading the WARC files to my server.... so hopefully you guys won't process them faster than I can upload them, heh. And here's a sample output from the processing of one file. You read this from the bottom up, as it shows up in the textbox and pushes the old info down the box (attached). example output.txt

I'm just... gonna do some name dropping here to try to get some people on board that have helped in the past... @Oubliette_Red @Zep @WanderingAries @Amatyr @justicebeliever @Cipher

...again. Here's try #2. If anyone wants to help in this project, now is your chance. Please follow the link below, read the page, download the program and start processing! http://www.cityofplayers.com/project-spelunker/

What is your Operating System? What is your system architecture? (x86 or x64)? Are you running any anti-virus or internet security programs? If so, they could be silently stopping the program without notification. Try disabling them temporally, and then trying again. If you're running Windows, please send me a few screenshots of ALL of the currently running processes from Task Manager IMMEDIATELY after trying to start the program. We can go from there.

Your post made no sense to me. If you're referring to the Modder tool, you DON'T have to register anything, I NEVER send out emails, and I have no idea why the program may not be starting, as you've provided no error messages. Basically, my response was to my inability to comprehend the meaning of your post in any way.

What operating system are you using? Did you have a previous installation of Homecoming, or is this a new installation? Did you play this installation of Homecoming successfully BEFORE using the Modder tool? How many and which mods have you installed through the modder program already? What is your Homecoming file path on your PC? Where can the homecoming.exe file be found on your PC? Where did you point the "CoH Root Directory" path to?