Project Spelunker

[WanderingAries]

JUST shy of 70% complete and roughly 3.7k remaining

[WanderingAries]

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-13337-20120904-211756.warc' ---> System.Exception: Failed to upload file. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.
   at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
   at System.Net.CommandStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.Net.ConnectionPool.Destroy(PooledStream pooledStream)
   at System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)
   at System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage)
   at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
   at System.Net.CommandStream.Abort(Exception e)
   at System.Net.CommandStream.CheckContinuePipeline()
   at System.Net.FtpDataStream.System.Net.ICloseEx.CloseEx(CloseExState closeState)
   at System.Net.FtpDataStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at COH_WARC_Processor.FTPclient.Upload(FileInfo fi, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 223
   --- End of inner exception stack trace ---
   at COH_WARC_Processor.FTPclient.Upload(FileInfo fi, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 227
   at COH_WARC_Processor.FTPclient.Upload(String localFilename, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 108
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 136
   --- End of inner exception stack trace ---

 

Oddly enough, this error is from a fresh instance: (silly forum posted it above)

 

Seems there was a matching pair on the other device

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-18836-20120904-155007.warc' ---> System.Exception: Failed to upload file. ---> System.Net.WebException: Unable to connect to the remote server
   at System.Net.FtpWebRequest.GetRequestStream()
   at COH_WARC_Processor.FTPclient.Upload(FileInfo fi, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 204
   --- End of inner exception stack trace ---
   at COH_WARC_Processor.FTPclient.Upload(FileInfo fi, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 227
   at COH_WARC_Processor.FTPclient.Upload(String localFilename, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 108
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 136
   --- End of inner exception stack trace ---

 

 

Edited August 19, 2021 by WanderingAries

[WanderingAries]

Next from the i7. I caught the Win11 VM having restarted itself on the iMini (likely updates) again, so I'm sure in the end you'll have a few left over to rerun. Oh and we're at just over 3k remaining and just shy of 75% complete.

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-27954-20120911-092547.warc' ---> System.Exception: Failed to download file. ---> System.Net.WebException: Unable to connect to the remote server
   at System.Net.FtpWebRequest.GetResponse()
   at COH_WARC_Processor.FTPclient.Download(String sourceFilename, FileInfo targetFI, Boolean PermitOverwrite) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 364
   --- End of inner exception stack trace ---
   at COH_WARC_Processor.FTPclient.Download(String sourceFilename, FileInfo targetFI, Boolean PermitOverwrite) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 431
   at COH_WARC_Processor.FTPclient.Download(String sourceFilename, String localFilename, Boolean PermitOverwrite) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 259
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 84
   --- End of inner exception stack trace ---

 

Edited August 20, 2021 by WanderingAries

[WanderingAries]

[Ura Hero]

Now at almost 84% complete.  Looks like we will be done somewhere around Wednesday.

[WanderingAries]

5 hours ago, Ura Hero said:

Now at almost 84% complete.  Looks like we will be done somewhere around Wednesday.

 

Give or take any individual processing of files like what @The Philotic Knight had to do at the end last time. I noticed this time we've been listed at a solid 150 instances for a bit. I am kinda curious as to the electrical impact though. My UPS reports around 250-300W (powering the PC + the iMini, TV, etc) of constant use, but IDK how to translate that.

[WanderingAries]

1123 left to go and a hair under 90% completed.

[_NOPE_]

I've moved all files from the "processing" queue back to the input queue, so needless to say, you'll probably get a bunch of failures this time around. Please post them here, because they'll probably be erroneous failures. It just means that while you were able to successfully upload the zip file, since I moved the processing file, it couldn't find it to  move it to the "processed" folder. I'll move them manually once I've confirmed that we have a good Zip file for the WARC. As it stands right now, we're at 91.53% complete, and I can TASTE it. So close.

 

I don't think I'm going to have to do any manual processing @WanderingAries, as all errors so far have been transmission errors, NOT programming/parsing errors. So I finally got this thing working correctly for all files. We might just have to retry a few that failed to upload or download.

 

Thank you EVERYONE for your work. When I make my "SiteMap" mini-app, I'll also write in there code to grab a final tally of how many files each username processed, and create a "Thanks To" file showing everyone's usernames and totals.

 

I'm excited! 😄

 

[WanderingAries]

So it looks like we're at 949 and 92.37%, but only 21 active instances (looks like my PC was idle after a point today), so we either slowed down a little or the purge knocked out some misreporting. I'll spin-up the four in a bit, but here are the errors that were triggered. The iMini 'appears' to have not been affected, but I never feel certain that the VMs aren't locked up (fewer resources).

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-13031-20120905-192906.warc' ---> System.Exception: Failed to upload file. ---> System.Net.WebException: Unable to connect to the remote server
   at System.Net.FtpWebRequest.GetRequestStream()
   at COH_WARC_Processor.FTPclient.Upload(FileInfo fi, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 204
   --- End of inner exception stack trace ---
   at COH_WARC_Processor.FTPclient.Upload(FileInfo fi, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 227
   at COH_WARC_Processor.FTPclient.Upload(String localFilename, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 108
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 136
   --- End of inner exception stack trace ---

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-12198-20120904-023830.warc' ---> System.IO.FileNotFoundException: File /processing/boards.cityofheroes.com-threads-range-12198-20120904-023830.warc not found
   at COH_WARC_Processor.FTPclient.FtpRename(String sourceFilename, String newName) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 575
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 148
   --- End of inner exception stack trace ---

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-11295-20120904-113010.warc' ---> System.IO.FileNotFoundException: File /processing/boards.cityofheroes.com-threads-range-11295-20120904-113010.warc not found
   at COH_WARC_Processor.FTPclient.FtpRename(String sourceFilename, String newName) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 575
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 148
   --- End of inner exception stack trace ---

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-23324-20120906-171725.warc' ---> System.Exception: Failed to upload file. ---> System.Net.WebException: Unable to connect to the remote server
   at System.Net.FtpWebRequest.GetRequestStream()
   at COH_WARC_Processor.FTPclient.Upload(FileInfo fi, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 204
   --- End of inner exception stack trace ---
   at COH_WARC_Processor.FTPclient.Upload(FileInfo fi, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 227
   at COH_WARC_Processor.FTPclient.Upload(String localFilename, String targetFilename) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 108
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 136
   --- End of inner exception stack trace ---

 

[_NOPE_]

Two of those have not had a successful zip upload file @WanderingAries, so their WARCs will remain in the input. The other three did, so I've moved them over to the processed folder. Thanks! 93.3% complete!

[Ura Hero]

I had a bunch of processes knocked offline.  I blame PK.  😉

[_NOPE_]

By the way, these files that we've been working with are the same files that I downloaded over two years ago, from here:

 

https://archive.org/details/archiveteam-city-of-heroes-main

 

Now, in between now, and when I got those files two years ago, there is a chance during all of my testing that I may have misplaced some files. I'm redownloading that same file now, unzipping my original "backup" of the files that I made about a year ago, and will compare all three directories together. If there's any missing files that I find in the original archive from archive.org, I'll add them into the queue.

[WanderingAries]

You broke the iMini's cluster. Will have to start back up later tonite. 😛

[WanderingAries]

Well sheet...good thing those had failed I suppose. My Parallels 17 trial expired (because I'm rethinking that software based on their pricing tactics) and when I tried to reinstall v16, it loaded, complained about needing to reduce the cores/ram I had configured, and quit, but it did something else to the vHDD because the VM got corrupted. Luckily, I'd made clones of the core VMs I had before the upgrade, so I'm restoring them. Unfortunately, this means two VMs with 2 instances each, so the iMini gets taxed a little more. IDK which four files were affected by this, but I wanted to point it out.

 

Ugh, I to go so far as to manually remove all the Parallels support files because even a new app install claimed the VMs were damaged. I'm still using my backup copies, but way easier than recreating the VMs from scratch.

Edited August 26, 2021 by WanderingAries

[_NOPE_]

Sooooooo.... It's a really damn good thing that I double checked the original source files a second time, because....

 

 

...ooops....

 

 

Found about another 6,000 files that I'd somehow lost between when I started this project two years ago and now. My best guess is that I was using a sample amount for testing and split them out.

 

I've also uploaded a version 2.2 of the parser, if y'all want to get it. The only difference is that I'm calculating the total files based on what's actually IN the directories now, instead of my previous rookie mistake of hard coding a "magic number" of the number of files I THOUGHT there were...

 

I apologize for any inconvenience this may cause. I'm still re-downloading the original file just to make SURE that I'm not missing anything else from my original download, and when I extracted the files from the three layers of archives they stored them under, and then re-archived them into the smaller 7z format.

[WanderingAries]

Yeah, the number went up a tad, but was rather excited to see the "98.24%".   :p

[_NOPE_]

14 minutes ago, WanderingAries said:

Yeah, the number went up a tad, but was rather excited to see the "98.24%".   😛

[_NOPE_]

I am also still re-downloading the ORIGINAL file, just to make TRIPLE sure that I didn't miss anything else, but I've got a food feeling that the 18,000 something number is the correct number of files:

 

Edited August 26, 2021 by The Philotic Knight

[WanderingAries]

I'm about ready to swap the current batches to 2.2, so we'll see what the numbers are for sure in a minute. You Can tell when something fails on that end and never gets uploaded right?

 

Heh, Just over 5600 right now @ the upper end of 69...% :p

Edited August 27, 2021 by WanderingAries

[_NOPE_]

Already back up to 80%, good job everyone!

[WanderingAries]

Yeah, I noticed it went pretty quick compared to before...was suspicious.

[Michiyo]

I've thrown a few instances at this on my 12c/24t and 32c/64t servers.  I've noticed a handful of processes actually fully hanging up (the date/time on the UI being a day or 2 old) and force closed them. I hope this hasn't caused any issues.

[WanderingAries]

37 minutes ago, Michiyo said:

I've thrown a few instances at this on my 12c/24t and 32c/64t servers.  I've noticed a handful of processes actually fully hanging up (the date/time on the UI being a day or 2 old) and force closed them. I hope this hasn't caused any issues.

 

I've noticed that they'll do that for up to a couple days, but then somehow clear themselves. But yeah, I never know if it's Truly locked up or not and takes ages to get Those to end manually as you have to hope you catch it in between batches (when the buttons become usable). The worst thing that would  happen is there'll be a few at the end that need to be thrown back in the bunch.

[WanderingAries]

Ok, so I caught the iMini VMs having done an unexpected restart (danger of betas), so hopefully those finished whatever they were doing, but I doubt it. So that's 4 back up and running. The i7 gave this error (below) and We're showing 355 remaining @ 97.8% complete. So that sounds like we're done with this round after everybody's chewed on 5 more each, give or take the ones in the failed/redo list.

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-27057-20120906-172820.warc' ---> System.Exception: Failed to move file to process from '/input/boards.cityofheroes.com-threads-range-27057-20120906-172820.warc' to '/processing/boards.cityofheroes.com-threads-range-27057-20120906-172820.warc'.
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 115
   --- End of inner exception stack trace ---

 

[WanderingAries]

Dude! We're at less than 200 already?!