Project Spelunker

[Paulus]

1 hour ago, WanderingAries said:

Dude! We're at less than 200 already?!

 

Just down to some processing now. I'm working on processing a single one on my end, hopefully it will be done soon.

[Paulus]

Finished with an error, pasting below:

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-12535-20120904-231445.warc' ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
   at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
   at System.Collections.Generic.List`1.get_Item(Int32 index)
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 70
   --- End of inner exception stack trace ---

 

Edited August 29, 2021 by Paulus

[WanderingAries]

I think we're at the end of the run. I've got an  "index error" to match the above. I still have to check the iMini now as well, but I think that give or take a few, that's it.

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-11198-20120904-102142.warc' ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
   at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
   at System.Collections.Generic.List`1.get_Item(Int32 index)
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 70
   --- End of inner exception stack trace ---

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-22431-20120905-031542.warc' ---> System.Exception: Failed to move file to process from '/input/boards.cityofheroes.com-threads-range-22431-20120905-031542.warc' to '/processing/boards.cityofheroes.com-threads-range-22431-20120905-031542.warc'.
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 115
   --- End of inner exception stack trace ---

 

 

[WanderingAries]

iMini has some matching results. I've told the current instances to wrap up, but so far they're all in that weird "I'm frozen but not" state, so I can't really tell them to safely stop yet.

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-12361-20120905-152158.warc' ---> System.Exception: Failed to move file to process from '/input/boards.cityofheroes.com-threads-range-12361-20120905-152158.warc' to '/processing/boards.cityofheroes.com-threads-range-12361-20120905-152158.warc'.
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 115
   --- End of inner exception stack trace ---

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-14748-20120904-201657.warc' ---> System.Exception: Failed to download file. ---> System.Net.WebException: The remote server returned an error: (550) File unavailable (e.g., file not found, no access).
   at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
   at System.Net.CommandStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.Net.ConnectionPool.Destroy(PooledStream pooledStream)
   at System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)
   at System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage)
   at System.Net.FtpWebRequest.GetResponse()
   at COH_WARC_Processor.FTPclient.Download(String sourceFilename, FileInfo targetFI, Boolean PermitOverwrite) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 364
   --- End of inner exception stack trace ---
   at COH_WARC_Processor.FTPclient.Download(String sourceFilename, FileInfo targetFI, Boolean PermitOverwrite) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 431
   at COH_WARC_Processor.FTPclient.Download(String sourceFilename, String localFilename, Boolean PermitOverwrite) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 259
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 84
   --- End of inner exception stack trace ---

 

[_NOPE_]

I'll push the ones stuck in processing back to the input folder sometime tomorrow. Thanks everyone. 

[WanderingAries]

One more for morning from the i7 and I think 3 total running.

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-28174-20120905-235500.warc' ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
   at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
   at System.Collections.Generic.List`1.get_Item(Int32 index)
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 70
   --- End of inner exception stack trace ---

 

[WanderingAries]

Checking in on the iMini and have what's probably the last from it.

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-21729-20120904-114415.warc' ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
   at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
   at System.Collections.Generic.List`1.get_Item(Int32 index)
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 70
   --- End of inner exception stack trace ---

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-12407-20120905-134240.warc' ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
   at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
   at System.Collections.Generic.List`1.get_Item(Int32 index)
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 70
   --- End of inner exception stack trace ---

 

[WanderingAries]

Ok, last error of my cluster:

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-28140-20120906-211234.warc' ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
   at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
   at System.Collections.Generic.List`1.get_Item(Int32 index)
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 70
   --- End of inner exception stack trace ---

 

[_NOPE_]

Version 2.3 is out on the server now if you want it. The only change is that I swapped the error out with a message saying basically that the input directory is empty. Another rookie mistake. I'm going through the last few now and will push them into the input folder if they don't have a good valid zip on my end. But I think we're about done here!

[_NOPE_]

Maybe... let's just throw two or three instances at these last few each, so there's less chance of bumping into each other? 😉

 

 

[WanderingAries]

8 hours ago, The Philotic Knight said:

Maybe... let's just throw two or three instances at these last few each, so there's less chance of bumping into each other? 😉

 

 

 

 

[WanderingAries]

Here's a new one from iMini:

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-21273-20120904-160852.warc' ---> System.Exception: Failed to handle large WARC file ---> System.Exception: Failed to get filename from string 'http://badge-hunter.com/castle_says.php?r1=V%20pehfu%20lbh.&r2=...&r3=Lbh%20yvxr%20guvf.'. ---> System.ArgumentException: String cannot be of zero length.
Parameter name: oldValue
   at System.String.ReplaceInternal(String oldValue, String newValue)
   at System.String.Replace(String oldValue, String newValue)
   at WHBennett.WarcExtractionRecord.GetSanitizedFileName(String input, String outputDir) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 555
   --- End of inner exception stack trace ---
   at WHBennett.WarcExtractionRecord.GetSanitizedFileName(String input, String outputDir) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 666
   at WHBennett.WarcExtractionRecord.StreamWARCFile(String sourceFileName, String outputDirectory, ToolStripStatusLabel& tssl) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 241
   --- End of inner exception stack trace ---
   at WHBennett.WarcExtractionRecord.StreamWARCFile(String sourceFileName, String outputDirectory, ToolStripStatusLabel& tssl) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 261
   at WHBennett.WarcExtractionRecord..ctor(String sourceFileName, String outputDirectory, ToolStripStatusLabel& tssl) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 33
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 126
   --- End of inner exception stack trace ---

 

[WanderingAries]

One more from the current Batch:

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-13284-20120904-200558.warc' ---> System.IO.FileNotFoundException: File /processing/boards.cityofheroes.com-threads-range-13284-20120904-200558.warc not found
   at COH_WARC_Processor.FTPclient.FtpRename(String sourceFilename, String newName) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 575
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 158
   --- End of inner exception stack trace ---

 

[WanderingAries]

Um, so it Says there's 8 reuploaded and only 1 active, but...yeah. The i7 has one and the iMini has 4, one of which I managed to set to end when done, but I figured the others would finish and just complain about not having more to grab eventually.

[Paulus]

I have one that’s still going, about 10 hours in since last activity.

[Paulus]

That last one I had processing crashed. No error message to paste.

[_NOPE_]

On 8/31/2021 at 7:59 AM, WanderingAries said:

Here's a new one from iMini:

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-21273-20120904-160852.warc' ---> System.Exception: Failed to handle large WARC file ---> System.Exception: Failed to get filename from string 'http://badge-hunter.com/castle_says.php?r1=V%20pehfu%20lbh.&r2=...&r3=Lbh%20yvxr%20guvf.'. ---> System.ArgumentException: String cannot be of zero length.
Parameter name: oldValue
   at System.String.ReplaceInternal(String oldValue, String newValue)
   at System.String.Replace(String oldValue, String newValue)
   at WHBennett.WarcExtractionRecord.GetSanitizedFileName(String input, String outputDir) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 555
   --- End of inner exception stack trace ---
   at WHBennett.WarcExtractionRecord.GetSanitizedFileName(String input, String outputDir) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 666
   at WHBennett.WarcExtractionRecord.StreamWARCFile(String sourceFileName, String outputDirectory, ToolStripStatusLabel& tssl) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 241
   --- End of inner exception stack trace ---
   at WHBennett.WarcExtractionRecord.StreamWARCFile(String sourceFileName, String outputDirectory, ToolStripStatusLabel& tssl) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 261
   at WHBennett.WarcExtractionRecord..ctor(String sourceFileName, String outputDirectory, ToolStripStatusLabel& tssl) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 33
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 126
   --- End of inner exception stack trace ---

 

I've moved this one over to my "manual processing" folder, and will get to it when I can. Hell, I think out of 18,000+ WARC files, that contain hundreds of URLs within them, I think not being able to turn ONE URL into a filename is pretty damn good odds there. I may just have to manually name that one myself.

[_NOPE_]

We're now down to FIVE files left, and one additional one that I'll have to process manually. I've moved the other five back to the input folder. Grab them while you can!

[Michiyo]

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-18341-20120905-142114.warc' ---> System.Exception: Failed to handle large WARC file ---> System.IO.FileNotFoundException: Could not find file 'C:\Users\Administrator.Mnemosyne\AppData\Local\Temp\2\boards.cityofheroes.com-threads-range-18341-20120905-142114.warc'.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean checkHost)
   at System.IO.StreamReader..ctor(String path, Encoding encoding)
   at WHBennett.WarcExtractionRecord.StreamWARCFile(String sourceFileName, String outputDirectory, ToolStripStatusLabel& tssl) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 45
   --- End of inner exception stack trace ---
   at WHBennett.WarcExtractionRecord.StreamWARCFile(String sourceFileName, String outputDirectory, ToolStripStatusLabel& tssl) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 261
   at WHBennett.WarcExtractionRecord..ctor(String sourceFileName, String outputDirectory, ToolStripStatusLabel& tssl) in C:\Projects\COH WARC Handler\WHB WARC Handler\WarcRecord.cs:line 33
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 116
   --- End of inner exception stack trace ---

 

[_NOPE_]

I'm just going to keep throwing these files back into the input queue over and over again until we either get them processed, or I get an actual coding error, rather than a transmission or "file not found" error (which is caused by multiple people trying to grab the same file usually). Down to THREE! Plus the one I'll have to manually process when I have time.

Edited September 1, 2021 by The Philotic Knight

[Michiyo]

I believe I'm running those 3 currently

[WanderingAries]

I've got one I 'know' is running and one I'm thinking isn't really doing anything as it's an instance that I wasn't able to tell to stop when done. I'm trying not to force them in the event they actually finish, but IDK.

[Michiyo]

Yeah... looking at mine I think they've been hung for a bit >_>

[_NOPE_]

These three files may have some unforseen issue that's making them lock up. I may have to process them manually. 🤨

[_NOPE_]

I'm now processing the last four in Debug mode, with byte-by-byte reporting, which is REALLY going to slow this down. But it's the only way to be SURE that it's still working, and to see the progress and know that it's not locked up. Refreshing this display every few hundred bytes makes things REALLY slow, but at least I'll know: