Project Spelunker

[_NOPE_]

Yuppers

[WanderingAries]

Are you wanting the errors that are unique in type or with different filenames? The latter would be a heck of a list unless you have a way of knowing what failed on your end.

[_NOPE_]

Just the different types of errors. Each error SHOULD have the filename listed, right?

 

I can use that file as a sample to debug the issue. No need for duplicate files with the same problem.

 

And as for what failed on my end?  Thats what the stack trace is for. 😉

[WanderingAries]

Heh, these are new 😛

 

This one seems to have an issue where it lost the '\' in the path

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-24760-20120904-203807.warc' ---> System.IO.FileNotFoundException: Could not find file 'C:\UsersUSERNAMEAppDataLocalTempCoH_Forums_Output2http.com_snapfish_viewsharedphoto_p15541346791138707_l6004733025_g4695186025_cobrandOid1000_campaignNameShareeNewReg_30FreePrints_2010Feb_otscSYE_otsiSPNRjsessionid7F5840A943EB6DAC642E8E3707FFDE14'.
   at Ionic.Zip.SharedUtilities.GetFileLength(String fileName) in C:\projects\dotnetzip-semverd\src\Zip.Shared\Shared.cs:line 43
   at Ionic.Zip.ZipEntry.MaybeUnsetCompressionMethodForWriting(Int32 cycle) in C:\projects\dotnetzip-semverd\src\Zip.Shared\ZipEntry.Write.cs:line 800
   at Ionic.Zip.ZipEntry.WriteHeader(Stream s, Int32 cycle) in C:\projects\dotnetzip-semverd\src\Zip.Shared\ZipEntry.Write.cs:line 1078
   at Ionic.Zip.ZipEntry.Write(Stream s) in C:\projects\dotnetzip-semverd\src\Zip.Shared\ZipEntry.Write.cs:line 2220
   at Ionic.Zip.ZipFile.Save() in C:\projects\dotnetzip-semverd\src\Zip.Shared\ZipFile.Save.cs:line 168
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 125
   --- End of inner exception stack trace ---

 

Not sure at all what this is about

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-28950-20120906-072354.warc' ---> System.Exception: Failed to move file to process from '/input/boards.cityofheroes.com-threads-range-28950-20120906-072354.warc' to '/processing/boards.cityofheroes.com-threads-range-28950-20120906-072354.warc'.
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 105
   --- End of inner exception stack trace ---

 

Duplicate file on server?

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-24096-20120905-085601.warc' ---> System.ApplicationException: Target file /processed/boards.cityofheroes.com-threads-range-24096-20120905-085601.warc already exists
   at COH_WARC_Processor.FTPclient.FtpRename(String sourceFilename, String newName) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 543
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 140
   --- End of inner exception stack trace ---

 

[WanderingAries]

This one sounds like a backend thing/

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-15632-20120906-135428.warc' ---> System.Exception: Failed to move file to process from '/input/boards.cityofheroes.com-threads-range-15632-20120906-135428.warc' to '/processing/boards.cityofheroes.com-threads-range-15632-20120906-135428.warc'.
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 105
   --- End of inner exception stack trace ---

 

[WanderingAries]

This one's interesting:

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-11187-20120904-135450.warc' ---> System.OverflowException: Value was either too large or too small for an Int32.
   at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
   at System.Int32.Parse(String s)
   at COH_WARC_Processor.FTPclient.Download(String sourceFilename, FileInfo targetFI, Boolean PermitOverwrite) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 320
   at COH_WARC_Processor.FTPclient.Download(String sourceFilename, String localFilename, Boolean PermitOverwrite) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 242
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 81
   --- End of inner exception stack trace ---

 

[_NOPE_]

I come back from the vacation weekend and.... HOLY CRAP PEOPLE, good job!

 

 

Now I gotta up my game and get the other phases figured out... I'll look into errors mentioned above. Thanks @WanderingAries

[_NOPE_]

On 6/12/2021 at 11:25 PM, WanderingAries said:

This one seems to have an issue where it lost the '\' in the path

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-24760-20120904-203807.warc' ---> System.IO.FileNotFoundException: Could not find file 'C:\UsersUSERNAMEAppDataLocalTempCoH_Forums_Output2http.com_snapfish_viewsharedphoto_p15541346791138707_l6004733025_g4695186025_cobrandOid1000_campaignNameShareeNewReg_30FreePrints_2010Feb_otscSYE_otsiSPNRjsessionid7F5840A943EB6DAC642E8E3707FFDE14'.
   at Ionic.Zip.SharedUtilities.GetFileLength(String fileName) in C:\projects\dotnetzip-semverd\src\Zip.Shared\Shared.cs:line 43
   at Ionic.Zip.ZipEntry.MaybeUnsetCompressionMethodForWriting(Int32 cycle) in C:\projects\dotnetzip-semverd\src\Zip.Shared\ZipEntry.Write.cs:line 800
   at Ionic.Zip.ZipEntry.WriteHeader(Stream s, Int32 cycle) in C:\projects\dotnetzip-semverd\src\Zip.Shared\ZipEntry.Write.cs:line 1078
   at Ionic.Zip.ZipEntry.Write(Stream s) in C:\projects\dotnetzip-semverd\src\Zip.Shared\ZipEntry.Write.cs:line 2220
   at Ionic.Zip.ZipFile.Save() in C:\projects\dotnetzip-semverd\src\Zip.Shared\ZipFile.Save.cs:line 168
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 125
   --- End of inner exception stack trace ---

 

This error was caused by DIS FAWKING GUY:

http_www5.snapfish.com_snapfish_viewsharedphoto_p15541346791138707_l6004733025_g4695186025_cobrandOid1000_campaignNameShareeNewReg_30FreePrints_2010Feb_otscSYE_otsiSPNRjsessionid7F5840A943EB6DAC642E8E3707FFDE14

 

Completely f***ing with my "SanitizeFileName" method, because of the .com followed by the super long freaking string afterwards. My method didn't know what to do with such a long "extension". So, I "hacked" the problem by truncating any extensions longer than 3 characters to just 3 characters long (but only when the filename with it would be over 250 characters.

 

So that monster file now becomes this: "http_www5.snapfish.com" whether it likes it or not... because that file is an asshole:

 

[_NOPE_]

On 6/12/2021 at 11:25 PM, WanderingAries said:

Not sure at all what this is about

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-28950-20120906-072354.warc' ---> System.Exception: Failed to move file to process from '/input/boards.cityofheroes.com-threads-range-28950-20120906-072354.warc' to '/processing/boards.cityofheroes.com-threads-range-28950-20120906-072354.warc'.
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 105
   --- End of inner exception stack trace ---

 

Another concurrency issue, but probably a rare one. I'm not going to deal with this one... basically two of your clients on the same PC tried to process the same file at the same time. I expect every random file to only be attempted to be processed once by a single PC... your 50,000 processors running at once caused this 😛 . When we get near the end of it, when there's like a dozen files left, we'll most likely have to make sure each PC is only processing one file at a time.

[_NOPE_]

On 6/12/2021 at 11:25 PM, WanderingAries said:

Duplicate file on server?

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-24096-20120905-085601.warc' ---> System.ApplicationException: Target file /processed/boards.cityofheroes.com-threads-range-24096-20120905-085601.warc already exists
   at COH_WARC_Processor.FTPclient.FtpRename(String sourceFilename, String newName) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 543
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 140
   --- End of inner exception stack trace ---

 

Same concurrency issue, just happening at a different part of the process. You're just trying to do too much at once on the same PC dude! 😛

[_NOPE_]

20 hours ago, WanderingAries said:

This one sounds like a backend thing/

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-15632-20120906-135428.warc' ---> System.Exception: Failed to move file to process from '/input/boards.cityofheroes.com-threads-range-15632-20120906-135428.warc' to '/processing/boards.cityofheroes.com-threads-range-15632-20120906-135428.warc'.
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 105
   --- End of inner exception stack trace ---

 

Same concurrency issue. (Seeing a trend?)

[_NOPE_]

10 hours ago, WanderingAries said:

This one's interesting:

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-11187-20120904-135450.warc' ---> System.OverflowException: Value was either too large or too small for an Int32.
   at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
   at System.Int32.Parse(String s)
   at COH_WARC_Processor.FTPclient.Download(String sourceFilename, FileInfo targetFI, Boolean PermitOverwrite) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 320
   at COH_WARC_Processor.FTPclient.Download(String sourceFilename, String localFilename, Boolean PermitOverwrite) in C:\Projects\COH WARC Handler\WHB WARC Processor\FTP Client.cs:line 242
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 81
   --- End of inner exception stack trace ---

 

THIS one is actually kind of an interesting and cool error, well at least to me because I'm a dork.

 

It's erroring out at this point in code, when I'm trying to instantiate the ProgressBar to show you the progress of the FTP download:

 

Well, the "Maximum", "Minimum", and "Value" properties of a ProgressBar object are all int32s.

 

int32s have a maximum allowable value of 2147483647 as per Microsoft.

 

So that means a progress bar can't handle any numerical values higher than that. And when I'm making these bars, I'm setting the max value to the total number of bytes for the file. Do you see where I'm going with this?  The 10 largest WARC files will OVERLOAD the maximum allowable values for a ProgressBar's int32 values:

 

So, since it's only 10 files that are going to have problems with this.... I'm going to "cheat", and... do this:

 

 

And this....

 

 

Basically just set the max of the bar to the max of an int32, and when it comes time to set the VALUE of the bar to higher than that, just reset it to 0. So, the bar will go back to empty and start filling again. It sucks, but it's only informational, so at least you'll still see that progress is happening, even though the bar's length won't be accurate. Sorry.

 

I'll release that in v 1.6 after ensuring that the new code works with this specific file without issue.

[WanderingAries]

47 minutes ago, The Philotic Knight said:

Same concurrency issue. (Seeing a trend?)

 

Digital example of my brain? X.X

[_NOPE_]

1.6 is now available for download with (sort of) "large file support". Also, if a file errors, it now automatically deletes the local WARC file, so that you don't have to.

[WanderingAries]

3 hours ago, The Philotic Knight said:

 

The more of these I see, the more that coding itch rears its head. 😛

[WanderingAries]

3 hours ago, The Philotic Knight said:

1.6 is now available for download with (sort of) "large file support". Also, if a file errors, it now automatically deletes the local WARC file, so that you don't have to.

 

If nothing else, you're getting some practice with the syntax. 😛

[WanderingAries]

Took a break (work), but we're at 91.75%.   :p

[WanderingAries]

On 6/7/2021 at 10:00 PM, The Philotic Knight said:

 

 

Just shy of 94% done. 😛

 

I just got a legit .Net error about instances trying to access the same files shortly after bumping up to 12 instances on the 3770k. Then I launched another 12 on the M1 and for the hell of it, another 12 on the 2012 MBP.  😄

Edited June 15, 2021 by WanderingAries

[WanderingAries]

Everything erred overnight, only 4 dozen left, so I'm gonna use the MBP to crunch through since I can run separate instances on isolated VMs.

[WanderingAries]

Down to four files. Had to drop to one OS/Instance due to what proport to be local errors, but pretty sure they were online (trying to access same file).

[_NOPE_]

Sorry to get your hopes up, but I just pushed the 32 files that were "stuck" in processing back to input. We can see if they'll process the second time. If not, I need to look at whatever errors are generated.

 

[WanderingAries]

3 minutes ago, The Philotic Knight said:

Sorry to get your hopes up, but I just pushed the 32 files that were "stuck" in processing back to input. We can see if they'll process the second time. If not, I need to look at whatever errors are generated.

 

 

I just saw that 😛

 

[WanderingAries]

How big are these last few files? Some of this stuff is probably bounce back from instances that "appear" frozen.

 

IE:

• boards.cityofheroes.com-threads-range-14589-20120911-131326.warc
• Has been extracting for a very long time in Win8.1. Have managed to handle a few files in Win10 since it started.

[WanderingAries]

New Error, multiple instances (moved back to 4 in Win10 when we hit 3 dozen). I see now there are only 5 "active" files, but 0 in the upload though. So. maybe it simply means "nothing to get". Killed the 4 instances that had the error as well as that VM. Still waiting on the file I mentioned above in Win8.1.

 

There was an error in the application. Please copy and paste the following text in a message to the program author:
System.Exception: Error while processing file 'boards.cityofheroes.com-threads-range-28315-20120906-190149.warc' ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
   at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
   at System.Collections.Generic.List`1.get_Item(Int32 index)
   at COH_WARC_Processor.MainForm.BtnProcess_Click(Object sender, EventArgs e) in C:\Projects\COH WARC Handler\WHB WARC Processor\MainForm.cs:line 75
   --- End of inner exception stack trace ---

 

Edited June 15, 2021 by WanderingAries

[_NOPE_]