Project Spelunker

[Widower]

I think you misunderstand. That's the COMPRESSED size.

 

Ah, you wound me, friend PK. I was thinking - why decompress the whole archive when I could decompress but a part of it and parse that, then rinse and repeat 'til there are no more files? A lot more work for me, yes, but it should get it done. (Unless, of course, my right honorable friend Cipher gets to it first!  ;D)

[_NOPE_]

Well, you COULD do that... but then you'd have to actually pay attention to the file names, and only uncompress the files that haven't been parsed already by using the Viewer as a cross reference... and those files are named... painfully.... I mean, if you want to go ahead and do that work, I'm all for whatever help I can get... but I'm telling you right now, it'd make my eyes BLEED to compare these file names against each other to find unparsed files:

 

 

 

 

 

Right now, the Parser program handles all this for you, by deleting all of the "previously parsed" files from your system, and then grabbing a random file that's left, on every loop.

[Widower]

Friend PK, there is nothing more pleasurable in the world than eye-bleed-inducing nonsense!

[Zep]

Good thing I have a 4TB Spinner that is almost empty -- not sure I have room on the NVME.

[Zep]

Any reason I could not make a whole bunch of copies of the executable and alias them to the same data folder? Or would they all try and work on the same file at the same time?

 

Or will it all be done by the time I get home from work today? :)

[_NOPE_]

Good thing I have a 4GB Spinner that is almost empty -- not sure I have room on the NVME.

 

I do hope you mean 4TB, instead of 4GB.

 

Any reason I could not make a whole bunch of copies of the executable and alias them to the same data folder? Or would they all try and work on the same file at the same time?

 

Or will it all be done by the time I get home from work today? :)

 

You don't even have to make copies of the executable. Just run the same executable as many times as you want, and point them to the same directory. The way the program works, on EVERY program loop, it:

 

• Checks for already parsed files in the database, and deletes them from the local folder
  
• Puts the list of all remaining files into an array and grabs a RANDOM file from that array to process
  

 

After it's done processing the file, it deletes the local file, and repeats the loop again, refreshing its own internal lists every time.

[WanderingAries]

A bit late in the game (having a hard time getting back to threads I've read), but what I was Trying to get at was something like having you put up say 1-10Gb chunks for people to munch on that would speed up the initial DL and make it easier for those who appear to be using SSDs. Personally, I'm sitting on a good ~6Tb so long as I don't go back to my AV project (been avoiding) in the meantime. I hadn't started anything because I first read this when it was like half a page long. :p

[Widower]

A bit late in the game (having a hard time getting back to threads I've read), but what I was Trying to get at was something like having you put up say 1-10Gb chunks for people to munch on that would speed up the initial DL and make it easier for those who appear to be using SSDs. Personally, I'm sitting on a good ~6Tb so long as I don't go back to my AV project (been avoiding) in the meantime. I hadn't started anything because I first read this when it was like half a page long. :p

 

It's a good idea, and I personally would take advantage, but I suspect it'd be difficult to manage - you wouldn't really want two people grabbing the same chunk. That would be... awkward.

[justicebeliever]

It's a good idea, and I personally would take advantage, but I suspect it'd be difficult to manage - you wouldn't really want two people grabbing the same chunk. That would be... awkward.

 

No one likes awkward chunks...

[WanderingAries]

It's a good idea, and I personally would take advantage, but I suspect it'd be difficult to manage - you wouldn't really want two people grabbing the same chunk. That would be... awkward.

 

Yes it would require some vetting. Perhaps instead of everyone grabbing, have OP Assign chunks as people ask to help. This would require the download folder be organized in such a way that there's a minimal amount of confusion as to who should be grabbing what.

 

Again, a bit late to the party, so I'd hate to undo some of the work already begun.

[_NOPE_]

I kinda really wanted it to be setup in such a way that would allow anyone at any time to grab all the files, start the parser, and start parsing. No fuss, no muss.

[WanderingAries]

Oh wow, I don't torrent so started up the TAR file for download on my first post (in here) today. I'm at 4% running <2MB/s, 11.3Gb file size right now. O.O

[_NOPE_]

Yeah, it's gonna take a while. That's why I'm trying to compress the whole archive into as small of a space as possible, and then re-upload it. I'll host it on CityOfPlayers.com, and I'll also host it via torrent, for as long as it takes to get this thing done. ETA, about 16 hours until the files are compressed.

[Zep]

Yes, TB not GB

 

Were gonna see how many threads this baby can handle.

[WanderingAries]

14% :p

[Zep]

Wow Win Zip sucks monkey balls -- like really salty smelly ones.

 

Should have listened and used 7z right off.

 

Estimating 90 min to uncompressed all the files.

[_NOPE_]

I've got some bad news for you, that's just the first round of unzipping.

[Zep]

Win zip did the first ok. Maybe 20-25 min.

 

It just wasnt willing to do more then 1 at a time with the second wave. I was not going to queue them up 1 by 1 time.

 

Second wave took 77 min.

 

Before bed I'll get it going.

[Zep]

In case any of you end up with all the files to be parced in separate directories too.

 

 

https://itstillworks.com/make-one-folder-out-multiple-folders-26192.html

 

Never deleted 18k+ empty folders before and that is saying something for me.

 

[Zep]

Looking at cpu use, disk use, net bandwidth, I could run over 200 of the app at the same time.

 

Could you make a local config file so I could set my name and data directory so I could just start clicking away?

[Zep]

Got an error about the server being busy 3 times - hope we arnt over loading you :)

 

Another error I can email you - dont have an image host set up to post it -- suppose we could buddy on discord and do it that way.

 

 

----

 

Morning and 19 errors

 

Can these be saves to a log?

[Hyperstrike]

Okay, even with Gig Internet, the torrent isn't stable enough to download in less than a day...

Downloading the TAR? A couple hours.

[Amatyr]

I'm just throwing the images onto imgur.

 

Last night's only error:

 

[hp]

16

[_NOPE_]

I'm just throwing the images onto imgur.

 

Last night's only error:

 

 

That's perfect, thank you. As I have time, I'll make new versions to fix these.