Jump to content

Project Spelunker


_NOPE_

Recommended Posts

  • City Council

I think you misunderstand. That's the COMPRESSED size.

 

Ah, you wound me, friend PK. I was thinking - why decompress the whole archive when I could decompress but a part of it and parse that, then rinse and repeat 'til there are no more files? A lot more work for me, yes, but it should get it done. (Unless, of course, my right honorable friend Cipher gets to it first!  ;D)

"We need Widower. He's a drop of sanity in a bowl of chaos - very important." - Cipher
 
Are you also a drop of sanity in a bowl of chaos? Consider applying to be a Game Master!
Link to comment
Share on other sites

Well, you COULD do that... but then you'd have to actually pay attention to the file names, and only uncompress the files that haven't been parsed already by using the Viewer as a cross reference... and those files are named... painfully.... I mean, if you want to go ahead and do that work, I'm all for whatever help I can get... but I'm telling you right now, it'd make my eyes BLEED to compare these file names against each other to find unparsed files:

 

t7Kgamm.png

 

 

 

 

Right now, the Parser program handles all this for you, by deleting all of the "previously parsed" files from your system, and then grabbing a random file that's left, on every loop.

I'm out.
Link to comment
Share on other sites

Good thing I have a 4TB Spinner that is almost empty -- not sure I have room on the NVME.

** Asus TUF x670E Gaming, Ryzen 7950x, AIO Corsair H150i Elite, TridentZ 192GB DDR5 6400, Sapphire 7900XTX, 48" 4K Samsung 3d & 56" 4k UHD, NVME Sabrent Rocket 2TB, MP600 Pro 8tb, MP700 2 TB. HDD Seagate 12TB **


** Corsair Voyager a1600 **

Link to comment
Share on other sites

Any reason I could not make a whole bunch of copies of the executable and alias them to the same data folder? Or would they all try and work on the same file at the same time?

 

Or will it all be done by the time I get home from work today? :)

** Asus TUF x670E Gaming, Ryzen 7950x, AIO Corsair H150i Elite, TridentZ 192GB DDR5 6400, Sapphire 7900XTX, 48" 4K Samsung 3d & 56" 4k UHD, NVME Sabrent Rocket 2TB, MP600 Pro 8tb, MP700 2 TB. HDD Seagate 12TB **


** Corsair Voyager a1600 **

Link to comment
Share on other sites

Good thing I have a 4GB Spinner that is almost empty -- not sure I have room on the NVME.

 

I do hope you mean 4TB, instead of 4GB. wCyyU1Z.gif

 

Any reason I could not make a whole bunch of copies of the executable and alias them to the same data folder? Or would they all try and work on the same file at the same time?

 

Or will it all be done by the time I get home from work today? :)

 

You don't even have to make copies of the executable. Just run the same executable as many times as you want, and point them to the same directory. The way the program works, on EVERY program loop, it:

 

  • Checks for already parsed files in the database, and deletes them from the local folder
  • Puts the list of all remaining files into an array and grabs a RANDOM file from that array to process

 

After it's done processing the file, it deletes the local file, and repeats the loop again, refreshing its own internal lists every time.

I'm out.
Link to comment
Share on other sites

A bit late in the game (having a hard time getting back to threads I've read), but what I was Trying to get at was something like having you put up say 1-10Gb chunks for people to munch on that would speed up the initial DL and make it easier for those who appear to be using SSDs. Personally, I'm sitting on a good ~6Tb so long as I don't go back to my AV project (been avoiding) in the meantime. I hadn't started anything because I first read this when it was like half a page long. :p

OG Server: Pinnacle  <||>  Current Primary Server: Torchbearer  ||  Also found on the others if desired  <||> Generally Inactive


Installing CoX:  Windows  ||  MacOS  ||  MacOS for M1  <||>  Migrating Data from an Older Installation


Clubs: Mid's Hero Designer  ||  PC Builders  ||  HC Wiki  ||  Jerk Hackers


Old Forums  <||>  Titan Network  <||>  Heroica! (by @Shenanigunner)

 

Link to comment
Share on other sites

  • City Council

A bit late in the game (having a hard time getting back to threads I've read), but what I was Trying to get at was something like having you put up say 1-10Gb chunks for people to munch on that would speed up the initial DL and make it easier for those who appear to be using SSDs. Personally, I'm sitting on a good ~6Tb so long as I don't go back to my AV project (been avoiding) in the meantime. I hadn't started anything because I first read this when it was like half a page long. :p

 

It's a good idea, and I personally would take advantage, but I suspect it'd be difficult to manage - you wouldn't really want two people grabbing the same chunk. That would be... awkward.

"We need Widower. He's a drop of sanity in a bowl of chaos - very important." - Cipher
 
Are you also a drop of sanity in a bowl of chaos? Consider applying to be a Game Master!
Link to comment
Share on other sites

It's a good idea, and I personally would take advantage, but I suspect it'd be difficult to manage - you wouldn't really want two people grabbing the same chunk. That would be... awkward.

 

No one likes awkward chunks...

"The opposite of a fact is falsehood, but the opposite of one profound truth may very well be another profound truth." - Niels Bohr

 

Global Handle: @JusticeBeliever ... Home servers on Live: Guardian ... Playing on: Everlasting

Link to comment
Share on other sites

It's a good idea, and I personally would take advantage, but I suspect it'd be difficult to manage - you wouldn't really want two people grabbing the same chunk. That would be... awkward.

 

Yes it would require some vetting. Perhaps instead of everyone grabbing, have OP Assign chunks as people ask to help. This would require the download folder be organized in such a way that there's a minimal amount of confusion as to who should be grabbing what.

 

Again, a bit late to the party, so I'd hate to undo some of the work already begun.

OG Server: Pinnacle  <||>  Current Primary Server: Torchbearer  ||  Also found on the others if desired  <||> Generally Inactive


Installing CoX:  Windows  ||  MacOS  ||  MacOS for M1  <||>  Migrating Data from an Older Installation


Clubs: Mid's Hero Designer  ||  PC Builders  ||  HC Wiki  ||  Jerk Hackers


Old Forums  <||>  Titan Network  <||>  Heroica! (by @Shenanigunner)

 

Link to comment
Share on other sites

Oh wow, I don't torrent so started up the TAR file for download on my first post (in here) today. I'm at 4% running <2MB/s, 11.3Gb file size right now. O.O

OG Server: Pinnacle  <||>  Current Primary Server: Torchbearer  ||  Also found on the others if desired  <||> Generally Inactive


Installing CoX:  Windows  ||  MacOS  ||  MacOS for M1  <||>  Migrating Data from an Older Installation


Clubs: Mid's Hero Designer  ||  PC Builders  ||  HC Wiki  ||  Jerk Hackers


Old Forums  <||>  Titan Network  <||>  Heroica! (by @Shenanigunner)

 

Link to comment
Share on other sites

Yeah, it's gonna take a while. That's why I'm trying to compress the whole archive into as small of a space as possible, and then re-upload it. I'll host it on CityOfPlayers.com, and I'll also host it via torrent, for as long as it takes to get this thing done. ETA, about 16 hours until the files are compressed.

I'm out.
Link to comment
Share on other sites

Yes, TB not GB

 

Were gonna see how many threads this baby can handle.

** Asus TUF x670E Gaming, Ryzen 7950x, AIO Corsair H150i Elite, TridentZ 192GB DDR5 6400, Sapphire 7900XTX, 48" 4K Samsung 3d & 56" 4k UHD, NVME Sabrent Rocket 2TB, MP600 Pro 8tb, MP700 2 TB. HDD Seagate 12TB **


** Corsair Voyager a1600 **

Link to comment
Share on other sites

14% :p

OG Server: Pinnacle  <||>  Current Primary Server: Torchbearer  ||  Also found on the others if desired  <||> Generally Inactive


Installing CoX:  Windows  ||  MacOS  ||  MacOS for M1  <||>  Migrating Data from an Older Installation


Clubs: Mid's Hero Designer  ||  PC Builders  ||  HC Wiki  ||  Jerk Hackers


Old Forums  <||>  Titan Network  <||>  Heroica! (by @Shenanigunner)

 

Link to comment
Share on other sites

Wow Win Zip sucks monkey balls -- like really salty smelly ones.

 

Should have listened and used 7z right off.

 

Estimating 90 min to uncompressed all the files.

** Asus TUF x670E Gaming, Ryzen 7950x, AIO Corsair H150i Elite, TridentZ 192GB DDR5 6400, Sapphire 7900XTX, 48" 4K Samsung 3d & 56" 4k UHD, NVME Sabrent Rocket 2TB, MP600 Pro 8tb, MP700 2 TB. HDD Seagate 12TB **


** Corsair Voyager a1600 **

Link to comment
Share on other sites

Win zip did the first ok. Maybe 20-25 min.

 

It just wasnt willing to do more then 1 at a time with the second wave. I was not going to queue them up 1 by 1 time.

 

Second wave took 77 min.

 

Before bed I'll get it going.

** Asus TUF x670E Gaming, Ryzen 7950x, AIO Corsair H150i Elite, TridentZ 192GB DDR5 6400, Sapphire 7900XTX, 48" 4K Samsung 3d & 56" 4k UHD, NVME Sabrent Rocket 2TB, MP600 Pro 8tb, MP700 2 TB. HDD Seagate 12TB **


** Corsair Voyager a1600 **

Link to comment
Share on other sites

In case any of you end up with all the files to be parced in separate directories too.

 

 

https://itstillworks.com/make-one-folder-out-multiple-folders-26192.html

 

Never deleted 18k+ empty folders before and that is saying something for me.

 

** Asus TUF x670E Gaming, Ryzen 7950x, AIO Corsair H150i Elite, TridentZ 192GB DDR5 6400, Sapphire 7900XTX, 48" 4K Samsung 3d & 56" 4k UHD, NVME Sabrent Rocket 2TB, MP600 Pro 8tb, MP700 2 TB. HDD Seagate 12TB **


** Corsair Voyager a1600 **

Link to comment
Share on other sites

Looking at cpu use, disk use, net bandwidth, I could run over 200 of the app at the same time.

 

Could you make a local config file so I could set my name and data directory so I could just start clicking away?

** Asus TUF x670E Gaming, Ryzen 7950x, AIO Corsair H150i Elite, TridentZ 192GB DDR5 6400, Sapphire 7900XTX, 48" 4K Samsung 3d & 56" 4k UHD, NVME Sabrent Rocket 2TB, MP600 Pro 8tb, MP700 2 TB. HDD Seagate 12TB **


** Corsair Voyager a1600 **

Link to comment
Share on other sites

Got an error about the server being busy 3 times - hope we arnt over loading you :)

 

Another error I can email you - dont have an image host set up to post it -- suppose we could buddy on discord and do it that way.

 

 

----

 

Morning and 19 errors

 

Can these be saves to a log?

** Asus TUF x670E Gaming, Ryzen 7950x, AIO Corsair H150i Elite, TridentZ 192GB DDR5 6400, Sapphire 7900XTX, 48" 4K Samsung 3d & 56" 4k UHD, NVME Sabrent Rocket 2TB, MP600 Pro 8tb, MP700 2 TB. HDD Seagate 12TB **


** Corsair Voyager a1600 **

Link to comment
Share on other sites

I'm just throwing the images onto imgur.

 

Last night's only error:

 

qH9uOqY.png

Excelsior Global Channel - for your server wide chat and forming TFs, Trials, Radios, Farms, whatever you want to do - /chan_join Excelsior today!

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...