Jump to content
The Philotic Knight

Project Spelunker

Recommended Posts

I think you misunderstand. That's the COMPRESSED size.

 

Ah, you wound me, friend PK. I was thinking - why decompress the whole archive when I could decompress but a part of it and parse that, then rinse and repeat 'til there are no more files? A lot more work for me, yes, but it should get it done. (Unless, of course, my right honorable friend Cipher gets to it first!  ;D)


"We need Widower. He's a drop of sanity in a bowl of chaos - very important." - Cipher

Share this post


Link to post
Share on other sites

Well, you COULD do that... but then you'd have to actually pay attention to the file names, and only uncompress the files that haven't been parsed already by using the Viewer as a cross reference... and those files are named... painfully.... I mean, if you want to go ahead and do that work, I'm all for whatever help I can get... but I'm telling you right now, it'd make my eyes BLEED to compare these file names against each other to find unparsed files:

 

t7Kgamm.png

 

 

 

 

Right now, the Parser program handles all this for you, by deleting all of the "previously parsed" files from your system, and then grabbing a random file that's left, on every loop.

Share this post


Link to post
Share on other sites

Friend PK, there is nothing more pleasurable in the world than eye-bleed-inducing nonsense!


"We need Widower. He's a drop of sanity in a bowl of chaos - very important." - Cipher

Share this post


Link to post
Share on other sites

Good thing I have a 4TB Spinner that is almost empty -- not sure I have room on the NVME.


** Asus Crosshair VI Hero, Ryzen 1800x, TridentZ 64GB DDR4 @ 3000, GTX 1080 ti, 48" 4K Samsung 3d & 56" 4k UHD, Storage:  WD-Blue 4TB, Intel M.2 NVME 500 GB On MOBO, StarTech PCIE 3.0 x4, 1x M.2 NVMe 1TB 2X M.2 SATA III [Crucial MX 500gb], [samsung Evo 850], Optical Asus BluRay Quad Layer 16x **

Share this post


Link to post
Share on other sites

Any reason I could not make a whole bunch of copies of the executable and alias them to the same data folder? Or would they all try and work on the same file at the same time?

 

Or will it all be done by the time I get home from work today? :)


** Asus Crosshair VI Hero, Ryzen 1800x, TridentZ 64GB DDR4 @ 3000, GTX 1080 ti, 48" 4K Samsung 3d & 56" 4k UHD, Storage:  WD-Blue 4TB, Intel M.2 NVME 500 GB On MOBO, StarTech PCIE 3.0 x4, 1x M.2 NVMe 1TB 2X M.2 SATA III [Crucial MX 500gb], [samsung Evo 850], Optical Asus BluRay Quad Layer 16x **

Share this post


Link to post
Share on other sites

Good thing I have a 4GB Spinner that is almost empty -- not sure I have room on the NVME.

 

I do hope you mean 4TB, instead of 4GB. wCyyU1Z.gif

 

Any reason I could not make a whole bunch of copies of the executable and alias them to the same data folder? Or would they all try and work on the same file at the same time?

 

Or will it all be done by the time I get home from work today? :)

 

You don't even have to make copies of the executable. Just run the same executable as many times as you want, and point them to the same directory. The way the program works, on EVERY program loop, it:

 

  • Checks for already parsed files in the database, and deletes them from the local folder
  • Puts the list of all remaining files into an array and grabs a RANDOM file from that array to process

 

After it's done processing the file, it deletes the local file, and repeats the loop again, refreshing its own internal lists every time.

Share this post


Link to post
Share on other sites

A bit late in the game (having a hard time getting back to threads I've read), but what I was Trying to get at was something like having you put up say 1-10Gb chunks for people to munch on that would speed up the initial DL and make it easier for those who appear to be using SSDs. Personally, I'm sitting on a good ~6Tb so long as I don't go back to my AV project (been avoiding) in the meantime. I hadn't started anything because I first read this when it was like half a page long. :p


OG Server: Pinnacle || Current Server: Torchbearer  || Also found on Indomitable & Excelsior (when needed)

Getting Started  ||  Dredd's Guide to Loading City of Heroes  ||  Install and Troubleshooting Guide v2

Mids' Reborn  ||  HeroStats  ||  Vidiot Maps  ||  QoL, Mods, etc  ||  Heroica! (by @Shenanigunner)

Old Forums  ||  Titan Network  ||  The City Representative (3rd party Info site for all servers)

Share this post


Link to post
Share on other sites

A bit late in the game (having a hard time getting back to threads I've read), but what I was Trying to get at was something like having you put up say 1-10Gb chunks for people to munch on that would speed up the initial DL and make it easier for those who appear to be using SSDs. Personally, I'm sitting on a good ~6Tb so long as I don't go back to my AV project (been avoiding) in the meantime. I hadn't started anything because I first read this when it was like half a page long. :p

 

It's a good idea, and I personally would take advantage, but I suspect it'd be difficult to manage - you wouldn't really want two people grabbing the same chunk. That would be... awkward.


"We need Widower. He's a drop of sanity in a bowl of chaos - very important." - Cipher

Share this post


Link to post
Share on other sites

It's a good idea, and I personally would take advantage, but I suspect it'd be difficult to manage - you wouldn't really want two people grabbing the same chunk. That would be... awkward.

 

No one likes awkward chunks...


"The opposite of a fact is falsehood, but the opposite of one profound truth may very well be another profound truth." - Niels Bohr

 

Global Handle: @JusticeBeliever ... Home servers on Live: Guardian ... Playing on: Everlasting

Share this post


Link to post
Share on other sites

It's a good idea, and I personally would take advantage, but I suspect it'd be difficult to manage - you wouldn't really want two people grabbing the same chunk. That would be... awkward.

 

Yes it would require some vetting. Perhaps instead of everyone grabbing, have OP Assign chunks as people ask to help. This would require the download folder be organized in such a way that there's a minimal amount of confusion as to who should be grabbing what.

 

Again, a bit late to the party, so I'd hate to undo some of the work already begun.


OG Server: Pinnacle || Current Server: Torchbearer  || Also found on Indomitable & Excelsior (when needed)

Getting Started  ||  Dredd's Guide to Loading City of Heroes  ||  Install and Troubleshooting Guide v2

Mids' Reborn  ||  HeroStats  ||  Vidiot Maps  ||  QoL, Mods, etc  ||  Heroica! (by @Shenanigunner)

Old Forums  ||  Titan Network  ||  The City Representative (3rd party Info site for all servers)

Share this post


Link to post
Share on other sites

Oh wow, I don't torrent so started up the TAR file for download on my first post (in here) today. I'm at 4% running <2MB/s, 11.3Gb file size right now. O.O


OG Server: Pinnacle || Current Server: Torchbearer  || Also found on Indomitable & Excelsior (when needed)

Getting Started  ||  Dredd's Guide to Loading City of Heroes  ||  Install and Troubleshooting Guide v2

Mids' Reborn  ||  HeroStats  ||  Vidiot Maps  ||  QoL, Mods, etc  ||  Heroica! (by @Shenanigunner)

Old Forums  ||  Titan Network  ||  The City Representative (3rd party Info site for all servers)

Share this post


Link to post
Share on other sites

Yeah, it's gonna take a while. That's why I'm trying to compress the whole archive into as small of a space as possible, and then re-upload it. I'll host it on CityOfPlayers.com, and I'll also host it via torrent, for as long as it takes to get this thing done. ETA, about 16 hours until the files are compressed.

Share this post


Link to post
Share on other sites

Yes, TB not GB

 

Were gonna see how many threads this baby can handle.


** Asus Crosshair VI Hero, Ryzen 1800x, TridentZ 64GB DDR4 @ 3000, GTX 1080 ti, 48" 4K Samsung 3d & 56" 4k UHD, Storage:  WD-Blue 4TB, Intel M.2 NVME 500 GB On MOBO, StarTech PCIE 3.0 x4, 1x M.2 NVMe 1TB 2X M.2 SATA III [Crucial MX 500gb], [samsung Evo 850], Optical Asus BluRay Quad Layer 16x **

Share this post


Link to post
Share on other sites

14% :p


OG Server: Pinnacle || Current Server: Torchbearer  || Also found on Indomitable & Excelsior (when needed)

Getting Started  ||  Dredd's Guide to Loading City of Heroes  ||  Install and Troubleshooting Guide v2

Mids' Reborn  ||  HeroStats  ||  Vidiot Maps  ||  QoL, Mods, etc  ||  Heroica! (by @Shenanigunner)

Old Forums  ||  Titan Network  ||  The City Representative (3rd party Info site for all servers)

Share this post


Link to post
Share on other sites

Wow Win Zip sucks monkey balls -- like really salty smelly ones.

 

Should have listened and used 7z right off.

 

Estimating 90 min to uncompressed all the files.


** Asus Crosshair VI Hero, Ryzen 1800x, TridentZ 64GB DDR4 @ 3000, GTX 1080 ti, 48" 4K Samsung 3d & 56" 4k UHD, Storage:  WD-Blue 4TB, Intel M.2 NVME 500 GB On MOBO, StarTech PCIE 3.0 x4, 1x M.2 NVMe 1TB 2X M.2 SATA III [Crucial MX 500gb], [samsung Evo 850], Optical Asus BluRay Quad Layer 16x **

Share this post


Link to post
Share on other sites

Win zip did the first ok. Maybe 20-25 min.

 

It just wasnt willing to do more then 1 at a time with the second wave. I was not going to queue them up 1 by 1 time.

 

Second wave took 77 min.

 

Before bed I'll get it going.


** Asus Crosshair VI Hero, Ryzen 1800x, TridentZ 64GB DDR4 @ 3000, GTX 1080 ti, 48" 4K Samsung 3d & 56" 4k UHD, Storage:  WD-Blue 4TB, Intel M.2 NVME 500 GB On MOBO, StarTech PCIE 3.0 x4, 1x M.2 NVMe 1TB 2X M.2 SATA III [Crucial MX 500gb], [samsung Evo 850], Optical Asus BluRay Quad Layer 16x **

Share this post


Link to post
Share on other sites

In case any of you end up with all the files to be parced in separate directories too.

 

 

https://itstillworks.com/make-one-folder-out-multiple-folders-26192.html

 

Never deleted 18k+ empty folders before and that is saying something for me.

 


** Asus Crosshair VI Hero, Ryzen 1800x, TridentZ 64GB DDR4 @ 3000, GTX 1080 ti, 48" 4K Samsung 3d & 56" 4k UHD, Storage:  WD-Blue 4TB, Intel M.2 NVME 500 GB On MOBO, StarTech PCIE 3.0 x4, 1x M.2 NVMe 1TB 2X M.2 SATA III [Crucial MX 500gb], [samsung Evo 850], Optical Asus BluRay Quad Layer 16x **

Share this post


Link to post
Share on other sites

Looking at cpu use, disk use, net bandwidth, I could run over 200 of the app at the same time.

 

Could you make a local config file so I could set my name and data directory so I could just start clicking away?


** Asus Crosshair VI Hero, Ryzen 1800x, TridentZ 64GB DDR4 @ 3000, GTX 1080 ti, 48" 4K Samsung 3d & 56" 4k UHD, Storage:  WD-Blue 4TB, Intel M.2 NVME 500 GB On MOBO, StarTech PCIE 3.0 x4, 1x M.2 NVMe 1TB 2X M.2 SATA III [Crucial MX 500gb], [samsung Evo 850], Optical Asus BluRay Quad Layer 16x **

Share this post


Link to post
Share on other sites

Got an error about the server being busy 3 times - hope we arnt over loading you :)

 

Another error I can email you - dont have an image host set up to post it -- suppose we could buddy on discord and do it that way.

 

 

----

 

Morning and 19 errors

 

Can these be saves to a log?


** Asus Crosshair VI Hero, Ryzen 1800x, TridentZ 64GB DDR4 @ 3000, GTX 1080 ti, 48" 4K Samsung 3d & 56" 4k UHD, Storage:  WD-Blue 4TB, Intel M.2 NVME 500 GB On MOBO, StarTech PCIE 3.0 x4, 1x M.2 NVMe 1TB 2X M.2 SATA III [Crucial MX 500gb], [samsung Evo 850], Optical Asus BluRay Quad Layer 16x **

Share this post


Link to post
Share on other sites

Okay, even with Gig Internet, the torrent isn't stable enough to download in less than a day...

Downloading the TAR? A couple hours.


If you want to be godlike, pick anything.

If you want to be GOD, pick a TANK!

Share this post


Link to post
Share on other sites

I'm just throwing the images onto imgur.

 

Last night's only error:

 

qH9uOqY.png


Excelsior Global Channel - for your server wide chat and forming TFs, Trials, Radios, Farms, whatever you want to do - /chan_join Excelsior today!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...