_NOPE_

I've updated the program again, and made the following changes (OP link updated, please re-download!): [*]Snazzy new icon based on the Spelunker badge! [*]Rudimentary Version Check - meaning, I've got a database table that just stores the version number, and before starting each new file, the program will check to make sure that the version number is the same as the number that I have recorded in the database. This way, when bug fixes need to go out, I can force the programs to stop working, but it won't automatically update without you doing it! [*]Deleted all of the old code that was referencing DataTables - this code was used back when I thought all of this data could fit into memory and then be pushed all at once to the database. Silly me! I plan to release the source code (minus the connectionString, for what should be obvious reasons), shortly so that y'all can be more confident that the program isn't doing anything shiesty. Then, I plan on polishing up the code for the front end viewer as well, and adding into the viewer a screen that allows to the the current progress of the project and the contributors, so that you know how far along the project is, and who's been helping. Call it a "Hall of Fame". I intend to rank the listing by number of files processed.

Parser Update: I just realized that by the default behavior, the program was going through all of the files in the directory by filename IN ORDER. So... um... that would mean that everyone would be working on the same file at the same time unless they pointed the parser into a directory that only contained certain files rather than all of them. OOPS! To reduce duplication of efforts, I added a randomizer into the program, so it'll now count the total number of files, and on every loop pick a random number and process that file. So please download and run the latest version. I might add a version check in the next version, if I have to make any more versions! But it'll just be a check and won't force an update. I've also added code so that if a file's already been found to be processed, it's automatically deleted from the computer. If it was deleted because you processed it, it'll still create the skip file so that you can know which files you processed if you want to report them. I also added the specific filename that you're processing into the console message now:

The TAR is a direct HTTP download from their website. The TORRENT uses a torrent program to download the same darn TAR. I suggest the torrent if you have a torrent program so that you can download from multiple sources at once. That being said, REDOWNLOAD THE PARSER if you have already, because I just pushed out a critical update that does the following: You no longer technically "have to" report completing the parsing of a file, I'm now recording that in the database as well, and I'm using a database check to see if a file's already been processed by someone before. Though, if you post here that you've parsed a file, people can still know to delete that file to save the database check. I've also added a "UserName" function that's required, so that I can give credit at the end of the project to all of the parsers, just give whatever screen name you want to be thanked: Tomorrow, if I have time at work, I'll think about the best way to release the front end, if I think it's ready. Heading to bed shortly.

Moving my chat on this project over to this thread now, and you can help if you want to now.

Project Spelunker Viewer This application will let you search, navigate, and view the results of Project Spelunker. Please keep in mind that this program is in BETA, and will be constantly updated and improved to add additional features as they are requested. The data is not going to be the best, but as more and more people contribute to the project and parse more and more files, more data will become available. And as more people request additional searches, we'll create additional indexing tables that will allow those searches to be performed in a timely manner (meaning, won't time out!). Here's the breakdown of the program: [*]Contributors Menu Item - Opens the Contributors window [*]Search Type ComboBox - Let's you choose which search to run [*]Value to Search TextBox - Lets you enter the value to search by [*]Search Button - Performs the search [*]Results Label - Shows the number of results found, when applicable [*]Results Grid - Displays the Results [*]Copy Context Menu - Lets you copy data from one selected cell of data [*]Export Context Menu - Lets you export the data currently shown onscreen to an Excel file [*]View HTML Button - Lets you view the HTML content associated with the current row, where applicable [*]View Wayback Button - Lets you view the related URL in the Internet Archive's Wayback Machine [*]Contributors Section - Displays the list of contributors, sorted by the number of files parsed/processed [*]File Processed Section - Displays information about the files that have already been processed [*]Files To Be Processed Section - Displays information about the files that have not yet been processed Not seeing what you want to see? Well, that's because currently only 0.2% of the forum's data has been Spelunked into the database. Help out by running the Parser whenever you can! It uses VERY few system resources! Download the Viewer here.

Project Spelunker ***UPDATED ON 07.27.2021*** Want to help bring back the old COH Forums in a readable, searchable way? Well, now you can! Just perform the following simple steps: Header over to the Project Spelunker homepage - http://www.cityofplayers.com/project-spelunker/, and download the "CoH WARC Processor" (you can also optionally download the source code if you want to peek in at the working guts of the program and/or want to make sure you can trust what the program is actually doing) Unzip the program to its own folder, wherever you want Run the program Type in your anonymous username (for credit on the eventual Credits page!) and click the big "Process" button: That's it! Just sit back and watch it spin! Here's what the program does, step by step: It picks a random number between 1 and the number of WARC files still left to process, then downloads that file number to your Temp directory It then creates a "CoH_Forums_Output" sub-directory inside of your temp directory (if it doesn't yet exist) It starts at the number 1, checks to see if there's a Temp\CoH_Forums_Output\1\ folder. If there isn't, it creates one. If there is, it increments the number until it finds an unused number, and creates a temp working directory. This allows multiple instances of the program to run without them crashing into each other. It then "extracts" the WARC data, restoring the ORIGINAL files that Archive.org archived from the old COH forums back in 2012, to that working directory. It then ZIPs all of those files up, and sends them back to my server. Finally, the program cleans up after itself, deleting all of the temp files, queries the FTP server to see how many files are left, picks a new random number and starts all over again It will do this until either we have processed all of the files, or if you click the "Stop After This Iteration" button, which as the button suggests, stops processing after the current iteration is complete. If you're curious to see what the program is extracting, you can find all of the output files under the Temp\CoH_Forums_Output\#\ folder, where # is the current running instance. But please, just don't mess with or touch these files, or it might interfere with the process! Any questions, comments, concerns, bugs, or errors, don't hesitate to contact me, either in this thread or via PM. And thanks for Spelunking! Old Outdated Info below... Want to be able to search the old CoH forums? Me too! So did Zep! So, I decided to take the data from the Internet Archive, parse it, and try to provide access to it to everyone! I decided to start "Project Spelunker" in order to coordinate these efforts. What is there to coordinate? Well, as I've discovered, it's going to be a LONG long process to get all of this data parsed and uploaded to a remote server accessible by all. There's over 18 THOUSAND files, and they each take quite some time to parse and upsert into the database. Who knows how long this would take me if I tried to parse it all alone on my own PC. So, I've made a Parsing program that I've released for anyone to download, that'll let YOU help in the parsing process to bring the old CoH Forums back... sort of... at least it'll be searchable... If you don't want to help in the project, and just want to see what we've done so far, check out the second post below to see about the Project Spelunker Viewer. However, if you'd like to help with Project Spelunker, here's what you can do: [*]Support my Patreon! http://www.cityofplayers.com/2019/07/04/please-support-city-of-players/ (not required to do the rest, but we have to meet the goal if we want to have a server!) [*]Download the Internet Archive copy of the forums. There's several options - CityOfPlayers.com hosted Torrent (453 KB), CityOfPlayers.com Direct HTTP Download (95GB), Internet Archive Direct HTTP Download (230 GB), Internet Archive hosted Torrent (1.1 GB). These are in order of recommended download. You only have to download one of these, as EACH of them contains a FULL copy of the archived forums! Note: the entire archived forums will be about 735 GB, so if you have that space, feel free to extract it all! If you don't have that much space, you'll need to fire up the Viewer program and pick and choose which file(s) to extract that have NOT already been processed, one by one to save your space! [*]Extract the Archives, using your favorite archive extractor program (I like 7Zip myself) into one directory. If you use the Internet Archive version, you'll have a extract a couple of times. [*]Download the CoH Forums Parser program here. I'm going to trust the community not to decompile the program, steal from it my SQL server information, and screw with my server. That would just be... unheroic. [*]Run the program, pointing it to the location where you have the archive extracted to (better make sure that you extracted them into one directory, or that'll be a slow process!) [*]Enter the anonymous UserName by which you want to receive credit for your work. [*]As it's running, the program will parse the WARC files, upsert the data into my database, and then when it's done with a file, it'll delete the local copy and notify the database that the file has been processed, thus telling everyone else's parsers to also delete their copy of that file on the next loop. [*]If you run across any errors with the Parser, it will create an "\Errors\" subdirectory underneath the directory you chose. Please share the Errors.log text file that you find there in this thread! That's it! After you've fired up the program, just let it sit and do it's thing, that's all there is to it! When you want to stop it, just close the window and you can restart it at any time. If you want to run multiple instances of the Parser at the same time, just go ahead and start the program multiple times, and point all of those same instances to whatever folders you want to parse. On every loop, the program automatically checks for and deletes already parsed files, so there's no danger of the programs conflicting with each other. I'll work on patches to fix up the parser when it inevitably breaks. It's literally parsing archived HTML text stored off the internet seven years ago... there are GOING to be some problems with the data! The thread title is a reference to the Spelunker mission and badge, because we're literally spelunking into the old archives and trying to pull something back up! Let's get Spelunking!

If it's so terrible, why do you have so many FF alts?

I get that, but it's another layer of damage reduction, functionally. One of the few weaknesses of Force Fields and its Defense based abilities is that it just had ONE layer, that was also what made it balanced, because it is SO GOOD at that one layer. If you add more layers of protection, it becomes OP. That's why I'd rather the other powers be buffed to encourage their use in a more active fashion by a larger portion of the player base.

I'm pretty much a hero side only player, so I can't speak for VEATs. Over Cold? Honestly, knockback/down and repel. I'm not sold on Absorb. It seems to be stepping on the toes of +Resistence sets and might just make FF TOO powerful, by adding another layer of protection. So what should FF do thematically? When I think of Force Fields, I think Sue Storm, as the quintessential bubbler. And her fields were used to defend and to control. She made shields of course, but she also used them for flight, and to push people around, and to hold people in place. She did damage by the force knocking straight into enemies. That's why I want a damage buff on Force Bolt and Repulsion Bomb, and that's why I want the other non-shield powers to be buffed to be more "controllery" . Add the disorient to Force Bolt, make Detention Field more useful to the team by allowing them to attack the target, but make the target useless. Make Force Bubble virtually IMPENETRABLE, but make it smaller so it's not a burden. That's what I want to see.

Maybe that'd be incentive NOT to do DFB? ;D

Out of long-standing appreciation for your contributions to CoH community and play, I'm happy to help, you snotty-faced heap of parrot droppings.

Incarnate Style wasn't my idea, it was Adeon's.

+1 Another possible idea... have some "lowbie" invasion events that are triggered based on lowbie task force/trial completion. Something like: Death From Below Trial - Hydras Spawn in Atlas Park in retaliation Cavern of Transcendence - Igneous invade Atlas Park and Skyway City Positron 1 - Circle of Thorns Demons and Spectres take over Steel Canyon Synapse - Clockwork overrun Skyway City And so on, and so on... just an idea.

Is trying to insert the data REALLY faster than doing a check? Aren't they both essentially just one database call each? I mean, I can try it to see if that speeds things up, but... I'm skeptical.

I love it too... and it misses WAY less than I remember it missing. It's just really noticeable when it does, because the animation is so long. But there's a really good amount of damage in that power.

I'd also stopped and restarted the process, and didn't realize that it was restarting from SCRATCH every time, so I'm now making a "skip" file for every file that I process - if, when I come across a file, I find its "skip" brother/sister, I'm skipping processing the file. A poor man's resume "tool", but it lets me deal with issues with power/internet outages going forward.

FYI, I'm going to try to post a daily update on how the data upsert process is going. Here's what we've got so far, so you can see it'll be a while, but hey, at least I have the files themselves, they aren't going anywhere (just on my PC), so the only thing slowing things right now is my mandatory database checks to ensure that we're NOT adding the same data into the database twice... because for some reason this archive contains a LOT of duplicated data....

Oh yes. +1. I've seen some messages from more modern missions that have different colored text within the same sentence, so this is already in the code somewhere to highlight in some fashion at least.

I know, right? Best. Curse. EVAR.

Yes... [glow=white,2,300]Welcome Home.[/glow]

The Slapper! We meet again!!!

I want to see an Uncle Sam superhero NPC, smashing down either British Red Coats, or maybe Barbary Coasts pirates...

+1. Also... I'd like to see dual wielding salami... for a chef character concept I have...

https://forums.homecomingservers.com/index.php/topic,2356.msg15257.html