[Discussion] General Data Storage

[WanderingAries]

So this came to light as a thing to revisit because started working on expanding / adjusting how I use my available cloud storage for shared data access, but the more I look into What I was doing, the less I think I want to rely on the cloud for now.

 

The Online Stuff:

Spoiler

As it stands, I have access to 1 Tb of OneDrive for each of the 6 accounts you get with the Family sub. I have mine and share with my parents (one each), leaving 3 more Tb otherwise. Since dad's gone (and I still manage all his remaining online accounts), his account has been used to store the little bit that he created. Mom's account probably doesn't even have anything in it besides settings, but I should probably change that to a secondary data backup method.

 

That said, I Did just spend most the week reorganizing my own data as my fixation with ISOs and OSes has pushed me over the 800 Gb mark. Now keep in mind, I probably don't Need half that data as most the OSes are no longer really in use, but (aside from time) it's one of my cheaper things to do. At this point though, my Mac data accounts for about 55-60% of what I have, so I took one of those slots and created a Mac only account and nested a shared folder link into my main account. Add the fact that I did some cleanup / reorganization, and you find out Real quick how slow 10-15 Mbps upload truly is. I think it took most of the week to push the 460+ Gb of data back up to the new account. The upside being I now have a TON of room for my non-Apple stuff.

 

The Offline Stuff:

Spoiler

On the local side of things, I'm starting to reconsider how I'm using my drives as well in the Main PC. Especially since I've maxed out my SATA buses (including the built-in dock for the tower). See below for the current arrangement:

Now keep in mind that I've upgraded the primary OS drive to a 500 Gb SSD and moved it to Win10. I still have the Win7 (cut down in size) on the 250 Gb drive that's attached to the external dock (using internal header) and I have my hacked Win11 install sitting on an internal 250 Gb SSD, but I may give that up until I can upgrade the core components to something modern. Everything else is the same. I've got a 250 Gb cache SSD, 4 Tb backup drive (multiple user backups), 2 Tb data drive, and the 12 Tb archive drive.

 

I have a few things in mind right now and they all focus on the data drives. The first idea is to reduce the extra work for archiving data and merge the smaller drive into the data on the large drive since that's basically what I end up doing later (whenever I remember to do it). There's really no reason to keep that much redundancy. Now even if I do that, however, I may want to consider where the OneDrive data will sit as well since that's a large chunk of data. I should probably look into checking out just how much storage is being used for Photos, Music, and Videos. I know it's in the Tb range, but still. I thought about putting That in the cloud, but I'm pretty sure I'd need to comb through the family movies and cleanup any redundant archiving and even look at downsampling the uploaded content in order for it to fit. I really would like to be able to just push the content to be shareable with the relevant family members.

 

The Network Concept:

Spoiler

Thinking ahead when designing things (as always), I made sure to get a router with the ability to manage drives attached to the USB 3.0 ports on the back. Now I've yet to Do it, however, as I know there's setup involved and then there's the whole needing physical space near the router, etc, but the option is there. Beyond that, there are two available Gigabit ports that could be used for a true NAS. That leaves me with a big decision. Do I want to deal with learning NAS/RAID or do I want to just build something with bare data drives?

 

I more than likely will go with what's easiest for me to deal with vs cost (of course), but I also want to just dip my toe into whichever path I'm going as I kind of lack a budget (seeing as how I've already financed a computer, my regular debts, etc). Springing for an external drive enclosure wouldn't be that bad, but a NAS has actual computing inside that adds a lot to cost should I go that route. Whatever happens, I have the existing 12 Tb bare drive, along with whatever else is sitting in my ESD stack available for doing this. Note: Without changing anything right now, my current internal Archive drive only has about 3 Tb in use, but that doesn't include OneDrive nor my Music.

 

Let me know your thoughts..

[Hyperstrike]

You could PROBABLY get away with a it.
But for data durability I don't recommend it.
You're looking at a hard drive enclosure stuck on your router.
It's generally bad engineering-wise.  As you really have no way to MANAGE the setup.  And you're always a single drive away from data loss.

If you're looking to get something you can manage and use with a backup solution (network drives get EXPENSIVE to back up FAST), look at an actual NAS device.

https://www.amazon.com/Synology-bay-DiskStation-DS218-Diskless/dp/B077PJX8TH

 

Of you want to save a couple bucks, get the Play version, which isn't as upgradeable.

https://www.amazon.com/Synology-Disk-Station-DS218play-Diskless/dp/B076G1G2ZT/

 

While the person who's experienced both "I've got my data backed up" and "What's in the box?  WHAT'S IN THE BOX!!!", I shy away from 2 disk setups.  It's still doable.

Synology has a technology called Synology Hybrid RAID.  Helps keep you from losing data and makes your setup expandable at a later date.  Just by trading up drives.
This makes the function similar to the old Drobos.

Also, as I said, the cost and hassle of dealing with mapping network drives (most backup solutions charge you for CLOUD STORAGE, rather than just backup).
With the Synology setup, you can actually mount your drive setup as a local drive through iSCSI.
Meaning it appears like a local drive to the OS.

So, for an online backup like Backblaze (can't recommend it enough), it simply backs it up like it's another drive in your system.

So with SHR, you can start with a single drive, expand it to two later, and as I said, upgrade it down the road.

And the management interface is web based and fairly straightforward.

[Hyperstrike]

And yes, I KNOW this isn't a "cheap" solution.
I've seen WAY too many people try the "cheap" solution and get themselves burned.
BADLY.

Suddenly that data they "didn't really need" is irreplaceable.
And do you know how much hard drive recovery costs?

 

[WanderingAries]

In the many things I watched, one of the big tech reviewers (GN I think they're called) said it seems like the Synology units have power issues after not very long in service. They had multiple units (of different models / owners) fail all within like 6 months of service. Otherwise, they looked nice with the whole hot-swap / rebuild method. The data's not really being accessed much at all, so it'd be more of an archive. Give or take me adding ISOs or Finally getting on that media archive project. It sounds like the recommendation here is to get the 4-bay unit though, even if I'm not yet using that many drives.

 

As for RAID itself, I thought it was best to have these things in pairs +1 for parity? I've read that RAID 5/6 are designs to be avoided and I think I remember RAID 10 (1+0?) being the best for redundancy / recoverability. I stumbled into the software RAID world around the time I came across the (don't laugh) LTT videos on the horror stories Linus has heard about standard raid (parity drive pushes corrupted data to good drive, etc). The RAID F1 vs SHR was interesting to read through and the advantage (when using the boxes) seems there for sure.

[Hyperstrike]

Also

https://youtu.be/l55GfAwa8RI

[WanderingAries]

8 hours ago, Hyperstrike said:

Also

https://youtu.be/l55GfAwa8RI

 

I went that Whole video like "I know this voice..." and then their page demo vid had the list of names. "Oh, that's Wendell!". I'm not sure if he worked with Linus a while back or what, but I remember him being like the IT version of that Tool Time neighbor in that we never saw his face. Might have been working somewhere that really wouldn't want to see him on youtube.

 

NeWay, yeah, LTT talks about ZedFS a lot (hurts my ears everytime it's said >.<), but certainly, it would make more sense for the FS to be the thing to handle stuff since it's already handling the data itself. I know my current board has the option for RAID built into the controllers too, but I don't think each controller can handle more than like 4 drives anyway. In the end, I think it'll probably be the board handling all this (once I can afford a newer build and scavenge from this one) later.