this post was submitted on 26 Aug 2024
30 points (85.7% liked)

Selfhosted

40198 readers
751 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
ruud@lemmy.world
Loki@lemmy.world
CannaVet@lemmy.world
devve@lemmy.world
HybridSarcasm@lemmy.world
HybridSarcasm@lemmy.hybridsarcasm.xyz
30
Data HDD with SSD catch drive (lemm.ee)
submitted 2 months ago* (last edited 2 months ago) by rambos@lemm.ee to c/selfhosted@lemmy.world
73 comments fedilink hide all child comments
 

After upgrading my internet connection I immediatelly noticed that my HDD tops 40 MB/s and bottlnecking download speed in qbittorrent. Is it possible to use SSD drive as a catch drive for 12 TB HDD so it uses SSD speeds when downloading and moves files to HDD later on? If yes, does it make sense? Anyone using anything simmilar? Would 512 GB be enough or could I benefit from 2TB SSD?

HDD is just for jellyfin (movies/shows), not in raid, dont need backup for that drive, I can afford risking data if that matters at all

All suggestions are welcome, Thx in advance

EDIT: I obviously have upset some of you, wasn't my intention, I'm sorry about that. I love to tinker and learn new things, but I could live with much lower speeds tho... Please don't hate me if I couldn't understand your comment or not being clear with my question.

HDD being bottleneck at 40 MB/s was wrong assumption (found out in meantime). I'm still trying to figure out what was the reason for download to be that slow, but I'm interested in learning about the main question anyway. I just thought I'm experiencing the same issue like many people today, having faster internet than storage. Some of you provided solutions I will look into, but need time for that and also have to fix whatever else I'm having issue with.

Keep this community awesome because it is <3

you are viewing a single comment's thread
view the rest of the comments
[–] ShortN0te@lemmy.ml 36 points 2 months ago (4 children)

40MB/s is very very low even for a HDD. I would eventually debug why it's that low.

Yes it's possible. FS like zfs btrfs etc. support that.

[–] catloaf@lemm.ee 13 points 2 months ago (2 children)

It's probably a 5400rpm drive, and/or SMR. Both are going to make it slower.

[–] Appoxo@lemmy.dbzer0.com 4 points 2 months ago

5.4k + smr would explain it at write but not at read.

[–] Markaos@lemmy.one 3 points 2 months ago

In my very limited experience with my 5400rpm SMR WD disk, it's perfectly capable of writing at over 100 MB/s until its cache runs out, then it pretty much dies until it has time to properly write the data, rinse and repeat.

40 MB/s sustained is weird (but maybe it's just a different firmware? I think my disk was able to actually sustain 60 MB/s for a few hours when I limited the write speed, 40 could be a conservative setting that doesn't even slowly fill the cache)

[–] acosmichippo@lemmy.world 9 points 2 months ago* (last edited 2 months ago)

agreed, I think there is something else going on here. test the write speed with another application, I doubt the drive actually maxes out at 40MB/s unless it's severely fragmented or failing.

incidentally what OP wants is how most people set up Unraid servers. SSD cache takes incoming files for write speed, then at a later time the OS moves the files to the spinning disk array.

[–] rambos@lemm.ee 4 points 2 months ago (1 children)

Its the cheapest drive I could find (refurbished seagate from amazon), I thought thats the reason for being slow, but wasnt aware its that low. Im also getting 25-40 MB/s (200-320 Mbps) when copying files from this drive over network. Streaming works great so its not too slow at all. Is there better way of debugging this? What speeds can I expect from good drive or best drive?

Ill research more about BTRFS and ZFS, thx

[–] acosmichippo@lemmy.world 2 points 2 months ago (1 children)

can you copy files to it from another local disk?

[–] rambos@lemm.ee 4 points 2 months ago (1 children)

Yeah, but need to figure out how to see transfer speed using ssh. Sorry noob here :)

[–] not_fond_of_reddit@lemm.ee 2 points 2 months ago (1 children)

If you use scp (cp over ssh) you should see the transfer speed.

[–] rambos@lemm.ee 4 points 2 months ago (2 children)

I have managed to copy with rsync and getting 180 MB/s. I guess my initial assumption was wrong, HDD is obviously not bottleneck here, it can get close to ISP speed. Thank you for pointing this out, Ill do more testing these days. Im kinda shocked because I never knew HDD can be that fast. Gonna reread all the comments as well

[–] not_fond_of_reddit@lemm.ee 2 points 2 months ago (1 children)

The cool thing about rsync is that it goes ”BRRRRRRRRR!” like a warthog… the plane… and it can saturate the receiving drive or array depending on your network and client. And getting 180 with rsync.. on a SATA drive, can’t really hope for more.

And you can run a quick n dirty test is using dd

$> dd if=/dev/zero of=1g-testfile bs=1g count=1

[–] rambos@lemm.ee 2 points 2 months ago (1 children)

Thx. Ive seen dd commands in guides how to test drive speed, but I'm not sure how can I specify what drive I want to test. I see I could change "if" and "of", but don't trust myself enough to use my own modified commands before understanding them better. Will read more about that. Honestly I'm surprised drive speed test is not easier, but its probably just me still being noob xD

[–] not_fond_of_reddit@lemm.ee 2 points 2 months ago (1 children)

Let’s say you want to test a drive that is mounted on /tmp… you just cd into that directory and you can use my example.

You can use

$> df -h or $> mount

to check how your drive is mounted in the OS Most ”default ” installations will have 1-4 partitions and / being partition 3 or 4.

So if you look at the mount command and / is /dev/sdX3 (where X can be a-z depending on how many drives you have connected) and no other mounts are in the output then every directory under / is on that drive… so you can run my example from your home-directory if you fancy that.

[–] rambos@lemm.ee 1 points 2 months ago (1 children)

Thank you a lot for being patient with me :D

dd if=/dev/zero of=1g-testfile bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.62269 s, 232 MB/s

This HDD is obviously working fine and much faster than I thought it can. I guess I have to find bottleneck elswhere

[–] not_fond_of_reddit@lemm.ee 1 points 2 months ago (1 children)

If I can at least help on stranger on the internet… well, then I have helped one stranger on the internet 😂

[–] rambos@lemm.ee 1 points 2 months ago

Hehe you are awesome 😂

[–] ShortN0te@lemmy.ml 2 points 2 months ago

The limitation of HDDs was never sequential Read/Write when it comes to day to day use on a PC.

The huge difference to an SSD is when data is written or read not sequentially, often referred to random I/O.

[–] possiblylinux127@lemmy.zip 1 points 2 months ago

Btrfs doesn't support using a cache drive