this post was submitted on 16 Mar 2025
831 points (98.8% liked)

Programmer Humor

21564 readers
2252 users here now

Welcome to Programmer Humor!

This is a place where you can post jokes, memes, humor, etc. related to programming!

For sharing awful code theres also Programming Horror.

Rules

founded 2 years ago
MODERATORS
Feyter@programming.dev
anzo@programming.dev
BurningTurtle@programming.dev
pylapp@programming.dev
831
muskrat's data eng expert's hard drive overheats while processing 60k rows (lemmy.dbzer0.com)
submitted 1 day ago by SchwertImStein@lemmy.dbzer0.com to c/programmer_humor@programming.dev
267 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] wise_pancake@lemmy.ca 4 points 12 hours ago (1 children)

60k rows is generally very usable with even wide tables in row formats.

I’ve had pandas work with 1M plus rows with 100 columns in memory just fine.

After 1M rows move on to something better like Dask, polars, spark, or literally any DB.

The first thing I’d do with whatever data they’re running into issues with is rewrite it as partitioned and sorted parquet.

[–] vk6flab@lemmy.radio 3 points 12 hours ago

My go-to tool of late is duckdb, comes with binaries for most platforms, works out of the box, loads any number of database formats and is FAST.