this post was submitted on 16 Mar 2025
850 points (98.7% liked)

Programmer Humor

21564 readers
2167 users here now

Welcome to Programmer Humor!

This is a place where you can post jokes, memes, humor, etc. related to programming!

For sharing awful code theres also Programming Horror.

Rules

founded 2 years ago
MODERATORS
 
you are viewing a single comment's thread
view the rest of the comments
[–] wise_pancake@lemmy.ca 4 points 13 hours ago (1 children)

60k rows is generally very usable with even wide tables in row formats.

I’ve had pandas work with 1M plus rows with 100 columns in memory just fine.

After 1M rows move on to something better like Dask, polars, spark, or literally any DB.

The first thing I’d do with whatever data they’re running into issues with is rewrite it as partitioned and sorted parquet.

[–] vk6flab@lemmy.radio 3 points 13 hours ago

My go-to tool of late is duckdb, comes with binaries for most platforms, works out of the box, loads any number of database formats and is FAST.