Science Memes

10940 readers

1761 users here now

Welcome to c/science_memes @ Mander.xyz!

A place for majestic STEMLORD peacocking, as well as memes about the realities of working in a lab.

Rules

Don't throw mud. Behave like an intellectual and remember the human.
Keep it rooted (on topic).
No spam.
Infographics welcome, get schooled.

This is a science community. We use the Dawkins definition of meme.

Research Committee

[email protected]

Other Mander Communities

Science and Research

Biology and Life Sciences

Physical Sciences

Humanities and Social Sciences

Practical and Applied Sciences

Memes

Miscellaneous

founded 2 years ago

MODERATORS

[email protected]

348

Pandas (mander.xyz)

submitted 3 months ago by [email protected] to c/[email protected]

40 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 68 points 3 months ago (3 children)

You havent seen anything until you need to put a 4.2gb gzipped csv into a pandas dataframe, which works without any issues I should note.

[–] [email protected] 5 points 3 months ago

It's good to see the occult is still alive and well

[–] [email protected] 12 points 3 months ago (1 children)

I really don't think that's a lot either. Nowadays we routinely process terabytes of data.

[–] [email protected] 9 points 3 months ago

Yeah, it was just a simple example. Although using just pandas (without something like dask) for loading terabytes of data at once into a single dataframe may not be the best idea, even with enough memory.

[–] [email protected] 24 points 3 months ago

I raise you thousands of gzipped files (total > 20GB) combined into one dataframe. Frankly, my work laptop did not like it all that much. But most basic operations still worked fine tho