this post was submitted on 02 Aug 2024
348 points (97.5% liked)

Science Memes

10340 readers
1931 users here now

Welcome to c/science_memes @ Mander.xyz!

A place for majestic STEMLORD peacocking, as well as memes about the realities of working in a lab.

Rules

  1. Don't throw mud. Behave like an intellectual and remember the human.
  2. Keep it rooted (on topic).
  3. No spam.
  4. Infographics welcome, get schooled.

Research Committee

Other Mander Communities

Science and Research

Biology and Life Sciences

Physical Sciences

Humanities and Social Sciences

Practical and Applied Sciences

Memes

Miscellaneous

founded 2 years ago
MODERATORS
[email protected]
[email protected]
[email protected]
348
Pandas (mander.xyz)
submitted 1 month ago by [email protected] to c/[email protected]
40 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 68 points 1 month ago (4 children)

You havent seen anything until you need to put a 4.2gb gzipped csv into a pandas dataframe, which works without any issues I should note.

[–] [email protected] 12 points 1 month ago (1 children)

I really don't think that's a lot either. Nowadays we routinely process terabytes of data.

[–] [email protected] 9 points 1 month ago

Yeah, it was just a simple example. Although using just pandas (without something like dask) for loading terabytes of data at once into a single dataframe may not be the best idea, even with enough memory.

load more comments (2 replies)