Working with large datasets in Python often leads to a common problem: you load your data with Pandas, and your program slows to a crawl or crashes entirely. This typically occurs because you are attempting to load everything into memory simultaneously.
Most memory issues stem from how you load and process data. With a handful of practical techniques, you can handle datasets much larger than your available memory.
In this article, you will learn seven techniques for working with large datasets efficiently in Python. We will start simply and build up, so by the end, you will know exactly which approach fits your use case.
🔗 You can find the code on GitHub. If you’d like, you can run this sample data generator Python script to get sample CSV files and use the code snippets to process them.
The most beginner-friendly approach is to process your data in smaller pieces instead of loading everything at once.
Consider a scenario where you have a large sales dataset and you want to find the total revenue. The following code demonstrates this approach:
Instead of loading all 10 million rows at once, we are loading 100,000 rows at a time. We calculate the sum for each chunk and add it to our running total. Your RAM only ever holds 100,000 rows, no matter how big the file is.
When to use this: When you need to perform aggregations (sum, count, average) or filtering operations on large files.
Often, you do not need every column in your dataset. Loading only what you need can reduce memory usage significantly.
Suppose you are analyzing customer data, but you only require age and purchase amount, rather than the numerous other columns:
By specifying usecols, Pandas only loads those three columns into memory. If your original file had 50 columns, you have just cut your memory usage by roughly 94%.
When to use this: When you know exactly which columns you need before loading the data.
By default, Pandas might use more memory than necessary. A column of integers might be stored as 64-bit when 8-bit would work fine.
For instance, if you are loading a dataset with product ratings (1-5 stars) and user IDs:
By converting the rating column from the probable int64 (8 bytes per number) to int8 (1 byte per number), we achieve an 8x memory reduction for that column.
Common conversions include:
When a column contains repeated text values (like country names or product categories), Pandas stores each value separately. The category dtype stores the unique values once and uses efficient codes to reference them.
Suppose you are working with a product inventory file where the category column has only 20 unique values, but they repeat across all rows in the dataset:
This conversion can substantially reduce memory usage for columns with low cardinality (few unique values). The column still functions similarly to standard text data: you can filter, group, and sort as usual.
When to use this: For any text column where values repeat frequently (categories, states, countries, departments, and the like).
Sometimes you know you only need a subset of rows. Instead of loading everything and then filtering, you can filter during the load process.
For example, if you only care about transactions from the year 2024:
We are combining chunking with filtering. Each chunk is filtered before being added to our list, so we never hold the full dataset in memory, only the rows we actually want.
When to use this: When you need only a subset of rows based on some condition.
For datasets that are truly massive, Dask provides a Pandas-like API but handles all the chunking and parallel processing automatically.
Here is how you would calculate the average of a column across a huge dataset:
Dask does not load the entire file into memory. Instead, it creates a plan for how to process the data in chunks and executes that plan when you call .compute(). It can even use multiple CPU cores to speed up computation.
When to use this: When your dataset is too large for Pandas, even with chunking, or when you want parallel processing without writing complex code.
When you are just exploring or testing code, you do not need the full dataset. Load a sample first.
Suppose you are building a machine learning model and want to test your preprocessing pipeline. You can sample your dataset as shown:
The first approach loads the first N rows, which is suitable for rapid exploration. The second approach randomly samples rows throughout the file, which is better for statistical analysis or when the file is sorted in a way that makes the top rows unrepresentative.
When to use this: During development, testing, or exploratory analysis before running your code on the full dataset.
Handling large datasets does not require expert-level skills. Here is a quick summary of techniques we have discussed:
The first step is knowing both your data and your task. Most of the time, a combination of chunking and smart column selection will get you 90% of the way there.
As your needs grow, move to more advanced tools like Dask or consider converting your data to more efficient file formats like Parquet or HDF5.
Now go ahead and start working with those massive datasets. Happy analyzing!
Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she’s working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more. Bala also creates engaging resource overviews and coding tutorials.
No, thanks!