Python: fastest way to stream a large CSV into MongoDB?

Stream the file and write in batches — never materialise the whole thing.

1import csv
2from itertools import islice
3from pymongo import MongoClient
4
5BATCH = 5_000
6
7col = MongoClient(uri).devflow.events
8
9with open("big.csv", newline="") as f:
10    reader = csv.DictReader(f)
11    while batch := list(islice(reader, BATCH)):
12        col.insert_many(batch, ordered=False)

1import csv
2from itertools import islice
3from pymongo import MongoClient
4
5BATCH = 5_000
6
7col = MongoClient(uri).devflow.events
8
9with open("big.csv", newline="") as f:
10    reader = csv.DictReader(f)
11    while batch := list(islice(reader, BATCH)):
12        col.insert_many(batch, ordered=False)

ordered=False lets the server keep going past individual duplicate-key failures rather than aborting the batch. Drop secondary indexes before the load and rebuild them afterwards — maintaining them per-insert is usually the real bottleneck.

Python: fastest way to stream a large CSV into MongoDB?

1 Answer