Question 1

How do I stream-process a 5 MB text file line by line in Python?

Accepted Answer

Use a file object as a lazy iterator: with open('file.txt', encoding='utf-8') as f: for line in f: process(line.rstrip('
')). Python's file iterator reads one line at a time from the OS buffer — memory usage is proportional to one line, not the full file. For parallel processing of independent lines, use concurrent.futures.ThreadPoolExecutor: with open('file.txt') as f: with ThreadPoolExecutor(max_workers=4) as executor: list(executor.map(process_line, f)). At 75,000 lines, single-threaded processing completes in under 500 ms for most per-line operations.

Question 2

What is the fastest way to search for a phrase in a 5 MB text file?

Accepted Answer

Command line: grep -n 'lorem ipsum' file.txt finds all occurrences with line numbers in under 50 ms. Python: [i for i,l in enumerate(open('file.txt')) if 'lorem ipsum' in l] — enumerate-based search takes 50–150 ms. For repeated queries on the same file, build a full-text index: meilisearch or elasticsearch store term offsets for O(1) lookup. For regex-based multi-pattern search, use ripgrep (rg) — it uses SIMD-accelerated string matching and is typically 5–20× faster than grep for complex patterns.

Format	TXT
Extension	.txt
MIME type	text/plain
Size	5 MB
Exact bytes	5,242,880
Filename	TestFile-txt-5mb.txt
Format version	Plain text (UTF-8)
Contents	~75,000 lines of lorem ipsum body text; paragraphs separated by blank lines; '##' headings every 10 paragraphs; UTF-8
Encoding	UTF-8 (no BOM); Unix LF line endings; lorem ipsum vocabulary

Sample TXT 5MB File

File details

What is a 5MB TXT file useful for?

What's inside this file?

Download via curl

Related formats

Other TXT sizes

Frequently asked questions