Python Generators & Iterators – Complete Guide with yield

Every time you write a for loop in Python, iterators are quietly doing the work. And when you need to produce a sequence of values lazily — one at a time, without building a giant list in memory — generators are the tool. Together they are the backbone of Python's data-processing power, letting you handle streams and even infinite sequences in constant memory.

This guide explains the iterator protocol from the ground up, builds a custom iterator, then shows how generators and the yield keyword make the same thing dramatically simpler. We cover generator expressions, lazy evaluation, infinite generators, yield from, and real-world memory savings — all with runnable code and output.

Iterables vs Iterators

An iterable is anything you can loop over (a list, string, dict). An iterator is the object that actually produces values one at a time via next(). You get an iterator from an iterable using iter().

nums = [10, 20, 30]      # iterable
it = iter(nums)          # get an iterator from it
print(next(it))
print(next(it))
print(next(it))
# print(next(it))        # StopIteration - no more items

Output:

10
20
30

A for loop is just this pattern automated: it calls iter() once, then next() repeatedly until it catches StopIteration.

The Iterator Protocol: __iter__ and __next__

To make your own iterator, implement two dunder methods: __iter__ (returns the iterator object, usually self) and __next__ (returns the next value or raises StopIteration).

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

for n in Countdown(3):
    print(n)

Output:

3
2
1

Generators: The Easy Way

Writing that whole class is tedious. A generator does the same thing with a normal function and the yield keyword. Each yield pauses the function, hands back a value, and remembers where it left off for the next call.

def countdown(start):
    while start > 0:
        yield start          # pause and return a value
        start -= 1

for n in countdown(3):
    print(n)

Output:

3
2
1

That's the entire iterator above — the __iter__/__next__/StopIteration machinery is generated for you automatically.

How yield Pauses and Resumes

Calling a generator function does not run it — it returns a generator object. The body runs only as you pull values out, pausing at each yield.

def steps():
    print("start")
    yield 1
    print("resumed after 1")
    yield 2
    print("resumed after 2")

g = steps()
print(next(g))
print(next(g))

Output:

start
1
resumed after 1
2

Generator Expressions

Just like a list comprehension, but with parentheses instead of brackets. It produces a generator — values are computed lazily, so nothing is stored up front.

squares_list = [x * x for x in range(5)]   # list: built immediately
squares_gen  = (x * x for x in range(5))   # generator: lazy

print(squares_list)
print(squares_gen)
print(sum(x * x for x in range(5)))        # pass straight into sum()

Output:

[0, 1, 4, 9, 16]
<generator object <genexpr> at 0x7f3c1a2b3d40>
30

Why Generators Save Memory

A list of one million squares stores a million integers at once. A generator stores almost nothing — it computes each value on demand. The size difference is enormous.

import sys

list_version = [x * x for x in range(1_000_000)]
gen_version  = (x * x for x in range(1_000_000))

print("list bytes:", sys.getsizeof(list_version))
print("generator bytes:", sys.getsizeof(gen_version))

Output:

list bytes: 8448728
generator bytes: 200

A few hundred bytes vs eight megabytes — for the same logical sequence. This is why generators are ideal for large files and data streams.

Infinite Generators

Because values are produced lazily, a generator can represent an infinite sequence safely — you just stop pulling when you have enough.

def naturals():
    n = 1
    while True:          # never ends on its own
        yield n
        n += 1

gen = naturals()
print([next(gen) for _ in range(5)])

Output:

[1, 2, 3, 4, 5]

Delegating with yield from

yield from lets one generator delegate to another (or any iterable), flattening nested sources without a manual inner loop.

def chain(*iterables):
    for it in iterables:
        yield from it          # yield every item from each iterable

print(list(chain([1, 2], (3, 4), "ab")))

Output:

[1, 2, 3, 4, 'a', 'b']

Real-World Example: Streaming a Large File

Reading a huge log file with file.read() can exhaust memory. A generator yields one line at a time, so you process a 10 GB file in constant memory. Here we count error lines lazily.

def read_lines(path):
    with open(path) as f:
        for line in f:        # files are already lazy iterators
            yield line.rstrip()

def count_errors(path):
    return sum(1 for line in read_lines(path) if "ERROR" in line)

# print(count_errors("server.log"))   # works even on a 10 GB file

Each line is read, checked, and discarded before the next one is loaded — memory usage stays flat regardless of file size.

Common Mistakes to Avoid

• Reusing an exhausted generator — once consumed, it is empty; create a new one to iterate again.
• Calling len() on a generator — it has no length; convert with list() first (which loads everything).
• Using return with a value mid-generator expecting it to yield — return stops the generator (raising StopIteration).
• Indexing a generator like gen[0] — not supported; use next() or itertools.islice.
• Building a list when you only iterate once — wastes memory; prefer a generator expression.

Summary Table

Conclusion

Iterators define how Python loops; generators are the simplest way to build them. Reach for yield or a generator expression whenever you process data sequentially, stream large inputs, or model endless sequences — you trade a tiny amount of laziness for huge memory savings.

Next time you write [... for ... in ...] and only loop over it once, switch the brackets to parentheses. That one-character change turns an eager list into a lazy generator.