Python Multiprocessing

Why Multiprocessing?

Because Python's Global Interpreter Lock stops multiple threads from executing Python code truly simultaneously, CPU-heavy tasks like large calculations, image processing, or data crunching don't speed up with threading. The multiprocessing module sidesteps the GIL entirely by running separate processes, each with its own interpreter and memory.

The Process Class

You can spawn a new process much like a thread, using the Process class with a target function, then start and join it the same way.

import multiprocessing

def square_numbers(numbers):
    result = [n * n for n in numbers]
    print(result)

if __name__ == "__main__":
    p = multiprocessing.Process(target=square_numbers, args=([1, 2, 3, 4],))
    p.start()
    p.join()
    print("Done")

Pool — Running Many Tasks in Parallel

For running the same function across many inputs, Pool manages a group of worker processes for you and distributes the work automatically using methods like map().

import multiprocessing

def square(n):
    return n * n

if __name__ == "__main__":
    with multiprocessing.Pool(processes=4) as pool:
        results = pool.map(square, [1, 2, 3, 4, 5])
    print(results)

Multiprocessing vs Threading

Choosing between the two comes down to whether your bottleneck is CPU computation or waiting on I/O. Processes have separate memory (safer, but heavier to create and communicate between), while threads share memory (lighter, but limited by the GIL for CPU work).