Simple Python Cluster

This is a lightweight, cross-platform cluster lib for simple “embarrassingly parallel” tasks, built entirely with Python’s standard library (multiprocessing).

Quick Start

Install

cd cluster
pip install .

Master Node:

from cluster import MasterNode
master = MasterNode(HOST='0.0.0.0', PORT=5000, AUTHKEY='secret')
master.start_master_server()

Worker Node:

from cluster import WorkerNode
worker = WorkerNode('master_host', 5000, 'secret', nprocs=4)
worker.join_cluster()

Define Tasks

1. Load environment (functions & dependencies):

master.load_envir("f = lambda x: x * 2", from_file=False)

2. Register target function:

master.register_target_function("f")

3. Load arguments (must be unique):

master.load_args(range(10))

Examples

1. Pi estimation

from random import random
envir = "throw = lambda _: (random()*2-1)**2 + (random()*2-1)**2 < 1"
master.load_envir(envir, from_file=False)
master.register_target_function("throw")
master.load_args(range(int(1e6)))
res = master.execute()
print("Pi ≈", 4.0 * sum(res.values()) / 1e6)

2. Factorization

envir = '''
def factorize(n):
    factors, p = [], 2
    while n > 1:
        if n % p == 0: factors.append(p); n //= p
        else: p += 1 if p == 2 else 2
    return factors
'''
master.load_envir(envir, from_file=False)
master.register_target_function("factorize")
master.load_args([999999999999 + 2*i for i in range(5000)])
res = master.execute()

3. Multiple arguments

master.load_envir("f = lambda x: x[0] + x[1]", from_file=False)
master.register_target_function("f")
master.load_args([(1,2), (3,4)])
print(master.execute())

Shutdown

master.shutdown()