Parallelizer
Abstract base class to help manage parallelism.
Provides the basic API that all parallelizers can be expected
to conform to.
Provides effectively the union of operations supported by
mp.Pool
and MPI
.
There is also the ability to lookup and register ‘named’
parallelizers, since we expect a single program to not
really use more than one.
This falls back gracefully to the serial case.
default_printer: builtin_function_or_method
InMainProcess: InMainProcess
InWorkerProcess: InWorkerProcess
mode_map: dict
__init__(self, logger=None, contract=None, uid=None):
load_registry():
@property
parallelizer_registry(self):
get_default():
For compat.
:returns
:_
lookup(key):
Checks in the registry to see if a given parallelizer is there
otherwise returns a SerialNonParallelizer
.
key
:Any
:returns
:_
register(self, key):
Checks in the registry to see if a given parallelizer is there
otherwise returns a SerialNonParallelizer
.
key
:Any
:returns
:_
@property
active(self):
initialize(self):
Initializes a parallelizer if necessary
:returns
:_
finalize(self, exc_type, exc_val, exc_tb):
Finalizes a parallelizer (if necessary) if necessary
:returns
:_
__enter__(self):
Allows the parallelizer context to be set
using with
:returns
:_
__exit__(self, exc_type, exc_val, exc_tb):
Allows the parallelizer context to be unset
using with
exc_type
:Any
exc_val
:Any
exc_tb
:Any
:returns
:_
@property
on_main(self):
Returns whether or not the executing process is the main process or not
:returns
:_
main_restricted(func):
A decorator to indicate that a function should only be run when on the main process
func
:Any
:returns
:_
worker_restricted(func):
A decorator to indicate that a function should only be run when on a worker process
func
:Any
:returns
:_
send(self, data, loc, **kwargs):
Sends data to the process specified by loc
data
:Any
kwargs
:Any
:returns
:_
receive(self, data, loc, **kwargs):
Receives data from the process specified by loc
data
:Any
kwargs
:Any
:returns
:_
broadcast(self, data, **kwargs):
Sends the same data to all processes
data
:Any
kwargs
:Any
:returns
:_
scatter(self, data, **kwargs):
Performs a scatter of data to the different
available parallelizer processes.
NOTE: unlike in the MPI case, data
does not
need to be evenly divisible by the number of available
processes
data
:Any
kwargs
:Any
:returns
:_
gather(self, data, **kwargs):
Performs a gather of data from the different available parallelizer processes
data
:Any
kwargs
:Any
:returns
:_
map(self, function, data, extra_args=None, extra_kwargs=None, **kwargs):
Performs a parallel map of function over the held data on different processes
function
:Any
data
:Any
kwargs
:Any
:returns
:_
starmap(self, function, data, extra_args=None, extra_kwargs=None, **kwargs):
Performs a parallel map with unpacking of function over the held data on different processes
function
:Any
data
:Any
kwargs
:Any
:returns
:_
apply(self, func, *args, main_kwargs=None, **kwargs):
Runs the callable func
in parallel
func
:Any
args
:Any
kwargs
:Any
:returns
:_
run(self, func, *args, comm=None, main_kwargs=None, **kwargs):
Calls apply
, but makes sure state is handled cleanly
func
:Any
args
:Any
kwargs
:Any
:returns
:_
from_config(mode=None, **kwargs):
@property
nprocs(self):
Returns the number of processes the parallelizer has to work with
:returns
:_
get_nprocs(self):
Returns the number of processes
:returns
:_
@property
id(self):
Returns some form of identifier for the current process
:returns
:_
@property
pid(self):
get_id(self):
Returns the id for the current process
:returns
:_
@property
printer(self):
main_print(self, *args, **kwargs):
Prints from the main process
args
:Any
kwargs
:Any
:returns
:_
worker_print(self, *args, **kwargs):
Prints from a main worker process
args
:Any
kwargs
:Any
:returns
:_
print(self, *args, where='both', **kwargs):
An implementation of print that operates differently on workers than on main processes
args
:Any
kwargs
:Any
:returns
:_
wait(self):
Causes all processes to wait until they’ve met up at this point.
:returns
:_
__repr__(self):
share(self, obj):
Converts obj
into a form that can be cleanly used with shared memory via a SharedObjectManager
obj
:Any
:returns
:_