Parallelizer

Abstract base class to help manage parallelism. Provides the basic API that all parallelizers can be expected to conform to. Provides effectively the union of operations supported by mp.Pool and MPI. There is also the ability to lookup and register ‘named’ parallelizers, since we expect a single program to not really use more than one. This falls back gracefully to the serial case.

default_printer: builtin_function_or_method
InMainProcess: InMainProcess
InWorkerProcess: InWorkerProcess
mode_map: dict

 

__init__(self, logger=None, contract=None, uid=None): 

 

load_registry(): 

 

@property
parallelizer_registry(self): 

 

get_default(): 

For compat.

  • :returns: _

 

lookup(key): 

Checks in the registry to see if a given parallelizer is there otherwise returns a SerialNonParallelizer.

  • key: Any
  • :returns: _

 

register(self, key): 

Checks in the registry to see if a given parallelizer is there otherwise returns a SerialNonParallelizer.

  • key: Any
  • :returns: _

 

@property
active(self): 

 

initialize(self): 

Initializes a parallelizer if necessary

  • :returns: _

 

finalize(self, exc_type, exc_val, exc_tb): 

Finalizes a parallelizer (if necessary) if necessary

  • :returns: _

 

__enter__(self): 

Allows the parallelizer context to be set using with

  • :returns: _

 

__exit__(self, exc_type, exc_val, exc_tb): 

Allows the parallelizer context to be unset using with

  • exc_type: Any
  • exc_val: Any
  • exc_tb: Any
  • :returns: _

 

@property
on_main(self): 

Returns whether or not the executing process is the main process or not

  • :returns: _

 

main_restricted(func): 

A decorator to indicate that a function should only be run when on the main process

  • func: Any
  • :returns: _

 

worker_restricted(func): 

A decorator to indicate that a function should only be run when on a worker process

  • func: Any
  • :returns: _

 

send(self, data, loc, **kwargs): 

Sends data to the process specified by loc

  • data: Any
  • kwargs: Any
  • :returns: _

 

receive(self, data, loc, **kwargs): 

Receives data from the process specified by loc

  • data: Any
  • kwargs: Any
  • :returns: _

 

broadcast(self, data, **kwargs): 

Sends the same data to all processes

  • data: Any
  • kwargs: Any
  • :returns: _

 

scatter(self, data, **kwargs): 

Performs a scatter of data to the different available parallelizer processes. NOTE: unlike in the MPI case, data does not need to be evenly divisible by the number of available processes

  • data: Any
  • kwargs: Any
  • :returns: _

 

gather(self, data, **kwargs): 

Performs a gather of data from the different available parallelizer processes

  • data: Any
  • kwargs: Any
  • :returns: _

 

map(self, function, data, extra_args=None, extra_kwargs=None, **kwargs): 

Performs a parallel map of function over the held data on different processes

  • function: Any
  • data: Any
  • kwargs: Any
  • :returns: _

 

starmap(self, function, data, extra_args=None, extra_kwargs=None, **kwargs): 

Performs a parallel map with unpacking of function over the held data on different processes

  • function: Any
  • data: Any
  • kwargs: Any
  • :returns: _

 

apply(self, func, *args, main_kwargs=None, **kwargs): 

Runs the callable func in parallel

  • func: Any
  • args: Any
  • kwargs: Any
  • :returns: _

 

run(self, func, *args, comm=None, main_kwargs=None, **kwargs): 

Calls apply, but makes sure state is handled cleanly

  • func: Any
  • args: Any
  • kwargs: Any
  • :returns: _

 

from_config(mode=None, **kwargs): 

 

@property
nprocs(self): 

Returns the number of processes the parallelizer has to work with

  • :returns: _

 

get_nprocs(self): 

Returns the number of processes

  • :returns: _

 

@property
id(self): 

Returns some form of identifier for the current process

  • :returns: _

 

@property
pid(self): 

 

get_id(self): 

Returns the id for the current process

  • :returns: _

 

@property
printer(self): 

 

main_print(self, *args, **kwargs): 

Prints from the main process

  • args: Any
  • kwargs: Any
  • :returns: _

 

worker_print(self, *args, **kwargs): 

Prints from a main worker process

  • args: Any
  • kwargs: Any
  • :returns: _

 

print(self, *args, where='both', **kwargs): 

An implementation of print that operates differently on workers than on main processes

  • args: Any
  • kwargs: Any
  • :returns: _

 

wait(self): 

Causes all processes to wait until they’ve met up at this point.

  • :returns: _

 

__repr__(self): 

 

share(self, obj): 

Converts obj into a form that can be cleanly used with shared memory via a SharedObjectManager

  • obj: Any
  • :returns: _

Feedback

Examples

Templates

Documentation