Distributed memory¶

Warning

doxygenfunction: Cannot find function “ttor::Comms_Init” in doxygen xml output for project “TaskTorrent” from directory: doxygen_output/xml/

Warning

doxygenfunction: Cannot find function “ttor::Comms_Finalize” in doxygen xml output for project “TaskTorrent” from directory: doxygen_output/xml/

Warning

doxygenfunction: Cannot find function “ttor::Comms_World_Barrier” in doxygen xml output for project “TaskTorrent” from directory: doxygen_output/xml/

Warning

doxygenfunction: Cannot find function “ttor::Comms_World_Rank” in doxygen xml output for project “TaskTorrent” from directory: doxygen_output/xml/

Warning

doxygenfunction: Cannot find function “ttor::Comms_World_Size” in doxygen xml output for project “TaskTorrent” from directory: doxygen_output/xml/

Warning

doxygenfunction: Cannot find function “ttor::make_Communicator_World” in doxygen xml output for project “TaskTorrent” from directory: doxygen_output/xml/

class ttor::Communicator_Base¶

Handles all inter-ranks communications.

Object responsible for communications accross ranks.

All MPI calls will be funelled through this object
UPC++ calls are processed by this object on the receiver

Subclassed by ttor::Communicator_MPI, ttor::Communicator_UPCXX

Public Functions

Communicator_Base(int verb)¶

Creates a communicator.

Parameters

[in] verb: the verbosity level. 0 is quiet, > 0 is more and more chatty.

void set_logger(Logger *logger)¶

Sets the logger.

Not thread safe

Pre

logger should be a pointer to a valid Logger, that should not be destroyed while this is in use.

Parameters

[in] logger: a pointer to the logger. The logger is not owned by this.

virtual void progress() = 0¶

Makes progress on the communications.

Should be called from the main thread (the one that called MPI_Init_Thread or upcxx::init) Not thread safe.

virtual bool is_done() const = 0¶

Checks for local completion.

Return: true if all queues are empty, false otherwise.

virtual llint get_n_msg_processed() const = 0¶

Number of locally processed active messages.

An active message is processed on the receiver when the associated LPC has finished running.

Return: The number of processed active message.

virtual llint get_n_msg_queued() const = 0¶

Number of locally queued active messages.

An active message is queued on the sender after a call to am->send(...).

Return: The number of queued active message.

virtual int comm_rank() const = 0¶

The rank within the communicator.

Return: The MPI/UPC++ rank of the current processor within its communicator/team

virtual int comm_size() const = 0¶

The size of the communicator.

Return: The number of MPI/UPC++ ranks within the communicator/team

virtual ~Communicator_Base()¶: Destroys the communicator.

class ttor::Communicator_MPI : public ttor::Communicator_Base ¶

Handles all inter-ranks communications.

Object responsible for communications accross ranks. All MPI calls will be funneled through that object.

Public Functions

Communicator_MPI(MPI_Comm comm = MPI_COMM_WORLD, int verb_ = 0, size_t break_msg_size_ = Communicator_MPI::max_int_size)¶

Creates an Communicator_MPI.

Pre

verb >= 0.

Parameters

[in] comm: the MPI communicator to use in communications.
[in] verb: the verbose level: 0 = no printing. > 0 = more and more printing.
[in] break_msg_size: the size at which to break large messages into MPI messages. Mainly used for testing.

template<typename ...Ps> ActiveMsg<char, Ps...> *make_active_msg(std::function<void(Ps...)> fun)¶: Creates an active message.

template<typename T, typename ...Ps> ActiveMsg<T, Ps...> *make_large_active_msg(std::function<void(Ps...)> fun, std::function<T*(Ps...)> fun_ptr, std::function<void(Ps...)> fun_complete)¶: Creates a large active message.

template<typename F> details::AM_t<ActiveMsg, decltype(&F::operator())> *make_active_msg(F f)¶: Creates an active message.

template<typename F, typename G, typename H> details::Large_AM_t<ActiveMsg, decltype(&G::operator())> *make_large_active_msg(F f, G g, H h)¶: Creates a large active message.

virtual void progress() override¶

Makes progress on the communications.

Asynchronous (queue rpcs & in-flight lpcs) progress.
Polls in Irecv and Isend request.
Should be called from thread that called MPI_Init_Thread.
Not thread safe.

virtual bool is_done() const override¶

Checks for local completion.

Return: true if all queues are empty, false otherwise.

virtual llint get_n_msg_processed() const override¶

Number of locally processed active messages.

An active message is processed on the receiver when the associated LPC has finished running.

Return: The number of processed active message.

virtual llint get_n_msg_queued() const override¶

Number of locally queued active messages.

An active message is queued on the sender after a call to am->send(...).

Return: The number of queued active message.

virtual int comm_rank() const override¶

The rank within the MPI communicator.

Communicator_MPI

Return: The MPI rank of the current processor

virtual int comm_size() const override¶

The size of the MPI communicator.

Return: The number of MPI ranks within the communicator

template<typename T, typename ...Ps> class ActiveMsg : public ttor::Communicator_MPI::ActiveMsg_Base ¶

Public Functions

ActiveMsg(std::function<void(Ps...)> fun, std::function<T*(Ps...)> ptr_fun, std::function<void(Ps...)> complete_fun, Communicator_MPI *comm, size_t id, )¶: ActiveMsg

class ActiveMsg_Base¶

Subclassed by ttor::Communicator_MPI::ActiveMsg< T, Ps >

Public Functions

size_t get_id() const¶: ActiveMsg_Base

class ttor::Communicator_UPCXX : public ttor::Communicator_Base ¶

Public Functions

Communicator_UPCXX(int verb_ = 0)¶: Communicator_UPCXX

template<typename ...Ps> ActiveMsg<char, Ps...> *make_active_msg(std::function<void(Ps...)> fun)¶: Creates an active message.

template<typename T, typename ...Ps> ActiveMsg<T, Ps...> *make_large_active_msg(std::function<void(Ps...)> fun, std::function<T*(Ps...)> fun_ptr, std::function<void(Ps...)> fun_complete)¶: Creates a large active message.

template<typename F> details::AM_t<ActiveMsg, decltype(&F::operator())> *make_active_msg(F f)¶: Creates an active message.

template<typename F, typename G, typename H> details::Large_AM_t<ActiveMsg, decltype(&G::operator())> *make_large_active_msg(F f, G g, H h)¶: Creates a large active message.

virtual llint get_n_msg_processed() const override¶

Number of locally processed active messages.

An active message is processed on the receiver when the associated LPC has finished running.

Return: The number of processed active message.

virtual llint get_n_msg_queued() const override¶

Number of locally queued active messages.

An active message is queued on the sender after a call to am->send(...).

Return: The number of queued active message.

virtual void progress() override¶

Makes progress on the communications.

Should be called from the main thread (the one that called MPI_Init_Thread or upcxx::init) Not thread safe.

virtual bool is_done() const override¶

Checks for local completion.

Return: true if all queues are empty, false otherwise.

virtual int comm_rank() const override¶

The rank within the world team.

Return: The UPC++ rank of the current processor within the world team

virtual int comm_size() const override¶

The size of the world team.

Return: The size of the UPC++ world team

template<typename T, typename ...Ps> class ActiveMsg : public ttor::Communicator_UPCXX::ActiveMsg_Base ¶

Public Functions

ActiveMsg(Communicator_UPCXX *comm, std::function<void(Ps...)> fun, std::function<T*(Ps...)> fun_ptr, std::function<void(Ps...)> fun_complete)¶: ActiveMsg

class ActiveMsg_Base¶

Subclassed by ttor::Communicator_UPCXX::ActiveMsg< T, Ps >

Public Functions

virtual ~ActiveMsg_Base()¶: ActiveMsg_UPCXX_Base

template<typename Communicator_t> class ttor::Threadpool_dist : public ttor::Threadpool_shared ¶

A threadpool associated to active messages and a communicator.

This class behaves exactly like Threadpool_shared except that join() is overloaded to complete when all threadpools on all ranks have completed and there are no in-flight active messages (i.e. all queued rpcs have been processed).

Public Functions

Threadpool_dist(int n_threads, Communicator_t *comm, int verb = 0, std::string basename = "Wk_", bool start_immediately = true)¶

Creates a Threadpool associated with a communicator.

Pre

n_threads >= 1

Pre

verb >= 0

Pre

comm points to a valid Communicator which should not be destroyed while this is in use.

Parameters

[in] n_threads: the number of threads.
[in] comm: the communicator (either Communicator_MPI or Communicator_UPCXX)
[in] verb: the verbosity level. 0 means not printing; > 0 prints more and more to stdout.
[in] basename: the prefix to be used to identity this in all logging operations.
[in] start_immediately: if true, this starts immediately. Otherwise, the user should call tp.start() before tp.join().

void join()¶

Complete accross all ranks.

Returns when

There are no tasks running or in any queues in all Threadpools associated with the Communicator;
There are no messages in flight

Pre

this has been started (through start() or the start_immediately constructor field).

Post

After tp.join,

Threadpool_dist::is_done() returns true on all ranks;
Communicator::is_done() returns true on all ranks;
All queued active message on a sender have been processed on a receiver.