Distributed memory¶

int ttor::comm_rank(MPI_Comm comm)¶

Return this processor’s rank within comm

Return

the rank of this processor within comm

Parameters

[in] comm: the MPI communicator

int ttor::comm_size(MPI_Comm comm)¶

Return comm’s size

Return

the number of processors in comm

Parameters

[in] comm: the MPI communicator

std::string ttor::processor_name()¶

Return the hostname

Return: the hostname of this processor

class ttor::ActiveMsgBase¶

Base Active Message class.

An active message is two (or five) things:

A payload (header) to be send from the sender to the receiver rank.
A function to be run on the receiver rank, when the header and the (optional) body have arrived
[Optional] A payload (body) to be send from the sender to the receiver rank, without any temporary copy.
[Optional] When using a body, a function to be run on the receiver rank indicating where to store the body
[Optional] When using a body, a function to be run on the sender rank when the send operation has completed and the body can be reused or freed.

The function is serialized accross ranks using its ID. The payload (header) is sent as a buffer of bytes, using an intermediary copy where the payload is serialized. When the send operation returns, the header can be immediately reused. The payload (body) is directly send (without any intermadiary copy). The body can only be reused when the completion function has finished.

Subclassed by ttor::ActiveMsg< int, int >, ttor::ActiveMsg< int, int, int >, ttor::ActiveMsg< Ps >

Public Functions

size_t get_id() const¶

Return the ID of the active message.

Return: The global ID of the active message

void run(char *payload, size_t size) = 0¶

Deserialize the (header) payload and run the associated function.

Pre

payload should be a valid buffer of size bytes.

Parameters

[in] payload: a pointer to the payload
[in] size: the number of bytes in the payload

char *get_user_buffers(char *payload, size_t size) = 0¶

Returns the location of where the body should be stored.

Pre

payload should be a valid buffer of size bytes.

Parameters

[in] payload: a pointer to the payload (header)
[in] size: the number of bytes in the payload (header)

void complete(char *payload, size_t size) = 0¶

Function to run when body send operation has completed.

Pre

payload should be a valid buffer of size bytes.

Parameters

[in] payload: a pointer to the payload (header)
[in] size: the number of bytes in the payload (header)

ActiveMsgBase(size_t id)¶

Creates an active message.

Pre

id should be a unique id for that active message, and should be the same for that active message accross all ranks.

Parameters

id: the global id of that active message.

~ActiveMsgBase()¶: Destroys the active message.

template<typename ...Ps> class ttor::ActiveMsg : public ttor::ActiveMsgBase ¶

Implementation of Active Message for a payload of type Ps....

An active message is a pair of

A function
A payload (header) tied to an Communicator instance. The active message also had an optional payload (body) and a function to indicate where to store the body on the receiver.

Public Functions

template<typename T> ActiveMsg(std::function<void(Ps&...)> fun, std::function<T*(Ps&...)> ptr_fun, std::function<void(Ps&...)> complete_fun, Communicator *comm, size_t id, )¶

Creates an active message.

Pre

comm should be a valid pointer to a Communicator, which should not be destroyed while the active message is in used.

Parameters

[in] fun: the function to be run on the receiver.
[in] ptr_fun: the function to be run on the receiver, giving the location of where the body should be stored.
[in] complete_fun: the function to be run on the sender when the body send operation has completed
[in] comm: the communicator instance to use for communications. The active message does not take ownership of comm.
[in] id: the active message unique ID. User is responsible to never reuse ID’s, and all ranks should use the same ID’s to refer to the same active function

void run(char *payload_raw, size_t size)¶

Deserialize the (header) payload and run the associated function.

Pre

payload should be a valid buffer of size bytes.

Parameters

[in] payload: a pointer to the payload
[in] size: the number of bytes in the payload

char *get_user_buffers(char *payload_raw, size_t size)¶

Returns the location of where the body should be stored.

Pre

payload should be a valid buffer of size bytes.

Parameters

[in] payload: a pointer to the payload (header)
[in] size: the number of bytes in the payload (header)

void complete(char *payload_raw, size_t size)¶

Function to run when body send operation has completed.

Pre

payload should be a valid buffer of size bytes.

Parameters

[in] payload: a pointer to the payload (header)
[in] size: the number of bytes in the payload (header)

void blocking_send(int dest, Ps&... ps)¶

Immediately sends payload to destination.

The function returns when the payload has been sent. This is not thread safe and can only be called by the MPI master thread.

Pre

dest != ttor::comm_rank()

Parameters

[in] dest: the destination rank
[in] ps: the payload

void send(int dest, Ps&... ps)¶

Queue the payload to be send later.

This is thread-safe and can be called by any thread.

Parameters

[in] dest: the destination rank
[in] ps: the payload

template<typename T> void send_large(int dest, view<T> body, Ps&... ps)¶

Queue the payload to be send later, with an accompanying body.

This is thread-safe and can be called by any thread.

Parameters

[in] dest: the destination rank
[in] body: a view to the body
[in] ps: the payload

template<typename T> void blocking_send_large(int dest, view<T> body, Ps&... ps)¶

Immediately send the payload, with an accompanying body.

The function returns when the payload (the header, not the body) has been sent. This is not thread safe and can only be called by the MPI master thread.

Pre

dest != ttor::comm_rank()

Parameters

[in] dest: the destination rank
[in] body: a view to the body
[in] ps: the payload

~ActiveMsg()¶: Destroys the ActiveMsg.

class ttor::Communicator¶

Handles all inter-ranks communications.

Object responsible for communications accross ranks. All MPI calls will be funneled through that object.

Public Functions

Communicator(MPI_Comm comm = MPI_COMM_WORLD, int verb_ = 0, size_t break_msg_size_ = Communicator::max_int_size)¶

Creates an Communicator.

Pre

verb >= 0.

Parameters

[in] comm: the MPI communicator to use in communications.
[in] verb: the verbose level: 0 = no printing. > 0 = more and more printing.
[in] break_msg_size: the size at which to break large messages into MPI messages. Mainly used for testing.

template<typename ...Ps> ActiveMsg<Ps...> *make_active_msg(std::function<void(Ps&...)> fun)¶

Creates an active message tied to function fun.

Return

A pointer to the active message. The active message is stored in this and should not be freed by the user.

Parameters

[in] fun: the active function to be run on the receiver rank.

template<typename T, typename ...Ps> ActiveMsg<Ps...> *make_large_active_msg(std::function<void(Ps&...)> fun, std::function<T*(Ps&...)> fun_ptr, std::function<void(Ps&...)> fun_complete)¶

Creates an active message tied to function fun and body pointer function fun_ptr.

Return

A pointer to the active message. The active message is stored in this and should not be freed by the user.

Parameters

[in] fun: the active function to be run on the receiver rank.
[in] fun_ptr: the active function to be run on the receiver rank to retreive the body buffer location.
[in] fun_complete: function to be run on the sender when the send operation has complete.

template<typename F> ActiveMsg_type<decltype(&F::)>::type *make_active_msg(F fun)¶

Creates an active message tied to function fun.

Return

A pointer to the active message. The active message is stored in this and should not be freed by the user.

Parameters

[in] fun: the active function to be run on the receiver rank.

template<typename F, typename G, typename H> ActiveMsg_type<decltype(&F::)>::type *make_large_active_msg(F fun, G fun_ptr, H fun_complete)¶

Creates an active message tied to function fun and body pointer function fun_ptr.

Return

A pointer to the active message. The active message is stored in this and should not be freed by the user.

Parameters

[in] fun: the active function to be run on the receiver rank.
[in] fun_ptr: the active function to be run on the receiver rank to retreive the body buffer location.
[in] fun_complete: function to be run on the sender when the send operation has complete.

void set_logger(Logger *logger)¶

Set the logger.

Pre

logger should be a pointer to a valid Logger, that should not be destroyed while this is in use.

Parameters

[in] logger: a pointer to the logger. The logger is not owned by this.

void recv_process()¶

Blocking-receive & process a message.

Should be called from thread that called MPI_Init_Thread. Not thread safe.

void progress()¶

Makes progress on the communications.

Asynchronous (queue rpcs & in-flight lpcs) progress. Polls in Irecv and Isend request. Should be called from thread that called MPI_Init_Thread. Not thread safe.

bool is_done()¶

Check for local completion.

Return: true if all queues are empty, false otherwise.

int get_n_msg_processed()¶

Number of locally processed active messages.

An active message is processed on the receiver when the associated LPC has finished running.

Return: The number of processed active message.

int get_n_msg_queued()¶

Number of locally queued active messages.

An active message is queued on the sender after a call to am->send(...).

Return: The number of queued active message.

int comm_rank()¶

The rank within the communicator.

Communicator

Return: The MPI rank of the current processor

int comm_size()¶

The size of the communicator.

Return: The number of MPI ranks within the communicator

class ttor::Threadpool_dist : public ttor::Threadpool_shared ¶

A threadpool associated to active messages and a communicator.

This class behaves exactly like Threadpool_shared except that join() is overloaded to complete when all threadpools on all ranks have completed and there are no in-flight active messages (i.e. all queued rpcs have been processed).

Public Functions

Threadpool_dist(int n_threads, Communicator *comm, int verb = 0, std::string basename = "Wk_", bool start_immediately = true)¶

Creates a Threadpool associated with a communicator.

Pre

n_threads >= 1

Pre

verb >= 0

Pre

comm points to a valid Communicator which should not be destroyed while this is in use.

Parameters

[in] n_threads: the number of threads.
[in] comm: the communicator.
[in] verb: the verbosity level. 0 means not printing; > 0 prints more and more to stdout.
[in] basename: the prefix to be used to identity this in all logging operations.
[in] start_immediately: if true, this starts immediately. Otherwise, the user should call tp.start() before tp.join().

void join()¶

Complete accross all ranks.

Returns when

There are no tasks running or in any queues in all Threadpools associated with the Communicator;
There are no messages in flight

Pre

this has been started (through start() or the start_immediately constructor field).

Post

After tp.join,

Threadpool_dist::is_done() returns true on all ranks;
Communicator::is_done() returns true on all ranks;
All queued active message on a sender have been processed on a receiver.