Abstraction layer around MPI, this issues and marshals the lower level communication details. More...

Data Types
interface	usleep

Functions/Subroutines
subroutine, public	initialise_mpi_communication (provided_threading)
	Initialises MPI communication. More...

subroutine, public	lock_mpi ()
	If we are explicitly managing MPI thread safety (SERIALIZED mode) then locks MPI. More...

subroutine, public	unlock_mpi ()
	If we are explicitly managing MPI thread safety (SERIALIZED mode) then unlocks MPI. More...

subroutine, public	pause_for_mpi_interleaving ()
	Pauses for a specific number of ms to allow for MPI interleaving, this is to avoid starvation. More...

subroutine, public	wait_for_mpi_request (request, status)
	Waits for a specific MPI request to complete, either by managing thread safety and interleaving or just a call to MPI if we are in multiple mode. More...

subroutine, public	waitall_for_mpi_requests (requests, count)
	Waits for all MPI requests to complete, either by managing thread safety and interleaving or just a call to MPI if we are in multiple mode. More...

integer function, public	get_number_io_servers (io_comm)
	Retrieves the number of IO servers that are running in total. More...

integer function, public	get_my_io_rank (io_comm)
	Retrieves my IO server rank out of the number of IO servers that are running. More...

subroutine, public	register_command_receive ()
	Registers a request for receiving a command from any MONC process on the command channel. More...

integer function, public	data_receive (mpi_datatype, num_elements, source, dump_data, data_dump_id, description_data)
	Awaits some data on the data channel. This is of the type, size from the source provided and can either be written into a byte buffer or integer buffer depending upon the arguments provided. More...

subroutine, public	cancel_requests ()
	Cancels all outstanding communication requests. More...

subroutine	cancel_request (req)
	Cancels a specific communication request. More...

logical function, public	test_for_command (command, source)
	Tests for a command message based upon the request already registered. More...

logical function, public	test_for_inter_io (inter_io_communications, number_of_inter_io, io_communicator, command, source, data_buffer)
	Tests for inter IO server communication. More...

subroutine, public	free_mpi_type (the_type)
	Frees an MPI type, used in clean up. More...

integer function, public	build_mpi_datatype (data_definition, data_size_info, data_size, field_start_locations, field_end_locations, field_dimensions)
	Builds the MPI type that corresponds to the data which will be received from a specific MONC process. Two factors determine the structure and size of this - the XML configuration which has been parsed and also specific details of array sizes sent by each process as part of its registration process. More...

Variables
integer, parameter	ms_wait_between_tests =100
	Interface to the C usleep Linux call which allows us to sleep for a specific number of MS. More...

integer	command_buffer
	Buffer used to receive the command data into when it arrives on that channel. More...

integer	command_request_handle
	Request handle representing the asynchronous P2P command request. More...

integer	mpi_threading_mode

integer, volatile	mpi_mutex

logical	manage_mpi_thread_safety

Detailed Description

Abstraction layer around MPI, this issues and marshals the lower level communication details.

Function/Subroutine Documentation

◆ build_mpi_datatype()

integer function, public mpi_communication_mod::build_mpi_datatype	(	type(io_configuration_data_definition_type), intent(in)	data_definition,
		type(data_sizing_description_type), dimension(:), intent(in)	data_size_info,
		integer, intent(out)	data_size,
		type(map_type), intent(out)	field_start_locations,
		type(map_type), intent(out)	field_end_locations,
		type(map_type), intent(out), optional	field_dimensions
	)

Builds the MPI type that corresponds to the data which will be received from a specific MONC process. Two factors determine the structure and size of this - the XML configuration which has been parsed and also specific details of array sizes sent by each process as part of its registration process.

Parameters

io_configuration	IO server representation of the configuration which contains data structure layout
array_sizes	Sizes of each data array which has been received from a MONC process when it registeres
data_size	The data size corresponding to this type is returned (i.e. the buffer size in bytes required to hold it)
field_start_locations	For the MONC process the start location for each field, keyed on field name
field_end_locations	For the MONC process the end location for each field, keyed on field name
field_dimensions	Optional map of dimensions, if provided will store the number of dimensions for each field

Returns: The MPI data type representation

Definition at line 291 of file mpicommunication.F90.

     type(io_configuration_data_definition_type), intent(in) :: data_definition
     type(data_sizing_description_type), dimension(:), intent(in) :: data_size_info
     integer, intent(out) :: data_size
     type(map_type), intent(out) :: field_start_locations, field_end_locations
     type(map_type), intent(out), optional :: field_dimensions
  
     integer :: type_extents(5), type_counts, i, j, field_start, data_type, field_array_sizes, &
          temp_size, prev_data_type, old_types(20), offsets(20), block_counts(20), new_type, current_location, ierr, field_ignores
     logical :: field_found
     type(data_sizing_description_type) :: field_size_info
  
     type_extents=populate_mpi_type_extents()
  
     field_start=1
     data_type=0
     type_counts=0
     field_array_sizes=0
     field_ignores=0
     current_location=1
     do i=1,data_definition%number_of_data_fields
       if (data_type == 0) then        
         prev_data_type=data_type        
         data_type=data_definition%fields(i)%data_type
       else
         if (data_type .ne. data_definition%fields(i)%data_type) then
           ! For efficient type packing, combine multiple fields with the same type - therefore when the type changes work the previous one pack
           call append_mpi_datatype(field_start, i-1-field_ignores, field_array_sizes, data_type, &
                type_extents, prev_data_type, type_counts+1, old_types, offsets, block_counts)          
           field_start=i
           field_array_sizes=0
           field_ignores=0
           prev_data_type=data_type                   
           data_type=data_definition%fields(i)%data_type
           type_counts=type_counts+1
         end if
       end if
       call c_put_integer(field_start_locations, data_definition%fields(i)%name, current_location)
       if (data_definition%fields(i)%field_type .eq. array_field_type .or. &
            data_definition%fields(i)%field_type .eq. map_field_type) then
         ! Grab the field info based upon the field name
         field_found=get_data_description_from_name(data_size_info, data_definition%fields(i)%name, field_size_info)
         if (.not. field_found .or. field_size_info%dimensions == 0) then
           ! If no field info, or the dimension is 0 then this MONC process is not sending that field - check it is optional
           if (.not. data_definition%fields(i)%optional) then
             call log_log(log_error, "Non optional field `"//trim(data_definition%fields(i)%name)//&
                  "' omitted from MONC IO server registration")
           end if
           field_ignores=field_ignores+1
         else
           ! If the field is specified then use the size data to assemble the field size and append to current size
           temp_size=1
           do j=1, field_size_info%dimensions
             temp_size=temp_size*field_size_info%dim_sizes(j)
           end do          
           if (data_definition%fields(i)%field_type .eq. map_field_type) then
             field_array_sizes=(field_array_sizes+temp_size*string_length*2)-1
             current_location=current_location+temp_size*string_length*2
           else
             field_array_sizes=(field_array_sizes+temp_size)-1
             current_location=current_location+temp_size*type_extents(data_type)
           end if
         end if
       else
         if (data_definition%fields(i)%optional) then
           if (get_data_description_from_name(data_size_info, data_definition%fields(i)%name)) then
             if (data_type==string_data_type) then
               field_array_sizes=(field_array_sizes+string_length)-1
               current_location=current_location+type_extents(data_type)*string_length
             else
               current_location=current_location+type_extents(data_type)
             end if
           else
             field_ignores=field_ignores+1
           end if
         else
           if (data_type==string_data_type) then
             field_array_sizes=(field_array_sizes+string_length)-1
             current_location=current_location+type_extents(data_type)*string_length
           else
             current_location=current_location+type_extents(data_type)
           end if
         end if
       end if
       call c_put_integer(field_end_locations, data_definition%fields(i)%name, current_location-1)
       if (present(field_dimensions)) then
         call c_put_integer(field_dimensions, data_definition%fields(i)%name, field_size_info%dimensions)
       end if
     end do
     if (field_start .le. i-1) then
       ! If there are outstanding fields to process then we do this here
       call append_mpi_datatype(field_start, i-1, field_array_sizes, data_type, &
                type_extents, prev_data_type, type_counts+1, old_types, offsets, block_counts)
       type_counts=type_counts+1
     end if
     call lock_mpi()
     call mpi_type_struct(type_counts, block_counts, offsets, old_types, new_type, ierr) 
     call mpi_type_commit(new_type, ierr)
     call unlock_mpi()
     call mpi_type_size(new_type, data_size, ierr)
     build_mpi_datatype=new_type

Here is the call graph for this function:

Here is the caller graph for this function:

◆ cancel_request()

subroutine mpi_communication_mod::cancel_request ( integer, intent(in) req )

private

Cancels a specific communication request.

Parameters

req	Handle of the request to cancel

Definition at line 202 of file mpicommunication.F90.

     integer, intent(in) :: req
  
     integer :: ierr
  
     if (req .ne. mpi_request_null) then
       call lock_mpi()
       call mpi_cancel(req, ierr)
       call unlock_mpi()
     end if

Here is the call graph for this function:

Here is the caller graph for this function:

◆ cancel_requests()

subroutine, public mpi_communication_mod::cancel_requests

Cancels all outstanding communication requests.

Definition at line 196 of file mpicommunication.F90.

197 call cancel_request(command_request_handle)

Here is the call graph for this function:

Here is the caller graph for this function:

◆ data_receive()

integer function, public mpi_communication_mod::data_receive	(	integer, intent(in)	mpi_datatype,
		integer, intent(in)	num_elements,
		integer, intent(in)	source,
		character, dimension(:), intent(inout), optional, allocatable	dump_data,
		integer, intent(in), optional	data_dump_id,
		type(data_sizing_description_type), dimension(:), intent(inout), optional	description_data
	)

Awaits some data on the data channel. This is of the type, size from the source provided and can either be written into a byte buffer or integer buffer depending upon the arguments provided.

Parameters

mpi_datatype	The MPI type of the data we are receiving
size	Number of elements to receive
source	The PID of the MONC process to receieve this data from
dump_data	(Optional) byte data buffer, for the data dump
description_data	(Optional) integer data buffer, for data description

Definition at line 165 of file mpicommunication.F90.

     integer, intent(in) :: mpi_datatype, num_elements, source
     integer, intent(in), optional :: data_dump_id
     character, dimension(:), allocatable, intent(inout), optional :: dump_data
     type(data_sizing_description_type), dimension(:), intent(inout), optional :: description_data
     integer :: ierr, request, status(MPI_STATUS_SIZE), recv_count, tag_to_use
  
     if (present(dump_data)) then
       tag_to_use=data_tag
       if (present(data_dump_id)) tag_to_use=tag_to_use+data_dump_id
       call lock_mpi()
       call mpi_irecv(dump_data, num_elements, mpi_datatype, source, tag_to_use, mpi_comm_world, request, ierr)
       call unlock_mpi()
       call wait_for_mpi_request(request, status)
       call lock_mpi()
       call mpi_get_count(status, mpi_datatype, recv_count, ierr)      
       call unlock_mpi()
       data_receive=recv_count
     else if (present(description_data)) then
       call lock_mpi()
       call mpi_irecv(description_data, num_elements, mpi_datatype, source, data_tag, mpi_comm_world, request, ierr)
       call unlock_mpi()
       call wait_for_mpi_request(request, status)
       call lock_mpi()
       call mpi_get_count(status, mpi_datatype, recv_count, ierr)      
       call unlock_mpi()
       data_receive=recv_count
     end if    

Here is the call graph for this function:

Here is the caller graph for this function:

◆ free_mpi_type()

subroutine, public mpi_communication_mod::free_mpi_type ( integer, intent(in) the_type )

Frees an MPI type, used in clean up.

Parameters

the_type The MPI type to free up

Definition at line 273 of file mpicommunication.F90.

     integer, intent(in) :: the_type
  
     integer :: ierr
  
     call mpi_type_free(the_type, ierr)

Here is the caller graph for this function:

◆ get_my_io_rank()

integer function, public mpi_communication_mod::get_my_io_rank ( integer, intent(in) io_comm )

Retrieves my IO server rank out of the number of IO servers that are running.

Parameters

io_comm The IO server communicator

Returns: My IO server rank

Definition at line 139 of file mpicommunication.F90.

     integer, intent(in) :: io_comm
  
     integer :: number, ierr
  
     call mpi_comm_rank(io_comm, number, ierr)
     get_my_io_rank=number

Here is the caller graph for this function:

◆ get_number_io_servers()

integer function, public mpi_communication_mod::get_number_io_servers ( integer, intent(in) io_comm )

Retrieves the number of IO servers that are running in total.

Parameters

io_comm The IO server communicator

Returns: The number of running IO servers

Definition at line 127 of file mpicommunication.F90.

     integer, intent(in) :: io_comm
  
     integer :: number, ierr
  
     call mpi_comm_size(io_comm, number, ierr)
     get_number_io_servers=number

Here is the caller graph for this function:

◆ initialise_mpi_communication()

subroutine, public mpi_communication_mod::initialise_mpi_communication ( integer, intent(in) provided_threading )

Initialises MPI communication.

Parameters

provided_threading The provided threading mode

Definition at line 45 of file mpicommunication.F90.

     integer, intent(in) :: provided_threading
  
     mpi_threading_mode=provided_threading
     if (mpi_threading_mode .ne. mpi_thread_multiple .and. mpi_threading_mode .ne. mpi_thread_serialized) then
       call log_master_log(log_error, "You must run MONC in MPI thread serialized or thread multiple mode for the IO server")
     end if    
     manage_mpi_thread_safety=provided_threading == mpi_thread_serialized
     call check_thread_status(forthread_mutex_init(mpi_mutex, -1))

Here is the call graph for this function:

Here is the caller graph for this function:

◆ lock_mpi()

subroutine, public mpi_communication_mod::lock_mpi

If we are explicitly managing MPI thread safety (SERIALIZED mode) then locks MPI.

Definition at line 57 of file mpicommunication.F90.

58 if (manage_mpi_thread_safety) call check_thread_status(forthread_mutex_lock(mpi_mutex))

Here is the call graph for this function:

Here is the caller graph for this function:

◆ pause_for_mpi_interleaving()

subroutine, public mpi_communication_mod::pause_for_mpi_interleaving

Pauses for a specific number of ms to allow for MPI interleaving, this is to avoid starvation.

Definition at line 67 of file mpicommunication.F90.

68 call usleep(int(ms_wait_between_tests, c_int32_t))

Here is the caller graph for this function:

◆ register_command_receive()

subroutine, public mpi_communication_mod::register_command_receive

Registers a request for receiving a command from any MONC process on the command channel.

Definition at line 149 of file mpicommunication.F90.

     integer :: ierr
  
     call lock_mpi()
     call mpi_irecv(command_buffer, 1, mpi_int, mpi_any_source, command_tag, &
          mpi_comm_world, command_request_handle, ierr)
     call unlock_mpi()

Here is the call graph for this function:

Here is the caller graph for this function:

◆ test_for_command()

logical function, public mpi_communication_mod::test_for_command	(	integer, intent(out)	command,
		integer, intent(out)	source
	)

Tests for a command message based upon the request already registered.

Parameters

command	The command which is received is returned to the caller
source	The PID of the source MONC process is returned to the caller

Returns: Whether a message is received or not

Definition at line 218 of file mpicommunication.F90.

     integer, intent(out) :: command, source
  
     integer :: ierr, status(MPI_STATUS_SIZE), complete
  
     call lock_mpi()
     call mpi_test(command_request_handle, complete, status, ierr)
     call unlock_mpi()
  
     if (complete .eq. 1) then
       command = command_buffer
       source = status(mpi_source)
       call register_command_receive()
       test_for_command=.true.
     else
       test_for_command=.false.
     end if    

Here is the call graph for this function:

Here is the caller graph for this function:

◆ test_for_inter_io()

logical function, public mpi_communication_mod::test_for_inter_io	(	type(io_configuration_inter_communication_description), dimension(:), intent(inout)	inter_io_communications,
		integer, intent(in)	number_of_inter_io,
		integer, intent(in)	io_communicator,
		integer, intent(out)	command,
		integer, intent(out)	source,
		character, dimension(:), intent(inout), allocatable	data_buffer
	)

Tests for inter IO server communication.

Parameters

inter_io_communications	Data structures representing the possible inter IO communication sources
number_of_inter_io	Number of inter IO communication descriptions registered
command	The command which is received is returned to the caller
source	The source of the inter IO communication, which is set to the index of the inter descriptor

Returns: Whether a message is received or not

Definition at line 243 of file mpicommunication.F90.

     integer, intent(in) :: number_of_inter_io, io_communicator
     integer, intent(out) :: command, source
     type(io_configuration_inter_communication_description), dimension(:), intent(inout) :: inter_io_communications
     character, dimension(:), allocatable, intent(inout) :: data_buffer
  
     integer :: i, ierr, status(MPI_STATUS_SIZE), message_size
     logical :: message_pending
  
     call lock_mpi()
     do i=1, number_of_inter_io
       call mpi_iprobe(mpi_any_source, inter_io_communications(i)%message_tag, io_communicator, message_pending, status, ierr)
       if (message_pending) then
         call mpi_get_count(status, mpi_byte, message_size, ierr)
         allocate(data_buffer(message_size))
         call mpi_recv(data_buffer, message_size, mpi_byte, mpi_any_source, inter_io_communications(i)%message_tag, &
              io_communicator, mpi_status_ignore, ierr)
         call unlock_mpi()
         command=inter_io_communication
         source=i
         test_for_inter_io=.true.
         return
       end if
     end do
     call unlock_mpi()
     test_for_inter_io=.false.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ unlock_mpi()

subroutine, public mpi_communication_mod::unlock_mpi

If we are explicitly managing MPI thread safety (SERIALIZED mode) then unlocks MPI.

Definition at line 62 of file mpicommunication.F90.

63 if (manage_mpi_thread_safety) call check_thread_status(forthread_mutex_unlock(mpi_mutex))

Here is the call graph for this function:

Here is the caller graph for this function:

◆ wait_for_mpi_request()

subroutine, public mpi_communication_mod::wait_for_mpi_request	(	integer, intent(inout)	request,
		integer, dimension(mpi_status_size), intent(inout), optional	status
	)

Waits for a specific MPI request to complete, either by managing thread safety and interleaving or just a call to MPI if we are in multiple mode.

Parameters

request The MPI request handle

Definition at line 74 of file mpicommunication.F90.

     integer, intent(inout) :: request
     integer, intent(inout), optional :: status(MPI_STATUS_SIZE)
  
     integer :: ierr, flag
  
     if (manage_mpi_thread_safety) then
       flag=0
       do while (flag .ne. 1)
         call lock_mpi()
         if (present(status)) then
           call mpi_test(request, flag, status, ierr)
         else
           call mpi_test(request, flag, mpi_status_ignore, ierr)
         end if
         call unlock_mpi()      
         if (flag .ne. 1) call pause_for_mpi_interleaving()
       end do
     else
       if (present(status)) then
         call mpi_wait(request, status, ierr)
       else
         call mpi_wait(request, mpi_status_ignore, ierr)
       end if
     end if

Here is the call graph for this function:

Here is the caller graph for this function:

◆ waitall_for_mpi_requests()

subroutine, public mpi_communication_mod::waitall_for_mpi_requests	(	integer, dimension(:), intent(inout)	requests,
		integer, intent(in)	count
	)

Waits for all MPI requests to complete, either by managing thread safety and interleaving or just a call to MPI if we are in multiple mode.

Parameters

requests	The MPI request handles to wait for
count	The number of request handles to wait for

Definition at line 105 of file mpicommunication.F90.

     integer, dimension(:), intent(inout) :: requests
     integer, intent(in) :: count
  
     integer :: ierr, flag
  
     if (manage_mpi_thread_safety) then
       flag=0
       do while (flag .ne. 1)
         call lock_mpi()
         call mpi_testall(count, requests, flag, mpi_statuses_ignore, ierr)
         call unlock_mpi()      
         if (flag .ne. 1) call pause_for_mpi_interleaving()
       end do
     else
       call mpi_waitall(count, requests, mpi_statuses_ignore, ierr)
     end if

Here is the call graph for this function:

Here is the caller graph for this function:

Variable Documentation

◆ command_buffer

integer mpi_communication_mod::command_buffer

Buffer used to receive the command data into when it arrives on that channel.

Definition at line 32 of file mpicommunication.F90.

   integer :: command_buffer,& !< Buffer used to receive the command data into when it arrives on that channel
        command_request_handle, & !< Request handle representing the asynchronous P2P command request
        mpi_threading_mode

◆ command_request_handle

integer mpi_communication_mod::command_request_handle

private

Request handle representing the asynchronous P2P command request.

Definition at line 32 of file mpicommunication.F90.

◆ manage_mpi_thread_safety

logical mpi_communication_mod::manage_mpi_thread_safety

private

Definition at line 36 of file mpicommunication.F90.

36 logical :: manage_mpi_thread_safety

◆ mpi_mutex

integer, volatile mpi_communication_mod::mpi_mutex

private

Definition at line 35 of file mpicommunication.F90.

35 integer, volatile :: mpi_mutex

◆ mpi_threading_mode

integer mpi_communication_mod::mpi_threading_mode

private

Definition at line 32 of file mpicommunication.F90.

◆ ms_wait_between_tests

integer, parameter mpi_communication_mod::ms_wait_between_tests =100

private

Interface to the C usleep Linux call which allows us to sleep for a specific number of MS.

Definition at line 21 of file mpicommunication.F90.

21 integer, parameter :: MS_WAIT_BETWEEN_TESTS=100

Data Types

Functions/Subroutines

Variables

Detailed Description

Function/Subroutine Documentation

◆ build_mpi_datatype()

◆ cancel_request()

◆ cancel_requests()

◆ data_receive()

◆ free_mpi_type()

◆ get_my_io_rank()

◆ get_number_io_servers()

◆ initialise_mpi_communication()

◆ lock_mpi()

◆ pause_for_mpi_interleaving()

◆ register_command_receive()

◆ test_for_command()

◆ test_for_inter_io()

◆ unlock_mpi()

◆ wait_for_mpi_request()

◆ waitall_for_mpi_requests()

Variable Documentation

◆ command_buffer

◆ command_request_handle

◆ manage_mpi_thread_safety

◆ mpi_mutex

◆ mpi_threading_mode

◆ ms_wait_between_tests