Just because an operation is non-blocking does not mean that it is somehow automatically more efficient than if it were blocking. Indeed, many of the same best practices that apply to blocking communication also apply to non-blocking communication. One such best practice judiciously pre-posting non-blocking receives. This method potentially helps an MPI implementation reduce the use of temporary buffers.

For example, if a message is received in an MPI process that is unexpected - meaning that the application did not [yet] post a corresponding receive - the MPI implementation may have to allocate a temporary buffer to receive it. If a matching receive is ever posted, the MPI implementation copies the message from the temporary buffer into the destination buffer.

However, if a non-blocking receive is posted before the message is received, once the message arrives, it is expected and can be received directly into the target buffer. No temporary buffer needs to be allocated and no extra memory copy is necessary. Hence, ensuring to pre-posting receives can increase the efficiency of an MPI application.

ROMIO is a popular implementation of many of the MPI-2 I/O function calls from Argonne National Laboratory (e.g., MPI_FILE_OPEN, MPI_FILE_READ, etc.). ROMIO's implementation is layered on top of MPI-1 point-to-point communication; it is specifically designed to be used as an add-on to existing MPI implementations (such as LAM/MPI, LA-MPI, FT-MPI, and MPICH, to name a few). This layering creates problems because ROMIO cannot re-define the underlying type MPI_Request since it has already been defined by the underlying MPI implementation. Moreover, the back-end of MPI_Request is different in every MPI implementation; ROMIO can't extend it in a portable way.

ROMIO's solution was to create a new type: MPIO_Request. All MPI_FILE* functions that are supposed to take an MPI_Request as a parameter instead take an MPIO_Request. This situation means that ROMIO technically does not conform to the MPI-2 standard, but this detail is usually overlooked for the sake of portability and functionality.

There is a notable side effect, however. Since MPI_TEST and MPI_WAIT (and their variants) take MPI_Request arguments, they cannot accept ROMIO MPIO_Requests. Hence, ROMIO implements its own MPIO_TEST and MPIO_WAIT functions. As such, MPI implementations that use ROMIO generally do not support invoking the various TEST and WAIT functions with arrays of point-to-point and I/O requests.