Collective operation

A collective operation is a concept in parallel computing in which data is simultaneously sent to or received from many nodes.

Common examples of collective operations are gather (in which data is collected from all nodes), scatter (in which a set of data is broken up into pieces, and a different piece is sent to all nodes), and broadcast (in which the same data is sent to all nodes).