One collective call used two generations

The first two-machine run printed nothing.

The program was supposed to be the smallest possible HPX collective:

auto comm = hpx::collectives::create_communicator(
    "/demo/all_reduce",
    hpx::collectives::num_sites_arg(num),
    hpx::collectives::this_site_arg(id));

auto result = hpx::collectives::all_reduce(
    comm, id + 1, std::plus<std::uint32_t>{}).get();

Locality zero contributes 1. Locality one contributes 2. Both should get 3. The same source worked with one locality, where it triumphantly proved that (1 = 1), but the two-locality process sat there until it was interrupted.

The collective was not broken. Only one locality had entered hpx_main. Locality zero waited for locality one, while locality one was running the HPX runtime without running the user’s function that contained the collective. This configuration line was missing:

std::vector<std::string> const cfg = {"hpx.run_hpx_main!=1"};

hpx::init_params init_args;
init_args.cfg = cfg;
return hpx::init(argc, argv, init_args);

After rebuilding, the same launcher produced:

Locality 0: reduced=3 expected=3 [PASS]
Locality 1: reduced=3 expected=3 [PASS]

and four localities produced four copies of 10.

That hang is a better entrance to hierarchical collectives than a clean architecture diagram. A collective is a promise made by a group. Correct arithmetic on one process is not enough. Every participant must execute the same operation, under the same name, with the same membership, at the same logical generation. Later in this investigation a single public call will use two internal generations, an all-to-all will use three logical phases, a callback will throw after moving its state, and an apparently finished runtime will still have an MPI receive alive during shutdown. They are all versions of the same mistake: one participant believes the protocol is at a different point from another.

The implementation discussed here is frozen at HPX commit 5328e6438681ee56c52dca3b3252987a8fe96d16. The endpoint repair in #7412 is kept separate because it was still open on July 28, 2026. All collective source files on that open branch are unchanged from the frozen commit. Historical timings below are identified as historical; the fresh local runs are correctness checks, not performance claims.

The second locality never arrived

Before touching a tree, it helps to make the small demo honest. HPX calls one operating-system process a locality. A locality normally owns a set of worker threads, participates in HPX’s global address space, and can send work or data to other localities. In this article the words site and locality usually refer to the same participant. HPX’s collective API uses site numbers because one process can also simulate several sites locally in a test.

The corrected program is complete enough to build:

#include <hpx/hpx_init.hpp>
#include <hpx/modules/collectives.hpp>

#include <cstdint>
#include <functional>
#include <iostream>
#include <string>
#include <utility>
#include <vector>

constexpr char const* name = "/article/hierarchical_all_reduce";

int hpx_main()
{
    using namespace hpx::collectives;

    std::uint32_t const id = hpx::get_locality_id();
    std::uint32_t const num =
        hpx::get_num_localities(hpx::launch::sync);

    auto communicator = create_hierarchical_communicator(
        name, num_sites_arg(num), this_site_arg(id), arity_arg(2),
        generation_arg(), root_site_arg(0),
        flat_fallback_threshold_arg(0));

    std::uint32_t local_value = id + 1;
    std::uint32_t const result =
        all_reduce(communicator, std::move(local_value),
            std::plus<std::uint32_t>{}, this_site_arg(id),
            generation_arg(1))
            .get();

    std::uint32_t const expected = num * (num + 1) / 2;
    std::cout << "Locality " << id << ": reduced=" << result
              << " expected=" << expected << " ["
              << (result == expected ? "PASS" : "FAIL") << "]\n";

    return hpx::finalize();
}

int main(int argc, char* argv[])
{
    std::vector<std::string> const cfg = {"hpx.run_hpx_main!=1"};
    hpx::init_params init_args;
    init_args.cfg = cfg;
    return hpx::init(argc, argv, init_args);
}

Several pieces of C++ are doing useful work here:

• auto communicator asks the compiler to use the type returned by create_hierarchical_communicator. The exact type is long, but it is not dynamic or typeless. It is still fixed at compile time.
• std::plus<std::uint32_t>{} constructs a tiny function object whose call operator adds two unsigned 32-bit integers.
• std::move(local_value) permits HPX to consume the value. It does not itself move bytes. It changes the expression so a move-aware overload can be used.
• .get() waits for an hpx::future<std::uint32_t> and either returns its integer or rethrows its stored exception.
• wrappers such as num_sites_arg stop two otherwise identical integers from being swapped accidentally.

The startup failure can be drawn without knowing anything about communicator internals:

broken run

process 0                           process 1
---------                           ---------
start HPX                           start HPX
enter hpx_main                      do not enter hpx_main
call all_reduce
arrive as site 0
wait for site 1  <................. site 1 never calls

corrected run: hpx.run_hpx_main!=1

process 0                           process 1
---------                           ---------
enter hpx_main                      enter hpx_main
arrive as site 0  ................. arrive as site 1
                 gate opens
get 3                               get 3

The test for this chapter is intentionally unglamorous: build the exact source, run one locality, then run two and four. One locality catches compile, link, registration, and trivial value errors. Two catches missing participation. Four makes the reduction result less likely to pass by accident. The durable probe and commands are linked at the end.

There were two earlier failures worth retaining. The old demo binary could not start because its runtime path named an HPX build directory that no longer existed. Reconfiguring against the live HPX build repaired that. The first sandboxed distributed run could not bind a loopback socket, which was an environment restriction rather than an HPX bug. Only after separating those failures did the silent protocol hang become visible.

Checkpoint. If locality zero is waiting in all_reduce, adding more threads to locality zero cannot replace a missing call from locality one. Membership is counted in sites, not worker threads.

One line has more contracts than arguments

The successful line still hides five agreements:

auto future = all_reduce(
    communicator,
    std::move(local_value),
    std::plus<std::uint32_t>{},
    this_site_arg(id),
    generation_arg(1));

First, every caller must refer to the same communicator. Second, callers must use distinct this_site values in the half-open range [0, num_sites). Third, they must perform the same collective operation. Fourth, their data types and combining operations must be compatible. Fifth, they must agree that this is generation one. A half-open range includes its left endpoint and excludes its right endpoint, so ten sites are numbered zero through nine.

A generation is a logical use number. Reusable synchronization objects need one because the same site can arrive again after an earlier operation completed. Without a generation, a late packet from yesterday’s operation could be mistaken for an early packet from today’s. The number is not a wall clock and it is not a globally unique ID. It only orders uses of one registered communicator name.

The typed argument wrappers make absence visible. A default-constructed generation_arg() means “automatic generation”, while generation_arg(1) means exactly one. The early contribution history repaired documentation and made is_default() checks consistent in #7136 and #7141. A one-line predicate looks small until one branch interprets the default as “ask the runtime” and another interprets it as a real site or generation.

The final public contract became stricter as the hierarchy grew:

flat communicator
  automatic generation: allowed
  explicit generation : allowed, subject to mode rules

hierarchical two-phase collective
  automatic generation: rejected
  explicit generation : required, positive, consecutive

That distinction is not ceremony. A two-phase operation must derive two internal positions from one user number. If the user number were silently allocated inside several sub-communicators, different branches could allocate in different orders.

Input validation belongs before participation in a gate. Suppose site 10 joins a communicator of ten sites. The valid IDs are zero through nine. If the code merely classifies site 10 into “no group” and follows a non-representative branch, the other nine sites can wait forever. A clear exceptional future is not only friendlier than a hang. It prevents an invalid participant from poisoning a distributed protocol.

The implementation now checks:

• the communicator exists and contains usable sub-communicators;
• this_site < num_sites;
• the supplied site equals the site used to create the hierarchical handle;
• hierarchical rooted operations use site zero as their root;
• the arity is at least two;
• operations that need phase generations receive an explicit positive number.

The validation work accumulated through #7340 and #7359, after the main algorithms already worked. That order is common in systems code. The happy path teaches the data movement. Review and adversarial tests teach which invalid states otherwise become hangs.

Checkpoint. For every collective call, you should be able to name the communicator identity, participant count, caller site, operation, value type, and generation. If any one is ambiguous, the one-line call is not fully specified.

The future waits on a named server

A flat communicator is the smallest place where those agreements meet. The call path looks like this:

application thread
      |
      | all_reduce(comm, value, op, site, generation)
      v
communicator client handle
      |
      | HPX action, local or remote
      v
communicator_server registered in AGAS
      |
      | handle_data(step, finalizer)
      v
and_gate waits for num_sites arrivals
      |
      | last arrival marks generation ready
      v
finalizer computes or reads shared result
      |
      v
hpx::future<T> becomes ready at each caller

AGAS is HPX’s Active Global Address Space. For this investigation its useful property is simple: a component can be registered under a name, and another locality can resolve that name to an HPX ID without knowing which process owns the object. The basename supplied to create_communicator is the rendezvous name. Site zero normally creates the server component and registers it. Other sites find it.

An HPX component is a C++ object that the runtime can address. An action is the remotely invocable wrapper around a component member function or free function. If caller and component are on the same locality, HPX can use a local path. If they are on different localities, HPX serializes the action arguments, sends a parcel, invokes the action at the owner, and sends the result or exception back. A parcel is HPX’s message envelope.

The communicator client is therefore not the server itself. It is a handle containing an HPX ID and some local metadata. Copying the handle does not copy the distributed server. Calling a collective through it eventually invokes the one server associated with its registered name and generation.

The flat server uses an and_gate. Think of it as a reusable barrier with a future attached:

generation g

site 0 --set/get--\
site 1 --set/get---\
site 2 --set/get----> [ and_gate: 3 of 4 ] -- closed
site 3 --set/get---/  [ and_gate: 4 of 4 ] -- ready

generation g + 1 has separate arrival state

The gate does not know how to add numbers or transpose a matrix. It knows how many arrivals are required and when the shared state is ready. The collective supplies two callbacks to generic server machinery:

1. a step that records one participant’s contribution;
2. a finalizer that produces the value seen by one participant after all arrivals.

For a simplified all_reduce, the step stores site which in slot which. The first finalizer invocation reduces all slots into slot zero. A boolean says whether that final value has already been produced, so later result requests reuse it.

// Shape of the real implementation, with error handling omitted.
[&value](auto& data, std::size_t which) {
    data[which] = std::move(value);
},
[op = std::move(op)](
    auto& data, bool& data_available, std::size_t) mutable {
    if (!data_available) {
        data[0] = hpx::reduce(
            data.begin() + 1, data.end(),
            std::move(data[0]), std::move(op));
        data_available = true;
    }
    return data[0];
}

The brackets introduce C++ lambda expressions. A lambda is an unnamed function object. The capture [&value] stores a reference to value; [op = std::move(op)] stores an owned operation object initialized from op. The trailing mutable permits the lambda to change its captured copy. auto& data makes this a generic lambda, meaning the compiler creates the necessary call operator for the actual data container.

Those details matter later. A reference capture is only safe while the referenced object lives. An owned capture can be moved only once unless its state is deliberately preserved. Generic code can accidentally require copies even when the public signature appears move-aware.

Different collectives reuse the same server shape:

• broadcast stores the root value and returns it to each site;
• gather stores one value per site and returns the vector at the root;
• reduce stores values and combines them only at the root;
• all_gather returns the stored vector to every site;
• all_reduce combines once and returns the answer to every site;
• a barrier carries readiness but no user payload.

All-to-all and scans need more specialized finalizers, but the same gate still defines when one generation is complete.

The failure mode of a flat server is concentration. Every participant reaches one component. Its action queue, lock, gate, stored values, and outgoing results all live at one locality. That is not automatically slow. At small site counts, one meeting can be cheaper than coordinating several meetings. It is, however, a structural bottleneck whose load grows with membership.

The flat call path is tested by the existing collectives unit tests and by forcing the hierarchical factory to collapse into one communicator. Merely checking a result is not enough to prove which path ran, so later tests inspect arity and use thresholds that force either flat or hierarchical behavior.

Checkpoint. The communicator client is a handle, AGAS resolves its name, the component owns the shared generation state, the gate counts arrivals, and the future is how completion or failure returns to the caller.

One server became a vector of smaller meetings

The first hierarchical contribution, #7160, did not replace the flat communicator server. It composed several of them.

create_hierarchical_communicator returns a handle containing a vector of flat communicator handles. Each site stores only the communicators it needs along its path through the tree. The leftmost site in a group is that group’s representative. A representative participates in the group’s local communicator and in the next communicator above it.

For eight sites and arity two, the useful picture is:

                         [0, 4]
                       representatives
                         /     \
                    rep 0       rep 4
                    /             \
               [0, 2]             [4, 6]
               /    \              /    \
            rep 0  rep 2        rep 4  rep 6
             /       \            /       \
          [0,1]     [2,3]      [4,5]     [6,7]
           / \       / \        / \       / \
          0   1     2   3      4   5     6   7

representatives are always the leftmost site of their group

The brackets name flat communicator groups. The exact internal naming includes the left and right site bounds so groups at different levels do not resolve to the same component. A site walks upward only if it represents the lower group.

Arity is the maximum fan-out requested for the tree. A binary tree has arity two. Higher arity makes the tree shallower but gives each internal communicator more participants. No arity is universally best:

• low arity means more levels and more sequential phase handoffs;
• high arity means fewer levels but wider contention inside each flat server;
• payload size, transport, process placement, and site count move the crossover.

The original PR reported a historical DGX H100 experiment with one integer, one thread per locality, and 100 iterations. At 32 processes, arity two reported 346 microseconds while the flat path reported 465 microseconds. Arity four regressed. These numbers explain why a tree was worth pursuing and why “larger arity is faster” would have been an unsafe conclusion. They are not fresh measurements and should not be used to predict another cluster.

The tree constructor is recursive. Given a half-open range of sites, it divides the range into at most arity groups. A group’s leftmost site joins the representative communicator. Every site then recurses into the one child group that contains it. Recursion stops when the current range is small enough to become one flat leaf communicator.

In C++, recursion means a function calls itself with a smaller problem. Termination must be visible. Here the range shrinks at every descent, and a leaf condition handles a range whose width is below the arity. An arity below two would not form a useful shrinking tree, so it is rejected.

The design was tested in two ways. Distributed tests launch real localities and exercise parcels, AGAS registration, and process-level participation. Local tests create several logical sites as asynchronous tasks inside one locality, which makes it cheap to sweep many site counts and arities. Neither replaces the other. Local simulation catches topology arithmetic quickly; distributed runs catch startup and networking contracts like the missing hpx_main call.

Checkpoint. A hierarchical communicator is not a new kind of global server. It is a per-site path through a tree of ordinary flat servers. The representative links one level to the next.

Ten sites refused to become a power of three

Perfect trees make diagrams pleasant and implementations suspicious. Ten sites at arity three cannot split into three equal integer groups. The final rule uses quotient and remainder:

count     = min(num_sites, arity)
quotient  = num_sites / count
remainder = num_sites % count

size(group) = quotient + (group < remainder ? 1 : 0)
left(group) = group * quotient + min(group, remainder)

For ten and three:

count     = 3
quotient  = 10 / 3 = 3
remainder = 10 % 3 = 1

group 0: left 0, size 4, sites 0 1 2 3, representative 0
group 1: left 4, size 3, sites 4 5 6,   representative 4
group 2: left 7, size 3, sites 7 8 9,   representative 7

The first remainder groups get one extra site. The left boundary includes all extra sites assigned before the current group, which is why it adds min(group, remainder).

Here is the resulting top level:

                           [0, 4, 7]
                         representative group
                         /        |       \
                        /         |        \
                 sites 0..3   sites 4..6  sites 7..9
                    rep 0        rep 4       rep 7
                   / | | \       / | \       / | \
                  0  1 2  3     4  5  6     7  8  9

Each child group can recurse again if it is still too wide. The real tree stores bounds and local indices, not drawn edges. classify_site walks the groups and returns the one containing a given site. is_top_level_rep compares that site to the group’s left boundary.

Several off-by-one errors are possible:

• treating the right boundary as exclusive in one helper and inclusive in another;
• assigning the remainder to the last groups while computing offsets as if it went to the first;
• using global site IDs in a child communicator that expects local indices;
• failing to reject a site at exactly num_sites;
• assuming the representative’s local index is always its global ID.

PR #7198 expanded coverage to uneven site counts, and later tests exercised arity four at 5, 6, 7, 9, 10, 11, 13, and 15 logical sites. The point is not a magical list. It is to hit both zero and nonzero remainders, leaf and recursive paths, and groups whose sizes differ by one.

The durable mechanics probe asserts the ten-site result exactly. An assertion is a claim checked by the executable. If the groups differ from {0,4}, {4,3}, and {7,3}, the probe stops instead of printing a persuasive but wrong diagram.

Checkpoint. For (N) sites and (A) requested groups, the first (N \bmod A) groups receive one extra site. You should be able to compute any group’s size, left boundary, representative, and a member’s local index.

The sum traveled up before it could travel down

A hierarchy is only useful once an operation follows it. Hierarchical all-reduce is the cleanest example because its two logical phases match its English definition:

1. reduce every site’s value to the root;
2. broadcast the reduced value back to every site.

With ten sites contributing one through ten, the answer is 55. At arity three, an illustrative top-level trace is:

local reductions

group 0: 1 + 2 + 3 + 4 = 10  -> representative 0
group 1: 5 + 6 + 7     = 18  -> representative 4
group 2: 8 + 9 + 10    = 27  -> representative 7

representative reduction

10 + 18 + 27 = 55              -> root site 0

broadcast down

root 0 sends 55 to representatives 0, 4, 7
each representative sends 55 through its subtree
every site returns 55

The actual recursive code handles each level with the site’s vector of communicators. A non-representative stops climbing once it has contributed to its local group. A representative takes the completed group value to the next level. On the downward phase the direction reverses.

Why not gather all ten values at site zero and reduce them there? That is the flat design. The tree reduces partial values early. If the operation combines two integers into one integer, the payload remains one integer at each level. The root receives one partial result per top-level group instead of one value per site.

The combining operation must be associative for arbitrary regrouping:

$$(a \mathbin{\circ} b) \mathbin{\circ} c = a \mathbin{\circ} (b \mathbin{\circ} c)$$

Integer addition is associative until overflow enters the model. Floating point addition is not exactly associative because rounding happens after each operation. A tree can therefore produce a slightly different floating point answer from a flat left-to-right fold while both follow the requested operation. Distributed reduction APIs generally require callers to accept that regrouping or to choose a numerically stronger strategy.

Commutativity, (a \circ b = b \circ a), is a separate property. Addition is both associative and commutative. String concatenation is associative but not commutative. A correct ordered reduction can regroup concatenations while preserving site order. Treating “associative” and “commutative” as synonyms would become disastrous for scans.

The C++ implementation forwards the operation and moves intermediate values where ownership permits. Forwarding means preserving whether an argument arrived as a temporary or as an existing object, usually through a template parameter and std::forward. It avoids forcing one copy policy on every value type. The implementation still has to be careful not to move an object and then invoke a callback again with its emptied state.

Tests from #7160 and #7189 cover repeated explicit generations, several arities, distributed execution, and local logical sites. The fresh two-locality focused test ran 500 generations and returned zero. Its emitted timing was ignored because a unit-test loop on one laptop is not a controlled benchmark.

The main failure mode is phase disagreement. If one site begins broadcast while a peer still believes the communicator is in reduce, both operations may be individually valid yet rendezvous at different gate positions. The next section makes the phase numbering explicit.

Checkpoint. Hierarchical all-reduce is reduce upward plus broadcast downward. The tree changes grouping and communication, not the promised value. One public call still needs a way to keep its two internal phases distinct.

The gathered vector grew on the way back

All-gather has the same broad skeleton:

1. gather contributions to the root in site order;
2. broadcast the full gathered vector to every site.

For values one through ten, every site receives:

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

The difference from all-reduce is payload growth. A reduction can combine two fixed-size values into one fixed-size value. A gather must retain each contribution. A representative for four sites carries four values upward. The root eventually owns ten values. The downward broadcast carries all ten through every branch.

all-reduce upward payload           all-gather upward payload

[1,2,3,4] -> 10                     [1,2,3,4] -> [1,2,3,4]
[5,6,7]   -> 18                     [5,6,7]   -> [5,6,7]
[8,9,10]  -> 27                     [8,9,10]  -> [8,9,10]

root receives 3 integers             root receives 10 integers
root returns 1 integer               root returns 10 integers

This distinction is why a statement like “the hierarchical path is (O(\log N))” is incomplete. The number of sequential levels may be logarithmic, while the bytes carried by a high-level message still grow with the number of sites represented below it. Latency, bytes, temporary allocations, and root work need separate ledgers.

Ordering is part of all-gather’s result. A subtree vector cannot simply be appended in arrival order, because parcels may arrive in a different order on each run. The communicator associates a contribution with a site position. Subtree results are placed according to group bounds, so the final vector is ordered by site ID.

Review of the early hierarchical work asked for broader node measurements and helped make the flat fallback configurable. That is an important boundary between source reasoning and performance proof. The source proves the hierarchical data path and ordered output. It does not prove which path wins for a payload on a particular fabric.

The focused all-gather test exercises real two-locality communication, and the local suite sweeps uneven topologies. A useful adversarial value is the site ID itself. If the result should be [0,1,2,...], any accidental arrival-order concatenation is visible immediately.

Checkpoint. All-reduce can keep a fixed-size partial value. All-gather cannot discard any contribution, so its payload grows upward and the full vector travels downward.

One public generation reserved two slots

Return to the public call:

all_reduce(
    hcomm, value, op,
    this_site_arg(site),
    generation_arg(k));

The caller sees generation (k). The two internal phases use:

$$g_{\text{first}} = 2k - 1$$ $$g_{\text{second}} = 2k$$

The first three calls therefore map like this:

public call       first phase       second phase
-----------       -----------       ------------
k = 1             generation 1      generation 2
k = 2             generation 3      generation 4
k = 3             generation 5      generation 6

The mapping is cheap arithmetic, but the invariant behind it is strong: every hierarchical collective advances every sub-communicator it touches by exactly two internal generations per public call.

Some operations only need one real communication phase on a particular sub-communicator. They still consume the second position by advancing the gate without another round trip. This uniform step is what lets all-reduce, all-gather, all-to-all, scans, barrier, and rooted collectives share one hierarchical communicator without each operation maintaining a different clock.

The gate supports two step modes. A double step performs the two normal phase arrivals. A single-phase operation performs its work and skips the unused position in a controlled way. Review on #7326 replaced a magic integer mode with a named enum because “1” and “2” did not explain whether the number described phases, increments, or callbacks.

Why require explicit generations on two-phase hierarchical calls? Imagine two subtrees allocate an automatic counter when their first local participant arrives. Network scheduling can make those allocations occur in different orders relative to another collective. A user-supplied (k) gives every phase a deterministic pair before any parcel moves.

The mode rules became even more precise in #7369:

fresh communicator
      |
      +-- explicit generations 1,2,3,... --> explicit sequence remains known
      |
      +-- automatic generation -----------> internal position hidden
                                               |
                                               +-- later explicit rejected

Once a caller uses automatic generation, it cannot reliably reconstruct the internal counter’s position and switch to an explicit number. The reverse transition, explicit to automatic, is allowed because the communicator can continue from its own known state. Every participant in one operation must use the same mode.

A tempting alternative is one registered communicator name per collective operation. That avoids cross-operation generation sharing, but multiplies registration, ownership, and API state. The final design permits one hierarchical communicator to be reused as long as every operation consumes the same two-step budget.

Two dedicated tests matter here. cross_collective_hierarchical_test mixes operations on one communicator and verifies that the shared sequence stays aligned. cross_collective_hierarchical_mixed_test deliberately mixes generation modes and expects rejection rather than a hang. Both passed in the fresh two-locality run.

Checkpoint. Public generation (k) owns internal generations (2k-1) and (2k). Even a one-phase internal path must leave the shared communicator at the same next position.

Small jobs flattened the tree

A hierarchy adds component lookups, representative handoffs, intermediate futures, and more branch logic. For a small group, the cure can cost more than the flat server.

The factory accepts flat_fallback_threshold_arg, whose default is 16. The comparison is strict:

num_sites < threshold  -> collapse to one flat communicator
num_sites >= threshold -> build the requested hierarchy
threshold == 0         -> disable fallback

The implementation collapses the tree by setting the effective arity to the site count. The root range then satisfies the leaf condition and becomes one flat communicator. Hierarchical operations recognize arity >= num_sites and dispatch directly to their flat equivalent.

The generation contract must survive that optimization. Suppose generation one uses a tree and consumes internal positions one and two, but generation two falls back and consumes only position three. Generation three would derive five and six while the flat gate expected four. The flat fast path therefore advances by the same two-step budget even when it performs one direct collective.

PR #7193 introduced the configurable fallback after early performance review. The number 16 is a default heuristic, not a theorem. A scalar on a low-latency shared-memory machine and a megabyte vector across racks have different crossovers.

The important tests force both paths:

// Always build a tree.
flat_fallback_threshold_arg(0)

// Force a flat path for this site count.
flat_fallback_threshold_arg(num_sites + 1)

They then run the same operation and compare results. Without forcing the path, a “hierarchical” test at four sites can silently exercise only the default flat fallback and prove none of the tree code. That exact testing trap is more valuable than another happy-path example.

The historical #7160 benchmark is the only timing evidence used in this chapter, and it is labeled with its hardware and method. The fresh hierarchical_flat_fallback_test establishes value and generation behavior, not a crossover threshold.

Checkpoint. Flat fallback changes the communication plan but not the public result or the two-step generation budget. A test must force the path it claims to cover.

Every source needed a private route to every destination

All-to-all is where the tree stops being a pleasant reduce-and-broadcast composition. With (N) sites, every source contributes (N) pieces. Piece (d) is intended for destination (d). Every destination receives one piece from every source.

It helps to write values that encode their route:

value(source, destination) = 100 * source + destination;

For ten sites, source 3 sends:

[300, 301, 302, 303, 304, 305, 306, 307, 308, 309]

Destination 7 must receive column 7:

[7, 107, 207, 307, 407, 507, 607, 707, 807, 907]

The source-to-destination matrix makes the transpose visible:

                 destination
source      0      1      2      3    ...      9
  0         0      1      2      3    ...      9
  1       100    101    102    103    ...    109
  2       200    201    202    203    ...    209
  .         .      .      .      .             .
  9       900    901    902    903    ...    909

destination d receives column d

A flat communicator can store all rows at one server and return column which from its finalizer. A hierarchy should avoid funneling every full row through one global component. The merged path from #7307 has three logical phases:

1. gather each subtree’s source rows at its representative;
2. exchange destination-group slices among top-level representatives;
3. scatter each received destination column down the owning subtree.

For the ten-site, arity-three tree:

phase 1: gather inside subtrees

sites 0..3  -> representative 0 owns rows from sources 0..3
sites 4..6  -> representative 4 owns rows from sources 4..6
sites 7..9  -> representative 7 owns rows from sources 7..9

phase 2: representative exchange

rep 0 sends pieces for destinations 4..6 to rep 4
rep 0 sends pieces for destinations 7..9 to rep 7
rep 4 sends pieces for destinations 0..3 to rep 0
rep 4 sends pieces for destinations 7..9 to rep 7
rep 7 sends pieces for destinations 0..3 to rep 0
rep 7 sends pieces for destinations 4..6 to rep 4

each representative retains its own diagonal destination group locally

phase 3: scatter inside subtrees

rep 0 sends completed columns to destinations 0..3
rep 4 sends completed columns to destinations 4..6
rep 7 sends completed columns to destinations 7..9

The word diagonal refers to a representative’s own source-group to destination-group block. Representative 4 already holds the pieces from sources 4 through 6 that are destined for sites 4 through 6. Sending that block through the representative exchange and back would add traffic without moving ownership, so it remains local.

The unequal group sizes make indexing more than a matrix transpose. Group zero has four sources and four destinations, while groups one and two have three. A representative needs the left boundary and size of every destination group to carve each source row correctly. It also needs source-group boundaries to reassemble columns in global source order.

Here is one concrete route. Value 207 begins at source 2 and belongs to destination 7:

source 2
   |
   | local subtree gather
   v
representative 0
   |
   | representative exchange: group 0 -> group 2
   v
representative 7
   |
   | local subtree scatter
   v
destination 7, position for source 2

Value 807, by contrast, begins and ends within top-level group two. It reaches representative 7 during the gather and then goes down during scatter. It does not cross the representative exchange.

The helper code separates topology from payload transformation. Topology says which group owns a site and where the group begins. Payload helpers slice rows, prepare representative messages, retain diagonal blocks, and rebuild destination columns. This separation was strengthened during review. Comments on #7307 asked for simpler helpers, clearer division arithmetic, stronger assertions, module registration, less obscure naming, move iterators where ownership transferred, and synchronization that remained generic.

A std::vector<T> owns a contiguous dynamic array of T. Iterators act like positions in that array. A move iterator makes dereferencing yield an rvalue, which permits an algorithm to move elements into a destination rather than copy them. That is appropriate only when the source container is no longer needed. Moving too early can leave a diagonal block empty before it is retained locally.

The focused test constructs route-identifying values, checks every destination column, sweeps uneven trees, and exercises repeated generations. It passed with two real localities in the fresh run. Local tests cover larger logical site counts cheaply. Failure cases include the wrong row length, an invalid site, a root other than zero, and inconsistent generations. These should become exceptions before a partial exchange waits forever.

Checkpoint. For any value 100 * source + destination, you should be able to name its gather representative, whether it crosses the representative exchange, its scatter representative, and its final position in the destination column.

Three logical phases did not require three generations

The phrase “three-phase all-to-all” suggests generations (3k-2), (3k-1), and (3k). That design was explored and rejected.

PR #7222 proposed a stride-three communicator design in a 648-line design note. It closed without merge. The reason is subtle: the three logical phases do not all contend for the same communicator.

The subtree gather and representative exchange operate on different communicators in the site’s hierarchy. They can use the same first internal generation because their registered component identities differ. The subtree scatter reuses the subtree communicator after gather and therefore needs the second internal generation.

public generation k

internal 2k - 1:
  subtree communicator       -> gather
  representative communicator -> all-to-all exchange

internal 2k:
  subtree communicator       -> scatter
  representative communicator -> skip to keep clock aligned

This is why counting English phases is not enough. A generation disambiguates repeated use of one synchronization object. Two operations on different objects can share the same number safely. Two consecutive operations on the same object cannot.

The rejected design still had value. It forced an explicit inventory of communicator reuse and made the two-step invariant testable. This is a better review outcome than merging extra protocol state because the proposal was detailed.

All-to-all hardening continued in #7321. Flat fallback, validation, and broader tree tests exposed an important requirement: a single-phase representative exchange must still advance its second generation position. Otherwise the representative communicator falls one step behind the subtree communicators even though the public all-to-all completed.

One could instead give gather, exchange, and scatter distinct basenames. That would avoid reuse but would increase registration and lifetime state, and it would not solve cross-collective sharing generally. The uniform two-step protocol keeps one public rule across the final operation set.

The test is a mixed sequence on one hierarchical communicator:

generation 1: all_reduce
generation 2: all_to_all
generation 3: all_gather
generation 4: inclusive_scan

Each operation touches a different subset of phase shapes, yet every sub-communicator must be ready for public generation five afterward.

Checkpoint. Count generations per reused communicator, not per English phase. Gather and representative exchange can share an internal number because they meet at different registered servers.

Different collectives shared one clock

Before the uniform protocol, operations advanced gates according to their individual implementation shapes. That worked while a communicator was used only for one repeated collective. It failed as an abstraction: the public type claimed to be a hierarchical communicator, not an all-reduce-only communicator.

Consider a small sequence:

all_reduce:   reduce + broadcast       two uses of subtree communicator
broadcast:    one downward pass         one use
all_to_all:   gather + exchange + scatter, mixed communicator reuse
barrier:      arrival + release         two logical sides

If each operation advances only when it communicates, the internal positions after one call differ:

without a uniform budget

subtree after all_reduce  -> +2
subtree after broadcast   -> +1
rep after all_to_all      -> +1
subtree after all_to_all  -> +2

Now let a broadcast follow an all-to-all on the same handle. A representative belongs to both the representative and subtree communicators. Its vector of handles no longer points at a uniform public generation. The next operation can rendezvous with stale state on one level.

The repair in #7326 made “one public call advances two internal positions” the shared law. A single-pass operation uses a skip, not a fake parcel exchange. The public generation remains gap-free: 1, 2, 3, rather than exposing the internal 1, 2, 3, 4, 5, 6 sequence.

The implementation represents the choice with a generation mode enum. An enum gives names to a closed set of integer values. Compared with a boolean such as double_step, it can express intent at call sites and leave room for validation without making true mean different things in different functions.

The communicator server also records whether automatic generation has ever been used. This is a small state machine:

                 explicit call
             +--------------------+
             |                    v
        [fresh/explicit] ----> [explicit sequence]
             |
             | automatic call
             v
        [automatic used]
             |
             | later explicit call
             v
        exceptional future

The rule rejects only the transition whose numeric position the caller cannot know. It does not reject automatic generation forever as a moral judgment.

Failure needs to be uniform here too. If one site uses automatic mode and another supplies generation four, both must learn that the operation is invalid. Allowing one branch to reject locally while another enters the gate replaces a clear contract error with a distributed wait. Validation therefore has to occur consistently before meaningful phase progress.

Review asked that the generation helpers be exported through the module rather than copied into operation headers. That reduces a dangerous kind of duplication: two implementations with the same formula today but independent future edits.

Fresh tests ran both the valid cross-collective sequence and the invalid mixed mode sequence. The latter returning zero means it observed the expected failure, not that the invalid operation succeeded.

Checkpoint. Sharing one hierarchical handle is safe only because every public call has the same internal budget and every participant agrees on generation mode.

A prefix was ordered, not merely reduced

A scan produces a different result at each site. Given values [1,2,3,4,5], an inclusive sum scan returns:

site             0   1   2   3   4
input            1   2   3   4   5
inclusive        1   3   6  10  15

Site (i) includes its own value. An exclusive scan excludes it:

site             0   1   2   3   4
input            1   2   3   4   5
exclusive        0   1   3   6  10

The zero shown here is an explicit additive initial value. APIs must specify site zero carefully because an exclusive scan without an initial value has no input before site zero.

PR #7343 added hierarchical inclusive and exclusive scans. Their shared plan is:

1. gather values to root in site order;
2. construct the full prefix vector at root;
3. scatter prefix result (i) to site (i).

inputs by site
  1   2   3   4   5
   \  |  /     \  |
     gather through hierarchy
              |
              v
       root: [1,2,3,4,5]
              |
       ordered prefix fold
              |
       [1,3,6,10,15]
              |
       scatter through hierarchy
        /   /   |   \    \
       1   3    6   10   15

This is not the only possible parallel scan algorithm. It is a clear composition from existing hierarchical primitives. Its root temporarily owns the whole input and output vectors, so it does not provide the same scaling shape as a distributed upsweep and downsweep scan. Correctness and reuse came first; the implementation does not pretend otherwise.

Order is non-negotiable. Use string concatenation to see why:

inputs:     ["a", "b", "c", "d"]
inclusive:  ["a", "ab", "abc", "abcd"]

Concatenation is associative but not commutative. Reordering values can produce "bacd", which contains the same letters and is still wrong. A scan helper therefore cannot borrow an optimization that assumes a commutative reduction unless the public contract adds that requirement.

The implementation factors common scan mechanics into hierarchical scan helpers. Review on #7343 pushed unrelated changes out of the patch, asked for constexpr helpers where evaluation could happen at compile time, and reduced duplicated inclusive and exclusive branches. constexpr means a function is eligible for compile-time evaluation when its inputs and context allow it. It does not force every call to run during compilation.

The ten-site mechanics probe verifies:

inclusive: 1 3 6 10 15 21 28 36 45 55
exclusive: 0 1 3 6 10 15 21 28 36 45

Focused distributed tests cover both scan types. Root validation rejects a nonzero hierarchical root because the current tree helpers designate site zero as the local root at every level. That is a documented implementation limit, not a mathematical limit of scans.

Checkpoint. A reduction gives one combined answer. A scan gives an ordered prefix answer per site. Associativity permits regrouping; it does not permit reordering.

The tempting fix lived in the wrong layer

One scan test exposed a generation problem. The first proposed repair modified the shared and_gate. That was attractive because the gate is where generation readiness becomes visible. It was also too broad.

The test reused a communicator in a way that let two logically distinct scan cases interfere. Changing the gate could make that test pass by changing generation behavior for every collective, including paths that were already correct. Review redirected the repair in #7364: use separate communicators for the independent test cases.

This is a systems debugging pattern worth naming:

observed failure
      |
      v
shared low-level mechanism looks suspicious
      |
      +-- change mechanism -> broad semantic blast radius
      |
      +-- inspect caller ownership and test isolation
                           |
                           v
                separate communicators

The low-level layer is often the first place where an inconsistency becomes observable, not the layer that created it. A gate sees mismatched generations, but the mismatch can originate in API use, communicator lifetime, or a test that accidentally shares state.

A useful repair test asks two questions:

1. Does the original failing scenario pass?
2. Do unrelated users of the shared mechanism retain exactly the same semantics?

The final diff for #7364 was small: 44 additions and 12 deletions across four test files. Small does not mean trivial. It means the repair boundary moved from a general synchronization primitive to the fixtures that owned the accidental sharing.

The failed approach remains educational because it was plausible. “Never change and_gate” would be cargo cult advice. The real rule is to state the gate invariant that is supposedly wrong, construct a minimal independent counterexample, and measure the blast radius before editing shared machinery.

Checkpoint. The layer that detects a protocol mismatch is not necessarily the layer that caused it. Before changing a shared primitive, reproduce the wrong behavior without the original caller.

Nested vectors became the next bottleneck

The first hierarchical payloads used natural C++ shapes. A collection of rows became std::vector<std::vector<T>>. It is easy to read:

std::vector<std::vector<int>> rows = {
    {10, 11, 12},
    {20},
    {30, 31}
};

It is not one contiguous allocation. The outer vector owns three inner vector objects. Each inner vector can own a separate heap allocation:

outer vector

+----------+----------+----------+
| ptr,len  | ptr,len  | ptr,len  |
+----|-----+----|-----+----|-----+
     |          |          |
     v          v          v
 [10 11 12]    [20]      [30 31]

at least four logical allocations:
one outer buffer plus three row buffers

For hierarchical gather, scatter, and all-to-all, intermediate nested vectors were built, sliced, serialized, moved, and rebuilt. The costs include:

• one allocation per row in the general case;
• pointer-rich metadata with poor spatial locality;
• serialization that walks separate buffers;
• more exception points during partial construction;
• harder overflow accounting because row counts and total elements are computed separately;
• special trouble for std::vector<bool>, whose elements are packed proxy bits rather than ordinary bool& references.

The first flattening proposal, #7370, changed 1,377 lines and deleted 135 across eleven files. It attempted several operations and representations in one patch. It closed without merge.

The replacement split the work:

• #7375 flattened gather and scatter;
• #7377 extended the representation to all-to-all.

This was not merely smaller commits for easier review. Gather/scatter can often use uniform rows, while all-to-all must preserve more general ragged group shapes. Splitting exposed the two data contracts instead of hiding them behind one large conversion.

Flattening does not eliminate all copies automatically. It creates a representation in which contiguous moves, serialization, and checked sizing are possible. Whether a particular path copies or moves still depends on value ownership and the serializer.

A performance claim would require allocation and byte measurements. The source establishes fewer row allocations in the carrier and contiguous storage. The focused payload tests establish round-trip values and shape. This article does not turn those facts into an unmeasured speedup.

Checkpoint. A nested vector is a convenient logical shape, not contiguous storage. Count allocations, metadata, serialization walks, and exception points before calling it free.

Uniform rows needed one number, ragged rows needed offsets

Two carriers replaced the nested shape.

If every row has the same length, store one contiguous data vector and one row size:

uniform rows

logical:
  [10 11 12]
  [20 21 22]
  [30 31 32]

stored:
  data     = [10 11 12 20 21 22 30 31 32]
  row_size = 3

row i begins at i * row_size

If row sizes differ, store contiguous data and prefix offsets:

ragged rows

logical:
  [10 11 12]
  [20]
  [30 31]

stored:
  data    = [10 11 12 20 30 31]
  offsets = [0, 3, 4, 6]

row i is data[offsets[i] .. offsets[i + 1])

There is one more offset than rows. The final offset equals the element count. An empty row is represented by two equal adjacent offsets.

The uniform carrier needs checked multiplication:

$$\text{elements} = \text{rows} \times \text{row size}$$

The ragged carrier needs checked addition while accumulating row lengths. A std::size_t overflow can wrap to a small number, allocate too little memory, and turn later writes into memory corruption. Validation therefore proves the size arithmetic before allocating or copying.

Reconstruction should validate:

• offsets are not empty when a row count is expected;
• the first offset is zero;
• offsets never decrease;
• the final offset equals data.size();
• uniform data size is divisible by the row size, with explicit handling for a zero row size;
• source and destination group dimensions match the collective topology.

The carrier owns its buffers. Constructors establish the relation between data and shape so invalid states are harder to create. Review moved repeated setup from call sites into constructors and members, removed duplicate branches, and used moves when a source payload had completed its role.

std::vector<bool> requires deliberate handling because its iterator does not yield a normal bool&. Generic code that takes an element address or assumes a reference type equals T& can fail to compile or behave unexpectedly. The flattened payload tests include it so the carrier’s generic claim is exercised with the standard library’s most notorious vector specialization.

The durable probe flattens the ragged example and asserts:

data:    10 11 12 20 30 31
offsets: 0 3 4 6

The HPX focused test goes further, round-tripping uniform and ragged collective payloads and checking overflow and shape errors. It passed in the fresh two-locality run.

Checkpoint. Uniform rows need a row width. Ragged rows need (R+1) offsets for (R) rows. The final offset must equal the contiguous data length.

The basename outlived the string it pointed into

A communicator server needs its basename after construction. It uses the name in diagnostics and registration-related state. Earlier code stored a char const*.

A pointer does not own the characters it addresses:

hpx::collectives::communicator make_comm()
{
    std::string name = make_name();
    return create_communicator(name.c_str(), ...);
} // name is destroyed here

name.c_str() points into name’s internal buffer. When name is destroyed, the pointer dangles. Reading it later is undefined behavior. Undefined behavior means the C++ standard imposes no required result. A test can pass, print stale text, or crash depending on unrelated allocation timing.

PR #7378 changed communicator storage to own a std::string:

before

temporary std::string buffer ----> communicator stores char const*
          |
          +-- destroyed
                    |
                    v
              dangling pointer

after

caller text --copied/moved--> communicator std::string
                              owns characters for its lifetime

Ownership answers a specific question: which object’s destructor is responsible for releasing the resource? With std::string, the communicator’s string member owns its character buffer. Moving the server or constructing it from a temporary transfers or creates valid ownership according to string’s rules.

Why did this surface late? Distributed lifetimes stretch ordinary C++ bugs. The server component can outlive the stack frame that created the client handle. Registration and remote lookup introduce time between storing and reading the name. A local synchronous test is less likely to overwrite the freed buffer before diagnostics use it.

The repair touched three files, added 55 lines, and deleted 15. The important test creates communicators from names whose original storage goes away, then uses the server later. Sanitizers can make this class of bug more visible, but the correct ownership model is the actual fix.

This also explains why public functions may still accept char const*. Borrowing for the duration of a call is fine if the callee copies into owned storage before returning. The error was retaining the borrowed pointer as server state.

Checkpoint. A pointer names storage; it does not own that storage. If a component needs text after its constructor returns, the component needs an owning value such as std::string.

A generic reduction could not demand a copy

The all-reduce API accepts a value and returns a value. It should work for a movable type even when copying is disabled:

struct move_only
{
    explicit move_only(int value) : value(value) {}

    move_only(move_only&&) = default;
    move_only& operator=(move_only&&) = default;

    move_only(move_only const&) = delete;
    move_only& operator=(move_only const&) = delete;

    int value;
};

Deleting the copy constructor asks the compiler to reject any hidden copy. That makes a move-only test a compile-time probe of the implementation’s value semantics.

The reduction seed is the first stored value. A call shaped like this copies it:

hpx::reduce(begin, end, data[0], op);

The move-aware shape consumes it:

hpx::reduce(begin, end, std::move(data[0]), std::move(op));

PR #7398 moved the all-reduce seed and added a move-only regression. The change matters beyond performance. Without it, the template advertises a broader generic surface than its implementation supports.

There is a boundary. Returning the same final result to every site may still require serialization or construction that the type supports. “Move-only locally” does not mean an arbitrary process-local object can be sent across a network. HPX needs serialization for remote transport, and multiple recipients need a defined way to construct their results.

Move semantics also create state hazards. After std::move(data[0]) feeds the reduction, data[0] remains a valid object but its value is unspecified. The implementation must assign the completed result back before later finalizers read it. It must not invoke the reduction callback twice from the emptied seed.

The move-only test is stronger than searching source for std::move. std::move can appear while a downstream function still copies. Deleting the copy operations makes the full instantiated path prove that no copy is required.

Checkpoint. std::move is a cast that permits consumption. A move-only test proves the instantiated path does not secretly require copying.

One failed callback had to fail every participant

Collective success is shared. Collective failure must be shared too.

Suppose the combining operation throws while reducing values:

struct throwing_plus
{
    int operator()(int left, int right) const
    {
        if (right == 3) {
            throw std::runtime_error("three is rejected");
        }
        return left + right;
    }
};

Several result callbacks can run after the gate opens. If the first finalizer moves op, throws, and leaves the generation half-consumed, a later finalizer must not call the moved-from operation again. It also must not return a value while the first participant receives an exception.

PR #7401 made the communicator cache the first exception for a generation:

all sites arrive
       |
       v
first finalizer attempts result
       |
       +-- success --> cache result --> every waiter receives result
       |
       +-- throws --> cache exception_ptr
                           |
                  +--------+--------+
                  |        |        |
                site 0   site 1   site 2 ...
                  |        |        |
               rethrow same first failure

std::exception_ptr is an owning handle to a captured exception. It lets code store a thrown exception and rethrow it later, possibly on another thread. The “same failure” guarantee means the first captured exception becomes the generation’s outcome. It does not mean every site gets the same in-memory exception object address across processes.

The communicator must clear generation-specific failure state when the next generation begins. Otherwise one failed collective would poison every future operation. Cleanup belongs to the same transition that resets data and gate state, so result, exception, and readiness cannot drift apart.

There are several adversarial cases:

• the step throws while recording a contribution;
• the finalizer throws while combining;
• one participant reaches the failure before others ask for their result;
• the operation object has already been moved;
• the next generation succeeds;
• different sites could observe different local exceptions unless the first is cached.

collectives_throwing_op_test covers shared failure and recovery, and passed in the fresh distributed run. This is not merely “the program did not crash.” The test inspects that every participant receives the expected exception and that a following generation can complete.

Checkpoint. A collective generation has one outcome: a value or the first exception. All waiters observe that outcome, and the next generation begins with neither cached.

Shutdown was another distributed protocol

A program can compute every correct answer and still fail during MPI_Finalize.

HPX can use MPI as a parcel transport. The parcelport keeps an asynchronous receive active so incoming parcels can make progress. It can also duplicate an MPI communicator to isolate HPX traffic from other MPI users. Those resources remain live after the last user collective returns.

An MPI request represents an operation that may still be pending. Finalizing MPI while a receive or duplicated communicator remains active violates the runtime’s lifetime order. Depending on the MPI implementation, the result can be a warning, an abort, a hang, or a crash during process teardown.

PR #7404 repaired the order:

unsafe order

user collective returns
        |
        v
MPI_Finalize
        |
        +-- receive request may still be active
        +-- duplicated communicator may still be owned

repaired order

stop accepting new parcelport work
        |
cancel outstanding receive
        |
wait/test until receive is quiescent
        |
serialize parcelport stop
        |
release duplicated communicator
        |
synchronize participating runtime state
        |
MPI_Finalize

Quiescent means no work is still in flight through the resource being destroyed. Cancellation is a request, not always immediate completion. The code must drive or observe the request to a completed state according to MPI’s rules before releasing related state.

The repair touched:

• libs/core/mpi_base/src/mpi_environment.cpp;
• libs/full/parcelport_mpi/include/hpx/parcelport_mpi/receiver.hpp;
• libs/full/parcelport_mpi/src/parcelport_mpi.cpp.

The environment owns MPI initialization and finalization. The receiver owns the outstanding receive state. The parcelport owns transport stop sequencing. No single file can repair the protocol by pretending the other lifetimes do not exist.

A mutex or serialized stop path prevents two threads from racing through teardown. A barrier before finalization ensures one participant does not tear down global MPI state while another still uses it. Releasing the duplicated communicator before MPI_Finalize matches the resource stack in reverse order: acquire MPI, duplicate communicator, start receive; then cancel receive, release duplicate, finalize MPI.

This is RAII in a distributed setting. RAII means Resource Acquisition Is Initialization: tie a resource’s lifetime to an object’s lifetime so destruction releases it. RAII remains necessary, but destructor order across threads and runtime subsystems still needs an explicit shutdown protocol. Destructors cannot infer that every remote or asynchronous user is done.

The local build-codex used for the collective probes has the TCP parcelport enabled and MPI parcelport disabled, so this article does not claim a fresh MPI teardown run. The evidence is the merged code, its PR review, and the tests reported with that contribution. That boundary is important. A TCP correctness run cannot verify MPI request quiescence.

Checkpoint. Returning from the last collective proves the user operation completed. It does not prove transport receives, duplicated communicators, or progress threads are quiescent.

A free port was not a usable endpoint

The next failure lived in a regression test rather than the collective library. The test starts another HPX locality as a child process, waits for it to depart, and checks AGAS behavior. It needs an endpoint at which the child can start.

Hard-coding a port makes parallel tests collide. PR #7405 asked the operating system for port zero. For TCP, binding to port zero asks the OS to choose an available local port.

The first improvement had this shape:

probe process
  bind(address, port 0)
  OS chooses port P
  read P
  close probe socket
  launch child with port P

It avoids guessing, but it does not reserve (P) across the close-and-rebind gap. Another process can bind it first. That is the familiar time-of-check to time-of-use race:

time ------------------------------------------------------------->

probe:  bind P ---- read P ---- close
other:                              bind P
child:                                      try to bind P -> failure

The still-open #7412 found a different observed error: EADDRNOTAVAIL, “address not available”, rather than the more obvious EADDRINUSE. A port number alone is not the full endpoint. An endpoint contains an address and a port. If the probe binds one address but the child constructs another from environment or hostname resolution, the chosen port can be free and the pair can still be unusable.

The proposed repair passes the exact endpoint the probe actually bound:

port-only handoff

probe binds (address A, port P)
child reconstructs (?, port P) from inherited state
                        |
                        +-- may become address B

exact-endpoint handoff

probe binds (address A, port P)
          |
          +-- serialize A:P directly into child launch configuration

The final commit on the open branch also replaces inherited environment entries instead of appending duplicates. Process environments are arrays of NAME=value strings. If both an old and a new value for the same variable are present, which value a consumer observes can depend on its lookup behavior. Replacement makes the child configuration singular.

The branch head used here is 88f070e94e0d71c424f4686bea07ccb4f2f60ea1. The regression binary was older than that commit, so it was rebuilt. CMake reported the same Git revision, and departed_locality_7384_test --hpx:threads=2 returned zero on this macOS host.

That result has limits:

• it proves the proposed head compiles and its regression passes locally;
• it does not merge the PR;
• it does not reproduce the original endpoint failure locally;
• it does not eliminate the generic close-and-rebind race.

The PR author explicitly reported that the original endpoint failure did not reproduce locally, so CI remains decisive for that observed environment. A good status ledger says “open and locally passing” rather than quietly writing the proposal as released behavior.

Checkpoint. A port is one field of a network endpoint. Preserve the exact address-port pair and replace inherited configuration deterministically.

Two small regressions widened the lesson

The audited contribution set touches two nearby areas outside the final collective and transport paths.

One actions regression covers a non-default-constructible argument. Distributed actions serialize arguments into a parcel and reconstruct them at the destination. Generic deserialization code sometimes assumes it can default construct an object and assign into it:

T value;          // fails if T has no default constructor
archive >> value;

A type can be perfectly movable and serializable without having a meaningful empty state. The regression in non_default_constructible_argument_5998.cpp makes the false requirement visible. The broader lesson matches the move-only reduction: template signatures are promises, and adversarial value types test whether the implementation added hidden constructors or copies.

The other detour touched run_loop.hpp in HPX’s execution module and was later reverted. A run loop schedules units of work and completes futures or sender operations. It is adjacent to collectives because collective continuations eventually run as tasks, but it is not part of the final hierarchical algorithm. Keeping the detour in the audit prevents a polished history from pretending every authored commit marched directly toward one design.

These small changes also clarify asynchronous failure. A function that returns a future can fail in two places:

call operation
   |
   +-- throw before returning a future
   |
   +-- return future
           |
           +-- future becomes exceptional later

Hierarchical collective overloads return futures, but internal tree handoffs may wait or throw before the final future is delivered. “Returns a future” does not automatically mean “fully non-blocking until .get().” A caller must know whether argument validation is reported by an exceptional future, by an immediate throw, or both along different internal paths.

Review scope matters as much as code proximity. An execution-loop tweak in a collectives PR makes failures harder to attribute and reviewers less able to evaluate the semantic blast radius. The scan review and the reverted run-loop detour both support the same discipline: split unrelated mechanisms before claiming one test proves them together.

Checkpoint. Test generic code with types that delete conveniences, and separate adjacent runtime changes unless one invariant genuinely requires both.

Generation two, reconstructed from the call site

Now the original ten-site call can be followed without skipping machinery. Use arity three and public generation two:

auto hcomm = create_hierarchical_communicator(
    "/app/exchange",
    num_sites_arg(10),
    this_site_arg(site),
    arity_arg(3),
    generation_arg(),
    root_site_arg(0),
    flat_fallback_threshold_arg(0));

auto result = all_to_all(
    hcomm,
    std::move(outgoing),
    this_site_arg(site),
    generation_arg(2))
    .get();

The complete trace is:

1. process startup
   hpx.run_hpx_main!=1 makes every locality enter hpx_main

2. communicator creation
   all sites use basename /app/exchange
   root site 0 creates and registers required flat servers
   each other site resolves the servers along its tree path

3. topology
   arity 3 divides ten sites into:
     group 0: sites 0..3, representative 0
     group 1: sites 4..6, representative 4
     group 2: sites 7..9, representative 7

4. validation
   communicator valid
   site in [0,10)
   supplied site matches creation site
   explicit generation 2 is positive
   every source row has ten destination pieces

5. public-to-internal generation mapping
   public 2 -> internal first 3, internal second 4

6. internal generation 3
   subtree communicators gather source rows
   representative communicator exchanges off-diagonal group slices
   each representative retains its diagonal block

7. internal generation 4
   subtree communicators scatter completed destination columns
   representative communicator advances its unused second position

8. payload representation
   uniform or ragged carriers hold contiguous values and shape
   checked arithmetic proves allocation sizes
   moves transfer payloads that are no longer needed

9. completion
   every gate reaches its expected arrivals
   each site's future receives its source-ordered destination column
   if any finalizer threw, every waiter would rethrow the cached first failure

10. next operation
    all touched sub-communicators are aligned for public generation 3,
    whose internal pair will be 5 and 6

11. shutdown
    user completion precedes transport quiescence
    an MPI parcelport, if enabled, must cancel receives and release duplicated
    state before MPI_Finalize

Take source 2’s value for destination 7, encoded as 207. Site 2 belongs to group zero. During internal generation three, it moves to representative zero as part of the gathered rows. Representative zero sends the group-two slice to representative seven. During internal generation four, representative seven scatters the finished column to destination seven. 207 occupies position two because the result is ordered by source.

Take 807. Source eight and destination seven are both in group two. Representative seven obtains it during subtree gather, retains it in the diagonal block, and scatters it to site seven. It does not cross the representative communicator, but that communicator still finishes the same two-position public budget as every other operation.

Take an exception thrown while assembling destination seven. The first exception is cached for that generation. Sites that have already asked for their result and sites that ask later receive the same logical failure. State is reset at the next generation boundary.

Finally, take the process itself. A returned all-to-all future means the collective result is available. It says nothing about a pending MPI receive or a child process endpoint test. Those belong to later protocols with their own participants and lifetime order.

The one public generation was never merely a counter argument. It was the compact name for all of this agreement.

Checkpoint. Starting from the call alone, you should now be able to predict the tree groups, internal generation pair, route of one matrix element, payload shape, failure outcome, next valid public generation, and remaining shutdown work.

Reproducing the evidence

The article probes live in writing-notes/probes/hpx-hierarchy/ in the blog repository. The commands below are the durable description.

Build the standalone mechanics trace and HPX demo:

cmake -S writing-notes/probes/hpx-hierarchy \
  -B /tmp/hpx-hierarchy-probe \
  -DHPX_DIR=/path/to/hpx/build-codex/lib/cmake/HPX

cmake --build /tmp/hpx-hierarchy-probe --parallel 2
/tmp/hpx-hierarchy-probe/mechanics_probe

Run the real demo on two and four localities:

python3 /path/to/hpx/build-codex/bin/hpxrun.py \
  -l 2 -t 2 /tmp/hpx-hierarchy-probe/collective_demo

python3 /path/to/hpx/build-codex/bin/hpxrun.py \
  -l 4 -t 2 /tmp/hpx-hierarchy-probe/collective_demo

Run a focused HPX test by substituting its executable name:

python3 /path/to/hpx/build-codex/bin/hpxrun.py \
  -l 2 -t 2 \
  /path/to/hpx/build-codex/bin/all_to_all_hierarchical_test

The fresh verification set contained:

all_reduce_hierarchical_test
all_gather_hierarchical_test
all_to_all_hierarchical_test
collectives_throwing_op_test
cross_collective_hierarchical_mixed_test
cross_collective_hierarchical_test
exclusive_scan_hierarchical_test
flattened_collective_payloads_test
hierarchical_flat_fallback_test
inclusive_scan_hierarchical_test

All returned zero with two localities and two worker threads per locality. The demo passed at one, two, and four localities. The open #7412 regression was rebuilt from 88f070e… and returned zero locally. The source, commands, and limitations are preserved in the verification record rather than inferred from commit messages.

Source map

These links point at the frozen merged revision:

• communicator creation and recursive tree construction
• topology validation and uneven-group formulas
• communicator server, generation mode, owned basename, and first exception
• shared hierarchical phase helpers
• hierarchical all-reduce
• hierarchical all-gather
• hierarchical all-to-all
• all-to-all payload transformations
• hierarchical scan composition
• uniform and ragged flattened carriers
• MPI environment teardown
• MPI receiver shutdown
• MPI parcelport stop order
• open exact-endpoint regression proposal

Pull-request status ledger

The account has 24 public HPX pull requests in this contribution body. At the audit freeze, 21 were merged, two were closed without merge, and one was open.

PR	Status	What remains in the final explanation
#7136	merged	collectives documentation corrections
#7141	merged	consistent default-argument checks
#7160	merged	hierarchical all-reduce and all-gather
#7189	merged	dedicated hierarchy tests
#7193	merged	configurable flat fallback
#7198	merged	uneven-tree coverage
#7222	closed, not merged	rejected stride-three design
#7307	merged	hierarchical all-to-all
#7321	merged	all-to-all hardening, validation, and fallback
#7326	merged	uniform two-step generation protocol
#7340	merged	hierarchical validation
#7343	merged	inclusive and exclusive scans
#7359	merged	further validation and helper refactoring
#7364	merged	scan test communicator isolation
#7369	merged	rejection of hidden auto-to-explicit transitions
#7370	closed, not merged	monolithic flattening attempt
#7375	merged	flattened gather and scatter
#7377	merged	flattened all-to-all
#7378	merged	owned communicator basenames
#7398	merged	move-aware all-reduce seed
#7401	merged	shared first-exception propagation
#7404	merged	MPI teardown quiescence
#7405	merged	OS-selected regression port
#7412	open	exact endpoint and environment replacement

Glossary at the protocol boundary

Action. An HPX-wrapped function or component operation that can be invoked through the runtime, including from another locality.

AGAS. HPX’s Active Global Address Space, which maps global names and IDs to distributed objects.

Arity. The maximum number of child groups formed at one hierarchy level.

Associative operation. An operation whose grouping can change without changing its mathematical result.

Automatic generation. A mode in which the communicator advances its own hidden generation counter.

Barrier. A synchronization operation that completes only after all participants arrive.

Basename. The shared string used to register or resolve a communicator.

Branch. A movable Git name for one line of repository history.

CMake. The build-system generator used here to find HPX and create native compiler build rules.

Commit. A Git snapshot with parent history, author metadata, and a message.

Continuous integration (CI). Automated building and testing performed for a proposed repository change.

Collective. One coordinated operation performed by every member of a specified group.

Commutative operation. An operation whose operand order can be swapped without changing its result.

Communicator. A handle and associated server state that define collective membership, identity, and reusable generations.

Component. An HPX-addressable C++ object whose operations can be exposed as actions.

Endpoint. A network address and port considered together.

Exceptional future. A future that becomes ready with an exception instead of a value.

Exclusive scan. A per-site ordered prefix reduction that excludes the current site’s value.

Explicit generation. A generation number supplied by the caller.

Finalizer. The communicator callback that creates one participant’s result after all required arrivals.

Flat fallback. Replacing a requested hierarchy with one flat communicator below a configurable site-count threshold.

Forwarding. Preserving an argument’s value category through generic C++ so the eventual callee can copy or move appropriately.

Future. A handle to a value or exception that may become available later.

Gate. Here, the reusable and_gate that tracks arrivals for a communicator generation.

Generation. The logical use number that distinguishes repeated operations on one communicator identity.

Hierarchical communicator. A per-site vector of flat communicator handles forming that site’s path through a balanced tree.

Inclusive scan. A per-site ordered prefix reduction that includes the current site’s value.

Locality. An HPX runtime process participating in the distributed application.

Move-only type. A C++ type whose resources can be transferred but not copied.

MPI. The Message Passing Interface, a standard API used here as one possible HPX network transport.

Parcel. HPX’s message envelope for remote actions and their serialized arguments.

Parcelport. An HPX transport implementation that sends and receives parcels, such as the TCP or MPI parcelport.

Pull request (PR). A proposed repository change with review and merge status.

Quiescence. A state in which no operation remains in flight through a resource that is about to be destroyed.

Ragged rows. Rows with varying lengths, represented here by contiguous data and prefix offsets.

Representative. The leftmost site of a hierarchy group, responsible for carrying that group’s data to the next level.

Regression test. A test retained to make a previously observed failure fail again if the repair is lost.

Root site. The designated root of a rooted collective or communicator. The current hierarchical implementation requires site zero.

Sanitizer. Compiler-instrumented runtime checking for classes of C++ errors such as invalid memory use or undefined behavior.

Site. A numbered participant in a collective. In distributed examples here, one locality supplies one site.

Step. The communicator callback that records one participant’s contribution.

Uniform rows. Equal-length rows, represented here by contiguous data and one shared row width.

The exhaustive authored-commit and touched-file audit follows. It is long on purpose. A contribution inventory that lists only the clean final commits would erase the rejected designs, review repairs, merges, and adjacent regressions that explain how the final protocol acquired its boundaries.

Appendix: all 186 authored HPX commits

Generated from public refs under origin in /Users/anshumanagrawal/codes/workplace/2026_projects/hpx.

Author match: ^(iemAnshuman|Anshuman|Anshuman Agrawal)

Invariant: 186 unique commits.

The feature labels describe where each commit ended up conceptually. A merge or repair commit can therefore appear beside the final feature rather than beside the branch on which it happened.

Foundational API and documentation cleanup

Small API consistency and documentation repairs that made the later work easier to reason about.

Count: 1.

Commit	Date	Subject	Directly touched paths
12d4030152c7	2026-04-12	Fix argument type access: use implicit conversion instead of .get()	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp

Hierarchical all-reduce and all-gather

The first composed hierarchical collectives, their tests, and their early performance investigation.

Count: 17.

Commit	Date	Subject	Directly touched paths
eefc489af7ee	2026-04-02	Fix same copy-paste issue in all_gather.hpp Doxygen comments	libs/full/collectives/include/hpx/collectives/all_gather.hpp
18835139a589	2026-04-04	Use is_default() consistently in hierarchical gather_here	libs/full/collectives/include/hpx/collectives/gather.hpp
9fb2138f3df3	2026-04-10	Add hierarchical all_gather by composing gather + broadcast	libs/full/collectives/include/hpx/collectives/all_gather.hpp
e66113fbe8f7	2026-04-10	Add all_reduce to benchmark_collectives with adjustable message sizes and hierarchical support	libs/full/collectives/tests/performance/benchmark_collectives.cpp
28a94dd45897	2026-04-11	Add missing include for HPX_INVOKE in all_reduce.hpp	libs/full/collectives/include/hpx/collectives/all_reduce.hpp
03562915bb61	2026-04-11	Consolidate detail namespace for hierarchical collectives and fix modules build	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/tests/performance/benchmark_collectives.cpp
0969ff19299b	2026-04-11	Fix clang-format issues in hierarchical all_reduce	libs/full/collectives/include/hpx/collectives/all_reduce.hpp
b4ef72349267	2026-04-11	Remove conflicting agas_interface include from all_gather (C++20 modules fix)	libs/full/collectives/include/hpx/collectives/all_gather.hpp
308fd815b92c	2026-04-11	Remove conflicting agas_interface include from all_reduce (C++20 modules fix)	libs/full/collectives/include/hpx/collectives/all_reduce.hpp
266214824313	2026-04-12	Fix benchmark: use scalar op for hierarchical all_reduce (vector_reduce_op wraps it)	libs/full/collectives/tests/performance/benchmark_collectives.cpp
0521b56c193b	2026-04-12	Refactor hierarchical collectives to use non-blocking future continuations	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
adae294eac0f	2026-04-13	Revert non-blocking refactor: restore .get() for hierarchical collectives	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
c1bba7a0c175	2026-04-15	Add unit tests for hierarchical all_reduce and all_gather	libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/all_gather_hierarchical.cpp libs/full/collectives/tests/unit/all_reduce_hierarchical.cpp
5ea4664cccbc	2026-04-15	style: apply clang-format to resolve PR check failure	libs/full/collectives/tests/unit/all_gather_hierarchical.cpp libs/full/collectives/tests/unit/all_reduce_hierarchical.cpp
87abae99922b	2026-06-16	relax the hierarchical communicator cross-sharing restriction docs	libs/full/collectives/include/hpx/collectives/create_communicator.hpp
e91321186a27	2026-07-13	Unify hierarchical gather result construction	libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/tests/unit/flattened_collective_payloads.cpp
24ef5dfa7f12	2026-07-21	Move the all_reduce reduction seed instead of copying it	libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/all_reduce_move_seed.cpp

Topology, fallback, and validation hardening

Uneven trees, arity handling, flat fallback, caller validation, and helper cleanup.

Count: 41.

Commit	Date	Subject	Directly touched paths
79b39308a0fe	2026-04-02	Fix copy-paste errors in collectives Doxygen comments	libs/full/collectives/include/hpx/collectives/reduce.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp
10575655ecc0	2026-04-11	Refactor vector_reduce_op to deduce type and use HPX_INVOKE	libs/full/collectives/include/hpx/collectives/all_reduce.hpp
7321712e029c	2026-04-11	Move non-public collective functions to namespace detail	libs/full/collectives/include/hpx/collectives/all_reduce.hpp
6e806d1de9ec	2026-04-11	Match clang-format 20 formatting	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
21e571b4f142	2026-04-12	Fix reduce_there calls: remove op argument (only reduce_here takes op)	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/tests/performance/benchmark_collectives.cpp
c312a5857eeb	2026-04-12	Final clang-format alignment fixes	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
3fd2978cacb0	2026-04-12	Remove vector<t> overloads: user supplies operator matching their data type	libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/tests/performance/benchmark_collectives.cpp
cc3e4ef1ed01	2026-04-12	Fix reduce_there: remove op argument	libs/full/collectives/include/hpx/collectives/all_reduce.hpp
c9cab5e15e8a	2026-04-12	Fix missing op parameter in reduce_there and apply format	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
b28e088ad850	2026-04-15	Add adaptive flat fallback for hierarchical collectives below 16 sites	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
aa499a5ab6d6	2026-04-15	Sort test entries alphabetically in CMakeLists.txt	libs/full/collectives/tests/unit/CMakeLists.txt
0c6b7ac4864f	2026-04-18	Add non-power-of-arity tests for hierarchical all_reduce and all_gather	libs/full/collectives/tests/unit/all_gather_hierarchical.cpp libs/full/collectives/tests/unit/all_reduce_hierarchical.cpp
0b51d1188233	2026-04-19	Extend non-power-of-arity tests to remaining hierarchical collectives	libs/full/collectives/tests/unit/broadcast_hierarchical.cpp libs/full/collectives/tests/unit/gather_hierarchical.cpp libs/full/collectives/tests/unit/reduce_hierarchical.cpp libs/full/collectives/tests/unit/scatter_hierarchical.cpp
42d4c99849c4	2026-04-17	Add flat fallback for hierarchical_communicator below site-count threshold	libs/full/collectives/include/hpx/collectives/argument_types.hpp libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/hierarchical_flat_fallback.cpp
66b9fc29ff3e	2026-04-17	Extend benchmark_collectives with all_gather and fallback_threshold CLI	libs/full/collectives/tests/performance/benchmark_collectives.cpp
37976baf3936	2026-06-01	fix CMakeLists.txt formatting destroyed by linter	libs/full/collectives/CMakeLists.txt libs/full/collectives/tests/unit/CMakeLists.txt
7cced25edd2d	2026-06-01	use std::div instead of seperate / and %	libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp
d1bc9131edc7	2026-06-01	restore macros.hpp and GLOBAL_HEADER_GEN in collectives CMakeLists	libs/full/collectives/CMakeLists.txt
0008eb79b4fc	2026-06-02	make flat_fallback explicit, drop the default argument	libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/src/create_communicator.cpp
9a41fc683ad1	2026-06-02	use collectives module header in unit tests	libs/full/collectives/CMakeLists.txt libs/full/collectives/tests/unit/hierarchical_helpers.cpp libs/full/collectives/tests/unit/subtree_gather_scatter.cpp
910e557d5ae1	2026-06-26	Remove redundant clang-format guards	libs/full/collectives/tests/unit/concurrent_collectives.cpp
9b147a710325	2026-06-26	Restore noexcept on communicator server constructor	libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/src/create_communicator.cpp
5bf95cef5e7c	2026-07-09	Merge branch ‘master’ into fix/collectives-hardening	merge/synchronization commit
f0a84859dd2f	2026-07-09	Merge branch ‘master’ into fix/collectives-hardening	merge/synchronization commit
575ec21b4b35	2026-07-10	Fix C++20 module export declarations	libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp
75f16a9d0da4	2026-07-11	Merge branch ‘master’ into fix/collectives-hardening	merge/synchronization commit
3cb78868a4d8	2026-07-11	Fix collectives helper symbol visibility	libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp
f893e77e68dc	2026-07-11	Fix collectives C++20 module tests	libs/full/collectives/tests/unit/hierarchical_helpers.cpp libs/full/collectives/tests/unit/subtree_gather_scatter.cpp
8df3ca586667	2026-07-11	Merge branch ‘master’ into fix/collectives-hardening	merge/synchronization commit
ba9e2814cbdb	2026-07-11	Fix collectives helper module exports	libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp libs/full/collectives/tests/unit/subtree_gather_scatter.cpp
0378254ab113	2026-07-12	Merge branch ‘master’ into fix/collectives-hardening	merge/synchronization commit
ab32bc73cfea	2026-07-13	Merge branch ‘master’ into fix/collectives-hardening	merge/synchronization commit
8bf53db7d39b	2026-07-13	Limit collectives helper module exports	libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp
c9538d57c019	2026-07-13	Merge branch ‘master’ into fix/collectives-hardening	merge/synchronization commit
2adde0d13495	2026-07-13	Rerun flaky partition test	merge/synchronization commit
c8b2cc68681e	2026-07-14	Expand uniform row carrier coverage	libs/full/collectives/tests/unit/hierarchical_helpers.cpp
5ecb5c71c8b2	2026-07-14	Merge branch ‘master’ into fix/collectives-hardening	merge/synchronization commit
9670df8b35e2	2026-07-14	Address final collectives review findings	libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/reduce.hpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/tests/unit/all_reduce.cpp libs/full/collectives/tests/unit/reduce.cpp
b53b5584c665	2026-07-14	Document local communicator validation	libs/full/collectives/include/hpx/collectives/create_communicator.hpp
ddbdfde563e1	2026-07-14	Fix private collectives tests with C++ modules	libs/full/collectives/tests/unit/flattened_collective_payloads.cpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp
e6324bd801e0	2026-07-24	Extend the failure caching to the collective step function	libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/tests/unit/collectives_throwing_op.cpp

Hierarchical all-to-all

Subtree gather, representative exchange, subtree scatter, and the design alternatives explored around them.

Count: 39.

Commit	Date	Subject	Directly touched paths
0f90f3b5c206	2026-04-25	Add hierarchical all-to-all design documentation	docs/hierarchical_all_to_all_design.md
5c8a0c0ed791	2026-05-26	Add partition helpers for hierarchical all_to_all collective	libs/full/collectives/CMakeLists.txt libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/hierarchical_helpers.cpp
e258aaf1aa99	2026-06-01	removed result construction duplication	libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp
2abcad6780f1	2026-06-01	add HPX_ASSERT for empty comm in subtree_send_to_top_rep	libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp
3a1b5eca4f32	2026-06-01	simplify subtree_receive_from_top_rep early return	libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp
1ab09dfca690	2026-06-01	add HPX_ASSERT for empty communicator in subtree_receive_from_top_rep	libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp
4bd91761ddf0	2026-06-01	fix non ASCII dash	libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp
74571d1ce6e6	2026-06-01	remove duplicate includes	libs/full/collectives/include/hpx/collectives/all_to_all.hpp
8d9bff379c56	2026-06-01	add HPX_CXX_EXPORT to hierarchical all_to_all overloads	libs/full/collectives/include/hpx/collectives/all_to_all.hpp
2ec60c1f5131	2026-06-01	rename is_rep to is_representative	libs/full/collectives/include/hpx/collectives/all_to_all.hpp
8ef70bbdabdb	2026-06-01	rename loop index h to group	libs/full/collectives/include/hpx/collectives/all_to_all.hpp
22ca8b8bd2c3	2026-06-01	fix left non ascii dash	libs/full/collectives/include/hpx/collectives/all_to_all.hpp
149b666b3212	2026-06-01	fix subtree_receive_from_top_rep return for nested payloads	libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp
460efc1a3068	2026-06-01	make all_to_all sync overload generic over communicator type	libs/full/collectives/include/hpx/collectives/all_to_all.hpp
6bc1099b53d1	2026-06-01	apply clang-format to hierarchical all_to_all sources	libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp
7c612f0a2fc3	2026-06-01	forward declare hierarchical all_to_all overload	libs/full/collectives/include/hpx/collectives/all_to_all.hpp
27bf5617f946	2026-06-01	rely on loop boundary for single-level subtree gather	libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp
5ebaa24a766f	2026-06-02	assert non-empty comms in subtree_scatter_at_top_rep	libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp
c279fb9f22fe	2026-06-02	return future from subtree to avoid make_ready_future	libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp libs/full/collectives/tests/unit/subtree_gather_scatter.cpp
f2d1d29471e1	2026-06-02	return signed index from classify_site to drop casts	libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp
a1466cc14c6d	2026-06-02	hoist classify_site, add arity assert in get_top_level_groups	libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp
110f0ab3bc82	2026-06-02	use std::move iterator range for Phase 2 block packing	libs/full/collectives/include/hpx/collectives/all_to_all.hpp
23ae8735607d	2026-06-02	add missing algorithm include for std::move iterator range	libs/full/collectives/include/hpx/collectives/all_to_all.hpp
9b5ee2af2bb8	2026-06-02	export hierarchical detail helpers for the module BMI	libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp
29bc4d2801ac	2026-06-11	dispatch hierarchical collectives flat fallback on arity >= num_sites	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/tests/unit/hierarchical_flat_fallback.cpp
6ca836ca1f45	2026-06-11	validate user input in hierarchical collectives entry points	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/src/create_communicator.cpp
3aa3ed4bef7f	2026-06-11	correct hierarchical collectives documentation and comments	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/argument_types.hpp libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/src/create_communicator.cpp
1b0e1d86bade	2026-06-11	add all_to_all to the hierarchical collectives benchmark	libs/full/collectives/tests/performance/benchmark_collectives.cpp
43c683cf21ed	2026-06-12	add missing hpx/assert.hpp includes flagged by inspect	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp
3fd375456c53	2026-06-22	Harden collectives validation paths	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/barrier.hpp libs/full/collectives/include/hpx/collectives/broadcast.hpp libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/tests/unit/all_to_all.cpp libs/full/collectives/tests/unit/all_to_all_sync.cpp libs/full/collectives/tests/unit/barrier.cpp libs/full/collectives/tests/unit/barrier_hierarchical.cpp libs/full/collectives/tests/unit/concurrent_collectives.cpp
315e1059e8ef	2026-06-26	Apply clang-format to collectives changes	libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/tests/unit/all_to_all.cpp libs/full/collectives/tests/unit/barrier.cpp libs/full/collectives/tests/unit/concurrent_collectives.cpp
4bccc97eaab8	2026-06-26	Simplify all_to_all test value initialization	libs/full/collectives/tests/unit/all_to_all.cpp
e9338e1a7d6d	2026-06-26	Remove redundant all_to_all reserve	libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp
8709e86b4e0b	2026-06-26	Remove redundant single-element reserves	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp libs/full/collectives/include/hpx/collectives/gather.hpp
69a08e2598f4	2026-07-08	Harden collectives validation paths	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/barrier.hpp libs/full/collectives/include/hpx/collectives/broadcast.hpp libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/include/hpx/collectives/reduce.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp libs/full/collectives/src/barrier.cpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/tests/unit/barrier.cpp libs/full/collectives/tests/unit/barrier_hierarchical.cpp
cfdca53b85eb	2026-07-09	Harden hierarchical collective validation	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/barrier.hpp libs/full/collectives/include/hpx/collectives/broadcast.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/include/hpx/collectives/reduce.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp libs/full/collectives/src/create_communicator.cpp
29435fc64da2	2026-07-09	Address hierarchical validation review feedback	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp
773c38e53498	2026-07-09	Move hierarchical helpers out of header	libs/full/collectives/CMakeLists.txt libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/src/detail/hierarchical_helpers.cpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp libs/full/collectives/tests/unit/subtree_gather_scatter.cpp
21b35a9ba15e	2026-07-15	Address additional hierarchical all-to-all review feedback	libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp

Shared generation protocol

The uniform two-step protocol, explicit generation rules, and cross-collective communicator reuse.

Count: 21.

Commit	Date	Subject	Directly touched paths
d2c15b6a01db	2026-04-10	Add hierarchical all_reduce overloads with 2k/2k+1 generation mapping and vector support	libs/full/collectives/include/hpx/collectives/all_reduce.hpp
a2e8c66680a7	2026-04-11	Reject default generation for hierarchical overloads, remove benchmark files from history	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
32d0c715199c	2026-04-11	Add copyright lines, reject default generation for hierarchical overloads	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/tests/performance/benchmark_collectives.cpp
fcdb2f0be963	2026-04-11	Refactor generation guards to use is_default()	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
5d4cb50b1fa5	2026-04-11	Inline generation math by removing intermediate k variable	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
d84ee700f4f3	2026-04-11	Hoist generation checks in hierarchical collectives	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
7f1f8c4cec20	2026-04-14	Fix generation mapping: use 2k-1/2k to ensure sequential generations starting from 1	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
b4c0f35f56d3	2026-04-14	Fix comments: generation mapping is 2k-1/2k, not 2k/2k+1	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp
9678c580ff44	2026-06-11	restore tree coverage in hierarchical collectives unit tests	libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/all_gather_hierarchical.cpp libs/full/collectives/tests/unit/all_reduce_hierarchical.cpp libs/full/collectives/tests/unit/broadcast_hierarchical.cpp libs/full/collectives/tests/unit/cross_collective_hierarchical.cpp libs/full/collectives/tests/unit/gather_hierarchical.cpp libs/full/collectives/tests/unit/hierarchical_flat_fallback.cpp libs/full/collectives/tests/unit/reduce_hierarchical.cpp libs/full/collectives/tests/unit/scatter_hierarchical.cpp
0f94fac8512d	2026-06-15	advance the collectives gate by a configurable number of generations	libs/full/collectives/include/hpx/collectives/detail/communicator.hpp
0914f1beb667	2026-06-15	step the hierarchical all_to_all inter-group exchange by two generations	libs/full/collectives/include/hpx/collectives/all_to_all.hpp
d58a07d6d3a0	2026-06-16	advance single-phase hierarchical collectives by two generations per call	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/broadcast.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/include/hpx/collectives/reduce.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp
62f42652b15d	2026-06-16	test cross-collective sharing of one hierarchical communicator instance	libs/full/collectives/tests/unit/cross_collective_hierarchical.cpp
c6703e6513d1	2026-06-16	test mixed and default-generation cross-collective sharing across tree shapes	libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/cross_collective_hierarchical_mixed.cpp
9b8f10bce612	2026-06-16	step the hierarchical collectives flat fast path by two generations	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/tests/unit/cross_collective_hierarchical_mixed.cpp
42816cd202c5	2026-06-16	reject a zero generation in the hierarchical collectives	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/barrier.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/tests/unit/cross_collective_hierarchical_mixed.cpp
35f62a39c5ee	2026-06-17	Merge branch ‘master’ into feat/unify-generation-step	merge/synchronization commit
c7aa65eb5faf	2026-06-18	collectives: hide the per-call generation step behind a generation_mode enum	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/argument_types.hpp libs/full/collectives/include/hpx/collectives/broadcast.hpp libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/include/hpx/collectives/reduce.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp
e8739ccfd8de	2026-06-19	Merge branch ‘master’ into feat/unify-generation-step	merge/synchronization commit
080c9d68bd03	2026-06-26	Document generation data serialization cursor	libs/full/collectives/tests/unit/concurrent_collectives.cpp
cd8b7d3b5efd	2026-07-11	Reject explicit generations after auto-generation use	libs/full/collectives/include/hpx/collectives/argument_types.hpp libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/tests/unit/cross_collective_hierarchical_mixed.cpp

Hierarchical scans

Inclusive and exclusive scan composition, ordering, root restrictions, and scan-specific tests.

Count: 14.

Commit	Date	Subject	Directly touched paths
911bb92fae04	2026-06-01	replaced linear scan with lower_bound	libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp
d867b917766f	2026-07-02	Add hierarchical scan collectives	libs/full/collectives/docs/index.rst libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/include/hpx/collectives/exclusive_scan.hpp libs/full/collectives/include/hpx/collectives/inclusive_scan.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp libs/full/collectives/tests/performance/benchmark_collectives.cpp libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/cross_collective_hierarchical.cpp libs/full/collectives/tests/unit/cross_collective_hierarchical_mixed.cpp libs/full/collectives/tests/unit/exclusive_scan_.cpp libs/full/collectives/tests/unit/exclusive_scan_hierarchical.cpp libs/full/collectives/tests/unit/exclusive_scan_sync.cpp libs/full/collectives/tests/unit/inclusive_scan_.cpp libs/full/collectives/tests/unit/inclusive_scan_hierarchical.cpp libs/full/collectives/tests/unit/inclusive_scan_sync.cpp
22de4f4b0d3d	2026-07-03	Address hierarchical scan review feedback	libs/full/collectives/docs/index.rst libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/include/hpx/collectives/exclusive_scan.hpp libs/full/collectives/include/hpx/collectives/inclusive_scan.hpp libs/full/collectives/tests/unit/cross_collective_hierarchical.cpp
3612dcfa491b	2026-07-03	Use shared collectives bool helper	libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/broadcast.hpp libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/include/hpx/collectives/exclusive_scan.hpp libs/full/collectives/include/hpx/collectives/inclusive_scan.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp
1d23d7d79c5b	2026-07-04	Unify scan result helpers	libs/full/collectives/include/hpx/collectives/exclusive_scan.hpp libs/full/collectives/include/hpx/collectives/inclusive_scan.hpp
cf587202c4ff	2026-07-06	Address hierarchical scan review feedback	libs/full/collectives/CMakeLists.txt libs/full/collectives/include/hpx/collectives/broadcast.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_scan_helpers.hpp libs/full/collectives/include/hpx/collectives/exclusive_scan.hpp libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/include/hpx/collectives/inclusive_scan.hpp libs/full/collectives/include/hpx/collectives/reduce.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp libs/full/collectives/tests/unit/cross_collective_hierarchical.cpp libs/full/collectives/tests/unit/cross_collective_hierarchical_mixed.cpp
c9f2a3aa0c18	2026-07-06	Merge branch ‘master’ into feat/inclusive-exclusive-scan	merge/synchronization commit
5f2f0473e15d	2026-07-06	Remove redundant future waits from collectives tests	libs/full/collectives/tests/unit/cross_collective_hierarchical_mixed.cpp libs/full/collectives/tests/unit/exclusive_scan_.cpp
7336967baa7c	2026-07-07	Merge branch ‘master’ into feat/inclusive-exclusive-scan	merge/synchronization commit
dbf793ed1e4d	2026-07-09	Avoid communicator reuse between scan test phases	libs/full/collectives/tests/unit/exclusive_scan_.cpp libs/full/collectives/tests/unit/exclusive_scan_sync.cpp libs/full/collectives/tests/unit/inclusive_scan_.cpp libs/full/collectives/tests/unit/inclusive_scan_sync.cpp
a59354fba985	2026-07-11	Harden hierarchical scan validation	libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_scan_helpers.hpp libs/full/collectives/tests/unit/inclusive_scan_hierarchical.cpp
122663f4f73d	2026-07-12	Fix hierarchical scan clang-tidy warning	libs/full/collectives/tests/unit/inclusive_scan_hierarchical.cpp
573e9eef006b	2026-07-14	Address remaining collectives review findings	libs/full/collectives/include/hpx/collectives/all_gather.hpp libs/full/collectives/include/hpx/collectives/all_reduce.hpp libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/barrier.hpp libs/full/collectives/include/hpx/collectives/broadcast.hpp libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_scan_helpers.hpp libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/include/hpx/collectives/reduce.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/src/detail/hierarchical_helpers.cpp libs/full/collectives/tests/unit/barrier_hierarchical.cpp libs/full/collectives/tests/unit/broadcast_hierarchical.cpp libs/full/collectives/tests/unit/exclusive_scan_hierarchical.cpp libs/full/collectives/tests/unit/gather_hierarchical.cpp libs/full/collectives/tests/unit/inclusive_scan_hierarchical.cpp libs/full/collectives/tests/unit/reduce_hierarchical.cpp libs/full/collectives/tests/unit/scatter_hierarchical.cpp
c37e7f2c95c2	2026-07-14	Improve communicator validation coverage	libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_scan_helpers.hpp libs/full/collectives/src/barrier.cpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/tests/unit/barrier.cpp libs/full/collectives/tests/unit/broadcast_hierarchical.cpp libs/full/collectives/tests/unit/gather_hierarchical.cpp libs/full/collectives/tests/unit/reduce_hierarchical.cpp libs/full/collectives/tests/unit/scatter_hierarchical.cpp

Contiguous and ragged payloads

Flattened gather, scatter, and all-to-all carriers, including overflow and vector<bool> edge cases.

Count: 19.

Commit	Date	Subject	Directly touched paths
de66e729ae61	2026-07-13	Add a uniform hierarchical payload carrier	libs/full/collectives/CMakeLists.txt libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp
40732d08d2eb	2026-07-13	Flatten hierarchical gather and scatter payloads	libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/flattened_collective_payloads.cpp
7f90f9c302fa	2026-07-13	Move uniform row operations into the carrier	libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp libs/full/collectives/tests/unit/flattened_collective_payloads.cpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp
6483e21b1134	2026-07-13	Correct hierarchical gather row unwrapping	libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp
ce944c515d12	2026-07-14	Refine uniform row carrier operations	libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp
4190c4dd15bb	2026-07-14	Keep uniform scatter details local	libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp
5e2644732c5e	2026-07-14	Harden uniform row carrier invariants	libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp libs/full/collectives/tests/unit/flattened_collective_payloads.cpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp
d0cf0ecc7104	2026-07-14	Refine hierarchical collective internals	libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/include/hpx/collectives/scatter.hpp
f8e0be99d7f5	2026-07-14	Merge branch ‘master’ into feat/flatten-hierarchical-gather-scatter	merge/synchronization commit
34ff6a4e848b	2026-07-14	Export flattened collective module details	libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/tests/unit/flattened_collective_payloads.cpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp
8125f51f656a	2026-07-14	Merge master into flattened gather and scatter	merge/synchronization commit
5bf213218a38	2026-07-14	Merge branch ‘master’ into feat/flatten-hierarchical-gather-scatter	merge/synchronization commit
8e5e843d8928	2026-07-15	Flatten hierarchical all-to-all exchange payloads	libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/flattened_collective_payloads.cpp libs/full/collectives/tests/unit/hierarchical_flat_fallback.cpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp libs/full/collectives/tests/unit/subtree_gather_scatter.cpp
780d3bf25eca	2026-07-15	Fix flattened carrier element extraction	libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp
c617282402dc	2026-07-15	Address hierarchical all-to-all review feedback	libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp
1e4f699530e8	2026-07-15	Merge branch ‘master’ into feat/flatten-hierarchical-all-to-all	merge/synchronization commit
7bbc7bc18e4b	2026-07-15	Merge branch ‘master’ into feat/flatten-hierarchical-all-to-all	merge/synchronization commit
1e0acdd11f79	2026-07-15	Address follow-up hierarchical all-to-all review feedback	libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp libs/full/collectives/include/hpx/collectives/gather.hpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp
1da48b927b2a	2026-07-15	Merge branch ‘master’ into feat/flatten-hierarchical-all-to-all	merge/synchronization commit

Ownership and failure convergence

Owned communicator names, move-aware reduction seeds, and caching the first exception for every participant.

Count: 11.

Commit	Date	Subject	Directly touched paths
c3fb3500af53	2026-04-11	Fix broken line split in make_exceptional_future call	libs/full/collectives/include/hpx/collectives/all_reduce.hpp
4c1fa1cabfde	2026-06-26	Keep communicator basename as pointer	libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/src/create_communicator.cpp
ee002c8d52b7	2026-07-11	Fix dangling communicator basename in diagnostics	libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/src/create_communicator.cpp
9145fc55ee65	2026-07-11	Harden communicator basename ownership for diagnostics	libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/src/create_communicator.cpp
edb3e7aad782	2026-07-15	Move communicator basenames into owned storage	libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/tests/unit/barrier.cpp
f1c8eed40963	2026-07-15	Merge branch ‘master’ into fix/collectives-basename-move	merge/synchronization commit
a64eef2b60b9	2026-07-23	Merge branch ‘master’ into perf/all-reduce-move-seed	merge/synchronization commit
cd694679adc1	2026-07-24	Deduplicate the throwing/recovering lambdas in the collectives test	libs/full/collectives/tests/unit/collectives_throwing_op.cpp
da14e69341e1	2026-07-25	Merge branch ‘master’ into perf/all-reduce-move-seed	merge/synchronization commit
e66bbd460993	2026-07-25	Merge branch ‘master’ into perf/all-reduce-move-seed	merge/synchronization commit
71cbd8ff93b9	2026-07-25	Merge branch ‘master’ into perf/all-reduce-move-seed	merge/synchronization commit

MPI shutdown reliability

Quiescing outstanding receives and releasing duplicated MPI state before finalization.

Count: 9.

Commit	Date	Subject	Directly touched paths
9a103ef0eb63	2026-07-23	Cache the first collective finalizer failure and rethrow it for every site	libs/full/collectives/include/hpx/collectives/detail/communicator.hpp libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/collectives_throwing_op.cpp
832b6ab24c60	2026-07-23	Merge branch ‘master’ into fix/collectives-finalizer-exception-caching	merge/synchronization commit
9e101b7b7791	2026-07-24	Rename finalizer_error_ to operation_error_	libs/full/collectives/include/hpx/collectives/detail/communicator.hpp
781d1340b03a	2026-07-24	Synchronize entry into MPI_Finalize across all ranks	libs/core/mpi_base/src/mpi_environment.cpp
c20946d45acc	2026-07-24	Cancel the pending wildcard header receive before finalizing MPI	libs/full/parcelport_mpi/include/hpx/parcelport_mpi/receiver.hpp libs/full/parcelport_mpi/src/parcelport_mpi.cpp
f710f19e5510	2026-07-24	Release the duplicated communicator before MPI_Finalize	libs/core/mpi_base/src/mpi_environment.cpp
e6d8379a664c	2026-07-24	Serialize receiver::stop against late header polling	libs/full/parcelport_mpi/include/hpx/parcelport_mpi/receiver.hpp
8a01393e2616	2026-07-25	Merge branch ‘master’ into fix/mpi-parcelport-finalize-quiesce	merge/synchronization commit
9025737c59dc	2026-07-25	Merge branch ‘master’ into fix/mpi-parcelport-finalize-quiesce	merge/synchronization commit

Departed-locality endpoint reliability

Port probing, exact endpoint propagation, and environment replacement in the AGAS regression.

Count: 10.

Commit	Date	Subject	Directly touched paths
b2a9f46bb773	2026-06-26	Simplify scatter data range endpoints	libs/full/collectives/include/hpx/collectives/scatter.hpp
909e8a793733	2026-07-25	Look up the AGAS address once in the departed locality test	libs/full/agas/tests/regressions/departed_locality_7384.cpp
26c8e7d34fd0	2026-07-25	Ask the OS for an unused port in the departed locality test	libs/full/agas/tests/regressions/departed_locality_7384.cpp
acf2394d5f3f	2026-07-25	Say plainly that the port probe cannot reserve the port	libs/full/agas/tests/regressions/departed_locality_7384.cpp
19e0ef561bfb	2026-07-25	Claim the probe port through the throwing asio overloads	libs/full/agas/tests/regressions/departed_locality_7384.cpp
690cfbd6d462	2026-07-25	Merge branch ‘master’ into fix/departed-locality-test-free-port	merge/synchronization commit
b99068e8ca7a	2026-07-25	Merge branch ‘master’ into fix/departed-locality-test-free-port	merge/synchronization commit
3a931cc90a6c	2026-07-28	Give the launched locality the endpoint the probe actually bound	libs/full/agas/tests/regressions/departed_locality_7384.cpp
2a1c4aa059f3	2026-07-28	Merge branch ‘master’ into fix/departed-locality-endpoint	merge/synchronization commit
88f070e94e0d	2026-07-28	Replace inherited environment entries instead of appending duplicates	libs/full/agas/tests/regressions/departed_locality_7384.cpp

Adjacent regressions and exploratory detours

Small work outside collectives that still supplied lessons about serialization, genericity, and review scope.

Count: 4.

Commit	Date	Subject	Directly touched paths
ec5552d7e7b7	2026-05-31	add hierarchical collective -> all_to_all	libs/core/execution/include/hpx/execution/algorithms/run_loop.hpp libs/full/collectives/CMakeLists.txt libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/all_to_all_hierarchical.cpp libs/full/collectives/tests/unit/hierarchical_helpers.cpp libs/full/collectives/tests/unit/subtree_gather_scatter.cpp
9cd9e603e151	2026-06-01	revert run_loop.hpp Apple Clang fix	libs/core/execution/include/hpx/execution/algorithms/run_loop.hpp libs/full/collectives/CMakeLists.txt libs/full/collectives/include/hpx/collectives/all_to_all.hpp libs/full/collectives/include/hpx/collectives/create_communicator.hpp libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp libs/full/collectives/src/create_communicator.cpp libs/full/collectives/tests/unit/CMakeLists.txt libs/full/collectives/tests/unit/all_to_all_hierarchical.cpp libs/full/collectives/tests/unit/subtree_gather_scatter.cpp
20bcd27e45dc	2026-06-18	Own the asynchronous action test argument before serialization.	libs/full/actions/tests/regressions/non_default_constructible_argument_5998.cpp
4dede97ead5a	2026-07-06	Fix non-default constructible action regression	libs/full/actions/tests/regressions/non_default_constructible_argument_5998.cpp

Appendix: all 59 touched HPX paths

Generated from the same 186-commit public-ref audit as the commit ledger.

Invariant: 59 unique paths.

Foundational API and documentation cleanup

Path	Authored commits touching it	Role in the final work
libs/full/collectives/docs/index.rst	2	Public collectives documentation and argument semantics.
libs/full/collectives/include/hpx/collectives/argument_types.hpp	4	Strong wrapper arguments such as arity, generation, root site, and flat-fallback threshold.

Hierarchical all-reduce and all-gather

Path	Authored commits touching it	Role in the final work
libs/full/collectives/include/hpx/collectives/all_gather.hpp	33	Public all gather API, documentation, validation, or hierarchical dispatch.
libs/full/collectives/include/hpx/collectives/all_reduce.hpp	40	Public all reduce API, documentation, validation, or hierarchical dispatch.
libs/full/collectives/tests/performance/benchmark_collectives.cpp	9	Historical flat-versus-hierarchical performance harness.
libs/full/collectives/tests/unit/all_gather_hierarchical.cpp	4	Focused test coverage for all gather hierarchical.
libs/full/collectives/tests/unit/all_reduce_hierarchical.cpp	4	Focused test coverage for all reduce hierarchical.
libs/full/collectives/tests/unit/all_reduce.cpp	1	Focused test coverage for all reduce.
libs/full/collectives/tests/unit/barrier_hierarchical.cpp	3	Focused test coverage for barrier hierarchical.
libs/full/collectives/tests/unit/broadcast_hierarchical.cpp	4	Focused test coverage for broadcast hierarchical.
libs/full/collectives/tests/unit/gather_hierarchical.cpp	4	Focused test coverage for gather hierarchical.
libs/full/collectives/tests/unit/reduce_hierarchical.cpp	4	Focused test coverage for reduce hierarchical.
libs/full/collectives/tests/unit/scatter_hierarchical.cpp	4	Focused test coverage for scatter hierarchical.

Topology, fallback, and validation hardening

Path	Authored commits touching it	Role in the final work
libs/full/collectives/CMakeLists.txt	9	Collectives module source registration.
libs/full/collectives/include/hpx/collectives/barrier.hpp	5	Public barrier API, documentation, validation, or hierarchical dispatch.
libs/full/collectives/include/hpx/collectives/broadcast.hpp	8	Public broadcast API, documentation, validation, or hierarchical dispatch.
libs/full/collectives/include/hpx/collectives/create_communicator.hpp	16	Public create communicator API, documentation, validation, or hierarchical dispatch.
libs/full/collectives/include/hpx/collectives/detail/hierarchical_helpers.hpp	22	Shared generation mapping, topology declarations, and hierarchical run parameters.
libs/full/collectives/include/hpx/collectives/gather.hpp	15	Public gather API, documentation, validation, or hierarchical dispatch.
libs/full/collectives/include/hpx/collectives/reduce.hpp	8	Public reduce API, documentation, validation, or hierarchical dispatch.
libs/full/collectives/include/hpx/collectives/scatter.hpp	18	Public scatter API, documentation, validation, or hierarchical dispatch.
libs/full/collectives/src/barrier.cpp	2	Hierarchical barrier implementation and generation stepping.
libs/full/collectives/src/create_communicator.cpp	20	Recursive communicator-tree construction, uneven group division, and flat fallback.
libs/full/collectives/src/detail/hierarchical_helpers.cpp	2	Hierarchy validation and exact top-level group formulas.
libs/full/collectives/tests/unit/barrier.cpp	5	Focused test coverage for barrier.
libs/full/collectives/tests/unit/CMakeLists.txt	14	Registration of focused unit and regression executables.
libs/full/collectives/tests/unit/concurrent_collectives.cpp	4	Focused test coverage for concurrent collectives.
libs/full/collectives/tests/unit/hierarchical_flat_fallback.cpp	4	Focused test coverage for hierarchical flat fallback.
libs/full/collectives/tests/unit/hierarchical_helpers.cpp	19	Focused test coverage for hierarchical helpers.
libs/full/collectives/tests/unit/reduce.cpp	1	Focused test coverage for reduce.

Hierarchical all-to-all

Path	Authored commits touching it	Role in the final work
docs/hierarchical_all_to_all_design.md	1	Rejected design-note branch that records the stride-3 communicator alternative.
libs/full/collectives/include/hpx/collectives/all_to_all.hpp	35	Public all to all API, documentation, validation, or hierarchical dispatch.
libs/full/collectives/include/hpx/collectives/detail/hierarchical_all_to_all_helpers.hpp	16	All-to-all gather/exchange/scatter payload transformations.
libs/full/collectives/tests/unit/all_to_all_hierarchical.cpp	2	Focused test coverage for all to all hierarchical.
libs/full/collectives/tests/unit/all_to_all_sync.cpp	1	Focused test coverage for all to all sync.
libs/full/collectives/tests/unit/all_to_all.cpp	3	Focused test coverage for all to all.
libs/full/collectives/tests/unit/subtree_gather_scatter.cpp	8	Focused test coverage for subtree gather scatter.

Shared generation protocol

Path	Authored commits touching it	Role in the final work
libs/full/collectives/tests/unit/cross_collective_hierarchical_mixed.cpp	7	Focused test coverage for cross collective hierarchical mixed.
libs/full/collectives/tests/unit/cross_collective_hierarchical.cpp	5	Focused test coverage for cross collective hierarchical.

Hierarchical scans

Path	Authored commits touching it	Role in the final work
libs/full/collectives/include/hpx/collectives/detail/hierarchical_scan_helpers.hpp	4	Gather/prefix/scatter composition shared by inclusive and exclusive scans.
libs/full/collectives/include/hpx/collectives/exclusive_scan.hpp	5	Public exclusive scan API, documentation, validation, or hierarchical dispatch.
libs/full/collectives/include/hpx/collectives/inclusive_scan.hpp	5	Public inclusive scan API, documentation, validation, or hierarchical dispatch.
libs/full/collectives/tests/unit/exclusive_scan_.cpp	3	Focused test coverage for exclusive scan .
libs/full/collectives/tests/unit/exclusive_scan_hierarchical.cpp	2	Focused test coverage for exclusive scan hierarchical.
libs/full/collectives/tests/unit/exclusive_scan_sync.cpp	2	Focused test coverage for exclusive scan sync.
libs/full/collectives/tests/unit/inclusive_scan_.cpp	2	Focused test coverage for inclusive scan .
libs/full/collectives/tests/unit/inclusive_scan_hierarchical.cpp	4	Focused test coverage for inclusive scan hierarchical.
libs/full/collectives/tests/unit/inclusive_scan_sync.cpp	2	Focused test coverage for inclusive scan sync.

Contiguous and ragged payloads

Path	Authored commits touching it	Role in the final work
libs/full/collectives/include/hpx/collectives/detail/flattened_data.hpp	12	Uniform and ragged contiguous payload carriers with checked size arithmetic.
libs/full/collectives/tests/unit/flattened_collective_payloads.cpp	7	Focused test coverage for flattened collective payloads.

Ownership and failure convergence

Path	Authored commits touching it	Role in the final work
libs/full/collectives/include/hpx/collectives/detail/communicator.hpp	15	Communicator server state, gate stepping, generation mode, owned name, and first-exception cache.
libs/full/collectives/tests/unit/all_reduce_move_seed.cpp	1	Focused test coverage for all reduce move seed.
libs/full/collectives/tests/unit/collectives_throwing_op.cpp	3	Focused test coverage for collectives throwing op.

MPI shutdown reliability

Path	Authored commits touching it	Role in the final work
libs/core/mpi_base/src/mpi_environment.cpp	2	MPI initialization/finalization ordering and duplicated communicator lifetime.
libs/full/parcelport_mpi/include/hpx/parcelport_mpi/receiver.hpp	2	MPI parcel receiver cancellation and shutdown state.
libs/full/parcelport_mpi/src/parcelport_mpi.cpp	1	Parcelport stop ordering before MPI finalization.

Departed-locality endpoint reliability

Path	Authored commits touching it	Role in the final work
libs/full/agas/tests/regressions/departed_locality_7384.cpp	6	Multi-locality endpoint regression, port probe, child environment, and launch command.

Adjacent regressions and exploratory detours

Path	Authored commits touching it	Role in the final work
libs/core/execution/include/hpx/execution/algorithms/run_loop.hpp	2	Execution run-loop detour; useful evidence about keeping unrelated changes out of a collective patch.
libs/full/actions/tests/regressions/non_default_constructible_argument_5998.cpp	2	Actions regression for serializing an argument without a default constructor.