Numbering/counting entities

[a-jp]

I have a simple requirement to find out the total number of vertices and elements in the mesh in parallel. I could make each processor count over the number of cells on that processor and bake in an mpi gather/reduce but it feels this should be done using the numbering API. So how would I find the total number of vertices and total number of cells in the mesh in parallel via the apf API?

I would imagine the suggested approach would generalise to edges/faces also if I wanted to globally count those? I would also like to exclude counting ghost entities if they have been generated.

Thanks, Andy

[jacobmerson]

This is the function to count an entity of a given dimension on the current part. I'm not sure if there is a global version of this.

https://github.com/SCOREC/core/blob/b48c0df2347c07d0072b318c836a9f523036dfd6/apf/apfMesh.h#L571-L573

[cwsmith]

In the PUMI api (since ghosting was mentioned), the following functions count then return global owned ents:

count: https://github.com/SCOREC/core/blob/14d26129e5054a8fef9b6983753bf30e46309906/pumi/pumi.h#L234

query: https://github.com/SCOREC/core/blob/14d26129e5054a8fef9b6983753bf30e46309906/pumi/pumi.h#L236

[a-jp]

If I call the following code on a 2D hex mesh with 4 hex cells, on 4 processors, I find that the vertices are globally numbered, sequentially from processor 0->4, as are the cells.

Is sequential numbering monotonically increasing with processor number always guaranteed? Is this still the case after mesh renumbering and after refinement etc?

void ShowNumbering(apf::Mesh *m)
{
  {
    apf::GlobalNumbering *gn = nullptr;
    gn = apf::makeGlobal(apf::numberOwnedNodes(m, "node-nums"));
    apf::synchronize(gn);

    for (int i = 0; i < PCU_Comm_Peers(); ++i)
    {
      if (i == PCU_Comm_Self())
      {
        apf::MeshIterator *vertIter = m->begin(0);
        apf::MeshEntity *vert = nullptr;
        while ((vert = m->iterate(vertIter)))
        {
          if (m->isOwned(vert) && m->getOwner(vert) == i)
          {
            const auto n = apf::getNumber(gn, vert, 0);
            std::cout << "Rank [vertex]: " << i << " " << n << " " << m->isOwned(vert) << " " << m->getOwner(vert) << std::endl;
          }
        }
        m->end(vertIter);

        destroyGlobalNumbering(gn);
      }
      PCU_Barrier();
    }
  }
  std::cout << std::endl;
  PCU_Barrier();
  {
    apf::GlobalNumbering *gn = nullptr;
    gn = apf::makeGlobal(apf::numberElements(m, "element-nums"));

    for (int i = 0; i < PCU_Comm_Peers(); ++i)
    {
      if (i == PCU_Comm_Self())
      {
        apf::MeshIterator *cellIter = m->begin(m->getDimension());
        apf::MeshEntity *cell = nullptr;
        while ((cell = m->iterate(cellIter)))
        {
          if (m->isOwned(cell) && m->getOwner(cell) == i)
          {
            const auto n = apf::getNumber(gn, cell, 0);
            std::cout << "Rank [element]: " << i << " " << n << " " << m->isOwned(cell) << " " << m->getOwner(cell) << std::endl;
          }
        }
        m->end(cellIter);

        destroyGlobalNumbering(gn);
      }
      PCU_Barrier();
    }
  }
}

The result of that function call on a 4 processor run for 4 hex cells is:

Rank [vertex]: 0 0 1 0
Rank [vertex]: 0 1 1 0
Rank [vertex]: 0 2 1 0
Rank [vertex]: 0 3 1 0
Rank [vertex]: 1 4 1 1
Rank [vertex]: 1 5 1 1
Rank [vertex]: 2 6 1 2
Rank [vertex]: 2 7 1 2
Rank [vertex]: 3 8 1 3

Rank [element]: 0 0 1 0
Rank [element]: 1 1 1 1
Rank [element]: 2 2 1 2
Rank [element]: 3 3 1 3

Can I rely on this always being the case? Thanks, Andy

[cwsmith]

Yes. A prefix sum (exscan(...) on line 560 below) is used to globalize the numbering.

https://github.com/SCOREC/core/blob/b48c0df2347c07d0072b318c836a9f523036dfd6/apf/apfNumbering.cc#L535-L569