plato icon indicating copy to clipboard operation
plato copied to clipboard

plato的vid映射没有保留原始数据的局部性?

Open jievince opened this issue 4 years ago • 2 comments

https://github.com/Tencent/plato/blob/master/plato/graph/structure/vid_encoder.hpp

    spread_message<VID_T, vid_t>(
      cache,
      [&](const push_context_t& context, size_t i, edge_unit_spec_t *edge) {
        if (opts_.src_need_encode_) {
          bool upserted = used.upsert(edge->src_, [](vid_t&){}, 0);
          if (upserted) {
            auto send_to = murmur_hash2(&(edge->src_), sizeof(VID_T)) % cluster_info.partitions_;
            context.send(send_to, edge->src_);
          }
        }
        if (opts_.dst_need_encode_) {
          bool upserted = used.upsert(edge->dst_, [](vid_t&){}, 0);
          if (upserted) {
            auto send_to = murmur_hash2(&(edge->dst_), sizeof(VID_T)) % cluster_info.partitions_;
            context.send(send_to, edge->dst_);
          }
        }
      },
      [&](VID_T& msg) {
        table.upsert(msg, [](vid_t&){}, 0);
        return 0;
      }
    );

jievince avatar Apr 27 '21 09:04 jievince

是的,plato在做数据映射时并没有考虑原始数据的分布情况。

skyssj avatar Apr 28 '21 02:04 skyssj

是的,plato在做数据映射时并没有考虑原始数据的分布情况。

在丢失掉输入数据的局部性的情况下,plato的通信开销会增加很多吧,不知道相比保留输入数据的局部性的,性能会降低多少?

jievince avatar Apr 28 '21 05:04 jievince