OpenSimulationInterface
Scenario generator
Reference to a related issue in the repository
This PR is a first draft of the scenario generator to benchmark the performance of OSI as mentioned here (https://github.com/OpenSimulationInterface/open-simulation-interface/issues/428).
Add a description
You can generator variable amount of moving object with variable amount of timestamps/messages.
Further TODOs for this PR
- [x] support unlimited moving object generation
- [x] support unlimited stationary object generation
- [ ] support unlimited pedestrain generation
- [ ] basic model to transform into SD
- [ ] add test for each functionality
- [ ] modular refactoring
- [ ] add pipeline into current CI as benchmark
Check the checklist
- [ ] I have made corresponding changes to the documentation.
- [x] My code and comments follow the style guidelines and contributors guidelines of this project.
- [x] I have performed a self-review of my own code.
- [x] My changes generate no new warnings.
- [ ] I have added tests that prove my fix is effective or that my feature works.
- [ ] New and existing unit tests / travis ci pass locally with my changes.
Added stationary objects and some more dynamics which can be visualized in osi-visualizer:
Looking like a very good start to me, will form the basis of our further performance work going forward.
During the Q4-meeting I became attention to this pull request. I have done a similar approach generating random objects for performance testing some time ago.
My intention was a bit different, as I have compared different programming languages and implementations of protobuf, butthere was no storing of generated scenarios and no sequential movement of the objects. I compared the native C++ and Python implementation and the third-party C implementation nanopb, which has no dependencies to external libraries. The original German publication (Aufbau einer Mixed-Reality-Versuchsumgebung zur Absicherung autonomer Systeme) is available at Springer (paywall), but the presentation is public available. The similar talk (Design of a Mixed Reality Test Environment to Validate Autonomous Vehicles) was also presented at the Webinar / Crash Course -- ASAM OSI.
If there are interests ofthe performance tests, I can publish the code in the next weeks.
@DerBaertige thanks for the info. I am quite interested in the code for the performance test of the creation of the data structures. I think it could be added as a seperate unit test into the CI then. Feel free to link it here from your repository. Ofc I would add attribution in the comments if I use any of the code in this PR.
I added the C++ and Python implementation here, as it was not in a public repository yet. The nanopb implementation has to be rewritten in parts, as I do not have the ownership of parts of the code. If I find time, I will submit it later.
I modified the performance program to make a flatbuffers version. The flatbuffers version of OSI was created based on # 427.
Environment: Ubuntu 20.04 Open Simulation Interface 3.3.1 google Flatbuffers 2.0.0
// \file performance.cpp // \brief Simple performance evaluation example // // Simple test application, which measures the execution time of different // parts of the open simulation interface and the protobuf implementation. // // The regions of interests are the filling of the structure of a // OSI GroundTruth message, the serialization and the parsing process using // the native C++ interfaces of protobuf. // // \author Georg Seifert [email protected] // \version 0.1 // \date 2021 // // The regions of interests are the filling of the structure of a // OSI GroundTruth message, the serialization and the parsing process using // the native C++ interfaces of flatbuf. // // \author M.Kohda [email protected] // \date 2021 // // This Source Code Form is subject to the terms of the Mozilla // Public License v. 2.0. If a copy of the MPL was not distributed // with this file, you can obtain one at http://mozilla.org/MPL/2.0/ //
#include <random> #include <ctime> #include <iostream> #include <limits> #include <unistd.h> #include "flatbuffers/flatbuffers.h" #include <osi_groundtruth_generated.h> #include <osi_object_generated.h> #include "clock.h"
// \def ITERATIONS // Default number of iterations of the benchmark // #define ITERATIONS (1000)
// \def OBJECTS // Default number of moving objects used to fill the structure // #define OBJECTS (10)
int main(int argc, char** argv) { uint64_t start, stop; int c; size_t i, num_object, num_object_max = OBJECTS, num_iterations_max = ITERATIONS; size_t *ts_ser; size_t *ts_deser; size_t *ts_gen;
// FlatBuffer allocated buffer
flatbuffers::FlatBufferBuilder flatbuf;
std::mt19937 generator(time(0));
std::uniform_real_distribution<> distribution(0.0, 1.0);
while ((c = getopt(argc, argv, "o:i:h")) != -1) {
switch (c) {
case 'o':
num_object_max = atoi(optarg);
break;
case 'i':
num_iterations_max = atoi(optarg);
break;
case '?':
case 'h':
std::cout << "Argumente:" << std::endl <<
"\t-o <num>\tCount of the moving objects, default = " << OBJECTS << std::endl <<
"\t-i <num>\tIteration of the benchmark, default = " << ITERATIONS << std::endl;
return 1;
default:
break;
}
}
std::cout << "Iterationen: " << num_iterations_max <<
", Moving Objects: " << num_object_max << std::endl;
ts_ser = new size_t[num_iterations_max];
ts_deser = new size_t[num_iterations_max];
ts_gen = new size_t[num_iterations_max];
for(i = 0; i < num_iterations_max; i++)
{
const osi3::GroundTruth* output;
flatbuffers::Offset<osi3::GroundTruth> gt_ser;
flatbuffers::Offset<flatbuffers::Vector<flatbuffers::Offset<osi3::MovingObject>>> moving_object;
start = clock_get_ns();
for(num_object = 0; num_object < num_object_max; num_object++)
{
auto moving_type = osi3::MovingObject_::Type::TYPE_VEHICLE;
auto strategy_type = osi3::StationaryObject_::Classification_::Type::TYPE_OTHER;
auto classification_type = osi3::MovingObject_::VehicleClassification_::Type::TYPE_SMALL_CAR;
auto vehicle_classification = osi3::MovingObject_::CreateVehicleClassification(flatbuf, moving_type, 0, (bool)false);
// osi3::Identifier *
flatbuffers::Offset<osi3::Identifier> id = osi3::CreateIdentifier(flatbuf, num_object);
// length = 12.0m (StVZO 32, 3,1)
// width = 2.55 (StVZO 32, 3.1)
// height = 4.0 (STVZO 32, 2),
// osi3::Dimension3d*
flatbuffers::Offset<osi3::Dimension3d> dimension = osi3::CreateDimension3d(flatbuf,
/* length */
(12.0 * distribution(generator)),
/* width */
(2.55 * distribution(generator)),
/* height */
(4.0 * distribution(generator))
);
// position: x, h, z (-512 -- 512)
// osi3::Vector3d*
flatbuffers::Offset<osi3::Vector3d> position = osi3::CreateVector3d(flatbuf,
/* x */
(1024 * distribution(generator) - 512),
/* y */
(1024 * distribution(generator) - 512),
/* z */
(1024 * distribution(generator) - 512)
);
// roll, pitch, yaw (0 -- 2pi)
// osi3::Orientation3d*
flatbuffers::Offset<osi3::Orientation3d> orientation = osi3::CreateOrientation3d(flatbuf,
/* roll */
(6.28 * distribution(generator)),
/* pitch */
(6.28 * distribution(generator)),
/* yaw */
(6.28 * distribution(generator))
);
// velocity (-30 -- 100)
// osi3::Vector3d*
flatbuffers::Offset<osi3::Vector3d> velocity = osi3::CreateVector3d(flatbuf,
/* x */
(130.0 * distribution(generator) - 20.0),
/* y */
(130.0 * distribution(generator) - 20.0),
/* z */
(130.0 * distribution(generator) - 20.0)
);
// acceleration (-12 -- 10), based on Pkw-Pkw-Unfalle
// https://doi.org/10.1007/978-3-8348-9974-3_12
// osi3::Vector3d*
flatbuffers::Offset<osi3::Vector3d> acceleration = osi3::CreateVector3d(flatbuf,
/* x */
(22.0 * distribution(generator) - 12),
/* y */
(22.0 * distribution(generator) - 12),
/* z */
(22.0 * distribution(generator) - 12)
);
// osi3::BaseMoving
auto base = osi3::CreateBaseMoving(flatbuf, dimension, position, orientation, velocity, acceleration, 0, 0, 0);
// osi3::MovingObject
auto tmp_moving_object = osi3::CreateMovingObject(flatbuf, id, base, strategy_type, 0, 0, vehicle_classification, 0, 0, 0);
moving_object = (flatbuffers::Offset<flatbuffers::Vector<flatbuffers::Offset<osi3::MovingObject>>> &) tmp_moving_object;
}
// Make GroundTruth
gt_ser = osi3::CreateGroundTruth(flatbuf, 0, 0, 0, 0, moving_object, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
stop = clock_get_ns();
ts_gen[i] = stop - start;
start = clock_get_ns();
osi3::FinishGroundTruthBuffer(flatbuf, gt_ser);
stop = clock_get_ns();
ts_ser[i] = stop - start;
start = clock_get_ns();
uint8_t *buf = flatbuf.GetBufferPointer();
output = osi3::GetGroundTruth(buf);
stop = clock_get_ns();
ts_deser[i] = stop - start;
}
double min = std::numeric_limits<double>::max(), max = 0, mean = 0, variance = 0, stddev = 0;
for(i = 0; i < num_iterations_max; i++)
{
min = (min > ts_gen[i]) ? ts_gen[i] : min;
max = (max < ts_gen[i]) ? ts_gen[i] : max;
mean += ts_gen[i];
variance += ts_gen[i]*ts_gen[i];
}
mean /= num_iterations_max;
variance = variance/num_iterations_max - mean*mean;
stddev = sqrt( variance );
std::cout << "Fill:" << std::endl <<
"\tmin = " << min << " ns" << std::endl <<
"\tmax = " << max << " ns" << std::endl <<
"\tmean = " << mean << " ns" << std::endl <<
"\tstandard deviation = " << stddev << " ns" << std::endl;
min = std::numeric_limits<double>::max(), max = 0, mean=0, variance=0, stddev = 0;
for(i=0; i < num_iterations_max; i++)
{
min = (min > ts_ser[i]) ? ts_ser[i] : min;
max = (max < ts_ser[i]) ? ts_ser[i] : max;
mean += ts_ser[i];
variance += ts_ser[i]*ts_ser[i];
}
mean /= num_iterations_max;
variance= variance/num_iterations_max - mean*mean;
stddev = sqrt( variance );
std::cout << "Serialize:" << std::endl <<
"\tmin = " << min << " ns" << std::endl <<
"\tmax = " << max << " ns" << std::endl <<
"\tmean = " << mean << " ns" << std::endl <<
"\tstandard deviation = " << stddev << " ns" << std::endl;
min = std::numeric_limits<double>::max(), max = 0, mean = 0, variance = 0, stddev = 0;
for(i = 0; i < num_iterations_max; i++)
{
min = (min > ts_deser[i]) ? ts_deser[i] : min;
max = (max < ts_deser[i]) ? ts_deser[i] : max;
mean += ts_deser[i];
variance += ts_deser[i]*ts_deser[i];
}
mean /= num_iterations_max;
variance = variance/num_iterations_max - mean*mean;
stddev = sqrt( variance );
std::cout << "Parse:" << std::endl <<
"\tmin = " << min << " ns" << std::endl <<
"\tmax = " << max << " ns" << std::endl <<
"\tmean = " << mean << " ns" << std::endl <<
"\tstandard deviation = " << stddev << " ns" << std::endl;
delete [] ts_ser;
delete [] ts_deser;
delete [] ts_gen;
return 0;
}
