Extend collision checking example

[simonschmeisser]

I would like to test some optimizations on the FCL collision checking and this seems like a very nice way to validate that my changes hopefully speed things up.
What I mostly want is a comparison plot of the different collision check types of FCL with MoveIt master compared to my branch. Is it easy to do that? Switching branches does not need to be automated, I can do that

I think what I mostly need help with/an example is what to put into the

      metrics:
        - type: boxplot
          names:
            - stage_solution_costs_grasp pose IK
            - stage_solution_costs_place pose IK
        

section.

[simonschmeisser]

I found #57 and made some small progress but I think I would prefer bargraphs with checks/second or similar

current graph:

current config:

gnuplot_config:
  plots:
    - #filters:
      #  - {key: /hw/cpu/model, val: 158, predicate: '='}
      filters:
        - ns: query/scene
          val: box-100-4c
          predicate: "="
      legends:
        - query/collision_detector
      labels:
        - query/request
      metrics:
        - type: boxplot
          name: time
  options:
    terminal: QT
    n_row: 1
    n_col: 1

[captain-yoshi]

What I mostly want is a comparison plot of the different collision check types of FCL with MoveIt master compared to my branch. Is it easy to do that? Switching branches does not need to be automated, I can do that

Each profiler/test has a virtual function that implements retrieval of version metadata at runtime (you just have to be careful that you build before running the test). This is not a problem when using a script when forcing to compile after each checkout. This would look like this:

    software:
      moveit_core:
        version: 1.1.0
        git_branch: HEAD
        git_commit: 3cc363372a6e52d378e6907096979ccff49cdccf
        pkg_manager: ROS Package
      moveit_benchmark_suite:
        version: 0.0.0
        git_branch: HEAD
        git_commit: 7cb2a5ae6b9017e10d21d03e71b31da1ae319bb3
        pkg_manager: ROS Package
      fcl:
        version: 0.6.1
        pkg_manager: ROS Package

You can then use the different moveit_core commits version in the legends or the X axis. The git branch name is wrong at the moment. You proposed this to fix that with this.

[captain-yoshi]

I found #57 and made some small progress but I think I would prefer bargraphs with checks/second or similar

The profilers currently only saves raw metrics, so no statistics is saved. But there are ways to achieve this at runtime (will show you the way to store statistics automatically). Just add the aggregate_config section underneath your benchmark/profiler config:

benchmark_config:
  ...
profiler_config:
  ...
  
aggregate_config:
  # filters:
  #   - {ns: /hw/cpu/model, val: 158, predicate: '='}
  statistics:
    - raw_metric: time
      new_metric: collision_check
      type: frequency

    - raw_metric: collision
      new_metric: avg_collision
      type: average

    - raw_metric: contact_count
      new_metric: avg_contact_count
      type: average

There are no documentation yet for this. The problem with this is when you have multiple queries with 10 000 trials it will slow down the yaml import and filtering process (because added to the same dataset). If you only want the statistics, you can filter out all lines that are greater then a threshold using sed (check this secret snippet). There are other usefull tools like yq to filter the dataset with more options then what I implemented with the filter options.

[captain-yoshi]

In the end you will have 2 datasets (regardless if it's in the same file or separated). Just load them with this config for plotting:

gnuplot_config:
  plots:
    - filters:
        - ns: query/scene
          val: box-100-4c
          predicate: "="
      labels:
        - query/collision_detector
      legends:
        - /software/moveit_core/version
      metrics:
        - type: bargraph
          name: collision_check
          title: "MoveIt regression: Collision check (s^{-1}) with blablabla"
        - type: bargraph
          name: avg_contact_count
          title: "MoveIt regression: Contact counts"
  options:
    terminal: QT
    n_row: 2
    n_col: 1
    # size:
    #   x: 1920
    #   y: 1080

You could also set the benchmark name in the dataset to something like default and my-fix and use the /name in the labels or the legends field to see the comparison between the 2 commits. As long as the names of the fields are different.

Forgot to mention, you MUST apply this snippet to your dataset:

sed 's/- name: \(.*\)/\1:/' dataset.yaml > output.yaml

this will change the software sequence into a map. Sequences cannot be targeted for a label or a legend. It's cleaner for the dataset to have a sequence but harder to use the way the configuration is done...