Submitting to Benchmark

This is a simple guide on how to prepare your model for submission to our benchmarking system SEANavBench. We have the following requirements for any submission:

  • It must be a zipped file that contains a single folder in the root of the zip. This single folder can contain one or more ROS packages.
  • The submission must contain a submission.launch file inside the launch folder of a package inside the zip, the benchmark server will run roslaunch $package_name/launch/submission.launch for the first submission.launch file found in the extracted folder.
  • If dependencies need to be installed, a setup.sh can be included in top-level folder. The benchmark server will look for and run setup.sh, if it exists, to install any required system-level or pip3 packages. Note that the sudo command can be used inside setup.sh, if required by a given command.

Turning Controller in Python into a ROS Package

Previously, in the tutorial on implementing your own controller, we have created own tmux configuration file and our own model in a python script. Here we continue with the example and turn it into a ROS package ready for submission. Here is the example code.

First, create a new ROS package. A detailed tutorial can be found here. Below, we give the basic commands we used to create our package:

cd ~/sim_ws/src
catkin_create_pkg example_submission std_msgs rospy roscpp

You should have generated an empty package. As you can see, in example_submission/package.xml, we have the template for package information and dependencies. Now, copy the tutorial_controller.py script (in sim_ws/src/social_sim_ros/social_sim_ros/src) into the example_submission/src folder, make sure it is executable.

chmod +x ~/sim_ws/src/example_submission/src/tutorial_controller.py

Create folder example_submission/launch to place the launch files, and create submission.launch inside the launch folder. We want to use this launch file to run tutorial_controller.py. Here is a tutorial on how to write a launch file. For our simple model, we only need to launch one node, so this suffices:

<launch>
  <node pkg="example_submission" name="move_base" type="tutorial_controller.py" output="screen" />
</launch>

To test this package in sim_ws, you need to first rebuild sim_ws and source the setup file:

cd ~/sim_ws
catkin_make
. ~/sim_ws/devel/setup.bash

Then, go to ~/sim_ws/tmux/sean_tutorial/.tmuxinator.yml (note the dot in .tmuxinator.yml), comment out line 14 and uncomment line 16. Namely, replace rosrun social_sim_ros tutorial_controller.py with

roslaunch --wait example_submission submission.launch

Then, you can run tmuxinator again as we did in the previous tutorials. This should run the model.

Finally, compress the ROS package into a zip file:

zip -r submission.zip example_submission

Your package is now ready for submission. Please upload the zip to the benchmark website: https://benchmark.interactive-machines.com/submissions/new

Another Example

In the example_submission folder, we also included the default ROS Navstack. This is a more complex model that uses ROS Parameter Server and ROS Console Config, and you can look at the launch file navstack.launch to understand how to load in the logging config and parameter files. If you want to submit the ROS Navstack for benchmarking, just rename navstack.launch to submission.launch and the benchmarking system will run this model.