Submitting to Benchmark
This is a simple guide on how to prepare your model for submission to our benchmarking system SEANavBench. We have the following requirements for any submission:
- It must be a zipped file that contains a single folder in the root of the zip. This single folder can contain one or more ROS packages.
- The submission must contain a
submission.launchfile inside the
launchfolder of a package inside the zip, the benchmark server will run
roslaunch $package_name/launch/submission.launchfor the first
submission.launchfile found in the extracted folder.
- If dependencies need to be installed, a
setup.shcan be included in top-level folder. The benchmark server will look for and run
setup.sh, if it exists, to install any required system-level or
pip3packages. Note that the
sudocommand can be used inside
setup.sh, if required by a given command.
Turning Controller in Python into a ROS Package
Previously, in the tutorial on implementing your own controller, we have created own
tmux configuration file and our own model in a python script. Here we continue with the example and turn it into a ROS package ready for submission. Here is the example code.
First, create a new ROS package. A detailed tutorial can be found here. Below, we give the basic commands we used to create our package:
catkin_create_pkg example_submission std_msgs rospy roscpp
You should have generated an empty package. As you can see, in
example_submission/package.xml, we have the template for package information and dependencies. Now, copy the
tutorial_controller.py script (in
sim_ws/src/social_sim_ros/social_sim_ros/src) into the
example_submission/src folder, make sure it is executable.
chmod +x ~/sim_ws/src/example_submission/src/tutorial_controller.py
example_submission/launch to place the launch files, and create
submission.launch inside the
launch folder. We want to use this launch file to run
tutorial_controller.py. Here is a tutorial on how to write a launch file. For our simple model, we only need to launch one node, so this suffices:
<node pkg="example_submission" name="move_base" type="tutorial_controller.py" output="screen" />
To test this package in
sim_ws, you need to first rebuild
sim_ws and source the setup file:
Then, go to
~/sim_ws/tmux/sean_tutorial/.tmuxinator.yml (note the dot in
.tmuxinator.yml), comment out line 14 and uncomment line 16. Namely, replace
rosrun social_sim_ros tutorial_controller.py with
roslaunch --wait example_submission submission.launch
Then, you can run
tmuxinator again as we did in the previous tutorials. This should run the model.
Finally, compress the ROS package into a zip file:
zip -r submission.zip example_submission
Your package is now ready for submission. Please upload the zip to the benchmark website: https://benchmark.interactive-machines.com/submissions/new
example_submission folder, we also included the default ROS Navstack. This is a more complex model that uses ROS Parameter Server and ROS Console Config, and you can look at the launch file
navstack.launch to understand how to load in the logging config and parameter files. If you want to submit the ROS Navstack for benchmarking, just rename
submission.launch and the benchmarking system will run this model.