NOTE: this software is part of the BenchBot software stack, and not intended to be run in isolation (although it can be installed independently through pip if desired). For a working BenchBot system, please install the BenchBot software stack by following the instructions here.
The BenchBot Add-ons Manager allows you to use BenchBot with a wide array of additional content, and customise your installation to suite your needs. Semantic Scene Understanding not your thing? Install the Semantic Question Answering add-ons instead. Want to create your own content? Write some basic YAML files to make your own add-ons. Need to re-use existing content? Simply include a dependency on that add-on. Add-ons are all about making BenchBot whatever you need it to be—build a BenchBot for your research problems, exactly as you need it.
Add-ons come in a variety of types. Anything that you may need to customise for your own experiments or research, should be customisable through an add-on. If not, let us know, and we'll add more add-on enabled functionality to BenchBot!
The list of currently supported types of add-ons are:
benchbot_batch
script.See the sections below for details of how to interact with installed add-ons, how to create your own add-ons, and formalisation of what's required in an add-on.
In general, you won't use the add-ons manager directly. Instead you interact with the BenchBot software stack, which uses the add-ons manager to manage and access add-ons.
The manager is a Python package if you do find you want to use it directly, and installable with pip. Run the following in the root directory where the repository was cloned:
u@pc:~$ pip install .
The manager can then be imported and used to manage installation, loading, accessing, processing, and updating of add-ons. Some samples of supported functionality are shown below:
from benchbot_addons import manager as bam
# Check if example with 'name' = 'hello_scd' exists
bam.exists('examples', [('name', 'hello_scd')])
# Find all installed environments
bam.find_all('environments')
# Get a list of the names for all installed tasks
bam.get_field('tasks', 'name')
# Get a list of (name, variant) pairs for all installed environments
bam.get_fields('environments', ['name', 'variant'])
# Find a robot with 'name' = 'carter'
bam.get_match('robots', [('name', 'carter')])
# Get the 'results_format' value for the task called 'scd:passive:ground_truth'
bam.get_value_by_name('tasks', 'scd:passive:ground_truth', 'results_format')
# Load YAML data for all installed ground truths
bam.load_yaml_list(bam.find_all('ground_truths', extension='json'))
# Install a list of comma-separated add-ons
bam.install_addons('benchbot-addons/ssu,benchbot-addons/sqa')
# Install a specific add-on (& it's dependencies)
bam.install_addon('tasks_ssu')
# Print the list of currently installed add-ons, & officially available add-ons
bam.print_state()
# Uninstall all add-ons
bam.remove_addons()
# Uninstall a string separated list of add-ons
bam.remove_addon('benchbot-addons/ssu,benchbot-addons/sqa')
Add-ons are designed to make it easy to add your own local content to a BenchBot installation. You can add your own local content to the "local add-ons" folder provided with your install. The location on your machine can be printed via the following:
from benchbot_addons import manager as bam
print(bam.local_addon_path())
BenchBot expects add-on content to be in named folders denoting the type of content. For example, robots must be in a folder called 'robots'
, tasks in a folder called 'tasks'
, and so on. A list of valid content types is available via the SUPPORTED_TYPES
field in the add-ons manager.
Below is an example of the process you would go through to create your own custom task locally:
u@pc:~$ python3 -c 'from benchbot_addons import manager as bam; print(bam.local_addon_path())'
/home/ben/repos/benchbot/addons/benchbot_addons/.local/my_addons
/home/ben/repos/benchbot/addons/benchbot_addons/.local/my_addons/tasks/my_task.yaml
Done. Your new custom task should now be available for use in your BenchBot system (e.g. benchbot_run --list-tasks
).
Custom add-on content can be grouped together into an add-on package, of which there are two different types: 'official' and third-party.
'Official' packages are those we've verified, and are stored in our benchbot-addons GitHub organisation. You can get a full list of official add-on packages through the manager.official_addons()
helper function, or benchbot_install --list-addons
script in the BenchBot software stack.
Third-party add-on packages differ only in that we haven't looked at them, and they can be hosted anywhere on GitHub you please.
Creating all add-on packages is exactly the same process, the only difference is whether the repository is inside or outside of the benchbot-addons GitHub organisation:
environments
directory at the root)..remote
metadata file described in the next section).dependencies*
filesNote: it's a good idea to only include one type of add-on per repository as it makes your add-on package more usable for others. It's not a hard rule though, so feel free to add multiple folders to your add-on if you require.
Feel free to have a look at any of the official add-ons for help and examples of how to work with add-ons.
Here are the technical details of what's expected in add-on content. The BenchBot system will assume these specifications are adhered to, and errors can be expected if you try to use add-ons that don't match the specifications.
An add-on package has the following structure (technically none of the files are required, they just determine what functionality your add-on includes):
Filename | Description |
---|---|
.dependencies |
A list of add-on packages that must be installed with this package. Packages are specified by their GitHub identifier (i.e. github_username/repository_name ), with one per line |
.dependencies-python |
A list of Python dependencies for your add-on. Syntax for file is exactly the same as requirements.txt files. |
.remote |
Specifies content that should be installed from a remote URL, rather than residing in this repository. A remote resource is specified as a URL and target directory separated by a space. One resource is specified per line. The add-ons manager will fetch the URL specified, and extract the contents to the target directory (e.g. http://myhost/my_content.zip environments ) |
<directory>/ |
Each named directory corresponds to an add-on type described below. The directory will be ignored if its name doesn't exactly match any of those below. |
A YAML file, that must exist in a folder called batches
in the root of the add-on package (e.g. batches/my_batch.yaml
).
The following keys are supported for batch add-ons:
Key | Required | Description |
---|---|---|
'name' |
Yes | A string used to refer to this batch (must be unique!). |
'environments' |
Yes | A list of environment strings of the format 'name':'variant' (e.g. 'miniroom:1' ). |
A YAML file, that must exist in a folder called environments
in the root of the add-on package (e.g. environments/my_environment.yaml
).
The following keys are supported for environment add-ons:
Key | Required | Description |
---|---|---|
'name' |
Yes | A string used to refer to this environment's name (the ('name', 'variant') pair must be unique!). |
'variant' |
Yes | A string used to refer to this environment's variant (the ('name', 'variant') pair must be unique!). |
'type' |
Yes | A string describing the type of this environment ('sim_unreal' & 'real' are the only values currently used). |
'map_path' |
Yes | A path to the map for this environment, which will be used by either the simulator or real world system to load the environment. |
'start_pose' |
Yes | The start pose of the robot that will be provided to users through the BenchBot API. The pose is specified as a list of 7 numbers: quarternion_w, quarternion_x, quarternion_y, quarternion_z, position_x, position_y, position_z. This must be accurate! |
'trajectory_poses' |
No | A list of poses for the robot to traverse through in order. Each pose is a list of 7 numbers: quarternion_w, quarternion_x, quarternion_y, quarternion_z, position_x, position_y, position_z. This environment won't be usable for tasks that use the 'move_next' action if this parameter isn't provided. |
'robots' |
No | A list of supported names for robot that are supported in this environment. If this list isn't included, all robots with the same 'type' as this environment will be able to run. |
'object_labels' |
No | A list of labels for the objects that exist in the scene. Can be used with simulated sensors like segmentation sensors. |
A YAML file, that must exist in a folder called evaluation_methods
in the root of the add-on package (e.g. evaluation_methods/my_evaluation_method.yaml
).
The following keys are supported for evaluation method add-ons:
Key | Required | Description |
---|---|---|
'name' |
Yes | A string used to refer to this evaluation method (must be unique!) |
'valid_result_formats' |
Yes | List of strings denoting results formats supported by the evaluation method. Ideally these format definitions should also be installed. |
'valid_ground_truth_formats' |
Yes | List of strings denoting ground truth formats supported by the evaluation method. Ideally these format definitions should also be installed. |
'functions' |
Yes | Dictionary of named functions provided by the evaluation method. The named methods are key value pairs where the key is the function name, and the value is a string describing how the function can be imported with Python. For example, evaluate: "omq.evaluate_method" declares a function called 'evaluate' that is imported via from omq import evaluate_method . Likewise "omq.submodule.combine_method" translates to from omq.submodule import combine_method . See below for the list of functions expected for evaluation methods. |
'description' |
No | A string describing what the evaluation method is and how it works. Should be included if you want users to understand where your method can be used. |
Evaluation methods expect the following named functions:
Name | Signature | Usage |
---|---|---|
'evaluate' |
fn(dict: results, list: ground_truths) -> dict |
Evaluates the performance using a results dictionary, and returns a dictionary of containing the scores. It also takes a list of dictionaries containing each ground truth that will be used in evaluation. |
'combine' |
fn(list: scores) -> dict |
Takes a list of scores dictionaries, and returns an aggregate score. If this method isn't declared, benchbot_eval won't return a summary score. |
A YAML file, that must in a folder called examples
in the root of the add-on package (e.g. examples/my_example.yaml
).
The following keys are supported for example add-ons:
Key | Required | Description |
---|---|---|
name |
Yes | A string used to refer to this example (must be unique!) |
native_command |
Yes | A string describing the command used to run your example natively, relative to the directory of this YAML file! For example running your my_example.py file which is in the same director as this YAML would be python3 ./my_example.py . |
container_directory |
No | Directory to be used for Docker's build context. The submission process will automatically look for a file called Dockerfile in that directory unless the 'container_filename' key is also provided. |
container_filename |
No | Custom filename for your example's Dockerfile. Dockerfile in container_directory will be used if this key is not included. This path is relative to this YAML file, not 'container_directory' . |
description |
No | A string describing what the example is and how it works. Should be included if you want users to understand how your example can be expanded. |
A YAML file, that must exist in a folder called formats
in the root of the add-on package (e.g. formats/my_format.yaml
).
The following keys are supported for format add-ons:
Key | Required | Description |
---|---|---|
'name' |
Yes | A string used to refer to this format (must be unique!) |
'functions' |
Yes | Dictionary of named functions for use with this format. The named methods are key-value pairs where the key is the function name, and the value is a string describing how the function can be imported with Python. For example, create: "object_map.create_empty" declares a function called 'create' that is imported via from object_map import create_empty . Likewise "object_map.submodule.validate" translates to from object_map.submodule import validate . See below for the list of functions expected for format definitions. |
'description' |
No | A string describing what the format is and how it works. Should be included if you want users to understand what your format is supposed to capture. |
Format definitions expect the following named functions:
Name | Signature | Usage |
---|---|---|
'create' |
fn() -> dict |
Function that returns an empty instance of this format. As much as possible should be filled in to make it easy for users to create valid instances (especially when a format is used for results). |
'validate' |
fn(dict: instance) -> None |
Takes a proposed instance of this format and validates whether it meets the requirements. Will typically use a series of assert statements to confirm fields are valid. |
A JSON file, that must exist in a folder called ground_truths
in the root of the add-on package (e.g. ground_truths/my_ground_truth.json
).
The following keys are supported for ground truth add-ons:
Key | Required | Description |
---|---|---|
'environment' |
Yes | A dictionary containing the definition data for the ground truth's reference environment. The data in this field should be a direct copy of an environment add-on. |
'format' |
Yes | A dictionary containing the definition data for the ground truth's format. The data in this field should be a direct copy of a format definition add-on. |
'ground_truth' |
Yes | A valid instance of the format described by the 'format' field. This is where your actual ground truth data should be stored. |
A lot of these keys should be copied from other valid definitions. Please see the GroundTruthCreator
helper class in BenchBot Evaluation for assistance in creating valid ground truths.
A YAML file, that must exist in a folder called robots
in the root of the add-on package (e.g. robots/my_robot.yaml
).
The following keys are supported for robot add-ons:
Key | Required | Description |
---|---|---|
'name' |
Yes | A string used to refer to this robot (must be unique!). |
'type' |
Yes | A string describing the type of this robot ('sim_unreal' & 'real' are the only values currently used). |
'address' |
Yes | A string for the address where a running BenchBot Robot Controller can be accessed (e.g. 'localhost:10000' ) |
'global_frame' |
Yes | The name of the global TF frame. All poses reported by the BenchBot API will be with respect to this frame. |
'robot_frame' |
Yes | The name of the robot's TF frame. |
'poses' |
Yes | A list of named poses that this robot provides. This list of poses will be available in observations provided by the BenchBot API. |
persistent_cmds |
Yes | A list of commands that will be run and kept alive for the lifetime of the robot controller. The commands will be run in parallel, and executed via bash -c <your_command_string> |
persistent_status |
Yes | A command used to check the status of your persistent_cmds . This command should execute quickly, and terminate on completion, with the return code being used to evaluate the status. The command string is executed via bash -c <your_command_string> |
run_cmd |
Yes | A single command issued by the controller to run a simulation. This command must terminate on completion. The command string is executed via bash -c <your_command_string> |
stop_cmd |
Yes | A single command issued by the controller to stop a simulation. This command must terminate on completion. The command string is executed via bash -c <your_command_string> |
'connections' |
Yes | A dictionary of connections that your robot makes available to the BenchBot ecosystem. The name of the key-value pair is important, and should follow the recommendations provided on standard channels in the BenchBot API documentation. A description of connection definitions is provided below. |
Connections are the lifeblood of interaction between BenchBot and robot platforms. They are defined by named entries, with the following fields:
Key | Required | Description |
---|---|---|
'connection' |
Yes | Connection type string, used by the BenchBot Robot Controller. Supported values are 'api_to_ros' (used for actions), 'ros_to_api' (used for observations), and 'roscache_to_api' (special value used for caching observation values). |
'ros_topic' |
Yes | Topic name for the ROS side of the connection. |
'ros_type' |
Yes | Topic type for the ROS side of the connection. |
'callback_api' |
No | A callback that is run on the HTTP encoded data received / sent on the API end of the connection. It takes in data, and returns transformed data based on the callback's action. Callbacks are specified by a string denoting how the callback can be accessed (e.g. 'api_callbacks.convert_to_rgb = from api_callbacks import convert_to_rgb ). No data transformation occurs if no callback is provided. |
'callback_ros' |
No | A callback that is run on the ROS data received / sent on the robot controller end of the connection. It takes in data and a reference to the robot controller. 'api_to_ros' connections use this data to act on the robot, whereas 'ros_to_api' connections turn this data into a dictionary that can be serialised into HTTP traffic. Callbacks are specified by a string denoting how the callback can be accessed (e.g. 'api_callbacks.convert_to_rgb = from api_callbacks import convert_to_rgb ). No action occurs at the ROS level if no callback is provided. |
A YAML file, that must exist in a folder called tasks
in the root of the add-on package (e.g. tasks/my_task.yaml
).
The following keys are supported for task add-ons:
Key | Required | Description |
---|---|---|
'name' |
Yes | A string used to refer to this task (must be unique!). |
'actions' |
Yes | A list of named connections to be provided as actions through the BenchBot API. Running this task will fail if the robot doesn't provide these named connections. |
'observations' |
Yes | A list of named connections to be provided as observations through the BenchBot API. Running this task will fail if the robot doesn't provide these named connections. |
'localisation' |
No | A string describing the level of localisation. Only supported values currently are 'ground_truth' and 'noisy' . The default value is 'ground_truth '. |
'results_format' |
No | A string naming the format for results. The format must be installed, as BenchBot API will use the format's functions to provide the user with empty results. |
'description' |
No | A string describing what the task is, and how it works. Should be included if you want users to understand what challenges your task is trying to capture. |
'type' |
No | A string describing what robot / environment types are valid for this task. For example, a task that provides a magic image segmentation sensor would only be made available for 'sim_unreal' type robots / environments. |
'scene_count' |
No | Integer representing the number of scenes (i.e. environment variations required for a task). If omitted, a default value of 1 will be used for the task. |
CRICOS No. 00213J