Automatic algorithm configuration tools

Irace

Irace (Iterated Race) is an autoconfiguration tool that focuses on tuning the parameters of optimization algorithms. It employs a racing mechanism to iteratively select the best configurations among a set of candidate configurations based on their performance on a set of problem instances.

Please note that you need to have a working installation of R on your system. Then install the irace R package via the R console by running:

install.packages("remotes")
install.packages("irace", version = "3.5", repos = "https://cloud.r-project.org")

After that you need to leave the R terminal and install the Irace Python package via

pip install git+https://github.com/DimitriWeiss/up-iracepy@main

The algorithm configuration implementation will then access irace via the python package rpy2.

For further details on Irace refer to the irace GitHub and the python implementation of irace.

Functionalities for managing and calling configurators.

class up_ac.Irace_configurator.IraceConfigurator

Configurator functions.

get_feedback_function(gaci, engine, metric, mode, gray_box=False)

Generate a function to run the planning engine and obtain feedback.

Parameters:

  • gaci (object) – AC interface object.

  • engine (str) – Name of the planning engine.

  • metric (str) – Metric type, either ‘runtime’ or ‘quality’.

  • mode (str) – Type of planning mode.

  • gray_box (bool, optional) – True if using a gray box approach (optional).

Returns:

A function to provide feedback based on the specified parameters.

Return type:

function

Raises:

ValueError – If the provided engine, metric, or mode is not supported.

optimize(feedback_function=None, gray_box=False)

Run the algorithm configuration process.

Parameters:

  • feedback_function (function, optional) – A function to run the engine and obtain feedback (optional).

  • gray_box (bool, optional) – True if using a gray box approach (optional).

Returns:

A tuple containing: - dict: The best configuration found. - None: If there is no feedback function.

Return type:

tuple or None

set_scenario(engine, param_space, gaci, configuration_time=120, n_trials=400, min_budget=1, max_budget=3, crash_cost=0, planner_timelimit=30, n_workers=1, instances=[], instance_features=None, metric='runtime', patience=10)

Set up the algorithm configuration scenario.

Parameters:

  • engine (str) – Name of the engine.

  • param_space (ConfigSpace.ConfigurationSpace) – The ConfigSpace object defining the parameter space.

  • gaci (object) – AC interface object.

  • configuration_time (int, optional) – Overall configuration time budget (optional).

  • n_trials (int, optional) – Maximum number of engine evaluations (optional).

  • min_budget (int, optional) – Minimum number of instances to use (optional).

  • max_budget (int, optional) – Maximum number of instances to use (optional).

  • crash_cost (int, optional) – The cost to use if the engine fails (optional).

  • planner_timelimit (int, optional) – Maximum runtime per evaluation (optional).

  • n_workers (int, optional) – Number of cores to utilize (optional).

  • instances (list, optional) – List of problem instance paths (optional).

  • instance_features (dict, optional) – Dictionary containing instance names and lists of features (optional).

  • metric (str, optional) – Optimization metric, either ‘runtime’ or ‘quality’ (optional).

  • patience (int) – Extra time in seconds to grant engine to terminate gracefully.

Raises:

ValueError – If the provided metric is not supported.

Irace algorithm configuration interface for unified planning.

class up_ac.Irace_interface.IraceInterface

Irace AC interface.

get_ps_irace(param_space)

Retrieve parameter space information for configuring irace.

Parameters:

param_space (ConfigSpace.ConfigurationSpace) – The ConfigSpace object defining the parameter space.

Returns:

A tuple containing: - dict: Default values for parameters. - bool: Indicates if there are forbidden parameter value combinations.

Return type:

tuple

transform_conf_from_ac(engine, configuration, plantype='OneshotPlanner')

Transform configuration from algorithm configuration format to engine format.

This function takes a configuration in the algorithm configuration format, specific to the provided engine, and transforms it into the corresponding format for the given engine.

Parameters:

  • engine (str) – Name of the engine for which the configuration is being transformed.

  • configuration (dict) – A dictionary containing parameter names with their values.

Returns:

A dictionary containing the transformed configuration for the specified engine.

Return type:

dict

Raises:

ValueError – If the specified engine is not supported.

Note:

The transformation process varies based on the engine type and specific configurations.

OAT

The Optano algorithm tuner (OAT) executes tuning on optimization functions using different algorithms like GGA, GGA++, JADE, and active CMA-ES. While it is able to run on a single computing node it also supports multiple workers. Before being able to use OAT first execute the following code after having installed up-ac.

from up_ac.utils.download_OAT import get_OAT, copy_call_engine_OAT, delete_OAT

get_OAT()
copy_call_engine_OAT()

The first function generates a directory for OAT, downloads compiled code for OAT and saves it in the up_ac directory. The second function moves code to the OAT directory. Once you have run these functions, you do not need to run them again, except if you have removed the OAT directory.

To remove the OAT directory run:

delete_OAT()

For more details on OAT refer to the documentation.

Functionalities for managing and calling configurators.

class up_ac.OAT_configurator.OATConfigurator

Configurator functions.

get_feedback_function(gaci, engine, metric, mode, gray_box=False)

Generate the function to run the engine and obtain feedback.

Parameters:

  • gaci – AC interface object.

  • engine (str) – Engine name.

  • metric (str) – Optimization metric (‘runtime’ or ‘quality’).

  • mode (str) – Type of planning.

  • gray_box (bool) – True if gray box to use.

Returns:

Planner feedback function.

Return type:

function

optimize(feedback_function=None, gray_box=False)

Run the algorithm configuration.

Parameters:

  • feedback_function (function) – Function to run the engine and get feedback.

  • gray_box (bool) – True if gray box usage.

Returns:

Tuple containing the best configuration found and None.

Return type:

tuple

set_scenario(engine, param_space, gaci, configuration_time=120, n_trials=400, min_budget=1, max_budget=3, crash_cost=0, planner_timelimit=30, n_workers=1, instances=[], instance_features=None, metric='runtime', popSize=128, evalLimit=2147483647, tourn_size=8, winner_percent=0.125, racing=False, numGens=None, goalGen=None, patience=10)

Set up algorithm configuration scenario.

Parameters:

  • engine (str) – Engine name.

  • param_space – ConfigSpace object.

  • gaci – AC interface object.

  • configuration_time (int) – Overall configuration time budget.

  • n_trials (int) – Maximum number of engine evaluations.

  • min_budget (int) – Minimum number of instances to use.

  • max_budget (int) – Maximum number of instances to use.

  • crash_cost (int) – Cost to use if engine fails.

  • planner_timelimit (int) – Maximum runtime per evaluation.

  • n_workers (int) – Number of cores to utilize.

  • instances (list) – Problem instance paths.

  • instance_features – Dictionary of instance names and lists of features.

  • metric (str) – Optimization metric.

  • popSize (int) – Population size of configs per generation (OAT).

  • evalLimit (int) – Maximum number of evaluations (OAT).

  • patience (int) – Extra time in seconds to grant engine to terminate gracefully.

OAT algorithm configuration interface for unified planning.

class up_ac.OAT_interface.OATInterface

OAT AC interface.

OAT does not handle forbidden parameter value combinations. OAT can handle multiple parent and 1 child conditionals, but not one parent multiple children conditionals. We naively just take the first one in the list. OAT does not support conditionals that are conditional. We leave them out naively. OAT does not support conditionals with value ranges. We naively only use the first value.

Note: Although this is suboptimal, invalid configurations will lead to crash or bad results such that OAT will rate them as subpar.

get_ps_oat(param_space)

Generate the OAT parameter tree in XML format.

OAT does not handle forbidden parameter value combinations. OAT can handle multiple parent and 1 child conditionals, but not one parent multiple children conditionals. We naively just take the first one in the list. OAT does not support conditionals that are conditional. We leave them out naively. OAT does not support conditionals with value ranges. We naively only use the first value.

Note: Although this is suboptimal, invalid configurations will lead to crash or bad results such that OAT will rate them as subpar.

Parameters:

param_space (ConfigSpace.ConfigurationSpace) – ConfigSpace object.

Returns:

OAT parameter tree in XML format.

Return type:

str

transform_conf_from_ac(engine, configuration, plantype)

Transform a configuration to the UP engine format.

Parameters:

  • engine (str) – The name of the engine.

  • configuration (dict) – Parameter names with values.

Returns:

The transformed configuration.

Return type:

dict

SMAC

SMAC3 (Sequential Model-Based Algorithm Configuration) optimizes algorithm parameters by employing a model-based approach, specifically Bayesian Optimization, to predict the performance of different configurations. It then uses an aggressive racing mechanism to efficiently compare configurations and iteratively refine the model, directing the search towards regions of the space where better configurations are likely to be found. In the autoconfiguration Smac can also make use of instance features to improve the predictions.

You can install it via

pip install swig
pip install smac==2.0.1

For more details on SMAC refer to the SMAC3 GitHub.

Functionalities for managing and calling SMAC.

class up_ac.Smac_configurator.SmacConfigurator

Configurator functions.

get_feedback_function(gaci, engine, metric, mode, gray_box=False)

Generate a function to run the planning engine and obtain feedback.

Parameters:

  • gaci (ACInterface) – Algorithm Configuration Interface object.

  • engine (str) – Name of the planning engine.

  • metric (str) – Metric to optimize, either ‘runtime’ or ‘quality’.

  • mode (str) – Type of planning.

  • gray_box (bool) – True if using a gray box, False otherwise (optional).

Returns:

A planner feedback function that takes configuration, instance, seed, and reader.

Return type:

function

Raises:

ValueError – If the provided engine is not supported for the given metric and mode.

optimize(feedback_function=None, gray_box=False)

Run the algorithm configuration optimization.

Parameters:

  • feedback_function (function) – A function to run the engine and obtain feedback (optional).

  • gray_box (bool) – True if gray box usage is enabled, False otherwise (optional).

Returns:

A tuple containing the best configuration found and additional information (if available). Returns None if feedback_function is not provided.

Return type:

tuple or None

set_scenario(engine, param_space, gaci, configuration_time=120, n_trials=400, min_budget=1, max_budget=3, crash_cost=10000, planner_timelimit=30, n_workers=1, instances=[], instance_features=None, output_dir='smac3_output', metric='runtime', patience=10)

Set up the algorithm configuration scenario for SMAC (Sequential Model-based Algorithm Configuration).

Parameters:

  • engine (str) – The name of the planning engine.

  • param_space (ConfigSpace.ConfigurationSpace) – ConfigSpace object defining the parameter space.

  • gaci (ACInterface) – AC interface object.

  • configuration_time (int) – Overall configuration time budget in seconds (default is 120).

  • n_trials (int) – Maximum number of engine evaluations (default is 400).

  • min_budget (int) – Minimum number of instances to use (default is 1).

  • max_budget (int) – Maximum number of instances to use (default is 3).

  • crash_cost (int) – The cost to use if the engine fails (default is 0).

  • planner_timelimit (int) – Maximum runtime per evaluation for the planner (default is 30).

  • n_workers (int) – Number of cores to utilize (default is 1).

  • instances (list) – List of problem instance paths (default is empty list, uses train_set).

  • instance_features (dict) – Dictionary containing instance names and lists of features (default is None).

  • metric (str) – The optimization metric, either ‘runtime’ or ‘quality’ (default is ‘runtime’).

  • patience (int) – Extra time in seconds to grant engine to terminate gracefully.

Raises:

ValueError – If an unsupported metric is provided.

Smac algorithm configuration interface for unified planning.

class up_ac.Smac_interface.SmacInterface

Generic Smac interface.

transform_conf_from_ac(engine, configuration, plantype)

Transform a configuration to the format expected by the planning engines.

Parameters:

  • engine (str) – Name of the planning engine.

  • configuration (dict) – The configuration with parameter names and values.

Returns:

The transformed configuration in the engine’s expected format.

Return type:

dict

Raises:

ValueError – If the provided engine list is empty or contains non-string elements.

Selector

Selector is an ensemble-based automated algorithm configurator. The current implementation includes functionalities and models from CPPL, GGA and SMAC.

For more details on Selector refer to the Selector GitHub.

You can install Selector via

pip install swig
pip install selector-ac

Functionalities for managing and calling selector.

class up_ac.Selector_configurator.SelectorConfigurator

Configurator functions.

get_feedback_function(gaci, engine, metric, mode, gray_box=False)

Generate a function to run the planning engine and obtain feedback.

Parameters:

  • gaci (ACInterface) – Algorithm Configuration Interface object.

  • engine (str) – Name of the planning engine.

  • metric (str) – Metric to optimize, either ‘runtime’ or ‘quality’.

  • mode (str) – Type of planning.

  • gray_box (bool) – True if using a gray box, False otherwise (optional).

Returns:

A planner feedback function that takes configuration, instance, seed, and reader.

Return type:

function

Raises:

ValueError – If the provided engine is not supported for the given metric and mode.

optimize(feedback_function=None, gray_box=False)

Run the algorithm configuration optimization.

Parameters:

  • feedback_function (function) – A function to run the engine and obtain feedback (optional).

  • gray_box (bool) – True if gray box usage is enabled, False otherwise (optional).

Returns:

A tuple containing the best configuration found and additional information (if available). Returns None if feedback_function is not provided.

Return type:

tuple or None

set_scenario(engine, param_space, gaci, configuration_time=120, n_trials=400, min_budget=1, max_budget=3, crash_cost=0, planner_timelimit=30, n_workers=1, instances=[], instance_features=None, tourn_size=8, metric='runtime', memory_limit=2048, patience=10, output_dir='up_ac_selector', ray_mode='desktop')

Set up the algorithm configuration scenario for Selector.

Parameters:

  • engine (str) – The name of the planning engine.

  • param_space (ConfigSpace.ConfigurationSpace) – ConfigSpace object defining the parameter space.

  • gaci (ACInterface) – AC interface object.

  • configuration_time (int) – Overall configuration time budget in seconds (default is 120).

  • n_trials (int) – Maximum number of engine evaluations (default is 400).

  • min_budget (int) – Minimum number of instances to use (default is 1).

  • max_budget (int) – Maximum number of instances to use (default is 3).

  • crash_cost (int) – The cost to use if the engine fails (default is 0).

  • planner_timelimit (int) – Maximum runtime per evaluation for the planner (default is 30).

  • n_workers (int) – Number of cores to utilize (default is 1).

  • instances (list) – List of problem instance paths (default is empty list, uses train_set).

  • instance_features (dict) – Dictionary containing instance names and lists of features (default is None).

  • metric (str) – The optimization metric, either ‘runtime’ or ‘quality’ (default is ‘runtime’).

  • memory_limit (int) – At which amount of MB used by a planning engine to terminate the run.

  • patience (int) – Extra time in seconds to grant engine to terminate gracefully.

  • output_dir (str) – Name of the directory selector writes to (directory is at .)

  • ray_mode (str) – Set to ‘desktop’ if running locally, or ‘cluster’ if running in SLURM

Raises:

ValueError – If an unsupported metric is provided.

class up_ac.Selector_interface.SelectorInterface

Using Smac interface.

Generic configuration tools

Here is an overview of the general functions and attributes used by all automated algorithm configuration tools.

These can be used to create custom automated algorithm configurators not included in this project.

Functionalities for managing and calling configurators.

class up_ac.configurators.Configurator

Configurator functions.

evaluate(metric, engine, mode, incumbent, gaci, planner_timelimit=300, crash_cost=10000, instances=[])

Evaluate performance of found configuration on training set.

Parameters:

  • metric (str) – Optimization metric.

  • engine (str) – Engine name.

  • mode (str) – Planning mode.

  • incumbent (dict) – Parameter configuration to evaluate.

  • gaci (ACInterface) – AC interface object.

  • planner_timelimit (int, optional) – Max runtime per evaluation, optional.

  • crash_cost (int, optional) – Cost if engine fails, optional.

  • instances (list, optional) – Instance paths, optional.

Returns:

Average performance on the instances.

Return type:

float

get_feedback_function(gaci, engine, metric, mode, gray_box=False)

Generate the function to run the engine and get feedback.

Parameters:

  • gaci (ACInterface) – Algorithm Configuration interface object.

  • engine (str) – Engine name.

  • metric (str) – Metric, either ‘runtime’ or ‘quality’.

  • mode (str) – Type of planning.

  • gray_box (bool, optional) – True if gray box to be used, optional.

Returns:

Planner feedback function or None if not supported.

Return type:

function or None

get_instance_features(instance_features=None)

Save instance features.

Parameters:

instance_features (dict, optional) – Instance names and their features in lists.

get_process_by_name(names)

Finds all processes matching given names.

Parameters:

names (str) – List of names of processes to get.

Returns:

Processes that match the names.

Return type:

list of psuitl.Process

kill_process_tree(pid, wt)

Terminate/kill all child processes.

Parameters:

  • pid – PID of parent Process.

  • wt – Time to wait for process to terminate.

optimize(feedback_function=None, gray_box=False)

Run the algorithm configuration.

Parameters:

  • feedback_function (function, optional) – Function to run engine and get feedback, optional.

  • gray_box (bool, optional) – True if gray box usage, optional.

Returns:

The best configuration found during optimization.

Return type:

dict

print_feedback(engine, instance, feedback)

Print feedback from the engine.

Parameters:

  • engine (str) – Name of the engine.

  • instance (str) – Name of the instance.

  • feedback – Feedback from the engine.

save_config(path, config, gaci, engine, plantype)

Save configuration in json file.

Parameters:

  • path (str) – Path where to save.

  • config (dict) – Configuration to save.

  • gaci (ACInterface) – AC interface object.

  • engine (str) – Engine name.

set_scenario(engine, param_space, gaci, configuration_time=120, n_trials=400, min_budget=1, max_budget=3, crash_cost=0, planner_timelimit=30, n_workers=1, instances=[], instance_features=None, metric='runtime', popSize=128, evalLimit=2147483647)

Set up algorithm configuration scenario.

Parameters:

  • engine (str) – Engine name.

  • param_space (ConfigSpace) – ConfigSpace object.

  • gaci (ACInterface) – AC interface object.

  • configuration_time (int, optional) – Overall configuration time budget, optional.

  • n_trials (int, optional) – Maximum number of engine evaluations, optional.

  • min_budget (int, optional) – Minimum number of instances to use, optional.

  • max_budget (int, optional) – Maximum number of instances to use, optional.

  • crash_cost (int, optional) – Cost to use if the engine fails, optional.

  • planner_timelimit (int, optional) – Maximum runtime per evaluation, optional.

  • n_workers (int, optional) – Number of cores to utilize, optional.

  • instances (list, optional) – Problem instance paths, optional.

  • instance_features (dict, optional) – Instance names and lists of features, optional.

  • metric (str, optional) – Optimization metric, optional.

  • popSize (int, optional) – Population size of configs per generation (OAT), optional.

  • evalLimit (int, optional) – Maximum number of evaluations (OAT), optional.

set_test_instance_set(test_set)

Set test instance set.

Parameters:

test_set (list) – List of instance paths.

set_training_instance_set(train_set)

Save training instance set.

Parameters:

train_set (list) – List of instance paths.

trigger_gc_for_children(pid)

Triggers garbage collection for child processes.

Parameters:

pid – PID of parent Process.

Generic algorithm configuration interface for unified planning.

class up_ac.AC_interface.GenericACInterface

Generic AC interface.

compute_instance_features(domain, instance)

Compute instance features of a given PDDL instance.

Parameters:

  • domain (str) – PDDL string representing the problem domain.

  • instance (str) – PDDL string representing the problem instance.

Returns:

Computed instance features.

Return type:

list

compute_plan_cost(problem, res)

Compute plan cost based on up problem and result.

Parameters:

  • problem (up.model.Problem) – up problem

  • res (up.engines.results.Result) – Result provided by a planning engine

Returns:

Cost of the plan

Type:

int

get_available_engines()

Get planning engines installed in up.

get_feedback(engine, fbtype, result)

Get feedback from a planning engine after a run.

Parameters:

  • engine (str) – Name of the planning engine.

  • fbtype (str) – Type of feedback: ‘quality’ or ‘runtime’.

  • result (object) – Planning result.

Returns:

Feedback based on the specified feedback type.

Return type:

object

Raises:

ValueError – If an unsupported feedback type is provided.

read_engine_pcs(engines, pcs_dir)

Read parameter configuration space (PCS) files for specified engines.

Parameters:

  • engines (list of str) – Names of the engines.

  • pcs_dir (str) – Path to the directory containing the PCS files.

run_engine_config(config, metric, engine, plantype, problem, timelimit, gb_listener=None)

Execute a configured engine run.

Parameters:

  • config (dict) – Configuration of the engine.

  • metric (str) – Metric for the evaluation: ‘runtime’ or ‘quality’.

  • engine (str) – Name of the engine.

  • plantype (str) – Type of planning: ‘OneshotPlanner’ or ‘AnytimePlanner’.

  • problem (str) – Path to the problem instance.

  • gb_listener (bool) – True if using a gray box approach (optional).

Returns:

Result from the configured engine run.

Return type:

object

Raises:

ValueError – If an unsupported planning type is provided.