selector.tournament_performance

This modulke contains functions to aggregate and discern performances.

Functions

get_censored_runtime_for_instance_set(...)

For a configuration compute the total runtime needed only for instances in a set.

get_censored_runtime_of_configuration(...)

Get total runtime of a conf not conditioned on an instance set.

get_conf_time_out(results, configuration_id, ...)

Determine if a configuration timed out on any instance in a set

get_instances_no_results(results, ...)

For a configuration get a list of instances we have no results for yet.

get_runtime_for_instance_set_with_timeout(...)

For a configuration compute the total runtime needed only for instances in a set.

overall_best_update(tournaments, results, ...)

Over all tournaments get the best finisher with the most instance runs and shortest runtime and save that conf to a file.
selector.tournament_performance.get_censored_runtime_for_instance_set(results, configuration_id, instances_set)[source]

For a configuration compute the total runtime needed only for instances in a set. If there are no results for the conf, return 0. Note that runs that were canceled by the monitor are not included since we count them as NaNs.

Parameters:

  • results (dict) – Dictionary of results: {conf_id: {instance: runtime}}

  • configuration_id (uuid.UUID) – ID of the configuration

  • instances_set (list) – List of instances

Returns:

Runtime of the configuration on instances

Return type:

float

selector.tournament_performance.get_censored_runtime_of_configuration(results, configuration_id)[source]

Get total runtime of a conf not conditioned on an instance set. Note that runs that were canceled by the monitor are not included since we count them as nan’s.

Parameters:

  • results (dict) – Results dictionary in the format {conf_id: {instance: runtime}}.

  • configuration_id (int) – ID of the configuration.

Returns:

Total runtime of the configuration.

Return type:

float

selector.tournament_performance.get_conf_time_out(results, configuration_id, instances_set)[source]

Determine if a configuration timed out on any instance in a set

Parameters:

  • results (dic) – Dictionary of results: {conf_id: {instance: runtime}}

  • configuration_id (uuid.UUID) – ID of Configuration.

  • instances_set (list) – Set of instances

Returns:

None if no results are present, True if conf timed out on any instance, False if it did not.

Return type:

None or bool

selector.tournament_performance.get_instances_no_results(results, configuration_id, instance_set)[source]

For a configuration get a list of instances we have no results for yet.

Parameters:

  • results (dict) – Dic of results: {conf_id: {instance: runtime}}.

  • configuration_id (int) – Id of the configuration.

  • instance_set (list) – List of instances.

Returns:

List of configurations the conf has not been run on.

Return type:

list

selector.tournament_performance.get_runtime_for_instance_set_with_timeout(results, configuration_id, instances_set, timeout, par_penalty=1)[source]

For a configuration compute the total runtime needed only for instances in a set. If there are no results for the conf, return 0. Note that runs that were canceled by the monitor are not included since we count them as NaNs.

Parameters:

  • results (dict) – Dictionary of results: {conf_id: {instance: runtime}}

  • configuration_id (uuid.UUID) – ID of the configuration

  • instances_set (list) – List of instances

  • timeout (int) – Time limit for target algorithm runs in seconds.

  • par_penalty (int) – PAR penalty for timeout.

Returns:

Runtime of the configuration on instances

Return type:

float

selector.tournament_performance.overall_best_update(tournaments, results, scenario, ac_runtime)[source]

Over all tournaments get the best finisher with the most instance runs and shortest runtime and save that conf to a file.

Parameters: