Benchmark

Note

A benchmark object can be created by calling jcmwave.optimizer.create_benchmark().

class jcmwave.client.Benchmark(host, benchmark_id, session, num_average)

This class provides methods for benchmarking different optimization studies against each other. Example:

benchmark = Benchmark(num_average=6)
benchmark.add_study(study1)
benchmark.add_study(study2)
benchmark.set_objective(objective)
benchmark.run()
data = benchmark.get_data(x_type='num_evaluations',y_type='objective',
           average_type='mean')
fig = plt.figure(figsize=(8,4))
for idx,name in enumerate(data['names']):
    X = data['X'][idx]
    Y = np.array(data['Y'][idx])
    std_error = np.array(data['sdev'][idx])/np.sqrt(6)
    p = plt.plot(X,Y,linewidth=2.0, label=name)
    plt.fill_between(X, Y-std_error, Y+std_error, alpha=0.2, color = p[0].get_color())
plt.legend(loc='upper right',ncol=1)
plt.grid()
plt.ylim([0.1,10])
plt.rc('font',family='serif')
plt.xlabel('number of iterations',fontsize=12)
plt.ylabel('average objective',fontsize=12)
plt.show()

add_study(study)

Adds a study to the benchmark. Example:

benchmark.add_study(study1)

Parameters:study – A Study() object.

add_study_results(study)

Adds the results of a benchmark study at the end of an optimization run. Example:

benchmark.add_study_results(study1)

Parameters:study – A Study() object after the study was run.

get_data(**kwargs)

Get benchmark data. Example:

data = benchmark.get_data( x_type='num_evaluations', y_type='objective',
     average_type='mean')
plt.plot(data['X'][0],data['Y'][0])

Parameters:

• x_type (str) – Data on x-axis. Can be either ‘num_evaluations’ or ‘time’
• y_type (str) – Data type on y-axis. Can be either ‘objective’, ‘distance’, (i.e. accumulated minimum distance off all samples to overall minimum), or ‘min_distance’ (i.e. distance of current minimum to overall minimum).
• average_type (str) – Type of averaging over study runs. Can be either ‘mean’ w.r.t. x-axis data or ‘median’ w.r.t. y-axis data
• invert (bool) – If True, the objective is multiplied by -1. (Parameter not available for distance average types)
• log_scale (bool) – If True, the ouput of Y and sdev are determined as mean and standard deviations of the natural logarithm of the considered y_type.
• minimum (list) – Vector with minimum position. (Only available for distance average types)
• scales (list) – Vector with positive weights for scaling distance in different directions. (Only available for distance average types)
• norm (str/int) – Order of distance norm as defined in numpy.linalg.norm. (Only available for distance average types)
• num_samples (int) – Number of samples on y-axis. (Only available for median average type or time on x-axis)

run()

Run the benchmark after the objective has been set (see set_objective()). Example:

benchmark.run()

set_objective(objective)

Set the objective function to be minimized. Example:

def objective(x1,x2): 
    observation = study.new_observation()
    observation.add(x1**2+x2**2)
    return observation
benchmark.set_objective(objective)

Note

Call this function only after all studies have been added to the benchmark.

Parameters:objective (func) – Function handle for a function of the variable parameters that returns a corresponding Observation() object.

property studies

A list of studies to be run for the benchmark.