geneva.evaluator

The Evaluator is charged with evaluating a given strategy and assigning a numerical fitness metric to it.

class evaluator.Evaluator(command, logger)

Bases: object

__init__(command, logger)

Initialize the global evaluator for this evolution.

Parameters:
  • command (list) – sys.argv or list of arguments
  • logger (logging.Logger) – logger passed in from the main driver to log from
assign_ids(ind_list)

Assigns random environment ids to each individual to be evaluated.

Parameters:ind_list (list) – List of individuals to assign random IDs to
canary_phase(canary)

Learning phase runs the client against the censor to collect packets.

Parameters:canary (actions.strategy.Strategy) – A (usually empty) strategy object to evaluate
Returns:canary id used (“canary”)
Return type:str
create_test_environment(worker_id)

Creates a test environment in docker.

Parameters:worker_id (int) – Worker ID of this worker
Returns:Environment dictionary to use
Return type:dict
dump_logs(environment_id)

Dumps client, engine, server, and censor logs, to be called on test failure at ERROR level.

Parameters:environment_id (str) – Environment ID of a strategy to dump
evaluate(ind_list)

Perform the overall fitness evaluation driving.

Parameters:ind_list (list) – list of individuals to evaluate
Returns:Population list after evaluation
Return type:list
exec_cmd(command, timeout=60)

Runs a subprocess command at the correct log level.

Parameters:
  • command (list) – Command to execute.
  • timeout (int, optional) – Timeout for execution
exec_cmd_output(command, timeout=60)

Runs a subprocess command at the correct log level. This is a separate method from the above exec_cmd, since that is used to stream output to the screen (so check_output is not appropriate).

Parameters:
  • command (list) – Command to execute.
  • timeout (int, optional) – Timeout for execution
Returns:

Output of command
Return type:

str
get_ip()

Gets IP of evaluator computer.

Returns:Public IP provided
Return type:str
get_log(remote, worker, log_name, logger)

Retrieves a log from the remote server and writes it to disk.

Parameters:
  • remote – A Paramiko SSH channel to execute over
  • worker (dict) – Dictionary describing external client worker
  • log_name (str) – Log name to retrieve
  • logger (logging.Logger) – A logger to log with
get_pid(container)

Returns PID of first actively running python process.

Parameters:container – Docker container object to query
Returns:PID of Python process
Return type:int
initialize_base_container(name)

Builds a base container with a given name and connects it to a given network. Also retrieves lower level settings and the IP address of the container.

Parameters:name (str) – Name of this docker container
Returns:Dictionary containing docker container object and relevant information
Return type:dict
parse_ip(container, iface)

Helper method to parse an IP address from ifconfig.

Parameters:
  • container – Docker container object to execute within
  • iface (str) – Interface to parse from
Returns:

IP address
Return type:

str
read_fitness(ind)

Looks for this individual’s fitness file on disk, opens it, and stores the fitness in the given individual.

Parameters:ind (actions.strategy.Strategy) – Individual to read fitness for
remote_exec_cmd(remote, command, logger, timeout=15, verbose=True)

Given a remote SSH session, executes a string command. Blocks until command completes, and returns the stdout and stderr. If the SSH connection is broken, it will try again.

Parameters:
  • remote – Paramiko SSH channel to execute commands over
  • command (str) – Command to execute
  • logger (logging.Logger) – A logger to log with
  • timeout (int, optional) – Timeout for the command
  • verbose (bool, optional) – Whether the output should be printed
Returns:

(stdout, stderr) of command, each is a list
Return type:

tuple
run_client(args, environment, logger)

Runs the plugin client given the current configuration

Parameters:
  • args (dict) – Dictionary of arguments
  • environment (dict) – Dictionary describing environment configuration for this evaluation
  • logger (logging.Logger) – A logger to log with
Returns:

Fitness of individual
Return type:

float
run_docker_client(args, environment, logger)

Runs client within the docker container. Does not return fitness; instead fitness is written via the flags directory and read back in later.

Parameters:
  • args (dict) – Dictionary of arguments
  • environment (dict) – Dictionary describing environment configuration for this evaluation
  • logger (logging.Logger) – A logger to log with
run_docker_server(args, environment, logger)

Runs server and censor in their respective docker containers.

Parameters:
  • args (dict) – Dictionary of arguments
  • environment (dict) – Dictionary describing environment configuration for this evaluation
  • logger (logging.Logger) – A logger to log with
run_local_client(args, environment, logger)

Runs client locally. Does not return fitness.

Parameters:
  • args (dict) – Dictionary of arguments
  • environment (dict) – Dictionary describing environment configuration for this evaluation
  • logger (logging.Logger) – A logger to log with
run_local_server(args, environment, logger)

Runs local server.

Parameters:
  • args (dict) – Dictionary of arguments
  • environment (dict) – Dictionary describing environment configuration for this evaluation
  • logger (logging.Logger) – A logger to log with
run_remote_client(args, environment, logger)

Runs client remotely over SSH, using the given SSH channel

Parameters:
  • args (dict) – Dictionary of arguments
  • environment (dict) – Dictionary describing environment configuration for this evaluation
  • logger (logging.Logger) – A logger to log with
Returns:

Fitness of individual
Return type:

float
run_test(environment, ind)

Conducts a test of a given individual in the environment.

Parameters:
  • environment (dict) – Dictionary of environment variables
  • ind (actions.strategy.Strategy) – A strategy object to test with
Returns:

(ind.environment_id, ind.fitness) environment ID of strategy and fitness
Return type:

tuple
setup_remote()

Opens an SSH tunnel to the remote client worker.

shutdown()

Shuts down all active environments

shutdown_container(container)

Tries to shutdown a given container and eats a NotFound exception if the container has already exited.

Parameters:container – docker container object to call stop() on
shutdown_environment(environment)

Shuts down the evaluation environment. If Docker, shuts down server and client container. If a remote SSH connection, the connection is shut down.

start_censor(environment, environment_id)

Starts the censor in the server environment container.

Parameters:
  • environment (dict) – Environment dictionary
  • environment_id (str) – Environment ID of the censor to stop
start_server(args, environment, logger)

Launches the server.

Parameters:
  • args (dict) – Dictionary of arguments
  • environment (dict) – Dictionary describing environment configuration for this evaluation
  • logger (logging.Logger) – A logger to log with
Returns:

fitness of individual (if one is provided)
Return type:

float
stop_censor(environment)

Send SIGKILL to all remaining python processes in the censor container. This is done intentionally over a SIGINT or a graceful shutdown mecahnism - due to dynamics with signal handling in nfqueue callbacks (threads), SIGINTs can be ignored and graceful shutdown mechanisms may not be picked up (or be fast enough).

The output this method parses is below:

# ps aux
USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root          1  0.1  0.0  21944  3376 pts/0    Ss+  13:30   0:00 /bin/bash
root         14 24.0  0.4 200376 38564 ?        Ss   13:30   0:00 python code/censor_driver.py censor2 jgsko1rf trials/2018-10-30_06:30:48 60181
root         32  0.0  0.0  19188  2400 ?        Rs   13:30   0:00 ps aux
Parameters:environment (dict) – Environment dictionary
stop_server(environment, server)

Stops server.

Parameters:
  • environment (dict) – Environment dictionary
  • server (plugins.plugin_server.ServerPlugin) – A plugin server to stop
terminate_docker()

Terminates any hanging running containers.

update_ports(environment)

Checks that the chosen port is open inside the docker container - if not, it chooses a new port.

Parameters:environment (dict) – Dictionary describing docker environment
worker(ind_list, worker_id, environment)

Perform the actual fitness evaluation as a multithreaded worker. The worker pops off an individual from the list and evaluates it.

Parameters:
  • ind_list (list) – List of strategy objects to evaluate
  • worker_id (int) – ID of this worker
  • environment (dict) – Environment dictionary
evaluator.collect_plugin(test_plugin, plugin_type, command, full_args, plugin_args)

Import a given plugin

Parameters:
  • test_plugin (str) – Plugin name to import (“http”)
  • plugin_type (str) – Component of plugin to import (“client”)
  • command (list) – sys.argv or list of arguments
  • full_args (dict) – Parsed full list of arguments already maintained by the parent plugin
  • plugin_args (dict) – Dictionary of args specific to this plugin component
Returns:

Imported plugin class for instantiation later
evaluator.get_arg_parser(single_use=False)

Sets up argparse. This is done separately to enable collection of help messages.

Parameters:single_use (bool, optional) – whether this evaluator will only be used for one strategy, used to configure sane defaults
evaluator.get_args(cmd, single_use=False)

Creates an argparser and collects arguments.

Parameters:single_use (bool, optional) – whether this evaluator will only be used for one strategy, used to configure sane defaults
Returns:parsed args
Return type:dict
evaluator.get_random_open_port()

Selects a random ephemeral port that is open.

Returns:Open port
Return type:int