Adding New Plugins

This section will describe the process to add a new application plugin to Geneva. Application plugins serve as the fitness function for Geneva during evolution, and allow it to evolve strategies to defeat certain types of censorship.

Plugins are run by the Evaluator; if you have not yet read how the Evaluator works, see the Strategy Evaluation section.

There are three types of plugins: clients, servers, and overriding plugins. A developer can choose to implement any one of, or all three of these plugins.

For this section, we will build an example plugin and walk through the existing plugins to tour through the plugin API.

Plugins are expected to be in the plugins/ folder in geneva’s repo. The folder name is the plugin name, and Geneva will discover these automatically. Plugins are specified to the evaluator or evolve with the --test-type flag. Within the plugin folder, plugins must adhere to the following naming scheme:

  • client.py - for plugin clients
  • server.py - for plugin servers [optional]
  • plugin.py - for an overriding plugin to customize logic [optional]

server.py and plugin.py are optional. The server plugin is required to do server-side evaluation, but the overriding plugin definition is only required to if a developer wishes to override the evaluator’s default behavior.

If an overriding plugin is provided, the evaluator will simply invoke it at the start of strategy evaluation and the overriding plugin will be responsible for calling the client and server. This section will assume that no overriding plugin is specified to describe the evaluator’s default behavior with plugins, and cover use cases for overriding plugins at the end.

Depending on the evaluation setup, some (or all) of these plugins will be used during evaluation. For example, during an exclusively client-side evaluation, only the client plugin is needed.

Client Plugins

During exclusively client-side evolution, the evaluator will start the engine with the strategy under evaluation, and then run the client plugin. During server-side evolution, the evaluator will run the engine on the server-side, start the server plugin, and then start the client plugin via an SSH session to the remote client worker. (See Adding a Worker on how external workers can be used).

The client plugin subclasses from the PluginClient object. To tour through the API, we will walk through the development of a custom client plugin.

Writing Our Own

Let us write a fitness function to test Iran’s whitelisting system.

Iran’s protocol whitelister was a recently deployed new censorship mechanism to censor non-whitelisted protocols on certain ports (53, 80, 443). We deployed Geneva against the whitelister, and discovered multiple ways to evade it in just one evolution of the genetic algorithm. (The results of that investigation is located here).

The whitelister worked by checking the first 2 packets of a flow, and if they did not match a fingerprint, it would destroy the flow.

In order to run Geneva against whitelister, we will define a client plugin that will try to trigger the whitelister and record whether the whitelister successfully censored its connection or not.

First, let’s make a new folder in the plugins/ directory called “whitelister”. We’ll create a “client.py” and create the plugin object as a subclass of ClientPlugin.

class WhitelisterClient(ClientPlugin):
    """
    Defines the whitelister client.
    """
    name = "whitelister"

    def __init__(self, args):
        """
        Initializes the whitelister client.
        """
        ClientPlugin.__init__(self)
        self.args = args

Done! Next, let’s define argument parsing for this plugin. Geneva uses a pass-through system of argument parsing: when command-line arguments are specified, evolve.py parses the options it knows and passes the rest to the evaluator. The evaluator prases the options it knows, and passes the list to the plugins. This allows developers to easily add their own arguments just to their plugin and use them from the command-line without changing any of the intermediate code.

In this case, we need our client to make a TCP connection to a server located outside of Iran to send our whitelister triggering messages to. Let’s add an argument so the user can specify which server to connect to.

We can do this by adding a get_args static method. The evaluator will call this method when the plugin is created and give it the full command line list, so the plugin is free to parse it how it chooses. For this example, we will use the standard argparse library.

Since the superclass also defines args, we’ll pass the command line list up to the super class as well to collect those arguments.

@staticmethod
def get_args(command):
    """
    Defines args for this plugin
    """
    super_args = ClientPlugin.get_args(command)
    parser = argparse.ArgumentParser(description='Whitelister Client')

    parser.add_argument('--server', action='store', help="server to connect to")

    args, _ = parser.parse_known_args(command)
    args = vars(args)

    super_args.update(args)
    return super_args

Now, we just need to define a run() method. The run() method is called by the evaluator to run the plugin. It provides the parsed arguments, a logger to log with, and a reference to an instance of the strategy engine that is running the strategy (see Engine for more information on how the engine works.)

Let’s start by defining the run method. We’ll pull out the argument for the server we defined earlier, connect to it with a python socket, and then just send “G”, “E”, and “T” in separate messages to trigger the whitelister. Since the whitelister censors connections by blackholing them, if the strategy failed to defeat the whitelister, we would expect our network connection to timeout; if we can send our messages and get a response from the server, the strategy under evaluation may have defeated the whitelister.

def run(self, args, logger, engine=None):
     """
     Try to open a socket, send two messages, and see if the messages
     time out.
     """
     fitness = 0
     port = int(args["port"])
     server = args["server"]
     try:
         client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
         client.settimeout(3)
         client.connect((server, port))
         client.sendall(b"G")
         time.sleep(0.25)
         client.sendall(b"E")
         time.sleep(0.25)
         client.sendall(b"T\r\n\r\n")
         server_data = client.recv(1024)
         logger.debug("Data recieved: %s", server_data.decode('utf-8', 'ignore'))
         if server_data:
             fitness += 100
         else:
             fitness -= 90
         client.close()
         # ...

Now we just need to define the error handling for this code. This is critical to the fitness function: we want to kill off strategies that damage the underlying TCP connection, so Geneva does not waste time searching this space of strategies.

Our goal is to set the fitness metric such that a censorship event has a higher fitness than the strategy damaging the connection. Since we can distinguish these cases based on the socket error, we will set a lower fitness if any other exception is raised besides the timeout.

Lastly, we’ll inflate the numerical fitness metric to make it a larger number. The evaluator does additional punishments to the fitness score based on the strategy (see Strategy Evaluation), so we want the number to be sufficiently large to not push succeeding strategies to negative numbers.

    except socket.timeout:
        logger.debug("Client: Timeout")
        fitness -= 90
    except socket.error as exc:
        fitness -= 100
        logger.exception("Socket error caught in client echo test.")
    finally:
        logger.debug("Client finished whitelister test.")
return fitness * 4

Putting it all together:

class WhitelisterClient(ClientPlugin):
    """
    Defines the whitelister client.
    """
    name = "whitelister"

    def __init__(self, args):
        """
        Initializes the whitelister client.
        """
        ClientPlugin.__init__(self)
        self.args = args

    @staticmethod
    def get_args(command):
        """
        Defines args for this plugin
        """
        super_args = ClientPlugin.get_args(command)
        parser = argparse.ArgumentParser(description='Whitelister Client')

        parser.add_argument('--server', action='store', help="server to connect to")

        args, _ = parser.parse_known_args(command)
        args = vars(args)

        super_args.update(args)
        return super_args

    def run(self, args, logger, engine=None):
        """
        Try to open a socket, send two messages, and see if the messages
        time out.
        """
        fitness = 0
        port = int(args["port"])
        server = args["server"]
        try:
            client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            client.settimeout(3)
            client.connect((server, port))
            client.sendall(b"G")
            time.sleep(0.25)
            client.sendall(b"E")
            time.sleep(0.25)
            client.sendall(b"T\r\n\r\n")
            server_data = client.recv(1024)
            logger.debug("Data recieved: %s", server_data.decode('utf-8', 'ignore'))
            if server_data:
                fitness += 100
            else:
                fitness -= 90
            client.close()
        except socket.timeout:
            logger.debug("Client: Timeout")
            fitness -= 90
        except socket.error as exc:
            fitness -= 100
            logger.exception("Socket error caught in client echo test.")
        finally:
            logger.debug("Client finished whitelister test.")
    return fitness * 4

Server Plugins

Coming soon!

Override Plugins

Coming soon!