Running the Evaluator¶
Since Geneva only needs to run on one side of the connection, Geneva is flexible as to where and how it can be run. Strategy evaluation can be done either on the server-side or the client-side. When used on the server-side, it can drive an external client to itself (or another server) for testing, or even act as a NAT-ing middlebox to interpose between the external client and true server.
The evaluator also allows plugins to override its default logic for single strategy evaluation, or for evaluating the entire strategy pool. See Adding New Plugins for more detail on writing your own plugin.
Client-side Evaluation¶
The most common use case for Geneva is client-side evaluation. In this mode, Geneva is run at the client-side inside a censored regime, trying to make a query to a forbidden resource located outside the censored regime. For this example, we will use the HTTP plugin. The HTTP plugin creates a forbidden HTTP GET request (such as example.com?q=ultrasurf in China).
The evaluator will start the engine, launch the client http plugin (which will make a request), and then the client plugin will record if the request succeeded or not. Under the hood, the GET request generated by the client HTTP plugin is caught by the engine (which modifies the traffic according to the strategy it is running) before sending it.
This effectively looks like this:
Censored Regime
---------------------------------------------------------+
|
Client - Geneva |
+---------------------------------------------+ |
| evaluator | | Server
| +-----------+ strategy | | +-----------------+
| | http | GET /bad +----------+ | +----------+ | |
| | plugin <--------------> engine <-----------> censor <--------> forbidden.org |
| | | +----------+ | +----------+ | |
| +-----------+ | | +-----------------+
| | |
+---------------------------------------------+ |
|
|
|
---------------------------------------------------------+
Note
For the purpose of this guide, examples will be given using evolve.py. evolve.py has an option --eval-only to run the evaluator on a strategy with given parameters and exit. As usual, --eval-only can also be replaced with parameters to the genetic algorithm to start the full genetic algorithm, instead of a single strategy evaluation.
To accomplish this, we can simply run:
# python3 evolve.py --eval-only "\/" --test-type http --server forbidden.org --log debug
This will start evolve.py, which will launch the evaluator with the
http plugin, configured to make a request to forbidden.org, in debug mode.
Server-side Evaluation¶
Beyond using Geneva from the client-side, we can also use it from the server-side.
In this mode, Geneva can learn strategies that work from the server-side, subverting censorship on client’s behalf.
This requires a copy of Geneva to be checked out on both sides of the connection. To do this, we must specify the --server-side flag.
Geneva will first start up a server for the specified plugin, and start the engine with the given strategy in front of the server. It next SSHes into an external client worker located inside a censored regime and runs the specified plugin through the SSH session. That plugin will generate a forbidden request to our server, and the plugin on the external client records if the request succeeded or not; the evaluator retrieves this fitness over the SSH session and evaluation is complete. This looks like this:
Censored Regime
+------------------------------+
| Server - Geneva
| +---------------------------------+
Client | XXXXXXXXXX | |
+------------+ | XX XX | strategy +-----------+ |
| | GET /bad +----------+ X X | +--------+ | | |
| curl <-------------> censor <------> Internet <----------> engine <---> evaluator | |
| | +----------+ X X | +--------+ | | |
+-----^------+ | XX XX | +-----+-----+ |
| | XXXXXXXXXX | | |
| | +-----------------------+---------+
| | |
+-----+------------------------+ |
| |
| 1. SSHs into the client and drives the request |
+----------------------------------------------------------------------------+
To run this with evolve.py to evaluate an empty strategy (\/), we can do:
# python3 evolve.py --test-type http --public-ip <mypublicip> --external-client example
--log debug --eval-only "\/" --server-side
Note that we specified two additional options here: the public IP address of the test computer (used for making the request), and the external client worker we want to use.
Note
See Adding a Worker on how to configure external workers for Geneva.
External Clients with Local Servers¶
Alternatively, we can evaluate the strategies on the client side, but have Geneva host the server for us so we can observe the traffic on both sides of the connection. By simply not specifying the --server-side flag, we can accomplish this.
Censored Regime
+------------------------------------------+
| Server + Geneva
| +-------------------+
External Client | | |
+------------+ strategy | | +-----------+ |
| | GET /bad +--------+ +----+-----+ | | | |
| curl <------------> engine <----> censor <--------> evaluator | |
| | +--------+ +----+-----+ | | | |
+-----^------+ | | +-----+-----+ |
| | | | |
| | +-------------------+
| | |
+------------------------------------------+ |
| |
| 1. SSHs into the client and drives the request |
+--------------------------------------------------------+
# python3 evolve.py --test-type http --public-ip <mypublicip> --external-client example
--log debug --eval-only "\/" --server-side
External Clients with External Servers¶
Alternatively, the client can be driven to a different server from outside the censored regime with the '--external-server' flag, and the server can be specified with '--server':
# python3 evolve.py --test-type http --public-ip <mypublicip> --external-client example
--log debug --eval-only "\/" --external-server --server http://wikipedia.org
Geneva
+--------------------------+
| |
| +-----------+ |
| | evaluator | |
| +-----+-----+ |
| | |
+-------------+------------+
|
SSHes & drives request |
+----------------+
|
| Censored Regime
+---------+----------------------------------------------+
| |
| External Client |
+---------+-----------------------------------+ |
| | | | Server
| +-----v-----+ strategy | | +-----------------+
| | http | GET /bad +----------+ | +-----+----+ | |
| | plugin <--------------> engine <-----------> censor <--------> forbidden.org |
| | | +----------+ | +-----+----+ | |
| +-----------+ | | +-----------------+
| | |
+---------------------------------------------+ |
|
|
|
+--------------------------------------------------------+
Engine as a Middlebox¶
There are many cases in which we cannot trigger a censor while communicating with servers we control. In these cases, we can use Geneva as a middlebox, and interpose between the client and server and apply the strategy in between.
To accomplish this, we will specify --act-as-middlebox, and specify three additional routing options:
- Routing options:
--routing-ip: Internal IP address of the middlebox--forward-ip: IP address we are forwarding to--sender-ip: IP address we are forwarding from
Note
Because Geneva operates at the packet-level, it cannot be used out of the box as a general purpose NAT, as it does not do connection tracking.
In this mode, the external client will communicate directly with our IP address, which will in turn forward the packets to the specified destination after they are modified according to the strategy under evaluation.
To run the strategy engine as a middlebox and drive an external client, we can add additional routing options:
# python3 evolve.py --test-type http --public-ip <mypublicip> --external-client example --log
debug --eval-only "\/" --server-side --act-as-middlebox --routing-ip <myinternalip>
--forward-ip <iptoforwardto> --sender-ip <ipofexternalclient>
1. SSHs into the client and drives the request
+------------------------------------------------+
| |
| +---------------------+
| Censored Regime | | |
+----------------------------+ | +-----+-----+ |
| | | | evaluator | |
v | | +-----------+ |
External Client | | | Server
+------------+ | | strategy | +---------+
| | GET /bad +----+-----+ | +--------+ | | |
| curl <------------> censor <------------> engine <------------> bad.com |
| | +----+-----+ | +--------+ | | |
+------------+ | | | +---------+
| +---------------------+
|
+----------------------------+
Note
The --routing-ip is NOT the public IP address of your machine, its the internal address. This means that if you’re running on an EC2 machine, the --public-ip is your external IP, but the --routing-ip is the IP you see when you run 'ifconfig'.
Internal Evaluation with Docker¶
For the purpose of internal testing and fitness function development, Geneva provides a set of simple mock censors (see censors/). The evaluator can be configured to train against these censors.
Note
In order to do internal evaluation against these mock censors, you must have set up Geneva’s base Docker container. See Setup for how to do this.
When used with Docker, Geneva will spin up three docker containers: a client, a censor, and a server, and configure the networking routes such that the client and server communicate through the censor. To evaluate strategies, Geneva will run the plugin client inside the client and attempt to communicate with the server through the censor.
Each docker container used by the evaluator runs out of the same base container.
evaluator
+--------------------------------------------------------------+
| |
| Client Censor Server |
| +----------+ +----------+ +----------+ |
| | | | | | | |
| | client | | censor | | server | |
| | <-----+-----> <-----+-----> | |
| | plugin | ^ | | ^ | plugin | |
| | | | | | | | | |
| +----------+ | +----------+ | +----------+ |
| container | container | container |
+--------------------------------------------------------------+
| |
| |
+------------------+
engine can be run on either side
We can accomplish this by specifying the --censor <censor> option. This will enable running with Docker, start the censor, and perform evaluation.
Depending on whether --server-side is specified, the engine will be run on either the client or server side.
# python3 evolve.py --eval-only "" --test-type echo --censor censor2 --log debug
Note
--censor cannot be used with --external-client or --external-server.