A framework to perform Chaos experiments built on top of EnvoyProxy
Chaos Experimentation Framework consists of a few parts - frontend, a backend server and a xDS management server.
The Frontend uses Clutch's core frontend. It can be customized by using the frontend config.
The Backend server is responsible for performing CRUD operations of the experimentation package - CreateExperiment, GetExperiments, CancelExperimentRun, etc. It stores Chaos experiments in its tables in the Postgres database.
The xDS management server uses go-control-plane library and serves two Envoy APIs - Runtime Discovery Service (RTDS) and Extension Configuration Discovery Service (ECDS). Either of these two APIs can be used to perform fault injection tests. With RTDS, you can make changes to runtime specific to faults whereas with ECDS you can make changes to the entire fault filter to perform any custom Chaos experiments.
Below components are responsible to perform Chaos experiments starting from storing the data in the Postgres database for each incoming request all the way to passing the experiment values to the Envoys to inject faults.
|Module that supports CRUD API for managing Chaos experiments like Create, Get, List, Cancel, etc|
|Module responsible for orchestrating server fault Chaos experiments.|
|Module which runs Envoy xDS management server which is responsible for propagating chaos experiment configurations to Envoys|
|Service that defines the data layer to perform all database operations for chaos experiments|
|Service used to connect to Postgres database|
In order to use Chaos Experimentation Framework, registration of all the above components is required.
It is recommended to run Envoy xDS management server (
clutch.module.chaos.experimentation.xds) on a separate host.
The frontend of the framework is completely configurable. Below is an example frontend config which will show the list of Chaos experiments and as well as workflow to start/stop an experiment.
Below configuration will spin up all the required modules and services to store the data coming from frontend into Postgres database.
Below is the configuration for spinning up xDS server. For details about the fields, take a look at the xds config proto.
Keep in mind that both backend config and xDS config need to connect to the same Postgres database.
When Envoy in the mesh boots up, it creates a bi-directional gRPC stream with the management server. Below is the sample Envoy configs for RTDS and ECDS which will initiate the connection to the xDS server. Checkout Envoy Proxy docs for details on Envoy's support for Fault Injection.
To perform Redis Chaos experiments, there is a specific module that is used to process the Redis Chaos experiments data. You will need below component in addition to the above Experimentation components. Also, keep in mind that Redis experiments can be only be run with the use of RTDS (and not ECDS).
|Module which is responsible for orchestrating the Redis Chaos experiments|