Saving each event performed with Clutch for security auditing is a top-level concern, and the processes to accomplish this fit into the normal component architecture.
Clutch has configurable middleware to save data on each incoming request, a module to query for previous events, and services to store and forward events to new sinks. What information it stores for the request comes from API annotations on the request itself. The below table has the components ordered as they are in a call-chain.
|Module to retrieve audit events, called from outside of Clutch and calls the audit service.|
|Middleware that runs on each request. Calls the audit service to store request information.|
|Handler in front of storage services. Called to save or retrieve request information.|
|Calls the audit service to retrieve request information. Call out of Clutch to forward the events.|
The annotations are read by middleware for each request and response, which forwards the information to the audit service for storage. The audit service is responsible for persistence as well as forwarding to other sinks to fan-out the messages. Each step of saving and forwarding is configurable with filters based on event attributes.
Clutch's auditing middleware reads the value of two API annotations: the request "type" to determine what sort of action is being performed on a resource, and the resource name to identify what item is being modified.
These annotations let customizable data on each request be logged and reviewed.
Clutch will save the the "type" of action that was performed (create, read, update, or delete or CRUD operations) for each request. The request type comes from an API annotation on each RPC.
AmiiboAPI example proto, to show that the
GetCharacters endpoint is a "read" operation, an extra
option annotation would be needed.
The annotation would look similar for the other supported types of changes.
Clutch will also resolve what resources were modified by each request to their unique identifiers. This is also accomplished through API annotations.
There are two different annotations used to mark the resources in a single request:
fieldsannotation will indicate what fields contain resource names.
patternsannotation will try template substitution with field values in order to create a name.
If we wanted to add these annotations to the Amiibo example, where the identifier for a character is a combination of its name and game series, we would make the following modifications to the API definition.
The first annotation (
fields = "characters") says that to get the set of resources contained in this message, to look at the
The second annotation (
patterns = ...) says that the identifier for this message is built by combining the value of the
name field with the value of the
On each request, the above annotations are read along with what endpoint was called, by whom, and when. When a response is being sent, any additional resources or final identifiers are merged with the initial list from the request.
All of this information is passed along to the audit service to persist.
The configuration for the audit middleware looks like:
The audit service has two behaviors: write requests somewhere, and read them back out. It takes events from the middleware and saves them, and it also pushes them to later "sinks" for further processing.
Clutch ships with an implementation that uses Postgres as backing storage. With this, events that are saved can be filtered. At Lyft, we use this to prevent saving healthcheck information, since it is not interesting to us from a security perspective.
The configuration for this looks like:
Clutch's audit events can also be viewed by querying the audit module if it is enabled.
Clutch ships with an in-memory storage for events which allows it to be used without setting up a Postgres database. It is not recommended to run this way outside of a trial or adhoc temporary use. All history of actions taken with Clutch will be lost with the process shutting down.
Below is sample configuration to show how the services described are enabled. Note that because services are instantiated in the order they are listed, order matters! Since the audit service depends on both the database and the sink, it needs to be listed after them.
Sinks asynchronously propagate events to other systems after they are persisted to Clutch's database.
Clutch ships with a logging sink as a scaffold for your own, as well as a sink for Slack.
Adding and customizing audit sinks lets you save or process infrastructure events however appropriate for your needs.
By default, the Slack sink creates a formatted Slack message using a subset of information saved in an audit event. The default Slack message provides a summary, answering questions such as what operation was performed, who performed the operation, and what resources were operated on.
The Slack sink requires your Slack app’s bot token and the channel to post the messages. You can optionally provide filter rules to control what kinds of Slack audits are sent to the channel.
Custom Slack Messages
A custom Slack message can be created for a given
/service/method using the available metadata (the API request and response body) in an audit event. The custom message will then be appended to the default Slack message for a richer Slack audit.
Creating a Custom Slack Messages
The feature is powered by the Golang
template package. In clutch-config, you can provide a template with the field names from the API request and/or response, which will be replaced with the field values at parse time. The template can also include Slack
mrkdwn to add useful visual highlights to the custom message.
Creating the template:
.Request.<key_name>to obtain data from the API request
.Response.<key_name>to obtain data from the API response
- Clutch-specific templating tokens in lieu of the standard Golang Template Action and Variable syntax
- Any of the Golang Template functions can be used in the template