Commit 42adf8b1 authored by Jochen Kressin's avatar Jochen Kressin
Browse files

Merge branch 'pr/signals-amendments' into feature/signals

parents 669daf53 a46dbcda
......@@ -19,7 +19,7 @@ description:
Conditions are used to control the execution flow. A condition can be used anywhere in the execution chain for watches and actions.
A condition must return a boolean. If the condition returns true, the execution continues. If the condition returns false, the execution is stopped.
A condition must return a boolean value. If the condition returns true, the execution continues. If the condition returns false, the execution is stopped.
In watches, a condition controls whether a certain value or threshold is reached, to decide whether the watch should continue execution.
......
......@@ -44,7 +44,8 @@ The following condition tests whether the total hits of a search, stored in the
|---|---|
| type | condition.script, defines this conditions as script condition. Mandatory. |
| name | name of this condition. Can be chosen freely. Mandatory. |
| source | The painless script to execute. Mandatory |
| source | The script to execute. Mandatory |
| lang | The scripting language to be used. Optional, defaults to painless. Other scripting languages may be provided by Elasticsearch plugins. |
## Stored scripts
......@@ -64,13 +65,15 @@ To run a stored script, refer to it by using it's id:
| name | name of this condition. Can be chosen freely. Mandatory. |
| script_id | The ID of the stored script. Mandatory |
## Accessing the execution context data
**Note:** When using stored scripts, keep in mind that stored scripts are not subject to multi-tenancy or Signals permissions and may be thus changed independently of the watch. Thus, you should review whether the ability to edit scripts is adequately restricted.
All scripts have full access to the data stored in the execution context, for example Elasticsearch or HTTP inputs.
## Accessing the runtime data
The data in the execution context is available via the `data` prefix, followed by the target name of the data.
All scripts have full access to the runtime data, gathered for example by Elasticsearch or HTTP inputs.
For example, the following watch runs a query against the serverlogs index to find entries where the statuscodeis 500. The result of the query is stored in the target with name `http_error_500`. The script condition accesses the data by using the `data.http_error_500` prefix and only continues if the total hits is above 10.
The runtime data is available via the `data` prefix.
For example, the following watch runs a query against the serverlogs index to find entries where the statuscode is 500. The target property of the input is configured to be `http_error_500`; thus the document read by the input is put under this property name into the runtime data. The script condition accesses the data by using the `data.http_error_500` prefix and only continues if the total hits is above 10.
```
{
......
......@@ -11,7 +11,7 @@ description:
<!--- Copyright 2019 floragunn GmbH -->
# Execution chain and execution context payload
# Execution chain and execution runtime data
{: .no_toc}
{% include toc.md %}
......@@ -20,7 +20,7 @@ description:
Each watch can define as many inputs, transformations, calculations and conditions as required, in any order.
Each step in the execution chain is called a `check` and needs to have a unique name. Example:
Each step in the execution chain is called a *check*. Example:
```
{
......@@ -57,11 +57,11 @@ Each step in the execution chain is called a `check` and needs to have a unique
}
```
## Execution context: Payload
## Execution runtime data
All `check`s operate on the watch execution context. The execution context stores all runtime data of the watch.
All checks and actions operate on the watch runtime data.
[Input](inputs.md) checks add data to the context under a unique name. [Transformations](transformations_transformations.md) transform existing data, [Calculations](transformations_calculations.md) add data based on existing data, and [Conditions](conditions.md) control the execution flow based on the runtime data.
[Input](inputs.md) checks can add data to the context; either under a specific property name or at the top level, replacing all data that was possibly stored before. [Transformations](transformations_transformations.md) transform existing data, [Calculations](transformations_calculations.md) add data based on existing data, and [Conditions](conditions.md) control the execution flow based on the runtime data.
[Actions](actions.md) send out notifications based on the runtime data, or store all or parts of the runtime data on a data sink, like Elasticsearch.
......@@ -69,9 +69,9 @@ All `check`s operate on the watch execution context. The execution context store
<img src="runtime_context.png" style="width: 40%" class="md_image"/>
</p>
### Adding data to the execution context
### Adding data to the runtime data
An [input](inputs.md) fetches data and places it in the execution context under a name specified by the `target` of the check. Example:
[Inputs](inputs.md) and [Transformations](transformations_transformations.md) fetch data and place it in the runtime data under a name specified by the `target` of the check. Example:
```
{
......@@ -95,11 +95,11 @@ An [input](inputs.md) fetches data and places it in the execution context under
}
```
This input executes an Elasticsearch query and stores the result of the query under the name `avg_ticket_price`.
This input executes an Elasticsearch query and stores the result of the query under the name `avg_ticket_price`. Using the special target name `_top` places the data at the top level of the runtime data. Any data that was present before gets erased.
### Accessing data in the execution context
### Accessing runtime data
Transformations, calculations and conditions access data in the execution context by using the target name:
Transformations, calculations and conditions access runtime data by using the prefix `data` followed by the respective property name. The property name is, of course, defined by the target value used in the input before.
```
{
......@@ -122,9 +122,9 @@ Format:
data.<target name>.path.to.data
```
### Accessing data in moustache templates
### Accessing data in Mustache templates
Actions format their messages by using moustache templates. Moustache templates have access to the execution context data as well. Example:
Actions format their messages by using Mustache templates. Mustache templates have access to the runtime data as well. Example:
```
......@@ -157,6 +157,3 @@ Format:
{% endraw %}
```
## Top-level context
If an input does not define any target, the data is stored in the top-level of the execution context.
\ No newline at end of file
......@@ -30,13 +30,13 @@ Assume you are ingesting log files from production systems to Elasticsearch. The
You can use Signals to:
* run an aggregaton periodically on the logs index that counts the amount of errors in the last 5 minutes
* run an aggregation periodically on the logs index that counts the amount of errors in the last 5 minutes
* implement a condition that checks whether the error level is above a certain threshold
* if the condition is met, send out notifications via Email or Slack to inform your DevOps team.
## Audit logging example
Asume you use the [Search Guard audit log feature](audit-logging-compliance) and want to be able to detect brute-force attempts to your cluster.
Assume you use the [Search Guard audit log feature](audit-logging-compliance) and want to be able to detect brute-force attempts to your cluster.
You can use Signals to
......@@ -47,32 +47,34 @@ You can use Signals to
* send out an email and/or Slack notification
* send an addition escalation email if the issue persist for more than 1 hour
## Components of a Signals watch
## Building blocks of a Signals watch
A Signals watch consists of the following components:
The basic working principle of a watch goes as follows:
* Trigger
* A [trigger](triggers.md) defines when a watch will be executed. Each watch needs to have at least on trigger
* Inputs
* An [input](inputs.md) pulls in data from a source. Can be Elasticsearch, HTTP or constants.
* Execution context payload
* Data pulled in by inputs is stored in the runtime execution payload under a unique name. All subsequent steps have access to this payload.
* Transformations and Calculations
* If required you can run painless scripts to [transform data in the payload, or calculate new values](transformations.md)
* Conditions
* [Conditions](conditions.md) control the execution flow. Conditions can be defined for watches and also actions.
* If a watch-level condition is not met, execution of the watch is aborted.
* If an action-level condition is not me, execution of this action is skipped.
* Actions
* [Actions](actions.md) are executed if all conditions are met.
* Actions be used to alert users via [Email](actions_email.md), [Slack](actions_slack.md), [Webhooks](actions_webhook.md) or PagerDuty (coming soon).
* Actions can be used to write the payload data back to data sinks like [Elasticsearch](actions_index.md).
After a watch has been *triggered*, it *checks* for certain conditions, and takes *action* if necessary.
## Execution chain: Checks
These three elements also form the three major building blocks of a Signals watch:
Inputs, transformations and conditions can be configured freely and in any order in the execution chain of the watch. Each step in the execution chain is called a `check`. You can configure as many checks as required.
* **[Triggers](triggers.md)** define when a watch will be executed. Each watch should have at least on trigger
* **Checks** are constructs meant for analyzing the situation to be watched. For doing so, Signals offers
* *[Inputs](inputs.md)* which pull in data from a source such as an Elasticsearch index or a HTTP service;
* *[Conditions](conditions.md)* to analyze the gathered data using scripts and decide whether to proceed with execution or to abort;
* *[Transformations and calculations](transformations.md)* to transform the gathered data into a format that subsequent operations may require.
* Each watch can have several checks, which are executed as a chain. Each action of a watch can have a further chain of checks.
* **[Actions](actions.md)** are executed if all preceding conditions are met.
* Actions be used to alert users via [Email](actions_email.md), [Slack](actions_slack.md), or PagerDuty (coming soon).
* Actions can be used to write the runtime data back to data sinks like an [Elasticsearch index](actions_index.md).
* Using the [Webhook action](actions_webhook.md), it is actually possible to invoke any kind of operation as result of a Signals watch.
* Each watch can have several actions. The action-specific checks can be used to select which actions are to be executed in which situation.
## Watch Runtime Data
All watch operations operate on the so-called watch runtime data. Index inputs put the gathered data into the runtime data; conditions can read it and transforms can modify it. Actions read from the runtime data as well.
The runtime data is formed like a hierarchical key/value document, quite similar to a document stored in an Elasticsearch index.
The checks of a watch subsequently modify the runtime data. If action-specific checks are defined, these will be operating on isolated copies of the runtime data. So, modifications of the runtime data done for one action does have no effect on the runtime data visible for other actions.
In addition, you can also configure `checks` for actions. For example, you can implement a condition for an action, so it is only executed if the condition is met, not affecting other actions. Before the action is executed, you can implement an action-specific transform, which cleans up and formats data prior to using it in a notification.
## Overview
......@@ -141,7 +143,7 @@ In addition, you can also configure `checks` for actions. For example, you can i
{
"type": "webhook",
"name": "myslack",
"throttle_period": "1s",
"throttle_period": "10m",
"request": {
"method": "POST",
"url": "https://hooks.slack.com/services/token",
......@@ -151,9 +153,6 @@ In addition, you can also configure `checks` for actions. For example, you can i
}
}
}
],
"active": true,
"log_runtime_data": false,
"_id": "avg_ticket_price"
]
}
```
\ No newline at end of file
......@@ -15,7 +15,7 @@ description:
# Inputs
{: .no_toc}
Each watch can have one or more inputes. Inputs can be freely defined anywhere in the execution chain.
Each watch can have one or more inputs. Inputs can be freely defined anywhere in the execution chain.
Each input will fetch data from a data source, and place it in the runtime data context under a configurable key for later usage.
......
......@@ -14,7 +14,7 @@ description:
# Search input
{: .no_toc}
A search input can be used to pull in data from Elasticsearch.
A search input can be used to pull in data from an Elasticsearch index.
You can use the full power of the Elasticsearch query syntax to query, filter and aggregate your data.
......@@ -68,7 +68,7 @@ Example:
| target | the name under which the data is available in later execution steps. |
| request | The search request to execute |
| request.indices | The indices to execute the `request.query` against. **The user that defines the watch needs to have a role that has access to the specified index / indices.**|
| request.body | The body of the search request. You can use all features of the Elasticsearch query and aggregation DSL here. |
| request.body | The body of the search request. You can use all features of the Elasticsearch query and aggregation DSL here. All attributes of the request body can be dynamically defined using Mustache templates.|
......
......@@ -48,6 +48,7 @@ For example, if you aggregate data from the [Search Guard Audit Log](auditlog),
| request.url | The URL for this HTTP input |
| request.method | One of GET|PUT|POST|DELETE |
| request.auth | Optional. The authentication method for the HTTP request. |
| request.body | The body of the HTTP request. Optional. Mustache templates can be used to render attributes from the watch runtime data. |
## Accessing HTTP input data in the execution chain
......@@ -57,7 +58,45 @@ In this example, the return values from the HTTP call can be accessed in later e
data.samplejson.mykey
```
## Dynamic Endpoints
The HTTP endpoint in the `request.url` attribute cannot be changed dynamically directly. However, you can use the configuration attributes `request.path` and `request.query_params` to define the respective parts of the URL using Mustache templates. The resulting path and/or query parameters then override the respective parts of the URL defined in `request.url`.
```json
{
"trigger": { ... },
"checks": [{
"type": "http",
"name": "testhttp",
"target": "samplejson",
"request": {
"method": "GET",
"url": "https://jsonplaceholder.typicode.com/",
"path": "todos/{{data.todo_no}}",
"auth": {"type":"basic","username":"admin","password":"admin"}
}
}],
"actions": [ ... ]
}
```
## Authentication
Authentication credentials are configured in the `auth` section if the `request` configuration. At the time of writing, the only authentication method is HTTP Basic Authentication.
**Note:** In the current version of the tech preview, the password is stored unencrypted and returned in verbatim when the watch is retrieved using the REST API. Future versions will provide a more secure way of storing authentication data.
## Advanced Functionality
Furthermore, HTTP inputs provide these configuration options:
**connection_timeout:** Specifies the time after which the try to create an connection shall time out. Optional. Specified in seconds.
**read_timeout:** Specifies the timeout for reading the response data after a connection has been already established. Optional. Specified in seconds.
## Security Considerations
Keep in mind that webhook actions allow to send arbitrary HTTP requests from Elasticsearch nodes. We are still working on mechanisms to define restrictions on the use of webhook actions and the allowed endpoints.
......@@ -25,7 +25,7 @@ Access control to the Signals API is governed by Search Guard roles. Signals shi
| SGS\_SIGNALS\_ALL | Grants access to all Watch APIs|
| SGS\_SIGNALS\_WATCH\_MANAGE | Grants permission to manage watches. Includes create, read, write and delete operations, and execute, activate/deactivate and acknowledging watches. |
| SGS\_SIGNALS\_WATCH\_READ | Grants read-only access to all Watch APIs. Includes retrieving watches and search for watches, and excludes everything else.|
| SGS\_SIGNALS\_WATCH\_EXECUTE | Grants permissions to execute watches. |
| SGS\_SIGNALS\_WATCH\_EXECUTE | Grants permissions to manually execute watches using the REST API. |
| SGS\_SIGNALS\_WATCH\_ACTIVATE | Grants permissions to activate and deactivate watches. |
| SGS\_SIGNALS\_WATCH\_ACKNOWLEDGE | Grants permissions to acknowledge watches. |
......
......@@ -50,36 +50,6 @@ In this example, regardless how often the condition in the watch definition fire
On other words, if you define a throttle period, the action will not fire if the last execution time is in the time window of the throttle period.
You can also access the data in the execution context to define a throttle period:
```json
{% raw %}
{
"checks": [
{
"type": "static",
"name": "constants",
"target": "myconstants",
"value": {
"throttleperiod": "1h",
}
}
],
"actions": [
{
"type": "email",
"name": "my_email_action",
"throttle_period": "{{data.myconstants.throttleperiod}}",
"account": "internal_mail",
"to": "notify@example.com",
"subject": "...,
"text_body": "..."
}
]
}
{% endraw %}
```
## Acknowledgement
......
......@@ -22,7 +22,7 @@ A transformation is a script that
* performs one or more painless statements
* replaces the
As opposed to [Caclulations](transformations_calculations.md), Transformation scripts have a return statement and need to define the target context where the transformed values are written back to.
As opposed to [Calculations](transformations_calculations.md), Transformation scripts have a return statement and need to define the target context where the transformed values are written back to.
If the target context already exists, it is overwritten. If not, a new one is created.
......@@ -53,7 +53,8 @@ For example, the next transformation accesses a runtime context that has stored
| type | transform, defines this script as transformation. Mandatory. |
| name | name of this transformation. Can be chosen freely. Mandatory. |
| target | Under which context name to store the result of the transformation in the runtime data. If the context already exists, it is replaced. If it does not exist, a new contect is created. If omitted, the top-level context will be used. |
| source | The painless script to execute. Mandatory |
| source | The script to execute. Mandatory |
| lang | The scripting language to be used. Optional, defaults to painless. Other scripting languages may be provided by Elasticsearch plugins. |
## Using stored scripts
......@@ -73,9 +74,9 @@ For example, the next transformation accesses a runtime context that has stored
| target | Under which context name to store the result of the transformation in the runtime data. If the context already exists, it is replaced. If it does not exist, a new contect is created. If omitted, the top-level context will be used. |
| script_id | The ID of the stored script. Mandatory. |
## Accessing the execution context data
## Accessing the runtime data
All scripts have full access to the data stored in the execution context. The data in the execution context is available via the `data` prefix, followed by the target name of the data.
All scripts have full access to the runtime data. The data in the execution context is available via the `data` prefix.
## Using transformations with actions
......
......@@ -48,7 +48,7 @@ Example:
## Trigger execution
Each trigger gets registered with the Trigger Execution Engine. The execution engine makes sure thar
Each trigger gets registered with the Trigger Execution Engine. The execution engine makes sure that
* Each trigger is executed on exactly one node at a time
* You can specify node filters to define on which nodes Signals Alerting should run
......
......@@ -32,5 +32,5 @@ You can specify the timezone of each trigger by adding a `timezone` ID to the s
}
```
A timezone ID can be either something like "Europe/Berlin" or "UTC+01:00". For details on timezoine IDs, refer to the [JavaDocs on timezones](https://docs.oracle.com/javase/8/docs/api/java/time/ZoneId.html).
A timezone ID can be either something like "Europe/Berlin" or "UTC+01:00". For details on timezone IDs, refer to the [JavaDocs on timezones](https://docs.oracle.com/javase/8/docs/api/java/time/ZoneId.html).
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment