Commit 93f1cbc7 authored by Jochen Kressin's avatar Jochen Kressin
Browse files

reworked documentation for SG14, preliminary work for SG15

parent bf5c806d
<!---
Copryight 2016 floragunn GmbH
Copryight 2016-2017 floragunn GmbH
-->
# Search Guard Documentation
This is the official documentation for Search Guard 2 and Search Guard 5. If you find any errors, or want to contribute, feel free to fork this repository and issue pull requests.
This is the official documentation for Search Guard 5 and Search Guard 2. If you find any errors, or want to contribute, feel free to fork this repository and issue pull requests.
Unless stated otherwise, all instructions apply for Search Guard 2 and Search Guard 5. If there are differences between the versions, this will be made clear in the documentation.
floragunn GmbH is not affiliated with Elasticsearch BV.
Unless stated otherwise, all instructions apply for Search Guard 5 and Search Guard 2. If there are differences between the versions, this will be made clear in the documentation.
Search Guard is a trademark of floragunn GmbH, registered in the U.S. and in other countries
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries.
floragunn GmbH is not affiliated with Elasticsearch BV.
Copyright 2016-2017 floragunn GmbH
## Table of contents
......@@ -22,36 +22,45 @@ Copyright 2016-2017 floragunn GmbH
* [Frontmatter](frontmatter.md)
* [Quickstart](quickstart.md)
* [Search Guard Overview](overview.md)
* [Search Guard Main Concepts](overview.md)
### Installation
* [Installing Search Guard](installation.md)
* [Upgrading Search Guard](upgrading.md)
* [Removing Search Guard](removing.md)
* [Compatibility with other plugins](compatibility.md)
### Configuring Search Guard SSL
### Configuring TLS
* [Generating TLS certificates](tls_overview.md)
* [Generating demo TLS certificates](tls_generate_demo_certificates.md)
* [Configuring TLS](tls_configuration.md)
* [TLS node certificates](tls_node_certificates.md)
* [Moving TLS to production](tls_certificates_production.md)
* [Using OpenSSL](tls_openssl.md)
* [Expert features: Custom SSLContext and PrincipalExtractor](tls_expert.md)
* [Troubleshooting TLS problems](tls_troubleshooting.md)
* [Expert features: Custom implementations and SSLContext](tls_expert.md)
### Configuring Search Guard
### Configuring Users, Roles and Permissions
* [Using sgadmin](sgadmin.md)
* [sgadmin Examples](sgadmin_examples.md)
* [Troubleshooting sgadmin](sgadmin_troubleshooting.md)
* [Configuring authentication and authorisation](configuration_auth.md)
* [Adding users to the internal user database](configuration_internalusers.md)
* [Mapping users to Search Guard roles](configuration_roles_mapping.md)
* [Defining roles and permissions](configuration_roles_permissions.md)
* [Using and defining action groups](configuration_action_groups.md)
* [Moving the configuration to production](configuration_production.md)
* [Search Guard configuration](configuration.md)
* [Using sgadmin to change configuration](sgadmin.md)
* [Search Guard index management](sgindex.md)
* [Running Search Guard behind a proxy](proxies.md)
### Authentication and Authorisation
### Configuring Authentication and Authorisation modules
* [Internal user database](internalusers.md)
* [HTTP Basic Authentication](httpbasic.md)
* [LDAP and Active Directory](ldap.md)
* [Kerberos](kerberos.md)
* [JSON Web token](jwt.md)
* [Proxy Authentication](proxy_auth.md)
* [Client Certificate Authentication](proxy_auth.md)
### Document- and Field-Level-Security
* [Document and field level security](dlsfls.md)
......@@ -62,13 +71,23 @@ Copyright 2016-2017 floragunn GmbH
### REST Management API
* [REST management API](managementapi.md)
### Search Guard index management
* [Search Guard index management](sgindex.md)
### Running Search Guard behind a Proxy
* [Running Search Guard behind a proxy](proxies.md)
### Integrating with the Elastic Stack
* [Using Search Guard with Kibana](kibana.md)
* [Kibana Multitenancy](multitenancy.md)
* [Using Kibana Multitenancy](multitenancy.md)
* [Using Search Guard with logstash](logstash.md)
* [Using Search Guard with X-Pack Monitoring](monitoring.md)
* [Tribe nodes](tribenodes.md)
* [Snapshot & Restore](snapshots.md)
* [Tribe nodes](tribenodes.md)
### Integrating with X-Pack
* [Using Search Guard with X-Pack Monitoring](x_pack_monitoring.md)
* [Using Search Guard with X-Pack Alerting](x_pack_alerting.md)
* [Using Search Guard with X-Pack Machine Learning](x_pack_machine_learning.md)
### Integrating with X-Pack Alternatives
* [Using Search Guard with ElastAlert](elastalert.md)
......
<!---
Copryight 2017 floragunn GmbH
-->
# Client certificate based authentication
Search Guard can use a client TLS certificate in the HTTP request to authenticate users and assign roles and permissions.
## Installation
Search Guard already ships with client certificate based authentication. No additional installation steps are required.
## Configuration
In order for Search Guard to pick up client certificate on the REST layer, you need to set the `clientauth_mode` in `elasticsearch.yml` to either `OPTIONAL` or `REQUIRED`:
```
searchguard.ssl.http.clientauth_mode: OPTIONAL
```
The configuration for the client certificate authenticator is very minimal:
```
clientcert_auth_domain:
enabled: false
order: 1
http_authenticator:
type: clientcert
config:
username_attribute: cn
challenge: false
authentication_backend:
type: noop
```
| Name | Description |
|---|---|
| username_attribute | String, the part of the certificate's DN that is used as username. If not specified, the complete DN is used.|
## Mapping DNs to roles
To map a certificate based user to a role, just use the username as specified by `username_attribute` in `sg_roles_mapping.yml`, for example:
```
sg_role_starfleet:
users:
- 'cn=ldaprole,ou=groups,dc=example,dc=com'
backendroles:
- ...
hosts:
- ...
```
\ No newline at end of file
## Compatibility
### Compatibility with other plugins
If you have other plugins installed, please check the compatibility with Search Guard.
As a rule of thumb, if a plugin is compatible with Shield, it is also compatible with Search Guard. Specifically:
If the plugin talks to Elasticsearch using REST and you have REST TLS enabled, the plugin must also support TLS and HTTP Basic Authentication.
If the plugin talks to Elasticsearch on the transport layer, you need to be able to add the Search Guard SSL plugin and its configuration settings to the transport client. You can read more about using transport clients with a Search Guard secured cluster [in this blog post](https://floragunn.com/searchguard-elasicsearch-transport-clients/).
#### Compatible plugins and tools
The following plugins and tools have been tested for compatibility with Search Guard:
* [Kibana](https://www.elastic.co/de/products/kibana) (with the Search Guard Kibana plugin installed)
* [Logstash](https://www.elastic.co/de/products/logstash)
* [Beats](https://www.elastic.co/de/products/beats)
* [X-Pack Monitoring](https://www.elastic.co/guide/en/x-pack/current/xpack-monitoring.html)
* [X-Pack Alerting](https://www.elastic.co/guide/en/x-pack/current/xpack-alerting.html)
* [X-Pack Machine Learning](https://www.elastic.co/guide/en/x-pack/current/xpack-ml.html)
* [Elastalert](https://github.com/Yelp/elastalert)
* [Curator](https://github.com/elastic/curator)
* [Kibi](https://siren.solutions/kibi/)
* [syslog-ng](https://syslog-ng.org/)
* Kopf / [Cerebro](https://github.com/lmenezes/cerebro)
* [Grafana](https://grafana.com/)
* ES-Hadoop / Spark
#### Incompatible plugins and tools
* [Graylog](https://www.graylog.org/)
* [JDBC Importer](https://github.com/jprante/elasticsearch-jdbc)
<!---
Copryight 2016 floragunn GmbH
-->
# Configuring Search Guard
## Configuring authentication and authorisation
The main configuration file for Search Guard is `sg_config.yml`. This file is used to specify both authentication and authorisation and has three main parts:
```
searchguard:
dynamic:
http:
...
authc:
...
authz:
...
```
In short, these sections are used to specify how Search Guard retrieves the user credentials, how it verifies these credentials, and where to get the (additional) users roles from. The latter is optional.
### Authentication
Let's first look at the `authc` section. This section has the following format:
```
<name>:
enabled: <true|false>
order: <integer>
http_authenticator:
...
authentication_backend:
...
```
An entry in the `authc` section of the config file is called an `authentication domain`. It specifies where to get the user credentials from, and against which backend they should be authenticated.
The first thing you'll notice is that you can define more than one authentication domain in this section. Each authentication domain has a name (e.g. "basic\_auth\_internal"), an `enabled` flag and an `order`. This makes it possible to chain authentication domains together. Search Guard will execute them in the order provided. Let's concentrate on a single entry first.
The two sections you need to fill are `http_authenticator` and the `authentication_backend`. The `http_authenticator` specifies which authentication method you want to use on the HTTP layer.
The syntax for defining an authenticator on the HTTP layer is:
```
http_authenticator:
type: <type>
challenge: <true|false>
config:
...
```
Allowed values for `type` are:
* basic—HTTP basic authentication. This type needs `challenge` set to `true`. No additional configuration is needed. See [HTTP Basic Authentication](httpbasic.md) for further details.
* kerberos—Kerberos authentication. This type needs `challenge` set to `false`. Additional, [Kerberos-specific configuration](kerberos.md) is needed.
* clientcert—authentication via a client TLS certificate. This certificate must be trusted by one of the Root CAs in the truststore of your nodes. See [TLS Client Certification](httpbasic.md) for further details.
* jwt-Use JSON web tokens as authentication method. Needs additonal settings.
* proxy-Use an external, proxy based authentication. This type needs `challenge` set to `false`. Additional, proxy-specific configuration is needed, and the "X-forwarded-for" module has to be enabled as well. See [Proxy authentication](proxy.md) for further details.
The config section contains specific configuration settings for the selected type. At the moment, only Kerberos, JWT and Proxy need additional configuration. See these sections in the documentation for further information.
After the HTTP authenticator was executed, you need to specify against which backend system you want to authenticate the user. This is specified in the `authentication_backend` section and has the following format:
```
authentication_backend:
type: <type>
config:
...
```
Possible vales for `type` are:
* noop—This means that no authentication against a backend system is performed. This setting only makes sense if the HTTP authenticator already authenticated the user or if the request carries some credentials that are implicitly trusted. The former is true if you choose Kerberos as HTTP authentication type. The latter is true if you choose Proxy as HTTP authentication type.
* internal—use the users and roles defined in `sg_internal_users` for authentication. This requires you to specify users and roles in the file `sg_internal_users.yml` and load them into Search Guard by using the `sgadmin` command line tool.
* ldap—authenticate users against an LDAP server. This requires additional configuration settings, see [LDAP and Active Directory](ldap.md) for further details.
### Authorization
After the user has been authenticated, Search Guard can optionally collect additional user roles from backend systems. Authorisation configuration has the following format:
```
authz:
<name>:
enabled: <true|false>
authorization_backend:
type: <type>
config:
...
```
You can also define multiple entries in this section the same way as you can for authentication entries. The execution order is not relevant here, hence there is no `order` field.
Possible vales for `type` are:
* noop
* Used for skipping this step altogether
* ldap
* Fetch additional roles from an LDAP server. This requires additional configuration settings, see section "LDAP and Active Directory" for further details.
#### Examples
Please refer to the [Addendum A](addendum_a_configuration_examples.md) of this documentation for some common configuration examples.
## Map users, backend roles and hosts to Search Guard roles
As outlined in chapter [Search Guard main concepts](concepts.md), Search Guard:
* Retrieves the user credentials.
* Validates them against the configured authentication backend(s).
* Collects additional roles from the configured authorization backend(s).
Depending on your configuration, you can now use the following data to assign the request to one or more Search Guard role:
* username—the name of the user.
* backend roles—the additional roles fetched by the authorization backend(s).
* hostname / IP—the hostname or IP the request originated from.
* Common name—rhe DN of the client certificate sent with the request.
Backend users, roles and hosts need to be mapped to Search Guard roles. This is done in the file `sg_roles_mapping.yml`.
```
sg_read_write:
backendroles:
- management
- operations
- 'cn=ldaprole,ou=groups,dc=example,dc=com'
hosts:
- "*.devops.company.com"
users:
- janedoe
- johndoe
```
A request can be assigned to one or more Search Guard roles. If a request is mapped to more than one role, the permissions of these roles are combined with `AND`.
## Define roles and their associated permissions
Search Guard roles and their associated permissions are defined in the file `sg_roles.yml`. You can define as many roles as you like. The syntax to define a role, and associate permissions with it, is as follows:
```
<sg_role_name>:
cluster:
- '<permission>'
indices:
'<indexname or alias>':
'<type>':
- '<permission>'
_dls_: '<querydsl query>'
_fls_:
- '<field>'
- '<field>'
```
The `cluster` entry is used to define permissions on cluster level. The `indices` entry is used to define permissions as well as [Document- and field-level security](dlsfls.md) on index level.
For `<permission>`, `<indexname or alias>` and `<type>` simple wildcards are possible:
* An asterisk (\*) will match any character sequence (or an empty sequence).
* A question mark (?) will match any single character (but NOT empty character).
Example: `\*my\*index` will match `my_first_index` as well as `myindex` but not `myindex1`.
Example: `?kibana` will match `.kibana` but not `kibana`.
For <permission>, <indexname or alias> and <type> also regular expressions are possible. You have to pre- and append a `/` to use regex instead of simple wildcards: `/<java regex>/`.
Example: `/\S*/` will match any non whitespace characters.
**Note: You cannot have a dot (.) in the <permission>, <indexname or alias> or <type> regex. Use `(\S|\s)` instead.**
Example
* `.` == `(\S|\s)` and `.*` == `(\S|\s)*`
* `[.]`, `[\.]` or `\.` == `\x2E`
See [https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html](https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html)
For `<indexname or alias>` also the placeholder `${user.name}` is allowed to support indices or aliases, which contains the name of the user. During evaluation of the permissions, the placeholder is replaced with the actual username.
### Defining permissions
Permissions can be applied on the cluster and index level. Cluster-level permissions always start with `cluster:`, while index-level permissions start with `indices:`. After that, a REST-style path further defines the exact action the permission grants.
For example, this permission would grant the right to execute a search on an index:
```
indices:data/read/search
```
While this permission grants the right to write to the index:
```
indices:data/write/index
```
On cluster-level, this permission grants the right to display the cluster health:
```
cluster:monitor/health
```
There is a plethora of permissions you can set. Search Guard is compatible with the permission definition of Shield up to version 2.1. [Here](https://www.elastic.co/guide/en/shield/2.1/reference.html#ref-actions-list) is a complete list.
Since there are so many permissions you can use, we strongly recommend to use action grouping (see next section) and work with action group aliases in `sg_roles.yml`.
## Defining action groups
Since defining permissions based on fine-grained actions can be quite verbose, Search Guard comes with a feature called *action groups*. As the name implies, this is a named group of actions. After an action group has been defined, you can refer to the group of actions simply by its name.
Action groups are defined in the file `sg_action_groups.yml`. The file structure is very simple:
```
<action group name>:
- '<permission>'
- '<permission>'
- '<permission>'
- ...
```
The definition of actions is the same as outlined in the chapter "Defining permissions". Wildcards are also supported. You can use any name you want. And you can also reference an action group from within another action group:
```
SEARCH:
- "indices:data/read/search*"
- "indices:data/read/msearch*"
- SUGGEST
SUGGEST:
- "indices:data/read/suggest*"
```
In this case, the action group `SEARCH` includes the (wildcarded) `search*` and `msearch*` permissions, and also all permissions defined by the action group `SUGGEST`.
You can then reference these action groups in the file `sg_roles.yml` simply by name:
```
sg_readall:
indices:
'*':
'*':
- SEARCH
```
# Using and defining action groups
An action group is simply a collection of permissions with a telling name. Action groups are defined in the file `sg_action_groups.yml` and can be referred to in `sg_roles.yml`. Action groups can be nested.
The file structure is very simple:
```
<action group name>:
- '<permission or action groups>'
- '<permission or action group>'
- ...
```
Using action groups is the preferred way of assigning permissions to a role.
## Pre-defined action groups
Search Guard ships with a list of pre-defined action groups that are suitable for most use cases.
### General
| Name | Description |
|---|---|
| UNLIMITED | Grants complete access, can be used on index- and cluster-level. Equates to `"*"`.|
### Index-level action groups
| Name | Description |
|---|---|
| INDICES_ALL | Grants all permissions on the index. Equates to `indices:*`|
| GET | Grants permission to use get and mget actions (only) |
| READ | Grants read permissions like get, mget or getting field mappings, but exludes search permissions |
| WRITE | Grants write permissions to documents |
| DELETE | Grants permission to delete documents |
| CRUD | Combines the READ, WRITE and DELETE action groups |
| SEARCH | Grants permission to search documents |
| SUGGEST | Grants permission to use the suggest API. Already included in the READ action group. |
| CREATE_INDEX | Grants permission to create indices and mappings|
| INDICES_MONITOR | Grants permission to execute all actions regarding index monitoring, e.g. recovery, segments info, index stats & status |
| MANAGE_ALIASES | Grants permission to manage aliases |
| MANAGE | Grants all `monitor` and index administration permissions |
### Cluster-level action groups
| Name | Description |
|---|---|
| CLUSTER_ALL | Grants all cluster permissions. Equates to `cluster:*`|
| CLUSTER_MONITOR | Grants all cluster monitoring permissions. Equates to `cluster:monitor/*`|
| CLUSTER\_COMPOSITE\_OPS\_RO | Grants read-only permissions to execute multi requests like mget, msearch or mtv, plus permission to query for aliases. |
| CLUSTER\_COMPOSITE\_OPS | Same as `CLUSTER\_COMPOSITE\_OPS\_RO`, but also grants bulk write permissions and all aliases permissions. |
| MANAGE_SNAPSHOTS | Grants full permissions to manage snapshots and repositories. |
### Multi- and bulk requests
To execute mult- and bulk requests, the respective user needs to have
* multi and/or bulk permission on cluster level.
* the respective permission(s) on index level
For example, if a user exeuctes a bulk request containing a delete request for index1 and an update request for index2, three permissions are required
* bulk permission on cluster level
* delete permission for index1
* write permission for index2
## Defining your own action groups
You can define your own action groups in `sg_action_groups.yml`. You can use any name you want. And you can also reference an action group from within another action group:
```
SEARCH:
- "indices:data/read/search*"
- "indices:data/read/msearch*"
- SUGGEST
SUGGEST:
- "indices:data/read/suggest*"
```
In this case, the action group `SEARCH` includes the (wildcarded) `search*` and `msearch*` permissions, and also all permissions defined by the action group `SUGGEST`.
You can then reference these action groups in the file `sg_roles.yml` simply by name:
```
sg_readall:
indices:
'*':
'*':
- SEARCH
```
# Configuring authentication and authorisation
Search Guard comes with pluggable authentication and authorisation modules. Depending on your use case and infrastructure, you can use one or multiple authentication and authorisation modules like:
* [Search Guard Internal user database](internalusers.md)
* [LDAP and Active Directory](ldap.md)
* [Kerberos](kerberos.md)
* [JSON Web token](jwt.md)
* [Proxy Authentication](proxy_auth.md)
The main configuration file for authentication and authorization modules is `sg_config.yml`. It defines how Search Guard retrieves the user credentials, how it verifies these credentials, and how additional user roles are fetched from backend systems (optional).
It has two main parts:
```
searchguard:
dynamic:
authc:
...
authz:
...
```
## Authentication
The `authc` section has the following format:
```
<name>:
enabled: <true|false>
order: <integer>
http_authenticator:
...
authentication_backend:
...
```
An entry in the `authc` section is called an `authentication domain`. It specifies where to get the user credentials from, and against which backend they should be authenticated.
You can use more than one authentication domain. Each authentication domain has a freely selectable name (e.g. `basic_auth_internal`), an `enabled` flag and an `order`. This makes it possible to chain authentication domains together. Search Guard will execute them in the order provided.
The `http_authenticator` specifies which authentication method you want to use on the HTTP layer.
The syntax for defining an authenticator on the HTTP layer is:
```
http_authenticator:
type: <type>
challenge: <true|false>
config:
...
```
Allowed values for `type` are:
* basic
* HTTP basic authentication. No additional configuration is needed. See [HTTP Basic Authentication](httpbasic.md) for further details.
* kerberos
* Kerberos authentication. Additional, [Kerberos-specific configuration](kerberos.md) is needed.
* jwt
* JSON web token authentication. Additional, [JWT-specific configuration](jwt.md) is needed.
* proxy
* Use an external, proxy based authentication. Additional, proxy-specific configuration is needed, and the "X-forwarded-for" module has to be enabled as well. See [Running Search Guard behind a proxy](proxyies.md) and [Proxy authentication](proxy_auth.md) for further details.
* clientcert
* Authentication via a client TLS certificate. This certificate must be trusted by one of the Root CAs in the truststore of your nodes.
After the HTTP authenticator was executed, you need to specify against which backend system you want to authenticate the user. This is specified in the `authentication_backend` section and has the following format:
```
authentication_backend:
type: <type>
config:
...
```
Possible vales for `type` are:
* noop
* This means that no further authentication against any backend system is performed. Use `noop` if the HTTP authenticator has already authenticated the user completely, as in the case of JWT, Kerberos, Proxy or Client certificate authentication.
* internal
* use the users and roles defined in `sg_internal_users` for authentication.
* ldap
* authenticate users against an LDAP server. This requires [additional, LDAP specific](ldap.md) configuration settings.
## Authorization
After the user has been authenticated, Search Guard can optionally collect additional user roles from backend systems. The authorization configuration has the following format:
```
authz:
<name>:
enabled: <true|false>
authorization_backend:
type: <type>
config:
...
```
You can also define multiple entries in this section the same way as you can for authentication entries. The execution order is not relevant here, hence there is no `order` field.
Possible vales for `type` are:
* noop
* Used for skipping this step altogether
* ldap
* Fetch additional roles from an LDAP server. This requires [additional, LDAP specific](ldap.md) configuration settings.
## Examples