Feature/docs 3628 customisable variability in new apid

docs/5.0/api-discovery/bola-protection.md

@@ -1 +1,15 @@
---8<-- "latest/api-discovery/bola-protection.md"
+# Automatic Protection Against BOLA Attacks <a href="../../about-wallarm/subscription-plans/#waap-and-advanced-api-security"><img src="../../images/api-security-tag.svg" style="border: none;"></a>
+
+Behavioral attacks such as [Broken Object Level Authorization (BOLA)](../attacks-vulns-list.md#broken-object-level-authorization-bola) exploit the vulnerability of the same name. This vulnerability allows an attacker to access an object by its identifier via an API request and either read or modify its data bypassing an authorization mechanism.
+
+Potential targets of the BOLA attacks are endpoints with variability. Wallarm can automatically discover and protect such endpoints among the ones explored by the [API Discovery](overview.md) module.
+
+To enable automatic BOLA protection, proceed to Wallarm Console → [**BOLA protection**](../admin-en/configuration-guides/protecting-against-bola.md) and turn the switch to the enabled state:
+
+![BOLA trigger](../images/user-guides/bola-protection/trigger-enabled-state.png)
+
+Each protected API endpoint will be highlighted with the corresponding icon in the API inventory, e.g.:
+
+![BOLA trigger](../images/about-wallarm-waf/api-discovery/endpoints-protected-against-bola.png)
+
+You can filter API endpoints by the BOLA auto protection state. The corresponding parameter is available under the **Others** filter.

docs/5.0/api-discovery/dashboard.md

@@ -3,4 +3,13 @@ search:
   exclude: true
 ---
 
---8<-- "latest/api-discovery/dashboard.md"
+[apid-overview]:                overview.md
+[apid-risk-score]:              risk-score.md
+[apid-track-changes]:           track-changes.md
+[apid-rogue]:                   rogue-api.md
+[check-attack]:                 ../user-guides/events/check-attack.md
+[img-api-discovery-widget]:     ../images/user-guides/dashboard/api-discovery-widget.png
+
+# API Discovery Dashboard <a href="../../../about-wallarm/subscription-plans/#waap-and-advanced-api-security"><img src="../../../images/api-security-tag.svg" style="border: none;"></a>
+
+--8<-- "../include/api-discovery/dashboard-5.0.md"

docs/5.0/api-discovery/exploring.md

@@ -1 +1,143 @@
---8<-- "latest/api-discovery/exploring.md"
+# Exploring API Inventory <a href="../../about-wallarm/subscription-plans/#waap-and-advanced-api-security"><img src="../../images/api-security-tag.svg" style="border: none;"></a>
+
+As soon as the [API Discovery](overview.md) module has built the catalog of your endpoints (your API inventory), you can explore it in the **API Discovery** section of Wallarm Console. Learn from this article how to go through the discovered data.
+
+## Endpoints
+
+Explore your discovered API inventory using the **API Discovery** section in the [US](https://us1.my.wallarm.com/api-discovery) or [EU](https://my.wallarm.com/api-discovery) Cloud.
+
+![Endpoints discovered by API Discovery](../images/about-wallarm-waf/api-discovery/discovered-api-endpoints.png)
+
+Each time you open the **API Discovery** section, you see all discovered endpoints and their [changes](track-changes.md) for the last week. With **Changes since** filter, you can change `Last week` to any other period.
+
+By default, endpoints are sorted by host/endpoint names (and grouped by hosts). If you sort by **Hits** or **Risk**, grouping goes away - to get back to the default, click hosts/endpoint column again.
+
+### External vs. internal
+
+The endpoints accessible from the external network are the main attack directions. Thus, it is important to see what is available from the outside and pay attention to these endpoints in the first place.
+
+Wallarm automatically splits discovered APIs to external and internal. The host with all its endpoints is considered to be internal if it is located on:
+
+* A private IP or local IP address
+* A generic top-level domain (for example: localhost, dashboard, etc.)
+
+In the remaining cases the hosts are considered to be external.
+
+By default, a list with all API hosts (external and internal) is displayed. In the built API inventory, you can view your internal and external APIs separately. To do this, click **External** or **Internal**.
+
+### Filtering
+
+Among a wide range of API endpoint filters, you can choose the ones corresponding to your analysis purpose, e.g.:
+
+* Only attacked endpoints that you can sort by the number of hits.
+* Find the most vulnerable endpoints characterized by processing sensitive data and active vulnerabilities of the high [risk level](risk-score.md). Exploiting vulnerabilities of a high risk level allows attackers to perform many malicious actions with the system including stealing sensitive data that the endpoint processes/stores.
+* Find [rogue endpoints](rogue-api.md): shadow, orphan and zombie.
+* Find the endpoints that have been changed or newly discovered in the last week and that process PII data. This kind of request can help you to stay up to date with critical [changes in your APIs](track-changes.md).
+* Find the endpoints being used to upload data to your server by the PUT or POST calls. Since such endpoints are a frequent attack target, they should be well secured. Using this kind of request you can check that endpoints are known to the team and are well secured from attacks.
+* Find the endpoints processing customers' bank card data. With this request, you can check that sensitive data is processed only by secured endpoints.
+* Find the endpoints of a deprecated API version (e.g. by searching `/v1`) and make sure that they are not used by clients.
+
+All filtered data can be exported in the OpenAPI v3 for additional analysis.
+
+## Endpoint details
+
+<a name="params"></a>By clicking the endpoint, you can also find the endpoint details, including request statistics, headers and parameters of requests and responses with the relevant data types:
+
+![Request parameters discovered by API Discovery](../images/about-wallarm-waf/api-discovery/discovered-request-params-4.10.png)
+
+Each request/response parameter information includes:
+
+* Parameter name and the part of request/response this parameter belongs to
+* Information about parameter changes (new, unused)
+* Presence and type of sensitive data transmitted by this parameter, including:
+
+    * Technical data like IP and MAC addresses
+    * Login credentials like secret keys and passwords
+    * Financial data like bank card numbers
+    * Medical data like medical license number
+    * Personally identifiable information (PII) like full name, passport number or SSN
+
+* [Type/format](#format-and-data-type) of data sent in this parameter
+* Date and time when parameter information was last updated
+
+!!! info "Availability of response parameters"
+    Response parameters are only available when using node 4.10.1 or higher.
+
+### Format and data type
+
+In the **Type** column, Wallarm indicates the data format identified through traffic analysis or, if not specific, a general data type.
+
+Wallarm attempts to detect various data formats such as `Int32`, `Int64`, `Float`, `Double`, `Datetime`, `IPv4`/`IPv6`, among others. If a value does not conform to any recognized data format, Wallarm classifies it under a general data type, such as `Integer`, `Number`, `String`, or `Boolean`.
+
+This data allows checking that values of the expected format are passed in each parameter. Inconsistencies can be the result of an attack or a scan of your API, for example:
+
+* The `String` values ​​are passed to the field with `IP`
+* The `Double` values are passed to the field where there should be a value no more than `Int32`
+
+### Variability
+
+URLs can include diverse elements, such as ID of user, like:
+
+* `/api/articles/author/author-a-0001`
+* `/api/articles/author/author-a-1401`
+* `/api/articles/author/author-b-1401`
+
+The **API Discovery** module unifies such elements into the `{parameter_X}` format in the endpoint paths, so for the example above you will not have 3 endpoints, but instead there will be one:
+
+* `/api/articles/author/{parameter_1}`
+
+Click the endpoint to expand its parameters and view which type was automatically detected for the diverse parameter.
+
+![API Discovery - Variability in path](../images/about-wallarm-waf/api-discovery/api-discovery-variability-in-path-4.10.png)
+
+Note that the algorithm analyzes the new traffic. If at some moment you see addresses, that should be unified but this did not happen yet, give it a time. As soon as more data arrives, the system will unify endpoints matching the newly found pattern with the appropriate amount of matching addresses.
+
+## Endpoint activities
+
+### Attacks
+
+Number of attacks on API endpoints for the last 7 days are displayed in the **Hits** column. You can request displaying only attacked endpoints by selecting in filters: **Others** → **Attacked endpoints**.
+
+To see attacks to some endpoint, click number in the **Hits** column:
+
+![API endpoint - open events](../images/about-wallarm-waf/api-discovery/endpoint-open-events.png)
+
+The **Attacks** section will be displayed with the [filter applied](../user-guides/search-and-filters/use-search.md):
+
+```
+attacks last 7 days endpoint_id:<YOUR_ENDPOINT_ID>
+```
+
+You can also copy some endpoint URL to the clipboard and use it to search for the events. To do this, in this endpoint menu select **Copy URL**.
+
+### All activities
+
+The number of all requests related to the endpoint is displayed in the **Requests** column. Click this number to open the [**API Sessions**](../api-sessions/overview.md) section with the list of user sessions for the last week with these requests.
+
+Within each found session, only requests to your endpoint will be initially displayed - in session, remove filter by endpoint to see all requests for context.
+
+A structured view of session activity helps in understanding your endpoint place in malicious and legitimate activities, its relation to sensitive business flows and required protection measures.
+
+## Creating rules for API endpoints
+
+You can quickly create a new [custom rule](../user-guides/rules/rules.md) from any endpoint of API inventory: 
+
+1. In this endpoint menu select **Create rule**. The create rule window is displayed. The endpoint address is parsed into the window automatically.
+1. In the create rule window, specify rule information and then click **Create**.
+
+![Create rule from endpoint](../images/about-wallarm-waf/api-discovery/endpoint-create-rule.png)
+
+## Exporting API inventory data
+
+The API Discovery UI provides you with an option to export the current filtered list of endpoints as the [OpenAPI v3](https://spec.openapis.org/oas/v3.0.0) specification or CSV file.
+
+To export, in Wallarm Console → **API Discovery**, use the **OAS/CSV** option. Consider the following:
+
+* For **OAS**, Wallarm returns the `swagger.json` with filtered endpoints. You can also use the **Download OAS** button in an individual endpoint menu
+
+    By utilizing the downloaded specification with other applications like Postman, you can conduct endpoints' vulnerability and other tests. In addition, it allows for a closer examination of the endpoints' capabilities to uncover the processing of sensitive data and the presence of undocumented parameters.
+
+* For **CSV**, Wallarm returns filtered endpoints data in a simple text comma-separated format, making it easy to export it into other programs.
+
+!!! warning "API host information in downloaded Swagger file"
+    If a discovered API inventory contains several API hosts, endpoints from all API hosts will be included in the downloaded file. Currently, the API host information is not included in the file.

docs/5.0/api-discovery/overview.md

@@ -1 +1,122 @@
---8<-- "latest/api-discovery/overview.md"
+# API Discovery Overview <a href="../../about-wallarm/subscription-plans/#waap-and-advanced-api-security"><img src="../../images/api-security-tag.svg" style="border: none;"></a>
+
+The **API Discovery** module of the Wallarm platform builds your application REST API inventory based on the actual API usage. The module continuously analyzes the real traffic requests and builds the API inventory based on the analysis results.
+
+The API built inventory includes the following elements:
+
+* API endpoints
+* Request methods (GET, POST, and others)
+* Required and optional GET, POST, and header parameters of requests and responses including:
+    * [Type/format](./exploring.md#format-and-data-type) of data sent in each parameter    
+    * Date and time when parameter information was last updated
+
+!!! info "Availability of response parameters"
+    Response parameters are only available when using node 4.10.1 or higher.
+
+<div>
+    <script src="https://js.storylane.io/js/v1/storylane.js"></script>
+    <div class="sl-embed" style="position:relative;padding-bottom:calc(60.95% + 27px);width:100%;height:0;transform:scale(1)">
+        <iframe class="sl-demo" src="https://wallarm.storylane.io/demo/cgqrxqwhmgyp" name="sl-embed" allow="fullscreen" style="position:absolute;top:0;left:0;width:100%!important;height:100%!important;border:1px solid rgba(63,95,172,0.35);box-shadow: 0px 0px 18px rgba(26, 19, 72, 0.15);border-radius:10px;box-sizing:border-box;"></iframe>
+    </div>
+</div>
+
+## Issues addressed by API Discovery
+
+**Building an actual and complete API inventory** is the main issue the API Discovery module is addressing.
+
+Keeping API inventory up-to-date is a difficult task. There are multiple teams that use different APIs and it is a common case that different tools and processes are used to produce the API documentation. As a result, companies struggle in both understanding what APIs they have, what data they expose and having up-to-date API documentation.
+
+Since the API Discovery module uses the real traffic as a data source, it helps to get up-to-date and complete API documentation by including to the API inventory all endpoints that are actually processing the requests.
+
+**As you have your API inventory discovered by Wallarm, you can**:
+
+* Have a full visibility into the whole API estate including the list of [external and internal](exploring.md#external-vs-internal) APIs.
+* See [what data](exploring.md#endpoint-details) is going into and out of the APIs.
+* Get a list of endpoints with the open vulnerabilities.
+* Get a list of the threats that occurred over the past 7 days per any given API endpoint.
+* Filter out only attacked APIs, sort them by number of hits.
+* Filter APIs that consume and carry [sensitive data](#sensitive-data-detection).
+* View visualized summary on your API inventory structure and problems on a handy [dashboard](dashboard.md).
+* Understand which endpoints are [most likely](risk-score.md) to be an attack target.
+* Find [shadow, orphan and zombie APIs](rogue-api.md).
+* [Track changes](track-changes.md) in API that took place within the selected period of time.
+* Filter API endpoints by the [BOLA auto protection state](bola-protection.md).
+* Provide your developers with [access](../user-guides/settings/users.md#user-roles) to the built API inventory reviewing and downloading.
+
+## How does API Discovery work?
+
+API Discovery relies on request statistics and uses sophisticated algorithms to generate up-to-date API specs based on the actual API usage.
+
+### Traffic processing
+
+API Discovery uses a hybrid approach to conduct analysis locally and in the Cloud. This approach enables a [privacy-first process](#security-of-data-uploaded-to-the-wallarm-cloud) where request data and sensitive data are kept locally while using the power of the Cloud for the statistics analysis:
+
+1. API Discovery analyzes legitimate traffic locally. Wallarm analyzes the endpoints to which requests are made and what parameters are passed and returned.
+1. According to this data, statistics are made and sent to the Cloud.
+1. Wallarm Cloud aggregates the received statistics and builds an [API description](exploring.md) on its basis.
+
+    !!! info "Noise detection"
+        Rare or single requests are [determined as noise](#noise-detection) and not included in the API inventory.
+
+### Noise detection
+
+The API Discovery module bases noise detection on the two major traffic parameters:
+
+* Endpoint stability - at least 5 requests must be recorded within 5 minutes from the moment of the first request to the endpoint.
+* Parameter stability - the occurrence of the parameter in requests to the endpoint must be more than 1 percent.
+
+The API inventory will display the endpoints and parameters that exceeded these limits. The time required to build the complete API inventory depends on the traffic diversity and intensity. 
+
+Also, the API Discovery performs filtering of requests relying on the other criteria:
+
+* Only those requests to which the server responded in the 2xx range are processed.
+* Requests that do not conform to the design principles of the REST API are not processed.
+
+    This is done by controlling the `Content-Type` header of responses: if it does not contain `application/json` (like `Content-Type: application/json;charset=utf-8`), the request is considered to be a non-REST API and is not analyzed.
+
+    If the header does not exist, API Discovery analyzes the request.
+
+* Standard fields such as `Accept` and alike are discarded.
+
+### Sensitive data detection
+
+API Discovery [detects and highlights](sensitive-data.md) sensitive data consumed and carried by your APIs:
+
+* Technical data like IP and MAC addresses
+* Login credentials like secret keys and passwords
+* Financial data like bank card numbers
+* Medical data like medical license number
+* Personally identifiable information (PII) like full name, passport number or SSN
+
+API Discovery provides the ability to configure the detection process and add your own sensitive data patterns (requires NGINX Node 5.0.3 or Native Node 0.7.0 or higher).
+
+### Security of data uploaded to the Wallarm Cloud
+
+API Discovery analyzes most of the traffic locally. The module sends to the Wallarm Cloud only the discovered endpoints, parameter names and various statistical data (time of arrival, their number, etc.) All data is transmitted via a secure channel: before uploading the statistics to the Wallarm Cloud, the API Discovery module hashes the values of request parameters using the [SHA-256](https://en.wikipedia.org/wiki/SHA-2) algorithm.
+
+On the Cloud side, hashed data is used for statistical analysis (for example, when quantifying requests with identical parameters).
+
+Other data (endpoint values, request methods, and parameter names) is not hashed before being uploaded to the Wallarm Cloud, because hashes cannot be restored to their original state which would make building API inventory impossible.
+
+!!! warning "Important"
+    Wallarm does not send the values that are specified in the parameters to the Cloud. Only the endpoint, parameter names and statistics on them are sent.
+
+## API Discovery demo video
+
+Watch API Discovery demo video:
+
+<div class="video-wrapper">
+  <iframe width="1280" height="720" src="https://www.youtube.com/embed/0bRHVtpWkJ8" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+## Checking API Discovery in playground
+
+To try the module even before signing up and deploying the node to your environment, explore [API Discovery in Wallarm Playground](https://playground.wallarm.com/api-discovery/?utm_source=wallarm_docs_apid).
+
+In Playground, you can access the API Discovery view like it is filled with real data and thus learn and try out how the module works, and get some useful examples of its usage in the read-only mode.
+
+![API Discovery – Sample Data](../images/about-wallarm-waf/api-discovery/api-discovery-sample-data.png)
+
+## Enabling and configuring API Discovery
+
+To start using API Discovery, enable and configure it as described in [API Discovery Setup](setup.md).