OP5 Log Analytics allows you to create alerts, i.e. monitoring queries. These are constant queries that run in the background and when the conditions specified in the alert are met, the specify action is taken.
For example, if you want to know when more than 20 „status:500” responscode from on our homepage appear within an one hour, then we create an alert that check the number of occurrences of the „status:500” query for a specific index every 5 minutes. If the condition we are interested in is met, we send an action in the form of sending a message to our e-mail address. In the action, you can also set the launch of any script.
Enabling the Alert Module¶
To enabling the alert module you should:
generate writeback index for Alert service:
/opt/alert/bin/elastalert-create-index --config /opt/alert/config.yaml
configure the index pattern for alert*
start the Alert service:
systemctl start alert
To create the alert, click the “Alerts” button from the main menu bar.
We will display a page with tree tabs: Create new alerts in „Create alert rule”, manage alerts in „Alert rules List” and check alert status „Alert Status”.
In the alert creation windows we have an alert creation form:
- Name - the name of the alert, after which we will recognize and search for it.
- Index pattern - a pattern of indexes after which the alert will be searched.
- Role - the role of the user for whom an alert will be available
- Type - type of alert
- Description - description of the alert.
- Example - an example of using a given type of alert. Descriptive field
- Alert method - the action the alert will take if the conditions are met (sending an email message or executing a command)
- Any - additional descriptive field.# List of Alert rules #
The “Alert Rule List” tab contain complete list of previously created alert rules:
In this window, you can activate / deactivate, delete and update alerts by clicking on the selected icon with the given alert: .
In the “Alert status” tab, you can check the current alert status: if it activated, when it started and when it ended, how long it lasted, how many event sit found and how many times it worked.
Also, on this tab, you can recover the alert dashboard, by clicking the “Recovery Alert Dashboard” button.# Type of the Alert module rules #
The various RuleType classes, defined in OP5-Log-Aalytics. An instance is held in memory for each rule, passed all of the data returned by querying Elasticsearch with a given filter, and generates matches based on that data.
- Any - The any rule will match everything. Every hit that the query returns will generate an alert.
- Blacklist - The blacklist rule will check a certain field against a blacklist, and match if it is in the blacklist.
- Whitelist - Similar to blacklist, this rule will compare a certain field to a whitelist, and match if the list does not contain the term.
- Change - This rule will monitor a certain field and match if that field changes.
- Frequency - his rule matches when there are at least a certain number of events in a given time frame.
- Spike - This rule matches when the volume of events during a given time period is spike_height times larger or smaller than during the previous time period.
- Flatline - This rule matches when the total number of events is under a given threshold for a time period.
- New Term - This rule matches when a new value appears in a field that has never been seen before.
- Cardinality - This rule matches when a the total number of unique values for a certain field within a time frame is higher or lower than a threshold.
- Metric Aggregation - This rule matches when the value of a metric within the calculation window is higher or lower than a threshold.
- Percentage Match - This rule matches when the percentage of document in the match bucket within a calculation window is higher or lower than a threshold.
Example of rules¶
Unix - Authentication Fail¶
num_events: 4 timeframe: minutes: 5 filter: - query_string: query: "program: (ssh OR sshd OR su OR sudo) AND message: \"Failed password\""
Windows - Firewall disable or modify¶
- query_string: query: "event_id:(4947 OR 4948 OR 4946 OR 4949 OR 4954 OR 4956 OR 5025)"
OP5 Log Analytics has a set of predefined set of rules and activities (called Playbook) that can be attached to a registered event in the Alert module. Playbooks can be enriched with scripts that can be launched together with Playbook.
To add a new playbook, go to the Alert module, select the Playbook tab and then Create Playbook
In the Name field, enter the name of the new Playbook.
In the Text field, enter the content of the Playbook message.
In the Script field, enter the commands to be executed in the script.
To save the entered content, confirm with the Submit button.
To view saved Playbook, go to the Alert module, select the Playbook tab and then Playbooks list:
To view the content of a given Playbook, select the Show button.
To enter the changes in a given Playbook or in its script, select the Update button. After making changes, select the Submit button.
To delete the selected Playbook, select the Delete button.
When creating an alert or while editing an existing alert, it is possible that the system will indicate the most-suited playbook for the alert. For this purpose, the Validate button is used, which starts the process of searching the existing playbook and selects the most appropriate ones.
OP5 Log Analytics allows you to estimate the risk based on the collected data. The risk is estimated based on the defined category to which the values from 0 to 100 are assigned.
Information on the defined risk for a given field is passed with an alert and multiplied by the value of the Rule Importance parameter.
To add a new risk Category, go to the Alert module, select the Risks tab and then Create Cagtegory.
Enter the Name for the new category and the category Value.
To view saved Category, go to the Alert module, select the Risks tab and then Categories list:
To view the content of a given Category, select the Show button.
To change the value assigned to a category, select the Update button. After making changes, select the Submit button.
To delete the selected Category, select the Delete button.
To add a new playbook, go to the Alert module, select the Playbook tab and then Create Playbook
In the Index pattern field, enter the name of the index pattern. Select the Read fields button to get a list of fields from the index. From the box below, select the field name for which the risk will be determined.
From the Timerange field, select the time range from which the data will be analyzed.
Press the Read valules button to get values from the previously selected field for analysis.
Next, you must assign a risk category to the displayed values. You can do this for each value individually or use the check-box on the left to mark several values and set the category globally using the Set global category button. To quickly find the right value, you can use the search field.
After completing, save the changes with the Submit button.
To view saved risks, go to the Alert module, select the Risks tab and then Risks list:
To view the content of a given Risk, select the Show button.
To enter the changes in a given Risk, select the Update button. After making changes, select the Submit button.
To delete the selected Risk, select the Delete button.
Risk calculation algorithms¶
The risk calculation mechanism performs the aggregation of the risk field values. We have the following algorithms for calculating the alert risk (Aggregation type):
- min - returns the minimum value of the risk values from selected fields;
- max - returns the maximum value of the risk values from selected fields;
- avg - returns the average of risk values from selected fields;
- sum - returns the sum of risk values from selected fields;
- custom - returns the risk value based on your own algorithm
Adding a new risk calculation algorithm¶
The new algorithm should be added in the
./elastalert_modules/playbook_util.py file in the
calculate_risk method. There is a sequence of conditional statements for already defined algorithms:
#aggregate values by risk_key_aggregation for rule if risk_key_aggregation == "MIN": value_agg = min(values) elif risk_key_aggregation == "MAX": value_agg = max(values) elif risk_key_aggregation == "SUM": value_agg = sum(values) elif risk_key_aggregation == "AVG": value_agg = sum(values)/len(values) else: value_agg = max(values)
To add a new algorithm, add a new sequence as shown in the above code:
elif risk_key_aggregation == "AVG": value_agg = sum(values)/len(values) elif risk_key_aggregation == "AAA": value_agg = BBB else: value_agg = max(values)
where AAA is the algorithm code, BBB is a risk calculation function.
Using the new algorithm¶
After adding a new algorithm, it is available in the GUI in the Alert tab.
To use it, add a new rule according to the following steps:
- Select the
customvalue in the
- Enter the appropriate value in the
The following figure shows the places where you can call your own algorithm:
Additional modification of the algorithm (weight)¶
Below is the code in the
calcuate_risk method where category values are retrieved - here you can add your weight:
#start loop by tablicy risk_key for k in range(len(risk_keys)): risk_key = risk_keys[k] logging.info(' >>>>>>>>>>>>>> risk_key: ') logging.info(risk_key) key_value = lookup_es_key(match, risk_key) logging.info(' >>>>>>>>>>>>>> key_value: ') logging.info(key_value) value = float(self.get_risk_category_value(risk_key, key_value)) values.append( value ) logging.info(' >>>>>>>>>>>>>> risk_key values: ') logging.info(values) #finish loop by tablicy risk_key #aggregate values by risk_key_aggregation form rule if risk_key_aggregation == "MIN": value_agg = min(values) elif risk_key_aggregation == "MAX": value_agg = max(values) elif risk_key_aggregation == "SUM": value_agg = sum(values) elif risk_key_aggregation == "AVG": value_agg = sum(values)/len(values) else: value_agg = max(values)
Risk_key is the array of selected risk key fields in the GUI.
A loop is made on this array and a value is collected for the categories in the line:
value = float(self.get_risk_category_value(risk_key, key_value))
Based on, for example, Risk_key, you can multiply the value of the value field by the appropriate weight. The value field value is then added to the table on which the risk calculation algorithms are executed.