The Deduplicate operator identifies entries with identical properties, ensuring unique and duplicate inputs are handled as intended. This improves automation efficiency and eliminates the need for manual deduplication logic.
The Deduplicate operator checks for duplicate inputs each time the workflow runs (workflow executions). It uses a specified backward-looking time range (e.g., past days, hours, or minutes) to determine if an input was previously submitted within that period.
With Torq's Deduplicate operator, you can perform a wide range of use cases, including:
Filter out duplicate trigger events, including:
Events burst handling
Repeated Failed Login Attempts
Phishing Email Reports
etc.
Deduplicate array content and object data, including:
Cross Product Aggregation
Repeated IOC Matches for Known Threat
Deduplicate based on enrichment data
Suppress repetitive step executions:
Avoid repeat notifications to users
Avoid reopening identical cases
and more!
How it Works
Use the Operator:
Add to Canvas: Drag the Deduplicate operator onto the canvas.
Choose Your Input: Define what input expression you want to define as unique.
The input can be anything:
an entire object or a specific field within an object
an entire array or a specific value within an array,
a value (string, integer, boolean, etc.)
etc.
You can define multiple expressions in your Input by clicking Add Expression. Note that these will be referenced in the AND capacity, which means that both the values must be present in the execution to be declared "Unique."
Select a Time Range: Define how far backward you want the operator to check for matching values.
The maximum evaluation period is 31 days.
After the designated period has passed, the uniqueness count will reset.
Pick Your Number of Executions: Specify how many instances of the input should be allowed to be defined as Unique across all the workflow executions for the given time range.
The default is 1, and the maximum is 1000 instances.
Build Your Branches: Steps on the Unique branch will execute for inputs classified as unique, including the input parameter, and all other values within the given object/array. Steps on the Duplicate branch will execute only for inputs classified as duplicates by the input parameter.
A General Example:
You specify in the input parameter IP address
192.168.1.1with a time range of2 daysand a number of executions of3.Every IP address that arrives that is not
192.168.1.1(i.e10.0.0.25, etc.)will be defined and treated as Unique.
On the first day, the workflow receives two events with the IP address
192.168.1.1, both of which are defined as unique and pass through to the Unique branch.On the second day, only the first event with the IP address
192.168.1.1is passed through the Unique branch; each following execution event of192.168.1.1is passed through the Duplicate branch.However, on the third day, the count will restart and allow
192.168.1.1through three times over the next2 days.
Use Cases
Below are some examples of use cases for the Deduplicate operator. Many other use cases are possible, as the operator is versatile and can be used skillfully within Loops, Switch, etc.
Triage Detections with Deduplication of IoCs
In the following use case, the Deduplicate operator is used in a workflow triggered by a Crowdstrike event of at least a medium or high severity. The Deduplicate operator ensures that Cases with the same information will not be created, and updates already existing Cases with repeating event information.
The Dedpulicate operator is then given the input {{$.extract_sha256_ioc_from_cs_detection.result}} with the time range of 5 minutes and the number of executions of 1.
If the event is unique to the last five minutes, the event information is then passed down the Unique branch, where the data is normalized, and a Case is opened and processed depending on the level of maliciousness.
If the event has already been sent within the last five minutes, the event information is passed down the Duplicate branch. The duplicate event is then added to a matching Case, and the information is added as a new event and new note.
Avoid Message Overload
In the following use case, the Deduplicate operator is used in an Assigned to a teammate triggered workflow. When a workflow is triggered upon a Case being assigned to a teammate, the Deduplicate operator checks to ensure that a message regarding that case has been sent to the assignee over the last hour. If not, a Slack message is sent. If a message has already been sent, no action is taken.
This use case is important to ensure communication is not repetitive and messages to teammates or users are not overwhelming.
