Loops let you repeat actions. Depending on the loop type, actions can continue until a certain condition is met over every item in a dataset or a set number of times.
When to Use Loops: Examples
In: Go through a dataset and do something with each item, like updating each IP address in a list.
Range: Repeat actions a certain number of times.
Until Break: Keep going until something specific happens or go through many pages of results (pagination).
You need to set up a condition and include the Break operator to stop Until Break loops.
Until Break loops can't execute iterations at the same time (no Parallel mode).
In and Range loops can do up to 50,000 iterations, while Until Break loops go up to 1,000. If you hit these limits, the loop stops with an error message. If you need more iterations, contact Torq support.
Understanding Loops: Key Concepts
Before you start using loops, here are some basics:
Loop Operator: To begin using loops, drag the Loop operator onto the workflow designer. Use the Collect and Break operators, accessible by hovering over the Loop operator, only within loops.
Mode: This tells you if the loop iterations are executed sequentially or in parallel.
Gathering Loop Iteration Results: Use the Collect operator:
Input: The information to collect for a single loop iteration. This can be a string, number, JSON object, or array.
Flatten results: If you're working with arrays and choose Yes, the operator combines all arrays into one. If not, each array stays separate.
Ending a Loop Early: Use the Break operator to stop a loop early based on certain conditions.
How to use a loop: examples
In: Loop on a dataset (Basic)
In: Loop on a dataset (Basic)
This workflow is triggered by a scheduled task that pulls threat intel data from Recorded Future daily. You can substitute Recorded Future with your threat intel service.
Loop Over IP Addresses
After extracting and de-duplicating the IP addresses retrieved from Recorded Future, loop over the list by providing it as input for the Loop operator: {{ $.unique_ips.result }}
Enrich each IP address with VirusTotal
The action performed on each IP address is IP enrichment using VirusTotal. Access the IP address by using {{ $.loop_value }}. Each iteration results in a separate output entry in the execution log.
In: Loop on a dataset: loop within a loop (Advanced)
In: Loop on a dataset: loop within a loop (Advanced)
Implementing a loop within a loop will require you to manipulate the execution options of the Loop operator so you can access your data and act on it.
This example will show how you can cross-check the IP addresses that came back benign with another platform, and check if the IP address is associated with any problematic domains.
Get information on the domain associated with the benign IP
The IP address that came back as not malicious can be run through an additional engine, in this case, AbuseIPDB. In the IP field, you can use {{ $.loop_value }} as that contains the IP address from the initial loop, which came back as not malicious at first glance.
AbuseIPDB provides the domain information for the IP address, which can be passed on to VirusTotal for a domain lookup.
Loop over the DNS records of a domain
VirusTotal returns all of the information about the domain, including the DNS records for the domain. You can loop over those records.
It’s important to note that you can change the loop name and Execution Options to help identify each loop and make it easy to use each loop's Key and Value in context.
Change the Key name to key_dns_record.
Change the Value name to value_dns_record.
Check if the domain is associated with other IPs
To act on the object you looped over, reference $.value_dns_record in subsequent steps.
You can now create a condition whereby if the type under $.value_dns_record is A, the IP address of that record is checked. If the IP address for the Type A record is the same as the initial IP address that was already checked by VirusTotal, do nothing, as there is no point in checking the same IP address twice.
However, if the IP address is different from the initial address, the IP address in the type A DNS record should be checked by VirusTotal.
After handling the type A DNS record, you can break the loop since, for the purposes of this tutorial, the assumption is there is only one type A record.
Range: Polling
Range: Polling
The following example shows how to use Torq to perform polling, i.e. waiting for a process to complete before moving on with your workflow. It is based on extracting and analyzing URLs from a suspected phishing email.
Since the analysis and the respective report can take some time to complete, you will need to implement a loop waiting for the results.
Submit extracted URLs for analysis
After extracting URLs from a suspicious email, you can loop over them. In the input for the loop, you should enter the results of the extraction utility step.
The action you want to take on each of these URLs is scanning with VirusTotal. Enter the value from the key:value pair of the loop step as the URL to scan. In this example, that is: {{ $.loop_2_value }}
Wait for analysis results retrieval to complete
Create another loop within this loop using the Range type. In this example, the loop runs a maximum of 20 times (or less if the results are retrieved sooner).
You will then pass the ID of the URL scan to a Get Analysis Results step. This attempts to retrieve the report for the URL that was submitted earlier.
If the analysis has been completed, you can check the results to determine if the URL is malicious. Otherwise, configure the workflow to wait a few seconds before checking again. Once that is completed, you can move back to the outer loop and check the next URL.
Until Break: loop for pagination
Until Break: loop for pagination
When you're getting large volumes of data and can divide it into pages, use the page size option to keep the data amounts manageable. If you have more data than one page can show, you'll get a next page token. This token points to the following page's data. Use this token in the next run to keep getting more pages. If this token is empty or not there, it means you've got all the data.
In the example below, the activity log is retrieved in pages of 10 entries.
Retrieve the first page of results and check whether pagination is required.
If pagination is required, add a loop and set the loop type to Until Break.
The action performed in each loop iteration is to retrieve the next page of results by specifying the next page token provided by the previous execution of the same step.
Use the Collect operator to accumulate the output of each execution of the step responsible for retrieving the relevant data.
After retrieving each page of results, check if there are more pages to retrieve.
If the next page token is empty or null, it means there are no additional pages to retrieve, and the loop should be broken (by using the Break operator). Otherwise, continue with the loop to retrieve the next page of results.
