Skip to main content
All CollectionsBuild AutomationsWorkflowsData
Advanced JSONPath and Golang Templates
Advanced JSONPath and Golang Templates

Learn how to efficiently retrieve and manipulate data in Torq with advanced JSONPath and Golang templates.

Updated over a week ago

This tutorial describes the advanced mechanisms available in Torq to retrieve and process the data provided by event triggers and workflow steps.

JSONPath is the notation used in Torq to access the data, and, using the advanced syntax explained below, it can also process complex data (such as a collection of objects, for example) to provide just the relevant part. Golang templates syntax can be used for more advanced data manipulation.

All data processing operations are also available as no-code steps.

All examples below refer to this collection of data: 

{
   "hosts":[
      {
         "name":"host1",
         "interfaces":[
            {
               "name":"Interface 1 of Host 1",
               "address":"10.10.10.1",
               "weight":10
            },
            {
               "name":"Interface 2 of Host 1",
               "address":"10.10.10.2",
               "weight":20
            }
         ]
      },
      {
         "name":"host2",
         "interfaces":[
            {
               "name":"Interface 1 of Host 2",
               "address":"20.20.20.1",
               "weight":10
            },
            {
               "name":"Interface 2 of Host 2",
               "address":"20.20.20.2",
               "weight":20
            },
            {
               "name":"Interface 3 of Host 2",
               "address":"20.20.20.3",
               "weight":30
            }
         ]
      }
   ]
}

Handling Arrays and Collections

The workflow context can contain collections of several variables, for example, lists of users, devices, vulnerabilities, and more. In order to process the data stored in these, Golang templates offer these operators:

In Torq, all collections are zero-based, meaning that the first member of the collection has an index of 0 (and not 1).

  • len: returns the length (number of items) of the array provided as an argument.

    • For the collection above, {{ len $.hosts }} will return 2.

  • index: returns a single item from a map or an array. The provided index can either be a numerical location of an item in an array or a named value inside a map. The index function can be used more than once in a single template in order to address a field of an object inside a collection.

    • {{ (index (index $.hosts 0).interfaces 0).name }} will return the name of the first interface of the first host in the collection. For the collection above, the result will be Interface 1 of Host 1.

  • []: returns either a single item or a range of items. {{ $.hosts[0] }} returns the same as {{ index $.hosts 0 }}. It can be used across fields {{ $.hosts[0].name }} and inside collections {{ $.hosts[0].interfaces[0].name }}.

    • For the collection above, {{ $.hosts[0].interfaces[0].name }} will return: Interface 1 of Host 1.

Accessing and Slicing Arrays and Collections

Torq supports the following JSONPath expressions for performing operations on arrays/collections. Reminder: Indexes start at 0.

Expression

Description

Example

[n]

Returns the n-th element in the array.

$.hosts[0].interfaces[0].name will return Interface 1 of Host 1

[x, y]

Returns a list consisting of the array items in indexes x and y.

$.hosts[1].interfaces[1,2][0].name will return Interface 2 of Host 2

[:x]

Returns a list consisting of the first x items of the array.

$.hosts[1].interfaces[:2][1].name will return Interface 2 of Host 2

[-x:]

Returns a list consisting of the last x items of the array.

$.hosts[1].interfaces[-2:] [1].name will return Interface 3 of Host 2

[? (expression)]

Returns a list consisting of all elements in an array that match the specified expression.

$.hosts[1].interfaces[?(@.weight > 10)][0] will return Interface 2 of Host 2

[:].field_name

Returns a new list where each element is a sub-element/field from the corresponding member of the original list.

$.get_aws_auto_scaling_groups.AutoScalingGroups[:] .AutoScalingGroupName will return a list of auto-scaling group names.

Apply Filters to Arrays and Collections

  • Filtering complete objects: the expression $.hosts[1].interfaces[?(@.weight > 10)] will return a list of interfaces where the value of the weight key exceeds 10.

  • Retrieving specific fields: the expression $.hosts[1].interfaces[?(@.weight > 10)].name will return a list of names rather than a list of complete interface objects. For the example above, $.hosts[1].interfaces[?(@.weight > 10)][0] will return Interface 2 of Host 2.

  • Matching by JavaScript Regular Expression: the expression $.hosts[1].interfaces[?(@.name=~ /(Interface [23])/)] will return a list of interfaces that contains interfaces 2 and 3.

  • Matching by multiple expressions: can be performed by combining expressions using && (logical AND) and || (logical OR) operators. For example, the expression $.hosts[1].interfaces[?(@.weight > 20 && @.name=~ /(Interface [23])/)] will return a list containing only Interface 3 of Host 2.

Comparisons

Comparison operators are useful either when defining a conditional execution of a step or when a step argument value depends on a specific condition. Each of the below is a boolean comparison operator:

  • eq - Returns the boolean truth of arg1 == arg2

  • ne - Returns the boolean truth of arg1 != arg2

  • lt - Returns the boolean truth of arg1 < arg2

  • gt - Returns the boolean truth of arg1 > arg2

  • ge - Returns the boolean truth of arg1 >= arg2

  • ieq - Returns the case-insensitive string comparison of arg1 and arg2 (==)

  • ine - Returns the negation of a case-insensitive string comparison of arg1 and arg2 (!=)

  • isNull - Returns true if the operand arg1 is Null (the actual word Null, not an empty argument)

  • empty - Returns true when a JSON path is not defined or present in a step output

For example, if using these comparisons with the collection provided above:

{{ eq (len $.hosts) 2 }} and {{ ge (len $.hosts) 0 }} will return true, whereas {{ isNull $.hosts }} or {{ lt (len $.hosts[0].interfaces) 1}} will return false.

Logical Operators

Logical operators allow executing boolean logic using execution context fields/variables as arguments. Here are examples of such cases:

  • and: returns the boolean AND of its arguments by returning the first empty argument or the last argument, that is, "and x y" behaves as "if x then y else x". All the arguments are evaluated.

    • For example: {{ and (eq $.event.Scope_Action "Firing") (eq $.event.Scope_Env "Production") }}

  • or: returns the boolean OR of its arguments by returning the first non-empty argument or the last argument, that is, "or x y" behaves as "if x then x else y". All the arguments are evaluated.

    • For example: {{ or (eq $.event.Scope_Action "Firing") (eq $.event.Scope_Action "Warning") }}

  • not: returns the boolean negation of its single argument.

    • For example: {{ not (eq $.event.Scope_Action "Firing") }}

Short-Circut Evaluation

Don't count on short-circuit evaluation, therefore, if one of the conditions can be in a state that prevents its evaluation, advanced conditional structures should be used. For example, in the case of an array that can be empty, the following clause will not work: {{ and (gt (len $.array) 0) (eq $.array[1] 8)}}, because when the array is empty, the index evaluation will fail.

Useful Template Functions

The table below summarizes a number of useful functions that can be used inside Torq as templates. All of these capabilities are also available as full no-code steps. The usage of template functions is an advanced low-code capability that can make the workflow more concise.

Function

Description

Example

upper / lower

Converts a string to uppercase/lowercase

{{ upper $.somestep.somevalue}}
{{ lower $.somestep.somevalue}}

nospace

Removes all whitespaces from a string

{{ nospace $.somestep.somevalue}}

add / sub / mul / div

Addition/subtraction/multiplication/division. Requires numeric values (doesn't operate on strings).

{{ add $.somestep.num 1 }}

toPrettyJson

Prints out the JSON data structure in a tabulated visual format that is easy to read

{{ toPrettyJson $.somestep.data }}

b64enc / b64dec

Encode (or decode) the provided data using Base64 encoding

{{ b64enc $.somestep.data }}

int / toString

Converts the provided data to an integer or to a string

{{ int $.somestep.num_in_string }}

trim / trimAll / trimSuffix / trimPrefix

Trim removes spaces from either side of the string, whereas trimAll can remove specific characters. Similarly, trimSuffix and trimPrefix remove just the matching side of the string.

{{ trimSuffix "-" $.somestep.some_string }}

replace

Performs substring replacement within a given string. Receives three arguments: the string to replace, the string to replace with, and the source string.

{{ replace " " "-" $.somestep.string_data }}

substr

Returns a substring from a provided string. It receives three arguments: the substring start index, the substring end index, and the original string.

{{ substr 5 12 $.somestep.string_data }}

now / date

Now returns the current timestamp, while date converts a timestamp into a format of choice (according to Golang time formatting).

{{ now }}
{{ now | date "2006-01-02" }}

snakecase / camelcase / kebabcase

Converts a given string to snake_case, camelCase, or kebab-case.

{{ snakecase $.somestep.some_string }}

trunc

Truncates a string by keeping the first X characters from the beginning.

{{ trunc 5 $.somestep.string}}

jsonEscape

Escape whitespaces and special characters to keep the data in JSON format.
​

{{ jsonEscape $.somestep.string}}
​

default

If the evaluated value is empty, the given default will be returned.

{{ default "foo" $.event.param1 }}

empty

Returns true if value is empty, and false otherwise.

{{ empty $.event.param1 }}

ternary

Takes two values and a thrid test value. If the test value is true, the first of the two values will be returned. If the test value is empty, the second of the two values will be returned.

{{ ternary "foo" "bar" true }}

Advanced Templates

Torq supports advanced templates that generate output (text, markdown, script) that is dynamic and gets filled by the data coming from the workflow context.

The advanced syntax described in the Golang Template package allows iterating over collections of data/objects using the range function.
For example, you can create an advanced text template that would mention the number of hosts and list their names.

Template:

We currently have {{ len $.hosts }} hosts and their names are: {{ range $index, $host:= $.hosts }} {{ $host.name }} {{ end }}

The above template would produce the following text:

We currently have 2 hosts and their names are: host1 host2

The range template presented above performs the following actions:

  1. Creates text lines for each member of $.hosts (will not create any if it's empty).

  2. Defines two temporary variables that can be referenced inside the template (note that they don't have an extra dot between the $sign and the variable name):

    • $index will contain the index of the object in the map/array.

    • $host will contain the actual object.

  3. For each host, prints its name field.

Another example is an advanced template using an If/else example:

{{ if (gt (len $.hosts) 1) -}} Multiple hosts found {{- else -}} Single host found {{- end}}

If there are multiple hosts, the above template will produce the following:

Multiple hosts found

If there is only one host found in the array, the template will produce: 

Single host found

Use $index to Improve the Template Verbosity

You can modify the template to create a more verbose output.

Template with improved verbosity:

We currently have {{ len $.hosts }} hosts and their names are: {{ range $index, $host:= $.hosts }} The name of host {{$index}} is {{ $host.name }} {{ end }}

Text output:

We currently have 2 hosts and their names are: The name of host 0 is host1 The name of host 1 is host2

As seen from the example, the index is zero-based.

Nested Range Constructs

The principle of nesting range constructs can be applied to arrays within arrays (such as our interfaces, which are, in turn, located inside hosts).

Template using nested range constructs:

We currently have {{ len $.hosts }} hosts and their names are: {{ range $index, $host:= $.hosts }} ---------------------------------------------- The name of host {{$index}} is {{ $host.name }} It has {{ len $host.interfaces }} interfaces: {{ range $interfaceidx, $interface:= $host.interfaces }} Interface number {{ $interfaceidx }} is named "{{$interface.name}}" and has an IP address of {{$interface.address}} {{end}} ---------------------------------------------- {{ end }}

The above template will generate the following dynamic text:

Text output:

We currently have 2 hosts and their names are: ---------------------------------------------- The name of host 0 is host1 It has 2 interfaces: Interface number 0 is named "Interface 1 of Host 1" and has an IP address of 10.10.10.1 Interface number 1 is named "Interface 2 of Host 1" and has an IP address of 10.10.10.2 ---------------------------------------------- ---------------------------------------------- The name of host 1 is host2 It has 3 interfaces: Interface number 0 is named "Interface 1 of Host 2" and has an IP address of 20.20.20.1 Interface number 1 is named "Interface 2 of Host 2" and has an IP address of 20.20.20.2 Interface number 2 is named "Interface 3 of Host 2" and has an IP address of 20.20.20.3 -----------------------------------------------

The construct described above can be combined with functions and variables described in the previous sections to achieve additional data manipulation. For example, the variable containing an index of the range can be used to retrieve a matching member of a different array:

{{index $.some_other_step.list_of_responses $idx}}

In this example, the index function is used to retrieve a member from an array stored in $.some_other_step.list_of_responses, and the $idx variable contains the index (0,1,2, ...) of the retrieved member.

Troubleshooting

When using inline go functions such as (jsonEscape, replace, lower... ) with dynamic input not in your control, the input might include Torq templating such as {{ $. }} which might in some conditions, fail the go template function, which may result in errors such as:

evaluating exit step success template (3): error parsing template: 1:18628: function \"trimprefix\" not defined
failed evaluating env at key \"VALUE_STRING\". unexpected \",\" in operand (3): error parsing template: 1:13: unexpected \",\" in operand

In such cases, report to Torq Support and Developers team. As a workaround, it is possible to migrate the logic from the inline function to an Escape JSON string step, Replace in String step, or the Escape Curly Brackets step.

Escape a URL

Using the urlquery inline function in Torq templates simplifies the process to escape a URL without needing a dedicated step. urlquery can be used within any Torq step.

Untitled1.png

Untitled2.png
Related Articles
Array Utilities
Collect Torq Audit or Activity Logs - Workflow Template
IBM QRadar
Gather Torq Audit or Activity Logs - Workflow Template
Utils Utilities
Did this answer your question?