Use these utility steps to transform, format and present data in various formats. Below are examples of each step's input and output.
The steps are grouped into the following categories based on their primary functionality.
Category | Description |
Data conversion | Converts data from one format to another |
Data extraction | Cleans up and extracts meaningful text from unstructured sources |
Data formatting & representation | Makes human-readable or presentation-ready output |
Data transformation | Filters, modifies, or reshapes data |
Miscellaneous | Prints to |
Data conversion
Convert CEF to JSON
Converts an array of CEF records to an array of JSON objects.
Input
CEF:0|Security|Threat|1.0|100|Attack Detected|10|src=192.168.1.1 dst=10.0.0.1
Output
[
{
"deviceVendor": "Security",
"deviceProduct": "Threat",
"deviceVersion": "1.0",
"name": "Attack Detected",
"severity": 10,
"src": "192.168.1.1",
"dst": "10.0.0.1"
}
]
Convert CLF to JSON
Converts CLF logs to JSON objects.
Input
127.0.0.1 - user [22/Nov/2024:10:00:00 +0000] "GET /index.html HTTP/1.1" 200 1024
192.168.1.1 - user1 [22/Nov/2024:10:01:00 +0000] "POST /login HTTP/1.1" 401 512
10.0.0.1 - user2 [22/Nov/2024:10:02:00 +0000] "GET /dashboard HTTP/2.0" 200 2048
Output
{
"api_object": {
"output": [
{
"ip": "127.0.0.1",
"identity": "-",
"user": "user",
"timestamp": "22/Nov/2024:10:00:00 +0000",
"request": "GET /index.html HTTP/1.1",
"status": "200",
"size": 1024
},
{
"ip": "192.168.1.1",
"identity": "-",
"user": "user1",
"timestamp": "22/Nov/2024:10:01:00 +0000",
"request": "POST /login HTTP/1.1",
"status": "401",
"size": 512
},
{
"ip": "10.0.0.1",
"identity": "-",
"user": "user2",
"timestamp": "22/Nov/2024:10:02:00 +0000",
"request": "GET /dashboard HTTP/2.0",
"status": "200",
"size": 2048
}
],
"step_status": {
"code": 1,
"message": "",
"verbose": ""
}
}
}
Convert NDJSON to JSON
Converts newline-delimited JSON to JSON.
Input
{"name":"Alice","age":24}
{"name":"Bob","age":31}Output
[ {"name":"Alice","age":24}, {"name":"Bob","age":31} ]
Convert to JSON
Converts the output of a previously executed step to JSON format, using the jc CLI tool. This step accepts the following data types: XML, YAML, CSV, or the output of shell commands.
Input
name,age Alice,24 Bob,31
Output
[ {"name": "Alice", "age": 24}, {"name": "Bob", "age": 31} ]
JSON parser
Parses the given raw data as JSON and outputs it directly to the step context. Supports using output from generic Docker images as steps (e.g., aws-cli). If no input is provided, the parser will use the output from the previous step as input.
Data extraction
Extract text from PDF file
Extracts ASCII text from a PDf file. Extracted text is returned as ASCII text and a Base64-encoded string.
Output
{ "ascii_text": "This is the extracted text from the PDF.",
"base64": "JVBERi0xLjQKJc..."
}
Strip HTML tags
Strips HTML tags from the given input.
Input
<html>
<head>
<title>My Title</title>
</head>
<body>
<h1>My Heading</h1>
<img src="my_dog.jpg" alt="dogs.com" width="104" height="142">
</body>
</html>
Output
"result": "\n \n \n \n \n My Heading\n \n \n",
Data formatting & representation
Create ASCII table
Generates an ASCII or Markdown table and populates it with the given input.
Input
[{"name":"Alice", "age":"32"},{"name":"Bob", "age":"53"},{"name":"Jane", "age":"46"}]Output
+-----+-------+
| AGE | NAME |
+-----+-------+
| 32 | Alice |
| 53 | Bob |
| 46 | Jane |
+-----+-------+
Create Adaptive Card table
Creates a table element for Microsoft Adaptive Cards from JSON arrays.
Input
[{"name":"Alice","score":95},{"name":"Bob","score":82}]Output
{
"type": "Table",
"columns": ["name","score"],
"rows": [
["Alice",95],
["Bob",82]
]
}
Create CSV
Creates a CSV file from the given input.
Input
[
{"name": "Bob", "age": 54, "email": "bob@company.com"},
{"name": "Alice", "age": 45, "email": "alice@company.com"}
]
Output
"age","email","name"
54,"bob@company.com","Bob"
45,"alice@company.com","Alice"
Create HTML table
Creates HTML table from given input.
Input
[
{"name": "Bob", "age": 54, "email": "bob@company.com"},
{"name": "Alice", "age": 45, "email": "alice@company.com"}
]
Output
<table class="torq-table">
<thead><tr><th>age</th><th>email</th><th>name</th></tr></thead>
<tr><td>54</td><td>bob@company.com</td><td>Bob</td></tr>
<tr><td>45</td><td>alice@company.com</td><td>Alice</td></tr>
</table>
Create Markdown table
Generates a Markdown table and populates it with the given input.
Input
[{"name":"Alice", "age":"32"},{"name":"Bob", "age":"53"},{"name":"Jane", "age":"46"}]Output
| AGE | NAME |
|-----|--------|
| 32 | Alice |
| 53 | Bob |
| 46 | Jane |
Generate a Markdown table from a JSON dictionary
Generate a Markdown table where the first column is the key and the second column is the value. The input should be a JSON dictionary and not an array.
Input
{"name":"Alice","age":24}
Output
| Key | Value |
|------|-------|
| name | Alice |
| age | 24 |
Generate an HTML unordered list from JSON
Converts a JSON to an unordered HTML list.
Input
["apple","banana","cherry"]
Output
<ul>
<li>apple</li>
<li>banana</li>
<li>cherry</li>
</ul>
Data transformation
Data transformer
Parses given data using various UNIX shell utilities, such as sed, grep, jq, etc.
Miscellaneous
Add data to context
Adds the given JSON object or list to the flow's context.
Input
{"addresses": ["google.com", "192.168.1.5"]}
Print a message to stdout
Prints a message to stdout.