Torq's Data Transformation operator streamlines JSON handling, ensuring seamless data interoperability for robust workflows. By leveraging AI, it simplifies the process of building data extractions and transformations for complex JSON structures, converting natural language prompts into JQ commands. These commands are executed on the input data during each workflow run, with AI involvement ending once your instructions are saved, ensuring consistent and reliable executions.
The Data Transformation operator leverages JQ, a high-performance language for processing and transforming JSON, to simplify complex transformations and accelerate workflow development. It supports the application of multiple piped transformation instructions — where the output of one instruction serves as the input for the next —streamlining multi-instruction transformations within a single operator. Users can create and edit these instructions without requiring JQ or programming expertise while retaining the flexibility to modify commands and leverage advanced JQ features.
With Torq's Data Transformation operator, you can perform a wide range of JSON transformations, including:
Mapping & Extraction
Renaming keys
Converting data types
Filtering & sorting
If/else conditions
Math functions
and much more!
The Data Transformation operator is available to organizations that have opted into Torq AI features.
As with Torq's other responsible AI products, no data inputted into the Data Transformation operator will be used for training.
How to Use
Break down complex data extraction and transformation tasks into smaller micro-transformations with step-by-step instructions. The operator combines these, simplifying output creation and allowing edits at any stage.
Add the Operator to the Canvas: Drag the Transform operator onto the canvas.
Set the Input: Pass the input from the workflow context and click Define Transformation.
The operator will not work with raw and/or pasted content.
Enter Your Prompt: Write your prompt in natural language in the transformation window that appears.
The Data Transformation Operator supports prompts in multiple languages, not just English.
When referencing a string, be sure to write case-sensitive.
When creating the prompt, you can click on
key:valuepairs in the input to reference them within the prompt (and the subsequent instructions/code).The more specific your prompt, the more accurate the command generated will be.
Generate and Review Output: Click the Generate icon and review the output JSON.
The output cannot be edited from within the transformation window.
Ctrl+Z undoes the last command created.
Edit JQ Commands: You can edit a JQ command directly by clicking the Code heading, hovering over the desired command, and selecting the edit icon.
Note that after editing the Code, the original natural language Instruction will no longer be editable, as it has been overwritten. The Code, however, will still be editable.
Edit Instructions: You can edit an instruction after generation by hovering over it and clicking the edit icon. Then click the Generate icon, and the Data Transformation operator will rewrite the generated code.
Add More Instructions: Enter additional prompts as needed and review the results. You can create as many instructions as you desire within one operator instance.
Save Your Transformation: Once your output is satisfactory, click Save.
If you do not click Save before navigating away from the transformation window, your prompts and generated transformation will not be saved.
Edit After Saving: After saving, you can edit the instructions by clicking the step and then Define Transformation. The transformation window will pop up again.
Save as a Custom Step: Optionally, frequently used data transformations can be saved as custom steps (see below).
The Data Transformation operator provides versatile options for data transformation, addressing most use cases comprehensively. For specific, targeted tasks, Torq's Utility steps offer an alternative solution.
Save as a Custom Step
Save your Data Transformation as a custom step to share and reuse throughout your workspace or organization.
If you generate a Data Transformation that could apply to multiple workflows or instances, you can save it as a custom step by accessing the three-dot menu at the top right of the operator.
The saved custom step will then be accessible in the Custom tab of your Steps Library.
You can share the custom step by sharing the workflow it is a part of, enabling its use across any workspace in your organization.
Example Use Case
The following example demonstrates a five-stage transformation of an audit log.
1. Extraction of Specific Properties
Extract the specified properties from each audit log entry in a list.
Use-Case | Prompt (Instruction) | Output Code (JQ) |
Extracting properties with specific key names from a JSON object. | get "department", "email", "action", "IP address", "location", and "eventID" | .[] | {email: .user.email, department: .user.department, action: .action, location: .deviceDetails.location} |
2. Sorting and Filtering
Remove audit log entries that match the specified value for the given key.
Use-Case | Prompt (Instruction) | Output Code (JQ) |
Filter data based on conditions. | remove results where"department" is equal to Engineering | .[] | select(.department != "Engineering") |
3. Data Aggregation
Group audit log entries by the specified keys and count the number of entries in each group.
Use-Case | Prompt (Instruction) | Output Code (JQ) |
Summarize data by calculating counts, sums, averages, or other aggregations. | group by "department", "email", and count how many actions each person took | group_by(.email, .department) | map({email: .[0].email, department: .[0].department, action_count: map(.action | length) | add}) |
4. String Manipulation
Extract data using string manipulations, including Regex support, and store the result as a new property.
Use-Case | Prompt (Instruction) | Output Code (JQ) |
Perform string matching on key values | extract from "email" the text before @ split the text into two using "." as a separator save the first string as a new property first_name | [.[] | .first_name = (.email | split("@")[0] | split(".")[0]) ] |
5. Modify JSON format
Modify the output format and include only the specified keys and values.
Use-Case | Prompt (Instruction) | Output Code (JQ) |
Restructure or modify the JSON format, such as renaming keys, changing the nesting of objects, or converting objects to arrays. | get results as an array where each element is a key:value pair where the key is first_name and value is action_count
extract key:values from each object | [.[] | {(.first_name): .action_count}]
.[] | to_entries[] | "\(.key):\(.value)” |
Additional Example Prompts
Prompt | Action | Output Code (JQ) |
extract vulnerabilities | Extraction | .vulnerabilities |
remove descriptions, weaknesses, configurations, references, and cvssData | Mapping | map(del(.cve.descriptions, .cve.weaknesses, .cve.configurations, .cve.references, .cve.metrics.cvssMetricV2[].cvssData)) |
map cvssMetricV2 fields as fields under the cve parent object and delete metrics | Mapping and Deletion | [.[] | .cve += (.cve.metrics.cvssMetricV2[0] // {}) | del(.cve.metrics)] |
if baseSeverity equal "HIGH" then change to "10" else if equal "MEDIUM" then change to "6" else if equal "LOW" then change to "3” | Conditional If/Else | [.[] | if .baseSeverity == "HIGH" then .baseSeverity = "10" elif .baseSeverity == "MEDIUM" then .baseSeverity = "6" elif .baseSeverity == "LOW" then .baseSeverity = "3" else . end] |
convert baseSeverity to number | Convert Data Type | .[] | .cve.baseSeverity |= tonumber |
add averageScore equal to the average of baseSeverity, exploitabilityScore and impactScore | Math | .[] | .averageScore = ((.baseSeverity + .exploitabilityScore + .impactScore) / 3) |
sort by averageScore | Sort | sort_by(.averageScore) |
Filter averageScore smaller than 8 | Filter | .[] | select(.averageScore < 8) |