Torq's Transform data operator streamlines JSON handling, ensuring seamless data interoperability for robust workflows. It uses JQ, a high-performance language for processing and transforming JSON, to perform the transformations during workflow executions.
A visual tool guides users in defining data transformations through multiple instructions, each representing a phase of the transformation, with real-time previews of each outcome. Instructions are piped, allowing one output to seamlessly feed into the next, enabling efficient, multi-instruction transformations within a single operator.
Powered by AI, the operator enables users to define transformations using natural language, which is automatically converted into JQ commands. Once the transformation is defined, the AI’s role ends, and the deterministic JQ code executes consistently in every workflow run. Users can create and edit JQ commands without programming expertise while retaining the flexibility to adjust the code and leverage advanced JQ features.
With Torq's Transform data operator, you can perform a wide range of JSON transformations, including:
Mapping & Extraction
Renaming keys
Converting data types
Filtering & sorting
If/else conditions
Math functions
and much more!
The Transform data operator is available to organizations that have opted into Torq AI features.
As with Torq's other responsible AI products, no data inputs into the Transform data operator will be used for training.
How to use
Add the operator: Drag and drop the Transform operator into your workflow.
Set the input: Pass the data from the workflow context, or paste a JSON, into the Input parameter
Define the transformation: Click Define Transformation to open the data transformation window.
Enter your prompt: Write your prompt in natural language in the centre pane of the transformation window that appears.
The Transform data operator supports prompts in multiple languages, not just English.
When referencing a string, note that this is case-sensitive.
When creating the prompt, you can click on
key:valuepairs in the input to reference them within the prompt (and in the subsequent instructions).The more specific your prompt, the more accurate the command generated will be.
Reference workspace or workflow-level data by using
.$to access the context, including metadata and workspace variables.For example: "Change the base to
{{ $.workspace_variables.example_number }}."After the prompt is processed, the Instructions generated by the prompt will reflect the value, not the reference. For the example above, the Instructions will be "Change the base to 100" after generation.
Generate and review output: Click the Generate icon to generate a preview in the output pane on the right.
The output cannot be edited from within the transformation window.
Ctrl+Z undoes the last command created.
Edit JQ commands: You can edit a JQ command directly by clicking the Code tab, hovering over the desired command, and selecting the edit icon.
Edit the code and click the tick icon to confirm your changes. Click the x icon to discard edits.
After editing the code, the original natural language instruction in the Instructions tab will no longer be editable, as it has been overwritten. The code entries, however, will still be editable.
Edit instructions: You can edit an instruction by hovering over it and clicking the edit icon. Then click the Generate icon, and the Transform data operator will rewrite the generated code.
Add more instructions: Enter additional prompts as needed and review the results. You can create as many instructions as you desire within one operator instance.
Save your transformation: Once your output is satisfactory, click Save.
If you do not click Save before navigating away from the transformation window, your prompts and generated transformation will not be saved.
Execute the transformation: After saving, execute the operator to apply the transformation to the input data and utilize the output in subsequent workflow steps.
Edit after saving: After saving, you can edit the instructions by clicking the step and then Define Transformation. The transformation window will open again.
Save as a custom step: Optionally, frequently used data transformations can be saved as custom steps.
The Transform data operator offers versatile options for data transformation, comprehensively addressing most use cases. For specific, targeted tasks, Torq's Utility steps provide an alternative solution.
Save as a Custom Step
Save your Transform data as a custom step to share and reuse throughout your workspace or organization.
If you generate a Transform data that could apply to multiple workflows or instances, you can save it as a custom step by accessing the three-dot menu at the top right of the operator.
The saved custom step will then be accessible in the Custom tab of the Builderbox.
You can share the custom step by sharing the workflow it is a part of, enabling its use across any workspace in your organization.
Use case
The following example demonstrates a five-stage transformation of an audit log.
1. Extraction of specific properties
Extract the specified properties from each audit log entry in a list.
Use-Case | Prompt (Instruction) | Output Code (JQ) |
Extracting properties with specific key names from a JSON object. | get "department", "email", "action", "IP address", "location", and "eventID" | .[] | {email: .user.email, department: .user.department, action: .action, location: .deviceDetails.location} |
2. Sorting and filtering
Remove audit log entries that match the specified value for the given key.
Use-Case | Prompt (Instruction) | Output Code (JQ) |
Filter data based on conditions. | remove results where"department" is equal to Engineering | .[] | select(.department != "Engineering") |
3. Data aggregation
Group audit log entries by the specified keys and count the number of entries in each group.
Use-Case | Prompt (Instruction) | Output Code (JQ) |
Summarize data by calculating counts, sums, averages, or other aggregations. | group by "department", "email", and count how many actions each person took | group_by(.email, .department) | map({email: .[0].email, department: .[0].department, action_count: map(.action | length) | add}) |
4. String manipulation
Extract data using string manipulations, including Regex support, and store the result as a new property.
Use-Case | Prompt (Instruction) | Output Code (JQ) |
Perform string matching on key values | extract from "email" the text before @ split the text into two using "." as a separator save the first string as a new property first_name | [.[] | .first_name = (.email | split("@")[0] | split(".")[0]) ] |
5. Modify JSON format
Modify the output format and include only the specified keys and values.
Use-Case | Prompt (Instruction) | Output Code (JQ) |
Restructure or modify the JSON format, such as renaming keys, changing the nesting of objects, or converting objects to arrays. | get results as an array where each element is a key:value pair where the key is first_name and value is action_count
extract key:values from each object | [.[] | {(.first_name): .action_count}]
.[] | to_entries[] | "\(.key):\(.value)” |
Additional example prompts
Use-Case | Prompt (Instruction) | Output Code (JQ) |
Extraction | extract vulnerabilities | .vulnerabilities |
Mapping | remove descriptions, weaknesses, configurations, references, and cvssData | map(del(.cve.descriptions, .cve.weaknesses, .cve.configurations, .cve.references, .cve.metrics.cvssMetricV2[].cvssData)) |
Mapping and Deletion | map cvssMetricV2 fields as fields under the cve parent object and delete metrics | [.[] | .cve += (.cve.metrics.cvssMetricV2[0] // {}) | del(.cve.metrics)] |
Conditional If/Else | if baseSeverity equal "HIGH" then change to "10" else if equal "MEDIUM" then change to "6" else if equal "LOW" then change to "3” | [.[] | if .baseSeverity == "HIGH" then .baseSeverity = "10" elif .baseSeverity == "MEDIUM" then .baseSeverity = "6" elif .baseSeverity == "LOW" then .baseSeverity = "3" else . end] |
Convert Data Type | convert baseSeverity to number | .[] | .cve.baseSeverity |= tonumber |
Math | add averageScore equal to the average of baseSeverity, exploitabilityScore and impactScore | .[] | .averageScore = ((.baseSeverity + .exploitabilityScore + .impactScore) / 3) |
Sort | sort by averageScore | sort_by(.averageScore) |
Filter | Filter averageScore smaller than 8 | .[] | select(.averageScore < 8) |