Guide - Transforms and pipelines

The term 'pipeline' represents a simple idea: transform data by chaining steps where each step does one thing. Flow represents pipelines graphically - every step represents a transform and the connecting lines show how data flows from one step to the next.

Streamscript pipelines are self-contained inside one script, and use the pipe | symbol to join commands.

# Streamscript
$results = $external_companies
         | Filter $_.IsOverdue
         | Sort $_.Amount

Types of Transforms

Declarative transforms are about stating what you want in a way that closely resembles the requirement and allows Streamscript to work out how to do it - for example, the commands Sort, Reverse, Max and Copy are declarative.

Imperative transforms are about telling Streamscript how to perform the task. You supply a logic snippet (such as a filter condition) to fully express the intent of your transformation. The Filter and Project commands are imperative.

For example - assume we wanted the three highest values from a collection of EU tax rates:

$eu_rates = [
    20, 21, 20, 19, 21, 19, 25, 20, 24,
    21, 24, 20, 25, 27, 23, 22, 21, 17,
    21, 18, 21, 23, 23, 19, 25, 22, 20
]

Here is the pipeline transform using declarative Streamscript:

$top_rates = $eu_rates | Unique | Sort | Reverse | Top 3    # top rates: 27, 25, 24

Each command (Unique, Sort, etc) performs a specific operation on the data handed to it and then hands its output to the next command as indicated by the pipe | operation.

Here is the same example showing the results at each step of the transform:

$top_rates = $eu_rates # 20, 21, 20, 19, 21, 19, 25, 20, 24, 21, 24 ...
           | Unique    # 20, 21, 19, 25, 24, 27, 23, 22, 17, 18
           | Sort      # 17, 18, 19, 20, 21, 22, 23, 24, 25, 27
           | Reverse   # 27, 25, 24, 23, 22, 21, 20, 19, 18, 17
           | Top 3     # 27, 25, 24

'Top' is the final command. The result of the pipeline is assigned to the variable $top_rates.

$top_rates = ... | Top 3    # results of Top are assigned to $top_rates

All collection commands work as standalone calls. Each command has enough information to process the collection handed to it from the left. Some commands like 'Top' may take an optional parameter (in this example, the number of items to select)

$top_rates = ... | Top      # selects 1 item
$top_rates = ... | Top 3    # selects 3 items

Logic Parameters

These transforms accept a logic param ${...}

# Transforms that modify the list:
Sort ${...}    # Sort the list using the sort logic in ${...}
Apply ${...}   # Run the logic in ${...} on each item in the list

# Transforms that return a new list:
ToMap ${...}   # Return a map, keyed on the logic in ${...}
Filter ${...}  # Return only items that meet the condition in ${...}
Project ${...} # Return new list items transformed by the logic in ${...}

# Transforms that aggregate a result:
Sum ${...}     # Sum across each number retrieved by logic in ${...}
Min ${...}     # Return the lowest number retrieved by logic in ${...}
Max ${...}     # Return the highest number retrieved by logic in ${...}
Avg ${...}     # Average across each number retrieved by logic in ${...}

The logic param is a placeholder for your own business logic. For example, supply your Filter logic to exclude items, or supply your Sort logic to specify the order. In your logic param, use the special $_ variable to refer to the current item in the collection or pipeline.

Logic params can take the form of block logic, or concise logic. With concise logic, only a single value is specified, which becomes the implied return value. Block logic uses many statements followed by a return.

Example 1. Assume we want the high rates from the following EU tax rates:

$eu_taxes = [ 20, 21, 19, 25, 24, 27, 23, 22, 17, 18 ]

This transforms the input list filtering for items greater than 20:

$high_rates = $eu_taxes | Filter $_.gt(20)                # 21, 25, 24, 27, 23, 22

Your logic only needs a surrounding block ${...} when it consists more than one statement.

$high_rates = $eu_taxes | Filter           $_.gt(20)   # valid 
$high_rates = $eu_taxes | Filter ${ return $_.gt(20) } # also valid

Logic params are similar to JavaScript arrow functions.

Example 2. Consider this monthly expense feed:

$expenses = [
    {Code: 'Heat',  Amount: 1100}
    {Code: 'Rent',  Amount: 8800}
    {Code: 'Water', Amount: 9910}
]

Here are the required steps to turn this into a double-entry accounting journal.

Copy the collection - this creates a balancing item from every expense.
Apply the ledger code 'Bank' to each balancing item.
Negate the amount on each balancing item, using the Apply command.

# Steps 1, 2, 3:
$balancing = $expenses
           | Copy
           | Apply ${ $_.Code = 'Bank' }
           | Apply ${ $_.Amount = $_.Amount * -1 }

Finally:

Combine the expense list and balancing list - this forms a balanced journal.
Project the journal into a collection of Salesforce records.

# Step 4: combine all items
$items = $balancing + $expenses

# Step 5: convert into records (using map as logic)
$JournalLine__c = $items | Project {
    Symbol__c: 'USD'
    Code__c: $_.Code
    Amount__c: $_.Amount
    attributes: {type: 'JournalLine__c'}
}

Result: records (
    Symbol__c: USD,   Code__c: Bank,   Amount__c: -1100
    Symbol__c: USD,   Code__c: Heat,   Amount__c: 1100
    Symbol__c: USD,   Code__c: Bank,   Amount__c: -8800
    Symbol__c: USD,   Code__c: Rent,   Amount__c: 8800
    Symbol__c: USD,   Code__c: Bank,   Amount__c: -9910
    Symbol__c: USD,   Code__c: Water,   Amount__c: 9910
)

Pipes and loops can achieve the same result - use the approach that suits you best.

Special Variables

Reference the index (for a list) or the key (for a map) using the special $_ and $__ variables:

In the context of a list:

$_ represents the item value,
$__ represents the item index.

In the context of a map:

$_ represents the map value,
$__ represents the map key.

Example 3. Consider a webhook API that transforms a JSON shopping cart into a list of opportunity line items and preserves the order of the original list.

$cart_items = Json-Decode `[
    {
        "qty": 1,
        "price": 1000,
        "name": "Product 1",
        "description": "This is the first product"
   },
   {
        "qty": 2,
        "price": 2000,
        "name": "Product 2",
        "description": "This is the second product"
    }
]`

Use Project to:

assert each item's original shopping cart order,
divide each item's price by 100 to convert from cents to dollars,
set the SObject type on each item, so that the returned data is recognised as a collection of records when inserted later in the flow.

# convert to opportunity line items (using map as logic)
$OpportunityLineItem = $cart_items | Project {
    Name: $_.name
    SortOrder: $__
    Quantity: $_.qty
    UnitPrice: $_.price / 100
    Description: $_.description
    attributes: {type: 'OpportunityLineItem'}
}

# output record collection
return $OpportunityLineItem

Result: records (
    UnitPrice: 10,   Quantity: 1,   SortOrder: 0,   Name: Product 1,   Description: This is the first product
    UnitPrice: 20,   Quantity: 2,   SortOrder: 1,   Name: Product 2,   Description: This is the second product
)

These commands work with lists, maps, and nested combinations of collections.

Transform Tips

To sort a list of rich data structures, specify the sort field(s) using the special $_ variable:

$list_applications = [
   { Course: 'Postgrad',  Student: {Score: 80, Name: 'Tom'} }
   { Course: 'Undergrad', Student: {Score: 60, Name: 'Liz'} }
   { Course: 'Postgrad',  Student: {Score: 70, Name: 'Jim'} }
   { Course: 'Undergrad', Student: {Score: 90, Name: 'Bob'} }
]

# primary sort then secondary sort (with nested attribute)
$by_course_score = $list_applications | Sort [ $_.Course, $_.Student.Score ]

Result: [
    { Course: 'Postgrad',   Student: {Score: 70, Name: 'Jim'} }
    { Course: 'Postgrad',   Student: {Score: 80, Name: 'Tom'} }
    { Course: 'Undergrad',   Student: {Score: 60, Name: 'Liz'} }
    { Course: 'Undergrad',   Student: {Score: 90, Name: 'Bob'} }
]

To convert a list to a map, use the ToMap command - the sequential index from the list will be used as the map key for each item. Conversely, to convert a map to a list, use the ToList command to discard the map keys.

$map_applications = $list_applications | ToMap

Result: {
    '0': { Course: 'Postgrad',   Student: {Score: 80, Name: 'Tom'} }
    '1': { Course: 'Undergrad',   Student: {Score: 60, Name: 'Liz'} }
    '2': { Course: 'Postgrad',   Student: {Score: 70, Name: 'Jim'} }
    '3': { Course: 'Undergrad',   Student: {Score: 90, Name: 'Bob'} }
}

The Sum, Avg, Min and Max commands are compatible across lists of primitives, lists of maps, or nested values - in any case, specify the aggregation field using the special $_ variable.

$average = $map_applications  | Avg $_.Student.Score # 75
$average = $list_applications | Avg $_.Student.Score # 75