Transforms and Pipelines

The term 'pipeline' represents a simple idea: transform data by chaining commands that each do one thing really well.

Tools such as Flow present a pipeline graphically - each element on the canvas represents a command and the line between elements represents the pipeline. Pipelines work because the output of one element matches the input format of the next.

Streamscript offers a finer grain of pipeline. Streamscript pipelines are self-contained in a single Flow step, and use the pipe | symbol to join commands.

# Streamscript
$results = $external_companies
         | Filter $_.IsOverdue
         | Sort $_.Amount

Types of Transforms

It is important to understand the difference between declarative and imperative transforms.

Declarative transforms are about stating what you want in a way that closely resembles the requirement and allows Streamscript to work out how to do it - for example, the commands Sort, Reverse, Max and Copy are declarative.

Imperative transforms are about telling Streamscript how to perform the task. You supply a logic snippet (such as a filter condition) to fully express the intent of your transformation. The Filter and Project commands are imperative.

For example - assume we wanted the three highest values from a collection of EU tax rates:

$eu_rates = [
    20, 21, 20, 19, 21, 19, 25, 20, 24,
    21, 24, 20, 25, 27, 23, 22, 21, 17,
    21, 18, 21, 23, 23, 19, 25, 22, 20
]

Here is the pipeline transform using declarative Streamscript:

$top_rates = $eu_rates | Unique | Sort | Reverse | Top 3    # top rates: 27, 25, 24

Each command (Unique, Sort, etc) performs a specific operation on the data handed to it and then hands its output to the next command as indicated by the pipe | operation.

Here is the same example showing the results at each step of the transform:

$top_rates = $eu_rates # 20, 21, 20, 19, 21, 19, 25, 20, 24, 21, 24 ...
           | Unique    # 20, 21, 19, 25, 24, 27, 23, 22, 17, 18
           | Sort      # 17, 18, 19, 20, 21, 22, 23, 24, 25, 27
           | Reverse   # 27, 25, 24, 23, 22, 21, 20, 19, 18, 17
           | Top 3     # 27, 25, 24

'Top' is the final command. The result of the pipeline is assigned to the variable $top_rates.

$top_rates = ... | Top 3    # results of Top are assigned to $top_rates

All collection commands work as standalone calls. Each command has enough information to process the collection handed to it from the left. Some like 'Top' also take an optional parameter (in this example, the number of items to select)

$top_rates = ... | Top      # selects 1 item
$top_rates = ... | Top 3    # selects 3 items

Logic Parameters

These transforms accept a logic param ${...}

# Transforms that modify the list:
Sort ${...}    # Sort the list using the sort logic in ${...}
Apply ${...}   # Run the logic in ${...} on each item in the list

# Transforms that return a new list:
ToMap ${...}   # Return a map, keyed on the logic in ${...}
Filter ${...}  # Return only items that meet the condition in ${...}
Project ${...} # Return new list items transformed by the logic in ${...}

# Transforms that aggregate a result:
Sum ${...}     # Sum across each number retrieved by logic in ${...}
Min ${...}     # Return the lowest number retrieved by logic in ${...}
Max ${...}     # Return the highest number retrieved by logic in ${...}
Avg ${...}     # Average across each number retrieved by logic in ${...}

The logic param is a placeholder for your own business logic. For example, supply your Filter logic to exclude items, or supply your Sort logic to specify the order. In your logic param, use the special $_ variable to refer to the current item in the collection or pipeline.

Logic params can take the form of block logic, or concise logic. With concise logic, only a single value is specified, which becomes the implied return value. Block logic uses many statements followed by a return.

For example - assume we want the high rates from the following EU tax rates:

$eu_taxes = [ 20, 21, 19, 25, 24, 27, 23, 22, 17, 18 ]

This transforms the input list filtering for items greater than 20:

$high_rates = $eu_taxes | Filter $_.gt(20)                # 21, 25, 24, 27, 23, 22

Your logic only needs a surrounding block ${...} when it consists more than one statement.

$high_rates = $eu_taxes | Filter           $_.gt(20)   # valid 
$high_rates = $eu_taxes | Filter ${ return $_.gt(20) } # also valid

Logic params are similar to JavaScript arrow functions.

Here is a more complex example - given a monthly feed of expenses, assume you wish to transform them into an accounting journal (double entry)

$expenses = [
    {Code: 'Heat',  Amount: 1100}
    {Code: 'Rent',  Amount: 8800}
    {Code: 'Water', Amount: 9910}
]

Here is the transform:

1. Copy to create a new balancing item from every expense item.
2. Apply a default ledger code to each balancing item.
3. Apply the reverse sign on each balancing item (so the journal sums to zero).

# Steps 1, 2, 3:
$balancing_items = $expenses
         | Copy
         | Apply ${ $_.Code = 'Bank' }
         | Apply ${ $_.Amount = $_.Amount * -1 }

Finally:

4. Combine the expense items and the balancing items to compose a balanced journal.
5. Project each item to convert it into a Salesforce journal line record.

Try# Step 4: combine all items
$items = $balancing_items + $expenses

# Step 5: convert into records (using map as logic)
$JournalLine__c = $items | Project {
    Symbol__c: 'USD'
    Code__c: $_.Code
    Amount__c: $_.Amount
    attributes: {type: 'JournalLine__c'}
}

Result: records (
    Symbol__c: USD,   Code__c: Bank,   Amount__c: -1100
    Symbol__c: USD,   Code__c: Heat,   Amount__c: 1100
    Symbol__c: USD,   Code__c: Bank,   Amount__c: -8800
    Symbol__c: USD,   Code__c: Rent,   Amount__c: 8800
    Symbol__c: USD,   Code__c: Bank,   Amount__c: -9910
    Symbol__c: USD,   Code__c: Water,   Amount__c: 9910
)

A loop delivers the same result - use the approach that suits you best.

Special Variables

When transforming lists and maps, it is often useful to reference the index (for a list) or the key (for a map). Streamscript provides the special $_ and $__ variables to help with this:

In the context of a list:

• $_ represents the item value,
• $__ represents the item index.

In the context of a map:

• $_ represents the map value,
• $__ represents the map key.

For example, let's say we receive JSON data from a webhook API containing a list of shopping cart items, and wish to transform them into opportunity line items explicitly preserving the order of each original item from the shopping cart.

$cart_items = Json-Decode `[
    {
        "qty": 1,
        "price": 1000,
        "name": "Product 1",
        "description": "This is the first product"
   },
   {
        "qty": 2,
        "price": 2000,
        "name": "Product 2",
        "description": "This is the second product"
    }
]`

We will use the Project transform:

• to assert each item's order as it originally appeared in the shopping cart,
• to divide each item's price by 100 to convert from cents to dollars,
• to set the SObject type on each item, so that the returned data is recognised as a collection of records when inserted later in the flow.

Try# convert to opportunity line items (using map as logic)
$OpportunityLineItem = $cart_items | Project {
    Name: $_.name
    SortOrder: $__
    Quantity: $_.qty
    UnitPrice: $_.price / 100
    Description: $_.description
    attributes: {type: 'OpportunityLineItem'}
}

# output record collection
return $OpportunityLineItem

Result: records (
    UnitPrice: 10,   Quantity: 1,   SortOrder: 0,   Name: Product 1,   Description: This is the first product
    UnitPrice: 20,   Quantity: 2,   SortOrder: 1,   Name: Product 2,   Description: This is the second product
)

We hope this helps you when working with collections in Streamscript. All these commands may be applied to primitives, lists, maps, or indeed any nested combination of collections.

Transform Tips

To sort a list of rich data structures, specify the sort field(s) using the special $_ variable:

$list_applications = [
   { Course: 'Postgrad',  Student: {Score: 80, Name: 'Tom'} }
   { Course: 'Undergrad', Student: {Score: 60, Name: 'Liz'} }
   { Course: 'Postgrad',  Student: {Score: 70, Name: 'Jim'} }
   { Course: 'Undergrad', Student: {Score: 90, Name: 'Bob'} }
]

# primary sort then secondary sort (with nested attribute)
$by_course_score = $list_applications | Sort [ $_.Course, $_.Student.Score ]

Result: [
    { Course: 'Postgrad',   Student: {Score: 70, Name: 'Jim'} }
    { Course: 'Postgrad',   Student: {Score: 80, Name: 'Tom'} }
    { Course: 'Undergrad',   Student: {Score: 60, Name: 'Liz'} }
    { Course: 'Undergrad',   Student: {Score: 90, Name: 'Bob'} }
]

To convert a list to a map, use the ToMap command - the sequential index from the list will be used as the map key for each item. Conversely, to convert a map to a list, use the ToList command to discard the map keys.

$map_applications = $list_applications | ToMap

Result: {
    '0': { Course: 'Postgrad',   Student: {Score: 80, Name: 'Tom'} }
    '1': { Course: 'Undergrad',   Student: {Score: 60, Name: 'Liz'} }
    '2': { Course: 'Postgrad',   Student: {Score: 70, Name: 'Jim'} }
    '3': { Course: 'Undergrad',   Student: {Score: 90, Name: 'Bob'} }
}

The Sum, Avg, Min and Max commands are compatible across lists of primitives, lists of maps, or nested values - in any case, specify the aggregation field using the special $_ variable.

$average = $map_applications  | Avg $_.Student.Score # 75
$average = $list_applications | Avg $_.Student.Score # 75

Getting started with Streamscript
Install from the Salesforce AppExchange
Package install link: /packaging/installPackage.apexp?p0=04tGA000005R8Ny