Creating a workflow

Workflows are packages that allows running sequence of tasks, such as executing SQL query or running a dataflow package. You can define the sequence and the conditions for executing a task, for example- task can be executed only in case that the previous task was completed successfully.

To create a workflow:

  1. On the main menu, click Packages.
  2. Click New package.
  3. Optionally, set a name and/or description
  4. Select workflow option from Type dropdown
  5. Click New Task + button
  6. Choose task
  7. Add additional tasks as required
  8. Connect tasks to create sequence of task execution. Click the connect icon on the dotted line to set the execution condition:
    • On success (default) - task will be executed once the preceding task was executed successfully
    • On failure - task will be executed once the preceding task execution failed
    • On completion - task will be executed once the preceding task completed, regardless to the completion status (failed/succeeded)

Execute SQL task

  1. Select your DB connection
  2. Write the SQL query that should be executed, and select the query result type from Result type dropdown. You can test the query by clicking Test Query
  3. If there are at least two preceding tasks, choose task execution condition:
    • all preceding conditions evaluate to true (AND)
    • one of the preceding conditions evaluate to true (OR)

Run package task

  1. Select the package to run
  2. Optionally, edit package variables If you want to override variable values before running the task.
  3. If there are at least two preceding tasks, choose task execution condition:
    • all preceding conditions evaluate to true (AND)
    • one of the preceding conditions evaluate to true (OR)

Using Variables in Workflows

User variables can be defined at the workflow package level. They can be used in the Execute SQL task within the query in which they'll be substituted for their values. Make sure to enclose the variable within curly brackets (i.e: '${var_name}'). Variables can also be assigned values by the Execute SQL task. 

Here's an example of using a variable within the Execute SQL task query:

SELECT * from todos where created_at > '${date_today}';

When setting up a Run package task, workflow package level variables can be used in expressions assigned to the dataflow variables. Note that if both workflow and dataflow variables have the same name, you will still have to assign the workflow variable to the dataflow variable in the Run package task (i.e. the workflow variable will not override an existing task variable). However, if a task dataflow uses a variable that isn't defined at the dataflow level but is assigned a value at the workflow level, the dataflow task will use the workflow variable value.


Creating packages

  1. Creating a new package in New Xplenty
  2. Creating a workflow
  3. Working in the new package designer
  4. Validating a package
  5. Using components: Amazon Redshift Source
  6. Using components: Bing Ads Source
  7. Using components: Database Source
  8. Using components: Facebook Ads Insights Source
  9. Using components: File Storage Source
  10. Using components: Google Adwords source
  11. Using components: Google Analytics Source
  12. Using components: Google BigQuery Source
  13. Using components: Google Cloud Spanner Source
  14. Using components: MongoDB Source
  15. Using components: NetSuite Source
  16. Using components: Salesforce source
  17. Using components: Rest API Source
  18. Using components: Aggregate Transformation
  19. Using components: Assert Transformation
  20. Using components: Clone transformation
  21. Using components: Cross Join Transformation
  22. Using components: Distinct Transformation
  23. Using components: Filter Transformation
  24. Using components: Join Transformation
  25. Using components: Limit Transformation
  26. Using components: Rank Transformation
  27. Using components: Select Transformation
  28. Using components: Sort Transformation
  29. Using components: Union Transformation
  30. Using components: Window Transformation
  31. Using components: Sample Transformation
  32. Using components: Cube transformation
  33. Using components: Amazon Redshift Destination
  34. Using components: Database Destination
  35. Using components: File Storage Destination
  36. Using components: Google BigQuery Destination
  37. Using components: Google Spanner Destination
  38. Using components: MongoDB Destination
  39. Using components: Salesforce Destination
  40. Using components: Snowflake Destination (beta)
  41. Using Components: Rest API Destination
  42. Using and setting variables in your packages
  43. System and pre-defined variables
  44. Using pattern-matching in source component paths
  45. Using ISO 8601 string functions
  46. Using Expressions in Xplenty
  47. Xplenty Functions

Feedback and Knowledge Base