Using components: Database Destination

Use the database destination component to store the output of a data flow in relational database table.

The following database platforms are supported: PostgreSQL, MySQL, Microsoft SQL Server and Oracle.

Connection

Select an existing database connection or create a new one.

Destination Properties

  • Target schema - the target table's schema. If empty, the default schema is used.
  • Target table - the name of the target table in your database. By default, if the table doesn't exist, it will be created automatically.
  • Automatically create table if it doesn't exist - if unchecked and the table doesn't exist, the job fails.
  • Automatically add missing columns - when checked, the job will check if each of the specified columns exist in the table and if one does not exist, it will add it. Key columns can't be automatically added to a table.

Operation type

Append (Insert only) - default behaviour. Data will only be appended to the target table

Overwrite (Truncate and insert) - truncate the target table before data is inserted into the target table. There may limitations on whether truncate can be used on the target table, depending on the destination database platform.

Overwrite (Delete and insert) - deletes all of the target table before the data flow executes. If a truncate statement can't be executed on the target table due to permissions or other constraints, you can use this instead.

Merge with existing data using delete and insert - incoming data is merged with existing data in the table by deleting target table data that exists in both the data sets and then inserting all the incoming data into the target table. Requires setting the merge keys correctly in field mapping. Merge is done in the following manner:

  1. First transaction - A staging table is created with a primary key according to your key mapping in the database's default schema.
  2. Second transaction - The dataflow's output is bulk copied into the staging table.
  3. Third transaction - Rows with keys that exist in the staging table are deleted from the target table. All rows in the staging table are inserted into the target table. The staging table is deleted.

Merge with existing data using update and insert - incoming data is merged with existing data in the table by updating existing data and inserting new data. Requires setting the merge keys correctly in field mapping. Merge is done in the following manner:

  1. First transaction - A staging table is created with a primary key according to your key mapping in the database's default schema.
  2. Second transaction - The dataflow's output is bulk copied into the staging table.
  3. Third transaction - Database specific command is issued to update existing records and insert. The staging table is deleted.
    PlatformSQL command
    PostgreSQL (9.5+)
    INSERT INTO <target> ("k1","c2",...,"cn") 
    SELECT "k1","c2",...,"cn" FROM <staging>
    ON CONFLICT ("k1") DO UPDATE SET 
        "c2" = excluded."c2", ...
        "cn" = excluded."cn";
    
    MySQL
    INSERT INTO <target>  (`k1`, `c2`, ..., `cn`)
    SELECT `k1`, `k2`, `c2`, `c3`, `c4` FROM <staging> AS stg
    ON DUPLICATE KEY UPDATE
        `c2` = stg.`c2`, ...
        `cn` = stg.`cn`;
    
    Microsoft SQL Server
    MERGE INTO  <target> as tgt
    USING <staging> as stg
    ON 
        tgt."k1" = stg."k1"
    WHEN MATCHED THEN
        UPDATE SET 
            tgt."c2" = stg."c2", ...
            tgt."cn" = stg."cn"
    WHEN NOT MATCHED BY TARGET THEN
        INSERT ("k1","c2",...,"c4")
        VALUES (stg."k1",stg."c2",...,stg."c4");
    
    Oracle
    MERGE INTO <target> AS tgt
    USING <staging>  AS stg
    ON 
        (tgt."k1" = stg."k1")
    WHEN MATCHED THEN
        UPDATE SET 
            tgt."c2" = stg."c2", ...
            tgt."cn" = stg."cn"
    WHEN NOT MATCHED THEN
        INSERT ("k1",c2",...,"cn")
        VALUES (stg."k1",stg."c2",...,stg."cn");
    
    Amazon Redshift
    START TRANSACTION;
    
    UPDATE <target>
    SET
        c2 =  <staging>.c2, ...
        cn =  <staging>.cn
    FROM <staging> 
    WHERE 
        <target>."k1" =  <staging>."k1";
    
    DELETE FROM  <staging>
    USING <target>
    WHERE  <staging>."k1" = <target>."k1";
    
    INSERT INTO <target> ("k1","k2","c2",...,"cn")
    SELECT "k1","c2",...,"cn"
    FROM  <staging>;
    
    COMMIT;
    
Note: The incoming data must have unique values in the key fields you selected. A possible workaround for this issue can be to use a Limit component and add the key field/s as a partition and limit it to 1 to remove duplicates.

Pre and post action SQL

Pre-action SQL - SQL code to execute before inserting the data into the target table. If a merge operation is selected, the sql code is executed before the staging table is created.

Post-action SQL - SQL code to execute after inserting the data into the target table. If a merge operation is selected, the sql code is executed after the staging table is merged into the target table.

Advanced options

  • Batch sizenumber of records that are inserted to the database in each batch (default 1000).
  • Transactions - by default, each connection to the database will use a single transaction. If "Transaction per batch" is selected, each batch will be committed on its own which may lead to partial data in the target table in case the job fails.
  • Max parallel connections - maximum number of concurrent connections to open when writing to the database (1 by default).
  • Split data insertion by input field - when using more than one connection, this field will be used to split the data to the different connections. Pick a field with low density (a unique key is best) to make sure that data split isn't skewed.

Schema Mapping

Map the dataflow fields to the target table's columns. Columns defined as key will be used as the sort key when Xplenty creates the table. If merge operation is used, you must select at least a field or multiple fields as keys, which will be used to uniquely identify rows in the table for the merge operation.

The data types in Xplenty are automatically mapped as follows when the table is created automatically. Note that since Xplenty doesn't have a notion of maximum string length, the string columns are created with the maximum length allowed in the database.
Xplenty PostgreSQL MySQL Microsoft
SQL Server
String varchar, char, text,  interval varchar, nvarchar, text,  varchar, nvarchar, text, ntext
Integer smallint, int bit, bool, tinyint, smallint, mediumint, int, integer tinyint, smallint, int
Long bigint bigint bigint
Float decimal, real decimal, float decimal, numeric, float
Double double precision double real
DateTime timestamp, date, time date, datetime, timestamp, time datetime, datetime2, smalldatetime, date, time, datetimeoffset
Boolean TINYINT(1) BOOLEAN BIT

    * Note - Use the ToDate function to cast a datetime string expression to datetime data type. Note that a datetime with timezone offset value will be adjusted to UTC when inserted into a database

Automatically creating and altering destination table

Xplenty can automatically create the destination table for you if it doesn't exist and it can also append columns to an existing table. If you define columns as key (regardless of the operation type), Xplenty defines them as the primary key in a new table. The data types in Xplenty are automatically mapped as follows. Note that since Xplenty doesn't have a notion of maximum string length, the string columns are created with the maximum length allowed in the database.

Xplenty PostgreSQL MySQL Microsoft
SQL Server
Oracle
String (key column) VARCHAR(100) NVARCHAR(100) NVARCHAR(100) NVARCHAR2(100)
String (non key column) VARCHAR(65535) TEXT NVARCHAR(MAX) NCLOB
Integer INT INT INT NUMBER
Long BIGINT BIGINT BIGINT NUMBER
Float REAL FLOAT REAL NUMBER
Double DOUBLE PRECISION DOUBLE FLOAT NUMBER
DateTime TIMESTAMP DATETIME DATETIME TIMESTAMP
Boolean BOOLEAN TINYINT(1) BIT NUMBER(1)

Creating packages

  1. Creating a new package in New Xplenty
  2. Creating a workflow
  3. Working in the new package designer
  4. Validating a package
  5. Using components: Amazon Redshift Source
  6. Using components: Bing Ads Source
  7. Using components: Database Source
  8. Using components: Facebook Ads Insights Source
  9. Using components: File Storage Source
  10. Using components: Google Adwords source
  11. Using components: Google Analytics Source
  12. Using components: Google BigQuery Source
  13. Using components: Google Cloud Spanner Source
  14. Using components: MongoDB Source
  15. Using components: NetSuite Source
  16. Using components: Salesforce source
  17. Using components: Rest API Source
  18. Using components: Aggregate Transformation
  19. Using components: Assert Transformation
  20. Using components: Clone transformation
  21. Using components: Cross Join Transformation
  22. Using components: Distinct Transformation
  23. Using components: Filter Transformation
  24. Using components: Join Transformation
  25. Using components: Limit Transformation
  26. Using components: Rank Transformation
  27. Using components: Select Transformation
  28. Using components: Sort Transformation
  29. Using components: Union Transformation
  30. Using components: Window Transformation
  31. Using components: Sample Transformation
  32. Using components: Cube transformation
  33. Using components: Amazon Redshift Destination
  34. Using components: Database Destination
  35. Using components: File Storage Destination
  36. Using components: Google BigQuery Destination
  37. Using components: Google Spanner Destination
  38. Using components: MongoDB Destination
  39. Using components: Salesforce Destination
  40. Using components: Snowflake Destination (beta)
  41. Using and setting variables in your packages
  42. System and pre-defined variables
  43. Using pattern-matching in source component paths
  44. Using ISO 8601 string functions
  45. Using Expressions in Xplenty
  46. Xplenty Functions

Feedback and Knowledge Base