Using components: Database Source

Use the Database Source component, to load data from a database table or view.

source settings

  1. Define the parameters for connecting to your database table as follows (for more information, see Allow Xplenty access to my database server):
    • database connection - either click the drop-down arrow and select an existing connection, or click (create new) to create a new connection.
    • access mode - select table to extract an entire table/view or query to execute a user query.
    • source table - the table or view name from which the data will be imported.
    • split by column/max connections (optional) - use these fields to specify the degree of parallelism used to import the source table (If you leave these fields blank, the source table data is imported in a single task).
      • split by column - specify the column name that will be used as a criterion to split the import workload. The total range between the low and high values of this column will be evenly split between the number of connections (tasks) specified in max connections. We recommend using a column that is uniformly distributed across its value range. We will use the primary key by default. 
        Note: Splitting by a textual column may result in a partial or duplicate records if your database sorts in a case-insensitive order.
      • max connections - an integer specifying how many tasks to assign to the import process.
        Note: Do not increase the number of tasks above what your database can reasonably support.
    • where clause - optional. You can add predicates clauses to the WHERE clause as part of the SQL query that is built in order to get the data from the database. Make sure to skip the keyword WHERE. e.g.:
      • good: prod_category = 1 AND prod_color = 'red'
      • bad: WHERE prod_category = 1 AND prod_color = 'red'
    • or query - type in a SQL query. Make sure to name all columns uniquely.
    • pre-process action - You can select copy to copy the data from the database source to an intermediate storage before processing the data or select none to read the data from the database and apply transformations or store it into the destination immediately. copy may keep the database connections open for shorter periods of time, but using none would usually result in quicker job execution times.

Then click the Test Connection button to help check that the connection is good and that the source table exists.

column mapping

  1. After defining the source settings you can use the green Auto-detect schema button to get the field names and data types or the Preview button to preview the data and fill in the fields manually.
  2. Define the columns you want to extract from the table as follows:
    • Define the column name in the table.
    • Define the alias you will use for the column as a field in the data-flow.
    • Define the data type for the field. Use the following table when matching database data types to Xplenty data types.
PostgreSQL MySQL Microsoft
SQL Server
Xplenty
varchar, char, text, time, interval varchar, nvarchar, text, time varchar, nvarchar, text, ntext, time, datetimeoffset String
smallint, int bit, bool, tinyint, smallint, mediumint, int, integer tinyint, smallint, int Integer
bigint bigint bigint Long
decimal, real decimal, float decimal, numeric, float Float
double precision double real Double
timestamp, date date, datetime, timestamp datetime, date, datetime2, smalldatetime DateTime

 Note: The query is executed in the read-committed transaction isolation level.

Creating packages

  1. Creating a new package
  2. Create a package from a template
  3. Working in the package designer
  4. Using Components: Facebook Ads Insights Source (Beta)
  5. Using components: File Storage Source
  6. Using components: Database Source
  7. Using components: Google AdWords Source
  8. Using components: NetSuite Source
  9. Using Components: Google Analytics Source
  10. Using Components: Google BigQuery Source
  11. Using components: Google Cloud Spanner Source
  12. Using Components: Bing Ads Source
  13. Using components: MongoDB Source
  14. Using components: Amazon Redshift Source
  15. Using Components: Rest API Source
  16. Using Components: Salesforce Source
  17. Using components: Select
  18. Using components: Sort
  19. Using components: Rank
  20. Using components: Limit
  21. Using components: Sample
  22. Using components: Join
  23. Using components: Cross Join
  24. Using components: Clone
  25. Using components: Cube and Rollup
  26. Using components: Union
  27. Using components: Filter
  28. Using Components: Window
  29. Using components: Assert
  30. Using components: Aggregate
  31. Using components: Distinct
  32. Using components: File Storage Destination
  33. Using components: Amazon Redshift Destination
  34. Using Components: Salesforce Destination (Beta)
  35. Using components: Google BigQuery Destination
  36. Using components: Google Cloud Spanner Destination
  37. Using components: Database Destination
  38. Using components: MongoDB Destination
  39. Using and setting variables in your packages
  40. Validating a package
  41. Using pattern-matching in source component paths
  42. Using ISO 8601 string functions
  43. Using Expressions in Xplenty
  44. Xplenty Functions

Feedback and Knowledge Base