Using components: Database Source

Use the database source component to read data stored in a database table, view or using a query.

Connection

Select an existing database connection or create a new one.

Source Properties

  • Access mode - select table to extract an entire table/view or query to execute a query.
  • Source schema - the source table's schema. If empty, the default schema is used.
  • Source table/view - the table or view name from which the data will be imported.
  • where clause - optional. You can add predicates clauses to the WHERE clause as part of the SQL query that is built in order to get the data from the database. Make sure to skip the keyword WHERE.
    Goodprod_category = 1 AND prod_color = 'red'
    BadWHERE prod_category = 1 AND prod_color = 'red'
  • Query - type in a SQL query. Make sure to name all columns uniquely.

Table access mode parallelization (used with table access mode only)

To parallelize a query, select a key to split the queries by and the maximum number of parallel connections. When parallelizing a query, a preliminary query will get the minimum and maximum values for the column and then queries will be issued from multiple connections with a where clause that splits the data to ranges. E.g.: pk >= 1 AND pk < 1000, pk >= 1001 AND pk < 2000.

  • Split query by key column - Specify the name of a source table column to split the query by or leave empty to use a single query. It is recommend to a column that is uniformly distributed across its value range (primary key column is a good choice).
  • Max parallel connections - an positive number specifying how many tasks to assign to the import process.

Note: Do not increase the number of tasks above what your database can reasonably support.

Source action

  • None - By default, data is read from the database and transformations are applied immediately.
  • Copy - Copy the data from the database source to intermediate storage before processing the data. This may keep the database connections open for shorter periods of time, but selecting None would usually result in quicker job execution times.

Source Schema

After defining the source table/view/query select the fields to use in the source.

With table access mode, the fields you select are used to build the query that will be executed to read the data.

With query access mode, select all the fields that are defined in the query and make sure to use the same column names

Define the data type for the field. Use the following table when matching database data types to Xplenty data types.

PostgreSQL MySQL Microsoft
SQL Server
Oracle Snowflake Xplenty
varchar, char, text, time, interval varchar, nvarchar, text, time varchar, nvarchar, text, ntext, time, datetimeoffset longnvarchar, nchar, nvarchar, longvarchar, char, varchar, clob, nclob varchar, char, character, string, text String
smallint, int bit, bool, tinyint, smallint, mediumint, int, integer tinyint, smallint, int tinyint, integer, smallint Integer
bigint bigint bigint bigint int, integer, bigint, smallint, tinyint, byteint, number(38,0) Long
decimal, real decimal, float decimal, numeric, float float, binary float, real Float
double precision double real numeric, decimal, binary double float, float4, float8, double, double precision, real, decimal, numeric Double
timestamp, date date, datetime, timestamp datetime, date, datetime2, smalldatetime date, time, timestamp, timestamptz, timestampltz date, datetime, timestamp, timestamptz, timestampltz, timestampntz DateTime

Note: The query is executed in the read-committed transaction isolation level.

Creating packages in New Xplenty

  1. Creating a new package in New Xplenty
  2. Creating a workflow
  3. Working in the new package designer
  4. Using components: Amazon Redshift Source
  5. Validating a package
  6. Using components: Bing Ads Source
  7. Using components: Database Source
  8. Using components: Facebook Ads Insights Source
  9. Using components: File Storage Source
  10. Using components: Google Adwords source
  11. Using components: Google Analytics Source
  12. Using components: Google BigQuery Source
  13. Using components: Google Cloud Spanner Source
  14. Using components: MongoDB Source
  15. Using components: NetSuite Source
  16. Using components: Salesforce source
  17. Using components: Rest API Source
  18. Using components: Aggregate Transformation
  19. Using components: Assert Transformation
  20. Using components: Clone transformation
  21. Using components: Cross Join Transformation
  22. Using components: Distinct Transformation
  23. Using components: Filter Transformation
  24. Using components: Join Transformation
  25. Using components: Limit Transformation
  26. Using components: Rank Transformation
  27. Using components: Select Transformation
  28. Using components: Sort Transformation
  29. Using components: Union Transformation
  30. Using components: Window Transformation
  31. Using components: Sample Transformation
  32. Using components: Amazon Redshift Destination
  33. Using components: Database Destination
  34. Using components: File Storage Destination
  35. Using components: Google BigQuery Destination
  36. Using components: Google Spanner Destination
  37. Using components: MongoDB Destination
  38. Using components: Salesforce Destination
  39. Using components: Snowflake Destination (beta)

Feedback and Knowledge Base