Using components: Rest API Source

Use the Rest API source provides access to read data from HTTP endpoints such as Rest Web Services. Use the Rest API source component to define the authentication method, request parameters and response fields to use in the package.

Authentication

  • None - No authentication
  • Basic - Use basic authentication (user and/or password).
  • Connection - Use an Xplenty connection to handle authentication automatically.

Request and response

Request

  • method - The HTTP method. Select GET (default) or POST.
  • URL - Enter the URL of your source endpoint.
  • Headers - Optionally add request headers (names and values)
  • Body - Optionally add a request body string

Pagination

Check Use pagination to have the source component make paginated requests. See here for list of supported API endpoints in addition to the standard link headers .

  • Pagination scheme - The method to paginate through response data (e.g. page number, offset+limit, cursor, link headers) and how to detect the final page.
    • Automatic (Default) - the source detects the pagination scheme to use by the URL host/path.
    • Jira - Use with hosted Jira installations.
    • Magento - Use with hosted Magento installations.
    • BigCommerce - Use with hosted BigCommerce installations.
    • Elasticsearch - Use with hosted Elasticsearch installations.
    • Link headers - the fallback option which will be used with the default automatic as well.
  • Sleep interval between paginated requests (ms) - use to throttle API requests to avoid rate limitations. Default is 0 (no sleep).
  • Maximum paginated requests - use to control the maximum number of pages for the source to go through. Default is unlimited, 0 or 1 means a single request will be made.

Response

Select the response type to determine how to parse the response from the API endpoint.

  • Raw - returns 3 fields: status code (integer), body (string) and headers (map).
  • JSON - Response is processed using a JSONPath expression.
    You can use a custom JSONPath expression to extract nested objects and arrays or the presets for object and array. Using the object preset, the response fields are the keys of the JSON object. Using the array preset, the response fields are the keys of the JSON objects within the array and each object is returned as a record in the component's output.
  • Line Delimited JSON - Response is a single JSON object per line. Note that the response as a whole is not a valid JSON. The response fields are the keys of the JSON objects and each object is processed as a record.
  • Line Delimited raw data - Response is broken to record per line. Useful when data contains records delimited by new line.

Schema

After defining the request and the response, select the fields to use in the source.

Creating packages

  1. Creating a new package in New Xplenty
  2. Creating a workflow
  3. Working in the new package designer
  4. Validating a package
  5. Using components: Amazon Redshift Source
  6. Using components: Bing Ads Source
  7. Using components: Database Source
  8. Using components: Facebook Ads Insights Source
  9. Using components: File Storage Source
  10. Using components: Google Adwords source
  11. Using components: Google Analytics Source
  12. Using components: Google BigQuery Source
  13. Using components: Google Cloud Spanner Source
  14. Using components: MongoDB Source
  15. Using components: NetSuite Source
  16. Using components: Salesforce source
  17. Using components: Rest API Source
  18. Using components: Aggregate Transformation
  19. Using components: Assert Transformation
  20. Using components: Clone transformation
  21. Using components: Cross Join Transformation
  22. Using components: Distinct Transformation
  23. Using components: Filter Transformation
  24. Using components: Join Transformation
  25. Using components: Limit Transformation
  26. Using components: Rank Transformation
  27. Using components: Select Transformation
  28. Using components: Sort Transformation
  29. Using components: Union Transformation
  30. Using components: Window Transformation
  31. Using components: Sample Transformation
  32. Using components: Cube transformation
  33. Using components: Amazon Redshift Destination
  34. Using components: Database Destination
  35. Using components: File Storage Destination
  36. Using components: Google BigQuery Destination
  37. Using components: Google Spanner Destination
  38. Using components: MongoDB Destination
  39. Using components: Salesforce Destination
  40. Using components: Snowflake Destination (beta)
  41. Using and setting variables in your packages
  42. System and pre-defined variables
  43. Using pattern-matching in source component paths
  44. Using ISO 8601 string functions
  45. Using Expressions in Xplenty
  46. Xplenty Functions

Feedback and Knowledge Base