The Rest API source provides access to Rest Web Services. Use the Rest Api source component to define the request parameters to use for reading data and which fields will be used in the package.
Source components are always the first components in a package.
To define the rest api source:
- Add a Rest Api source component to begin your dataflow.
- Open the component and name it.
request settingsDefine the request setting parameters:
- connection - Use an existing connection to have Xplenty handle authentication for you, or use No connection to use basic authentication or handle authentication on your own.
- url - Enter the url of your data source site.
- method - The HTTP method. Select GET (default) or POST.
- username - The username for basic authentication. Can be blank if the site does not require authentication.
- password - The password for basic authentication. Can be blank if the site does not require authentication.
- body - The string body of the request.
- headers - Click Add new header? to add header fields. You can then enter:
- header field name
- header field value
- use pagination - Check if the Rest API source uses pagination. See here for list of supported API endpoints in addition to the standard link headers.
- Sleep interval between paginated requests (ms) - use to throttle API requests to avoid rate limitations. Default is 0 (no sleep).
- Maximum paginated requests - use to control the maximum number of pages for the source to go through. Default is unlimited, 0 or 1 means a single request will be made.
Then click the Test Connection button to help check that the http request is good.
response typeDefine the type of your source object:
- raw -Raw response in 3 fields: status code (integer), body (string) and headers (map).
- json - Response is processed using a JSONPath expression.
You can use a custom JSONPath expression to extract nested objects and arrays or the presets for object and array. Using the object preset, the response fields are the keys of the JSON object. Using the array preset, the response fields are the keys of the JSON objects within the array and each object is returned as a record in the component's output.
- line delimited json - Response is a single JSON object per line. Note that the response as a whole is not a valid JSON. The response fields are the keys of the JSON objects and each object is processed as a record.
- line delimited data - Response is broken to record per line. Useful when data contains records delimited by new line.
- After defining the source location you can optionally either:
- click the green Auto-detect Schema button to automatically populate the field names and data types and open a window to preview the data
- click the Show data preview button to preview the data.
- Define the source data fields as follows:
- for a raw file, the fields are auto-populated and cannot be changed.
- for JSON and JSON array, you only need to define the fields that you wish to use in your package.
- For Json and Json array files:
- key - type the source field name in the Json file
- alias - type the alias you will use for this field in other components
- type - select the data type that matches the source field