Using Components: Rest API Source

The Rest API provides access to Rest Web Services. Use the Rest Api source component to define the request parameters to use for reading data and which fields will be used in the package.
Source components are always the first components in a package.

To define the rest api source:

  1. Add a Rest Api source component to begin your dataflow.
  2. Open the component and name it.

request settings

Define the request setting parameters:
  • url - Enter the url of your data source site.
  • method - The HTTP method. Select GET (default) or POST.
  • username - The username for basic authentication. Can be blank if the site does not require authentication.
  • password - The password for basic authentication. Can be blank if the site does not require authentication.
  • body - The string body of the request.
  • headers - Click Add new header? to add header fields. You can then enter:
    • header field name
    • header field value
  • use paginationCheck if the Rest API source uses pagination. The Rest API source currently supports link headers and the following services:
    • 8x8 (*.8x8.com/api)
    • Adform (api.adform.com)
    • Appsee (api.appsee.com)
    • Autopilot (api2.autopilothq.com/v1/*/contacts/)
    • Asana (*.asana.com)
    • Atlassian (*.atlassian.net)
    • Base CRM (api.getbase.com)
    • CallRail (api.callrail.com/v2)
    • CallTrackingMetrics (api.calltrackingmetrics.com)
    • Cloudability (api.cloudability.com)
    • CradlePoint (www.cradlepointecm.com)
    • Cratejoy (api.cratejoy.com)
    • Delighted.com (api.delighted.com)
    • Desk.com (*.desk.com)
    • e-conomic (restapi.e-conomic.com)
    • Facebook Graph (graph.facebook.com)
    • Freshdesk (*.freshdesk.com/api)
    • Factual (api.v3.factual.com)
    • Google Places (maps.googleapis.com/maps/api/place)
    • HubSpot (api.hubapi.com)
    • Intercom (api.intercom.io)
    • Jira (*.jira.com)
    • Leankit (*.leankit.com)
    • LogEntries (rest.logentries.com)
    • LoyaltyLion (api.loyaltylion.com)
    • MailChimp (*.api.mailchimp.com)
    • Mailgun (api.mailgun.net)
    • Marketo (*.mktorest.com)
    • Paid (api.paidlabs.com)
    • PerformanceHorizon (api.performancehorizon.com)
    • Pipedrive (api.pipedrive.com/v1)
    • Printfection (api.printfection.com)
    • Promoter.io (app.promoter.io)
    • Recurly (*.recurly.com)
    • Salesforce Desk (*.desk.com)
    • SalesforceIQ (api.salesforceiq.com)
    • ShipStation (ssapi.shipstation.com)
    • Shopify (*.myshopify.com)
    • Sparkpost (*.sparkpost.com/api/)
    • Stormpath (api.stormpath.com)
    • Stripe (api.stripe.com)
    • Twilio (api.twilio.com) (json output only)
    • VTEX OMS (*.vtexcommercestable.com.br*/oms/*)
    • Wordpress (public-api.wordpress.com)
    • Xplenty (api.xplenty.com)
    • Yotpo (api.yotpo.com)
    • Zendesk (*.zendesk.com)
    • Zopim (www.zopim.com)
    • Zuora (rest.zuora.com)

    • Sleep interval between paginated requests (ms) - use to throttle API requests to avoid rate limitations. Default is 0 (no sleep).
    • Maximum paginated requests - use to control the maximum number of pages for the source to go through. Default is unlimited, 0 or 1 means a single request will be made.

Then click the Test Connection button to help check that the http request is good.

response type

Define the type of your source object:
  • raw -Raw response in 3 fields: status code (integer), body (string) and headers (map). 
  • json - Response is processed using a JSONPath expression.
    You can use a custom JSONPath expression to extract nested objects and arrays or the presets for object and array. Using the object preset, the response fields are the keys of the JSON object. Using the array preset, the response fields are the keys of the JSON objects within the array and each object is returned as a record in the component's output.
  • line delimited json - Response is a single JSON object per line. Note that the response as a whole is not a valid JSON. The response fields are the keys of the JSON objects and each object is processed as a record.
response fields

  1. After defining the source location you can optionally either:
    • click the green Auto-detect Schema button to automatically populate the field names and data types and open a window to preview the data
    • click the Show data preview button to preview the data.
  2. Define the source data fields as follows:
    • for a raw file, the fields are auto-populated and cannot be changed.
    • for JSON and JSON array, you only need to define the fields that you wish to use in your package.
  3. Define how you will refer to these fields (alias) in the other components of your package. If you use illegal characters, we'll let you know before you close the dialog.
Defining fields for use in your components

By default, one row of text and list boxes are displayed for defining the first field's parameters.

  • For Json and Json array files:
    • key - type the source field name in the Json file
    • alias - type the alias you will use for this field in other components
    • type - select the data type that matches the source field

Creating packages

  1. Creating a new package
  2. Create a package from a template
  3. Working in the package designer
  4. Using Components: Facebook Ads Insights Source (Beta)
  5. Using components: File Storage Source
  6. Using components: Database Source
  7. Using components: Google AdWords Source
  8. Using components: NetSuite Source
  9. Using Components: Google Analytics Source
  10. Using Components: Google BigQuery Source
  11. Using components: Google Cloud Spanner Source
  12. Using Components: Bing Ads Source
  13. Using components: MongoDB Source
  14. Using components: Amazon Redshift Source
  15. Using Components: Rest API Source
  16. Using Components: Salesforce Source
  17. Using components: Select
  18. Using components: Sort
  19. Using components: Rank
  20. Using components: Limit
  21. Using components: Sample
  22. Using components: Join
  23. Using components: Cross Join
  24. Using components: Clone
  25. Using components: Cube and Rollup
  26. Using components: Union
  27. Using components: Filter
  28. Using Components: Window
  29. Using components: Assert
  30. Using components: Aggregate
  31. Using components: Distinct
  32. Using components: File Storage Destination
  33. Using components: Amazon Redshift Destination
  34. Using Components: Salesforce Destination (Beta)
  35. Using components: Google BigQuery Destination
  36. Using components: Google Cloud Spanner Destination
  37. Using components: Database Destination
  38. Using components: MongoDB Destination
  39. Using and setting variables in your packages
  40. Validating a package
  41. Using pattern-matching in source component paths
  42. Using ISO 8601 string functions
  43. Using Expressions in Xplenty
  44. Xplenty Functions

Feedback and Knowledge Base