Airbyte Declarative Connectors Specification (manifest.yaml)

List of strings defining the path where to add the value on the record.

Value of the new field. Use {{ record['existing_field'] }} syntax to refer to other fields in the record.

A schema type.

List of transformations (path and corresponding value) that will be added to the record.

Fields will be added if expression is evaluated to True.

The API key to inject in the request. Fill it in the user inputs.

The name of the HTTP header that will be set to the API key. This setting is deprecated, use inject_into instead. Header and inject_into can not be defined at the same time.

Specifies the key field or path and where in the request a component's value should be injected.

The type of auth to use

JSON path to a field in the connectorSpecification that should exist for the advanced auth to be applicable.

Value of the predicate_key fields for the advanced auth to be applicable.

Specification describing how an 'advanced' Auth flow would need to function.

The username that will be combined with the password, base64 encoded and used to make requests. Fill it in the user inputs.

The password that will be combined with the username, base64 encoded and used to make requests. Fill it in the user inputs.

Token to inject as request header for authenticating with the API.

Path of the field in config with selected authenticator name

Authenticators to select from.

Names of the streams to try reading from when running a check operation.

The dynamic stream name.

The number of streams to attempt reading from during a check operation. If stream_count exceeds the total number of available streams, the minimum of the two values will be used.

Numbers of the streams to try reading from when running a check operation.

Enables stream check availability. This field is automatically set by the CDK.

List of error handlers to iterate on to determine how to handle a failed response.

The amount of concurrency that will applied during a sync. This value can be hardcoded or user-defined in the config if different users have varying volume thresholds in the target API.

The maximum level of concurrency that will be used during a sync. This becomes a required field when the default_concurrency derives from the config, because it serves as a safeguard against a user-defined threshold that is too high.

Condition that will be evaluated to determine if a set of streams should be available.

Streams that will be used during an operation based on the condition.

Backoff time in seconds.

Value of the cursor defining the next page to fetch.

The number of records to include in each pages.

Template string evaluating when to stop paginating.

Fully-qualified name of the class that will be implementing the custom authentication strategy. Has to be a sub class of DeclarativeAuthenticator. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom backoff strategy. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom error handler. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom incremental sync. The format is source_<name>.<package>.<class_name>.

The location of the value on a record that will be used as a bookmark during sync.

Fully-qualified name of the class that will be implementing the custom pagination strategy. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom record extraction strategy. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom record filter strategy. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom requester strategy. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom retriever strategy. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom partition router. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom schema loader. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom normalization. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom state migration. The format is source_<name>.<package>.<class_name>.

Fully-qualified name of the class that will be implementing the custom transformation. The format is source_<name>.<package>.<class_name>.

The location of the value on a record that will be used as a bookmark during sync. To ensure no data loss, the API must return records in ascending order based on the cursor field. Nested fields are not supported, so the field must be at the top level of the record. You can use a combination of Add Field and Remove Field transformations to move the nested field to the top.

The value that determines the earliest record that should be synced.

Specifies the key field or path and where in the request a component's value should be injected.

The location of the value on a record that will be used as a bookmark during sync. To ensure no data loss, the API must return records in ascending order based on the cursor field. Nested fields are not supported, so the field must be at the top level of the record. You can use a combination of Add Field and Remove Field transformations to move the nested field to the top.

The datetime that determines the earliest record that should be synced.

The datetime format used to format the datetime values that are sent in outgoing requests to the API. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:

• %s: Epoch unix timestamp - 1686218963
• %s_as_float: Epoch unix timestamp in seconds as float with microsecond precision - 1686218963.123456
• %ms: Epoch unix timestamp (milliseconds) - 1686218963123
• %a: Weekday (abbreviated) - Sun
• %A: Weekday (full) - Sunday
• %w: Weekday (decimal) - 0 (Sunday), 6 (Saturday)
• %d: Day of the month (zero-padded) - 01, 02, ..., 31
• %b: Month (abbreviated) - Jan
• %B: Month (full) - January
• %m: Month (zero-padded) - 01, 02, ..., 12
• %y: Year (without century, zero-padded) - 00, 01, ..., 99
• %Y: Year (with century) - 0001, 0002, ..., 9999
• %H: Hour (24-hour, zero-padded) - 00, 01, ..., 23
• %I: Hour (12-hour, zero-padded) - 01, 02, ..., 12
• %p: AM/PM indicator
• %M: Minute (zero-padded) - 00, 01, ..., 59
• %S: Second (zero-padded) - 00, 01, ..., 59
• %f: Microsecond (zero-padded to 6 digits) - 000000
• %_ms: Millisecond (zero-padded to 3 digits) - 000
• %z: UTC offset - (empty), +0000, -04:00
• %Z: Time zone name - (empty), UTC, GMT
• %j: Day of the year (zero-padded) - 001, 002, ..., 366
• %U: Week number of the year (starting Sunday) - 00, ..., 53
• %W: Week number of the year (starting Monday) - 00, ..., 53
• %c: Date and time - Tue Aug 16 21:30:00 1988
• %x: Date standard format - 08/16/1988
• %X: Time standard format - 21:30:00
• %%: Literal '%' character

Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the Python documentation.

This option is used to adjust the upper and lower boundaries of each datetime window to beginning and end of the provided target period (day, week, month)

The possible formats for the cursor field, in order of preference. The first format that matches the cursor field value will be used to parse it. If not provided, the Outgoing Datetime Format will be used. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:

• %s: Epoch unix timestamp - 1686218963
• %s_as_float: Epoch unix timestamp in seconds as float with microsecond precision - 1686218963.123456
• %ms: Epoch unix timestamp - 1686218963123
• %a: Weekday (abbreviated) - Sun
• %A: Weekday (full) - Sunday
• %w: Weekday (decimal) - 0 (Sunday), 6 (Saturday)
• %d: Day of the month (zero-padded) - 01, 02, ..., 31
• %b: Month (abbreviated) - Jan
• %B: Month (full) - January
• %m: Month (zero-padded) - 01, 02, ..., 12
• %y: Year (without century, zero-padded) - 00, 01, ..., 99
• %Y: Year (with century) - 0001, 0002, ..., 9999
• %H: Hour (24-hour, zero-padded) - 00, 01, ..., 23
• %I: Hour (12-hour, zero-padded) - 01, 02, ..., 12
• %p: AM/PM indicator
• %M: Minute (zero-padded) - 00, 01, ..., 59
• %S: Second (zero-padded) - 00, 01, ..., 59
• %f: Microsecond (zero-padded to 6 digits) - 000000, 000001, ..., 999999
• %_ms: Millisecond (zero-padded to 3 digits) - 000, 001, ..., 999
• %z: UTC offset - (empty), +0000, -04:00
• %Z: Time zone name - (empty), UTC, GMT
• %j: Day of the year (zero-padded) - 001, 002, ..., 366
• %U: Week number of the year (Sunday as first day) - 00, 01, ..., 53
• %W: Week number of the year (Monday as first day) - 00, 01, ..., 53
• %c: Date and time representation - Tue Aug 16 21:30:00 1988
• %x: Date representation - 08/16/1988
• %X: Time representation - 21:30:00
• %%: Literal '%' character

Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the Python documentation.

Specifies the key field or path and where in the request a component's value should be injected.

The datetime that determines the last record that should be synced. If not provided, {{ now_utc() }} will be used.

Specifies the key field or path and where in the request a component's value should be injected.

Smallest increment the datetime_format has (ISO 8601 duration) that is used to ensure the start of a slice does not overlap with the end of the previous one, e.g. for %Y-%m-%d the granularity should be P1D, for %Y-%m-%dT%H:%M:%SZ the granularity should be PT1S. Given this field is provided, step needs to be provided as well.

• PT0.000001S: 1 microsecond
• PT0.001S: 1 millisecond
• PT1S: 1 second
• PT1M: 1 minute
• PT1H: 1 hour
• P1D: 1 day

A data feed API is an API that does not allow filtering and paginates the content from the most recent to the least recent. Given this, the CDK needs to know when to stop paginating and this field will generate a stop condition for pagination.

Set to True if the target API endpoint does not take cursor values to filter records and returns all records anyway. This will cause the connector to filter out records locally, and only emit new records from the last sync, hence incremental. This means that all records would be read from the API, but only new records will be emitted to the destination.

Set to True if the target API does not accept queries where the start time equal the end time. This will cause those requests to be skipped.

Setting to True causes the connector to store the cursor as one value, instead of per-partition. This setting optimizes performance when the parent stream has thousands of partitions. Notably, the substream state is updated only at the end of the sync, which helps prevent data loss in case of a sync failure. See more info in the docs.

Time interval (ISO8601 duration) before the start_datetime to read data for, e.g. P1M for looking back one month.

• PT1H: 1 hour
• P1D: 1 day
• P1W: 1 week
• P1M: 1 month
• P1Y: 1 year

Name of the partition start time field.

Name of the partition end time field.

The size of the time window (ISO8601 duration). Given this field is provided, cursor_granularity needs to be provided as well.

• PT1H: 1 hour
• P1D: 1 day
• P1W: 1 week
• P1M: 1 month
• P1Y: 1 year

Secret used to sign the JSON web token.

Algorithm used to sign the JSON web token.

When set to true, the secret key will be base64 encoded prior to being encoded as part of the JWT. Only set to "true" when required by the API.

The amount of time in seconds a JWT token can be valid after being issued.

The prefix to be used within the Authentication header.

JWT headers used when signing JSON web token.

Additional headers to be included with the JWT headers object.

JWT Payload used when signing JSON web token.

Additional properties to be added to the JWT payload.

The name of the property to use to refresh the access_token.

The OAuth client ID. Fill it in the user inputs.

The name of the property to use to refresh the access_token.

The OAuth client secret. Fill it in the user inputs.

The name of the property to use to refresh the access_token.

Credential artifact used to get a new access token.

The full URL to call to obtain a new access token.

The name of the property which contains the access token in the response from the token refresh endpoint.

The value of the access_token to bypass the token refreshing using refresh_token.

The name of the property which contains the expiry date in the response from the token refresh endpoint.

The name of the property to use to refresh the access_token.

Specifies the OAuth2 grant type. If set to refresh_token, the refresh_token needs to be provided as well. For client_credentials, only client id and secret are required. Other grant types are not officially supported.

Body of the request sent to get a new access token.

Headers of the request sent to get a new access token.

List of scopes that should be granted to the access token.

The access token expiry date.

The format of the time to expiration datetime. Provide it if the time is returned as a date-time string instead of seconds.

When the refresh token updater is defined, new refresh tokens, access tokens and the access token expiry date are written back from the authentication response to the config object. This is important if the refresh token can only used once.

Authenticator for requests using JWT authentication flow.

Enable using profile assertion as a flow for OAuth authorization.

Component used to coordinate how records are extracted across stream slices and request pages.

The stream name.

Component used to fetch data incrementally based on a time field in the data.

The stream field to be used to distinguish unique records. Can either be a single field, an array of fields representing a composite key, or an array of arrays representing a composite key where the fields are nested fields.

One or many schema loaders can be used to retrieve the schema for the current stream. When multiple schema loaders are defined, schema properties will be merged together. Schema loaders defined first taking precedence in the event of a conflict.

A list of transformations to be applied to each output record.

Array of state migrations to be applied on the input state

(experimental) Describes how to fetch a file

List of call rate policies that define how many calls are allowed.

The HTTP response header name that indicates when the rate limit resets.

The HTTP response header name that indicates the number of remaining allowed calls.

List of HTTP status codes that indicate a rate limit has been hit.

The time interval for the rate limit window.

The maximum number of calls allowed within the period.

List of matchers that define which requests this policy applies to.

List of rates that define the call limits for different time intervals.

List of matchers that define which requests this policy applies to.

List of matchers that define which requests this policy applies to.

The maximum number of calls allowed within the interval.

The time interval for the rate limit.

The HTTP method to match (e.g., GET, POST).

The base URL (scheme and host, e.g. "https://api.example.com") to match.

A regular expression pattern to match the URL path.

The query parameters to match.

The headers to match.

List of backoff strategies to use to determine how long to wait before retrying a retryable request.

The maximum number of time to retry a retryable request before giving up and failing.

List of response filters to iterate on when deciding how to handle an error. When using an array of multiple filters, the filters will be applied sequentially and the response will be selected if it matches any of the filter's predicate.

Strategy defining how records are paginated.

Specifies the key field or path and where in the request a component's value should be injected.

List of potentially nested fields describing the full path of the field to extract. Use "*" to extract all values from an array. See more info in the docs.

Multiplicative constant applied on each retry.

The name of the field on the record whose value will be used to group properties that were retrieved through multiple API requests.

Requester submitting HTTP requests and extracting records from the response.

The path in the response body returned from the login requester to the session token.

Authentication method to use for requests sent to the API, specifying how to inject the session token.

The duration in ISO 8601 duration notation after which the session token expires, starting from the time it was obtained. Omitting it will result in the session token being refreshed for every request.

• PT1H: 1 hour
• P1D: 1 day
• P1W: 1 week
• P1M: 1 month
• P1Y: 1 year

Component used to decode the response.

Specifies the key field or path and where in the request a component's value should be injected.

Deprecated, use the url instead. Base URL of the API source. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.

The URL of the source API endpoint. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.

Deprecated, use the url instead. Path the specific API endpoint that this stream represents. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.

The HTTP method used to fetch data from the source (can be GET or POST).

Authentication method to use for requests sent to the API.

Defines the behavior for fetching the list of properties from an API that will be loaded into the requests to extract records.

For APIs that require explicit specification of the properties to query for, this component specifies which property fields and how they are supplied to outbound requests.

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

Specifies how to populate the body of the request with a non-JSON payload. Plain text will be sent as is, whereas objects will be converted to a urlencoded form.

Specifies how to populate the body of the request with a JSON payload. Can contain nested objects.

Specifies how to populate the body of the request with a payload. Can contain nested objects.

Error handler component that defines how to handle errors.

Enables stream requests caching. This field is automatically set by the CDK.

Action to execute if a response matches the filter.

Failure type of traced exception if a response matches the filter.

Error Message to display if the response matches the filter.

Match the response if its error message contains the substring.

Match the response if its HTTP code is included in this list.

Match the response if the predicate evaluates to true.

List of potentially nested fields describing the full path of the field key to extract.

List of nested fields defining the schema field path to extract. Defaults to [].

List of potentially nested fields describing the full path of the field type to extract.

Component used to coordinate how records are extracted across stream slices and request pages.

(This component is experimental. Use at your own risk.) Identifies schema details for dynamic schema extraction and processing.

Responsible for filtering fields to be added to json schema.

A list of transformations to be applied to the schema.

Describes a streams' schema. Refer to the Data Types documentation for more details on which types are valid.

Path to the JSON file defining the schema. The path is relative to the connector module's root.

Whether to flatten lists or leave it as is. Default is True.

Prefix to add for object keys. If not provided original keys remain unchanged.

Suffix to add for object keys. If not provided original keys remain unchanged.

A path to field that needs to be flattened.

Whether to delete the origin value or keep it. Default is False.

Whether to replace the origin record or not. Default is False.

Old value to replace.

New value to set.

Fully-qualified name of the class that will be implementing the custom decoding. Has to be a sub class of Decoder. The format is source_<name>.<package>.<class_name>.

Parser to parse the decompressed data from the zipfile(s).

While iterating over list values, the name of field used to reference a list value. The partition value can be accessed with string interpolation. e.g. "{{ stream_partition['my_key'] }}" where "my_key" is the value of the cursor_field.

The list of attributes being iterated over and used as input for the requests made to the source API.

Specifies the key field or path and where in the request a component's value should be injected.

Datetime value.

Format of the datetime value. Defaults to "%Y-%m-%dT%H:%M:%S.%f%z" if left empty. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:

• %s: Epoch unix timestamp - 1686218963
• %s_as_float: Epoch unix timestamp in seconds as float with microsecond precision - 1686218963.123456
• %ms: Epoch unix timestamp - 1686218963123
• %a: Weekday (abbreviated) - Sun
• %A: Weekday (full) - Sunday
• %w: Weekday (decimal) - 0 (Sunday), 6 (Saturday)
• %d: Day of the month (zero-padded) - 01, 02, ..., 31
• %b: Month (abbreviated) - Jan
• %B: Month (full) - January
• %m: Month (zero-padded) - 01, 02, ..., 12
• %y: Year (without century, zero-padded) - 00, 01, ..., 99
• %Y: Year (with century) - 0001, 0002, ..., 9999
• %H: Hour (24-hour, zero-padded) - 00, 01, ..., 23
• %I: Hour (12-hour, zero-padded) - 01, 02, ..., 12
• %p: AM/PM indicator
• %M: Minute (zero-padded) - 00, 01, ..., 59
• %S: Second (zero-padded) - 00, 01, ..., 59
• %f: Microsecond (zero-padded to 6 digits) - 000000, 000001, ..., 999999
• %_ms: Millisecond (zero-padded to 3 digits) - 000, 001, ..., 999
• %z: UTC offset - (empty), +0000, -04:00
• %Z: Time zone name - (empty), UTC, GMT
• %j: Day of the year (zero-padded) - 001, 002, ..., 366
• %U: Week number of the year (Sunday as first day) - 00, 01, ..., 53
• %W: Week number of the year (Monday as first day) - 00, 01, ..., 53
• %c: Date and time representation - Tue Aug 16 21:30:00 1988
• %x: Date representation - 08/16/1988
• %X: Time representation - 21:30:00
• %%: Literal '%' character

Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the Python documentation.

Ceiling applied on the datetime value. Must be formatted with the datetime_format field.

Floor applied on the datetime value. Must be formatted with the datetime_format field.

OAuth specific blob. This is a Json Schema used to validate Json configurations used as input to OAuth. Must be a valid non-nested JSON that refers to properties from ConnectorSpecification.connectionSpecification using special annotation 'path_in_connector_config'. These are input values the user is entering through the UI to authenticate to the connector, that might also shared as inputs for syncing data via the connector. Examples: if no connector values is shared during oauth flow, oauth_user_input_from_connector_config_specification=[] if connector values such as 'app_id' inside the top level are used to generate the API url for the oauth flow, oauth_user_input_from_connector_config_specification={ app_id: { type: string path_in_connector_config: ['app_id'] } } if connector values such as 'info.app_id' nested inside another object are used to generate the API url for the oauth flow, oauth_user_input_from_connector_config_specification={ app_id: { type: string path_in_connector_config: ['info', 'app_id'] } }

The DeclarativeOAuth specific blob. Pertains to the fields defined by the connector relating to the OAuth flow.

Interpolation capabilities:

• The variables placeholders are declared as {{my_var}}.

• The nested resolution variables like {{ {{my_nested_var}} }} is allowed as well.

• The allowed interpolation context is:

  • base64Encoder - encode to base64, {{ {{my_var_a}}:{{my_var_b}} | base64Encoder }}
  • base64Decorer - decode from base64 encoded string, {{ {{my_string_variable_or_string_value}} | base64Decoder }}
  • urlEncoder - encode the input string to URL-like format, {{ https://test.host.com/endpoint | urlEncoder}}
  • urlDecorer - decode the input url-encoded string into text format, {{ urlDecoder:https%3A%2F%2Fairbyte.io | urlDecoder}}
  • codeChallengeS256 - get the codeChallenge encoded value to provide additional data-provider specific authorisation values, {{ {{state_value}} | codeChallengeS256 }}

Examples:

• The TikTok Marketing DeclarativeOAuth spec: { "oauth_connector_input_specification": { "type": "object", "additionalProperties": false, "properties": { "consent_url": "https://ads.tiktok.com/marketing_api/auth?{{client_id_key}}={{client_id_value}}&{{redirect_uri_key}}={{ {{redirect_uri_value}} | urlEncoder}}&{{state_key}}={{state_value}}", "access_token_url": "https://business-api.tiktok.com/open_api/v1.3/oauth2/access_token/", "access_token_params": { "{{ auth_code_key }}": "{{ auth_code_value }}", "{{ client_id_key }}": "{{ client_id_value }}", "{{ client_secret_key }}": "{{ client_secret_value }}" }, "access_token_headers": { "Content-Type": "application/json", "Accept": "application/json" }, "extract_output": ["data.access_token"], "client_id_key": "app_id", "client_secret_key": "secret", "auth_code_key": "auth_code" } } }

OAuth specific blob. This is a Json Schema used to validate Json configurations produced by the OAuth flows as they are returned by the distant OAuth APIs. Must be a valid JSON describing the fields to merge back to ConnectorSpecification.connectionSpecification. For each field, a special annotation path_in_connector_config can be specified to determine where to merge it, Examples: complete_oauth_output_specification={ refresh_token: { type: string, path_in_connector_config: ['credentials', 'refresh_token'] } }

OAuth specific blob. This is a Json Schema used to validate Json configurations persisted as Airbyte Server configurations. Must be a valid non-nested JSON describing additional fields configured by the Airbyte Instance or Workspace Admins to be used by the server when completing an OAuth flow (typically exchanging an auth code for refresh token). Examples: complete_oauth_server_input_specification={ client_id: { type: string }, client_secret: { type: string } }

OAuth specific blob. This is a Json Schema used to validate Json configurations persisted as Airbyte Server configurations that also need to be merged back into the connector configuration at runtime. This is a subset configuration of complete_oauth_server_input_specification that filters fields out to retain only the ones that are necessary for the connector to function with OAuth. (some fields could be used during oauth flows but not needed afterwards, therefore they would be listed in the complete_oauth_server_input_specification but not complete_oauth_server_output_specification) Must be a valid non-nested JSON describing additional fields configured by the Airbyte Instance or Workspace Admins to be used by the connector when using OAuth flow APIs. These fields are to be merged back to ConnectorSpecification.connectionSpecification. For each field, a special annotation path_in_connector_config can be specified to determine where to merge it, Examples: complete_oauth_server_output_specification={ client_id: { type: string, path_in_connector_config: ['credentials', 'client_id'] }, client_secret: { type: string, path_in_connector_config: ['credentials', 'client_secret'] } }

The number of records to include in each pages.

Using the offset with value 0 during the first request

The number of records to include in each pages.

Index of the first page to request.

Using the page number with value defined by start_from_page during the first request

Reference to the parent stream.

The primary key of records from the parent stream that will be used during the retrieval of records for the current substream. This parent identifier field is typically a characteristic of the child records being extracted from the source API.

While iterating over parent records during a sync, the parent_key value can be referenced by using this field.

Specifies the key field or path and where in the request a component's value should be injected.

Indicates whether the parent stream should be read incrementally based on updates in the child stream.

If set, this will enable lazy reading, using the initial read of parent records to extract child records.

Array of field paths to include as additional fields in the stream slice. Each path is an array of strings representing keys to access fields in the respective parent record. Accessible via stream_slice.extra_fields. Missing fields are set to None.

Describes the path to the field that should be extracted

Requester component that describes how to fetch the properties to query from a remote API endpoint.

The type used to determine the maximum number of properties per chunk

The maximum amount of properties that can be retrieved per request according to the limit type.

Record merge strategy that combines records according to fields on the record.

The set of properties that will be queried for in the outbound request. This can either be statically defined or dynamic based on an API endpoint

The list of properties that should be included in every set of properties when multiple chunks of properties are being requested.

For APIs with restrictions on the amount of properties that can be requester per request, property chunking can be applied to make multiple requests with a subset of the properties.

The predicate to filter a record. Records will be removed if evaluated to False.

Responsible for filtering records to be emitted by the Source.

Responsible for normalization according to the schema.

If true, transformation will be applied before record filtering.

Array of paths defining the field to remove. Each item is an array whose field describe the path of a field to remove.

The predicate to filter a property by a property value. Property will be removed if it is empty OR expression is evaluated to True.,

Configures where the descriptor should be set on the HTTP requests. Note that request parameters that are already encoded in the URL path will not be duplicated.

Configures which key should be used in the location that the descriptor is being injected into. We hope to eventually deprecate this field in favor of field_path for all request_options, but must currently maintain it for backwards compatibility in the Builder.

Configures a path to be used for nested structures in JSON body requests (e.g. GraphQL queries)

The name of the session token header that will be injected in the request

Path of the login URL (do not include the base URL)

Name of the key of the session token to be extracted from the response

Path of the URL to use to validate that the session token is valid (do not include the base URL)

Session token to use if using a pre-defined token. Not needed if authenticating with username + password pair

Username used to authenticate and obtain a session token

Password used to authenticate and obtain a session token

The stream name.

A stream whose behavior is described by a set of declarative low code components.

A stream whose behavior is described by a set of declarative low code components.

Requester component that describes how to prepare HTTP requests to send to the source API.

Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.

Component decoding the response so records can be extracted.

Paginator component that describes how to navigate through the API's pages.

If true, the partition router and incremental request options will be ignored when paginating requests. Request options set directly on the requester will not be ignored.

Used to iteratively execute requests over a set of values, such as a parent stream's records or a list of constant values.

Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.

Async Job Status to Airbyte CDK Async Job Status mapping.

Responsible for fetching the actual status of the async job.

Responsible for fetching the final result urls provided by the completed / finished / ready async job.

Requester component that describes how to prepare HTTP requests to send to the source API to create the async server-side job.

Requester component that describes how to prepare HTTP requests to send to the source API to fetch the status of the running async job.

Requester component that describes how to prepare HTTP requests to send to the source API to download the data provided by the completed async job.

Responsible for fetching the records from provided urls.

The time in minutes after which the single Async Job should be considered as Timed Out.

Requester component that describes how to prepare HTTP requests to send to the source API to extract the url from polling response by the completed async job.

Paginator component that describes how to navigate through the API's pages during download.

Requester component that describes how to prepare HTTP requests to send to the source API to abort a job once it is timed out from the source's perspective.

Requester component that describes how to prepare HTTP requests to send to the source API to delete a job once the records are extracted.

PartitionRouter component that describes how to partition the stream, enabling incremental syncs and checkpointing.

Component decoding the response so records can be extracted.

Component decoding the download response so records can be extracted.

A connection specification describing how a the connector can be configured.

URL of the connector's documentation page.

Additional and optional specification object to describe what an 'advanced' Auth flow would need to function.

• A connector should be able to fully function with the configuration as described by the ConnectorSpecification in a 'basic' mode.
• The 'advanced' mode provides easier UX for the user with UI improvements and automations. However, this requires further setup on the server side by instance or workspace admins beforehand. The trade-off is that the user does not have to provide as many technical inputs anymore and the auth process is faster and easier to complete.

The list of transformations that will attempt to be applied on an incoming unmigrated config. The transformations will be applied in the order they are defined.

The description/purpose of the config migration.

Specifies which parent streams are being iterated over and how parent records should be used to partition the child stream data set.

The name of the response header defining how long to wait before retrying.

Optional regex to apply on the header to extract its value. The regex should define a capture group defining the wait time.

Given the value extracted from the header is greater than this value, stop the stream.

The number of partitions to include in each group. This determines how many partition values are batched together in a single slice.

The partition router whose output will be grouped. This can be any valid partition router component.

If true, ensures that partitions are unique within each group by removing duplicates based on the partition key.

The name of the response header defining how long to wait before retrying.

Minimum time to wait before retrying.

Optional regex to apply on the header to extract its value. The regex should define a capture group defining the wait time.

A list of potentially nested fields indicating the full path where value will be added or updated.

The dynamic or static value to assign to the key. Interpolated values can be used to dynamically determine the value during runtime.

A schema type.

Determines whether to create a new path if it doesn't exist (true) or only update existing paths (false). When set to true, the resolver will create new paths in the stream template if they don't exist. When false (default), it will only update existing paths.

A condition that must be met for the mapping to be applied. This property is only supported for ConfigComponentsResolver.

Component used to coordinate how records are extracted across stream slices and request pages.

A list of potentially nested fields indicating the full path in source config file where streams configs located.

A list of default values, each matching the structure expected from the parsed component value.

A list of object of parameters for stream, each object in the list represents params for one stream.

(This component is experimental. Use at your own risk.) Represents a stream parameters definition to set up dynamic streams from defined values in manifest.

Reference to the stream template.

Component resolve and populates stream templates with components values.

The dynamic stream name.

Whether or not to prioritize parent parameters over component parameters when constructing dynamic streams. Defaults to true for backward compatibility.

Request body GraphQL query object

The GraphQL query to be executed

List of potentially nested fields describing the full path of the field to validate. Use "*" to validate all values from an array.

The condition that the specified config value will be evaluated against

The value to be validated. Can be a literal value or interpolated from configuration.

The validation strategy to apply to the value.