Meltano project definition (SchemaStore) JSON Schema

Type	object
File match	`meltano.yml` `meltano-manifest.json` `meltano-manifest..json`
Schema URL	https://catalog.lintel.tools/schemas/schemastore/meltano-project-definition/latest.json
Source	https://raw.githubusercontent.com/meltano/meltano/main/src/meltano/schemas/meltano.schema.json

Validate with Lintel

npx @lintel/lintel check

Type: object

Meltano (https://meltano.com) is an open source platform for building, running & orchestrating ELT pipelines

Properties

version const: 1

DEPRECATED: The meltano.yml version field is deprecated and will be removed in a future release. Please remove it from your meltano.yml file. To specify Meltano version requirements, use 'requires_meltano' instead.

Constant: 1

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

default_environment string

A default environment to be used when one is not present as an argument or as an environment variable.

send_anonymous_usage_stats boolean

Share anonymous usage data with the Meltano team to help them gauge interest and allocate development time

Default: true

project_id string

A globally unique project identifier. Defaults to a Meltano-generated unique GUID for each project.

requires_meltano string

A version specifier for the Meltano version required by this project. If the version of Meltano being used does not satisfy this requirement, Meltano will exit with an error.

Examples: ">=3,<4", ">=3.6,<3.7"

database_uri string

A database URI for the project system database. Defaults to a SQLite file stored at .meltano/meltano.db

Default: "sqlite:///$MELTANO_PROJECT_ROOT/.meltano/meltano.db"

python string

The python version to use for plugins, specified as a path, as the name of an executable to find within a directory in $PATH, or as a version number (e.g. '3.11'). If not specified, the python executable that was used to run Meltano will be used (within a separate virtual environment). This can be overridden on a per-plugin basis by setting the python property for the plugin.

Examples: "/usr/bin/python3.10", "python", "python3.11", "3.11"

state_backend object

Configuration for state backend.

6 nested properties

uri string

The URI pointing to the backend to use.

Default: "systemdb"

lock_timeout_seconds integer

The number of seconds a lock on a state file is considered valid.

lock_retry_seconds integer

The number of seconds to wait between retrying lock acquisition.

azure object

Configuration for Azure Blob Storage state backend.

1 nested properties

connection_string string

The azure connection string to use for connecting to Azure Blob Storage.

gcs object

Configuration for Google Cloud Storage state backend.

1 nested properties

application_credentials string

Path to a Google credentials file to use for authenticating to Google Cloud Storage.

s3 object

Configuration for AWS S3 state backend.

3 nested properties

aws_access_key_id string

AWS access key ID to use for authenticating to AWS S3.

aws_secret_access_key string

AWS secret access key to use for authenticating to AWS S3.

endpoint_url string

S3 endpoint URL to use.

venv object

Configuration for plugin virtual environments.

1 nested properties

backend string

The virtual environment backend to use.

Default: "virtualenv"

Values: "virtualenv" "uv"

environments environments[]

An array of environments (i.e. dev, stage, prod) with override configs for plugins based on the environment its run in.

include_paths string[]

An array of paths (relative to the project directory) of other Meltano config files to include in the configuration.

plugins object

8 nested properties

extractors extractors[]

loaders loaders[]

mappers mappers[]

orchestrators orchestrators[]

transformers transformers[]

files files[]

utilities utilities[]

transforms transforms[]

schedules schedules[]

Scheduled ELT jobs, added using 'meltano schedule'

env Record<string, string>

Mapping of environment variables for use in config.

Examples: {"DBT_PROFILES_DIR":"/usr/app/profile/","SNOWFLAKE_ACCOUNT":"my.snowflake.account.com"}

jobs jobs[]

Jobs, added using 'meltano job'

database_max_retries integer

The number of times to retry connecting to the Meltano database.

Default: 3

min=0

database_retry_timeout integer

The time in seconds to wait between Meltano database connection attempts.

Default: 5

min=0

project_readonly boolean

Whether the project is read-only.

Default: false

cli object

CLI related settings.

1 nested properties

log_config string

The path to the logging config, relative to the Meltano project root.

Default: "logging.yaml"

elt object

ELT related settings

1 nested properties

buffer_size integer

The size of the ELT buffer in bytes.

Default: 10485760

experimental boolean

Whether experimental features should be enabled.

Default: false

ff object

A feature flag, which can be used to enable or disable particular Meltano features.

2 nested properties

strict_env_var_mode boolean

Whether references to undefined environment variables cause Meltano to exit with an error.

Default: false

plugin_locks_required boolean

Whether plugin locks are required to run a job.

Default: false

hub_api_root string | null

The URL to the root of the Meltano Hub API.

Default: null

Examples: "https://hub.meltano.com/meltano/api/v1"

hub_url string

The URL for Meltano Hub.

Default: "https://hub.meltano.com"

hub_url_auth string | boolean | null

The value of the Authorization header when making requests to Meltano Hub. If null, false, or unset, no Authorization header is used.

Default: null

Examples: "Bearer $ACCESS_TOKEN"

snowplow object

Snowplow related settings.

1 nested properties

collector_endpoints string[]

Array of Snowplow collector endpoints.

Default:

[
  "https://sp.meltano.com"
]

auto_install boolean

Whether to automatically install any required plugins on command invocation. A plugin will be auto-installed when its virtual environment does not already exist or pip_url is changed.

Default: true

Definitions

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

env Record<string, string>

Mapping of environment variables for use in config.

Examples:

{ "DBT_PROFILES_DIR": "/usr/app/profile/", "SNOWFLAKE_ACCOUNT": "my.snowflake.account.com" }

base_setting object

name string required

The name of the setting

Examples: "account_id"

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

aliases string[]

Alternative setting names that can be used in 'meltano.yml' and 'meltano config set'

Examples: "accountId", "account_identifier"

label string

A user friendly label for the setting

Examples: "Account ID"

value

The default value of this setting if not otherwise defined

placeholder string

A placeholder value for this setting

Examples: "Ex. 18161"

kind string

The type of value this setting contains

One of: enum enum, enum enum

description string

A description for what this setting does

Examples: "The unique account identifier for your Stripe Account"

tooltip string

A phrase to provide additional information on this setting

Examples: "Here is some additional info..."

documentation string

A link to documentation on this setting

Examples: "https://meltano.com/"

hidden boolean

A hidden setting should not be user-configurable

Default: false

sensitive boolean

A sensitive setting is redacted in command output

Default: false

env string

An alternative environment variable name to populate with this settings value in the plugin environment. Meltano takes the value of the setting and injects it into the plugin's runtime environment as this environment variable, in addition to the default environment variable (of the form <PLUGIN_NAME>_<SETTING_NAME>, etc.).

Examples: "GITLAB_API_TOKEN", "FACEBOOK_ADS_ACCESS_TOKEN"

value_processor enum

Use with kind: object to pre-process the keys in a particular way.

Values: "nest_object" "upcase_string"

value_post_processor const: "nest_object" | const: "upcase_string" | const: "stringify" | const: "parse_date"

oauth object

1 nested properties

provider string required

The name of a Meltano-supported OAuth provider

Examples: "google-adwords"

options object[]

schedules object

https://docs.meltano.com/guide/orchestration

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

name string

The schedule's unique name

Examples: "gitlab-to-jsonl"

job string

The name of a meltano job

Examples: "some-custom-job"

extractor string

The name of the extractor plugin

Examples: "tap-gitlab"

env Record<string, string>

Mapping of environment variables for use in config.

Examples: {"DBT_PROFILES_DIR":"/usr/app/profile/","SNOWFLAKE_ACCOUNT":"my.snowflake.account.com"}

loader string

The name of the loader plugin

Examples: "target-jsonl"

interval string

A UNIX cron expression to represent the frequency the scheduled job should execute

Examples: "@hourly", "@daily", "@weekly", "0 0 * * *"

pattern=^((@(hourly|daily|weekly|monthly|yearly|once))|((((\d+,)+\d+|(\d+|\*)(\/|-)\d+|\d+|\*) ?){5,6}))$

transform string

Describes if transforms should run, be skipped, or if only transforms should execute (skip extractors and loaders)

Default: "skip"

Values: "run" "skip" "only"

start_date string

The date when the schedule should first execute

Examples: "2020-08-06 00:00:00"

jobs object

https://docs.meltano.com/reference/command-line-interface#job

name string required

This jobs unique name

Examples: "gitlab-to-jsonl"

tasks string | string | string[][] required

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

environments object

name string required

The name of the environment.

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

config object

An object of environment specific configurations.

1 nested properties

plugins object

An object of environment specific plugin configurations.

7 nested properties

extractors array

loaders plugins[]

orchestrators plugins[]

transformers plugins[]

files plugins[]

utilities plugins[]

transforms plugins[]

env Record<string, string>

Mapping of environment variables for use in config.

Examples: {"DBT_PROFILES_DIR":"/usr/app/profile/","SNOWFLAKE_ACCOUNT":"my.snowflake.account.com"}

requires object

A set of plugin requirements.

plugin_requirement object

A single plugin requirement

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

name string

The name of the required plugin

variant string

The variant of the required plugin

plugin_generic object

name string required

The name of the plugin.

Examples: "target-jsonl"

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

description string

A description of the plugin.

plugin_type string

The type of the plugin.

Values: "extractors" "loaders" "orchestrators" "transformers" "files" "utilities" "transforms" "mappers"

repo string

The URL of the repository containing the source code for the plugin.

ext_repo string

The URL of the repository containing the source code for a Meltano extension plugin that typically wraps some other tool, in which case the 'repo' property is used for the URL of the repository for that tool.

inherit_from string

An existing plugin to inherit from.

pip_url string

The pip hosted package name or URL

Examples: "target-jsonl", "git+https://gitlab.com/meltano/tap-facebook.git", "wtforms==2.2.1 apache-airflow==1.10.2"

variant string

The variant of the plugin.

namespace string

The namespace of this plugin

Examples: "tap_csv", "target_jsonl"

config object

Your plugin configuration, type 'meltano config list' for details

label string

A user friendly label describing the plugin

Examples: "Stripe", "Facebook Ads"

logo_url string

An optional logo URL for this plugin

python string

The python version to use for this plugin, specified as a path, as the name of an executable to find within a directory in $PATH, or as a version number (e.g. '3.11'). If not specified, the top-level python setting will be used, or if it is not set, the python executable that was used to run Meltano will be used (within a separate virtual environment).

Examples: "/usr/bin/python3.10", "python", "python3.11", "3.11"

executable string

The plugin's executable name, as defined in setup.py (if a Python based plugin)

Examples: "tap-stripe", "tap-covid-19"

settings base_setting[]

docs string

A URL to the documentation for this plugin

settings_group_validation string[][]

commands Record<string, object | string>

A mapping of command names to either a string command, or an object containing info about the command.

requires object

A set of plugin requirements.

requires_meltano string

A version specifier for the Meltano version required by this plugin. If the version of Meltano being used does not satisfy this requirement, Meltano will exit with an error.

Examples: ">=3,<4", ">=3.6,<3.7"

env Record<string, string>

Mapping of environment variables for use in config.

Examples: {"DBT_PROFILES_DIR":"/usr/app/profile/","SNOWFLAKE_ACCOUNT":"my.snowflake.account.com"}

extractor_shared object

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

catalog string

A path to a catalog file (relative to the project directory) to be provided to the extractor when it is run in sync mode.

load_schema string

The name of the database schema extracted data should be loaded into.

metadata object

An object containing Singer stream and property metadata

Default:

{}

schema object

An object representing override rules for a Singer stream schema

Default:

{}

state string

A path to a state file (relative to the project directory) to be provided to the extractor when it is run as part of a pipeline using meltano el or meltano elt.

select string[]

An array of entity selection rules in the form '<entity|>.<attribute|>'

Default:

[
  "*.*"
]

select_filter string[]

An array of entity selection filter rules that are applied to the extractor's discovered or provided catalog file when the extractor is run.

Default:

[]

use_cached_catalog boolean

A boolean that determines if the catalog cache should be used or ignored.

Default: true

log_parser string

The log parser to use for the extractor.

Values: "singer-sdk"

extractors object

https://docs.meltano.com/concepts/plugins#extractors

loaders object

https://docs.meltano.com/concepts/plugins#loaders

orchestrators object

name string required

The name of the plugin.

Examples: "target-jsonl"

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

description string

A description of the plugin.

plugin_type string

The type of the plugin.

Values: "extractors" "loaders" "orchestrators" "transformers" "files" "utilities" "transforms" "mappers"

repo string

The URL of the repository containing the source code for the plugin.

ext_repo string

inherit_from string

An existing plugin to inherit from.

pip_url string

The pip hosted package name or URL

Examples: "target-jsonl", "git+https://gitlab.com/meltano/tap-facebook.git", "wtforms==2.2.1 apache-airflow==1.10.2"

variant string

The variant of the plugin.

namespace string

The namespace of this plugin

Examples: "tap_csv", "target_jsonl"

config object

Your plugin configuration, type 'meltano config list' for details

label string

A user friendly label describing the plugin

Examples: "Stripe", "Facebook Ads"

logo_url string

An optional logo URL for this plugin

python string

Examples: "/usr/bin/python3.10", "python", "python3.11", "3.11"

executable string

The plugin's executable name, as defined in setup.py (if a Python based plugin)

Examples: "tap-stripe", "tap-covid-19"

settings base_setting[]

docs string

A URL to the documentation for this plugin

settings_group_validation string[][]

commands Record<string, object | string>

A mapping of command names to either a string command, or an object containing info about the command.

requires object

A set of plugin requirements.

requires_meltano string

A version specifier for the Meltano version required by this plugin. If the version of Meltano being used does not satisfy this requirement, Meltano will exit with an error.

Examples: ">=3,<4", ">=3.6,<3.7"

env Record<string, string>

Mapping of environment variables for use in config.

Examples: {"DBT_PROFILES_DIR":"/usr/app/profile/","SNOWFLAKE_ACCOUNT":"my.snowflake.account.com"}

transformers object

name string required

The name of the plugin.

Examples: "target-jsonl"

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

description string

A description of the plugin.

plugin_type string

The type of the plugin.

Values: "extractors" "loaders" "orchestrators" "transformers" "files" "utilities" "transforms" "mappers"

repo string

The URL of the repository containing the source code for the plugin.

ext_repo string

inherit_from string

An existing plugin to inherit from.

pip_url string

The pip hosted package name or URL

Examples: "target-jsonl", "git+https://gitlab.com/meltano/tap-facebook.git", "wtforms==2.2.1 apache-airflow==1.10.2"

variant string

The variant of the plugin.

namespace string

The namespace of this plugin

Examples: "tap_csv", "target_jsonl"

config object

Your plugin configuration, type 'meltano config list' for details

label string

A user friendly label describing the plugin

Examples: "Stripe", "Facebook Ads"

logo_url string

An optional logo URL for this plugin

python string

Examples: "/usr/bin/python3.10", "python", "python3.11", "3.11"

executable string

The plugin's executable name, as defined in setup.py (if a Python based plugin)

Examples: "tap-stripe", "tap-covid-19"

settings base_setting[]

docs string

A URL to the documentation for this plugin

settings_group_validation string[][]

commands Record<string, object | string>

A mapping of command names to either a string command, or an object containing info about the command.

requires object

A set of plugin requirements.

requires_meltano string

A version specifier for the Meltano version required by this plugin. If the version of Meltano being used does not satisfy this requirement, Meltano will exit with an error.

Examples: ">=3,<4", ">=3.6,<3.7"

env Record<string, string>

Mapping of environment variables for use in config.

Examples: {"DBT_PROFILES_DIR":"/usr/app/profile/","SNOWFLAKE_ACCOUNT":"my.snowflake.account.com"}

files object

name string required

The name of the plugin.

Examples: "target-jsonl"

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

description string

A description of the plugin.

plugin_type string

The type of the plugin.

Values: "extractors" "loaders" "orchestrators" "transformers" "files" "utilities" "transforms" "mappers"

repo string

The URL of the repository containing the source code for the plugin.

ext_repo string

inherit_from string

An existing plugin to inherit from.

pip_url string

The pip hosted package name or URL

Examples: "target-jsonl", "git+https://gitlab.com/meltano/tap-facebook.git", "wtforms==2.2.1 apache-airflow==1.10.2"

variant string

The variant of the plugin.

namespace string

The namespace of this plugin

Examples: "tap_csv", "target_jsonl"

config object

Your plugin configuration, type 'meltano config list' for details

label string

A user friendly label describing the plugin

Examples: "Stripe", "Facebook Ads"

logo_url string

An optional logo URL for this plugin

python string

Examples: "/usr/bin/python3.10", "python", "python3.11", "3.11"

executable string

The plugin's executable name, as defined in setup.py (if a Python based plugin)

Examples: "tap-stripe", "tap-covid-19"

settings base_setting[]

docs string

A URL to the documentation for this plugin

settings_group_validation string[][]

commands Record<string, object | string>

A mapping of command names to either a string command, or an object containing info about the command.

requires object

A set of plugin requirements.

requires_meltano string

A version specifier for the Meltano version required by this plugin. If the version of Meltano being used does not satisfy this requirement, Meltano will exit with an error.

Examples: ">=3,<4", ">=3.6,<3.7"

env Record<string, string>

Mapping of environment variables for use in config.

Examples: {"DBT_PROFILES_DIR":"/usr/app/profile/","SNOWFLAKE_ACCOUNT":"my.snowflake.account.com"}

utilities object

name string required

The name of the plugin.

Examples: "target-jsonl"

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

description string

A description of the plugin.

plugin_type string

The type of the plugin.

Values: "extractors" "loaders" "orchestrators" "transformers" "files" "utilities" "transforms" "mappers"

repo string

The URL of the repository containing the source code for the plugin.

ext_repo string

inherit_from string

An existing plugin to inherit from.

pip_url string

The pip hosted package name or URL

Examples: "target-jsonl", "git+https://gitlab.com/meltano/tap-facebook.git", "wtforms==2.2.1 apache-airflow==1.10.2"

variant string

The variant of the plugin.

namespace string

The namespace of this plugin

Examples: "tap_csv", "target_jsonl"

config object

Your plugin configuration, type 'meltano config list' for details

label string

A user friendly label describing the plugin

Examples: "Stripe", "Facebook Ads"

logo_url string

An optional logo URL for this plugin

python string

Examples: "/usr/bin/python3.10", "python", "python3.11", "3.11"

executable string

The plugin's executable name, as defined in setup.py (if a Python based plugin)

Examples: "tap-stripe", "tap-covid-19"

settings base_setting[]

docs string

A URL to the documentation for this plugin

settings_group_validation string[][]

commands Record<string, object | string>

A mapping of command names to either a string command, or an object containing info about the command.

requires object

A set of plugin requirements.

requires_meltano string

A version specifier for the Meltano version required by this plugin. If the version of Meltano being used does not satisfy this requirement, Meltano will exit with an error.

Examples: ">=3,<4", ">=3.6,<3.7"

env Record<string, string>

Mapping of environment variables for use in config.

Examples: {"DBT_PROFILES_DIR":"/usr/app/profile/","SNOWFLAKE_ACCOUNT":"my.snowflake.account.com"}

transforms object

https://docs.meltano.com/concepts/plugins#transforms

mapper_specific object

annotations Record<string, object>

Arbitrary annotations keyed by tool/vendor name - not processed by the core Meltano library or CLI

mappings object[]

Name of the invokable mappings and their associated configs.

mappers object

https://docs.meltano.com/concepts/plugins#mappers