Vector
Configuration file for Vector, an observability data pipeline agent and aggregator
| File match |
vector.json
vector.toml
vector.yaml
|
|---|---|
| Schema URL | https://catalog.lintel.tools/schemas/schemastore/vector/latest.json |
| Source | https://www.schemastore.org/vector.json |
Validate with Lintel
npx @lintel/lintel checkA complete Vector configuration.
All of
Definitions
The user configuration to choose the metric tag strategy.
Options for building a LengthDelimitedDecoder or LengthDelimitedEncoder.
Length field byte order (little or big endian)
Number of bytes representing the field length
Number of bytes in the header before the length field
Maximum frame length
Configures how events are decoded from raw bytes.
Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
Config used to build a CsvSerializer.
The CSV Serializer Options.
7 nested properties
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
Set the capacity (in bytes) of the internal buffer used in the CSV writer. This defaults to a reasonable setting.
The field delimiter to use when writing CSV.
This is enabled by default, but it may be disabled. When disabled, quotes in field data are escaped instead of doubled.
In some variants of CSV, quotes are escaped using a special escape character like \ (instead of escaping quotes by doubling them).
To use this, double_quotes needs to be disabled as well otherwise it is ignored.
The quote character to use when writing CSV.
The quoting style to use when writing CSV data.
Config used to build a JsonSerializer.
Options for the JsonSerializer.
{
"pretty": false
}1 nested properties
Whether to use pretty JSON formatting.
The user configuration to choose the metric tag strategy.
Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
A file path.
Configuration of the region/endpoint to use when interacting with an AWS service.
Many methods exist for matching events, such as using a VRL expression, a Datadog Search query string, or hard-coded matchers like "must be a metric" or "fields A, B, and C must match these constraints".
As VRL is the most common way to apply conditions to events, this type provides a shortcut to define VRL expressions directly in the configuration by passing the VRL expression as a string:
condition = '.message == "hooray"'
When other condition types are required, they can be specified with an enum-style notation:
condition.type = 'datadog_search'
condition.source = 'NOT "foo"'
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an HTTP header without any additional encryption beyond what is provided by the transport itself.
Configuration of the authentication strategy when interacting with NATS.
Configuration settings for InfluxDB v0.x/v1.x.
Configuration settings for InfluxDB v2.x.
All compression algorithms use the default compression level unless otherwise specified.
HTTP Basic authentication configuration.
Configuration of multi-line aggregation.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
Wrapper for sensitive strings containing credentials
List of allowed origin IP networks. IP addresses must be in CIDR notation.
TCP keepalive settings for socket-based components.
TLS configuration.
Configures the TLS options for incoming/outgoing connections.
TlsEnableableConfig for sources, adding metadata from the client certificate.
A wrapper around OwnedValuePath that allows it to be used in Vector config.
This requires a valid path to be used. If you want to allow optional paths,
use [optional_path::OptionalValuePath].
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
This can refer to any valid timezone as defined in the TZ database, or "local" which refers to the system local timezone. It will default to the globally configured timezone.
File position to use when reading a new file.
A span of time, in fractional seconds.
A span of time, in whole milliseconds.
A span of time, in whole seconds.
A file path.
An internet socket address, either IPv4 or IPv6.
AMQP connection options.
The URI has the format of
amqp://<user>:<password>@<host>:<port>/<vhost>?timeout=<seconds>.
The default vhost can be specified by using a value of %2f.
To connect over TLS, a scheme of amqps can be specified instead. For example,
amqps://.... Additional TLS settings, such as client certificate verification, can be
configured under the tls section.
Configuration of the authentication strategy for interacting with AWS services.
Configuration of the region/endpoint to use when interacting with an AWS service.
Custom endpoint for use with AWS-compatible services.
Configures how events are encoded into raw bytes.
Encoding configuration.
Configures how events are encoded into raw bytes.
Framing configuration.
Transformations to prepare an event for serialization.
List of fields that are excluded from the encoded event.
List of fields that are included in the encoded event.
Format used for timestamp fields.
Many methods exist for matching events, such as using a VRL expression, a Datadog Search query string, or hard-coded matchers like "must be a metric" or "fields A, B, and C must match these constraints".
As VRL is the most common way to apply conditions to events, this type provides a shortcut to define VRL expressions directly in the configuration by passing the VRL expression as a string:
condition = '.message == "hooray"'
When other condition types are required, they can be specified with an enum-style notation:
condition.type = 'datadog_search'
condition.source = 'NOT "foo"'
An input describes not only the type of event to insert, but also which transform within the configuration to insert it to.
The name of the transform to insert the input event to.
Only relevant when type is log.
Only relevant when type is metric.
Only relevant when type is vrl.
Can be either raw, vrl, log, or `metric.
Use this only when the input event should be a raw event (i.e. unprocessed/undecoded log
event) and when the input type is set to raw.
Configure output for component when generated with graph command
They are added to the node as provided
Fully resolved sink component.
Fully resolved source component.
Fully resolved transform component.
Configuration for the unit_test sink.
Name of the test that this sink is being used for.
List of names of the transform/branch associated with this sink.
Configuration for the unit_test source.
Configuration for the unit_test_stream sink.
Configuration for the unit_test_stream source.
Configuration for the file enrichment table.
File-specific settings.
2 nested properties
File encoding configuration.
A file path.
This is used to coerce log fields from strings into their proper types. The available types are listed in the Types list below.
Timestamp coercions need to be prefaced with timestamp|, for example "timestamp|%F". Timestamp specifiers can use either of the following:
- One of the built-in-formats listed in the
Timestamp Formatstable below. - The time format specifiers from Rust’s
chronolibrary.
Types
boolstringfloatintegerdatetimestamp(see the table below for formats)
Timestamp Formats
| Format | Description | Example |
|---|---|---|
%F %T | YYYY-MM-DD HH:MM:SS | 2020-12-01 02:37:54 |
%v %T | DD-Mmm-YYYY HH:MM:SS | 01-Dec-2020 02:37:54 |
%FT%T | ISO 8601/RFC 3339, without time zone | 2020-12-01T02:37:54 |
%FT%TZ | ISO 8601/RFC 3339, UTC | 2020-12-01T09:37:54Z |
%+ | ISO 8601/RFC 3339, UTC, with time zone | 2020-12-01T02:37:54-07:00 |
%a, %d %b %Y %T | RFC 822/RFC 2822, without time zone | Tue, 01 Dec 2020 02:37:54 |
%a %b %e %T %Y | ctime format | Tue Dec 1 02:37:54 2020 |
%s | UNIX timestamp | 1606790274 |
%a %d %b %T %Y | date command, without time zone | Tue 01 Dec 02:37:54 2020 |
%a %d %b %T %Z %Y | date command, with time zone | Tue 01 Dec 02:37:54 PST 2020 |
%a %d %b %T %z %Y | date command, with numeric time zone | Tue 01 Dec 02:37:54 -0700 2020 |
%a %d %b %T %#z %Y | date command, with numeric time zone (minutes can be missing or present) | Tue 01 Dec 02:37:54 -07 2020 |
{}Configuration for the geoip enrichment table.
Other databases, such as the country database, are not supported.
mmdb enrichment table can be used for other databases.
MaxMind includes localized versions of some of the fields within their database, such as country name. This setting can control which of those localized versions are returned by the transform.
More information on which portions of the geolocation data are localized, and what languages are available, can be found here.
Configuration for the mmdb enrichment table.
Configuration of the authentication strategy for interacting with GCP services.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Skip all authentication handling. For use with integration tests only.
Configuration of HTTP server keepalive parameters.
A value of 0.1 means that the actual duration will be between 90% and 110% of the specified maximum duration.
Only applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 requests.
A random jitter configured by max_connection_age_jitter_factor is added
to the specified duration to spread out connection storms.
Configuration of internal metrics for file-based components.
This is useful for distinguishing between different files while monitoring. However, the tag's cardinality is unbounded.
Kafka authentication configuration.
Configuration for SASL authentication when interacting with Kafka.
Configuration for the http provider.
Configuration for the aws_secrets_manager secrets backend.
Configuration for the directory secrets backend.
A file path.
Remove trailing whitespace from file contents.
Configuration for the exec secrets backend.
The path to the script or binary must be the first argument.
The timeout, in seconds, to wait for the command to complete.
Configuration for the file secrets backend.
A file path.
Configuration for the test secrets backend.
Fixed value to replace all secrets with.
Configurable sinks in Vector.
Supports AMQP version 0.9.1
Configuration for the appsignal sink.
Wrapper for sensitive strings containing credentials
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}3 nested properties
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
The maximum size of a batch before it is flushed.
The maximum age of a batch before it is flushed.
All compression algorithms use the default compression level unless otherwise specified.
Transformations to prepare an event for serialization.
3 nested properties
List of fields that are excluded from the encoded event.
List of fields that are included in the encoded event.
Format used for timestamp fields.
The URI for the AppSignal API to send data to.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
Configuration for the aws_cloudwatch_logs sink.
Configuration for the aws_cloudwatch_metrics sink.
Base configuration for the aws_kinesis_ sinks.
The actual specific sink configuration types should either wrap this in a newtype wrapper,
or should extend it in a new struct with serde(flatten).
Configuration for the aws_kinesis_firehose sink.
Configuration for the aws_kinesis_streams sink.
Configuration for the aws_s3 sink.
Base Configuration aws_s_s for sns and sqs sink.
Configures how events are encoded into raw bytes.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Configuration of the authentication strategy for interacting with AWS services.
This value is a template which should result in a unique string for each event. See the AWS documentation for more about how AWS does message deduplication.
Can be applied only to FIFO queues.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
Configuration for the aws_sns sink.
Configuration for the aws_sqs sink.
Configuration for the axiom sink.
The Axiom dataset to write to.
Wrapper for sensitive strings containing credentials
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
The batch settings for the sink.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}All compression algorithms use the default compression level unless otherwise specified.
Only required when using personal tokens.
Outbound HTTP request settings.
Optional, constrains TLS settings for this sink.
Only required if not using Axiom Cloud.
Configuration for the azure_blob sink.
Configuration for the azure_monitor_logs sink.
Can only contain letters, numbers, and underscores (_), and may not exceed 100 characters.
Wrapper for sensitive strings containing credentials
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}Transformations to prepare an event for serialization.
3 nested properties
List of fields that are excluded from the encoded event.
List of fields that are included in the encoded event.
Format used for timestamp fields.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
The setting of log_schema.timestamp_key, usually timestamp, is used here by default.
This field should be used in rare cases where TimeGenerated should point to a specific log
field. For example, use this field to set the log field source_timestamp as holding the
value that should be used as TimeGenerated on the Azure side.
Configuration for the blackhole sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
By default, there is no limit.
Configuration for the clickhouse sink.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}All compression algorithms use the default compression level unless otherwise specified.
The database that contains the table that data is inserted into.
Sets date_time_input_format to best_effort, allowing ClickHouse to properly parse RFC3339/ISO 8601.
Transformations to prepare an event for serialization.
3 nested properties
List of fields that are excluded from the encoded event.
List of fields that are included in the encoded event.
Format used for timestamp fields.
The format to parse input data.
Sets insert_distributed_one_random_shard, allowing ClickHouse to insert data into a random shard when using Distributed Table Engine.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
If left unspecified, use the default provided by the ClickHouse server.
Configuration for the console sink.
Configuration for the databend sink.
The table that data is inserted into.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
The username and password to authenticate with. Overrides the username and password in DSN.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}Compression configuration.
The database that contains the table that data is inserted into. Overrides the database in DSN.
Configures how events are encoded into raw bytes.
{
"codec": "json",
"json": {
"pretty": false
}
}Defines how missing fields are handled for NDJson. Refer to https://docs.databend.com/sql/sql-reference/file-format-options#null_field_as
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
The TLS configuration to use when connecting to the Databend server.
Shared configuration for Datadog sinks. Contains the maximum set of common settings that applies to all DD sink components.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
If an event has a Datadog API key set explicitly in its metadata, it takes precedence over this setting.
This value can also be set by specifying the DD_API_KEY environment variable.
The value specified here takes precedence over the environment variable.
The endpoint must contain an HTTP scheme, and may specify a hostname or IP address and port. The API path should NOT be specified as this is handled by the sink.
If set, overrides the site option.
This value can also be set by specifying the DD_SITE environment variable.
The value specified here takes precedence over the environment variable.
If not specified by the environment variable, a default value of
datadoghq.com is taken.
Configuration for the datadog_events sink.
Configuration for the datadog_logs sink.
Configuration for the datadog_metrics sink.
Configuration for the datadog_traces sink.
Configuration for the elasticsearch sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Amazon OpenSearch Serverless requires this option to be set to auto (the default).
Elasticsearch Authentication strategies.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}Elasticsearch bulk mode configuration.
{
"action": "index",
"index": "vector-%Y.%m.%d",
"template_fallback_index": null,
"version": null,
"version_type": "internal"
}5 nested properties
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
The default index to write events to if the template in bulk.index cannot be resolved
Version field value.
Possible values are internal, external or external_gt and external_gte.
All compression algorithms use the default compression level unless otherwise specified.
Elasticsearch data stream mode configuration.
Options for determining the health of an endpoint.
This is only relevant for Elasticsearch <= 6.X. If you are using >= 7.0 you do not need to set this option since Elasticsearch has removed it.
Transformations to prepare an event for serialization.
3 nested properties
List of fields that are excluded from the encoded event.
List of fields that are included in the encoded event.
Format used for timestamp fields.
The endpoint must contain an HTTP scheme, and may specify a hostname or IP address and port.
The endpoint must contain an HTTP scheme, and may specify a hostname or IP address and port.
[]By default, the _id field is not set, which allows Elasticsearch to set this
automatically. Setting your own Elasticsearch IDs can hinder performance.
Configuration for the metric_to_log transform.
Elasticsearch Indexing mode.
Amazon OpenSearch service type
The name of the pipeline to apply.
Custom parameters to add to the query string for each HTTP request sent to Elasticsearch.
Outbound HTTP request settings.
To avoid duplicates in Elasticsearch, please use option id_key.
The type field was deprecated in Elasticsearch 7.x and removed in Elasticsearch 8.x.
If enabled, the doc_type option is ignored.
Configuration for the file sink.
Configuration for the gcp_cloud_storage sink.
Configuration for the gcp_pubsub sink.
Configuration for the gcp_stackdriver_logs sink.
Configuration for the gcp_stackdriver_metrics sink.
Configuration for the gcp_chronicle_unstructured sink.
Configuration for the greptimedb_logs sink.
The endpoint of the GreptimeDB server.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}All compression algorithms use the default compression level unless otherwise specified.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
Transformations to prepare an event for serialization.
3 nested properties
List of fields that are excluded from the encoded event.
List of fields that are included in the encoded event.
Format used for timestamp fields.
Custom parameters to add to the query string for each HTTP request sent to GreptimeDB.
This is required if your instance has authentication enabled.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
Pipeline version to be used for the logs.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
This is required if your instance has authentication enabled.
Configuration items for GreptimeDB
This sink uses GreptimeDB's gRPC interface for data ingestion. By default, GreptimeDB listens to port 4001 for gRPC protocol.
The address must include a port.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}Default to public, the default database of GreptimeDB.
Database can be created via create database statement on
GreptimeDB. If you are using GreptimeCloud, use dbname from the
connection information of your instance.
Set gRPC compression encoding for the request
Default to none, gzip or zstd is supported.
This is to keep consistency with GreptimeDB's naming pattern. By
default, this sink will use val for value column name, and ts for
time index name. When turned on, greptime_value and
greptime_timestamp will be used for these names.
If you are using this Vector sink together with other data ingestion sources of GreptimeDB, like Prometheus Remote Write and Influxdb Line Protocol, it is highly recommended to turn on this.
Also if there is a tag name conflict from your data source, for
example, you have a tag named as val or ts, you need to turn on
this option to avoid the conflict.
Default to false for compatibility.
This is required if your instance has authentication enabled.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
This is required if your instance has authentication enabled.
Configuration items for GreptimeDB
This sink uses GreptimeDB's gRPC interface for data ingestion. By default, GreptimeDB listens to port 4001 for gRPC protocol.
The address must include a port.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}Default to public, the default database of GreptimeDB.
Database can be created via create database statement on
GreptimeDB. If you are using GreptimeCloud, use dbname from the
connection information of your instance.
Set gRPC compression encoding for the request
Default to none, gzip or zstd is supported.
This is to keep consistency with GreptimeDB's naming pattern. By
default, this sink will use val for value column name, and ts for
time index name. When turned on, greptime_value and
greptime_timestamp will be used for these names.
If you are using this Vector sink together with other data ingestion sources of GreptimeDB, like Prometheus Remote Write and Influxdb Line Protocol, it is highly recommended to turn on this.
Also if there is a tag name conflict from your data source, for
example, you have a tag named as val or ts, you need to turn on
this option to avoid the conflict.
Default to false for compatibility.
This is required if your instance has authentication enabled.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
This is required if your instance has authentication enabled.
Configuration for the honeycomb sink.
Wrapper for sensitive strings containing credentials
The dataset to which logs are sent.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}3 nested properties
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
The maximum size of a batch before it is flushed.
The maximum age of a batch before it is flushed.
All compression algorithms use the default compression level unless otherwise specified.
Transformations to prepare an event for serialization.
3 nested properties
List of fields that are excluded from the encoded event.
List of fields that are included in the encoded event.
Format used for timestamp fields.
Honeycomb's endpoint to send logs to
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
Configuration for the http sink.
Configuration for the humio_logs sink.
Configures how events are encoded into raw bytes.
Wrapper for sensitive strings containing credentials
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}All compression algorithms use the default compression level unless otherwise specified.
The scheme (http or https) must be specified. No path should be included since the paths defined
by the Splunk API are used.
If unset, Humio defaults it to none.
An optional path that deserializes an empty string to None.
In public-facing APIs, this must (if present) be equal to the repository used to create the ingest token used for authentication.
In private cluster setups, Humio can be configured to allow these to be different.
For more information, see Humio’s Format of Data.
Can be used to tag events by specifying fields starting with #.
For more information, see Humio’s Format of Data.
[]Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
Typically the filename the logs originated from. Maps to @source in Humio.
An optional path that deserializes an empty string to None.
Overrides the name of the log field used to retrieve the nanosecond-enabled timestamp to send to Humio.
Configuration for the humio_metrics sink.
Configuration for the influxdb_logs sink.
Configuration for the influxdb_metrics sink.
Configuration for the kafka sink.
Configuration for the loki sink.
Configures how events are encoded into raw bytes.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}3 nested properties
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
The maximum size of a batch before it is flushed.
The maximum age of a batch before it is flushed.
All compression algorithms use the default compression level unless otherwise specified.
Both keys and values are templateable, which enables you to attach dynamic labels to events.
Valid label keys include *, and prefixes ending with *, to allow for the expansion of
objects into multiple labels. See Label expansion for more information.
Note: If the set of labels has high cardinality, this can cause drastic performance issues with Loki. To prevent this from happening, reduce the number of unique label keys and values.
Some sources may generate events with timestamps that aren't in chronological order. Even though the sink sorts the events before sending them to Loki, there is a chance that another event could come in that is out of order with the latest events sent to Loki. Prior to Loki 2.4.0, this was not supported and would result in an error during the push request.
If you're using Loki 2.4.0 or newer, Accept is the preferred action, which lets Loki handle
any necessary sorting/reordering. If you're using an earlier version, then you must use Drop
or RewriteTimestamp depending on which option makes the most sense for your use case.
The path to use in the URL of the Loki instance.
Whether or not to delete fields from the event when they are used as labels.
Whether or not to delete fields from the event when they are used in structured metadata.
The timestamp is still sent as event metadata for Loki to use for indexing.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
Both keys and values are templateable, which enables you to attach dynamic structured metadata to events.
Valid metadata keys include *, and prefixes ending with *, to allow for the expansion of
objects into multiple metadata entries. This follows the same logic as Label expansion.
{}When running Loki locally, a tenant ID is not required.
Configuration for the mezmo (formerly logdna) sink.
Wrapper for sensitive strings containing credentials
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}The default app that is set for events that do not contain a file or app field.
The default environment that is set for events that do not contain an env field.
Transformations to prepare an event for serialization.
3 nested properties
List of fields that are excluded from the encoded event.
List of fields that are included in the encoded event.
Format used for timestamp fields.
The IP address that is attached to each batch of events.
The MAC address that is attached to each batch of events.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
The tags that are attached to each batch of events.
Configuration for the mezmo (formerly logdna) sink.
Wrapper for sensitive strings containing credentials
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}The default app that is set for events that do not contain a file or app field.
The default environment that is set for events that do not contain an env field.
Transformations to prepare an event for serialization.
3 nested properties
List of fields that are excluded from the encoded event.
List of fields that are included in the encoded event.
Format used for timestamp fields.
The IP address that is attached to each batch of events.
The MAC address that is attached to each batch of events.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
The tags that are attached to each batch of events.
Configuration for the mqtt sink
Configures how events are encoded into raw bytes.
MQTT server address (The broker’s domain name or IP address).
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
If set to true, the MQTT session is cleaned on login.
MQTT client ID.
Connection keep-alive interval.
MQTT password.
TCP port of the MQTT server to connect to.
Supported Quality of Service types for MQTT.
Whether the messages should be retained by the server
MQTT username.
Configuration for the nats sink.
Configures how events are encoded into raw bytes.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
The URL must take the form of nats://server:port.
If the port is not specified it defaults to 4222.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
If set, the subject must belong to an existing JetStream stream.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
{
"timeout_secs": 60,
"rate_limit_duration_secs": 1,
"rate_limit_num": 9223372036854775807,
"retry_attempts": 9223372036854775807,
"retry_max_duration_secs": 30,
"retry_initial_backoff_secs": 1,
"retry_jitter_mode": "Full",
"adaptive_concurrency": {
"initial_concurrency": 1,
"decrease_ratio": 0.9,
"ewma_alpha": 0.4,
"rtt_deviation_scale": 2.5,
"max_concurrency_limit": 200
}
}9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
Configuration for the new_relic sink.
Wrapper for sensitive strings containing credentials
New Relic API endpoint.
Wrapper for sensitive strings containing credentials
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}3 nested properties
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
The maximum size of a batch before it is flushed.
The maximum age of a batch before it is flushed.
All compression algorithms use the default compression level unless otherwise specified.
Transformations to prepare an event for serialization.
3 nested properties
List of fields that are excluded from the encoded event.
List of fields that are included in the encoded event.
Format used for timestamp fields.
New Relic region.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
Configuration for the OpenTelemetry sink.
Protocol configuration
Configuration for the papertrail sink.
Configures how events are encoded into raw bytes.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
Configures the send buffer size using the SO_SNDBUF option on the socket.
Configuration for the prometheus_exporter sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
An internet socket address, either IPv4 or IPv6.
[
0.005,
0.01,
0.025,
0.05,
0.1,
0.25,
0.5,
1.0,
2.5,
5.0,
10.0
]This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with an underscore (_).
It should follow the Prometheus naming conventions.
While distributions as a lossless way to represent a set of samples for a metric is supported, Prometheus clients (the application being scraped, which is this sink) must aggregate locally into either an aggregated histogram or aggregated summary.
[
0.5,
0.75,
0.9,
0.95,
0.99
]This can sometimes be useful when the source of metrics leads to their timestamps being too far in the past for Prometheus to allow them, such as when aggregating metrics over long time periods, or when replaying old metrics from a disk buffer.
Configuration for the prometheus_remote_write sink.
The endpoint should include the scheme and the path to write to.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Authentication strategies.
The batch config for remote write.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null,
"aggregate": true
}[
0.005,
0.01,
0.025,
0.05,
0.1,
0.25,
0.5,
1.0,
2.5,
5.0,
10.0
]All compression algorithms use the default compression level unless otherwise specified.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with an underscore (_).
It should follow the Prometheus naming conventions.
[
0.5,
0.75,
0.9,
0.95,
0.99
]Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
If set, a header named X-Scope-OrgID is added to outgoing requests with the value of this setting.
This may be used by Cortex or other remote services to identify the tenant making the request.
Configuration for the pulsar sink.
Configures how events are encoded into raw bytes.
The endpoint should specify the pulsar protocol and port.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Authentication configuration.
Event batching behavior.
{
"max_events": null,
"max_bytes": null
}2 nested properties
The maximum size of a batch before it is flushed.
Note this is an unsigned 32 bit integer which is a smaller capacity than many of the other sink batch settings.
Supported compression types for Pulsar.
Custom connection retry options configuration for the Pulsar client.
If the field does not exist in the log event or metric tags, a blank value will be used.
If omitted, the key is not sent.
Pulsar uses a hash of the key to choose the topic-partition or uses round-robin if the record has no key.
The name of the producer. If not specified, the default name assigned by Pulsar is used.
If omitted, no properties will be written.
Configuration for the redis sink.
Configures how events are encoded into raw bytes.
The URL must take the form of protocol://server:port/db where the protocol can either be
redis or rediss for connections secured via TLS.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}3 nested properties
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
The maximum size of a batch before it is flushed.
The maximum age of a batch before it is flushed.
Redis data type to store messages in.
List-specific options.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
{
"timeout_secs": 60,
"rate_limit_duration_secs": 1,
"rate_limit_num": 9223372036854775807,
"retry_attempts": 9223372036854775807,
"retry_max_duration_secs": 30,
"retry_initial_backoff_secs": 1,
"retry_jitter_mode": "Full",
"adaptive_concurrency": {
"initial_concurrency": 1,
"decrease_ratio": 0.9,
"ewma_alpha": 0.4,
"rtt_deviation_scale": 2.5,
"max_concurrency_limit": 200
}
}9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
The Sematext region to send data to.
Configuration for the sematext_logs sink.
Wrapper for sensitive strings containing credentials
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}Transformations to prepare an event for serialization.
3 nested properties
List of fields that are excluded from the encoded event.
List of fields that are included in the encoded event.
Format used for timestamp fields.
Setting this option overrides the region option.
The Sematext region to send data to.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
Configuration for the sematext_metrics sink.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with a period (.).
Wrapper for sensitive strings containing credentials
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}3 nested properties
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
The maximum size of a batch before it is flushed.
The maximum age of a batch before it is flushed.
Setting this option overrides the region option.
The Sematext region to send data to.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
Configuration for the socket sink.
Splunk HEC acknowledgement configuration.
Configuration for the splunk_hec_logs sink.
Wrapper for sensitive strings containing credentials
Configures how events are encoded into raw bytes.
The scheme (http or https) must be specified. No path should be included since the paths defined
by the Splunk API are used.
Splunk HEC acknowledgement configuration.
{
"indexer_acknowledgements_enabled": true,
"query_interval": 10,
"retry_limit": 30,
"max_pending_acks": 1000000
}This option is only relevant to Splunk v8.x and above, and is only applied when
endpoint_target is set to event.
Setting this to true causes Splunk to extract the timestamp from the message text
rather than use the timestamp embedded in the event. The timestamp must be in the format
yyyy-mm-dd hh:mm:ss.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}All compression algorithms use the default compression level unless otherwise specified.
Splunk HEC endpoint configuration.
By default, the global log_schema.host_key option is used if log
events are Legacy namespaced, or the semantic meaning of "host" is used, if defined.
If not specified, the default index defined within Splunk is used.
[]Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
This is typically the filename the logs originated from.
If unset, the Splunk collector sets it.
If unset, Splunk defaults to httpevent.
By default, either the global log_schema.timestamp_key option is used
if log events are Legacy namespaced, or the semantic meaning of "timestamp" is used, if defined.
Configuration of the splunk_hec_metrics sink.
Wrapper for sensitive strings containing credentials
The scheme (http or https) must be specified. No path should be included since the paths defined
by the Splunk API are used.
Splunk HEC acknowledgement configuration.
{
"indexer_acknowledgements_enabled": true,
"query_interval": 10,
"retry_limit": 30,
"max_pending_acks": 1000000
}Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}All compression algorithms use the default compression level unless otherwise specified.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with a period (.).
An optional path that deserializes an empty string to None.
If not specified, the default index defined within Splunk is used.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
This is typically the filename the logs originated from.
If unset, the Splunk collector sets it.
If unset, Splunk defaults to httpevent.
Configuration for the statsd sink.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Event batching behavior.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
The maximum size of a batch before it is flushed.
The maximum age of a batch before it is flushed.
Event batching behavior.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
The maximum size of a batch before it is flushed.
The maximum age of a batch before it is flushed.
Event batching behavior.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
The maximum size of a batch before it is flushed.
The maximum age of a batch before it is flushed.
Event batching behavior.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
The maximum size of a batch before it is flushed.
The maximum age of a batch before it is flushed.
All compression algorithms use the default compression level unless otherwise specified.
Compression level.
Outbound HTTP request settings.
The jitter mode to use for retry backoff behavior.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
Both IP addresses and hostnames/fully qualified domain names (FQDNs) are accepted formats.
The address must include a port.
Configuration for the vector sink.
Both IP address and hostname are accepted formats.
The address must include a port.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Event batching behavior.
{
"max_bytes": null,
"max_events": null,
"timeout_secs": null
}3 nested properties
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
The maximum size of a batch before it is flushed.
The maximum age of a batch before it is flushed.
If set to true, requests are compressed with gzip.
Various settings can be configured, such as concurrency and rate limits, timeouts, retry behavior, etc.
Note that the retry backoff policy follows the Fibonacci sequence.
9 nested properties
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
5 nested properties
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
The adaptive request concurrency limit will not go above this bound. This is put in place as a safeguard.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
Configuration for outbound request concurrency.
This can be set either to one of the below enum values or to a positive integer, which denotes a fixed concurrency limit.
The time window used for the rate_limit_num option.
The maximum number of requests allowed within the rate_limit_duration_secs time window.
The maximum number of retries to make for failed requests.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
The jitter mode to use for retry backoff behavior.
The maximum amount of time to wait between retries.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
Version of the configuration.
Configuration for the webhdfs sink.
Configuration for the websocket sink.
Configures how events are encoded into raw bytes.
This should include the protocol and host, but can also include the port, path, and any other valid part of a URI.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
If this option is not configured, pings are not sent on an interval.
If the ping_timeout is not set, pings are still sent but there is no expectation of pong
response times.
If a response is not received within this time, the connection is re-established.
Configurable sources in Vector.
Supports AMQP version 0.9.1
Configuration for the apache_metrics source.
The list of mod_status endpoints to scrape metrics from.
Disabled if empty.
Configuration for the aws_ecs_metrics source.
If empty, the URI is automatically discovered based on the latest version detected.
By default:
- The version 4 endpoint base URI is stored in the environment variable
ECS_CONTAINER_METADATA_URI_V4. - The version 3 endpoint base URI is stored in the environment variable
ECS_CONTAINER_METADATA_URI. - The version 2 endpoint base URI is
169.254.170.2/v2/.
Disabled if empty.
If empty, the version is automatically discovered based on environment variables.
By default:
- Version 4 is used if the environment variable
ECS_CONTAINER_METADATA_URI_V4is defined. - Version 3 is used if the environment variable
ECS_CONTAINER_METADATA_URI_V4is not defined, but the environment variableECS_CONTAINER_METADATA_URIis defined. - Version 2 is used if neither of the environment variables
ECS_CONTAINER_METADATA_URI_V4orECS_CONTAINER_METADATA_URIare defined.
Configuration for the aws_kinesis_firehose source.
An internet socket address, either IPv4 or IPv6.
If set to true, when incoming requests contains an access key sent by AWS Firehose, it is kept in the
event secrets as "aws_kinesis_firehose_access_key".
AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If
configured, access_key should be set to the same value. Otherwise, all requests are allowed.
AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If
configured, access_keys should be set to the same value. Otherwise, all requests are allowed.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
Configures how events are decoded from raw bytes.
Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
Configuration of HTTP server keepalive parameters.
2 nested properties
A value of 0.1 means that the actual duration will be between 90% and 110% of the specified maximum duration.
Only applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 requests.
A random jitter configured by max_connection_age_jitter_factor is added
to the specified duration to spread out connection storms.
The namespace to use for logs. This overrides the global setting.
Some services, like AWS CloudWatch Logs, compresses the events with gzip, before sending them AWS Kinesis Firehose. This option can be used to automatically decompress them before forwarding them to the next component.
Note that this is different from Content encoding option of the Firehose HTTP endpoint destination. That option controls the content encoding of the entire HTTP request.
Configuration for the aws_s3 source.
Configuration for the aws_sqs source.
Configuration for the datadog_agent source.
An internet socket address, either IPv4 or IPv6.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
Configures how events are decoded from raw bytes.
If this is set to true, logs are not accepted by the component.
If this is set to true, metrics (beta) are not accepted by the component.
If this is set to true, traces (alpha) are not accepted by the component.
Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
Configuration of HTTP server keepalive parameters.
2 nested properties
A value of 0.1 means that the actual duration will be between 90% and 110% of the specified maximum duration.
Only applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 requests.
A random jitter configured by max_connection_age_jitter_factor is added
to the specified duration to spread out connection storms.
The namespace to use for logs. This overrides the global setting.
For a source component named agent, the received logs, metrics (beta), and traces (alpha) can then be
configured as input to other components by specifying agent.logs, agent.metrics, and
agent.traces, respectively.
If this is set to true, when log events contain the field ddtags, the string value that
contains a list of key:value pairs set by the Agent is parsed and expanded into an array.
If this is set to true, when incoming events contain a Datadog API key, it is
stored in the event metadata and used if the event is sent to a Datadog sink.
Configuration for the demo_logs source.
Configuration for the dnstap source.
Configuration for the docker_logs source.
Enables automatic merging of partial events.
Use an HTTPS URL to enable TLS encryption.
If absent, the DOCKER_HOST environment variable is used. If DOCKER_HOST is also absent,
the default Docker local socket (/var/run/docker.sock on Unix platforms,
//./pipe/docker_engine on Windows) is used.
Matching is prefix first, so specifying a value of foo would match any container named foo as well as any
container whose name started with foo. This applies equally whether matching container IDs or names.
By default, the source collects logs for all containers. If exclude_containers is configured, any
container that matches a configured exclusion is excluded even if it is also included with
include_containers, so care should be taken when using prefix matches as they cannot be overridden by a
corresponding entry in include_containers, for example, excluding foo by attempting to include foo-specific-id.
This can be used in conjunction with include_containers.
By default, the global log_schema.host_key option is used.
Matching is prefix first, so specifying a value of foo would match any container named foo as well as any
container whose name started with foo. This applies equally whether matching container IDs or names.
By default, the source collects logs for all containers. If include_containers is configured, only
containers that match a configured inclusion and are also not excluded get matched.
This can be used in conjunction with exclude_containers.
If not provided, all images are included.
Labels should follow the syntax described in the Docker object labels documentation.
The namespace to use for logs. This overrides the global setting.
If not specified, multiline aggregation is disabled.
If auto_partial_merge is disabled, partial events are emitted with a log field, set by this
configuration value, indicating that the event is not complete.
Only relevant when connecting to Docker with an HTTPS URL.
If not configured, the environment variable DOCKER_CERT_PATH is used. If DOCKER_CERT_PATH is absent, then DOCKER_CONFIG is used. If both environment variables are absent, the certificates in ~/.docker/ are read.
Configuration for the eventstoredb_metrics source.
By default, eventstoredb is used.
Endpoint to scrape stats from.
Configuration for the exec source.
The command to run, plus any arguments required.
Mode of operation for running the command.
Whether or not to clear the environment before setting custom environment variables.
Configures how events are decoded from raw bytes.
Custom environment variables to set or update when running the command. If a variable name already exists in the environment, its value is replaced.
Whether or not the output from stderr should be included when generating events.
The namespace to use for logs. This overrides the global setting.
The maximum buffer size allowed before a log event is generated.
Configuration options for scheduled commands.
Configuration options for streaming commands.
The directory in which to run the command.
Configuration for the file source.
Array of file patterns to include. Globbing is supported.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
By default, the global data_dir option is used.
Make sure the running user has write permissions to this directory.
If this directory is specified, then Vector will attempt to create it.
Character set encoding.
Takes precedence over the include option. Note: The exclude patterns are applied after the attempt to glob everything
in include. This means that all files are first matched by include and then filtered by the exclude
patterns. This can be impactful if include contains directories with contents that are not accessible.
[]An optional path that deserializes an empty string to None.
This is important for checkpointing when file rotation is used.
{
"strategy": "checksum",
"bytes": null,
"ignored_header_bytes": 0,
"lines": 1
}A span of time, in whole milliseconds.
By default, the global log_schema.host_key option is used.
Set to "" to suppress this key.
Checkpoints are still written normally.
This may be useful when used with source directories containing dangling symlinks.
Ignore files with a data modification date older than the specified number of seconds.
Configuration of internal metrics for file-based components.
1 nested properties
This is useful for distinguishing between different files while monitoring. However, the tag's cardinality is unbounded.
String sequence used to separate one file line from another.
The namespace to use for logs. This overrides the global setting.
This protects against malformed lines or tailing incorrect files.
This allows distributing the reads more or less evenly across the files.
String value used to identify the start of a multi-line message.
How long to wait for more data when aggregating a multi-line message, in milliseconds.
If not specified, multiline aggregation is disabled.
The value is the byte offset of the start of the line within the file.
Off by default, the offset is only added to the event if this is set.
Instead of balancing read capacity fairly across all watched files, prioritize draining the oldest files before moving on to read data from more recent files.
File position to use when reading a new file.
If not specified, files are not removed.
Whether or not to start reading from the beginning of a new file.
Configuration for the file_descriptor source.
The file descriptor number to read from.
Configures how events are decoded from raw bytes.
By default, the global host_key option is used.
The namespace to use for logs. This overrides the global setting.
Messages larger than this are truncated.
Configuration for the stdin source.
Configures how events are decoded from raw bytes.
By default, the global log_schema.host_key option is used.
The namespace to use for logs. This overrides the global setting.
Messages larger than this are truncated.
Configuration for the fluent source.
If a socket address is used, it must include a port.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
The maximum number of TCP connections that are allowed at any given time.
The namespace to use for logs. This overrides the global setting.
This generally should not need to be changed.
Configuration for the gcp_pubsub source.
Configuration for heroku_logs source.
An internet socket address, either IPv4 or IPv6.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
Configures how events are decoded from raw bytes.
Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
Configuration of HTTP server keepalive parameters.
2 nested properties
A value of 0.1 means that the actual duration will be between 90% and 110% of the specified maximum duration.
Only applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 requests.
A random jitter configured by max_connection_age_jitter_factor is added
to the specified duration to spread out connection storms.
The namespace to use for logs. This overrides the global setting.
Accepts the wildcard (*) character for query parameters matching a specified pattern.
Specifying "*" results in all query parameters included in the log event.
These override any values included in the body with conflicting names.
[]Filtering configuration.
The patterns are matched using globbing.
The patterns are matched using globbing.
Configuration for the host_metrics source.
This collector is only available on Linux systems, and only supports either version 2 or hybrid cgroups.
{
"levels": 100,
"base": null,
"groups": {
"includes": null,
"excludes": null
}
}Defaults to all collectors.
[
"cpu",
"disk",
"filesystem",
"load",
"host",
"memory",
"network",
"process",
"cgroups"
]Options for the disk metrics collector.
{
"devices": {
"includes": null,
"excludes": null
}
}1 nested properties
Filtering configuration.
2 nested properties
The patterns are matched using globbing.
The patterns are matched using globbing.
Options for the filesystem metrics collector.
{
"devices": {
"includes": null,
"excludes": null
},
"filesystems": {
"includes": null,
"excludes": null
},
"mountpoints": {
"includes": null,
"excludes": null
}
}3 nested properties
Filtering configuration.
2 nested properties
The patterns are matched using globbing.
The patterns are matched using globbing.
Filtering configuration.
2 nested properties
The patterns are matched using globbing.
The patterns are matched using globbing.
Filtering configuration.
2 nested properties
The patterns are matched using globbing.
The patterns are matched using globbing.
Overrides the default namespace for the metrics emitted by the source.
Options for the network metrics collector.
{
"devices": {
"includes": null,
"excludes": null
}
}1 nested properties
Filtering configuration.
2 nested properties
The patterns are matched using globbing.
The patterns are matched using globbing.
Options for the process metrics collector.
{
"processes": {
"includes": null,
"excludes": null
}
}1 nested properties
Filtering configuration.
2 nested properties
The patterns are matched using globbing.
The patterns are matched using globbing.
Configuration for the http_client source.
The full path must be specified.
HTTP Authentication.
Configures how events are decoded from raw bytes.
Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
One or more values for the same header can be provided.
{}The namespace to use for logs. This overrides the global setting.
HTTP method.
One or more values for the same parameter key can be provided.
The parameters provided in this option are appended to any parameters
manually provided in the endpoint option.
{}A span of time, in fractional seconds.
TLS configuration.
Configuration for the http_server source.
An internet socket address, either IPv4 or IPv6.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
Configures how events are decoded from raw bytes.
For json and ndjson encodings, the fields of the JSON objects are output as separate fields.
Accepts the wildcard (*) character for headers matching a specified pattern.
Specifying "*" results in all headers included in the log event.
These headers are not included in the JSON payload if a field with a conflicting name exists.
[]An optional path that deserializes an empty string to None.
Configuration of HTTP server keepalive parameters.
2 nested properties
A value of 0.1 means that the actual duration will be between 90% and 110% of the specified maximum duration.
Only applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 requests.
A random jitter configured by max_connection_age_jitter_factor is added
to the specified duration to spread out connection storms.
The namespace to use for logs. This overrides the global setting.
HTTP method.
The URL path on which log event POST requests are sent.
An optional path that deserializes an empty string to None.
Accepts the wildcard (*) character for query parameters matching a specified pattern.
Specifying "*" results in all query parameters included in the log event.
These override any values included in the body with conflicting names.
[]Specifies the HTTP response status code that will be returned on successful requests.
If set to true, only requests using the exact URL path specified in path are accepted. Otherwise,
requests sent to a URL path that starts with the value of path are accepted.
With strict_path set to false and path set to "", the configured HTTP source accepts requests from
any URL path.
Configuration for the http_server source.
An internet socket address, either IPv4 or IPv6.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
Configures how events are decoded from raw bytes.
For json and ndjson encodings, the fields of the JSON objects are output as separate fields.
Accepts the wildcard (*) character for headers matching a specified pattern.
Specifying "*" results in all headers included in the log event.
These headers are not included in the JSON payload if a field with a conflicting name exists.
[]An optional path that deserializes an empty string to None.
Configuration of HTTP server keepalive parameters.
2 nested properties
A value of 0.1 means that the actual duration will be between 90% and 110% of the specified maximum duration.
Only applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 requests.
A random jitter configured by max_connection_age_jitter_factor is added
to the specified duration to spread out connection storms.
The namespace to use for logs. This overrides the global setting.
HTTP method.
The URL path on which log event POST requests are sent.
An optional path that deserializes an empty string to None.
Accepts the wildcard (*) character for query parameters matching a specified pattern.
Specifying "*" results in all query parameters included in the log event.
These override any values included in the body with conflicting names.
[]Specifies the HTTP response status code that will be returned on successful requests.
If set to true, only requests using the exact URL path specified in path are accepted. Otherwise,
requests sent to a URL path that starts with the value of path are accepted.
With strict_path set to false and path set to "", the configured HTTP source accepts requests from
any URL path.
Configuration for the internal_logs source.
By default, the global log_schema.host_key option is used.
Set to "" to suppress this key.
The namespace to use for logs. This overrides the global setting.
An optional path that deserializes an empty string to None.
Configuration for the internal_metrics source.
Overrides the default namespace for the metrics emitted by the source.
A span of time, in fractional seconds.
Tag configuration for the internal_metrics source.
{
"host_key": null,
"pid_key": null
}2 nested properties
The value is the peer host's address, including the port. For example, 1.2.3.4:9000.
By default, the global log_schema.host_key option is used.
Set to "" to suppress this key.
By default, this is not set and the tag is not automatically added.
Configuration for the journald source.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
This option limits the size of the batch.
Only include entries that occurred after the current boot of the system.
By default, the global data_dir option is used.
Make sure the running user has write permissions to this directory.
If this directory is specified, then Vector will attempt to create it.
If exclude_units is specified, it is merged into this list.
{}Unit names lacking a . have .service appended to make them a valid service unit
name.
[]If specified, it is merged to the command line arguments as-is.
[]If empty or not present, all journal fields are accepted.
If include_units is specified, it is merged into this list.
{}If empty or not present, all units are accepted.
Unit names lacking a . have .service appended to make them a valid service unit name.
[]If not set, journalctl uses the default system journal path.
This value is passed to journalctl through the --namespace option.
If not set, journalctl uses the default namespace.
If not set, a search is done for the journalctl path.
The namespace to use for logs. This overrides the global setting.
Has no effect unless the value of the field is already an integer.
Only include entries that appended to the journal after the entries have been read.
Configuration for the kafka source.
Configuration for the kubernetes_logs source.
Partial events are messages that were split by the Kubernetes Container Runtime log driver.
By default, the global data_dir option is used.
Make sure the running user has write permissions to this directory.
If this directory is specified, then Vector will attempt to create it.
A list of glob patterns to exclude from reading the files.
The built-in Node filter uses self_node_name to only watch Pods located on the same Node.
If your files share a common header that is not always a fixed size,
If the file has less than this amount of lines, it won’t be read at all.
A span of time, in whole milliseconds.
Ignore files with a data modification date older than the specified number of seconds.
A list of glob patterns to include while reading the files.
This is useful to compute the latency between important event processing
stages. For example, the time delta between when a log line was written and when it was
processed by the kubernetes_logs source.
Configuration of internal metrics for file-based components.
1 nested properties
This is useful for distinguishing between different files while monitoring. However, the tag's cardinality is unbounded.
If not set, a connection to Kubernetes is made using the in-cluster configuration.
The namespace to use for logs. This overrides the global setting.
This protects against malformed lines or tailing incorrect files.
This allows distributing the reads more or less evenly across the files.
Configuration for how the events are enriched with Namespace metadata.
{
"namespace_labels": ".kubernetes.namespace_labels"
}1 nested properties
An optional path that deserializes an empty string to None.
Configuration for how the events are enriched with Node metadata.
{
"node_labels": ".kubernetes.node_labels"
}1 nested properties
An optional path that deserializes an empty string to None.
Instead of balancing read capacity fairly across all watched files, prioritize draining the oldest files before moving on to read data from more recent files.
Configuration for how the events are enriched with Pod metadata.
{
"pod_name": ".kubernetes.pod_name",
"pod_namespace": ".kubernetes.pod_namespace",
"pod_uid": ".kubernetes.pod_uid",
"pod_ip": ".kubernetes.pod_ip",
"pod_ips": ".kubernetes.pod_ips",
"pod_labels": ".kubernetes.pod_labels",
"pod_annotations": ".kubernetes.pod_annotations",
"pod_node_name": ".kubernetes.pod_node_name",
"pod_owner": ".kubernetes.pod_owner",
"container_name": ".kubernetes.container_name",
"container_id": ".kubernetes.container_id",
"container_image": ".kubernetes.container_image",
"container_image_id": ".kubernetes.container_image_id"
}13 nested properties
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.
File position to use when reading a new file.
Configured to use an environment variable by default, to be evaluated to a value provided by Kubernetes at Pod creation.
The default time zone for timestamps without an explicit zone.
Determines if requests to the kube-apiserver can be served by a cache.
Configuration for the logstash source.
If a socket address is used, it must include a port.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
The maximum number of TCP connections that are allowed at any given time.
The namespace to use for logs. This overrides the global setting.
The size of the receive buffer used for each connection.
Configuration for the mongodb_metrics source.
Each endpoint must be in the Connection String URI Format.
If set to an empty string, no namespace is added to the metrics.
By default, mongodb is used.
Configuration for the nats source.
The URL takes the form of nats://server:port.
If the port is not specified it defaults to 4222.
Configures how events are decoded from raw bytes.
Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
The namespace to use for logs. This overrides the global setting.
The NATS queue group to join.
An optional path that deserializes an empty string to None.
This value determines how many messages the NATS subscriber buffers before incoming messages are dropped.
See the async_nats documentation for more information.
Configuration for the nginx_metrics source.
Each endpoint must be a valid HTTP/HTTPS URI pointing to an NGINX instance that has the
ngx_http_stub_status_module module enabled.
If set to an empty string, no namespace is added to the metrics.
By default, nginx is used.
Configuration for the opentelemetry source.
Configuration for the opentelemetry gRPC server.
2 nested properties
An internet socket address, either IPv4 or IPv6.
Configuration for the opentelemetry HTTP server.
4 nested properties
An internet socket address, either IPv4 or IPv6.
Accepts the wildcard (*) character for headers matching a specified pattern.
Specifying "*" results in all headers included in the log event.
These headers are not included in the JSON payload if a field with a conflicting name exists.
[]Configuration of HTTP server keepalive parameters.
2 nested properties
A value of 0.1 means that the actual duration will be between 90% and 110% of the specified maximum duration.
Only applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 requests.
A random jitter configured by max_connection_age_jitter_factor is added
to the specified duration to spread out connection storms.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
The namespace to use for logs. This overrides the global setting.
Configuration for the postgresql_metrics source.
Each endpoint must be in the Connection URI format.
Specifying "" includes metrics where datname is NULL.
This can be used in conjunction with include_databases.
If not set, metrics are collected from all databases. Specifying "" includes metrics where datname is
NULL.
This can be used in conjunction with exclude_databases.
Overrides the default namespace for the metrics emitted by the source.
Configuration of TLS when connecting to PostgreSQL.
Configuration for the prometheus_pushgateway source.
An internet socket address, either IPv4 or IPv6.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
Only applies to counters and histograms as gauges and summaries can't be meaningfully aggregated.
Configuration of HTTP server keepalive parameters.
2 nested properties
A value of 0.1 means that the actual duration will be between 90% and 110% of the specified maximum duration.
Only applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 requests.
A random jitter configured by max_connection_age_jitter_factor is added
to the specified duration to spread out connection storms.
Configuration for the prometheus_remote_write source.
An internet socket address, either IPv4 or IPv6.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
Configuration of HTTP server keepalive parameters.
2 nested properties
A value of 0.1 means that the actual duration will be between 90% and 110% of the specified maximum duration.
Only applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 requests.
A random jitter configured by max_connection_age_jitter_factor is added
to the specified duration to spread out connection storms.
Configuration for the prometheus_scrape source.
Endpoints to scrape metrics from.
The tag value is the endpoint of the scraped instance.
If true, the new tag is not added if the scraped metric has the tag already. If false, the conflicting tag
is renamed by prepending exported_ to the original name.
This matches Prometheus’ honor_labels configuration.
The tag value is the host and port of the scraped instance.
One or more values for the same parameter key can be provided. The parameters provided in this option are
appended to any parameters manually provided in the endpoints option. This option is especially useful when
scraping the /federate endpoint.
{}A span of time, in fractional seconds.
Configuration for the pulsar source.
The endpoint to which the Pulsar client should connect to.
The Pulsar topic names to read events from.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
Authentication configuration.
Max count of messages in a batch.
The Pulsar consumer name.
Dead Letter Queue policy configuration.
Configures how events are decoded from raw bytes.
Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
The namespace to use for logs. This overrides the global setting.
The broker follows descending priorities. For example, 0=max-priority, 1, 2,...
In Shared subscription type, the broker first dispatches messages to the max priority level consumers if they have permits. Otherwise, the broker considers next priority level consumers.
The Pulsar subscription name.
Configuration for the redis source.
The Redis key to read messages from.
The URL must take the form of protocol://server:port/db where the protocol can either be redis or rediss for connections secured using TLS.
The Redis data type (list or channel) to use.
Configures how events are decoded from raw bytes.
Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.
Options for the Redis list data type.
The namespace to use for logs. This overrides the global setting.
The value is the Redis key that the event was read from.
By default, this is not set and the field is not automatically added.
Configuration for the socket source.
Configuration for the splunk_hec source.
Acknowledgement configuration for the splunk_hec source.
{
"enabled": null,
"max_pending_acks": 10000000,
"max_number_of_ack_channels": 1000000,
"max_pending_acks_per_channel": 1000000,
"ack_idle_cleanup": false,
"max_idle_time": 300
}6 nested properties
A channel is idling if it is not used for sending data or querying acknowledgement statuses.
Enables end-to-end acknowledgements.
Channels can potentially idle for longer than this setting but clients should not rely on such behavior.
Minimum of 1.
Minimum of 1.
Equivalent to the max_number_of_acked_requests_pending_query Splunk HEC setting.
Minimum of 1.
Equivalent to the max_number_of_acked_requests_pending_query_per_ack_channel Splunk HEC setting.
Minimum of 1.
An internet socket address, either IPv4 or IPv6.
Configuration of HTTP server keepalive parameters.
2 nested properties
A value of 0.1 means that the actual duration will be between 90% and 110% of the specified maximum duration.
Only applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 requests.
A random jitter configured by max_connection_age_jitter_factor is added
to the specified duration to spread out connection storms.
The namespace to use for logs. This overrides the global settings.
If set to true, when incoming requests contain a Splunk HEC token, the token used is kept in the
event metadata and preferentially used if the event is sent to a Splunk HEC sink.
If supplied, incoming requests must supply this token in the Authorization header, just as a client would if
it was communicating with the Splunk HEC endpoint directly.
If not supplied, the Authorization header is ignored and requests are not authenticated.
If supplied, incoming requests must supply one of these tokens in the Authorization header, just as a client
would if it was communicating with the Splunk HEC endpoint directly.
If not supplied, the Authorization header is ignored and requests are not authenticated.
Configuration for the static_metrics source.
A span of time, in fractional seconds.
Tag configuration for the internal_metrics source.
[]Overrides the default namespace for the metrics emitted by the source.
Configuration for the statsd source.
Configuration for the syslog source.
HTTP method.
If a socket address is used, it must include a port.
Configuration for the vector source.
An internet socket address, either IPv4 or IPv6.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
1 nested properties
Whether or not end-to-end acknowledgements are enabled for this source.
The namespace to use for logs. This overrides the global setting.
Version of the configuration.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
Configurable transforms in Vector.
Configuration for the aggregate transform.
During this time frame, metrics (beta) with the same series data (name, namespace, tags, and so on) are aggregated.
Some of the functions may only function on incremental and some only on absolute metrics.
Configuration for the aws_ec2_metadata transform.
Overrides the default EC2 metadata endpoint.
A list of metadata fields to include in each transformed event.
[
"ami-id",
"availability-zone",
"instance-id",
"instance-type",
"local-hostname",
"local-ipv4",
"public-hostname",
"public-ipv4",
"region",
"subnet-id",
"vpc-id",
"role-name"
]Sets a prefix for all event fields added by the transform.
Configure to proxy traffic through an HTTP(S) proxy when making external requests.
Similar to common proxy configuration convention, you can set different proxies to use based on the type of traffic being proxied. You can also set specific hosts that should not be proxied.
4 nested properties
Enables proxying support.
Must be a valid URI string.
Must be a valid URI string.
Multiple patterns are allowed:
| Pattern | Example match |
|---|---|
| Domain names | example.com matches requests to example.com |
| Wildcard domains | .example.com matches requests to example.com and its subdomains |
| IP addresses | 127.0.0.1 matches requests to 127.0.0.1 |
| CIDR blocks | 192.168.0.0/16 matches requests to any IP addresses in this range |
| Splat | * matches all hosts |
[]Requires the transform to be able to successfully query the EC2 metadata before starting to process the data.
A list of instance tags to include in each transformed event.
[]Configuration for the dedupe transform.
Caching configuration for deduplication.
{
"num_events": 5000
}1 nested properties
Number of events to cache and use for comparing incoming events to previously seen events.
When no field matching configuration is specified, events are matched using the timestamp,
host, and message fields from an event. The specific field names used are those set in
the global log schema configuration.
Configuration for the route transform.
An array of named routes. The route names are expected to be unique.
Configuration for the filter transform.
Many methods exist for matching events, such as using a VRL expression, a Datadog Search query string, or hard-coded matchers like "must be a metric" or "fields A, B, and C must match these constraints".
As VRL is the most common way to apply conditions to events, this type provides a shortcut to define VRL expressions directly in the configuration by passing the VRL expression as a string:
condition = '.message == "hooray"'
When other condition types are required, they can be specified with an enum-style notation:
condition.type = 'datadog_search'
condition.source = 'NOT "foo"'
Configuration for the log_to_metric transform.
A list of metrics to generate.
This is an example JSON representation of a counter with the following properties:
counter: An object with a single propertyvaluerepresenting the counter value, in this case,10.0).kind: A string indicating the kind of counter, in this case, "incremental".name: A string representing the name of the counter, here set to "test.transform.counter".tags: An object containing additional tags such as "env" and "host".
Objects that can be processed include counter, histogram, gauge, set and summary.
Configuration for the lua transform.
Configuration for the metric_to_log transform.
If present, the value of the tag is set on the generated log event in the host field,
where the field key uses the global host_key option.
The namespace to use for logs. This overrides the global setting.
The user configuration to choose the metric tag strategy.
This overrides the global timezone option. The time zone name may be
any name in the TZ database or local to indicate system local time.
Configuration for the reduce transform.
If supplied, every time this interval elapses for a given grouping, the reduced value for that grouping is flushed. Checked every flush_period_ms.
If this condition resolves to true for an event, the current transaction is immediately
flushed with this event.
Each group with matching values for the specified keys is reduced independently, allowing you to keep independent event streams separate. When no fields are specified, all events are combined in a single group.
For example, if group_by = ["host", "region"], then all incoming events that have the same
host and region are grouped together before being reduced.
[]The maximum number of events to group together.
For each field specified, the given strategy is used for combining events rather than the default behavior.
The default behavior is as follows:
- The first value of a string field is kept and subsequent values are discarded.
- For timestamp fields the first is kept and a new field
[field-name]_endis added with the last received timestamp value. - Numeric values are summed.
- For nested paths, the field value is retrieved and then reduced using the default strategies mentioned above (unless explicitly specified otherwise).
{}If this condition resolves to true for an event, the previous transaction is flushed
(without this event) and a new transaction is started.
Configuration for the remap transform.
If a VRL program is manually aborted (using abort) when
processing an event, this option controls whether the original, unmodified event is sent
downstream without any modifications or if it is dropped.
Additionally, dropped events can potentially be diverted to a specially-named output for
further logging and analysis by setting reroute_dropped.
Normally, if a VRL program encounters an error when processing an event, the original,
unmodified event is sent downstream. In some cases, you may not want to send the event
any further, such as if certain transformation or enrichment is strictly required. Setting
drop_on_error to true allows you to ensure these events do not get processed any
further.
Additionally, dropped events can potentially be diverted to a specially named output for
further logging and analysis by setting reroute_dropped.
If a relative path is provided, its root is the current working directory.
Required if source is missing.
If a relative path is provided, its root is the current working directory.
Required if source or file are missing.
The user configuration to choose the metric tag strategy.
When using drop_on_error or drop_on_abort, events that are "dropped" are processed no
further. In some cases, it may be desirable to keep the events around for further analysis,
debugging, or retrying.
In these cases, reroute_dropped can be set to true which forwards the original event
to a specially-named output, dropped. The original event is annotated with additional
fields describing why the event was dropped.
The runtime to use for executing VRL code.
Required if file is missing.
This overrides the global timezone option. The time zone name may be
any name in the TZ database, or local to indicate system local time.
Configuration for the route transform.
Normally, if an event doesn't match any defined route, it is sent to the <transform_name>._unmatched
output for further processing. In some cases, you may want to simply discard unmatched events and not
process them any further.
In these cases, reroute_unmatched can be set to false to disable the <transform_name>._unmatched
output and instead silently discard any unmatched events.
The following identifiers are reserved output names and thus cannot be used as route IDs:
_unmatched_default
Each route can then be referenced as an input by other components with the name
<transform_name>.<route_id>. If an event doesn’t match any route, and if reroute_unmatched
is set to true (the default), it is sent to the <transform_name>._unmatched output.
Otherwise, the unmatched event is instead silently discarded.
Configuration for the sample transform.
For example, rate = 1500 means 1 out of every 1500 events are forwarded and the rest are
dropped.
A logical condition used to exclude events from sampling.
If left unspecified, or if the event doesn't have group_by, then the event is not
sampled separately.
Each unique value for the key creates a bucket of related events to be sampled together
and the rate is applied to the buckets themselves to sample 1/N buckets. The overall rate
of sampling may differ from the configured one if values in the field are not uniformly
distributed. If left unspecified, or if the event doesn’t have key_field, then the
event is sampled independently.
This can be useful to, for example, ensure that all logs for a given transaction are
sampled together, but that overall 1/N transactions are sampled.
An optional path that deserializes an empty string to None.
Configuration for the tag_cardinality_limit transform.
Configuration for the throttle transform.
Each unique key has its own threshold.
A span of time, in fractional seconds.
A logical condition used to exclude events from sampling.
Configuration of internal metrics for the Throttle transform.
{
"emit_events_discarded_per_key": false
}1 nested properties
If true, the counter will be incremented for each discarded event, including the key value
associated with the discarded event. If false, the counter will not be emitted. Instead, the
number of discarded events can be seen through the component_discarded_events_total internal
metric.
Note that this defaults to false because the key tag has potentially unbounded cardinality.
Only set this to true if you know that the number of unique keys is bounded.
If left unspecified, or if the event doesn't have key_field, then the event is not rate
limited separately.
Wildcards (*) are supported.
See configuration for more info.
Wrapper for sensitive strings containing credentials
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by all connected sinks before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
Configure to proxy traffic through an HTTP(S) proxy when making external requests.
Similar to common proxy configuration convention, you can set different proxies to use based on the type of traffic being proxied. You can also set specific hosts that should not be proxied.
Enables proxying support.
Must be a valid URI string.
Must be a valid URI string.
Multiple patterns are allowed:
| Pattern | Example match |
|---|---|
| Domain names | example.com matches requests to example.com |
| Wildcard domains | .example.com matches requests to example.com and its subdomains |
| IP addresses | 127.0.0.1 matches requests to 127.0.0.1 |
| CIDR blocks | 192.168.0.0/16 matches requests to any IP addresses in this range |
| Splat | * matches all hosts |
[]Metrics can be either absolute or incremental. Absolute metrics represent a sort of "last write wins" scenario, where the latest absolute value seen is meant to be the actual metric value. In contrast, and perhaps intuitively, incremental metrics are meant to be additive, such that we don't know what total value of the metric is, but we know that we'll be adding or subtracting the given value from it.
Generally speaking, most metrics storage systems deal with incremental updates. A notable exception is Prometheus, which deals with, and expects, absolute values from clients.
Metric value. Container for the actual value of a metric.
TLS configuration.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
This has no effect unless key_file is set.
Only relevant for outgoing connections.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Configures the TLS options for incoming/outgoing connections.
A wrapper around OwnedTargetPath that allows it to be used in Vector config
with prefix default to PathPrefix::Event
A wrapper around OwnedValuePath that allows it to be used in Vector config.
This requires a valid path to be used. If you want to allow optional paths,
use [optional_path::OptionalValuePath].
An optional path that deserializes an empty string to None.
An optional path that deserializes an empty string to None.