Bigconfig

Bigconfig is a yaml-based declarative monitoring as code solution. Data engineers can deploy Bigeye metrics from the command-line for convenient and scalable data quality monitoring

Type object
File match bigconfig.yml bigconfig.yaml
Schema URL https://catalog.lintel.tools/schemas/schemastore/bigconfig/latest.json
Source https://www.schemastore.org/bigconfig.json

Validate with Lintel

npx @lintel/lintel check
Type: object

Bigconfig is a yaml-based declarative monitoring as code solution. Data engineers can deploy Bigeye metrics from the command-line for convenient and scalable data quality monitoring.

Properties

type string required
Default: "BIGCONFIG_FILE"
Any of: const: "BIGCONFIG_FILE" const: "BIGCONFIG_FILE"
auto_apply_on_indexing boolean
Default: true
tag_definitions TagDefinition[]
row_creation_times
Default:
{
  "tag_ids": [],
  "column_selectors": []
}
All of: RowCreationTimes object
saved_metric_definitions object
1 nested properties
metrics SimpleMetricDefinition[] required
tag_deployments TagDeploymentSuite[]
table_deployments TableDeploymentSuite[]

Definitions

SimpleFieldType string
ColumnSelector object
name string
Default: "*.*.*.*"
type string
Values: "UNSPECIFIED" "STRING" "UUID" "BINARY" "NUMERIC" "DATE_LIKE" "TIMESTAMP_LIKE" "ANY" "BOOLEAN" "UNKNOWN" "ARRAY" "STRUCT"
exclude string
regex string
TagDefinition object
tag_id string required
column_selectors ColumnSelector[] required
Default:
[
  {
    "name": "*.*.*.*"
  }
]
RowCreationTimes object
tag_ids string[]
column_selectors ColumnSelector[]
NotificationChannels object
slack string
email string
webhook string
webhook_header_key string
webhook_header_value string
MetricParameters object
key string
string_value string
column_name string
number_value number
SimpleMetricDefinition object
saved_metric_id string
metric_type object
4 nested properties
predefined_metric enum required
Values: "PERCENT_NULL" "COUNT_NULL" "PERCENT_EMPTY_STRING" "COUNT_EMPTY_STRING" "PERCENT_UNIQUE" "PERCENT_VALUE_IN_LIST" "AVERAGE" "MIN" "MAX" "SUM" "COUNT_ROWS" "COUNT_DISTINCT" "HOURS_SINCE_MAX_DATE" "HOURS_SINCE_MAX_TIMESTAMP" "COUNT_TRUE" "PERCENT_TRUE" "COUNT_FALSE" "PERCENT_FALSE" "COUNT_USA_PHONE" "PERCENT_USA_PHONE" "COUNT_USA_ZIP_CODE" "PERCENT_USA_ZIP_CODE" "PERCENT_UUID" "COUNT_TIMESTAMP_STRING" "PERCENT_TIMESTAMP_STRING" "COUNT_DUPLICATES" "COUNT_USA_STATE_CODE" "PERCENT_USA_STATE_CODE" "VARIANCE" "SKEW" "KURTOSIS" "GEOMETRIC_MEAN" "HARMONIC_MEAN" "COUNT_UUID" "COUNT_CUSIP" "PERCENT_CUSIP" "COUNT_SEDOL" "PERCENT_SEDOL" "COUNT_ISIN" "PERCENT_ISIN" "COUNT_LEI" "PERCENT_LEI" "COUNT_FIGI" "PERCENT_FIGI" "COUNT_PERM_ID" "PERCENT_PERM_ID" "COUNT_NAN" "PERCENT_NAN" "COUNT_LONGITUDE" "PERCENT_LONGITUDE" "COUNT_LATITUDE" "PERCENT_LATITUDE" "COUNT_NOT_IN_FUTURE" "PERCENT_NOT_IN_FUTURE" "COUNT_DATE_NOT_IN_FUTURE" "PERCENT_DATE_NOT_IN_FUTURE" "MEDIAN" "PERCENTILE" "COUNT_NOT_NULL" "STRING_LENGTH_AVERAGE" "STRING_LENGTH_MIN" "STRING_LENGTH_MAX" "COUNT_SSN" "PERCENT_SSN" "COUNT_EMAIL" "PERCENT_EMAIL" "ROWS_INSERTED" "HOURS_SINCE_LAST_LOAD" "COUNT_READ_QUERIES" "PERCENT_NOT_NULL" "FRESHNESS" "VOLUME"
type string
Values: "PREDEFINED" "TEMPLATE"
template_id number
aggregation_type string
Values: "COUNT" "PERCENT"
metric_name string
description string
schedule_frequency object
Default:
{
  "interval_type": "HOURS",
  "interval_value": 24
}
2 nested properties
interval_type enum required
Default: "HOURS"
Values: "HOURS" "MINUTES"
interval_value number required
Default: 24
conditions string[]
group_by string[]
threshold object
7 nested properties
type enum required
Values: "AUTO" "RELATIVE" "STDDEV" "CONSTANT"
sensitivity enum

Only valid for autothreshold metrics

Values: "NARROW" "MEDIUM" "WIDE" "XWIDE"
upper_bound number

Only valid for constant, relative, and standard deviation metrics

lower_bound number

Only valid for constant, relative, and standard deviation metrics

upper_bound_only boolean

Only valid for autothreshold metrics

lower_bound_only boolean

Only valid for autothreshold metrics

reference_point object

Only valid for relative and standard deviation metrics

2 nested properties
interval_type enum required
Values: "DAYS"
interval_value number required
notification_channels NotificationChannels[]
parameters MetricParameters[]
lookback object
3 nested properties
lookback_type enum required
Values: "DATA_TIME" "METRIC_TIME" "CLOCK_TIME"
lookback_window object required
2 nested properties
interval_type enum required
Values: "DAYS"
interval_value number required
bucket_size enum

Only valid for data time window metrics

Values: "DAY" "HOUR"
metric_schedule object
1 nested properties
named_schedule object required
3 nested properties
name string required
cron string
id number
SavedMetricDefinitions object
metrics SimpleMetricDefinition[] required
TagDeployment object
metrics SimpleMetricDefinition[] required
column_selectors ColumnSelector[]
tag_id string
SimpleCollection object
name string required
description string required
notification_channels object
5 nested properties
slack string
email string
webhook string
webhook_header_key string
webhook_header_value string
TagDeploymentSuite object
collection object
3 nested properties
name string required
description string required
notification_channels object
5 nested properties
slack string
email string
webhook string
webhook_header_key string
webhook_header_value string
deployments TagDeployment[]
ColumnMetricDeployment object
column_name string required
metrics SimpleMetricDefinition[] required
TableDeployment object
fq_table_name string required
columns ColumnMetricDeployment[]
table_metrics SimpleMetricDefinition[]
row_creation_time string
TableDeploymentSuite object
deployments TableDeployment[] required
collection object
12 nested properties
saved_metric_id string
metric_type object
4 nested properties
predefined_metric enum required
Values: "PERCENT_NULL" "COUNT_NULL" "PERCENT_EMPTY_STRING" "COUNT_EMPTY_STRING" "PERCENT_UNIQUE" "PERCENT_VALUE_IN_LIST" "AVERAGE" "MIN" "MAX" "SUM" "COUNT_ROWS" "COUNT_DISTINCT" "HOURS_SINCE_MAX_DATE" "HOURS_SINCE_MAX_TIMESTAMP" "COUNT_TRUE" "PERCENT_TRUE" "COUNT_FALSE" "PERCENT_FALSE" "COUNT_USA_PHONE" "PERCENT_USA_PHONE" "COUNT_USA_ZIP_CODE" "PERCENT_USA_ZIP_CODE" "PERCENT_UUID" "COUNT_TIMESTAMP_STRING" "PERCENT_TIMESTAMP_STRING" "COUNT_DUPLICATES" "COUNT_USA_STATE_CODE" "PERCENT_USA_STATE_CODE" "VARIANCE" "SKEW" "KURTOSIS" "GEOMETRIC_MEAN" "HARMONIC_MEAN" "COUNT_UUID" "COUNT_CUSIP" "PERCENT_CUSIP" "COUNT_SEDOL" "PERCENT_SEDOL" "COUNT_ISIN" "PERCENT_ISIN" "COUNT_LEI" "PERCENT_LEI" "COUNT_FIGI" "PERCENT_FIGI" "COUNT_PERM_ID" "PERCENT_PERM_ID" "COUNT_NAN" "PERCENT_NAN" "COUNT_LONGITUDE" "PERCENT_LONGITUDE" "COUNT_LATITUDE" "PERCENT_LATITUDE" "COUNT_NOT_IN_FUTURE" "PERCENT_NOT_IN_FUTURE" "COUNT_DATE_NOT_IN_FUTURE" "PERCENT_DATE_NOT_IN_FUTURE" "MEDIAN" "PERCENTILE" "COUNT_NOT_NULL" "STRING_LENGTH_AVERAGE" "STRING_LENGTH_MIN" "STRING_LENGTH_MAX" "COUNT_SSN" "PERCENT_SSN" "COUNT_EMAIL" "PERCENT_EMAIL" "ROWS_INSERTED" "HOURS_SINCE_LAST_LOAD" "COUNT_READ_QUERIES" "PERCENT_NOT_NULL" "FRESHNESS" "VOLUME"
type string
Values: "PREDEFINED" "TEMPLATE"
template_id number
aggregation_type string
Values: "COUNT" "PERCENT"
metric_name string
description string
schedule_frequency object
Default:
{
  "interval_type": "HOURS",
  "interval_value": 24
}
2 nested properties
interval_type enum required
Default: "HOURS"
Values: "HOURS" "MINUTES"
interval_value number required
Default: 24
conditions string[]
group_by string[]
threshold object
7 nested properties
type enum required
Values: "AUTO" "RELATIVE" "STDDEV" "CONSTANT"
sensitivity enum

Only valid for autothreshold metrics

Values: "NARROW" "MEDIUM" "WIDE" "XWIDE"
upper_bound number

Only valid for constant, relative, and standard deviation metrics

lower_bound number

Only valid for constant, relative, and standard deviation metrics

upper_bound_only boolean

Only valid for autothreshold metrics

lower_bound_only boolean

Only valid for autothreshold metrics

reference_point object

Only valid for relative and standard deviation metrics

notification_channels NotificationChannels[]
parameters MetricParameters[]
lookback object
3 nested properties
lookback_type enum required
Values: "DATA_TIME" "METRIC_TIME" "CLOCK_TIME"
lookback_window object required
bucket_size enum

Only valid for data time window metrics

Values: "DAY" "HOUR"
metric_schedule object
1 nested properties
named_schedule object required