Module zillion.configs¶

Base schema for an adhoc field

The schema of an adhoc metric

Attributes:

• aggregation - (str, optional) A string representing the aggregation type to apply to this metric. See zillion.core.AggregationTypes.
• technical - (str or dict, optional) A string or dict that will be parsed as a TechnicalField to define a technical computation to be applied to the metric.
• rounding - (int, optional) If specified, the number of decimal places to round to
• weighting_metric - (str, optional) A reference to a metric to use for weighting when aggregating averages
• required_grain - (list of str, optional) If specified, a list of dimensions that must be present in the dimension grain of any report that aims to include this metric.

Base Schema with custom JSON module

Attributes:

• meta - (*dict, optional) A dict of additional custom attributes for the config object

Compute a rolling average and bollinger bands for a column. This adds additional columns to the input dataframe.

The schema of a column configuration

A marshmallow field for the column's field attribute

The schema of a column's field attribute

Attributes:

• name - (str) The name of the field
• ds_formula - (str) A formula used to calculate the field value at the datasource query level. It must use syntax specific to the datasource.

ZillionInfo for a column in a table. See ColumnInfoSchema for more details about fields.

The schema of column info that ends up in the zillion column metadata

Attributes:

• fields - (list of ColumnFieldConfigField, optional) A list of field names or definitions
• allow_type_conversions - (bool, optional) A flag denoting whether additional fields may be inferred from this column based on its column type (such as deriving year from a date).
• type_conversion_prefix - (str, optional) A prefix to apply to all fields defined through automated type conversions.
• active - (bool, optional) A flag denoting whether this column is active.
• required_grain - (list of str, optional) If specified, a list of dimensions that must be present in the dimension grain of any report that aims to include this column.

Mixin to allow validation against a marshmallow schema

The schema of a datasource configuration represented as a marshmallow Field

The schema of a datasource configuration

Attributes:

• connect - (str or dict) A connection string or dict for establishing the datasource connection. This may have placeholders that get filled in from the DATASOURCE_CONTEXTS of the zillion config. See DataSourceConnectField for more details on passing a dict.
• skip_conversion_fields - (bool, optional) Don't add any conversion fields when applying a config
• prefix_with - (str, optional) prefix all queries against this DataSource using SQLAlchemy's prefix_with function. The table-level prefix_with setting overrides this setting.
• metrics - (marshmallow field, optional) A list of MetricConfigSchema
• dimensions - (marshmallow field, optional) A list of DimensionConfigSchema
• tables - (marshmallow field, optional) A dict mapping of TableNameField -> TableConfigSchema

The schema of a datasource connect field

The schema of a technical configuration

A field for defining column-level criteria conversions. This allows for optimizing queries by converting values instead of applying a function on the column to evaluate criteria, which can otherwise prevent index usage.

A Technical that computes a periodic diff on a DataFrame

The schema of a dimension configuration

Common attributes and logic for dimension configs

Attributes:

• values - (str or list, optional) A list of allowed dimension values or a name of a callable to provide a list of values. If a string representing a callable is passed, it must be importable and the callable must accept two arguments: (warehouse ID, dimension object). An example callable would be zillion.field.values_from_db which reads allowed dimension values from the dimension_values table in the Zillion database.
• sorter - (str, optional) A reference to an importable callable that accepts three arguments: (warehouse ID, dimension object, values). Currently values is a pandas Series and the callable is expected to return a Series. See zillion.field.sort_by_value_order for an example.

A field for defining dimension values

The schema of a metric divisors field

The schema of metric divisor settings

Attributes:

• metrics - (list of str) A list of metric names to use as divisors
• rounding - (int, optional) If specified, the number of decimal places to round each new metric to.
• name - (str, optional) A template to use for the name where {divisor} can be substituted for the divisor metric name. Defaults to "{metric}per{divisor}". It will naively attempt to singularize the divisor name by stripping a trailing 's'.
• formula - (str, optional) A template to use for the formula where {divisor} can be substituted for the divisor metric name.

The base schema of a field configuration

Attributes:

• name - (str) The name of the field
• type - (str) A string representing the data type of the field. This will be converted to a SQLAlchemy type via ast.literal_eval.
• display_name - (str, optional) The display name of the field
• description - (str, optional) The description of the field

The schema of a warehouse's NLP settings in the meta dict

Attributes:

• enabled - (bool, optional) A flag denoting whether this field should be included in NLP embeddings.
• embedding_text - (str or list, optional) A string or list of texts to use when creating embeddings for this field. If not specified, the field's name will be used.

The schema of a formula dimension configuration

The base schema of a formula field configuration

Attributes:

• name - (str) The name of the field
• formula - (str, optional) A formula used to compute the field value. Formula fields are applied at the combined query layer, rather than in datasources queries, so the syntax must match that of the combined query layer database.
• display_name - (str, optional) The display name of the field
• description - (str, optional) The description of the field

The schema of a formula metric configuration

The schema of a metric configuration

Attributes:

• ifnull - (float, optional) A numeric value to use in place of NULLs in the Combined Layer query.
• divisors - (dict, optional) Divisor config for this metric. This is used to automatically add fields from this metric by dividing by other fields. See DivisorsConfigField for more info on the format.

Common attributes and logic for metric configs

Attributes:

• aggregation - (str, optional) A string representing the aggregation type to apply to this metric. See zillion.core.AggregationTypes.
• rounding - (int, optional) If specified, the number of decimal places to round to
• weighting_metric - (str, optional) A reference to a metric to use for weighting when aggregating averages
• technical - (str or dict, optional) A string or dict that will be parsed as a TechnicalField to define a technical computation to be applied to the metric.
• required_grain - (list of str, optional) If specified, a list of dimensions that must be present in the dimension grain of any report that aims to include this metric.
• ifnull - (float, optional) A numeric value to use in place of NULLs in the Combined Layer query.

A marshmallow field for the field's NLP embedding text setting

A generic Technical runs a pandas method

A polytype nested field that iterates through a list of possible types

A Technical specific to the pandas rank function

A Technical that uses the pandas rolling feature

The schema of a table configuration

Attributes:

• columns - (dict, optional) A dict mapping of column name to ColumnConfigSchema
• data_url - (str, optional) A url used to download table data if this is an adhoc table
• if_exists - (str, optional) Control whether to replace, fail, or ignore when the table data already exists.
• drop_dupes - (bool, optional) Drop duplicate primary key rows when loading a table from a data_url
• convert_types - (dict, optional) A mapping of column names to types to convert to when loading a table from a data url. The types must be strings representing valid sqlalchemy types. Ex: {"col1": "date", "col2": "integer"}
• primary_key - (list of str, optional) A list of fields representing the primary key of the table
• adhoc_table_options - (dict, optional) A dict of additional params to pass to the adhoc table class as kwargs

ZillionInfo for a table. See TableInfoSchema for more details about fields.

The schema of table info that ends up in the zillion table metadata

Attributes:

• type - (str) Specifies the TableType
• active - (bool, optional) A flag denoting whether this table is active or not.
• parent - (str, optional) A reference to the full name of a parent table. This impacts the possible join relationships of this table. It is assumed to be safe to join back to any parent or ancestor table via shared keys (the child table must have the primary key of the parent table).
• siblings - (list, optional) A list of references to the full names of sibling tables. This impacts the possible join relationships of this table. It is assumed to be safe to join back to any sibling table via shared keys (the table must have the primary key of the sibling table).
• create_fields - (bool, optional) If true, try to create Field objects from all columns in the table. Specifying the fields in a column config will override this behavior. Metric vs Dimension fields are inferred from the type. It is generally better to be explicit about your fields and field types, but this option provides convenience for special cases, particularly adhoc use cases.
• use_full_column_names - (bool, optional) If True and create_fields is True, fully qualify the created field names using the full table and column names. If false, assume it is safe to simply use the column name as the field name.
• primary_key - (list of str) A list of fields representing the primary key of the table
• incomplete_dimensions - (list of str, optional) If specified, a list of dimensions that are not safe to use for joins.
• priority - (int, optional) Set the priority of this table relative to other tables. All tables default to priority=1. When choosing the best table, lower numbers are considered higher priority. Tables at the same priority level use the length of their TableSet for the given query as the tie-breaker. See Warehouse._choose_best_table_set.
• prefix_with - (str, optional) prefix all queries against this Table using SQLAlchemy's prefix_with function. If a query contains multiple tables with prefix_with set, the first in the join takes precedence.

The schema of a table configuration represented as a marshmallow Field

A field for the type of a table

A technical computation on a DataFrame column

Parameters:

• type - (str) The TechnicalType
• params - (dict) Params for the technical computation
• mode - (str) The mode that controls how to apply the technical computation across the data's dimensions. See TechnicalModes for options. If None, the default mode will be set based on the technical type.

Attributes:

• allowed_params - (set) Define the allowed technical parameters

A field for defining technical calculations

The schema of a technical configuration

The schema of a warehouse configuration.

Attributes:

• includes - (marshmallow field, optional) A list of warehouse files to import. Later items in the list will override earlier items for overlapping keys. Any settings in the warehouse config will take precedence.
• metrics - (marshmallow field, optional) A list of MetricConfigSchema
• dimensions - (marshmallow field, optional) A list of DimensionConfigSchema
• datasources - (marshmallow field) A dict mapping of datasource name -> DataSourceConfigField

The schema of a warehouse's NLP settings in the meta dict

Attributes:

• collection_name - (str, optional) The name of the field embedding collection. If not provided the collection name will be set based on the warehouse name or as a fallback name if no warehouse name has been set.
• field_disabled_patterns - (list of str, optional) A list of regex patterns to control which fields to exclude from NLP processing.
• field_disabled_groups - (list of str, optional) A list of group names to control which fields to exclude from NLP processing based on the field's meta["group"] setting if present.

Information that defines a part of the zillion configuration. The information may come from a JSON config file or directly from the SQLALchemy object's info.zillion attribute. The JSON schema is parsed with a marshmallow schema object. See the particular schema used with each subclass for details on fields.

Parameters:

• kwargs - Parameters that will be parsed with the given marshmallow schema.

Attributes:

• schema - (marshmallow schema) A class attribute that specifies the marshmallow schema used to parse the input args on init.

Validate the NLP settings in the meta dict of a field config

Create a technical instance from the input object

Parameters:

• info - (str or dict) If str, parse as atechnical string. If a dict, parse as TechnicalInfoSchema.

Returns:

(Technical) - A Technical object based on the input.

Determine a default display name from the field name

Parameters:

• name - (str) The field name to process

Returns:

(str) - The field display name

Get the default field name from a SQLAlchemy column

Parameters:

• column - (SQLAlchemy column) A column to get the default field name for

Returns:

(str) - The default field name for the column

Replace characters with underscores if they are not in FIELD_NAME_ALLOWED_CHARS

Parameters:

• name - (str) The field name to process

Returns:

(str) - The "safe" field name

Given a metric with a divisor config, generate the formula metrics

Validate a mapping that has sqlalchemy type strings as values

Helper to test if an object is an active part of the zillion config

Validate aggregation type

Validate column field config

Validate technical type

Validate datasource config

Validate datasource connect value

Validate datasource criteria conversions

Validate dimension values

Validate metric divisors

Validate field display name

Validate field name

Validate field nlp embedding text config

Validate if_exists param

Validate SQLAlchemy type string

Validate table name

Validate table type

Validate technical

Validate technical mode

Validate technical type

Parse a datasource JSON config

Parameters:

• cfg - (dict, str, or buffer) A datasource config dict or a file path/buffer to read the config contents from.

Returns:

(dict) - The parsed datasource config

Parse a datasource JSON config from a location stored in an environment variable

Parse a warehouse JSON config

Parameters:

• cfg - (dict, str, or buffer) A warehouse config dict or a file path/buffer to read the config contents from.

Returns:

(dict) - The parsed warehouse config

Parse a warehouse JSON config from a location stored in an environment variable

Parse a marshmallow schema file

Parameters:

• f - (str or buffer) A file path or buffer to read the raw schema contents from. Both JSON and YAML are supported.
• schema - (marshmallow schema) The marshmallow schema to use to parse the data

Returns:

(dict) - A dict structure loaded from the schema file

Parse Technical args from a shorthand string

Parameters:

• val - (str) The technical string to parse. The general format is: type(*args):mode. The type must be a valid value in TechnicalTypes. The argument requirements vary by type, and are optional in some cases. The mode controls whether the computation is done across the last group or the full data. The mode is optional, and will default to a value specific to that technical type (usually "group" mode). Examples:

  • "mean(5)" for moving average, window=5
  • "mean(5,2)" for moving average, window=5, min_period=2
  • "cumsum" for cumulative sum (no args)
  • "cumsum:all" for cumulative sum across all data, regardless of dimension

Returns:

(dict) - A dict of Technical args

Replace characters with underscores if they are not in TABLE_NAME_ALLOWED_CHARS

Parameters:

• name - (str) The field name to process

Returns:

(str) - The "safe" table name