experimental

35 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Experimental Pipeline API

The pydantic.experimental.pipeline module contains the _Pipeline class in its public API.

Experimental Arguments Schema API

The pydantic.experimental.arguments_schema module contains the generate_arguments_schema function in its public API.

Partial validation limitation - BaseModel doesn't support partial validation

BaseModel doesn't support partial validation, so it won't forward the allow_partial instruction down to nested validators. For example, partial validation won't work on a list validator inside a BaseModel field.

Partial validation limitation - incomplete JSON acceptance

jiter cannot differentiate between complete JSON like '{"a": 1, "b": "12"}' and incomplete JSON like '{"a": 1, "b": "12'. This means some invalid JSON will be accepted by Pydantic when using experimental_allow_partial.

Pipeline API introduced in v2.8.0

Pydantic v2.8.0 introduced an experimental pipeline API that allows composing parsing (validation), constraints and transformations in a more type-safe manner than existing APIs. This API is subject to change or removal.

Pipeline API step types

Each step in the pipeline can be: a validation step that runs pydantic validation on the provided type; a transformation step that modifies the data; a constraint step that checks the data against a condition; or a predicate step that checks the data against a condition and raises an error if it returns False.

validate_as function for pipeline

The validate_as function is a more type-safe way to define BeforeValidator, AfterValidator and WrapValidator. When called with Ellipsis (...) as the first positional argument, validate_as(...) implies validate_as(<field type>). Use validate_as(Any) to accept any type. The function can be called before or after other steps to do pre or post processing.

Pipeline API validate_as example with str_lower

Example showing string lowercase transformation: Annotated[str, validate_as(str).str_lower()]

Pipeline API validate_as example with constraints

Example showing integer constraint: Annotated[int, validate_as(int).gt(0)]

Pipeline API validate_as example with regex pattern

Example showing string pattern constraint: Annotated[str, validate_as(str).str_pattern(r'[a-z]+')]

Pipeline API validate_as with transform and predicate

Example showing chained transform and predicate: Annotated[str, validate_as(str).transform(str.lower).predicate(lambda x: x != 'password')]

Pipeline API combining steps with operators

Pipeline steps can be combined using the | operator (logical OR) or & operator (logical AND) to compose multiple validation paths.

Pipeline API with len constraint

Example showing length constraint on list: Annotated[list[User], validate_as(...).len(0, 100)]

Pipeline API with transform and validate_as chaining

Example showing pre/post processing: Annotated[datetime, validate_as(int).transform(lambda x: x / 1_000_000).validate_as(...)]

Partial validation introduced in v2.10

Partial validation allows you to validate an incomplete JSON string, or a Python object representing incomplete input data. It is particularly helpful when processing the output of an LLM where the model streams structured responses.

Partial validation enabled on TypeAdapter methods

Partial validation can be enabled when using TypeAdapter.validate_json(), TypeAdapter.validate_python(), and TypeAdapter.validate_strings() methods via the experimental_allow_partial flag.

experimental_allow_partial parameter values

The experimental_allow_partial flag can take the following values (default is False): False or 'off' - disable partial validation; True or 'on' - enable partial validation but don't support trailing strings; 'trailing-strings' - enable partial validation and support trailing strings.

Trailing-strings mode in partial validation

The 'trailing-strings' mode allows for trailing incomplete strings at the end of partial JSON to be included in the output. For example, JSON input '{\'a\': \'hello\', \'b\': \'wor' would validate as {'a': 'hello', 'b': 'wor'}.

Partial validation mechanism - partial JSON parsing

The experimental_allow_partial flag is passed to jiter (the JSON parser used by Pydantic) via the allow_partial argument to enable partial JSON parsing.

Partial validation mechanism - ignore errors in last element

When using experimental_allow_partial, Pydantic ignores ALL errors in the last element of the input data, since only having access to part of the input data means errors commonly occur in the last element.

Partial validation limitation - TypeAdapter only

Partial validation via experimental_allow_partial can only be passed to TypeAdapter methods. It is not yet supported via other Pydantic entry points like BaseModel.

Partial validation supported collection types

Right now only a subset of collection validators support partial validation: list, set, frozenset, dict (as in dict[X, Y]), and TypedDict (only non-required fields may be missing via NotRequired or total=False).

Partial validation limitation - all errors in last field ignored

Any error in the last field of the input will be ignored during partial validation. This means clearly invalid data will pass validation if the error is in the last field of the input.

generate_arguments_schema function

The experimental generate_arguments_schema() function can be used to construct a core schema for validating a callable's arguments. This can later be used with a SchemaValidator to validate arguments loaded from other data sources such as JSON data, without actually calling the decorated callable.

generate_arguments_schema example

Example: from pydantic_core import SchemaValidator; from pydantic.experimental.arguments_schema import generate_arguments_schema; def func(p: bool, *args: str, **kwargs: int) -> None: ...; arguments_schema = generate_arguments_schema(func=func); val = SchemaValidator(arguments_schema, config={'coerce_numbers_to_str': True}); args, kwargs = val.validate_json('{"p": true, "args": ["arg1", 1], "kwargs": {"extra": 1}}'); print(args, kwargs) #> (True, 'arg1', '1') {'extra': 1}

generate_arguments_schema with parameters_callback

The generate_arguments_schema function accepts a parameters_callback argument which is called for every parameter. This callback can return 'skip' to ignore specific parameters based on index, name, or annotation.

MISSING sentinel value

The MISSING sentinel is a singleton imported from pydantic.experimental.missing_sentinel that indicates a field value was not provided during validation. It can be used as a default value as an alternative to None when it has an explicit meaning. During serialization, any field with MISSING as a value is excluded from the output.

MISSING sentinel in union types

The MISSING sentinel can be used in union types like int | None | MISSING to provide three distinct states: a value, null, or not provided.

MISSING sentinel doesn't appear in JSON Schema

When a field has MISSING as a default value, the MISSING value doesn't appear in the JSON Schema output. For example, a field with type int | None | MISSING and default MISSING will have schema {'anyOf': [{'type': 'integer'}, {'type': 'null'}], 'title': 'Timeout'}.

MISSING sentinel discrimination with is operator

The is operator can be used to discriminate between the MISSING sentinel and other values, for example: timeout = conf.timeout if conf.timeout is not MISSING else defaults['timeout'].

MISSING sentinel relies on PEP 661

The MISSING sentinel feature is marked as experimental because it relies on the draft PEP 661 which introduces sentinels in the standard library.

MISSING sentinel type checking limitation

Static type checking of sentinels is only supported with Pyright 1.1.402 or greater, and the enableExperimentalFeatures type evaluation setting should be enabled.

MISSING sentinel pickling limitation

Pickling of models containing MISSING as a value is not supported.

MISSING sentinel with field constraints

When applying constraints to a union containing the MISSING sentinel, such constraints are automatically applied to the remaining type(s) of the union.

missing_sentinel_error validation error

This error is raised when the experimental MISSING sentinel is the only value allowed, and wasn't provided during validation.

Give your agent this brain