Pipeline API experimental feature overview
The pipeline API, introduced in Pydantic v2.8.0, is experimental and allows composing of parsing, validation, constraints, and transformations in a type-safe manner. It is subject to change or removal. Each step in a pipeline can be a validation step (runs pydantic validation on a provided type), a transformation step (modifies the data), a constraint step (checks data against a condition), or a predicate step (checks data against a condition and raises an error if it returns False). Steps can be combined using the | operator (logical OR) or & operator (logical AND).
validate_as() pipeline method examples
The validate_as() method is the core API for the pipeline approach. Example usage: validate_as(str).str_lower() lowercases a string; validate_as(int).gt(0) constrains an integer to be greater than zero; validate_as(str).str_pattern(r'[a-z]+') constrains a string to match a regex pattern. The method .transform() applies custom transformations like .transform(str.lower). The method .predicate() checks a condition and raises an error if it returns False, e.g., .predicate(lambda x: x != 'password'). Calling validate_as(...) with Ellipsis implies validate_as(<field type>). Use validate_as(Any) to accept any type. validate_as() can be called before or after other steps for pre or post processing.
Pipeline API example with multiple transformations
Example showing pipeline API usage in a BaseModel with Annotated fields:
```python
from __future__ import annotations
from datetime import datetime
from typing import Annotated
from pydantic import BaseModel
from pydantic.experimental.pipeline import validate_as
class User(BaseModel):
name: Annotated[str, validate_as(str).str_lower()]
age: Annotated[int, validate_as(int).gt(0)]
username: Annotated[str, validate_as(str).str_pattern(r'[a-z]+')]
password: Annotated[
str,
validate_as(str)
.transform(str.lower)
.predicate(lambda x: x != 'password'),
]
favorite_number: Annotated[
int,
(validate_as(int) | validate_as(str).str_strip().validate_as(int)).gt(
0
),
]
friends: Annotated[list[User], validate_as(...).len(0, 100)]
bio: Annotated[
datetime,
validate_as(int)
.transform(lambda x: x / 1_000_000)
.validate_as(...),
]
```
Mapping validate_as to BeforeValidator, AfterValidator, WrapValidator
The validate_as method provides a more type-safe way to define the older validator types. BeforeValidator is equivalent to validate_as(str).str_strip().validate_as(...). AfterValidator is equivalent to transform(lambda x: x * 2). WrapValidator combines validation with transformation: validate_as(str).str_strip().validate_as(...).transform(lambda x: x * 2).
Partial Validation experimental feature added in v2.10
Partial validation allows validating incomplete JSON strings or Python objects representing incomplete input data. It is particularly helpful when processing LLM output that streams structured responses. Partial validation is currently experimental and may change in future versions. It can be enabled when using TypeAdapter methods: validate_json(), validate_python(), and validate_strings().
experimental_allow_partial parameter values
The experimental_allow_partial flag can take the following values (default is False): False or 'off' disables partial validation; True or 'on' enables partial validation but doesn't support trailing strings; 'trailing-strings' enables partial validation and supports trailing incomplete strings at the end of partial JSON.
Partial validation example with TypeAdapter
Example demonstrating partial validation:
```python
from typing import Annotated
from annotated_types import MinLen
from typing_extensions import NotRequired, TypedDict
from pydantic import TypeAdapter
class Foobar(TypedDict):
a: int
b: NotRequired[float]
c: NotRequired[Annotated[str, MinLen(5)]]
ta = TypeAdapter(list[Foobar])
v = ta.validate_json('[{"a": 1, "b"', experimental_allow_partial=True)
print(v)
#> [{'a': 1}]
v = ta.validate_json(
'[{"a": 1, "b": 1.0, "c": "abcd', experimental_allow_partial=True
)
print(v)
#> [{'a': 1, 'b': 1.0}]
v = ta.validate_json(
'[{"b": 1.0, "c": "abcde"', experimental_allow_partial=True
)
print(v)
#> []
v = ta.validate_json(
'[{"a": 1, "b": 1.0, "c": "abcde"},{"a": ', experimental_allow_partial=True
)
print(v)
#> [{'a': 1, 'b': 1.0, 'c': 'abcde'}]
v = ta.validate_python([{'a': 1}], experimental_allow_partial=True)
print(v)
#> [{'a': 1}]
v = ta.validate_python(
[{'a': 1, 'b': 1.0, 'c': 'abcd'}], experimental_allow_partial=True
)
print(v)
#> [{'a': 1, 'b': 1.0}]
v = ta.validate_json(
'[{"a": 1, "b": 1.0, "c": "abcdefg',
experimental_allow_partial='trailing-strings',
)
print(v)
#> [{'a': 1, 'b': 1.0, 'c': 'abcdefg'}]
```
How partial validation works: two main behaviors
Partial validation enables two pieces of behavior: (1) Partial JSON parsing: the jiter JSON parser used by Pydantic supports parsing partial JSON; experimental_allow_partial is passed to jiter via the allow_partial argument. (2) Ignore errors in the last element of the input: because only part of the input data is available, errors commonly occur in the last element. Pydantic ignores ALL errors in the last element of the input data during partial validation.
Partial validation limitations: TypeAdapter only
The experimental_allow_partial parameter can only be passed to TypeAdapter methods. It is not yet supported via other Pydantic entry points like BaseModel.
Partial validation supported collection types
Only a subset of collection validators currently support partial validation: list, set, frozenset, dict (as in dict[X, Y]), and TypedDict (with only non-required fields allowed to be missing, e.g., via NotRequired or total=False). Other collection validators will be validated "all or nothing", and partial validation will not work on nested types within them.
Partial validation limitation: some invalid JSON accepted
The jiter JSON parser cannot differentiate between complete JSON like {"a": 1, "b": "12"} and incomplete JSON like {"a": 1, "b": "12". This means some invalid JSON will be accepted by Pydantic when using experimental_allow_partial.
Partial validation limitation: any last field error ignored
During partial validation, ANY error in the last field of the input will be ignored, not just errors that could result from truncation. This means clearly invalid data will pass validation if the error is in the last field.
generate_arguments_schema() experimental function
The experimental generate_arguments_schema() function can be used to construct a core schema for function arguments that can later be used with a SchemaValidator. This allows validating arguments without actually calling the decorated callable, useful when loading from data sources like JSON. Unlike @validate_call, the core schema will only validate the provided arguments; the underlying callable will not be called.
generate_arguments_schema() example
Example using generate_arguments_schema():
```python
from pydantic_core import SchemaValidator
from pydantic.experimental.arguments_schema import generate_arguments_schema
def func(p: bool, *args: str, **kwargs: int) -> None: ...
arguments_schema = generate_arguments_schema(func=func)
val = SchemaValidator(arguments_schema, config={'coerce_numbers_to_str': True})
args, kwargs = val.validate_json(
'{"p": true, "args": ["arg1", 1], "kwargs": {"extra": 1}}'
)
print(args, kwargs)
#> (True, 'arg1', '1') {'extra': 1}
```
generate_arguments_schema() with parameters_callback
generate_arguments_schema() accepts a parameters_callback argument to ignore specific parameters. The callback is called for every parameter with signature (index: int, name: str, annotation: Any) -> Any and can return 'skip' to ignore a parameter.
MISSING sentinel overview
The MISSING sentinel is a singleton indicating a field value was not provided during validation. It can be used as a default value, as an alternative to None when it has an explicit meaning. During serialization, any field with MISSING as a value is excluded from the output. The MISSING value does not appear in the JSON Schema. The `is` operator can be used to discriminate between the sentinel and other values.
MISSING sentinel example
Example using MISSING:
```python
from pydantic import BaseModel
from pydantic.experimental.missing_sentinel import MISSING
class Configuration(BaseModel):
timeout: int | None | MISSING = MISSING
# configuration defaults, stored somewhere else:
defaults = {'timeout': 200}
conf = Configuration()
# `timeout` is excluded from the serialization output:
conf.model_dump()
# {}
# The `MISSING` value doesn't appear in the JSON Schema:
Configuration.model_json_schema()['properties']['timeout']
#> {'anyOf': [{'type': 'integer'}, {'type': 'null'}], 'title': 'Timeout'}}
# `is` can be used to discriminate between the sentinel and other values:
timeout = conf.timeout if conf.timeout is not MISSING else defaults['timeout']
```
MISSING sentinel limitations
The MISSING sentinel feature is marked as experimental because it relies on the draft PEP 661. Current limitations: (1) Static type checking of sentinels is only supported with Pyright 1.1.402 or greater, and the enableExperimentalFeatures type evaluation setting should be enabled. (2) Pickling of models containing MISSING as a value is not supported. When applying constraints to a union containing the MISSING sentinel, constraints are automatically applied to the remaining type(s) of the union.