type coercion

52 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Pydantic conversion table tabs

Pydantic provides a conversion table with multiple tabs for understanding type coercion during validation. The tabs are: All, JSON, JSON - Strict, Python, and Python - Strict. The Strict column in the conversion table indicates which type conversions are allowed when validating in Strict Mode.

Conversion table documentation

The conversion table is a reference that documents how Pydantic converts data during validation. It includes separate views for All conversions, JSON-only conversions, JSON conversions in strict mode, Python conversions, and Python conversions in strict mode.

JSON has no date or tuple types - Pydantic coerces automatically

When parsing JSON directly, Pydantic knows that JSON has no native date or tuple types, so it allows strings and arrays as inputs respectively. Strings are coerced to dates and arrays are coerced to tuples automatically during JSON parsing.

Data conversion and type coercion behavior

Pydantic may cast input data to force conformance to model field types, which can result in information loss. For example, 3.000 coerces to int 3, '2.72' coerces to float 2.72, and b'binary data' coerces to str 'binary data'. This is deliberate Pydantic behavior. Pydantic provides strict mode where no data conversion is performed and values must match declared field types exactly.

Concrete container types preferred over abstract types

Use concrete container types like list[int] instead of abstract types like Sequence. Pydantic converts input to the concrete type automatically (e.g., tuple input to list). Using abstract types can lead to poor validation performance.

Use Any type to skip validation when not needed

If a field does not require validation, use the Any type annotation to keep the value unchanged and avoid validation overhead.

Generic container implementation with MySequence

Custom generic container types like MySequence(Sequence[T]) can be created by implementing __get_pydantic_core_schema__. The implementation uses get_args(source) to extract type parameters, calls handler.generate_schema(Sequence[args[0]]) to generate schema for the parameterized sequence, and returns a union_schema combining is_instance_schema for existing instances and no_info_after_validator_function to convert lists to the custom type.

Accessing field name in __get_pydantic_core_schema__

As of Pydantic V2.4, the field name can be accessed within __get_pydantic_core_schema__ via handler.field_name, and is available to validators as info.field_name. This allows custom types to store or use the field name they were applied to during validation.

Accessing field name in Annotated validators

Markers used with Annotated, such as AfterValidator, can access the field name via info.field_name in their validator functions. For example, def my_validators(value: int, info: ValidationInfo): return f'<{value} {info.field_name!r}>' accesses the field name within an AfterValidator.

Named recursive type aliases

Named type aliases should be used whenever recursive type aliases are needed. Pydantic cannot support implicit recursive aliases due to inability to resolve forward annotations across modules. For example, Json = TypeAliasType('Json', 'dict[str, Json] | list[Json] | str | int | float | bool | None') creates a recursive JSON type using forward annotation syntax. Python 3.12 named type statements do not require forward annotations as the value is lazily evaluated.

__get_pydantic_core_schema__ for custom type validation

The __get_pydantic_core_schema__ method can be implemented on a custom type class to customize how Pydantic generates pydantic-core schema for validation and serialization. This is the Pydantic V2 equivalent of implementing __get_validators__ in V1. The method receives source_type (the type being validated) and handler (a callable to generate schema for other types), and returns a CoreSchema object. This is an advanced internal API that may change; prefer high-level hooks like Annotated markers when possible.

Type annotation patterns for custom types

Pydantic supports several ways to define custom types. Built-in and standard library types (int, str, date) can be used as is with optional strictness control and constraints. Pydantic provides extra types directly in the library (SecretStr) or via pydantic-extra-types. Strictness and constraints cannot be applied to these extra types. The main patterns for custom types are: using Annotated with constraints, creating named type aliases using TypeAliasType or PEP 695 type statement, and implementing __get_pydantic_core_schema__ for extensive customization.

Annotated pattern with Pydantic Field constraints

The Annotated pattern allows creating reusable custom types with validation constraints. For example, PositiveInt can be defined as Annotated[int, Field(gt=0)]. Constraints can alternatively use the annotated-types library to make definitions Pydantic-agnostic, such as Annotated[int, Gt(0)]. These custom types can be validated using TypeAdapter.

Adding validation and serialization to Annotated types

Pydantic markers like AfterValidator, PlainSerializer, and WithJsonSchema can be used within Annotated to customize validation, serialization, and JSON schemas on arbitrary types. For example, TruncatedFloat = Annotated[float, AfterValidator(lambda x: round(x, 1)), PlainSerializer(lambda x: f'{x:.1e}', return_type=str), WithJsonSchema({'type': 'string'}, mode='serialization')] applies rounding on validation, formats as scientific notation on serialization, and overrides the JSON schema for serialization mode.

__get_pydantic_core_schema__ method signature on custom type

When implementing __get_pydantic_core_schema__ as a classmethod on a custom type, the signature is: @classmethod def __get_pydantic_core_schema__(cls, source_type: Any, handler: GetCoreSchemaHandler) -> CoreSchema. The method should call handler(str) or similar to get the base schema, then wrap or modify it using pydantic_core schema functions.

Generic type variables in Annotated

Type variables can be used within Annotated types using typing.TypeVar. For example, ShortList = Annotated[list[T], Len(max_length=4)] creates a generic list type with a maximum length constraint, which can then be instantiated as ShortList[int]. Similarly, PositiveList = list[Annotated[T, Gt(0)]] creates a list of positive-constrained elements.

__get_pydantic_core_schema__ on Annotated metadata

__get_pydantic_core_schema__ can also be implemented on metadata intended for use in Annotated. This is useful for parametrizing custom types beyond generic type parameters or when the actual type instance is not needed. The implementation follows a middleware pattern: modify the type before calling handler, modify the returned core schema, or skip calling handler entirely. Metadata classes implementing this should typically be frozen dataclasses to remain hashable.

GetPydanticSchema for reducing boilerplate

GetPydanticSchema reduces boilerplate when creating marker classes for __get_pydantic_core_schema__. Instead of defining a class, pass a lambda to GetPydanticSchema within Annotated: Annotated[str, GetPydanticSchema(lambda tp, handler: core_schema.no_info_after_validator_function(lambda x: x * 2, handler(tp)))].

Custom validation for third-party types using Annotated

To add Pydantic validation to third-party types that were not designed with Pydantic integration, create an annotation class implementing __get_pydantic_core_schema__ and __get_pydantic_json_schema__, then wrap the third-party type with Annotated. This allows specifying how to validate and serialize instances, what Python and JSON inputs to accept, and how to represent the type in JSON schema without modifying the third-party code.

Generic classes with custom validation

Generic classes can be used as field types with custom validation based on type parameters via __get_pydantic_core_schema__. Use typing.get_args() to extract generic parameters from source_type, and call handler.generate_schema() on the parameter types to create their schemas. This pattern does not require arbitrary_types_allowed configuration.

Custom generic class example with Owner

A dataclass Owner(Generic[ItemType]) can implement __get_pydantic_core_schema__ to validate its item field according to the type parameter. The schema uses json_or_python_schema to handle both JSON (typed_dict) and Python inputs, chain_schema for sequential validation steps, and is_instance_schema to check for Owner instances. The handler.generate_schema(item_tp) generates appropriate schema for the item type parameter.

Callable Discriminator example with models

Example with callable discriminator: from typing import Annotated, Any, Literal from pydantic import BaseModel, Discriminator, Tag class Pie(BaseModel): time_to_cook: int num_ingredients: int class ApplePie(Pie): fruit: Literal['apple'] = 'apple' class PumpkinPie(Pie): filling: Literal['pumpkin'] = 'pumpkin' def get_discriminator_value(v: Any) -> str | None: if isinstance(v, dict): return v.get('fruit', v.get('filling')) return getattr(v, 'fruit', getattr(v, 'filling', None)) class ThanksgivingDinner(BaseModel): dessert: Annotated[ Annotated[ApplePie, Tag('apple')] | Annotated[PumpkinPie, Tag('pumpkin')], Discriminator(get_discriminator_value), ]

Union validation fundamentals

Unions require only one member to be valid, unlike other types that require all fields/items/values to be valid. This creates complexity around which member(s) to validate against and in which order, and which errors to raise when validation fails.

Three fundamental union validation approaches

Pydantic supports three approaches to validating unions: (1) left to right mode - tries each member in order and returns the first match; (2) smart mode - tries members in order but continues past the first match to find a better match, this is the default mode in Pydantic >=2; (3) discriminated unions - only one member is tried based on a discriminator.

union_mode='left_to_right' usage

Left to right mode is set via the Field() parameter: union_mode='left_to_right'. Validation is attempted against each member of the union in the order they are defined, and the first successful validation is accepted. If all members fail, the validation error includes errors from all members. This is not the default in Pydantic >=2.

Left to right mode pitfall with type coercion

In left to right mode with lax validation (the default), a numeric string like '456' will validate successfully against int (the first member) before being tried against str. For a union like int | str with union_mode='left_to_right', the input '456' becomes int(456) instead of remaining str('456'). This can lead to surprising results, which is why left to right is not the default.

Smart mode is default union validation mode

In Pydantic >=2, union_mode='smart' is the default mode for union validation. This mode attempts to select the best match for the input from union members. The algorithm uses metrics including number of valid fields set and exactness of the match. The exact algorithm may change between Pydantic minor releases.

Smart mode exactness scoring

Smart mode scores matches into three exactness groups from highest to lowest: (1) exact type match (e.g., int input to float | int union matches int exactly); (2) validation would succeed in strict mode; (3) validation would succeed in lax mode.

Smart mode algorithm for BaseModel, dataclass, TypedDict

For BaseModel, dataclass, and TypedDict in smart mode: (1) union members are attempted left to right, scoring successful matches into exactness categories and tallying valid fields set count; (2) the member with highest valid fields set count is returned; (3) if there is a tie, exactness score is used as tiebreaker; (4) if all fail, all errors are returned.

Smart mode algorithm for other data types

For all other data types in smart mode: (1) union members are attempted left to right, and if validation succeeds with exact type match, that member is returned immediately and following members are not attempted; (2) if validation succeeded on at least one member as a strict match, the leftmost strict match is returned; (3) if validation succeeded on at least one member in lax mode, the leftmost match is returned; (4) if validation failed on all members, all errors are returned.

Smart mode valid fields metric introduction

The valid fields metric for smart mode was introduced in Pydantic v2.8.0. Prior to this version, only exactness was used to determine the best match. This metric counts the number of valid fields set on models, dataclasses, and typed dicts, including nested models, which bubble up to the top-level union.

Discriminated unions overview

Discriminated unions are also called tagged unions. They validate more efficiently by using a discriminator to choose which union member to validate against. This makes validation more efficient and avoids proliferation of errors when validation fails. Discriminated unions in generated JSON schema implement the discriminator attribute from the OpenAPI specification.

String discriminator setup

For discriminated unions with string discriminators: each model in the union must have a common field (e.g., pet_type) typed as accepting one or more Literal values. The Field() function must specify the discriminator parameter with the field name as a string. Example: pet: Cat | Dog | Lizard = Field(discriminator='pet_type').

Callable Discriminator usage

Callable discriminators are used when there is no single uniform field across all union members. A callable discriminator function receives the input data and returns the discriminator tag value. The function should handle both dict and model type inputs, similar to mode='before' validators, since Pydantic uses callable discriminators for both validation and serialization.

Callable discriminator returns None for missing tags

If a callable discriminator function cannot determine a discriminator value, it should return None. When None is returned, a union_tag_not_found error is raised.

Nested discriminated unions

Only one discriminator can be set for a field, but multiple discriminators can be combined by creating nested Annotated types. This allows hierarchical discrimination: first discriminate by one field, then by another field within those cases.

Left to right mode example

Example with union_mode='left_to_right': from pydantic import BaseModel, Field, ValidationError class User(BaseModel): id: str | int = Field(union_mode='left_to_right') print(User(id=123)) # id=123 print(User(id='hello')) # id='hello' With invalid input [], both members fail and errors from both are returned.

Smart mode example with UUID

Example showing smart mode: from uuid import UUID from pydantic import BaseModel class User(BaseModel): id: int | str | UUID name: str user_01 = User(id=123, name='John Doe') # id=123 (exact int match) user_02 = User(id='1234', name='John Doe') # id='1234' (string, not coerced to int) user_03_uuid = UUID('cf57432e-809e-4353-adbd-9d5c0d733868') user_03 = User(id=user_03_uuid, name='John Doe') # id=UUID(...) (exact UUID match)

String discriminator example

Example with string discriminator: from typing import Literal from pydantic import BaseModel, Field, ValidationError class Cat(BaseModel): pet_type: Literal['cat'] meows: int class Dog(BaseModel): pet_type: Literal['dog'] barks: float class Lizard(BaseModel): pet_type: Literal['reptile', 'lizard'] scales: bool class Model(BaseModel): pet: Cat | Dog | Lizard = Field(discriminator='pet_type') n: int print(Model(pet={'pet_type': 'dog', 'barks': 3.14}, n=1)) # pet=Dog(pet_type='dog', barks=3.14) n=1

Nested discriminated unions example

Example with nested discriminators: from typing import Annotated, Literal from pydantic import BaseModel, Field class BlackCat(BaseModel): pet_type: Literal['cat'] color: Literal['black'] black_name: str class WhiteCat(BaseModel): pet_type: Literal['cat'] color: Literal['white'] white_name: str Cat = Annotated[BlackCat | WhiteCat, Field(discriminator='color')] class Dog(BaseModel): pet_type: Literal['dog'] name: str Pet = Annotated[Cat | Dog, Field(discriminator='pet_type')] class Model(BaseModel): pet: Pet n: int

Discriminator custom error parameters

The Discriminator constructor accepts optional parameters to customize error messages: custom_error_type (the type attribute of ValidationError), custom_error_message (the msg attribute of ValidationError), and custom_error_context (the ctx attribute of ValidationError). Example: Discriminator(func, custom_error_type='invalid_union_member', custom_error_message='Invalid union member', custom_error_context={'discriminator': 'str_or_model'})

Union validation error messages with discriminated unions

When union validation fails, untagged unions produce verbose error messages with validation errors for each case. Discriminated unions simplify error messages because validation errors are only produced for the case with a matching discriminator value. Using Tag annotations further simplifies error messages by labeling each union case.

TypeAdapter for union validation

TypeAdapter can be used to validate data against a union without inheriting from BaseModel. This is useful when you want to validate data solely against a union type. Example: type_adapter = TypeAdapter(Pet); pet = type_adapter.validate_python({'pet_type': 'cat', 'color': 'black', 'black_name': 'felix'})

Root models with Literal in discriminated unions

Root models with a Literal root type can be used in place of Literal types in discriminated unions. This feature was added in Pydantic v2.13.

Recommended union approach

Pydantic recommends using discriminated unions. They are both more performant and more predictable than untagged unions as they allow you to control which member of the union to validate against. For complex cases with untagged unions, use union_mode='left_to_right' if you need guarantees about validation order. For specialized behavior, use custom validators.

Callable Discriminator with models and primitives

Example combining models and primitive types: from typing import Annotated, Any from pydantic import BaseModel, Discriminator, Tag, ValidationError def model_x_discriminator(v: Any) -> str | None: if isinstance(v, int): return 'int' if isinstance(v, (dict, BaseModel)): return 'model' return None class SpecialValue(BaseModel): value: int class DiscriminatedModel(BaseModel): value: Annotated[ Annotated[int, Tag('int')] | Annotated['SpecialValue', Tag('model')], Discriminator(model_x_discriminator), ]

Type coercion with validate_call

By default, validate_call coerces types before passing them to the actual function. For example, a string '2000-01-01' will be converted to a date object if the parameter is annotated as date. Unannotated parameters are inferred as Any.

validate_call example with type coercion

Example showing validate_call with date coercion: A function parameter annotated as date will accept string input like '2000-01-01' and automatically convert it to a date object before passing to the function.

Avoid abstract collections types

Avoid using abstract collections such as collections.abc.Sequence if the goal is to accept both list and tuples, as this is inefficient. Use specific concrete types instead.

StringConstraints example

Example using StringConstraints for string-specific validation: from typing import Annotated from pydantic import BaseModel, StringConstraints class Model(BaseModel): a: Annotated[str, StringConstraints(strip_whitespace=True)] This applies strip_whitespace constraint to the field instead of writing a validator.

StringConstraints for string-specific constraints

String constraints such as strip_whitespace, to_upper, to_lower, and ascii_only can only be specified using pydantic.StringConstraints, not through the Field() function. Example: a: Annotated[str, StringConstraints(strip_whitespace=True)]

Avoid unions for type coercion

Avoid using unions such as int | str if the goal is to coerce str to int via a validator, because every use of the field will need to check for each type before doing anything with it.

Give your agent this brain