deployment/replication

10 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Multiple worker processes for handling more clients

A single FastAPI process using the fastapi command (Uvicorn) can serve multiple clients concurrently. To handle more clients than a single process can handle, especially with multiple CPU cores available, multiple worker processes of the same application can run simultaneously with requests distributed among them. These multiple processes are commonly called workers.

Single port requirement for multiple worker processes

Only one process can listen on one combination of port and IP address on a server. To have multiple worker processes, there must be a single process listening on the port that transmits communication to each worker process.

Memory consumption per process

Multiple processes normally do not share any memory. Each running process has its own variables and memory. If code loads a Machine Learning model of 1 GB in size, one process will consume at least 1 GB of RAM. If 4 processes are running, each will consume 1 GB of RAM, totaling 4 GB of RAM. This must be considered when sizing servers.

CPU versus memory utilization patterns

The percentage of CPU used by each process can vary significantly over time. Memory (RAM) normally stays more or less stable. If an API does a comparable amount of computations each time with many clients, CPU utilization will probably also be stable.

Replication strategies and tools

Replication can be achieved through: Uvicorn with --workers (one process manager listens on IP and port, starting multiple Uvicorn worker processes); Kubernetes and distributed container systems (container layer listens on IP and port with multiple containers each running one Uvicorn process); or cloud services that handle replication by running a single Uvicorn process which the cloud service replicates.

Multiple workers with --workers flag

You can start multiple Uvicorn worker processes using the `--workers` command line option with either the `fastapi` command or the `uvicorn` command directly. For example, `fastapi run --workers 4 main.py` or `uvicorn main:app --host 0.0.0.0 --port 8080 --workers 4` will start 4 worker processes.

Worker processes and process manager

When running with the `--workers` option, Uvicorn creates a parent process that acts as a process manager, with each `--workers` value creating an additional worker process. Each process has its own PID displayed in the logs.

Workers enable multi-core CPU utilization

Using multiple workers with the `--workers` option allows FastAPI applications to take advantage of multiple CPU cores, parallelize execution, and serve more requests concurrently.

Workers primarily address replication concept

Using workers mainly helps with the replication part of deployment concepts (the number of processes running). However, workers alone do not fully address other deployment concepts such as security (HTTPS), running on startup, automatic restarts, memory management, and previous steps before starting.

Single Uvicorn process per container in Kubernetes

When running FastAPI in Kubernetes, you should avoid using workers and instead run a single Uvicorn process per container. The container orchestration system handles process replication and management.

Give your agent this brain