Buf has released protobuf-py, a complete Protocol Buffers library for Python written from scratch. It passes every binary and JSON case in the Protobuf conformance suite across proto2, proto3, and editions. It supports extensions, custom options, unknown fields, dynamic messages, and well-known types. Generated code is readable, typed Python with no runtime dependencies. With the optional Rust accelerator, it matches the performance of upb, Google's C engine, in production workloads.

The Problem with Google's Python Protobuf

Google's protobuf Python package is complete and battle-tested, but the API and generated code feel like a binding around a C++/Java runtime. The engine is upb, written in C. Messages live in a C arena; the Python object is a handle. Reading a field crosses into C, finds the value, and materializes a Python object. This leaves fingerprints everywhere:

  • Generated _pb2.py files are hard to read — classes are assembled at import time to configure the C engine.
  • Methods like HasField, WhichOneof, and CopyFrom have a C++-shaped API. HasField raises on proto3 scalars. WhichOneof returns a string you hand back to getattr.
  • Generated imports are absolute and break in packages. There's a separate PyPI tool (fix-protobuf-imports) just to rewrite them.
  • Types register in a process-wide pool, so importing two builds of the same .proto raises at runtime.

These aren't bugs. They're the cost of sharing an engine across languages. The clumsy API and the speed shipped together — you took both or neither.

What protobuf-py Does Differently

protobuf-py keeps message data in Python. Messages are plain objects with __slots__, fields are ordinary Python values: ints, strings, lists, submessages. The Rust accelerator speeds up parsing and serialization, writing results directly into the object. After parsing, reading a field is just accessing a Python attribute.

Generated code is real, readable Python:

class User(Message[_UserFields]):
    __slots__ = ("first_name", "last_name", "active", "manager", "locations", "projects", "contact")
    if TYPE_CHECKING:
        def __init__(
            self,
            *,
            first_name: str = "",
            last_name: str = "",
            active: bool = False,
            manager: User | None = None,
            locations: list[str] | None = None,
            projects: dict[str, str] | None = None,
            contact: Oneof[Literal["email"], str] | Oneof[Literal["phone"], str] | None = None,
        ) -> None: ...
        first_name: str
        last_name: str
        active: bool
        manager: User | None
        locations: list[str]
        projects: dict[str, str]
        contact: Oneof[Literal["email"], str] | Oneof[Literal["phone"], str] | None

Working with it feels like normal Python:

import copy
from gen.user_pb import User
from protobuf import Oneof

user = User(first_name="Alice", active=True, locations=["NYC", "LDN"])
user.last_name = "Smith"
wire = user.to_binary()
parsed = User.from_binary(wire)

match parsed.contact:
    case Oneof(field="email", value=email):
        send(email)
    case Oneof(field="phone", value=phone):
        call(phone)

inactive = copy.replace(parsed, active=False)  # Python 3.13+

Oneofs become pattern-matchable values. Enums are real IntEnum members. Pyright, mypy, and pytype read the generated code without stubs. Generated files use relative imports. Types resolve via an explicit Registry, not a global pool.

Complete Spec Coverage

Other libraries like betterproto are nicer but drop half the spec. betterproto is proto3-only with no proto2, editions, extensions, or custom options. protobuf-py covers the entire spec: proto2, proto3, editions, extensions, custom options, groups, unknown-field preservation across round trips, packed and expanded repeated fields, and full ProtoJSON encoding of well-known types. It passes Google's conformance suite with zero failures — checked into the repo, and CI fails if that changes.

Performance: Fast Where It Counts

A benchmark that parses a message and throws it away makes upb look untouchable because it defers work. Production code parses once, branches on fields, reads a few values, copies, and serializes. upb pays the Python translation cost on every read. protobuf-py pays it zero times after the first read because parsing already produced a normal Python object.

Buf benchmarked a real-world workload: building a home page response for a social media site. Throughput (ops/sec) relative to upb (higher is faster):

Workloadupbprotobuf-py
Build home page response (end-to-end)1.0x1.06x
Parse (in isolation)1.0x0.22x
Serialize (in isolation)1.0x0.60x

In isolation, upb wins marshaling. End-to-end, protobuf-py comes out ahead. The payload is text-heavy (Reddit post bodies, bios, notifications), where keeping strings as Python objects pays off most. The benchmark is at test_bench.py in the repo.

The Rust accelerator is optional and automatic. Prebuilt wheels load it when present; a pure-Python path behaves identically. It runs on free-threaded Python 3.14. No runtime dependencies.

Getting Started

# In your existing UV project
uv add protobuf-py
uv add --dev protoc-gen-py buf-bin

Point a buf.gen.yaml at your .proto files:

version: v2
inputs:
  - directory: proto
plugins:
  - local: protoc-gen-py
    out: gen

Then generate:

uv run buf generate

You get a typed _pb.py you can open and read. Docs are live, source and benchmarks on GitHub, issues open.

Buf has a track record here: the Buf CLI, Schema Registry, ConnectRPC, Protovalidate, and protobuf-es for TypeScript. Buf engineers worked on Protobuf editions. This library is built by people who helped write the spec.