The Problem
At Banjo, our entire platform was event-driven. Every service communicated through Kafka or gRPC, and every payload was a Protocol Buffer message. We started on Protobuf 2 and eventually migrated to Protobuf 3. With services written in Ruby, Java, Python, and TypeScript, having a single schema language that all four could share was not a convenience. It was what made the platform coherent.
Protobufs solved the cross-language problem completely. Elasticsearch created a new one.
Elasticsearch has its own type system, its own JSON document format, and its own opinions about how data should be structured. Every time we needed to persist or query anything, we had to write a translation layer: Protobuf to Java POJO, POJO to Elasticsearch JSON, Elasticsearch JSON back to POJO, POJO back to Protobuf. Four hops to store something. Four hops to retrieve it.
We were disciplined enough to consolidate persistence logic as much as we could, but the translation code was still scattered. Nearly every usage had its own normalizer, its own mapping assumptions, its own edge cases. There was no standardized way to handle the conversion. Each service reinvented the wheel, and the bugs that came out of that inconsistency were a constant tax on the team.
The Realization
The breaking point was not a bug or an outage. It was a quieter kind of frustration: the recognition that we were spending real engineering time maintaining code that should not have to exist.
Protobufs have constraints that most developers treat as limitations. They are immutable. There is no null. Field numbers are permanent identifiers, not just positions. But if you are designing a data persistence layer, those constraints are actually features. Immutability means you are not accidentally mutating something mid-pipeline. No null means you never have to defend against a missing field in a way that silently corrupts a document. Permanent field numbers mean schema evolution has real rules instead of being left to convention.
The insight was this: if the Protobuf schema is already the canonical definition of your data, why maintain a parallel set of Java classes that say the same thing? Why write mapping code by hand when the type information to generate that mapping already exists in the proto file?
I could see a path to doing three things at once. Remove the POJO layer entirely and let services work with Protobuf objects directly. Build an Elasticsearch adapter that reads the Protobuf schema and generates index mappings automatically. Build a response translator that converts Elasticsearch JSON back into Protobuf objects. The two type systems are structured enough that, with the right adapter between them, they work together almost seamlessly.
The Design
The idea first took shape at Banjo, but the library went through several iterations across different contexts before it became what it is today. Each codebase I brought it into taught me something about where the original design had gaps.
The core principle stayed constant throughout: the .proto file is the single source of truth. When a service starts up, the library inspects the Protobuf message definitions and generates the corresponding Elasticsearch index mappings. No separate mapping files. No configuration that can drift out of sync with the schema. If the Protobuf changes, the index mapping reflects that change.
Custom annotations on Protobuf fields control indexing behavior where the defaults are not enough. You can specify analyzers for full-text fields, mark fields as keyword-only for exact matching and aggregations, or configure how nested objects should be mapped. The annotation vocabulary is intentional and minimal, enough to cover real cases without becoming a configuration language of its own.
A small, generic example shows the shape. This is an illustration written for this article, not the library’s code. The indexing hints are ordinary Protobuf custom options, so they live in the schema and travel with it:
syntax = "proto3";
package catalog.v1;
import "google/protobuf/descriptor.proto";
import "google/protobuf/timestamp.proto";
option java_multiple_files = true; // Product becomes its own Java class
// Declared once, reused by every message that needs them.
extend google.protobuf.FieldOptions {
string es_analyzer = 50001; // full-text field, analyzed this way
bool es_keyword = 50002; // exact match and aggregations only
}
message Product {
string id = 1 [(es_keyword) = true];
string title = 2 [(es_analyzer) = "english"];
string description = 3 [(es_analyzer) = "english"];
repeated string tags = 4 [(es_keyword) = true];
int64 price_cents = 5;
bool in_stock = 6;
google.protobuf.Timestamp listed_at = 7;
}Everything the index mapping needs is already in that descriptor. Field names carry over, scalar types map directly (int64 to long, bool to boolean), the well-known Timestamp becomes a date, and the options decide between text and keyword. repeated needs no special handling, because any Elasticsearch field can hold an array:
{
"mappings": {
"properties": {
"id": { "type": "keyword" },
"title": { "type": "text", "analyzer": "english" },
"description": { "type": "text", "analyzer": "english" },
"tags": { "type": "keyword" },
"price_cents": { "type": "long" },
"in_stock": { "type": "boolean" },
"listed_at": { "type": "date" }
}
}
}The round trip needs no hand-written mapper either. Protobuf’s own JSON support translates in both directions, with two settings that matter: keep the proto field names the mapping was built from, and write proto3 default values (false, 0) instead of omitting them, or a query for in_stock: false quietly finds nothing:
@Singleton
public class ProductStore {
// alwaysPrintFieldsWithNoPresence() needs protobuf-java-util 4.26+;
// on 3.x the equivalent is includingDefaultValueFields().
private static final JsonFormat.Printer TO_JSON =
JsonFormat.printer()
.preservingProtoFieldNames() // match the mapping's field names
.alwaysPrintFieldsWithNoPresence(); // keep false and 0, so they stay queryable
private static final JsonFormat.Parser FROM_JSON =
JsonFormat.parser().ignoringUnknownFields(); // tolerate fields from newer schemas
private final RestClient client; // Elasticsearch low-level REST client
public ProductStore(RestClient client) {
this.client = client;
}
public void save(Product product) throws IOException {
Request request = new Request("PUT", "/products/_doc/" + product.getId());
request.setJsonEntity(TO_JSON.print(product));
client.performRequest(request);
}
public Product findById(String id) throws IOException {
Response response = client.performRequest(new Request("GET", "/products/_source/" + id));
Product.Builder builder = Product.newBuilder();
FROM_JSON.merge(EntityUtils.toString(response.getEntity()), builder);
return builder.build();
}
}No POJO and no normalizer. Add a field to the message, and the mapping, the stored document, and the parsed result all follow it. One more reason the mapping has to be explicit rather than dynamic: JsonFormat prints int64 as a JSON string ("1999"), so _source holds a string, and it is the explicit long mapping that coerces it into a number for range queries and aggregations.
Micronaut’s compile-time dependency injection is what makes this practical at scale. Most ORM-style frameworks lean heavily on reflection, which means type mismatches and misconfigured repositories surface at runtime, often in production. Because Micronaut resolves repository interfaces at compile time, wiring errors surface during the build, and anything the generated mapping cannot accept surfaces at startup, when the mapping is built. Neither waits for a production query to find it.
Schema Evolution
This is where things get genuinely interesting, and where most homegrown translation layers eventually break down.
Protobufs are designed to evolve forward. The contract is that you do not change field numbers, do not rename fields in ways that break existing consumers, and do not change field types in backwards-incompatible ways. If you follow those rules, older clients can still read messages written by newer services, and newer services can still read messages written by older clients.
Elasticsearch index mappings are less forgiving. You cannot change the type of an existing field in a live index. Add a new field and Elasticsearch can handle it through dynamic mapping, but a type change or a structural reorganization requires a reindex.
The library handles the common evolution cases automatically. A new field on a Protobuf message adds the corresponding field to the index mapping. Backwards-compatible type changes propagate correctly. Index settings updates apply without requiring manual intervention.
For the cases that genuinely require a reindex, the ones that fall outside what Elasticsearch allows through an in-place mapping update, the library surfaces that explicitly rather than silently failing or producing a corrupt index. The contract mirrors the Protobuf contract: evolve forward, stay backwards-compatible, and migrations are smooth. Break that contract and the library tells you so at startup rather than letting bad data accumulate quietly.
The gRPC Pairing
One of the more satisfying outcomes of this design is how cleanly it pairs with gRPC.
When you define a Protobuf message, you have the data schema. When you define a gRPC service using that same message type, you have the API contract. With this library, that Protobuf message also gives you the Elasticsearch index automatically.
The practical result is a workflow that looks like this: write the .proto file, and you get a well-structured Elasticsearch index, a gRPC service stub ready to implement, and serialization handled in both directions without any additional code. No POJOs, no manual mapping, no normalizers.
For teams building services that need both a queryable data store and a gRPC API, this eliminates the majority of the boilerplate that usually comes before any real work can begin. The schema is the contract, the contract is the index, and the index stays in sync with the schema for the life of the service.
Where It Stands
I use this library across several of my own services today. It is one of a small set of tools I have built and rebuilt across different contexts over the years, the kind of problem that, once you have solved it well, you never want to solve again from scratch.
One piece of that toolbox is public. proto-faker is an open-source library on Maven Central (com.phatjam98:proto-faker) that starts from the same premise, that a message’s Protobuf descriptor already knows everything the tooling needs, and points it at tests instead of storage: it reads the descriptor and generates realistic fake instances, field-name aware (an email field gets an email address), with nested messages, repeated fields, maps, and oneof groups filled in. More of my open-source work is on the projects page.