Understanding Remote Procedure Calls (RPC): Architecture, Stubs, and Inter-Service Communication

In distributed systems, microservices frequently need to communicate over the network. Traditionally, this was dominated by resource-oriented RESTful APIs using JSON over HTTP/1.1. However, Remote Procedure Calls (RPC)—an architectural paradigm originally developed decades ago—have experienced a massive resurgence across modern cloud infrastructure, powering high-throughput systems at companies like Google, Netflix, and Slack.

At its core, RPC is designed to make a remote network call look and feel identical to a local function call, abstracting away serialization, deserialization, transport protocols, connection management, and error handling.

---

The Inter-Service Communication Problem

Consider an architectural pattern where an Authentication Service must instruct a Notification Service to dispatch a One-Time Password (OTP) whenever a user logs in.

The Standard REST Approach

In a typical REST-based implementation, the authentication handler might look like this:

def login(email, password):
    # 1. Validate credentials and generate access token
    token = generate_access_token(email, password)
    
    # 2. Trigger notification
    send_notification_otp(email)
    return token

Under the hood, the developer must implement send_notification_otp(email) using an HTTP client:

import requests

def send_notification_otp(email):
    payload = {"email": email}
    try:
        response = requests.post(
            "http://notification.internal.local/v1/email/otp", 
            json=payload,
            timeout=2.0
        )
        response.raise_for_status()
        return response.json()
    except requests.exceptions.RequestException as e:
        # Handle retries, exponential backoff, circuit breaking, etc.
        logger.error(f"Failed to send OTP: {e}")
        raise ServiceUnavailableError()

Where REST-Based Microservices Fall Short

1. Boilerplate Duplication: Every service calling the Notification Service must independently implement HTTP request handling, endpoint routing, serialization, connection management, and timeout configurations.
2. Language Fragmentation: In polyglot architectures (e.g., Python, Go, Java, Rust), every language has its own HTTP client primitives (requests in Python, net/http in Go, HttpClient in Java). There is no single enforced standard across client implementations.
3. Weak API Contracts: Payload structures are typically enforced at runtime. If the Notification Service changes a field or type in its JSON payload, downstream services only find out when requests fail in production or break JSON unmarshalling.
4. Serialization Overhead: JSON is human-readable text. Serializing large payloads to JSON strings, parsing strings back into objects, and transmitting non-compressed text over the wire wastes CPU cycles and bandwidth.

---

What is RPC?

Remote Procedure Call (RPC) solves this friction by redefining how functions are executed across a network boundary.

Instead of mentally mapping operations to HTTP verbs (GET, POST, PUT, DELETE) and URL paths (/v1/resource/:id), RPC treats remote services as an extension of your local application code. You simply invoke a method on an object, pass native typed objects as arguments, and receive native typed objects as a return value.

// Go client invoking a remote Java notification service as if it were a local package
response, err := notificationClient.SendOTP(ctx, &pb.OTPRequest{Email: email})
if err != nil {
    log.Fatalf("could not send OTP: %v", err)
}

Behind the scenes, the RPC framework handles:

• Converting native language objects into bytes (Marshalling / Serialization).
• Opening, maintaining, and pooling network connections.
• Transmitting packets across the wire.
• Parsing the wire payload back into the server’s native structures (Unmarshalling / Deserialization).
• Managing retries, timeouts, and cancellation signals.

Key Caveat: Although an RPC syntactically resembles a local function call, it is fundamentally a network call. A local procedure call executes in nanoseconds within CPU cache and RAM; a remote call takes milliseconds and can fail due to network partitions, host crashes, or packet loss. Engineers must always account for network latency and fault tolerance.

---

The Core Mechanism: Stubs and Marshalling

How does an RPC client turn a local method call into a remote execution and reconstruct the return value? It uses Stubs.

A stub is an auto-generated client-side or server-side adapter that handles the conversion between in-memory language constructs and wire-level transmission formats.

+-------------------------------------------------------------------------+
|                              CLIENT NODE                                |
|                                                                         |
|  +------------------------+             +----------------------------+  |
|  | Client Application     |             | Client Stub                |  |
|  | - Calls SendOTP(email) | ----------> | - Marshals Go Struct to    |  |
|  |   like a local method  |             |   Wire Format (Protobuf)   |  |
|  +------------------------+             +----------------------------+  |
|                                                        |                |
+--------------------------------------------------------|----------------+
                                                         | Network (TCP / HTTP/2)
                                                         v
+-------------------------------------------------------------------------+
|                              SERVER NODE                                |
|                                                                         |
|  +------------------------+             +----------------------------+  |
|  | Server Service Logic   |             | Server Stub (Skeleton)     |  |
|  | - Implements SendOTP() | <---------- | - Unmarshals Wire Format   |  |
|  | - Returns native class |             |   into Java Class Instance |  |
|  +------------------------+             +----------------------------+  |
+-------------------------------------------------------------------------+

Step-by-Step Execution Flow

1. Client Invocation: The client application invokes the procedure via the Client Stub, passing native language objects (e.g., a Go struct).
2. Marshalling (Client Side): The client stub converts the parameters into a standardized, compressed, language-agnostic wire format (e.g., binary Protocol Buffers).
3. Network Transport: The RPC runtime transmits the serialized byte stream over the network via its chosen transport layer (e.g., HTTP/2, raw TCP, WebSockets).
4. Unmarshalling (Server Side): The Server Stub (historically called a skeleton) receives the raw bytes, parses them, and instantiates the server’s native object model (e.g., a Java class instance).
5. Method Execution: The server stub calls the actual business logic implementation on the server.
6. Response Marshalling: The server implementation returns a response object to the server stub, which marshals it into bytes.
7. Return Path: The bytes travel back across the network, the client stub unmarshals them into a native client object, and the original client function receives the return value.

---

Interface Definition Language (IDL) and Code Generation

Because services in a distributed architecture are often written in different languages, RPC frameworks rely on an Interface Definition Language (IDL) to define the contract between client and server.

The IDL is a strictly typed, human-readable specification of:

• The available services.
• The callable RPC methods inside each service.
• The schema of the request and response messages.

Example: Protocol Buffers (.proto)

In gRPC (the most prevalent modern RPC implementation), the IDL uses Protocol Buffers:

syntax = "proto3";

package notification;

// Definition of the remote service
service NotificationService {
  rpc SendOTP (OTPRequest) returns (OTPResponse);
}

// Schema for the request payload
message OTPRequest {
  string email = 1;
  int64 timestamp = 2;
}

// Schema for the response payload
message OTPResponse {
  bool success = 1;
  string message_id = 2;
}

The Stub Generator

Once the .proto contract is defined, developers do not manually write the networking or parsing code. Instead, they run an automated Stub Generator (such as the protoc compiler):

# Generate Go client and server stubs
protoc --go_out=. --go-grpc_out=. notification.proto

# Generate Java client and server stubs
protoc --java_out=. --grpc-java_out=. notification.proto

The compiler produces target-language source files containing:

• Strongly typed data structures matching the messages.
• Serialization and deserialization routines.
• Client stubs equipped with connection pooling, deadlines, and invocation methods.
• Server base interfaces that developers simply extend with business logic.

---

Debunking the Transport Myth: RPC vs. HTTP

A frequent misconception in systems architecture is that HTTP means REST and RPC cannot use HTTP.

In reality:

• REST is an architectural style centered on resources, state representations, and uniform interfaces (typically mapped to HTTP verbs).
• HTTP is an application-layer network protocol (Layer 7).
• RPC is a communication paradigm completely decoupled from the underlying transport protocol.

An RPC framework can run on top of virtually any transport mechanism:

• Raw TCP / UDP: Used by specialized internal frameworks for raw speed.
• HTTP/1.1: Possible, but limited by head-of-line blocking and lack of native multiplexing.
• HTTP/2: Used by modern frameworks like gRPC. It provides multiplexed bidirectional streaming, binary framing, and HPACK header compression over a single long-lived TCP connection.
• WebSockets: Can be leveraged for persistent bidirectional RPC links.

+-------------------------------------------------------------+
|                     RPC Application Layer                   |
+-------------------------------------------------------------+
                               |
      +------------------------+------------------------+
      |                        |                        |
+------------+           +------------+           +------------+
|  HTTP/2    |           |  Raw TCP   |           | WebSockets |
|  (gRPC)    |           | (Custom)   |           | (Custom)   |
+------------+           +------------+           +------------+

---

Advantages of Modern RPC

Advantage	Description
Strong API Contracts	Contracts are enforced strictly at compile time through the IDL, preventing schema drift and runtime parsing bugs.
Polyglot Code Generation	Developers define an interface once and automatically generate fully typed client SDKs across dozens of programming languages.
Superior Wire Performance	Binary serialization formats like Protobuf are significantly smaller and faster to serialize/deserialize than plain-text JSON.
Transport Optimization	Out-of-the-box support for persistent connections, connection pooling, TCP connection reuse, multiplexing, and bidirectional streaming.
Abstracted Cross-Cutting Concerns	Deadlines/timeouts, exponential backoff retries, metadata propagation (distributed tracing), and mTLS encryption are handled at the framework layer.
Developer Productivity	Engineers write idiomatic native code with IDE autocompletion instead of crafting HTTP strings and configuring parsing logic manually.

---

Trade-offs and Operational Challenges

Despite its advantages, RPC introduces specific engineering challenges:

1. Coupled Deployment Workflows: Whenever a signature or field definition changes in an IDL file, stubs must be re-generated, and client applications must be updated and redeployed. Schema versioning rules must be strictly maintained (e.g., never changing field tag numbers in Protobuf).
2. Debugging and Testing Complexity: You cannot easily test an RPC endpoint using standard browser windows or simple curl commands without specialized reflection tools (like grpcurl) because the transport payload is binary rather than plain text.
3. Steeper Initial Setup: Starting a simple REST service takes minutes in almost any language framework. Setting up an RPC pipeline requires configuring an IDL compiler, managing .proto build artifacts, and setting up client stub distribution repositories.
4. Limited Direct Browser Support: Browsers do not expose low-level access to HTTP/2 frames required by pure gRPC. As a result, browser-to-backend communication typically relies on standard REST/GraphQL APIs or requires proxy translation layers like grpc-web or Envoy JSON-to-gRPC transcoding.

---

Summary

Remote Procedure Calls solve the operational and performance bottlenecks of microservice-to-microservice communication. By abstracting serialization, transport, and network handling behind auto-generated stubs, RPC frameworks allow polyglot microservices to communicate with high throughput, compile-time safety, and minimal development friction.