Understanding HTTP protocol using netcat

Majority of the traffic over the internet is HTTP Traffic. There is a HTTP Client which wants some data from HTTP Server, so it creates a HTTP Request Message in the protocol understandable by the server and sends it. Server reads the message, understands it, acts accordingly and replies back with HTTP Response.

This complete process is abstracted by the tools like curl, requests libraries and utilities like Postman. Instead of using these tools and utilities, we shall go by the hard way and see HTTP messages in action.

The Webserver

For experimentation purpose let’s create a very basic webserver in Python Flask framework that exposes a trivial Hello World end point.

Python webserver script

from flask import Flask
app = Flask(__name__)

@app.route('/hello')
def hello():
    return "Hello, World!"

app.run(port=3000)

Installing requirements

pip install flask

Start the webserver

python hello.py

The server listens on port 3000 . If you hit from the browser http://localhost:3000/hello, you should see Hello, World! rendered.

The HTTP Request Message

A HTTP Client talks to HTTP Server via a common protocol that is understandable by the two parties. A sample HTTP request message looks something like

GET /hello.html HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE5.01; Windows NT)
Host: www.sample-server.com
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

To understand more about HTTP Request messages, see references at the end of this article.

The HTTP Communication happens over a TCP Connection. So we create a TCP connection with the server and try to get response from it. To get a TCP connection I will use netcat.

Netcat

netcat is the utility that is used for just about anything under the sun involving TCP or UDP. It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning, and deal with both IPv4 and IPv6.

The webserver that was created above is listening on port 3000 . Lets create a TCP Connection and connect to it using netcat.

netcat localhost 3000

The command along with creating a TCP connection, will also open a STDIN. Anything passed in that input stream will reach the server via the connection. Lets see what happens when we provide This is a sample as input.

The input message given is not a valid HTTP message hence server responded with a status code of 400 which is for Bad Request. And if you closely observe the server logs on flask application, you will see an entry of our last request.

Since the server is a HTTP Server, so it understands HTTP request. Let’s create one to hit our exposed API endpoint /hello .

The HTTP request message for this request looks something like this

GET /hello HTTP/1.1

And you should see output like this

The HTTP Server understands the message sent from the client and it responded back as directed by the source code.

Complex Requests and HTTP Request Messages

GET method with query params and headers

Following method exposes an endpoint which accepts a query parameter named name, and returns a response with name in it.

from flask import request

@app.route('/user')
def get_user():
    name = request.args.get('name')
    return "Requested for name = %s" % name

HTTP Request Message

Provide the HTTP request message below when STDIN opens up after you execute netcat command and connect with the server.

GET /user?name=arpit HTTP/1.1

Output

Basic POST Method example

Following method accepts form data via HTTP POST method and returns a dummy response with username and password in it.

from flask import request

@app.route('/login', methods=['POST'])
def login():
    username = request.form.get('username')
    password = request.form.get('password')
    return "Login successful for %s:%s" % (username, password)

HTTP Request Message

Provide the HTTP request message below when STDIN opens up after you execute netcat command and connect with the server.

POST /login HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 32

username=arpit&password=welcome

Output

POST Method with JSON Request Body

Following method accepts JSON data that contains a field id with integer value via HTTP POST method and returns a dummy response with id in it.

from flask import request

@app.route('/save', methods=['POST'])
def save_user():
    user_data = request.json
    return 'Saving user with id = %d' % (user_data.get('id'))

HTTP Request Message

Provide the HTTP request message below when STDIN opens up after you execute netcat command and connect with the server.

POST /save HTTP/1.1
Content-Type: application/json
Content-Length: 30

{"id": 1092, "name": "Arpit"}

Output

Conclusion

The hard way to hit REST endpoints was not hard at all ;-) Stay curious and dive deep.

References:

1. HTTP/1.1: HTTP Message
2. HTTP Requests - Tutorialspoint
3. The TCP/IP Guide - HTTP Request Message Format
4. HTTP Status Codes
5. Netcat man page
6. HTTP Methods