1 Software Blocks

Generic HTTP Load Balancer

The generic HTTP load balancer could be a hardware balancer or a software balancer.

Clustering Edge Server

The Edge Servers are Red5 servers that are specially customized as reverse proxies and are running on Terracotta so that connection states and some routing information could be shared among nodes and fail-over is ensured.

Terracotta Server

The TC server is used to manage Edge nodes. It's part of the TC solution.

Origin Servers

The Origin Servers are independent and hopefully no information is shared among the nodes. The servers might access a common storage via NAS. The separation policy is defined by Edge Servers and for Live conference and Remote SharedObject communication like Text chat room, the separation is based on URL so that all participants are grouped into one Origin Server.

Links between clients and load balancer

The link is RTMPT protocol so that standard HTTP is used and firewall issue is gone.

Links between Edge Layer and Origin Layer

An Edge Server can make connections to all Origin Servers. The RTMPT connection from the client is multiplexed to one single connection to a specific Origin Server.

2 Deployment

The load balancer is shared with SAMP blades and the proper policy should be defined in it to route the Red5-related requests to the Edge Servers.

3 Design

Connection Establishment

Successful Connection Establishment

1. Client handshakes with Edge. This step includes three packets (client to server, server to client and client to server again) and the Edge will creates a connection object accordingly and maintain the handshake state. Edge puts the connection object to a local list for Keep-alive worker (a scheduled task) to check.
2. Client sends "connect" RTMP invocation packet to Edge with the URL indicating the Scope to be connected.
3. Edge verifies the "connect" request and chooses an Origin according to some load-balancing rules. Edge sends "Connect Message" to Origin containing client ID. Origin creates a connection object accordingly waiting for "connect" RTMP packet.
4. Edge sends "connect" RTMP invocation packet to Origin with the URL sent by client.
5. Origin verifies the "connect" request and sends back the "SUCCESS" RTMP packet to Edge.
6. Edge sends back the "SUCCESS" RTMP packet to client.

Connection Rejected by Edge

1. Client handshakes with Edge. This step includes three packets (client to server, server to client and client to server again) and the Edge will creates a connection object accordingly and maintain the handshake state. Edge puts the connection object to a local list for Keep-alive worker (a scheduled task) to check.
2. Client sends "connect" RTMP invocation packet to Edge with the URL indicating the Scope to be connected.
3. Edge verifies the "connect" request and rejects the connection by sending back the "REJECT" RTMP packet.
4. Edge closes the connection and destroys the connection object.

Connection Rejected by Origin

1. Client handshakes with Edge. This step includes three packets (client to server, server to client and client to server again) and the Edge will creates a connection object accordingly and maintain the handshake state. Edge puts the connection object to a local list for Keep-alive worker (a scheduled task) to check.
2. Client sends "connect" RTMP invocation packet to Edge with the URL indicating the Scope to be connected.
3. Edge verifies the "connect" request and chooses an Origin according to some load-balancing rules. Edge sends "Connect Message" to Origin containing client ID. Origin creates a connection object accordingly waiting for "connect" RTMP packet.
4. Edge sends "connect" RTMP invocation packet to Origin with the URL sent by client.
5. Origin verifies the "connect" request and rejects the connection by sending back the "REJECT" RTMP packet.
6. Origin sends "Close Message" to close the connection. Origin destroys the connection object created.
7. Edge passes the "REJECT" RTMP packet to client and close the connection.

Connection Closed by Timer

A timer runs on each Edge to check the aliveness of the connections. When the connection is created for the first time on an Edge, it is added to a local list (not shared) which will be checked periodically by the timer task. The connection object on Edge should expose some APIs for the timer task to check if it's ok to force the close of connection. Several criteria are defined here.

1. In connect phase, Edge waits too long for the client's first handshake packet.
2. In handshake phase, Edge waits too long for the client's second handshake packet.
3. In connected phase, timeout occurs for the next client request to come. When the connection is explicitly closed by Edge, the Origin will be notified and the corresponding resources on the Origin will be released too.
4. The connection is in closing state but the follow-up client's request never reaches the Edge. The connection will be automatically closed.

Connect Timeout

1. Client sends request to open the connection.
2. Edge waits timeout for the client's first handshake.

Handshake Timeout

1. The client handshakes with Edge and Edge replies the handshake.
2. Edge waits timeout for the client handshake response and close the connection.

Waiting for "Connect" Invocation from Client Timeout

1. The client handshakes with Edge
2. Edge waits timeout for the client's "Connection" Invocation request and close the connection.

Waiting timeout for client's request

1. Timeout for client's request. Edge sends "Close Message" to Origin to release the resources on Origin.
2. Edge closes the connection.

Waiting timeout in closing state

1. Timeout in closing state and the connection is automatically close.

Connection Closed by Origin

1. Send "Close Message" to Edge and release related resources.
2. Send "Close" packet to client (packet is actually put in client's buffer) and change to "closing" state.
3. Client gets the closing packet and the connection is closed.

Connection Closed by Edge

1. Send "Close Message" to Origin and the related resources are released on Origin.
2. Send "Close" packet to client (packet is actually put in client's buffer) and change to "closing" state.
3. Client gets the closing packet and the connection is closed.

Connection State Machine

Edge Connection State Machine

CONNECT

This is the initial state of a connection on Edge. A connection is created in response to the client's RTMPT "Open" request.

HANDSHAKE

On receiving the first handshake from the client, Edge sends the server handshake reply and changes into the HANDSHAKE phase waiting for the client's second handshake.

EDGE_CONNECTED

On receiving the second handshake from the client, Edge changes to EDGE_CONNECTED phase waiting for the "Connect" RTMP invocation from client.

EDGE_CONNECT_ORIGIN_SENT

On receiving the "Connect" invocation, Edge will choose an Origin to make connection and sends the "Connect Message" to that Origin.

ORIGIN_CONNECT_FORWARDED

If the "Connect Message" is accepted by the Origin, Edge will continue to forward the "Connect" invocation to Origin.

EDGE_ORIGIN_CONNECTED

The "Connect" is also accepted by the Origin and now the Edge and Origin are connected.

CLOSING

Due to the characteristic of HTTP, the server needs to wait for the client's retrieval of "Close" before actually closing the connection. The connection is in CLOSING state waiting for the actual close.
When the connection is in CLOSING state, all related resources on Origin are cleaned up.

CLOSED

The connection is dead and removed for GC.

Origin Connection State Machine

CONNECT

On receiving the "Connect Message", the connection object is created and the initial state is CONNECT.

CONNECTED

On receiving the "Connect" invocation, the connection is accepted and state is changed to CONNECTED.

CLOSED

On receiving the "Close Message" or the connection is closed by an explicit close, the connection is closed and all resources are released.

Application Data Flow

When the connection is established between Client and Edge and between Edge and Origin, the client's application requests are served by the server.

1. The client sends a RTMPT request to the server.
2. The request is distributed by the load-balancer to one of the Edges.
3. Edge verifies the validity of the request. If either the client id or the sequence number is wrong, go to step 8), or go to 4).
4. Edge extracts the RTMP packet from the request if the request type is SEND. If the request type is SEND and the Origin connection exists, go to 5), or go to 6).
5. Edge looks up the Origin in the connection object and forwards the packet to Origin. (This step is only for SEND and the Origin connection exists.)
6. Edge puts all the pending RTMP packets from the connection buffer to the RTMPT content and sends back to the client.
7. If the connection state is CLOSING, the connection is closed.
8. Done.

The Origin handles all the requests from the Edge and sends back the replies with push mode. The VOD is pushed by a background thread and Live/RSO is pushed by other requests. When the responses reach the Edge, they are pushed to the corresponding connection buffers.

Multiplexing RTMP - Protocol between Edge and Origin

The Multiplexing RTMP is a protocol between Edge and Origin. The protocol aims to multiplex connections for a specific Origin in one connection between Edge and Origin.
The protocol is based on TCP. The Origin acts as the server and listens on port standard port 1945 (should be configurable).

Multiplexing Packet

The protocol is composed of consecutive pieces of multiplexing packets (M packets). Each M packet has the layout described in the diagram below.

The M packet has a packet header and a packet body as Payload. The header is filled marked as dark blue and the body is marked as light blue in the diagram.
The packet header contains three fields. "Message Type" is the 32-bit type of M packet. "Connection Id" corresponds to the 64-bit id of the RTMPT connection. "Length" is the 32-bit size of the Payload. All fields are encoded as big-endian integer.
The content in the packet body as Payload is decided by the "Message Type" and might be of zero length.

Message Type

Currently three types of packet are defined.

1. OPEN, 0x01: Create a connection. Length is 0.
2. CLOSE, 0x02: Close a connection. Length is 0.
3. RTMP, 0x03: RTMP encoded packets.

Origin Load Balancing Mechanism

The mechanism used for load-balancing the request is by URL pattern. The connections are categorized as two groups: the group for Live classroom and Live chat and the group for VOD and recording.

Live and RSO Balancing

The connection sharing a common URL pattern will be put on a same Origin.

VOD and Recording Balancing

The VOD and recording connections will be assigned an Origin per the overall load of Origins.
For example a connection to URL "vod/vod1.flv" might be assigned to Origin A or Origin B per their load. The load is measured according to the number of connection served by an Origin collected by the statistics.

Edge Discovery

When a new Edge server is added, the notification will be broadcasted to all Edges in the cluster. This is a mechanism provided by Terracotta JMX. The join-in of a new Edge is seamless.

Origin Discovery

Edge should join a Multicasting group on startup and this group should be a configuration shared by all Edges and Origins. When an Origin starts up, it should send a notification to this Multicasting group letting any of the Edges know its startup. The notification will be continuously sent until any of the Edge connects to it. The Edges share all Origin information by Terracotta so it makes sure that all Edges know this new Origin when any Edge connects.
The notification message contains the IP address of the Origin, the port of M RTMP, the startup time etc.

Edge Affinity

As the Origin has many Edge connections connected at the same time, the reply messages can be sent via any of the Edges because the response buffer is shared across Edges. The best choice for this scenario is the Edge that accepts the RTMPT request so that we don't need to copy the response message via Terracotta.
We solve this problem by recording the last Edge that sends the request within the M RTMP connection and send the response message to that Edge. The assumption is the load-balancer should be clever enough to choose the same Edge to serve the request for affinity consideration.

Fail-over

Edge Fail-over

The Edge failover works based on the support of load-balancer failover. When an Edge fails and detected by the load-balancer, the load-balancer should try to find another Edge and forward the request again to that Edge. As the connection state is shared on all Edge servers, all Edges are equivalent in terms of accepting the RTMPT request.
All Origins will be notified by the connection failure of Edge when the Edge is down. The Origin, if it is sending messages to the failing Edge, will resend the unsent messages to another Edge.
Each RTMPT packet has a sequence id attached with the request and the id will be maintained in the connection state on Edge cluster. The id will be updated when the request has been successfully processed (after the request has been forwarded to Origin and before the sending buffer is checked for the response message). This makes sure the request won't be lost. There might be cases when the request is processed twice and this will cause problems if the request is not idempotent and this is not covered by this solution.

Origin Fail-over

All the connections on that Origin are broken and the corresponding connections on Edge are released as a result of the Multiplexing connection broken notification.
The client is aware of the failure and is responsible for the reconnection.

Security

Prevent Faked Request

Each client will be assigned a client id which will be mapped to a connection object on the server. We need to prevent another client trying the same client id and communicating with the server.

1. Edge saves the client's IP when the connection is created and needs to check the client's IP for the following requests for the connection.
2. Sequence id of the RTMPT packet is checked and the packets with wrong RTMPT sequence id are dropped.
3. RTMPS is recommended if the security needs ensured.

Authentication

The client's authentication messages are sent by the parameters of the "connect" RTMP invocation and the actually authentication process depends on the application.

Performance Consideration

The Edge has a pending RTMP message buffer for each connection and it is a Terracotta shared object. As Terracotta is not suitable for sharing raw binary data like video or audio data, it is not efficient to populate another node with raw data when the load balancer routes the request to a different Edge than the original one. It is recommended by Terracotta team experts to retrieve the needed raw binary data when they are not available.
The Application Data Flow needs updating and M RTMP needs extension to support this.

Extension to Pending RTMP Message Buffer

The pending RTMP message buffer will not only contain the actual RTMP packet but also the meta RTMP packet containing where to get the actual RTMP packet and the index of the packet into the whole RTMP packet chain. When the RTMP packet is to be sent, the actual RTMP packet will be retrieved from the location indicated.

Enhancement to the Application Data Flow

For some RTMP packet like video or audio packet, Origin will not send the actual content to the Edge but the meta packet. Origin will maintain a local buffer to hold these packets giving each an index number. When the Edge needs sending the packet, it will send the request for the actual data packet and Origin will then retrieve the packet from local buffer and send it to Edge.
The Edge should notify the Origin about the packet sent so that the Origin can release the packet from local buffer.
To minimize the round trip between Edge and Origin, a local packet cache could be maintained on Edge. The Origin will not only send to the Edge the meta packets but the actual packets as well and the Edge will save these packets in the cache so that it can serve the client's request as soon as possible. If the cache misses, the Edge will request the packet from the Origin.

Extension to the M RTMP

Meta Packet (from Origin to Edge)

Packet Request (from Edge to Origin)

Actual Data Packet (from Origin to Edge)

Sent Notification (from Edge to Origin)