Load Balancing & Worker Nodes

Overview

xOTT load balancing lets you add worker servers that run stream workloads away from the main application server. The main server owns the panel, database, service accounts, provider refreshes, orchestration, monitoring, licensing, and stable customer-facing output URLs.

This is true workload distribution, not only traffic forwarding. Eligible streams can be started on worker nodes so CPU, memory, stream process count, and network pressure are spread across the available infrastructure.

Architecture

The load balancing system uses a controller-worker model:

• Main server: Runs the dashboard, API, database, account and provider refresh logic, scheduler, placement engine, entry HLS routes, and license checks.
• Media origins: Run assigned streams, store HLS output, serve viewer playlists and segments, report health, and accept signed controller commands.
• Placement engine: Chooses where a stream should run based on global settings, service defaults, stream overrides, node health, limits, and strategy.
• Output layer: Keeps configured entry URLs stable while routing media directly to an assigned origin. Direct mode fails closed by default when an origin is unavailable; controller relay is an explicit compatibility option.

Requirements

Prepare each worker server before adding it to xOTT:

• Ubuntu server with root SSH access for the initial installation.
• Enough CPU, RAM, disk, and bandwidth for the stream workload you plan to assign.
• Network access from the main xOTT server to the worker API port.
• Network access from the worker back to the main xOTT server for heartbeats and controller communication.
• For Direct origin mode, either a public HTTPS worker origin (recommended) or the explicit Direct IP HTTP option with the worker port reachable by viewers.
• Firewall rules that allow only the required controller-to-worker traffic whenever possible.

Direct Origin Delivery

Direct origin keeps the playlist URL configured in players on the main xOTT server. The main server resolves the current assignment and returns a short-lived signed redirect; playlist and segment bandwidth then comes from the assigned worker.

• Worker domain and HTTPS: Recommended for public deployments and the broadest player, browser, ISP, and reverse-proxy compatibility.
• Direct IP media over HTTP: An explicit per-worker option for operators who cannot configure DNS or certificates. It uses the worker's literal public IP and worker port, opens that port when UFW is active, and never enables itself implicitly.
• Controller relay: Keeps all media flowing through the main server and therefore uses main-server bandwidth.

Add Worker

Worker installation is handled from the xOTT Server page. Add the server IP, SSH port, root username, and root password, then start the installer. xOTT connects to the server, installs the worker runtime, writes the node configuration, registers the worker, and generates a signed node token for secure command authentication.

Recommended first check

1. Add the worker from the Server page.
2. Wait for the first heartbeat to appear.
3. Click the health check icon for the node.
4. Assign a single non-critical stream to the worker.
5. Confirm the public HLS URL still uses the main server address.

Health & Heartbeats

Workers send regular heartbeat data to the main server. The controller uses these reports to decide whether a node is eligible for new stream assignments.

• Heartbeat interval: Workers report health frequently, usually around every 15 seconds.
• Stale threshold: A node that stops reporting for roughly 45 seconds is treated as unavailable for placement.
• Metrics: Host-normalized CPU, memory, stream count, ingress, egress, active viewers, node status, and last heartbeat time are used by placement.
• Health check: The Server page health check sends a signed command to confirm the worker API can be reached and authenticated.

Placement Policies

Stream placement is configurable at multiple levels. xOTT resolves the final assignment using this precedence:

1. Stream-specific override
2. Service account or service default
3. Global load balancing policy

Placement modes

• Auto / Inherit: Use the next applicable parent policy.
• Main server only: Force the stream to run on the main xOTT server.
• Any worker: Run the stream on any eligible healthy worker.
• Specific worker: Pin the stream to one worker node.
• Worker group: Restrict placement to nodes in selected groups.

Strategies

• Lowest load: Prefer the eligible node with the lowest current resource pressure and stream count.
• Round robin: Rotate assignments across eligible nodes for simple distribution.

Fallback behavior

• Main server fallback: If no worker is eligible, run the stream on the main server.
• Fail if unavailable: If the required worker or group is not available, do not start the stream.

Node Settings

Worker settings let you control capacity and routing without hardcoding service-specific behavior into the engine.

• Groups: Assign nodes to groups such as us, eu, directv, or vix for policy targeting.
• Max streams: Stop new assignments once the node reaches its stream process limit.
• Max CPU: Prevent new assignments when CPU pressure exceeds the configured limit.
• Max memory: Prevent new assignments when memory usage exceeds the configured limit.
• Max ingress and egress: Reserve network capacity for pending starts and stop new assignments before measured plus reserved traffic reaches the limit.
• Weight: Give stronger nodes more placement preference where supported by the strategy.
• Drain mode: Keep existing streams running but stop sending new starts to the node during maintenance.

Stream Settings

Individual stream settings are the most precise control point. Use them when a channel needs a specific node, a strict region, or a controlled fallback policy that should not affect the rest of the service.

• Use inherit for normal streams so global and service rules can keep balancing automatically.
• Use specific worker for streams that need a fixed host due to proxy, account, or regional constraints.
• Use worker group when several equivalent nodes can run the stream.
• Use fail if unavailable only when running on the wrong node is worse than not starting the stream.

Service Defaults

Service-level defaults are useful when a provider, account group, or region should use a predictable set of worker nodes. For example, you can keep a service on a regional worker group while still letting individual streams override that behavior when needed.

This keeps the engine universal: the load balancer does not need custom code per provider. Service behavior is expressed through policies, groups, limits, and fallback settings.

Operations

Adding capacity

Add a new worker, wait for heartbeat health, assign groups and limits, then let the placement strategy begin sending eligible streams to the node.

Maintenance

Enable drain mode before planned maintenance. The node stops receiving new placements. When evacuation is requested, assigned streams are fenced, stopped, and reconstructed on eligible nodes before maintenance proceeds.

Scaling guidance

• Start with conservative max stream, CPU, and memory limits.
• Set measured ingress and egress limits below the provider or NIC ceiling.
• Network metrics use the default-route interface to avoid double-counting bridge and container traffic. Set XOTT_METRICS_NETWORK_INTERFACES on workers with dedicated media interfaces.
• Use groups for geography, service families, or hardware classes.
• Keep main server fallback enabled while validating a new worker.
• Review failed stream starts for node eligibility problems before changing service scripts.

Troubleshooting

Worker does not heartbeat

• Confirm the worker service is running on the node.
• Confirm the controller URL in the worker environment points to the main xOTT server.
• Check outbound firewall access from the worker to the main server.
• Verify the node token is present in the worker environment.

Health check fails but heartbeat works

• Confirm the main server can reach the worker API host and port.
• Check any firewall rules between the controller and worker.
• Verify the worker API is using the current signed command token.

Streams still run on the main server

• Check whether the global, service, or stream policy is set to main server only.
• Confirm at least one worker is healthy and not in drain mode.
• Review max stream, CPU, and memory limits for the candidate workers.
• Check whether fallback to main server is enabled because no worker was eligible.

Output URL shows the main server

This is expected for the entry URL. In Direct origin mode, inspect the first response for a 307 redirect to the assigned HTTPS origin or explicitly enabled Direct IP HTTP origin. If no redirect is returned, verify the node public URL, selected delivery option, firewall, protocol version, and origin capability. For HTTPS origins, also verify the TLS certificate. Direct mode fails closed by default so sustained media traffic cannot silently move back to the controller. Controller relay is used only when explicitly selected under Direct Failure.

On-demand stream stops while viewers are active

• Confirm the worker is sending current heartbeats and advertises origin assets.
• Confirm the stream assignment generation matches the generation running on the worker.
• Verify playlist and segment requests reach the worker origin after the initial redirect.
• Update older workers so viewer activity is reported to the controller with heartbeats.

Security

Worker nodes use controller-issued node tokens, signed commands, generation fencing, short-lived media tickets, and replay protection. Keep command APIs restricted to the main xOTT server. Public access is required only for the signed origin media route.

• Keep the raw worker API port restricted to the controller unless Direct IP HTTP is explicitly enabled. In that mode, worker-role route isolation and signed command or media authentication remain mandatory.
• Rotate or reinstall a worker if its token may have been exposed.
• Use HTTPS for controller and origin URLs so stream credentials and refresh responses are encrypted in transit.
• Pin SSH host fingerprints and verify the installer SHA-256 in production.
• Use the Server page installer instead of manually copying credentials between systems.
• Keep the main server license active and assign a separate active xOTT Worker license to every worker node. Workers are not standalone panels.