Live Streaming Video Integration in C# and ASP.NET Core

Live video stopped being a feature you bolt on at the end of a project. It is the product surface itself for virtual classrooms, telehealth visits, live commerce, sports co-viewing, remote inspections, creator broadcasts, and AI-assisted production tooling. When you build any of those on the Microsoft stack, you eventually hit the same wall: ASP.NET Core is an outstanding HTTP framework, but live video is not an HTTP problem. It is a timing, buffering, and bandwidth problem that happens to need an HTTP control plane.

This guide covers a practical architecture for live streaming in C# and ASP.NET Core: protocol selection, signaling, media server integration, GPU-backed AI processing, capacity control, observability, and the mistakes that quietly destroy stream quality. The examples stay framework-level rather than vendor-specific so they apply whether you run a single virtual machine, a Kubernetes cluster, or a managed video service.

Why Live Video Breaks the Usual ASP.NET Core Assumptions

A typical ASP.NET Core API handles discrete requests. A client asks, the server answers, the connection closes, memory is released, and the request lifetime ends. Live streaming violates nearly every assumption in that model:

A single viewer session can hold a socket open for hours.
A steady trickle of small control messages matters far more than raw request throughput.
Events such as mute, camera switch, or a moderator removing a participant must propagate in well under a second.
One slow or disconnected client must never degrade anyone else's experience.
State lives outside the request: who is in the room, who may publish, and what is being recorded.

Then there is transport physics. HTTP runs on TCP, and TCP retransmits lost packets. That is exactly the wrong behavior for video. By the time a retransmitted packet arrives, the frame it belonged to has already been shown or skipped, so the retransmission consumes bandwidth for no visible benefit. Real-time media prefers UDP-based transports, forward error correction, and a willingness to discard data rather than delay it.

That is why mature systems split responsibilities:

The control plane stays in ASP.NET Core: authentication, room membership, permissions, chat, recording metadata, scheduling, notifications, and AI job orchestration.
The data plane moves to a media server or a selective forwarding unit: contribution protocols, transcoding, WebRTC delivery, segment packaging, and edge fan-out.

Your design skill is knowing exactly where that boundary sits, because every byte that crosses it adds latency and cost.

A one-question sanity test for your boundary

Ask one question about every endpoint you plan to write: does this request or response carry media payload? If the answer is yes, it belongs in the data plane. Controllers should return tokens, manifests, room state, and metrics, never frames, segments, or packets. Teams that ignore this rule ship a prototype that works beautifully for five viewers and collapses at five hundred, and the rewrite costs more than the original build.

Choosing a Transport Layer for .NET Applications

Protocol choice is the single decision that constrains everything downstream. Changing it later usually means rewriting the client, the ingest path, and half of your media server configuration, so it is worth a day of deliberate thought before the first line of streaming code.

The four transport families

Ingest protocols: RTMP and SRT. RTMP is old, but it is universally supported by OBS and hardware encoders, and because it runs over TCP it tolerates shaky uplinks at the cost of latency. SRT adds packet recovery and handles jitter better, which makes it the stronger default when contributors stream from mobile networks, hotels, or congested venues.

WebRTC. Round trips well under two hundred milliseconds, with encryption, congestion control, and echo cancellation built in. This is the right choice for anything interactive: co-hosting, live auctions, one-to-one coaching, remote inspections with a technician on the line, and multiplayer co-viewing. The catch is scale. A selective forwarding unit fans streams out to viewers, and CPU plus egress bandwidth grow with every participant, so mesh topologies stop being viable after about three peers.

Low-latency HLS. Two to five seconds of delay, cacheable at the edge, and playable natively in Safari and most mobile browsers. This is the pragmatic default for large audiences that can tolerate a small delay.

Standard HLS and DASH. Five to thirty seconds of delay, inexpensive to deliver at very large scale, and the correct answer for replays, archives, and near-live content.

A decision framework

Requirement	Recommended transport	Why it fits
Two-way interaction under 300 ms	WebRTC through an SFU	Built-in congestion control; mesh fails beyond a few peers
Thousands of concurrent viewers, 3 to 5 seconds of delay	Low-latency HLS behind a CDN	Cacheable segments absorb spikes without scaling origin
Publishing from OBS or a hardware encoder	RTMP, or SRT for unstable uplinks	Universal encoder support with optional packet recovery
Smart TVs, older browsers, social simulcast	HLS	Widest compatibility, highest latency
Server-side AI analysis of the live feed	Any, tapped server-side	Analyze media server output instead of the browser

Mixing transports is normal, not exotic

Serious platforms rarely commit to one transport. A typical layout looks like this: contributors publish over SRT; the media server transcodes and packages; an SFU delivers the interactive room over WebRTC; a parallel low-latency HLS rendition feeds passive viewers and social simulcasts; and the same packaged output feeds a recorder for later replay. Designing two egress paths from day one saves a painful migration, because the recording and the AI tap can both read from media server output rather than depending on a browser session that may close at any moment.

Designing the Control Plane in ASP.NET Core

The control plane is where most of your code lives, and it should be deliberately dull. Dull means predictable, testable, and cheap to operate under load.

Keep the API boring

Controllers and minimal API endpoints should do four things: authenticate, authorize, read or write room state, and enqueue work. Endpoints stay fast because nothing inside them touches a socket that could stall. A request that starts a stream should return a signed publish token and a room descriptor, not wait for the media server to confirm that publishing began. Confirmation arrives later through a webhook or a reconciler, and the user interface reflects it through the signaling channel.

Typed options and validated startup

Streaming services accumulate configuration: media server endpoints, ingest hostnames, token lifetimes, bitrate ladders, queue names, worker concurrency, and webhook secrets. Bind that configuration to typed option classes and validate it during startup, so a missing hostname fails the deployment instead of failing a live event. Validation should cover ranges as well as presence. A token lifetime of zero seconds or a bitrate ladder missing its bottom rung produces symptoms that are genuinely hard to trace later.

Background services for work that outlives a request

Anything that survives longer than a request belongs in a hosted background service. Four are almost always worth writing:

A stream health monitor that polls media server health endpoints every few seconds, marks rooms unhealthy, and alerts when a publisher stops sending data without closing cleanly.
A recording finalizer that reacts to recording-available events, moves files to object storage, writes metadata, and generates a playback manifest.
A job retry sweeper that requeues failed AI jobs with exponential backoff and a hard attempt ceiling.
A room reaper that closes rooms idle beyond a threshold and revokes their publish tokens.

Register each as a singleton hosted service, make each iteration idempotent, give each a bounded time budget, and log structured events with a correlation identifier so you can follow one room's story across processes.

Signaling and Room State with SignalR

SignalR is the natural signaling channel in .NET. It handles WebSocket negotiation, fallback transports, reconnection, and group semantics, which is precisely the surface a WebRTC handshake needs. It is also a convenient place to push AI job progress, chat, and presence events over a connection you already pay for.

A small, intention-revealing hub surface

Keep the hub tiny. Its vocabulary should be offer, answer, and network candidate exchange, plus room events such as join, leave, and role change. A handful of methods covers nearly everything:

A connection-ready message that returns the connection identifier so the client can address itself.
A join-room call that validates the caller against the room registry and adds the connection to a group.
A relay call that forwards a signaling payload to one specific connection.
A disconnect handler that removes the connection from the registry and notifies the room.

Resist the temptation to push media or large binary payloads through the hub. Control data only. If a message would be larger than a few kilobytes, it probably belongs in object storage or in the data plane.

Authentication and connection mapping

Authenticate the WebSocket handshake with the same token middleware that protects your REST endpoints, then read the user principal inside the hub. Maintain a server-side map from connection identifier to user, room, and role. That map is the source of truth for who may publish, who may only watch, and who must be evicted when a moderator acts. Because the map is authoritative, it also becomes the mechanism for idempotent joins: a duplicate join request updates the existing entry rather than creating a second one.

Scaling out without losing messages

A single-process hub does not survive horizontal scaling. Add a backplane such as Redis or a managed real-time service so group messages reach connections on every instance. Keep hub methods non-blocking: if a method needs to call a model, transcode a clip, or wait on the media server, hand the work to a queue and acknowledge immediately. Design clients to treat disconnects as routine, so a reconnect renegotiates the session rather than reloading the page.

Treat reconnection as a first-class flow

Mobile networks, laptop sleep, and tab backgrounding all cause disconnects that have nothing to do with failure. The client should attempt a staged rejoin: reconnect the socket, re-authenticate if the token expired, rejoin the room group, and renegotiate media with a fresh offer. On the server side, the registry should hold a short grace window before evicting a connection, so a two-second blip does not remove a panelist from the stage.

Integrating a Media Server as a Managed Dependency

Most teams integrate a media server rather than build one, which turns the problem into a control-plane problem with four recurring elements.

Idempotent commands

Every action, whether it starts a stream, stops a stream, changes a layout, or removes a publisher, should carry a client-generated request identifier. Store recent identifiers with a short expiry and short-circuit duplicates. Without this, a retried HTTP call creates a second room, and now two live sessions share a name, both consume GPU time, and your support queue grows while nobody can explain why.

Reconciliation instead of fire-and-forget

Do not assume a command succeeded because the HTTP response looked healthy. Run a periodic reconciler that compares your database's view of active rooms against the media server's actual state and repairs drift in both directions: close orphaned streams that waste CPU, and surface sessions the media server reports but your database does not know about. This single loop prevents the most common outage class in video platforms, the ghost room that holds participants while nobody can see them.

Webhooks, signatures, and secret hygiene

Media servers notify you when publishing starts, when it stops, when a recording becomes available, and when something fails. Verify the signature, deduplicate by event identifier, and process asynchronously. Webhook handlers should be fast and boring, and they should never transcode inline. Stream keys and API secrets belong in a secret store rather than source control, and rotation should be a scheduled routine rather than a reaction to an incident. Use a typed HTTP client with retry, jitter, and circuit breakers for outbound calls so a slow media server degrades gracefully instead of exhausting your thread pool.

Tapping the Live Feed for AI Without Adding Latency

This is where live streaming stops being purely a transport exercise. Live transcription, automatic highlight clipping, real-time moderation, scene detection, synthetic voice-over, and replay chaptering are all asynchronous by nature, and none of them may block the stream.

Model the job, not the call

Never call a model inline in a controller or a hub method. Persist a job record containing an identifier, type, target asset or time range, parameters, priority, status, attempt count, and timestamps. The API enqueues and returns immediately. A worker picks the job up, updates status, and publishes progress. Status transitions should be explicit and finite: queued, running, succeeded, failed, cancelled, expired.

Queues, deadlines, and cancellation

Use a durable queue: Redis streams, RabbitMQ, a cloud service bus, or a database-backed queue with a skip-locked claim. Give each job an idempotency key derived from its asset and parameters so duplicate submissions collapse into a single execution. Workers must be safe to restart at any moment, because they will be terminated mid-task eventually.

Every live-adjacent job should carry a deadline. A transcript delivered ten minutes after the segment ended is worse than no transcript: it consumes GPU time, confuses the timeline, and clutters the interface. Check cancellation tokens between pipeline stages, and cancel downstream work as soon as a newer job supersedes an older one.

Advisory AI instead of autonomous AI

AI that proposes camera framing, drafts lower-third graphics, or flags a likely highlight works best as an advisory system. Emit structured proposals into the room's event stream and let a human or a deterministic rule decide what goes on air. Latency stays predictable, and a model hallucination cannot become a broadcast incident. The same pattern applies to moderation: the model flags, a policy engine decides, and a human reviews appeals.

Where to tap the stream

Tap server-side, from media server output, for three reasons. It does not depend on client device performance, it adds no upload load for contributors, and recordings can be reprocessed later with better models without touching the original broadcast. Browser-side analysis remains useful for local effects such as background blur, but it should never be the source of truth for captions, moderation, or analytics.

GPU Capacity, Cost Control, and Fair Scheduling

GPU-backed inference and transcoding are the two workloads that inflate a bill fastest. Both need admission control.

Cap concurrency explicitly. Decide how many transcodes and how many model inferences may run at once per node, then enforce the limit with a semaphore or a weighted queue. Unbounded parallelism does not raise throughput; it raises latency, queue wait, and out-of-memory crashes.

Warm pools beat cold starts. Loading a model can take longer than the inference itself. Keep a small pool of preloaded workers sized to steady-state demand and let cold starts happen only during genuine surges.

Separate interactive and batch lanes. Captions for an ongoing stream must preempt highlight rendering for a broadcast that ended an hour ago. Two queues with different priorities handle that reliably; one queue with clever scheduling usually does not.

Measure utilization, not allocation. Track memory usage, utilization percentage, and queue wait time per job type. If wait time is consistently near zero you are over-provisioned. If the queue grows while utilization plateaus well below saturation, the bottleneck is elsewhere, often a single-threaded preprocessing step or a shared object store.

Set quotas per account. One tenant submitting thousands of clip jobs can starve everyone else. Per-account quotas with clear error messages protect the shared system far better than reactive throttling applied after complaints arrive.

Route interruptible work to cheaper capacity. Highlight rendering and replay transcription tolerate interruption; live captions do not. Matching workload to capacity class lowers cost without touching the interactive path.

Observability, Testing, and Failure Drills

Metrics that predict user complaints

Time to first frame, the best single proxy for perceived startup quality.
Glass-to-glass latency, measured with a timestamp overlay rather than estimated from logs.
Rebuffer ratio, calculated as reconnections divided by total playback time.
Join failure rate, segmented by region and device class.
Publish stability, covering dropped frames and bitrate collapse on the contributor side.
Job latency and queue depth per job type.

Emit all of it with OpenTelemetry so a trace connects one viewer's playback session to the signaling messages and API calls behind it. Dashboards should make the boundary between control plane and data plane visually obvious, because that is where most incidents originate.

Testing strategy that catches real failures

Unit tests cover token signing, authorization rules, and state machines. Integration tests run a short-lived media server in a container and push a synthetic clip through ingest, transcode, and packaging. Load tests simulate both sides of the platform: hundreds of publish sessions and thousands of viewers, driven by a load tool or scripted browsers. Contract tests pin the webhook payload shapes your reconciler depends on, so a media server upgrade cannot silently break state repair.

Practice failure on purpose

Kill the media server mid-stream. Drop the backplane and watch how clients recover. Throttle a connection to mobile speeds and confirm the bitrate ladder behaves. Revoke a publish token while a contributor is live. Recovery behavior is a feature: either you test it deliberately, or users discover it for you during your busiest hour.

Mistakes That Quietly Ruin Live Streams

Running media through the application. It survives staging and collapses with a real audience.
Leaving ingest unauthenticated. An open publish endpoint on the public internet will be discovered and abused within days.
Fire-and-forget media server commands. Drift accumulates until rooms stop responding.
One queue for everything. Batch work starves interactive features.
No bitrate ladder. Viewers on weak connections buffer constantly while you pay for high-quality egress nobody can decode smoothly.
Ignoring mobile browsers. Autoplay policies, background throttling, and codec support differ sharply from desktop.
Treating disconnects as errors. Reconnection is normal; design the client to renegotiate rather than reload.
Storing room truth inside the hub. In-memory hubs lose state on scale-out and on restart.
No recording finalizer. Streams end and recordings sit on a media server that later gets recycled.
Estimating latency instead of measuring it. Opinion is not telemetry.

FAQ

Can ASP.NET Core serve video directly?
It can serve static segments over HTTP, which is acceptable for HLS or DASH delivery at modest scale. It should not terminate WebRTC sessions or transcode. Use a media server or an SFU for that and keep your application on the control plane.

Do I need WebRTC if HLS already works?
Only if interaction matters. HLS is fine for one-way broadcast with a few seconds of delay. The moment two people must react to each other, in auctions, interviews, tutoring, or co-hosting, the latency difference becomes the deciding factor.

How do I scale SignalR for thousands of rooms?
Add a backplane or a managed real-time service, keep hub methods non-blocking, and store room state outside the hub in a registry backed by Redis or a database. Hubs should route messages, not hold the truth.

Where should AI processing tap into the stream?
Server-side, from the media server output. Browser-side analysis is fragile, device-dependent, and adds load to the client. A server-side tap also lets you reprocess recordings later with better models.

How do I keep GPU spending under control?
Cap concurrency, size warm pools to steady-state demand, separate interactive and batch lanes, and apply per-account quotas. Review utilization weekly, because most overspend comes from idle warm capacity provisioned for a peak that never arrives.

What is the smallest architecture that can support a real first release?
An ASP.NET Core API for authentication, rooms, and orchestration; SignalR for signaling and events; one media server instance for ingest and transcoding; low-latency HLS through a CDN for viewers; and a background worker for asynchronous jobs. That combination supports a genuine audience and leaves clean seams for scaling each layer independently.

How should I handle recording and replay?
Have the media server publish recording events, run a finalizer worker that moves files to object storage, and generate chapter metadata from the AI transcript so replays are searchable and skimmable instead of a single long timeline.

Do I need Kubernetes for this?
No. A single well-monitored virtual machine with a media server and an API can carry a surprisingly large audience. Container orchestration helps with rolling upgrades, worker autoscaling, and isolating GPU nodes, so adopt it when operational needs justify the complexity rather than in advance.

Build the control plane to be boring and the media layer to be replaceable, and everything else, including AI features, scale, and cost control, becomes an engineering problem you can solve incrementally instead of an architectural rewrite.