webrtc 视频

George shut his laptop and rubbed his sleep-deprived red eyes. "Customers continue to complain about stream freezing; the new fix package did not help at all! What do I do with this (censored) HLS?" he said.

乔治关上笔记本电脑，抚摸着睡眠不足的红眼睛。 “客户继续抱怨流冻结；新的修复程序包根本没有帮助！我如何处理这个(经过审查的)HLS？” 他说。

浏览器不仅是超文本，而且是流媒体 (The browser is not only hypertext, but also a streamer)

Browsers have had players for a long time, but the story is different with the video encoder and streaming. Now, in almost any browser of the latest version, we can find modules for encoding, streaming, decoding, and playback. These functions are available through the JavaScript API, and the implementation is called Web Real Time Communications or WebRTC. This library built into browsers can do quite a lot: capture video from a built-in, virtual or USB camera, compress it with H.264, VP8, and VP9 codecs, and send it to the network via SRTP protocol; i.e., it functions as a software streamer video encoder. As a result, we see a browser that has something similar to ffmpeg or gstreamer, compresses video well, streams on RTP, and plays video streams.

浏览器拥有播放器已有很长时间了，但是故事却与视频编码器和流媒体有所不同。 现在，在几乎所有最新版本的浏览器中，我们都可以找到用于编码，流传输，解码和回放的模块。 这些功能可通过JavaScript API获得，其实现称为Web实时通信或WebRTC。 这个内置在浏览器中的库可以做很多事情：从内置的虚拟或USB摄像机捕获视频，用H.264，VP8和VP9编解码器压缩，然后通过SRTP协议将其发送到网络； 即，它用作软件流视频编码器。 结果，我们看到了一个浏览器，它具有与ffmpeg或gstreamer类似的东西，可以很好地压缩视频，在RTP上播放流并播放视频流。

WebRTC gives us the freedom to implement a variety of streaming cases in JavaScript:

WebRTC使我们可以自由地在JavaScript中实现各种流情况：

• stream from the browser to the server for recording and subsequent distribution从浏览器流到服务器以进行记录和后续分发
• distribute peer-to-peer streams分发对等流
• play another user’s stream and send one’s own (video chat)播放其他用户的视频流并发送自己的视频流(视频聊天)
• convert other protocols by the server, for example RTMP, RTSP, etc., and play them in the browser as WebRTC服务器转换其他协议，例如RTMP，RTSP等，并以WebRTC的形式在浏览器中播放

Refined flow control scripts may look like this:

完善的流控制脚本可能如下所示：

//Launching broadcast from browser to serversession.createStream({name:”mystream”}).publish();//Playing broadcast by the browsersession.createStream({name:”mystream”}).play();

HLS在WebRTC不起作用的地方工作 (HLS works where WebRTC does not work)

WebRTC runs in the latest versions of browsers, however, there are the following two factors: 1) Not all users update their browsers in a timely manner and may well use Chrome’s old version for three years. 2) Updates and new browsers, WebView, as well as other clients and instant messengers helping users to surf the Internet are released almost once a week. Needless to say, not all of them have WebRTC support, and if they do, it can be limited. See how things are now:

WebRTC在最新版本的浏览器中运行，但是，有以下两个因素：1)并非所有用户都及时更新其浏览器，并且可能会使用Chrome的旧版本三年。 2)更新和新的浏览器，WebView以及帮助用户上网的其他客户端和即时通讯程序几乎每周发布一次。 不用说，并不是所有人都支持WebRTC，如果支持的话，它可能会受到限制。 看看现在情况如何：

Everyone's favorite devices by Apple can be a headache. They have begun to support WebRTC only recently and at times, their behavior compared to webkit browsers may seem surprising. Where WebRTC does not work or works not very well, HLS works fine. In this regard, compatibility is required, and something like a converter that allows us to convert WebRTC to HLS and play it on almost any device.

苹果公司最喜欢的设备可能让人头疼。 他们只是在最近才开始支持WebRTC，有时，与Webkit浏览器相比，它们的行为似乎令人惊讶。 如果WebRTC无法正常运行或运行不正常，则HLS可以正常运行。 在这方面，需要兼容性，并且需要类似转换器的工具，该转换器使我们可以将WebRTC转换为HLS并在几乎所有设备上播放。

HLS was not originally conceived for real-time streams. Indeed, how can we stream real-time video via HTTP? The task of HLS is to cut the video into pieces and deliver them to the player smoothly, without rushing, by downloading them one by one. A HLS player expects a strictly formed and smooth video stream. Here we have a conflict, since WebRTC, on the contrary, can afford to lose packets due to real-time requirements and low latency and have a floating FPS/GOP and a variable bit rate — be the exact opposite of HLS in terms of predictability and regularity of the stream.

HLS最初不是为实时流而设计的。 确实，我们如何通过HTTP流实时视频？ HLS的任务是将视频切成小块，并通过逐个下载将它们平稳地传递到播放器，而不会着急。 HLS播放器要求严格形成且流畅的视频流。 在这里，我们存在冲突，因为相反，WebRTC会由于实时要求和低延迟而承受丢包的风险，并且具有浮动FPS / GOP和可变比特率-就可预测性而言，这与HLS完全相反流的规律性。

An obvious approach — WebRTC depacketization (SRTP) and subsequent conversion to HLS may not work in a native Apple HLS player or work with freezing, which is a form unsuitable for production. The native player means a player that is used in Apple iOS Safari, Mac OS Safari, and Apple TV.

一个明显的方法-WebRTC拆包(SRTP)和随后的转换为HLS可能无法在本机Apple HLS播放器中使用，也不能与冻结配合使用，这是不适合生产的形式。 本机播放器是指Apple iOS Safari，Mac OS Safari和Apple TV中使用的播放器。

Therefore, if you notice the HLS freezing in the native player, maybe this is the case, and the source of the stream is WebRTC or another dynamic stream with uneven markup. In addition, in the implementation of the native Apple players, there is behavior that can only be understood empirically. For example, the server should start sending HLS segments immediately after the m3u8 playlist is returned. A 1-second delay may result in freezing. If the bitstream configuration changed in the process (which is fairly common during WebRTC streaming), there will also be freezing.

因此，如果您发现HLS在本机播放器中冻结，则可能是这种情况，并且流的源是WebRTC或标记不均匀的另一个动态流。 此外，在实施本地Apple播放器时，有些行为只能凭经验来理解。 例如，服务器应在返回m3u8播放列表后立即开始发送HLS段。 1秒的延迟可能会导致冻结。 如果在此过程中更改了比特流配置(这在WebRTC流式传输中很常见)，那么也会冻结。

与本土玩家抗争 (Fighting freezing in native players)

Thus, WebRTC depacketization and HLS packetization generally do not work. In the Web Call Server (WCS) streaming video server, we solve the problem in two ways, and we offer the third as an alternative:

因此，WebRTC拆包和HLS打包通常不起作用。 在Web呼叫服务器(WCS)流视频服务器中，我们以两种方式解决该问题，并且我们提供了第三种方法：

1) Transcoding.

1)转码。

This is the most reliable way to align a WebRTC stream to HLS requirements, set the desired GOP, FPS, etc. However, in some cases, transcoding is not a good solution; for example, transcoding 4k streams of VR video is indeed a bad idea. Such weighty streams are very expensive to transcode in terms of CPU time or GPU resources.

这是使WebRTC流符合HLS要求，设置所需的GOP，FPS等的最可靠方法。但是，在某些情况下，转码不是一个好的解决方案。 例如，对4k的VR视频流进行转码确实是个坏主意。 就CPU时间或GPU资源而言，这样的重量级流进行代码转换非常昂贵。

2) Adapting and aligning WebRTC flow on the go to match HLS requirements.

2)随时随地调整和调整WebRTC流程以符合HLS要求。

These are special parsers that analyze H.264 bitstream and adjust it to match the features/bugs of Apple’s native HLS players. Admittedly, non-native players like video.js and hls.js are more tolerant of streams with a dynamic bitrate and FPS running on WebRTC and do not slow down where the reference implementation of Apple HLS essentially results in freezing.

这些是特殊的解析器，用于分析H.264比特流并对其进行调整以匹配Apple本地HLS播放器的功能/错误。 不可否认，非本地播放器(例如video.js和hls.js)更能容忍在WebRTC上运行的具有动态比特率和FPS的流，并且不会放慢Apple HLS的参考实现导致冻结的速度。

3) Using RTMP as the stream source instead of WebRTC.

3)使用RTMP代替WebRTC作为流源。

Despite the fact that Flash player is already obsolete, the RTMP protocol is actively used for streaming; take OBS Studio, for example. We must acknowledge that RTMP encoders produce generally more even streams than WebRTC and therefore practically do not cause freezing in HLS, i.e. RTMP>HLS conversion looks much more suitable in terms of freezing, including in native HLS players. Therefore, if streaming is done using the desktop and OBS, then it is better to use it for conversion to HLS. If the source is the Chrome browser, then RTMP cannot be used without installing plugins, and only WebRTC works in this case.

尽管Flash Player已经过时，但RTMP协议已被积极地用于流式传输。 以OBS Studio为例。 我们必须承认，RTMP编码器产生的流通常比WebRTC还要均匀，因此实际上不会导致HLS冻结，即RTMP> HLS转换在冻结方面看起来更合适，包括在本机HLS播放器中。 因此，如果使用台式机和OBS完成流传输，则最好将其用于转换为HLS。 如果源是Chrome浏览器，则必须先安装插件才能使用RTMP，除非安装插件。

All three methods described above have been tested and work, so you can choose based on the task.

上面介绍的所有三种方法都已经过测试和工作，因此您可以根据任务选择。

WebRTC到CDN上的HLS (WebRTC to HLS on CDN)

There are some undesirables you're going to run into in a distributed system when there are several WebRTC stream delivery servers between the WebRTC stream source and the HLS player, namely CDN, in our case, based on a WCS server. It looks like this: There is Origin — a server that accepts WebRTC stream, and there is Edge — servers that distribute this stream including via HLS. There can be many servers, which enables horizontal scaling of the system. For example, 1000 HLS servers can be connected to one Origin server; in this case, system capacity scales 1000 times.

当WebRTC流源和HLS播放器之间有多个WebRTC流传递服务器时，在分布式系统中会遇到一些不良情况，在本例中， CDN基于WCS服务器。 看起来像这样：有一个Origin —一个接受WebRTC流的服务器，还有一个Edge —包括通过HLS分发此流的服务器。 可以有许多服务器，这可以实现系统的水平扩展。 例如，一台原始服务器可以连接1000个HLS服务器； 在这种情况下，系统容量可扩展1000倍。

The problem has already been highlighter above; it usually arises in native players: iOS Safari, Mac OS Safari, and Apple TV. By native we mean a player that works with a direct indication of the playlist url in the tag, for example <video src="https://host/test.m3u8"/>. As soon as the player requested a playlist – and this action is actually the first step in playing the HLS stream – the server must immediately, without delay, begin to send out HLS video segments. If the server does not start to send segments immediately, the player will decide that it has been cheated and stop playing. This behavior is typical of Apple’s native HLS players, but we can’t just tell users “please do not use iPhone Mac и Apple TV to play HLS streams.”

问题已经在上面突出显示了； 它通常出现在本机播放器中：iOS Safari，Mac OS Safari和Apple TV。 原生是指可以直接指示标签中播放列表网址的播放器，例如<video src="https://host/test.m3u8"/> 。 播放器请求播放列表后(实际上这是播放HLS流的第一步)，服务器必须立即无延迟地开始发送HLS视频片段。 如果服务器没有立即开始发送片段，则播放器将判定它已被欺骗并停止播放。 这种行为是Apple本地HLS播放器的典型现象，但是我们不能仅仅告诉用户“请不要使用iPhone Mac®Apple TV播放HLS流。”

So, when you try to play a HLS stream on the Edge server, the server should immediately start returning segments, but how is it supposed do it if it doesn’t have a stream? Indeed, when you try to play it, there is no stream on this server. CDN logic works on the principle of Lazy Loading – it won’t load the stream to the server until someone requests this stream on this server. There is a problem of the first connected user; the first one who requested the HLS stream from the Edge server and had the imprudence to do this from the default Apple player will get freezing for the reason that it will take some time to order this stream from the Origin server, get it on Edge, and begin HLS slicing. Even if it takes three seconds, this will not help. It will freeze.

因此，当您尝试在Edge服务器上播放HLS流时，服务器应立即开始返回段，但是如果没有流，应该怎么做呢？ 确实，当您尝试播放它时，此服务器上没有流。 CDN逻辑根据延迟加载的原理工作–除非有人在此服务器上请求此流，否则它不会将流加载到服务器。 第一个连接的用户有问题； 第一个从Edge服务器请求HLS流并不愿从默认Apple播放器执行此操作的人将被冻结，原因是从原始服务器订购该流并将其存储在Edge上需要一些时间，并开始HLS切片。 即使需要三秒钟，这也无济于事。 它将冻结。

Here we have two possible solutions: one is OK, and the other is less so. One could abandon the Lazy Loading approach in the CDN and send traffic to all nodes, regardless of whether there have viewers or not. A solution, possibly suitable for those who are not limited in traffic and computing resources. Origin will send traffic to all Edge servers, as a result of which, all servers and the network between them will be constantly loaded. Perhaps this scheme would be suitable only for some specific solutions with a small number of incoming flows. When replicating a large number of streams, such a scheme will be clearly inefficient in terms of resources. And if you recall that we are only solving the “problem of the first connected user from the native browser,” then it becomes clear that it is not worth it.

在这里，我们有两种可能的解决方案：一种可以，另一种则可以。 有人可以放弃CDN中的“延迟加载”方法，而将流量发送到所有节点，而不管是否有查看器。 一种解决方案，可能适用于流量和计算资源不受限制的用户。 源将向所有边缘服务器发送流量，因此，所有服务器和它们之间的网络将不断加载。 也许该方案仅适用于少量输入流的某些特定解决方案。 当复制大量流时，这种方案在资源方面显然效率低下。 而且，如果您还记得我们只是在解决“本机浏览器中第一个连接的用户的问题”，那么很明显，这是不值得的。

The second option is more elegant, but it is also merely an end-around. We give the first connected user a video picture, but this is still not the stream that they want to see – this is a preloader. Since we must give them something already and do it immediately, but we don’t have the source stream (it is still being ordered and delivered from Origin), we decide to ask the client to wait a bit and show them a video of the preloader with moving animation. The user waits a few seconds while the preloader spins, and when the real stream finally comes, the user starts getting the real stream. As a result, the first user will see the preloader, and those who connect after that will finally see the regular HLS stream coming from the CDN operating on the principle of Lazy Loading. Thus, the engineering problem has been solved.

第二种选择更加优雅，但这也只是一个终结。 我们为第一个连接的用户提供了一个视频图片，但这仍然不是他们想要看到的流-这是一个预加载器。 由于我们必须已经给他们一些东西并立即进行操作，但是我们没有源流(仍在从Origin订购并交付源流)，因此，我们决定请客户端稍等一下，然后向他们展示有关带有动画的预加载器。 当预加载器旋转时，用户等待几秒钟，然后当实际流最终到来时，用户开始获取实际流。 结果，第一个用户将看到预加载器，之后的用户将最终看到来自CDN的常规HLS流，该CDN按照延迟加载的原理进行操作。 因此，解决了工程问题。

但尚未完全解决 (But not yet fully solved)

It would seem that everything works well. The CDN is functioning, the HLS streams are loaded from the Edge servers, and the issue of the first connected user is solved. And here is another pitfall – we give the preloader in a fixed aspect ratio of 16:9, while streams of any formats can enter the CDN: 16:9, 4:3, 2:1 (VR video). And this is a problem, because if you send a preloader in 16:9 format to the player, and the ordered stream is 4:3, then the native player will once again face freezing.

似乎一切正常。 CDN正在运行，从边缘服务器加载了HLS流，并且解决了第一个连接的用户的问题。 这是另一个陷阱–我们给预加载器以16：9的固定宽高比，而任何格式的流都可以输入CDN：16：9、4：3、2：1(VR视频)。 这是一个问题，因为如果您将16：9格式的预加载器发送给播放器，并且有序流为4：3，则本机播放器将再次面临冻结。

Therefore, a new task arises – you need to know with what aspect ratio the stream enters the CDN and give the same ratio to the preloader. A feature of WebRTC streams is the preservation of aspect ratio when changing resolution and transcoding — if the browser decides to lower the resolution, it lowers it in the same ratio. If the server decides to transcode the stream, it maintains the aspect ratio in the same proportion. Therefore, it makes sense that if we want to show the preloader for HLS, we show it in the same aspect ratio in which the stream enters.

因此，出现了一个新任务–您需要知道流以什么纵横比进入CDN，并将相同的比率分配给预加载器。 WebRTC流的一个功能是在更改分辨率和转码时保留宽高比-如果浏览器决定降低分辨率，则以相同的比例降低分辨率。 如果服务器决定对流进行转码，则它将纵横比保持在相同的比例。 因此，有意义的是，如果要显示HLS的预加载器，请以与流进入时相同的纵横比显示它。

The CDN works as follows: when traffic enters the Origin server, it informs other servers on the network, including Edge servers, about the new stream. The problem is that at this point, the resolution of the source stream may not yet be known. The resolution is carried by H.264 bitstream configs along with the key frame. Therefore, the Edge server may receive information about a stream, but will not know about its resolution and aspect ratio, which will not allow it to correctly generate the preloader. In this regard, it is necessary to signal the presence of the stream in the CDN only if there is a key frame – this is guaranteed to give the Edge server size information and allow the correct preloader to be generated to prevent “first connected viewer issue.”

CDN的工作方式如下：当流量进入原始服务器时，它将向网络上的其他服务器(包括边缘服务器)通知新流。 问题在于，此时，源流的分辨率可能还未知。 分辨率由H.264比特流配置以及关键帧承载。 因此，边缘服务器可能会收到有关流的信息，但不会知道其分辨率和纵横比，这将使其无法正确生成预加载器。 在这方面，只有在存在关键帧的情况下，才有必要在CDN中发信号通知流的存在–这可以保证提供Edge服务器的大小信息，并允许生成正确的预加载器，以防止“首次连接的查看器出现问题”。 。”

摘要 (Summary)

Converting WebRTC to HLS generally results in freezing when played in default Apple players. The problem is solved by analyzing and adjusting the H.264 bitstream to Apple's HLS requirements, either by ranscoding, or migrating to the RTMP protocol and encoder as a stream source. In a distributed network with Lazy Loading of streams, there is the problem of the first connected viewer, which is solved using the preloader and determining the resolution on the Origin server side – the entry point of the stream in the CDN.

在默认的Apple播放器中播放时，将WebRTC转换为HLS通常会导致冻结。 通过对H.264比特流进行分析和调整使其符合Apple的HLS要求，可以解决该问题，方法是通过转码或迁移到RTMP协议和编码器作为流源。 在具有流的延迟加载的分布式网络中，存在第一个连接的查看器的问题，该问题使用预加载器并确定了原始服务器端(CDN中流的入口点)的分辨率来解决。

链接 (Links)

Web Call Server – WebRTC server

Web呼叫服务器 – WebRTC服务器

CDN for low latency WebRTC streaming — WCS based CDN

CDN用于低延迟WebRTC流传输 —基于WCS的CDN

Playing WebRTC and RTMP video streams via HLS — Server functions for converting streams from various sources to HLS

通过HLS播放WebRTC和RTMP视频流 -服务器功能，用于将各种来源的流转换为HLS

翻译自: https://habr.com/en/company/flashphoner/blog/481668/

webrtc 视频