Basic steps to troubleshoot HLS encryption in FMS

Http live streaming support in FMS provides you “right” to protect you content in its Interactive and Enterprise edition. You may refer technical documentation for the configurations required for this.

Here I would like to present some basic steps for troubleshooting:

How to validate that stream is protected or not:

  1. Place the m3u8 request in firefox/IE from Desktop.
  2. Download the m3u8. This is plain text file. So you may read it in any text editor.
  3. If downloaded m3u8 list another m3u8. Repeat the step 1 and 2 for the listed m3u8.
  4. Otherwise, check whether m3u8 contains the key file information. For this, check whether file has #EXT-X-KEY: tag.

In that case first few lines of your m3u8 must look like this:






Troubleshooting configurations:

1 . Check for hls-vod or hls-live (whichever is applicable) in httpd.conf, HLSEncryptionScope is either set to server or content. By default, this configuration is commented-out.

2. If set to server, check HLSEncryptCipherKeyFile configuration is set and points to valid path of the key file. Check whether key file exist at that path. In case path is relative, absolute path must be resolved as <Apache Installation Folder>/<relative path>.

3. If set to server, check HLSEncryptKeyURI configuration is mentioned. Note: since key file is stored scrambled on the disk, so the key file request must go through the module.

4. If set to content, check both KeyFile and KeyURI are mentioned in jit.conf (in case of hls-vod) or keyfile and keyuri are mentioned in application.xml or event.xml (in case of hls-live). Key file must point to the valid path. If relative, absolute path must resolve to <XML file path>/../<key-file path>.

5. If set to content, both key file and key uri configuration must be present in a single file. For example, in case of live either both must be present in application.xml or event.xml. If present in event.xml, application.xml should “allow” the encryption override in its encryption config.

6. If set to content, remember to validate that your xmls or jit.conf is a valid xml after edit. Opening file in any web-browser will do that.

7. If set to content, for hls-vod, jit.conf must be placed along with the content file.

8. In case you have configured a different key other than that comes with default installation, Note: key must have been generated via scrambler tool inside <FMS installation>/tools/scramble.

9. You may like to check the error logs inside <Apache>/logs folder for any other errors while encrypting your streams.

10. You may validate the m3u8 as described above to find whether stream is getting encrypted or not.

Troubleshooting key file serving.

You may find even though m3u8 has encryption information, you are not able to play the encrypted stream. So in that case, if you find no key request listed in access logs (inside <Apache>/logs) or key file wasn’t served with 200 http code, you need to debug the key file serving.

1. First try with a key uri on http protocol and not on https protocol.

2. Make sure httpd-hls-secure.conf file in included in httpd.conf. Uncomment “Include conf/httpd-hls-secure.conf

3. Make sure configurations inside <hls-key> tag inside httpd-hls-secure.conf file are good. If your key file is being served on http make sure these three setting are on  and disable other configurations inside hls-key tag:

HLSEncryptHostCipherKey true
HLSFmsDirPath “..”
HLSEncryptKeyRepository “../phls”

HLSEncryptKeyRepository must point to a folder where you key files are placed.

4. Restart Apache server and now try subscribing again. If things play fine.. then move to next step of serving key files on https protocol.

5. Make sure module is installed in <Apache>/modules folder and is enabled in httpd.conf. “LoadModule ssl_module modules/” should not be commented or absent.

6. Make sure you have generated the correct server certificates. Server certificates must have fully qualified domain name.

7. Make sure server certificates are properly installed on the ios-client.

8. Make sure SSLCertificateFile file in httpd-hls-secure.conf points to correct server certificate.

9. Make sure ServerName in httpd-hls-secure.conf is FQDN name of the server.

10. For other ssl specific errors, log ssl requests and errors. For this uncomment, “CustomLog logs/ssl_request.log “%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x %{SSL_CLIENT_CERT}x %{SSL_CLIENT_S_DN_CN}x \”%r\” %b” inside httpd-hls-secure.conf.

Hope this helps. Best of luck.. 🙂

DVR and disk management for HDS and HLS

Flash media server 4.5 provides you flexibility to control and manage the HTTP dymanic streaming content storage for the livestream. This helps one to go for 24X7 livestream by just keeping the required window of the content onto the disk and thus not flooding the it unnecessarily.

DVR can thus be actually limited to the content stored on to the disk. However, HLS and HDS both comes up with their own configurations to dictate respective player for the length of seekbar to be enabled for the DVR, but disk management provides a greater control to the central-adminstrator to overall limit the availibility of the content either at the application level or event level.

For the purpose, one can set the DiskManagementDuration. FMS ensure that any segments not having any fragment representing time within this duration with respect to the current time, are deleted off. This ofcourse require HDS recording be segmented into multiple segments.

In such a case, DVR length should be under (DiskManagementDuration – SegmentDuration).  That means DVRInfo tag inside manifest.xml and HLSSlidingWindow inside httpd.conf/Applicationxml/Event.xml which dictates the seekbar length to the respective clients (HDS and HLS clients), should follow this rule.

How to set up your external apache for HDS and HLS

FMS comes up with the bundled Apache version 2.2.17, supplied with respective modules and configuration to support HDS and HLS streaming. There is a complete workflow which includes, encoder, FMS and Apache as webserver that delivers your live stream on HTTP to iOS and Flash devices..

But some users might have their previous installation of Apache that they may not like to replace with the one that is bundled with FMS. Alright.. Here is how you can configure your external Apache to work along with FMS to stream your content to iOS devices..

1. Place following modules inside your <External Apache Dir>/modules

a. On windows : adbe_dms.dll adbe_license.dll asneu.dll hds.dll libeay32.dll libexpat.dll libf4f.dll

b. On linux:

You may find these files inside your <FMS installation dir>/Apache 2.2/modules on respective platforms..

2. Edit httpd.conf file for the configurations:

a. For HLS, pick following configurations from <FMS installation dir>/Apache 2.2/conf/httpd.conf and paste in <External Apache Dir>/conf/httpd.conf

LoadModule hlshttp_module modules/

<IfModule hlshttp_module>
<Location /hls-live>
    HLSHttpStreamingEnabled true
    HttpStreamingLiveEventPath “../applications”
    HttpStreamingContentPath “../applications”
    HLSMediaFileDuration 8000
    HLSSlidingWindowLength 6
    HLSFmsDirPath “..”
    HLSM3U8MaxAge 2
    HLSTSSegmentMaxAge -1
    Options -Indexes FollowSymLinks

<Location /hls-vod>
    HLSHttpStreamingEnabled true
    HLSMediaFileDuration 8000
    HttpStreamingContentPath “../webroot/vod”
    HLSFmsDirPath “..”
    Options -Indexes FollowSymLinks

b. Edit following configs value

HttpStreamingLiveEventPath : Point to your FMS application directory, provide full path

HttpStreamingContentPath : Point to your content directory, for hls-live, it can be FMS application directory (full path) and for hls-vod , it can be your vod content path

HLSFmsDirPath: Point to FMS installation directory. This is required for the licensing path.

c. Similarly for HDS, you may pick configuration under location tag hds-vod, hds-live and add following lines

LoadModule f4fhttp_module modules/
LoadModule jithttp_module modules/

You will have to edit the value of similar configurations here as well as in case of HLS..

3. To enable encryption, you may require to add more configuration as defined in FMS documentations..

( You may read more about the configurations in the FMS documentation.. But this post may serve you as starting point..)

4. Restart your Apache..

5. In case Apache restart gives issues, check

a. Your Apache version has compatibility with the modules. As I said that modules have been compiled against Apache versions 2.2.17 headers. You may like to check for the dll and modules dependency in something like dependency walker on windows and similarly for linux. Check whether you are still using some old version of openssl on windows or libcrypto on linux.

b. Check you have correct configurations at correct place.

Deploying mbr playlist for adaptive-bit-rate streaming

For HTTP streaming on flash and ios devices, FMS in 4.5 provides a categorical workflow on device-network basis. This is very easy to configure and deploy. Adaptive bitrate streaming was long solved problem by FMS, but now it goes one step ahead by providing the users facility of targeting individual device’s capabilities and requirements and let configure the multiple individual mbr playlists specific to them.

For example, if someone has a wifi enabled i-phone or android based phone, supporting a max bandwidth of 64 kbps in some region.. there it is foolish to assume that user will be able to get a smooth stream experience if served with 720 kbps stream. One may argue that switching logic will be good enough to not to switch to the 720 kbps stream even if listed in the mbr playlist. So what’s the harm in just preparing a consolidated list.. Yes, this is true.. but presenting a 720 kbps stream in this case will be good enough to confuse the switching logic.. And one may observe the unnecessary switching glitches if just for a fraction of second your network improves.. I am no expert in -“how player’s switching logic works”, but presenting a bandwidth which is out-of-range of device-network capability in its mbr playlist may be a very good idea in case of HTTP like delivery system.

So in case of variant playlist (HLS) and f4m (HDS), you may generate multiple versions of them depending upon targeted device-network situation. And there can present only those bit-rate encoded streams that may fall in the target network range..

For example, if publisher is publishing 64, 100, 200, 350, 500, 1000, 1500 kbps streams.. then for the wifi devices one may present only 64, 100, 200 kbps bit-rate streams, for desktop devices can list 350, 500, 1000, 1500 kbps streams and for devices which are on a middle kind of network can be presented playlist with 200, 350, 500 kbps bit-rate streams mentioned.

How to deploy it in system:

It is very easy from one web application to identify the device from which the page is being accessed. Depending upon the same, you may present the URL of mbr playlist (variant playlist in case of ios and f4m in case of HDS).. To assist creating a MBR playlist, FMS 4.5 provides a configurator tool..

Some basic step to troubleshoot for HLS live-streaming

Once your set-up is done and for some of your HLS live-stream, you find that it is not playing.. what to do?.. Here I present some of the basic steps to diagnose problem in the pipeline..

1. Check that your encoder has correct codec profile mentioned as per docs.

2. Check that your stream is being recorded inside, <your-streams-dir>/<instance-name>/<event-name> and its growing.. By default <your-stream-directory> is <application-dir>/livepkgr/streams. By default instance-name is _definst_. In case, you have connected to “livepkgr/myinstance”, instance name will be “myinstance”

3. Check your Apache is running fine and a simple Http request from any web browser is logged in the apache’s access log. You may try to access the index.html as http://<server&gt;:8134/index.html

4. Place a request for the m3u8 from any web browser’s address bar. It won’t play it, but should be able to serve you the m3u8 as text file.If not

4.a Check inside Apache’s module folder.. you have placed. Check httpd.conf, you have “LoadModule hlshttp_module modules/” enable.

4.b Check your hls-live specific settings inside the httpd.conf

4.b Check the access logs/error logs inside <Apache>/<logs> folder.

5. Play the m3u8 url from the ios-device and check the access logs/error logs inside <Apache>/<logs> folder. Be sure that your browser or player supports playing HLS stream. Refer specs for that.

Best practices for setting ios-media-file-duration

What is ios-media-file-duration :

We all know unlike RTMP, HTTP is request response model. HTTP delivers data in response to some request. So even if client is playing the livestream, it requires to send some request for data to receive. While in RTMP, you once say to play, and server keeps pushing data to the client as soon as its available to him from publisher. So if server has 1000 min of data, you may choose to send 1000 request to the server and get 1 min of data as response to one request.. or you may choose to send 100 request to the server and get 10 min of data as response to each request.

So ios-media-file-duration is nothing but “amount of data you want to receive in response to one request“.

Implications of ios-media-file-duration:

1. In case you choose to have ios-media-file-duration to be too low, then you are asking client to generate too many request and server to handle them.. This may not be good for your network. Server may clog handling so many request. Sending and handling each request has some overheads at client, server, network, router every point..

2. In case you choose to have high ios-media-file-duration, you are welcoming a huge latency in playback. Reason is that client will have to wait at least for that much amount of data to be available to him, before it can play it.. HLS specification says in its case, client will require 3 such chunks information before it could play..

3. Other than this, in order to achieve better seeking capabilities, you may want each segment starting with a keyframe. So, it is recommended to have ios-media-file-duration be a multiple of key frame interval.

Conclusion – Best practice:

Choose it intelligently depending upon how may concurrent clients will subscribe to your stream and how much latency is acceptable to you. FMS by default configures it to be 8 sec, considering by default publisher configures the keyframe interval to be 4 second.

FMS – video-on demand on iOS devices

Through recently introduced HLS packaging of the content in FMS 4.5, you can not only stream live streams, but video on demand on the iOS devices. What you have to take care is “Encode your video as per Apple’s specification for the iOS playback.”

Apache shipped with FMS 4.5 comes up with the modules that can stream your video-on demand on the iOS devices. FMS also offers you various nice-to-have configurations to you to decide how to manage the caches, number of requests on the server, latency etc..

How to actually set the game:

1. Place your video at any location inside webroot/vod . say at webroot/vod/your-video-path-with-filename.ext
2. Start Apache service, if not already started.
3. Open safari browser, place the URL for playlist file as http://server-name:port/hls-vod/your-video-path-with-filename.ext.m3u8
4. Watch and enjoy your Video..
5. Remember to share the playlist URL with your friends, to let them know the power of FMS you have 🙂

Http live streaming in FMS 4.5

FMS 4.5 has come up with support to deliver http live stream on the iOS devices. This is interesting step forward to cater multiple devices through a single FMS box. Solution provides easy-ness for the broadcasters to just publish single stream from their encoder and FMS will take care of delivering it to multiple platforms with different needs, for example, HDS stream on OSMF, HLS stream on iOS, and RTMP (+variants)

How it works:

  1. Encoder publishes live stream to FMS on livepkgr app. Say streamName : livestream, eventName: myevent
  2. Livepkgr app records the “livestream” in f4f format inside the event folder ( “myevent” here)
  3. Apache is used as the HTTP webserver for the HDS and HLS delivery.
  4. Apache can be configured for the f4f recording event path and content path in the apache configuration file, so that when it is asked for the HDS playlist (f4m) or HLS playlist (m3u8) or their content ts file or f4f fragments respectively , it know from where to pick and process the content from.
  5. Apache in FMS comes with two import module for the purpose mod_f4fhttp and mod_hlshttp. mod_f4fhttp welcomes request for the HDS playback while mod_hlshttp welcomes request for the HLS playback.
  6. Just fire the request for HDS or HLS playlist for the live stream access and wow Here’s the stream for you.