History | Log In     View a printable version of the current page. Get help!  
Issue Details (XML | Word)

Key: SN-39
Type: Bug Bug
Status: Open Open
Priority: Major Major
Assignee: Luke Hubbard
Reporter: Daniel Daley
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
Streaming and Networking

Buffer lockup for VOD clients.

Created: 19/Apr/07 04:45 PM   Updated: 28/Mar/08 05:17 PM
Component/s: None
Affects Version/s: None
Fix Version/s: None

File Attachments: 1. Text File jstack.txt (33 kb)



 Description  « Hide
Recently with revisions in the last month or so I've been having a condition where the server will stream an initial buffer to the client and then will stop and sit as if buffering. This will happen on all VOD clips at the same time and it will always play the same few seconds of video if you restart the stream. Restarting the server will fix the issues and it will come back after several hours. I only see it under fairly high loads (not maxed though) as the server will usually crash from OutOfMemory errors under less loads if it has been running for very long. Unfortunately, the logs look normal with no exceptions. In fact, the logs just seem to indicate all is normal and people just are watching a few seconds of everything. The version of Red5 I am running is r1898.

 All   Comments   Change History      Sort Order:
Daniel Daley [17/Jul/07 07:07 PM]
Hello, I apologize for my lack of updates on this ticket. The issue appeared to have somewhat gone away for quite a long period of time until just recently.

The issue is pretty much the same in that when pulling VOD content from the server, the first few frames of the video are sent and the player will sit as if buffering while no more data is sent from the server. This happens to every connection as I can see in my access logs tons of hits for a few seconds with very low data transfer (average around 22k). Restarting the server corrects the issue. With the randomness of it's occurrences and with having it disappear for some time now it makes me think there is perhaps a customer file that is causing the issue. There are hundreds of files or I'd try to narrow it down, and being that there are no indications whatsoever in the log I'm not sure what to look for anyways. I have upgraded to the latest svn version as of 2007-07-16 and the issue still occurs. I've monitored the memory usage closely as well and it doesn't seem to be an issue, but I'd probably expect to see some actual errors in that case.

I really wish I had some more information to throw out there on this. Do you have any ideas what I might try? Could there be some kind of resource management going awry that I could try to comment out in the server?

Thanks as always for your assistance!

Daniel Daley [19/Jul/07 08:18 PM]
After examining our usage logs I've noticed that it has gone up quite a bit since this started occurring. So this issue may simply be related to higher usage rather than a particular file or client like I thought. The server in question is doing well over 100GB per day of traffic.

Joachim Bauch [19/Jul/07 11:34 PM]
Could you please generate a jstack dump as described here [1] when the server is in this state and attach to this ticket? Perhaps some of the threads responsible for streaming get locked under high load.

[1] http://osflash.org/pipermail/red5_osflash.org/2007-May/011632.html

Daniel Daley [19/Jul/07 11:41 PM]
I was afraid that would come up. As far as I can find jstack doesn't run on FreeBSD (or bsd like system such as OSX). Do you know if there's an alternative?

Joachim Bauch [25/Sep/07 09:56 PM]
You can also use jconsole to view realtime informations about your running Java process and find deadlocks in threads. Jconsole can even be used remotely.
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html

Jered Davis [16/Jan/08 09:45 PM]
I too am currently seeing this issue. I HAVE installed the latest trunk and no go. I am attempting to stream VOD from a VP6 encoded file that is approximately 175 MB in size. The video WILL play fine after a restart of Red5 and then within an hour or so the buffering issue begins.

JStack output attached.

Bill [28/Mar/08 05:17 PM]
here's what the log says when it stops:

> > [INFO] [btpool0-2] org.red5.server.stream.ProviderService -
> > getVODProviderFile scope path: name: <vod file name>
> > [INFO] [btpool0-2] org.red5.server.stream.ProviderService -
> > getStreamFile null check - factory:
> > org.red5.io.StreamableFileFactory@1c190c99 name: flv:<vod file name>
> >