Thieves Hunter

Introduction

From the time I've started publishing articles on the web, I got lots of contacts from many different people around the world. As it usually happens some of these contacts are very interesting, but some of them you may want just to ignore by closing the e-mail as soon as possible after opening it. In this article I would like to tell the story of one of the "interesting" contacts I got recently and what has happened as a result ...

Not so long ago I got an e-mail from a person living in UK, who was asking for help. He was asking to help in finding thieves ... Well, it was the first time I was asked about such type of help, so it was quite intriguing. The story of that person was about stolen bicycle from a garage. And this was not just a bike - it was a bike, which had special value because of some memories (whatever) related with it. So there was a strong wish to find those thieves and get the bike back.

Luckily the area of the garage was monitored by two cameras and security guards finally agreed to provide 1 month recordings. Unluckily the records were made as small video clips of 120 seconds length each. This makes 720 records per day, 21600 records per month and in total 43200 video clips to review from both cameras. Unfortunately it was not known exactly when the bike was stolen - the time window was around 2-3 weeks. This means that without any special software, which could process all videos in a batch, it could take quite a lot of time to process all the videos manually. Personally I would be depressed by 43200 videos and not sure I would start viewing them manually. And we did not. Instead of this we decided to give this work to motion detection software ...

Playing video files

So, the decision was to write a small application, which could analyze video files in a specified folder (with subfolders) searching for motion and providing information about amount of motion in each file. Taking into account that AForge.NET framework seems to provide all we needed, it did not look so complex. Let's go step by step.

The first thing we need to do is to get access to video files, being able to process them frame by frame. The AForge.NET framework provides several classes for playing videos, which could be used for our purpose - FileVideoSource and AVIFileVideoSource. Both of them implement the same interface, which is quite easy to work with:

// create video source
IVideoSource videoSource = new FileVideoSource( fileName );
// or AVI video source
// IVideoSource source = new AVIFileVideoSource( "some file" );

// set NewFrame event handler
videoSource.NewFrame += new NewFrameEventHandler( video_NewFrame );
// start the video source
videoSource.Start( );
// ...
// signal to stop
videoSource.SignalToStop( );
// ...

// New frame event handler, which is invoked on each new available video frame
private void video_NewFrame( object sender, NewFrameEventArgs eventArgs )
{
    // get new frame
    Bitmap bitmap = eventArgs.Frame;
    // process the frame
}

If these classes implement the same interface and are aimed for the same purpose, then what is the difference actually? Well, they are really very similar in what they do, but very different in how they do it. These classes can be found in different namespaces and, although they do similar work, they do it using different technologies. The first one, FileVideoSource, uses DirectShow to get access to video files of different types. The second one, AVIFileVideoSource, uses Video for Windows interface to get access to AVI (only) video files. Video for Windows interface is quite old API for playing AVI video files, which is marked as deprecated (but still supported) by Microsoft. But DirectShow interface is relatively newer and stays as a standard interface for video processing on Microsoft systems.

At the first glance both of the above classes could suite our needs. We had thousands of AVI files and both of the classes could read them. Since DirectShow seems to be the newer interface, which is supported by all new video codecs, why not to use it (i.e. use FileVideoSource class)? We could do that, but there is one more difference between above two classes - the FileVideoSource class does not allow controlling playing frame rate, i.e. video files are played with the frame rate, which was set during saving. But AVIFileVideoSource class gives full control of frame rate - it is possible to specify which frame rate to use for playing: original or the one you like. It is even possible to set that you want to receive new video frames as fast as possible (see FrameInterval property)! (Note: all the above said does not mean that DirectShow does not allow controlling frame rate; it is just not yet supported by the AForge.NET framework).

Why playing frame rate is so important in our task? Because the total amount of time, which we need to spend to process all video files, depends on it. If we were processing video files using original frame rate we would spend 2 month to process them all - 1 month of savings for 2 cameras. This would be too long. So, we need to get rid of any delays between frames and process them as fast as we can. This means we go for AVIFileVideoSource this time, which gives the frame rate control.

But, before we go with AVIFileVideoSource, lets take a look at what we have else. The AVIFileVideoSource is nice since it implements IVideoSource interface, which unifies access to different video source. But, to simplify things, we may use also AVIReader class, which gives less overhead and suits better the task of processing collection of video files:

// instantiate AVI reader
AVIReader reader = new AVIReader( );
// open video file
reader.Open( "test.avi" );
// read the video file
while ( reader.Position - reader.Start < reader.Length )
{
    // get next frame
    Bitmap image = reader.GetNextFrame( );
    // .. process the frame somehow or display it
}
reader.Close( );

So, we decided to go with Video for Windows ... But this decision comes with its price. As I already mentioned Video for Windows is an old interface to access videos, but DrectShow is a newer one. This leads to the fact that it is much easer to find video codecs, which support DirectShow, since all new codecs support the newest technology. But it may be harder to find video codecs supporting older interface, which is Video for Windows. But it is still possible to find them. For example, you may try K-Lite Mega Codec pack or FFDShow package. Both of these provide plenty of codecs, which are accessible though both DirectShow and Video for Windows interfaces.

Personally I have FFDShow package installed on my system, so I may need to make a note about it. By default most of the FFDShow codecs are available through DirectShow interface only. But it is possible to enable Video for Windows interface as well. If you run "VFW configuration" utility, which comes with FFDShow package, you will be able to enable codecs (see Codec section) by selecting "libavcodec" in "Decoder" column:

Detecting motion

Once we have solution for video playing, the rest seems to be simple. AForge.NET framework provides set of motion detection classes, which do not depend on any particular video source. These classes deal with regular images, which are fed continuously to them, leaving all video related stuff to user. This makes motion detection classes to be highly reusable - you may choose any approach you like for reading video files.

The framework contains different type of motion detection classes, which differ in features they provide and performance they give. For the application we build - counting amount of frames with motion - we may select the simplest (and the fastest) motion detector, which is based on comparing of two consequent video frames - TwoFramesDifferenceMotionDetector.

So, all we need to do is to create motion detection class we want and continuously feed it with video frames. After each processed video frame we need to check MotionLevel property to get information about motion amount:

// create motion detector
TwoFramesDifferenceMotionDetector motionDetector =
    new TwoFramesDifferenceMotionDetector( false, true );

...
// for each video file to process
amountOfMotionFrames = 0;

// create reader
AVIReader reader = new AVIReader( );
// open file
reader.Open( videoFileName );
// process all frames in the video file
for ( int i = 0, n = reader.Length; i < n; i++ )
{
    Bitmap frame = reader.GetNextFrame( );
    motionDetector.ProcessFrame( frame );
    if ( motionDetector.MotionLevel > alarmLevel )
        amountOfMotionFrames++;

    frame.Dispose( );
}

motionDetector.Reset( );

The idea of the above code is quite simple - feed all video frames to motion detector and count amount of frames, where motion level is higher than specified threshold. That is all need!

Putting all together

Putting all the code together and adding some user interface, we got a relatively simple tool, with the next list of features:

• It may process video files from selected folder and all its subfolders;
• For each processed video file, it provides information about total amount of frames and amount of frames with motion detected;
• By double clicking a file in the list of files, it is possible to start playing the file, so it can be viewed manually;
• It allows to sort files by amount of detected motion;
• It allows selecting some files from the list and save all the information about them into CSV file.

Final result

As I mentioned in the beginning of the article, all the work we did was initiated by a real story. So, the developed tool did not end up somewhere in file system as a ZIP file, but passed real test. And it was successful! Unfortunately, as far as I am aware, the success was partial, but we got it.

Using the developed tool it was possible to automate processing of video archive provided by security department. Once processing was finished it was possible to analyze provided results, find video files with higher amount of motion and finally find video files, which show thieves stealing the bike. Those video files were given to police and thieves finally were caught! So, it was success for the tool. Unfortunately it was not success for the bike owner - it looks like those thieves have done something with the bike and did not own it any more ...

Conclusion

Well, I must say it was fun developing a tool for a real case and test it on real data. And it was even more fun, that the tool was successful. Yes, final aim was not obtained - bike was not returned. But at least thieves were caught by police!

The developed tool potentially may evolve further and be extended with even more features. If one real video surveillance setup had such configuration to write separate video file for each 2 minutes of inspection, then potentially such configurations may exist somewhere else as well. So, who knows, maybe it is not the last story of this application and it will find more work to do.