• Font Size:
  • A
  • A
  • A

Feature Articles

Intelligent Video in Security Applications

by Nello Zuech, Contributing Editor - AIA

Today one is finding the migration of machine vision technology into a number of non-manufacturing applications. Some of these non-manufacturing markets have been using different terminology to associate with their specific market, but not consistently. In the security market today one finds the term 'intelligent video' frequently used and more recently the term 'video analytics.' Essentially these terms reflect the application of image enhancement, image processing, segmentation and automated analysis to security related applications – the same underlying techniques used in machine vision.

As the security industry migrates to digital cameras, intelligent video/video analytics is an evolutionary sequence. With increased emphasis on security since 9/11, the market has recognized that things must be done differently. It is impossible for a single security operator to monitor more than a fixed number of camera displays. It is also impossible for any person to maintain consistent vigilance when viewing displays for any length of time. The same issues associated with people inspecting products visually apply – distractions, consistency between operators and adapting to subtle changes over time, etc.

Hence, the market needs intelligent video/video analytics to assess scene changes, identify those of consequence and deliver alerts to the operator to interactively take over and manage the specific scene change induced event. Analog-based video motion detectors have been around for a long time. The challenge has been (just as it was in analog-based machine vision) that electronic noise, subtle acceptable scene changes, etc. would result in flagging the operator with annoying frequency leading ultimately to ignoring potential conditions that do merit attention.

The digital world has changed this. By employing essentially machine vision technology, rather rigorous detection schemes have been developed to flag only serious scene changes that merit attention. Today’s technology has the ability to track the cause of the initial flag not only within the field-of-view of a single camera but also as the object (person, vehicle) moves from one camera’s field-of-view to another. Many now offer behavior analysis where once a person of interest has been identified his behavior can be monitored and if deemed outside of norm, security can be alerted to take action. This could be something as simple as a person leaving a package and moving from the area.

What is most interesting about these applications is that they are generally more demanding than most machine vision applications. For the most part, machine vision works because the scene can be constrained. In the security world, such constraints are generally not possible. In an outdoor environment the system has to be able to handle daily changes (dawn, day, dusk, night), weather changes, seasonal changes, temperature, humidity, vibration, wind blown debris, shadows, cloud coverage, etc. All these variables can contribute to scene changes, which are typically inconsequential. Often, this is managed by using infrared imaging.

Significantly, the very underlying technologies that have yielded more rigorous and cost effective solutions in the machine vision market are having the same impact on the applications in the security market: price/performance of computers, cameras, lighting, optics.

To gain additional insights on intelligent video/video analytics in security applications from a vendors’ perspective, an email was submitted to all known suppliers with specific questions. The answers provided (in a 'round-robin' format) follow. The following people provided answers:

  • Dr. Christoph Stadler, Senior Software Engineer, DynaPel
  • Dr. Dieter Koller, CTO, Dynapel
  • Sean Shankar, VP Marketing, IntelliVision
  • Paul Bodell, VP Sales & Marketing, IQinvision
  • Lyle Powers, President, Radiant Imaging

1. How would you describe your specific intelligent video implementation? Smart camera-based solution? Board-level solution? Software-based solution? Specific applications addressed?
[Christoph Stadler and Dieter Koller - DynaPel]
DynaPel currently provides board-level solutions, so-called video appliances, for the following products:

  • DynaPel CloseView: unattended video object tracking with PTZ (pan, tilt and zoom) camera control
  • DynaPel SteadyEye: video stabilization
  • DynaPel IntelligentCCTV: video enhancement

These solutions will eventually migrate into the camera to generate a smart camera. We also offer the respective 19' rack-mount solutions.

[Sean Shankar – IntelliVision] All of the above. Our core IP is primarily software. However, based on customer requirements, the applications are deployed on servers, devices, boards, and chips (DSPs). In terms of applications, we provide a wide range of applications such as intelligent motion detection, abandoned object detection, intrusion detection, automated PTZ tracking, asset protection, people counting, behavior analysis, and directional monitoring among others.

[Paul Bodell – IQinvision] Our intelligent video is smart-camera based, using our IQeye multi-megapixel network cameras.

[Lyle Powers – Radiant Imaging] Our core technology is implemented in a Video Processor module with the capability to detect and track moving targets in outdoor environments. We also offer a standalone product with one module and a multichannel product that can be configured with one to sixteen modules. Radiant offers the module to OEMs who embed it in their own products.

2. Can you provide a one or two paragraph general description of the underlying principles associated with your specific intelligent video implementation? Image processing/segmentation principles? And image analysis principles?
Intelligent Video can be described as having multiple layers of analysis - from motion detection to object detection, object classification, and behavior analysis. Our technologies address all these layers to provide robust solutions.

[Paul] We manufacturer the IQeye which is a distributed processing platform that provides video-analytics developers access to megapixel images at any point in the image processing stream.  This enables their algorithms to operate more effectively on more data.

[Lyle] Our Video Processor uses a variety of signal processing methods combined with a statistical detector to make the system robust in noisy environments. The emphasis in our algorithms is to make the first levels of image processing as smart as possible so they filter out normal background motion and environmental changes. The output from the low level detectors are segmented into objects which are tracked over time. The tracks are processed by size, age, velocity, distance moved, aspect ratio and other user set discriminators to determine whether to trigger an alarm.

[Christoph and Dieter]

  • DynaPel CloseView: Moving object detection and tracking with automatic control of a PTZ camera following the target object, e.g. an intruder. It uses noise-resilient motion detection, predictive object tracking and a closed camera control loop with an advanced dynamic model of the PTZ camera. It is implemented as a two-camera system with a fixed reference camera covering the scene under observation and a controlled PTZ camera. It integrates into existing installations and requires an initial calibration to correlate pixels in the reference camera with steering directions of the PTZ.
  • DynaPel SteadyEye: Video stabilization appliance that deploys real-time global motion estimation and statistical modeling to compute the ensuing motion compensation to align successive video frames and generate a stable video.
  • DynaPel IntelligentCCTV: Video enhancement appliance that applies contrast optimization, color enhancement, and temporal de-noising.

3. Are the image processing and analysis tools accessible for modification or addition by the user? If so, describe, how.
Our image processing tools are available to anyone who has the IQinVision SDK (software development kit).

[Lyle] Our API provides access to real-time track coordinates for eight targets, and complete control of settings and mask for fixed and PTZ cameras. Also, we customize the firmware for specific applications.

[Christoph and Dieter] No, the three products are intended as stand-alone appliances. Data from video analysis is only used internally, not communicated to any outside device. As security CCTV appliances, the systems require minimal configuration effort. Currently there is no software-API or DVR-Plug-In version of our technology available, although we provide customizations to certain OEM customers.

[Sean] Primarily available via configurations, users can navigate a menu to choose features and set parameters for detection. We provide SDK to our OEM partners and customers to integrate with other applications or products.

4. Can you provide a one or two paragraph general description of your graphic user interface/man-machine interface? What interface steps need to be taken to set up the application?
Our free Windows-based user interface allows changes to the settings and masks over serial or TCP/IP links.  These include the rules for triggering alarms, defining zones, and working with up to 10 pre-sets on PTZ cameras. 

[Christoph and Dieter] The common user interface of our three intelligent video appliances is a simple on-screen-menu. It can be operated via a six-button keypad integrated into the front panel of the appliance, or from any PTZ control joystick. The 19' rack mounted versions of our appliances can also be configured via a built-in web interface.

  • DynaPel CloseView Setup: Serial communication parameters for control joystick and PTZ camera, IP settings for built-in web server, calibration of reference and PTZ camera, and adjustments like tolerated noise level and masking areas.
  • DynaPel SteadyEye Setup: Minimal frequency of motion which is to be compensated, amount of digital zoom used to avoid black borders caused by motion compensation, and IP settings for built-in web server.
  • DynaPel IntelligentCCTV Setup: Maximum contrast gain, level of de-noising.

[Sean] We have standard products that provide an easy-to-use windows-application type of interface. This will present the user with specific parameters that need to be set based on the application. The advanced suite is menu-based, similar to MS Office type of applications.

[Paul]  You can interface with the IQeye on a number of different levels including Active-X and Java, Embedded C and Interpreted C.

5. What are the specific application issues that one must be attentive to when applying intelligent video in a security application? For example, optics issues, lighting issues, resolution issues? Day/night issues? Scene variables? Etc.?
[Christoph and Dieter]
DynaPel CloseView: Reference camera must not shake. It works with wide angle and even fisheye cameras. Automatic adjustment to changing lighting conditions. Objects require a certain minimum size (number of pixels) to be tracked.
DynaPel SteadyEye / IntelligentCCTV: No special issues. Works with any kind of analog camera (black&white, color, thermal, IR) in any environment. Due to internal A/D-D/A conversion of the video signal, any embedded invisible information like closed caption text or Pelco coaxitron PTZ control is lost or distorted.

[Sean] These challenges are very real, that is why simple motion detectors fail and cause a lot of false alarms.  Our Intelligent Video applications include dynamic learning capabilities that adapt to these environmental changes automatically. On identifying any threat conditions, the system uses advanced validations to filter out any false events or alarms.

[Paul] Lighting and number of pixels-on-target appear to be the most significant issues facing smart video developers.

[Lyle] These issues are key to the success of intelligent video in security applications. Time after time, we see system integrators stumble over designing video-based security systems that meet the customer’s requirements. First, there is a tendency to oversell the capabilities of intelligent video processing. These systems don’t have human vision capabilities; they don’t always work under all conditions. It is up to the integrator to design a system in terms of illumination, camera type, number, and placement that matches the characteristics of the intelligent video product.  We get so many questions regarding set up we developed an Applications Guide and posted it with information about our eval units on our website.  System integrators can read about what works and what fails, camera placement, lighting placement, and the use of PTZ cameras to backup the fixed camera array.  It also gives examples of high security system designs, shows some failures we have seen in the past and lists other resources that we use to make systems work including our 3D CAD site modeling service.

There are a lot of sales people that are over selling the capabilities of smart vision and it is hurting the system integrators that install and over promise to their customers.  The main issue is to clarify the perimeter problem, with a camera every 100 meters, with fields of view starting at 80 and ending at 180 meters, with illumination from the side (to avoid the auto-iris shutting down if it views an illuminator mounted incorrectly on the next camera pole), with illumination even within a 1:5 ratio as measured by a simple light meter on top of a yard stick looking down at the ground.  The extended dynamic range cameras (all seem to use the Pixim chipset) overcome some of these requirements, but in general lighting is very important to maintain high Pd and low NAR in VMD.  A 2' wide person should be 2% of the field of view (100' wide FOV at far field) and the ratio of the near to far FOV should not be >1:3).  A min. velocity of .15M/S is the old Air Force and Sandia standard, which is 1/2 ft/sec but the Israeli MOD sites are testing at .0015M/S because they have intruders that hide under a hollow architectural rock and move slowly across the FOV. 

6. What is it that you require from a prospective buyer to assure that what you provide will satisfy their application?
We work with customers to develop clear scope and requirements; this is very critical to the overall success of the deployment. In most cases, we also help customers by recommending cameras as well as optimal locations and positioning.

[Paul] A clearly defined problem with measurable parameters helps us solve the problem and satisfy the customer.

[Lyle] Not much is required from System Integrators.  Eighty percent of our users just press the 'Load Default Settings' button and the basic configuration works for them. We also sell through OEMS who must have EEs and programmers to design the Video Processor into their products.

[Christoph and Dieter] A prospective CloseView buyer should divide the monitoring area into viewing areas small enough to give sufficient resolution for intruder detection and object tracking. It should be combined with a PTZ camera type and protocol supported by our drivers. It requires an installation with a direct, bi-directional 4-wire communication between the PTZ and our appliance.

7. What are the skills required to integrate intelligent video in a security application?
To add smart video to the IQeye requires C programming experience.

[Lyle] Not much. If you can set up a DVR, you can set up our products.  Getting the target coordinates requires 'C' skills to modify and compile a simple API program, which is posted on our website. We don't provide behavior track style products or segment on boundaries or try to differentiate deer from coyotes based on FFTs of the spatial motion of the low side of the targets so we don't get wrapped around the axle as bad as some companies might.

[Christoph and Dieter] Since we spent some effort to keep the installation and configuration setup simple, there are no additional skills necessary on top of what is required by a regular CCTV installer. The CloseView appliance requires a calibration procedure to align the views of the reference camera and the PTZ camera mounted next to each other. Connecting your control joystick and your PTZ camera to the RS422 ports involves the correct assignment of 6 wires to 10 terminal blocks. An installer must also find out address and baud-rate of both joystick and PTZ.

[Sean] In the case of our standard products, they can be deployed by anyone who can install cameras and DVRs. For our advanced suite, the skill level needed is that of someone who is familiar with using a PC. In case any integration with other security applications is needed, it would require some knowledge of programming and APIs.

8. How do you support your products? E.g., help with set-up? Post service support? Training? Warranty? Documentation?
For System Integrators, we provide unlimited phone, email and Skype support. We’ve found that, while a user might need some help with the first application, they rarely need support after that.  For OEMs we provide technical support during the weeks while OEMs are designing the Video Processor into their product.  Then we train their technical support people and provide periodic sales training as needed.  We have very few support calls after that. We provide extensive documentation. The user manual, setup instructions, software license, API documentation, and full SDK are available for download at no cost. Our stated warranty is for 12 months, and we provide 5 year extended warranties.  However we have so few returns, less than 1%, that we usually just repair or replace any defective product.

[Christoph and Dieter] We provide demo units or on-site demo for potential customers. We provide training to sales representatives and dealers and first level support through our network of sales representatives. Furthermore we provide a telephone hotline with direct access to the development team, a one-year warranty and an optional extended two-year warranty. Our products come with comprehensive printed documentation. We also provide a download site for firmware updates, bug fixes and additional documents describing setup and trouble shooting.

[Sean] We provide a whole range of deployment services as well as extended warranty and support. We also provide training, tutorials, and documentation.

[Paul] Post sales support and a 1-year warranty.

9. What are the most important differentiators between competing products in the respective intelligent video applications in the security market today?
[Christoph and Dieter]
Reliability and simplicity. To implement sophisticated algorithms and publish a scientific paper about it is one thing. To port those algorithms into an embedded systems box and build a company around it is another story. Intelligent Video basically summarizes the transition from simple analog video security into an IT industry that will eventually be dominated by software companies. This transition is currently moving very fast, such that, not only customers but also integrators and the manufacturers have a hard time keeping up with the latest developments. We think that we understand the security market enough and that the most successful products are the ones that are powerful yet simple to use. We also understand the current limitations of current state-of-the-art video analysis algorithms and do not make unjustified claims. We provide reliable, powerful, yet simple products.

[Sean] Reliability and Ease-of-Integration. Reliability is key. Customers get turned off when they see too many false alarms coming from simple motion detection systems. They want to be alerted on certain events that they care about. Ease-of-Integration is very important for larger projects where systems integrators tie several systems and applications together. Our application has an open architecture and can integrate with existing systems and security applications easily. IntelliVision excels in both requirements.

[Paul] Reliability and accuracy

[Lyle] High Probability of Detection (Pd), Low Nuisance Alarm Rate (NAR), high Mean Time Between Failures (MTBF), high density, low power, low price, and ease of setup are all important selection factors.  Price is important if you have the rest, but a product that produces nuisance alarms and does not detect targets in low contrast situations is expensive because the guards will shut off the alarms and you have zero protection.  Happens a lot.

10. In your opinion, what are the main product features (of the specific products you offer) that customers are looking for today?
Our feeling is that the market is still at the high-end - primarily government agencies and large corporations concerned about security and safety. They need flexibility and robustness. Our Advanced Suite is targeted at this market. However, we see the intelligent video moving to the mass market and we are beginning to deliver our products to this market as well.

[Paul] On-Camera processing and a high number of pixels-on-target.

[Lyle] Bids for borders, pipelines, airports, military bases, nuclear power plants and infrastructure are getting more and more competitive.  Our customers demand high performance, high reliability, and high density at low cost. In Israel it is all about lost bags (euphemism for bombs in briefcases), in Europe and UK it is about package density and resolution because they don't have enough rack space and power, and in the US it has become high tech discriminators -- 3D background models or multiple camera VMDs. 

[Christoph and Dieter] CloseView provides completely unattended close-up views of intruders, cars or other moving objects utilizing power zoom lenses of PTZ cameras. It alerts a video guard and, therefore, takes a tremendous load off him. He can now watch many more cameras without missing any intrusion and, therefore, saves the company money without loss of security. It, therefore, can be considered an intelligent or smart camera. CloseView integrates with existing PTZ installations.

SteadyEye and IntelligentCCTV provide Plug-and-Play operation. Simply hook it up between your camera and your downstream devices and you make your video more useful or solve certain video problems. High powered zoom lenses of modern PTZ cameras or extremely long lenses in certain military applications dramatically emphasize miniscule camera or lens shake and generate shaky video that is electronically corrected in DynaPel’s SteadyEye appliance.

11. Are there some emerging changes associated with the underlying technology used in intelligent video in security applications that will impact performance? What will those impacts be on specific applications?
Smart multi-megapixel cameras will enhance the reliability and accuracy of smart video.

[Lyle] All sensors fail in certain scenarios.  Combining target information from extended dynamic range cameras with target information from infrared cameras will enable smart video to work in applications that require higher performance. Combining information from different types of sensors that sense different phenomenology (microwave/fence/long range PIR), overcome the weaknesses of each individual sensor.

In terms of algorithms, more powerful machine learning techniques: Bayesian Nets, Support Vector Machines, and Active Learning are being developed now.  Research in computational neuroscience promises to give us systems that more closely match human capabilities. That’s the goal for intelligent video: flexible, adaptive, and easy to train systems. 
PTZ tracking: Indoors (think distance learning) will be solved first. Outdoor PTZ tracking will continue to fail for a few more years, but eventually get better as the basic detectors are improved to work with panning and tilting without loosing Pd and NAR values.  Finally, full motion with rotating reference frame applications will be improved.  The worst case is tracking a car at night under the freeway lights.  The on/off pattern of illumination is tough.

  • Combination Technology: Extended dynamic range cameras combined with affordable infrared cameras are crucial for high-end applications looking over brush.  In August the person and the ground are 98 degrees, but the 6-foot chaparral waving in the breeze is cool.  This means the sensitivity (regardless of statistics) must hug the noise floor to detect the target, which is <3 luma levels away from the background.  The cool dark brush is >40 luma levels from the background.  When the wind blows the brush (think southern border and nuclear plant boundaries), there are waves of 'motion' that are the correct size, shape, aspect ratio, velocity, direction as the targets, and there is a LOT more energy (size of target times luma difference) in the brush than in the target.  At this time it is best to use the day/night camera.  Looking over vegetation in IR is one of the toughest apps.
  • Temperature Range:  We see -40C to +80C temperature ranges being discussed for border applications.  There will be specs for boot time, MTBF, temp range, and overall power requirements that eliminate PC based systems from the bids.
  • Bayesian Nets, Adaptive Learning, Differential Target/Background compression rates and multi-phenomenology will add to the effectiveness of these detectors.
  • Dynamic warping:  Each camera will be dynamically warped vertically and horizontally with warp tie points to allow adjacent camera views to be panned across a single monitor seamlessly.   This will require light and color balance so the images look smooth on the screen, which can easily hold 4 side-by-side cameras and pan smoothly, using fixed cameras around the perimeter.

Industries weakest area is the fence.  We are trying to put sensors on a very stupid fence.  Expect smart fences to be manufactured with an array of sensors, video, chemical, vibration, thermal, microwave, and capacitive, that all connect to a backbone integrated into the fence.  This is critical to visualize in seeing the future.

[Christoph and Dieter] Using networked cameras usually means compressed video (e.g. MPEG-4, H.264). Unavoidable compression artifacts might affect further analysis of the video. Additionally, the compression/decompression adds some delay to the video signal, which affects object tracking.   The typical installer of CCTV systems has limited experience with IP networks, while the typical IP geek knows little about CCTV installations. Although both are learning, it will take some time until the transition is completed.

[Sean] Emerging changes include newer algorithms in software, and faster processors in hardware. This will lead to wider deployment of Intelligent Video applications.

12. What are specific camera and camera connectivity trends associated with data capture for intelligent video applications in the security market?
We are designing new products that combine high density, low power, image stabilization, VMD, tracking, H.264 Compression and IP connection for fiber, copper, or RF connections.  The module is about the size of a small box of matches.

[Christoph and Dieter] Analog Video cabling and RS422 camera control will be replaced by far more flexible and superior IP based network solutions. Although Intelligent Video should not necessarily be associated with IP video, it is being used in the same context.

That’s because Intelligent Video denotes video that is enhanced or tagged with meta data provided by video analysis. And that meta data is easily being added into the IP video stream. Although video compression has gained some ground on reducing the bandwidth, there are no major improvements ahead that could further reduce bandwidth and storage space without scarifying video quality, which in turn is necessary for robust video analysis. This seems to be a typical IT dilemma: hardware improved much fast than software.

[Sean] Most of the market is analog, which will continue for the next few years. We see IP cameras gradually increasing over the next 2 to 3 years, after which we foresee a rapid increase in their deployment.

[Paul] Everything will be TCP IP and most of the processing will move to the edge of the network.

13. What are other major trends you see associated with future intelligent video in security applications?
[Christoph and Dieter]
The transition from video security into an IT industry requires a learning phase on both sides. I expect this transition to be very fruitful in the next few years since both sides can learn from each other. This will result in much more intelligent devices like smart cameras and the like. But I then expect a slowdown after that transition period has been settled. Just the same way in the IT Industry where there are dramatic improvements made to hardware performance in the past years, whereas intelligent software improvements are still limited. Comprehensive video and scene analysis relies on computer vision algorithms that fall within the area of artificial intelligence. Progress in artificial intelligence has been very limited in the past 20 years, despite very ambitious programs and projects. We, therefore, need human video guards in the loop for the foreseeable future.

[Sean] We see the market movi



There are currently no comments for this article.

Leave a Comment:

All fields are required, but only your name and comment will be visible (email addresses are kept confidential). Comments are moderated and will not appear immediately. Please no link dropping, no keywords or domains as names; do not spam, and please do not advertise.

First Name: *
Last Name: *
Your Email: *
Your Comment:
Please check the box below and respond as instructed.

Search AIA:

Browse by Products:

Browse by Company Type: