FPGA-based driver assistance systems

Embedded software FPGA development Measurement technology

Camera-based driver assistance systems must be able to process video images in real time. SCS has implemented various image processing algorithms such as SGM stereo, optical flow and Stixel on an FPGA. These are now in series production and make the road safer.

  • Problem definition

    Camera-based driver assistance systems must be able to process video images in real time. With newly developed algorithms, there is often a need to speed up the runtime.

  • SCS solution

    The customer's algorithm was available as a PC programme. SCS ported its functionality to an FPGA platform, which executes the algorithm in real time.

  • Added value

    The customer is now able to execute the algorithm in the test vehicle in real time and test it in real traffic situations.

Project insights

The development of driver assistance systems, such as a lane departure warning system or traffic jam assistant, involves solving a wide range of problems. For example, the system must be taught to transform pixels into known objects (road surface, road markings, vehicles, pedestrians, etc.). The algorithms required for this are implemented as PC programmes by our customer’s technical experts for research purposes.

As the performance of modern processors and graphics cards was not sufficient at the time of the project to process the video image of an automotive camera in real time with the algorithm, the technical expert was unable to test the functionality of his algorithm in the moving test vehicle. Several such algorithms were implemented by SCS engineers in an FPGA and integrated on our suitable hardware platform. The resulting system enables real-time processing of the video images. In each case, the feasibility was first examined in a feasibility study in close cooperation with the customer’s technical experts and the realisation costs were determined.

SCS then carried out the implementation on FPGA. With the implementation on FPGA, the customer’s technical experts are able to execute their algorithms in the test vehicle in real time. They can combine them into a complete system and test them in real traffic situations. Project insight using the example of the Stixel algorithm: Processing video images on a pixel basis requires the algorithm to handle considerable amounts of data. The Stixel algorithm helps to reduce the amount of data: it summarises the pixels of an image column into “columns”, so-called stixels. It forms pixels from vertical surfaces (in vehicle rear ends, kerbs, …), flat surfaces (roadway, pavement, …) and background.

In addition to data reduction, an initial, rough grouping of the pixels into sub-objects is achieved. This algorithm is mathematically complex and correspondingly computationally intensive. As part of a feasibility study, SCS has succeeded in mapping the algorithm onto a streaming architecture for a low-cost FPGA. It was then implemented on the SCS FPGA box and integrated with other algorithms. Two automotive cameras can now be connected directly. The result, a Stixel image, is available via a network connection for display or further processing on a PC.

The FPGA box from SCS was one of the subsystems of the Mercedes-Benz S-Class INTELLIGENT DRIVE, which was the first car in the world to drive a 100km overland route completely autonomously in a pioneering achievement. The selected route leads from Mannheim through villages and small towns to Pforzheim and has historical significance: exactly 125 years earlier, Bertha Benz demonstrated the suitability of the patented Benz motorised carriage on the same route. The modern S-Class successfully mastered numerous difficult traffic situations.

Text and images provide an insight on the following website:
Pioneering achievement: Autonomous long-distance drive in rural and: Mercedes-Benz S-Class INTELLIGENT DRIVE drives autonomously in the tracks of Bertha Benz

Related projects

Visual inspection of railway wagons

Is the block brake worn? How thick is the pantograph contact strip? Is an incorrectly fitted screw coupling hanging down? In future, SBB's "Visual ... More

Visual innovation through AI in ophthalmology

Optical coherence tomography (OCT) is a gentle laser microscope and is used by ophthalmologists to examine the retina for abnormalities. Trained AI ... More

Reliable AI for the measuring wheel set

SBB uses so-called measuring wheel sets to measure the dynamic contact forces between rail and wheel. Sensors on the wheels measure the forces. It is ... More

Smart Meter Toolkit

According to the Electricity Supply Ordinance, the grid operator must enable the end customer to receive real-time measurement data from the smart ... More

Radar Evaluation Kit

Versal Edge based Radar Demonstrator @ European Microwave ... More
Show all projects
Felix Eberli Embedded Systems & Automotive How can I help you?