AI is fast, powerful, runs virtually error-free, and does not pause. So it is already superior to humans in many areas where work processes must be carried out continuously with consistent high production and quality. These are the reasons why we want to use artificial intelligence in a machine vision environment in interaction with robots, in order to make operations more efficient and less expensive. A vision-guided robot use case shows how typical pick-and-place tasks can be intelligently automated with a robot and an AI-integrated vision camera that can even do without a computer.
For Smart Fist, the different disciplines must work optimally together. For example, if the task is to sort products of different sizes, shapes, materials or quality using a robot, it is not enough to pick them up, they must also be selected, analyzed and located in advance. With rule-based image processing systems, this is often not only very complex but also difficult to achieve economically, especially for small batch sizes. But in combination with AI-based reasoning, it is already possible today to teach industrial robots the necessary skills and product knowledge to a specialized worker.
On the production line, objects are randomly scattered on a conveyor belt. Objects must be recognized, selected and placed, for example, in their packaging or transported in the correct position to a processing or analysis station. The software company urobots GmbH has developed a computer-based solution for capturing objects and controlling robots. Their trained AI model was able to recognize the position and orientation of objects in camera images, from which the grip coordinates of the robot were determined. The goal now was to transfer this solution to the integrated AI-based vision system of IDS Imaging Development Systems GmbH. Because for the solution, urobots were mainly interested in two things:
1. The user must have the ability to easily adapt the system himself to different application cases without the need for special expertise in the field of artificial intelligence. And this is even if, for example, something changed in production, such as the lighting or appearance of objects or even if other types of objects had to be combined.
2. The whole system had to work without a computer thanks to the direct connection of the components of the device, to be economical, light and space-saving at the same time.
These two requirements have already been resolved in IDS using the IDS NXT Inference Camera System. The trained neural network identifies all the objects in the image and additionally discovers their position and orientation. Thanks to artificial intelligence, this is not only possible for things that are always static and identical, but also when there is a lot of natural variation, such as food, plants or other flexible things. This results in a very reliable detection of the position and direction of objects. Urobots GmbH has trained the network for the customer using its own software and knowledge, converting it to the correct format and then uploading it to the IDS NXT camera. To do this, it had to be translated into a special format resembling a kind of “linked list”. Transferring a neural network trained to be used in an inference cam was very easy thanks to the IDS NXT gateway tool provided by IDS. Then each CNN layer becomes a node descriptor that accurately describes each layer. Finally, a complete list associated with the CNN is generated in binary representation. This FPGA-based CNN accelerator specially designed for camera can implement these universal CNN formats in an optimized way.
Then the vision app developed by the urobots calculates the optimal grip positions from the detection data. But the job is not over yet. In addition to what, where and how to capture, a direct connection must be established between the IDS NXT camera and the robot. This task should not be underestimated. This is often where the time, money, and manpower to invest in a solution are identified. The urobots implemented an XMLRPC-based network protocol in a camera vision application using the IDS NXT Vision App Creator, in order to send concrete work instructions directly to the bot. The ultimate AI vision app detects objects in about 200 milliseconds and achieves a position accuracy of +/- 2 degrees.
It’s not just AI that makes this use case so smart. The fact that this solution works completely without an additional computer is also interesting in two respects. Since the camera itself generates image processing results and not only provides images, it is possible to do without computers and all associated infrastructure. Ultimately, this reduces installation acquisition and maintenance costs. But it is also often important that process decisions are made directly on site, i.e. “just in time”. Thus the following operations can be performed more quickly and without delay, which in some cases makes it possible to increase the rate.
To find out more