How to effectively store and manage machine vision images and data

The information generated by machine vision systems puts a whole new spin on the term “big data.” The raw image files output from vision systems are huge, and manufacturers can generate terabytes of image data in a month—even in a week! So, what do you do with all this data? 

Decrease the size of machine vision images and data files 

Those big, fat raw image files need only be stored if the intention is to reprocess them. If the primary need is to maintain an archive of images only for compliance purposes, storage needs can be dramatically reduced by reducing file size. This can be accomplished by changing resolution or using a compressed format. An image can be reduced to a tenth the size of the raw original, or even less, and the naked eye can barely notice a difference.

Store machine vision images and data in a central database

Getting your files into a central database will make them more accessible and manageable. This requires a network architecture for data transfer to get the files out of that inspection station silo. It could be a wireless network or a fixed Ethernet connection. 
In order to access reliable, real-time image data transfer, you’ll require a software gateway that can do the following: 

• Manage cameras and their connections: This could include managing multiple cameras, showing live camera status and running calibration routines
• Collect and convert images and data: Depending on your system, images may need to be converted into different file formats, merged with overlays, or compressed to minimize storage space
• Manage all of the files created from these processes: This includes managing image and image file data transfer, storing raw image files (either temporarily or permanently), and caching data for data surety (given that machine vision systems and machine vision cameras often don’t have local data storage)
• Facilitate reliable communication and data transfer: The software gateway must be able to communicate between your PLCs (programmable logic controllers) to ensure reliable data transfer, error handling and system health

Enable a user-friendly archive for your data

An archive is not a backup of your data. It’s a means of keeping the production data repository at a consistent size with the flexibility to “slice” the data into meaningful buckets—such as blocks of one, two, or three months, to make it easy to rotate older data into longer term storage.

This archive must be capable of storing terabytes of data in a way that supports easy and intuitive search and retrieval for rapid analysis. This means serialization—all datasets indexed by the serial number of the part/assembly to which they belong.

Avoid inconsistency in machine vision data 

Inconsistency in how data is tracked and managed can be a huge problem for manufacturers. Data must be stored in the same way, using the same types of data across the plant and/or enterprise in order to be easily accessed and put to use when a situation arises. 

We have worked with companies in this situation. It was taking them days to troubleshoot quality issues using custom query tools to identify the problems. If everything had been indexed by part/assembly serial number (a birth history record), days would be reduced to hours, if not minutes—without the need for custom query tools.

Consider long-term data management needs 

Have a clear archival policy because you never know when that vision data from six, 12 or even 18 months ago might become vitally important when faced with a flood of warranty claims or a recall. 

A clear retention policy must be determined before data collection begins because it will drive a number of requirements, such as the IT department procuring the necessary space and resources for data storage. You need to be proactive, not reactive, to ensure a smooth rollout. 

Examples of potential policies:

• Keep production data accessible for up to 12 months 
• Keep original, raw images for six months 
• Keep image data for three years 

To learn more about machine vision and data analysis on the modern production line, download our free ebook: "How to use machine vision for manufacturing 4.0".\

Sciemetric offers solutions for process monitoring, learn more about the sigPOD and Sciemetric Edge.