Harnessing the Speed of SQL

160 Million Records Daily

When tasked with creating a customized fuzzy-logic algorithm for duplicate invoice detection that could be used on big data, Kurt Johnson and Ricardo Murillo of Audimation Services Solutions Development Team were at a loss. They needed a solution that could not only handle processing roughly 160 million records at a time, but could also do it on a daily basis in a practical amount of time.

They looked at several applications that included specific analytics for duplicate testing but none of them fit the needs of the situation. That’s when they decided to try SQL (Structured Query Language), the language used by database professionals worldwide. They developed their SQL-based fuzzy duplicate invoice algorithms which match on variations of fields including vendor, date, amount, and invoice number. Their customized SQL algorithms performed successfully and at a speed that makes handling massive data sets practical.

Kurt and Ricardo believe their customized algorithms have great potential when deployed in continuous monitoring environments, such as CaseWare Monitor and hope to expand their usage into other analytics like address matching. Their goal is to achieve quality of output for fuzzy-matching in reasonable time frames no matter the size of the dataset. Their creativity and skills are making this goal a reality.