Better Business Through Data Analysis & Monitoring My Account

Harnessing the Speed of SQL

160 Million Records Daily

When tasked with creating a customized fuzzy-logic algorithm for duplicate invoice detection that could be used on big data, Kurt Johnson and Ricardo Murillo of Audimation Services Solutions Development Team were at a loss. They needed a solution that could not only handle processing roughly 160 million records at a time, but could also do it on a daily basis in a practical amount of time.

They looked at several applications that included specific analytics for duplicate testing but none of them fit the needs of the situation. That’s when they decided to try SQL (Structured Query Language), the language used by database professionals worldwide. They developed their SQL-based fuzzy duplicate invoice algorithms which match on variations of fields including vendor, date, amount, and invoice number. Their customized SQL algorithms performed successfully and at a speed that makes handling massive data sets practical.

Kurt and Ricardo believe their customized algorithms have great potential when deployed in continuous monitoring environments, such as CaseWare Monitor and hope to expand their usage into other analytics like address matching. Their goal is to achieve quality of output for fuzzy-matching in reasonable time frames no matter the size of the dataset. Their creativity and skills are making this goal a reality.

Print

Number of views (1313)/Comments (0)

Brad Newman

Brad Newman

Other posts by Brad Newman
Contact author

Please login or register to post comments.

Name:
Email:
Subject:
Message:
x
Identifying Churn in AR
19 April 2018

Identifying Churn in AR

IDEA Tech Tip

Customer churn (customer attrition, turnover, or defection) is when a customer decides to stop engagement with your company. IDEA can help easily identify “churn” in accounts receivable by following these steps...

Read More

Jeremy Clopton Joins Integration Partnership Program to Help Professionals State & Sustain Their Use of Data Analytics
18 April 2018

Jeremy Clopton Joins Integration Partnership Program to Help Professionals State & Sustain Their Use of Data Analytics

Data analytics & fraud detection expert Jeremy Clopton joins the Audimation Integration Partnership program to help organizations leverage data analytics.

Read More

Think Like a Thief to Catch One
11 April 2018

Think Like a Thief to Catch One

Apply Logic to Detect Peculiar Behavior

When working with rows & columns of data to spot suspicious behaviors, fraud & errors can look very similar to one another. Tips on sniffing out fraud.

Read More

5 Elements of a Successful Data Analytics Program
12 March 2018

5 Elements of a Successful Data Analytics Program

Building a successful, sustainable data analytics program takes effort…and a few other elements.

Read More

The Power of Field Statistics
7 March 2018

The Power of Field Statistics

IDEA Tech Tip

IDEA’s Field Statistics can save you a wealth of time. It should be the first thing you go to after importing a file. Why?

Read More

Using Data Analytics to Prepare for Fieldwork
20 February 2018

Using Data Analytics to Prepare for Fieldwork

Automating High-Risk Transactions with IDEA

See how a global corporation used IDEA to develop an automated process to prepare auditors for on-site work.

Read More

People. Processes. Tools.
5 February 2018

People. Processes. Tools.

Successful Integration of Data Analytics

Expert IDEA users share their experiences and advice for building and sustaining a successful data analytics program.

Read More

Planning & Scoping Your Audit with Data Visualization
1 February 2018

Planning & Scoping Your Audit with Data Visualization

Data-Driven Insights Save Hours of Effort

Here are some ways IDEA can help you plan and scope an audit using the data visualization features within IDEA.

Read More

Classroom Collaboration at University of Missouri - St. Louis
30 January 2018

Classroom Collaboration at University of Missouri - St. Louis

Student Learn to Develop Analytic Tests

Masters students attending the University of Missouri - St. Louis share their results using data analytics to assess risk, test 100% of the data population.

Read More

Do One Thing Every Day that Scares You
30 January 2018

Do One Thing Every Day that Scares You

Virginia Tech Students Gain Confidence Using Data Analytics

Learn how Associate Professor of Practice Nadia Rogers, CPA, introduced students to IDEA, including types of audit tests that can be performed within the software and various documentation aspects of an audit engagement.

Read More

New Features in IDEA 10.3
29 January 2018

New Features in IDEA 10.3

IDEA now runs faster and offers more flexibility than ever before

The latest version of IDEA introduced a host of new features and improvements.  IDEA now runs faster and offers more flexibility than ever before.

Read More

5 Ways Data Analytics Saves Time
26 January 2018

5 Ways Data Analytics Saves Time

Turn Hours of Work into Minutes

5 time-saving ways data analytics can help you shrink your audit time, while improving the quality of your work.

Read More

Tired of Wrangling Cumbersome Client Data?
24 January 2018

Tired of Wrangling Cumbersome Client Data?

10 Ways Our Importing Services Make it Easy

10 reasons to try our data importing services, which save you time, effort and undue frustration. 

Read More

A Journey in Using Data Analytics
24 January 2018

A Journey in Using Data Analytics

Testimonial from Brian Cullum

Read how Brian Cullum used IDEA to help identify a multi-million-dollar fraud based on the overstatement of the borrower’s lending base.

Read More

RSS