Better Business Through Data Analysis & Monitoring My Account

Harnessing the Speed of SQL

160 Million Records Daily

When tasked with creating a customized fuzzy-logic algorithm for duplicate invoice detection that could be used on big data, Kurt Johnson and Ricardo Murillo of Audimation Services Solutions Development Team were at a loss. They needed a solution that could not only handle processing roughly 160 million records at a time, but could also do it on a daily basis in a practical amount of time.

They looked at several applications that included specific analytics for duplicate testing but none of them fit the needs of the situation. That’s when they decided to try SQL (Structured Query Language), the language used by database professionals worldwide. They developed their SQL-based fuzzy duplicate invoice algorithms which match on variations of fields including vendor, date, amount, and invoice number. Their customized SQL algorithms performed successfully and at a speed that makes handling massive data sets practical.

Kurt and Ricardo believe their customized algorithms have great potential when deployed in continuous monitoring environments, such as CaseWare Monitor and hope to expand their usage into other analytics like address matching. Their goal is to achieve quality of output for fuzzy-matching in reasonable time frames no matter the size of the dataset. Their creativity and skills are making this goal a reality.

Print

Number of views (1160)/Comments (0)

Brad Newman

Brad Newman

Other posts by Brad Newman
Contact author

Please login or register to post comments.

Name:
Email:
Subject:
Message:
x
The Power of Field Statistics
7 March 2018

The Power of Field Statistics

IDEA Tech Tip

IDEA’s Field Statistics can save you a wealth of time. It should be the first thing you go to after importing a file. Why?

Read More

Using Data Analytics to Prepare for Fieldwork
20 February 2018

Using Data Analytics to Prepare for Fieldwork

Automating High-Risk Transactions with IDEA

See how a global corporation used IDEA to develop an automated process to prepare auditors for on-site work.

Read More

People. Processes. Tools.
5 February 2018

People. Processes. Tools.

Successful Integration of Data Analytics

Expert IDEA users share their experiences and advice for building and sustaining a successful data analytics program.

Read More

Planning & Scoping Your Audit with Data Visualization
1 February 2018

Planning & Scoping Your Audit with Data Visualization

Data-Driven Insights Save Hours of Effort

Here are some ways IDEA can help you plan and scope an audit using the data visualization features within IDEA.

Read More

Classroom Collaboration at University of Missouri - St. Louis
30 January 2018

Classroom Collaboration at University of Missouri - St. Louis

Student Learn to Develop Analytic Tests

Masters students attending the University of Missouri - St. Louis share their results using data analytics to assess risk, test 100% of the data population.

Read More

Do One Thing Every Day that Scares You
30 January 2018

Do One Thing Every Day that Scares You

Virginia Tech Students Gain Confidence Using Data Analytics

Learn how Associate Professor of Practice Nadia Rogers, CPA, introduced students to IDEA, including types of audit tests that can be performed within the software and various documentation aspects of an audit engagement.

Read More

New Features in IDEA 10.3
29 January 2018

New Features in IDEA 10.3

IDEA now runs faster and offers more flexibility than ever before

The latest version of IDEA introduced a host of new features and improvements.  IDEA now runs faster and offers more flexibility than ever before.

Read More

5 Ways Data Analytics Saves Time
26 January 2018

5 Ways Data Analytics Saves Time

Turn Hours of Work into Minutes

5 time-saving ways data analytics can help you shrink your audit time, while improving the quality of your work.

Read More

Tired of Wrangling Cumbersome Client Data?
24 January 2018

Tired of Wrangling Cumbersome Client Data?

10 Ways Our Importing Services Make it Easy

10 reasons to try our data importing services, which save you time, effort and undue frustration. 

Read More

A Journey in Using Data Analytics
24 January 2018

A Journey in Using Data Analytics

Testimonial from Brian Cullum

Read how Brian Cullum used IDEA to help identify a multi-million-dollar fraud based on the overstatement of the borrower’s lending base.

Read More

Planning & Scoping Your Audit
15 January 2018

Planning & Scoping Your Audit

With Data Visualization

Data visualization is changing the way auditors are communicating with their audiences. Easy planning, find risk areas and prioritize your scope of work.

Read More

Infusing IDEA into Your Organization
16 November 2017

Infusing IDEA into Your Organization

Building a successful and sustainable data analytics program requires a mix of people, processes and products...it all starts with collaboration.

Read More

Using Nested @Functions()
16 November 2017

Using Nested @Functions()

IDEA Tech Tip

Combining @functions in creative ways can open a wealth of new capabilities that you may not have known your IDEA software is capable of.

Read More

IDEA 10.3 Now Available
15 November 2017

IDEA 10.3 Now Available

CaseWare Analytics Launches Update

Newest release of CaseWare IDEA introduces key features and enhancements to help auditors, accounts and other finance professionals improve their audits.

Read More

RSS