Machine Learning, IDP and Chinese Bamboo Trees

When I was a young engineer a couple of years out of college, I was very keen on making a name for myself as an outstanding coder. Someone who was known to solve problems that others could not, someone who could write three lines of code to achieve the same results that others did with 15. I was once stuck with a problem for a whole day and failed to figure it out. I went to sleep bummed out with that problem. I clearly remember to this day that I woke up in the middle of the night with a solution. I took a shower, got ready, and rushed to the office to try the solution. (Unlike today, in those days working from home was not really an option; most of us used a desktop and had to go to the office to work.) I was delighted to see that the solution worked. And I was even more delighted at the wonder of having solved a tough problem in my sleep.

Apparently, it is a known phenomenon that if you focus on a select few problems and keep thinking about them constantly, then anything you come across while walking, reading, watching movies, listening to the lyrics of songs, etc. can trigger thoughts that can lead to the answer.

In our IDP conversations, some prospects want to know why a machine learning-based system has a longer takeoff time than other systems that are code-based. They also wonder why they should invest time up front instead of using a legacy system that gives them a clearly apparent ‘headstart’. We try to answer this question with data and anecdotal evidence from other customers.

This past week, I heard an unrelated interesting fact about the Chinese Bamboo Tree. Did you know that for the first 5 years of its life, the bamboo tree does not break the ground where it is planted? You water it, fertilize it, attend to it without any sign of it doing anything under the ground. You have to basically do this on faith for not just a few days but 5 years!

Five Years!

But when the bamboo does pop in its fifth year, it grows like it's nobody’s business! It grows from 0 to 90 feet in height in under 5 weeks! In those 5 weeks, it is the fastest growing plant on earth. This begs the question - did it really grow in 5 weeks or 5 years? What was it doing during the 5 years? It was building roots and establishing a strong foundation to support the crazy growth that was about to come. When I heard about this, I could not help but correlate it to the question about the upfront time investment needed in machine learning IDP systems like ours.

Machine learning systems are in some ways like these bamboo trees. You need to invest time upfront training it with your documents. It tries to learn variations, understand layouts, and build a foundational capability to support the prediction wave that is about to come. But once it takes off, it gives you accuracy that is no one’s business. It is a little different from the bamboo tree in that it does not need 5 years of training. Just a few weeks will do.

Here is reference data from an actual live implementation on Infrrd’s IDP platform for a document that had thousands of variations in it. It was impossible to create and manage templates for this type of document that did not follow a fixed format and data moved all around the document.

This is how the accuracy improved with each iteration of training:

By the time the customer started using it, it had a very high accuracy rate:

It built on this accuracy silently like a bamboo shoot hidden under the ground. While other traditional systems belted out 65% accuracy against the start accuracy of 63% of this model, in a few weeks, it outperformed every other accuracy benchmark that the customer had experienced.

With a machine learning-based IDP system, you do invest some effort upfront but you get the following benefits:

•   Pre-Process  Complex Documents: When there are multiple documents in an image, it may be challenging or impossible for OCR to detect the boundaries or coordinates of each document, but IDP systems excel in providing scalable and reliable chop, split, combine documents using computer vision algorithms.
  
•   Manage Diverse Documents: The traditional OCR does not have the expertise in recognizing and classifying documents based on the document type. IDP solutions use machine language algorithms and natural language processing (NLP) to classify documents based on their type, irrespective of whether they are structured, semi-structured, or unstructured.
  
•   Template Free Data Extraction: OCR systems are template-based while IDP solutions are template-free. It is expensive to scale and maintain a template-based solution, such as OCR because you need to invest time and effort to create or modify a template each time a new variation is introduced or detected. IDP learns from each extraction to gain improved performance over time and does not warrant you to invest in scaling, maintenance, or modification of templates.
  
•   Minimize Corrections: OCR systems require a manual review of extraction results and the correction effort remains the same unless you invest in updating or introducing templates now and then. However, IDP systems use machine learning to get smarter as they process more documents. You get Straight Through Processing (STP) for a higher percentage of documents so that a minimal effort is employed in human corrections as the system matures.
  
•   Use intelligence: OCR systems are code-based and template-based, which means they behave or perform as they are programmed. IDP solutions use advanced AI technologies, such as NLP, which means they behave and perform like humans. They learn each time a document is processed, trained, and corrected and use their intelligence to attain improved accuracy and quality for each extraction in the future.
  
•   Manage complexity: Now, let us say you never bothered to use an OCR system because you had complex documents and variations, machine learning will win your faith again.
  
  Much like the Chinese bamboo tree which needs an upfront investment of watering and fertilizing, the IDP system needs a small investment of training and learning based on past data. And very much like the tree which beats all records as it starts to grow, the IDP accuracy that you get after the initial investment beats any accuracy that you may have seen. Give it a go!