How Can I Use Computer Vision To Read Charts?
by Amit Jnagal, on May 12, 2017 1:45:00 PM PDT
Using computer vision or Artificial Intelligence (AI) algorithms to interpret charts is a fairly complex problem, which is precisely why the AI job market is almost always on the lookout for talent that not only has a deep understanding of machine learning, deep learning, natural language processing, data structures and graph neural networks in addition to the basics of computer science.
Getting back to the chart!
We believe you will agree that this is a two-step process - reading the chart and interpreting the values - both of which are full of complexity.
Let us explain it with an example. Take a look at this chart:
For computer vision to work on this chart, it needs to do the following:
Recognize that is a chart, and that is not a big problem. You can do that easily by training your algorithm with charts of different types as training data. In fact, by deploying basic machine learning techniques, it will also tell you what type of chart this is - perhaps a time series chart.
Understand that there are two axes for this chart and their values vary from 0 to 100 for the Y axis and Dec 2007 to October 2008 for the X axis. This is fairly complex - first, you need to employ OCR in some capacity to efficiently accomplish the data mining and figure out which data aligns to which axis.
Understand that there are two cities whose values are plotted on the graph - New York (NY) and San Francisco (SF). You may need to refer to external sources to figure out NY is New York and SF is San Francisco but this is fairly easy too.
Now, the most complex bit - see and calculate values of all variables at a particular point in a graph. For instance, in the given circle, it needs to figure out that:
In August 2008, the average temperature in NY was 80 degrees and for San Francisco, it was 60 degrees. This is not trivial, even for a human being. This is a relatively simple chart, here is a little more complex chart:
This is where all your fundamental vision algorithms come into play - pixel interpretation, border, edge detection, etc. Assuming you can do that, then the second half of the problem is to answer questions based on the interpretation of this data. They can get abstract very easily. Here is a sample set of questions in increasing order of complexity for the temperatures chart:
Which city is hotter in May - June?
This is relatively easy if you have solved the reading of the chart
Was this a harsh winter for New York?
The algorithm cannot answer this question without access to external data. It needs to figure out what temperatures NY has been over the past 10, 20 years and then match these values to arrive at an answer
When is the most comfortable time to visit New York?
Now, we are talking about stuff that is not even there on the chart. The algorithm first needs to figure out what comfortable time means in the context of this chart and then figure out an answer.
As you can see, it can get out of hand very quickly.
How to Enhance Data Interpretation Efficiency?
You can do three things to improve your success with chart interpretation problems:
Start with the end
A better way to solve this problem is to approach it from the other end - with the question that you want the computer vision algorithm to answer and model your recognition system around that.
Data & Interpretation Constraints
It also helps tremendously if you know the type of graph that you need to interpret. It is a lot easier to build relevant AI Index and algorithms that have to read weather charts with two cities' information in them rather than a generic time series chart. Try to create a solution with some constraints to improve accuracy.
Pre-Processing of External Data
Once you have figured out #2, you may need to get external data and train your system to have the context around the data it needs to interpret. For instance - basic elaboration or dictionary to understand NY=New York & SF=San Francisco, the knowledge of average highs and lows of temperature across different cities over a period of time, etc.
The good news is that AI is advancing fast and it is just a matter of time before computer vision algorithms will start interpreting complex charts with ease.
About this blog
AI can be a game-changer, but only if you know how to play the game, something that global private investment firms are slowly but steadily getting better at!
If anything, we have all learned the growing importance of Machine Learning and Artificial Intelligence in the past decade.
This blog is a practical guide to turning AI into real business value. For instance, predicting future growth prospects on the basis of the compound annual growth rate of the past few years. Learn how to:
- Make sense of complex documents and images.
- Extract the data you need to drive intelligent process automation.
- Apply AI to gain insights and knowledge from your business documents.