How Can I Use Computer Vision To Read Charts?
by Amit Jnagal, on May 12, 2017 1:45:00 PM PDT
Using computer vision or AI algorithms to interpret charts is a fairly complex problem. This is a two-step solution - reading the chart and interpreting the values - both of them are full of complexity.
Let me explain it with an example. Take a look at this chart:
- Recognize that is a chart, that is not a big problem. You can do that easily by training your algorithm with charts of different types. In fact, it will also tell you what type of chart this is - perhaps a time series chart.
- Understand that there are two axes for this chart and their values vary from 0 to 100 for the Y axis and Dec 2007 to October 2008 for X axis. This is fairly complex - first you need to employ OCR in some capacity to get the data properly and figure out which data aligns to which axis.
- Understand that there are two cities whose values are plotted on the graph - New York (NY) and San Francisco (SF). You may need to refer to external sources to figure out NY is New York and SF is San Francisco but this is fairly easy too.
- Now, the most complex bit - see and calculate values of all variables at a particular point in a graph. For instance, in the given circle, it needs to figure out that:
In August 2008, the average temperature in NY was 80 degrees and for San Francisco, it was 60 degrees. This is not trivial, even for a human being. This is a relatively simple chart, here is a little more complex chart:
This is where all your fundamental vision algorithms come into play - pixel interpretation, border, and edge detection, etc. Assuming you can do that, then the second half of the problem is to answer questions based on interpretation of this data. They can get abstract very easily. Here is a sample set of questions in increasing order of complexity for the temperatures chart:
- Which city is hotter in May - June? (This is relatively easy if you have solved the reading of the chart)
- Was this a harsh winter for New York? (The algorithm cannot answer this question without access to external data. It needs to figure out what are temperatures NY has had over the past 10, 20 years and then match these values to arrive at an answer)
- When is the most comfortable time to visit New York? (Now we are talking about stuff that is not even there on the chart. The algorithm first needs to figure out what comfortable time means in the context of this chart and then figure out an answer)
As you can see, it can get out of hand very quickly. You can do three things to improve your success with chart interpretation problems:
- Start with the end - A better way to solve this problem is to approach it from the other end - with the question that you want the computer vision algorithm to answer and model your recognition system around that.
- Data & Interpretation Constraints - It also helps tremendously if you know the type of graph that you need to interpret. It is a lot easier to build algorithms that have to read weather charts with two cities' information in them rather than a generic time series chart. Try to build a solution with some constraints to improve accuracy.
- Pre-Processing of External Data - Once you have figured out #2, you may need to get external data and train your system to have the context around the data it needs to interpret. For instance - basic elaboration or dictionary to understand NY=New York & SF=San Francisco, the knowledge of average highs and lows of temperature across different cities over a period of time, etc.
The good news is that AI is advancing fast and it is just a matter of time before computer vision algorithms will start interpreting complex charts with ease.