What is interpretability?
Interpretability is the ability to explain to a human being how a machine learning algorithm came to its decision. Some algorithms (e.g. deep learning / neural networks) are opaque and virtually impossible to decipher. Other algorithms such as Decision Trees and Regression can be explained to a person.
Why is interpretability important?
The importance of interpretability depends on the business case. For some use-cases accuracy is the most important criteria (e.g. online ads, computer vision) and interpretability is irrelevant. For other use-cases, interpretability is important because the consequences of getting it wrong are serious (e.g. medical diagnosis). There are also use-cases where interpretability is mandated by law to ensure fairness and to avoid potential discriminatory practices (e.g. loan approval).
This limits the specific machine learning algorithm that can be used. The image below shows various machine learning algorithms and shows that generally there is trade-off between the interpretability and the accuracy of the algorithm.
How this impacted businesses using Machine Learning
In many industries there are regulatory requirements that restrict the algorithms and data that could be used for machine learning. In these industries companies would use either Decision Trees or Regression Analysis for algorithms. As regards data, it is important to note that as well as not using fields that indicate race, religion, etc, other fields may be de facto proxies for these fields. This is not always obvious and highlights the importance of being able to analyse how the algorithm came to its decision. For example, zip codes can be a proxy for ethnic background and depending on the use-case may not be appropriate to use.
Even if every effort is made to comply with existing laws and regulations, companies should ensure that the machine learning tasks is seen as fair and non-discriminatory. Otherwise, it is likely to picked up on and can cause negative publicity, as in the example below:
Amazon built a machine learning model to decide the optimal areas that could avail of same day delivery. This had unfortunate results, for example one district in Boston was omitted while all surrounding districts were eligible. This district had a higher percentage of minority neighborhoods. Amazon made their decisions based on models that looked at the concentration of existing users and distance from distribution hubs. Although it was generally accepted that no profiling of customers involved, Amazon were forced to defend their decisions.