Responsible AI: What Businesses Should Know Before Adopting AI-Driven Tools
Can AI be too good to be true? Questions a business user should ask.
It used to be that the only places that we would expect widespread use of machine learning supported decision-making was in tech companies. We know that Amazon has used AI to help screen resumes¹ and that Netflix uses a machine learning driven recommender system to guess what other shows you would like to watch². This is changing.
Machine learning models are now embedded in our banking system in the form of automated loan assessments⁴ and in our criminal justice system through generating risk scores to inform sentencing⁵. Beyond these custom-made models, more organizations from law enforcement⁵ to retail⁶ are buying software that are silently driven by AI.
As these tools become widely adopted, business leaders need to be asking the right questions of their software providers, data scientists and machine learning specialists.
When poorly designed, deployed and integrated, machine learning tools can have serious consequences.
Serious consequences? Like what?
Discriminatory bias can occur if the application of your AI-driven tool assesses or make decisions on people or communities. Without you being aware, a machine learning algorithm can learn biases from the training data. In the past, this has lead to resume screening tools that are biased against women¹ or crime prediction models that lead to over-policing of ethnic communities⁵.
This can happen even with the best of intentions. Are you a university trying to leverage machine learning to better target recruitment activities? Are you a non-profit leveraging big data to identify communities for funding? If history has been biased against a certain population then the data you are training your model with reflects the outcomes of that bias. If you are not careful, you can be perpetuating historical biases even with the best of intentions.
Concept drift happens when the conditions that underpin your machine model changes leading to your predictions no longer being accurate. The world we work in is fast-changing and humans adapt our decision-making based on these changing conditions. If I am a corporate campus recruiter, I could adapt my resume screening criteria based on my knowledge of a university program’s curriculum change that led to different GPA distributions.
A trained model would ignore these differences. Over time, the difference between the world when the model was trained and the world the model is making predictions on widens. At best, business users notice the difference and adapt accordingly. At worst, a business could lose its competitiveness by entrusting a core function to an outdated model.
Every model (or sets of models) has its limitations in that they are highly accurate in some conditions and could be randomly guessing in others. Usually, a model performs well when there is a lot of training data to learn patterns from and performs poorly in examples that vary too much from what it has seen before.
Not knowing the specific situations in your context where a model works well and when it performs poorly means that you cannot build business processes that account for its shortcomings. For instance, if we know that an algorithm is likely to perform poorly from a university that we have never recruited from, then let’s hand-screen candidates from new universities.
As a business user, what can I do to prepare my organization?
Before adopting an AI-driven tool, have a discussion with your AI experts.
- Ask about the model or the tool’s performance metrics beyond accuracy. Sometimes, you care about the accuracy of different groups, for example, whether ethnic minorities receive similar predictions as the majority ethnic group. In other situations, you may care less about accuracy and more that we didn’t miss any potential cases (recall).
- Put together a plan for monitoring the model post-deployment and identify triggers for re-assessing and potentially re-training the model. Circumstances change and unless you are actively monitoring the situation, you and your organization may not notice it. Identify metrics to monitor the distribution of the underlying population. Implement feedback mechanisms to monitor the accuracy (or other performance metrics) of your model. Periodically review this feedback and be proactive about maintaining your model.
- Understand your model’s limitations and address them in the design of the surrounding business processes. Stress-test your model under different conditions. How does your model perform? As an example, if it’s a tool that impacts your customers, is the performance under these circumstances acceptable? If not, plan for these limitations by placing human interventions where it is needed.
Monique is a former management consultant and a trained data scientist. You can view her other writing on data science and business at medium.com/@monique_wong
[1] Jeffrey Dastin. (October 9, 2018). https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G
[2] Libby Plummer. (August 22, 2017). https://www.wired.co.uk/article/how-do-netflixs-algorithms-work-machine-learning-helps-to-predict-what-viewers-will-like
[3] James Vincent. (January 12, 2018). https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai
[4] Kathleen Walch. (July 26, 2019). https://www.forbes.com/sites/douglasmerrill/2019/04/04/ai-is-coming-to-take-your-mortgage-woes-away/#195ee2b87567
[5] Karen Hao. (January 21, 2019). https://www.technologyreview.com/2019/01/21/137783/algorithms-criminal-justice-ai/
[6] AI Startups. (March 12, 2020). https://www.ai-startups.org/top/retail/
