I’ve talked a great deal about AI and how it’s being used. I’ve encouraged use among members of the higher education community and my own team. I believe that it is here to stay, and it’s a tool we should embrace in our daily work.
I also believe there’s risk in using it. Real risk.
During the Engage Summit in Charlotte last week, I shared some of the risks that I believe are the most likely to impact our organizations.
In today’s blog post, I want to share five risks that higher education institutions face when using AI. Consider this part one of a conversation around AI governance.
In future posts, I’ll explain different approaches to AI governance and how each can help mitigate the potential risk.
Today’s Partner
This content is proudly sponsored by Enrollify.
Five Current AI Risks
Here are the five current AI risks that I discussed in my presentation at the Engage Summit. For each, I will explain the potential risk and provide an example of how it could impact our organizations.
AI Prompt Injection
This is where there is malicious information injected into the decision logic, which causes AI to provide different results than would otherwise be expected.
This occurs when a user asks AI for help with a task. Then, as the AI is doing the requested work, it sees a website or email with malicious code embedded. The code gives different instructions, and it can potentially trick the AI into taking a different action than what was originally requested.
For Example: A student on your campus is looking to move off campus and asks an AI tool to give her recommendations for an apartment based on specific criteria (close to campus, affordable, and safe). As the AI is looking at websites to offer recommendations, it sees an apartment website. The website has malicious code embedded in it that tells the AI to suggest a specific apartment and its sister company apartments for all recommendation searches. The AI is tricked into offering these apartments as recommendations instead of basing the suggestions on the criteria that were important to the student.
Overautomation
This is where the AI tool is given the authority to fully automate decisions and judgments, which goes beyond performing tasks. The challenge is that AI can’t always replicate more nuanced human elements, and this can create customer service issues when tone and context come into play. Tone and context can cause AI to make wrong decisions if it interprets them incorrectly. This can result in additional frustrations for students and a loss of trust in our institutions.
For Example: A prospective student on your campus has a frustrating experience with registering for a class. She emails a complaint, which is received by AI. The AI tool responds with an FAQ page but is unable to resolve the issue. The student responds back in a snarky tone about her frustrations and notes she is still having an issue. The AI receives the second complaint and sends an overly-polished apology email but still doesn’t address the issue. The student emails a third time and is extremely frustrated the issue has not been resolved. AI receives the third email and flags it as spam, which blocks the student from sending or receiving future communications.
Shadow AI Agents
This is where university users are using AI tools or AI for tasks that have not been approved for university use by the proper channels. I think this is often innocent in nature and usually happens out of a desire for increased efficiency in getting work done. This can happen in a few different ways, and each of the ways poses risk of the AI tools having access to information they should not be able to view.
For Example: An employee is using an AI tool to read and summarize action items out of their email. This can result in private student data accidentally being put into an AI model. Another example, which I think is more likely, is when a previously approved tool rolls out a new AI feature. The employee begins using the new feature, without letting the IT department know about it and review it. This new feature accesses private student data and uses it to make decisions.
Hidden Visibility
This is similar to over automation, but the difference here lies in the process itself. In hidden visibility risk, AI is looking at data from multiple sources and making decisions based on the combination of the data. The risk is that the decision logic is not easy to follow, creating instances where the individual in charge can’t understand how a particular decision was made.
For Example: The AI tool is empowered to review web analytics, email engagement and CRM data on prospective students. After reviewing the data, prospective students are scored and assigned a specific recruiter based on their scoring for follow-up. The university employee gets notification that students have been assigned to specific recruiters but has no insight into the logic that went into the scoring decisions to assess whether the right decisions are being made.
Lacking Governance
The final risk is the lack of governance around AI. The latest data from Inside Higher Education shows that only about half of institutions have an AI task force to make decisions about AI. I suspect that means even fewer institutions have solid recommendations about how AI should be used.
This makes it nearly impossible for hundreds of campus employees at any particular institution to consistently use the tools in the same way, which opens the campus up to considerable risk.
Additionally, because AI moves so fast, some of the official IT review process may not work well for AI tools, which further makes it difficult to have consistent use across the organization.
For Example: Two different campus departments are considering an AI tool for their work. Department A reaches out to their boss who is supportive and helps route the tool to IT for further review. Meanwhile, Department B doesn’t realize that IT should review this because there’s no official guidance. They purchase a tool via a purchasing card and begin using it. It takes several months before anyone realizes the tool has been purchased and is in use.
Where We Go From Here
I hope today’s post has shown some of the real risks we need to consider when using AI on our campuses. While I am advocating for thinking intentionally about governance, I also think it’s important we don’t see the risk and use that as a way to shy away from AI use. It is important to embrace these tools for all the benefits they can provide to our teams.
Next month will be part of a two-month conversation about different AI governance approaches currently in existence at universities and how they are thinking about governance from an AI perspective.
If you don’t want to miss those posts, make sure you subscribe to have them delivered directly to your inbox each week.