Iman Khwaja’s article “Choosing the Path of Most Resistance: Metacognitive AI Literacy and Mitigating Epistemic Risk” reimagines metacognition as both an individual tool and a component of AI literacy. Metacognition, Khwaja argues, can act as a critical pathway to mitigating epistemic threats posed by AI. Read the full article here in CJAI, Volume I, Issue 3.

What inspired you to write this paper?

This was my first ever research paper actually. I’m an independent researcher, and it was purely a matter of interest, but was also heavily inspired by witnessing an increasing dependence on AI tools while I was finishing my degree.

This alarmed me and I saw it as an aperture for a whole range of threats which could evolve to become quite serious. As far as I understood at the time, and still do, recognising major catastrophic existential risks is a high priority for mitigation efforts. But I feel very strongly that it is important to highlight the potential for epistemic risk to emerge as a significant threat of its own kind.

You identify several specific epistemic threats in your article. Which of these risks are you most concerned about and why?

The most concerning risk for me is what I call “perceptual fragmentation” in my paper. In a worst-case scenario, this would look like AI technology becoming sort of like an extension of ourselves, making it very difficult for us to discern between ourselves and AI-attributed knowledge. Maybe it’s an extreme case, but more generally, I’m concerned AI could affect our collective ability to perceptually engage with reality.

In a downsized context, that kind of threat would look like changes to our epistemic capabilities and values that affect our ability to respond organically to new challenges and new situations that emerge because of technological progress.

What would you say to a student using AI to complete their assignment? How do you convince them to care about epistemic risks?

I think the best way to convince anybody of a real risk is to let them see how it affects themselves. A lot of institutions are returning to pen and paper for examinations after we’ve seen a surge in dependence on AI tools. It probably would have been very interesting for a lot of those students to see their instinct to reach out for assistance in that context and then not be able to have it, and become aware of how it affects their performance.

Especially for older students, I think it would be really interesting to see the contrast in the content of their work pre-AI dependence and post. I think that would be a really great way to alert people to the risks posed to their own cognitive wellbeing.

I also acknowledge that AI tools offer a lot of value to students with various needs. But generally speaking, I would say that AI should be used as a last step in the process of any kind of epistemic challenge. Using it isn’t so much the problem, it’s more about at what point you use it in the process.

How do you use AI tools yourself in your work?

I try my best to refrain from engaging too much with AI tools. But my research interests have also led me to experiment with a lot of AI tools just to see both what risks they pose and what value they offer us. For my own work, I have consulted with AI tools for my editorial needs, like with referencing or summarizing large bodies of literature. Because I didn’t have a research background, it’s also been quite helpful in terms of the logistics of it. It was intimidating to learn how to screen literature, properly reference it and put together quality research articles by myself, and AI tools have been supportive for me with regards to condensing large amounts of information that can feel overwhelming to sift through. However, I do try to be as metacognitive as possible when engaging with AI. For me, that would look like fully completing a piece of work, at least conceptually, before I outsource any part of the process to an AI model. Then I know the ideas belong to me and I can trace them back to my own thought pathways before I ask for further input.

The metacognitive framework set out in your article can appear quite demanding for AI users and, as you write, “much of the success of this cognitive protection is reliant on individual willingness, ability and circumstance”. So is there also a need to design AI tools that promote metacognition?

I hope so. I think that a metacognitive AI would do at least half the heavy lifting of preserving our cognitive wellbeing. For an AI to be metacognitive, though, it’s also critical that we understand how model architectures contribute to cognition in AI, and I don’t think we’ve quite gotten there with understanding it yet. Eventually I think it would be ideal if we could embed the self-reflexive aspect of metacognitive practice into AI models to be able to discern between our own thought pathways and the reliability of model outputs.

If you were to write a follow-up article, what would the topic be?

I think a follow-up article could focus less on risk identification and address pathfinding opportunities to explicitly optimize AI to be metacognitive, or at least to promote our cognitive welfare. Since writing this article, my interests have shifted towards matters of cognitive architecture in AI models specifically, and I am curious about the very topical discussions being had about human, animal and AI consciousness, and ultimately finding a definition for it.

What is one thing you want readers to take away from your article?

I would advise readers to nurture their own relationships with knowledge and trust. These are going to radically shift as we develop stronger relationships with AI – things like what we consider intelligence to be, our shared epistemic floors, and our understanding of the world. So I think it is important to recognise that inquiry into these things, even on a personal level, is valuable. These are questions I think we’re all capable of asking, and we should continuously be asking, even in the face of unprecedented. I would hope that readers recognise the value of their own epistemic authority and continue to nurture it.

Footnote

Kathryn Conrad / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/