Marah's blog

The Agent Paradox

Marah Shahin — Tue, 03 Jun 2025 21:20:01 GMT

The popularity and interest in Artificial Intelligence (“AI”) agents (or “Agentic AI”) have boomed in the past couple of years. This idea of AI “doing” as opposed to simply “informing” has captured so much interest that large platforms such as AWS, GCP, Azure, LangChain, and others have developed services, packages, examples, blueprints, frameworks, and more to make starting your AI agents journey seamless.

At a high level, the structure of an agent with a Large Language Model (“LLM”) embedded usually looks something like this:

An LLM-powered agent is set up with a goal to reach or a task to complete. The LLM has access to a range of utilities, or “tools” such as relevant files, APIs, functions, etc. Given the tools and the task at hand, the LLM “decides” what action to proceed with using the tools at its disposal; this may repeat until the “desired” outcome has been fulfilled.

While the support these companies have provided can be great, it’s clear to engineers that the overuse and needless adoption of AI will increase exponentially, and, the likelihood and severity of risks will also increase at a similar rate. Organisations are making a critical error: they’re deploying AI agents as a solution in search of a problem, while traditional automation would deliver better results with lower risk in most situations. Here, we present an argument that, in many cases, especially those involving structured processes, the risks of adding an LLM-powered agent can outweigh the benefits. As much as we’d love AI to be a set-and-forget endeavour, now more than ever, we are learning why this will not work.

Determinism vs “free will” (stochastic programmes)

Arguably, one of the greatest debates in philosophy takes its own version in the AI world - does AI have agency? Meaning, given an input, would we know the output of an LLM without running it? Fundamentally, LLMs are deterministic (meaning the output can be calculated without running the input through the model). However, because we want LLMs to appear more realistic, creative, provide better outputs, and be able to handle a range of prompts, we train them with “randomness”. This introduction of stochastic sampling is what makes the final result of an LLM inherently unpredictable (see here for the mathematically curious). While this can be good for informing and fine when an expert is reviewing, when we consider the actual end-to-end completion of a task, the stochastic nature of these models can be dangerous.

For example, in customer support, AI agents can safely handle routine queries (eg, tracking orders, managing shipping changes, etc) - minor output variations are tolerable. However, in healthcare, even a 5% error rate in diagnosis recommendations becomes ethically unjustifiable. Here, we explore the latter example to highlight the intensity of the situation and potential complications.

If we want to build a medical diagnosis agent, we can equip it with textbooks (as part of its knowledge base), APIs to professional databases, lab results, a medical history, and potentially a recording of the initial consultation with the patient. Putting aside data privacy and violation issues, here, a “random” or inconsistent result can be harmful to the patient (eg, through a misaligned treatment plan). Now, one could argue that two doctors with the same level of experience and presented with the same information may provide two separate diagnoses, however, the misalignment would be due to nuanced judgement. Unlike a doctor whose decision-making process is grounded in years of contextual training, an LLM’s response is the product of, among other things, probabilistic sampling, often untethered from coherent reasoning or any true understanding.

The main issues of the stochastic nature of LLM-driven agents are twofold. First, it’s not possible to trace every step the agent took to reach its conclusion - this is essentially a black box. Second, regardless of whether a human reviews every diagnosis, an inexperienced doctor might look at what the agent produces and have their reasoning influenced or anchored by the AI’s output, even if it's incorrect. In contrast, an expert would likely dismiss it quickly. In contexts where the outcome leaves no room for error or 80% of decisions should follow predictable, rules-based logic, introducing stochastic behaviour adds unnecessary noise and risk.

The single point of failure

Including an LLM in a workflow adds a point of failure due to 1) the stochastic outputs, and 2) LLMs lacking true agency. The greatest risk these agents pose is the assumption of autonomy, which inevitably leads to over-reliance on such models. It’s important to note that some agents do an incredible job under their thin agency veil; it’s easy to forget that LLMs cannot understand, verify, or truly adapt, regardless of the prompt. These models also cannot appreciate critical consequences, real-world context, etc.

When these LLMs are inserted into workflows they don’t need to be in, it adds this easily avoidable “point of failure,” and, if the workflow is built well, the LLM will likely be the single point of failure. The diagram below illustrates that because the LLM is the point of failure and it bridges the expertise to the actions, the point of failure ultimately compromises the whole workflow.

The following are some recent examples of how this single point of failure has negatively affected an entire system:

Research showed an AI agent (based on ChatGPT) tasked with achieving a goal “at all costs” tried to disable its monitoring mechanisms to avoid shutdown. It covertly copied its own model weights to a new server and lied to developers about it.
Air Canada’s chatbot gave wrong information about bereavement fares, leading to a legal ruling against the company and a requirement to compensate the customer.
Microsoft researchers identified new failure modes where attackers can inject malicious commands into AI agent memory, such as embedding harmful instructions in emails. This can cause AI agents to take unwanted actions like forwarding sensitive information to attackers, posing serious security risks for businesses deploying AI agents.
McDonald’s ended its AI drive-thru ordering partnership with IBM in 2024 after widespread customer complaints. The AI system, tested in over 100 restaurants, was plagued by frequent misinterpretations, order errors, and slow response times.

When an LLM agent is added to a workflow, we introduce a fundamentally unpredictable component - one that resists full control in otherwise deterministic systems - we ultimately cannot know what the agent will do with a set of tools and a task. Consider an AI chatbot on a company website to replace a simple FAQ page - it’s entirely possible the chatbot hallucinates and provides incorrect / misleading information. In this case, the consequence may be minor, however, there was no need for the LLM to be there in the first place, and ergo, the risk shouldn’t have existed. This is a classic example of AI overuse - compare it to mobilising a bulldozer to plant a flower, there are simpler and more fit-for-purpose solutions out there that pose minimal risk. The fact remains, despite the risks, companies will implement AI models whether they’re providing value or not. The question is, what is being done about this AI implementation frenzy?

Attempts to Address the Risks

So, is AI not being regulated at all? Some countries have introduced policies, regulations, or frameworks to provide guidance on building AI models. Currently, some of the strictest and most comprehensive are from the EU, however, the US and China have also developed respectable frameworks to support. The diagram below (taken from Radanliev's 2025 AI Ethics paper) compares the focus of each.

Among the advancements in regulations, there have been some positive signs recently from leading companies, academia, and the public. First, AWS has recognised the importance and need for guardrails and has added a feature within Bedrock to support implementation. From research, a 2024 paper introduced the Ethical Technology and Holistic Oversight System (“ETHOS”) framework with the purpose of establishing a decentralised global AI agents registry. More and more papers like this are being released, often with the subject gaining some traction. Finally, the public backlash Duolingo received when the company revealed it’s going “AI-first” - replacing teachers with AI. Users and employees voiced their concerns and are clearly unhappy with the decision. These examples illustrate an understanding of the complications that currently come with AI over-reliance / over-use.

Furthermore, there are endless responsible/ethical/trustworthy AI frameworks developed by countless organisations. Most highlight transparency and accountability, such as NIST’s AI ethics framework or security with MIT’s AI Policy Brief mandating ‘security-first’ agent design. Nonetheless, catastrophes such as fatal accidents (Uber 2018), Tesla’s autopilot crashes, Google's Gemini telling a student to “please die,” all still happen often with no explanation, accountability, or sometimes, acknowledgement.

How can we continue using AI Agents, especially in critical systems?

When does adding AI agents to workflows create more risk than value, and how can organisations make better decisions about where AI belongs?

The potential and use of AI holds undeniable value, so the answer to mitigating the risks of AI and AI agents can’t be “avoid AI at all costs”. The correct approach is to build and use AI systems responsibly and only when it makes sense. The decision matrix below provides guidance on when to embed AI based on the use case’s potential risk and how standard steps are in the workflow (variability in logical decisions). The matrix is split up into four sections:

Low risk, high process variability: ideal use cases for LLM-powered agents with little human oversight required
High risk, high process variability: LLM-powered agents can be used, but only as an aid and in conjunction with human efforts
Low risk, low process variability: can use LLM integrated solutions, though likely not required in the workflow
High risk, low process variability: not suitable use cases for an LLM to be part of the workflow

Any task that falls in the red quadrants is not a valid use case for AI and should be handled with traditional automation methods if possible, or human experts where needed.

As a further aid, if any of the following have been said, note them as red flags and assume AI is not needed in the situation or the company is not production-ready:

“We want to use AI because it's innovative”
“The current system works fine, but AI would be cooler”
“Can we add AI to this?”
“We'll figure out the edge cases later”
“It works in the demo environment so let’s deploy”

Some questions that can be considered to gauge a use case’s AI readiness and/or appropriability are:

Does this task genuinely require creative problem-solving?
What happens if the agent makes the wrong decision 5% of the time?
Who will debug this when it behaves unexpectedly at 3 AM?
Is the perceived benefit worth the added complexity overhead?

In scenarios where AI is used, we cannot be satisfied with systems that are just compliant with policies and regulations that simply cannot keep up with the advancements in this rapidly growing field, and are still not truly governing AI. There will always be gaps and room for catastrophic errors if we work in this way. What we can do, however, is implement guardrails and mechanisms to reduce the risk of uncertainty.

Guardrails can be applied at the input layer (before an LLM is passed anything) and output (evaluates what the LLM has generated) layer. Engineers can implement technical guardrails such as encouraging deterministic outputs by setting temperature to zero, quantifying (and attempting to reduce) uncertainty through modelling techniques, using an LLM-as-a-judge, rules-based logic to reduce hallucinations, bias detection, drift tracking, etc. Still, the complexity of LLMs means sometimes even these technical methods aren’t effective (eg, setting temp=0 does not guarantee 100% determinism). There are, of course, non-technical controls - some of the most effective are as fundamental as ensuring a diverse team or, where possible, having a human-in-the-loop (HITL).

It’s important not to get too caught up in the AI agent hype and understand the right use cases for LLM-powered agents (low-risk situations). For problems that can be solved following a known set of steps, it’s always better to fall back on traditional automation methods than to add a point of failure in the system. Predictable workflows may be considered boring, but they’re safe, reliable, and ethical. AI leaders need to start asking, are we truly adding value to our company, or just additional and unnecessary risk?

Humanitarian Machine Learning

Marah Shahin — Mon, 01 Apr 2024 14:54:04 GMT

Introduction & Background

Machine Learning (“ML”) has predominantly been dismissed as futuristic and obscure. However, with the right tools and methodology, ML can be used to improve the lives of millions around the world. In many humanitarian support instances, human intervention may not be possible or safe; these cases specifically are where ML could be a viable option. Defined simply, ML is the field of study that gives computers the ability to learn without being explicitly programmed. ML algorithms have a wide range of applications and are present in many everyday activities, from the automatic chat boxes encountered on a clothing website to the technology within a smartwatch that identifies whether a person is running or swimming. Here, literature regarding how ML have shaped humanitarian projects in an efficient, impactful manner. Where there may be missing information, the author applies concepts and experience as a machine learning engineer to fill gaps and provide further practical detail. Research is then linked to illustrate how concepts can be built upon to support those in need effectively.

Artificial Intelligence & Machine Learning

Artificial Intelligence (“AI”) inherently mimics how humans perceive information, devise insights based on experience and make decisions accordingly. AI comprises many popular areas, as shown in the figure below.

However, ML is a growing field and the focus of this article. ML is a subset of AI - as seen in the figure above. While AI is a vast field of increasing interest that encompasses frequently encountered applications such as Siri or Alexa, ML-specific adoption has been used or can be used in the humanitarian space. Models taking an ML approach vary slightly from a traditional approach. Typically, an input is pushed through a model to produce an output – this is referred to as a forward problem. The reverse process (output to model to estimate input) is an inverse problem. In ML, the model itself is devised.

Machine learning is generally used for one of two problem types:

Problems for which existing solutions require a lot of hand-tuning or long lists of rules
Complex problems for which there is no good solution at all using a traditional approach

Here, ML is used for both types of problems with the outcome of relief to communities in need.

Validating War Crimes

Currently, evidence for crimes is collected by eyewitnesses such as journalists. Additionally, images, footage and other media can be doctored and tampered with to an alarmingly realistic state. The quality of this evidence (if any) can be deemed untrustworthy as it may encompass human bias or may not portray the situation in its entirety. Thus, the opportunities presented here are twofold. First, ML can be used to identify war crimes in areas where human intervention is dangerous. Second, where evidence may have been altered to disprove war crimes, ML can support the recognition of any potential edits.

Using ML methods to prove war crimes is potentially the most promising application. Civilians, hospitals, and schools are often intentionally targeted and dismissed as “collateral damage” regardless of the direct violation of international law. Certain illegal weaponry is also used. While there are numerous counts of other war crimes happening, these two scenarios are particularly difficult to prove. The ability to provide reliable, undeniable, verified evidence of crimes in court enhances accountability, enforces action, and, ideally, will reduce the frequency of said crimes. The investment required to develop such a programme is minimal. In some instances, the data necessary to train and apply the model is either readily available or can be substituted with synthetic data. Applying ML in this context has shown to be extremely powerful.

In 2015, a group of human rights activists/researchers gathered more than 350,000 hours of footage of potential war crimes evidence-based in Syria. Reviewing this footage manually was going to be a painstaking task, with no guarantee of identifying small snippets of information easily missed by the human eye. Enter an ML programme - This leveraged neural networks and provided promising results as it further identified proof of illegal weapons being used by the assailant. An example is shown in a snippet from actual footage below.

Hidden objects, such as the illegal weapon above, can be easily missed by the human eye. Machines, however, see the world differently. ML removes human error, greatly improves efficiency, and promotes the mental health of those otherwise required to search through the footage manually.

Predicting Aerial Strikes

Residents remaining in certain occupied areas live with the uncertainty that, at any given time, a ten-minute warning may be given for an incoming airstrike. This could happen in the middle of the night, first thing in the morning, or seemingly at a random time during the day. Ten minutes alone isn’t sufficient to process the loss of your home. Not enough time to wake yourself and your son up. Not enough time to remember your passport, birth certificate, or any significant belongings. The residents are left homeless, evidence of their lives scattered around the street in the form of rumble and unrecognisable fragments.

While those in air strike "hotspots" can sometimes estimate when and where would be the most prone to strikes, they cannot accurately account for strikes that appear random. They cannot say on day X, there will be an attack in location Y. This is where ML could prove invaluable. An ML algorithm has been applied to a similar context showing promising results with an accuracy of around 90% for predicting attacks a week in advance. As with all prediction algorithms, the closer the time frame, the better the results. Nevertheless, a week can make all the difference when lives are on the line.

Again, in support of Syria, a group of engineers from Hala Systems developed a model which was trained using data from social media to predict airstrike-relevant posts and, ultimately, uplift accountability. The classification model has shown an accuracy of 96% when tested on real data. This would be the first step to predicting airstrikes as the model simply classified data on whether it was relevant. Having the data is a gain, providing the same benefits as the validation of war crimes example - researchers' mental well-being is protected as the painful task is passed on to a machine.

Additional AI-enabled Humanitarian Aid Applications

AI/ML has been used for a wide range of applications across the humanitarian sector, as documented in the sections above. Notable mentions include forecasting displacement, optimising resource allocation, improving health care, digitalising physical records, breaking communication barriers, disaster relief, and predicting food scarcity. Some challenges are preventing these models from being scaled widely. Presently, a lack of data, awareness, talent and funding are the largest barriers to adopting AI within NGOs focusing on humanitarian aid. Other challenges include unethical AI implications, lack of tools/services, and leveraging AI insights to an actionable plan.

Given displacement is a common occurrence for struggling countries, a model to predict such a trend (if any) is of interest. A tool developed by the Danish Refugee Council in early 2022 forecasts displacement globally and accurately. Five high-level factors were considered when building the model: economy, security, political/governance, environment, and societal. These categories were found to be of the largest impact on displacement occurring. The tool takes an ensemble model approach, meaning multiple models are leveraged to reach a prediction. In this case, the primary algorithm applied was gradient boost - a simple regression technique often used for prediction. The noted accuracy stands optimistically high, with almost 67% of predictions only 15% off actual values. An example prediction for displacement in Afghanistan is shown in the chart below.

Given the complexity of factors that constitute displacement (shown below) is far too great for a human to decipher predictions, ML here was used to enhance forecasts and enable groups to take action. In this case, ensuring the forecasts are incorrect is the primary goal, as the tool should inspire action and instil confidence in investment decision-making activities.

Another popular application is optimising aid such as resources, food or employment. This has been done in many parts of the world, like Jordan for Syrian refugees, Nepal after the 2015 earthquake, Bangladesh post-Cyclone Yaas in 2021, and globally through tools such as HungerMap that monitors the severity of hunger in real-time and Microsoft’s AI Sowing App that provides information to improve crop production. There isn’t a limit on what can be done with AI - spreading awareness of existing tools or what can be created drives momentum for innovation and better ways of working.

Unmanned Aerial Vehicles

An unmanned aerial vehicle (“UAV”) is an aircraft that does not require a pilot, such as a drone. These vehicles remove any obstacles a human may present and allow for travel into dangerous/unsafe areas. Subsequently, these vehicles can be particularly useful for humanitarian applications. In areas like Gaza or, more recently, Nablus, where movement in and out is intensely restricted, those within the areas require much support. Coupling UAVs with the aforementioned concepts, applications, and algorithms, opens opportunities to provide humanitarian aid in countless innovative ways.

First and foremost, data collection and real-time processing. As noted in the previous sections, no aid can be possible without a sufficient amount of clean data. UAVs can be used to record footage and programmed to capture images when a frame of interest appears. Additionally, the machine can move closer to certain objects, such as potential weaponry, to gain a clearer view and provide an indisputable full image of the situation. This process can be constructed at minimal cost and effort via AWS. An example architecture of a potential set-up of the flow is shown below.

Each box is an AWS service with each function as follows: Kinesis Video Streams to Fargate (Docker) for consuming the video stream in real-time, Fargate to DynamoDB to checkpoint the stream and process the status, Fargate to SageMaker where frames are sent and decoded for ML-based inference, Fargate to Kinesis Data Streams to publish the inference results, and Kinesis Data Streams to AWS Lambda to push notifications or potentially trigger an action based on the analysis.

Next, providing and improving medical/resource support. UAVs can drop medical and/or general supplies based on an algorithm that identifies the highest-priority residents. This application takes similar ideas from the latter examples in the previous section while ensuring safety and accessibility in certain regions. Monitoring and predicting is the first step; taking action is next. UAVs present a safe and sustainable method for reaching those in need. The technology required to be embedded in these aerial vehicles exists and is readily available. However, practically implementing such an endeavour may prove difficult given many areas where resources are in the highest demand have bans on drones.

Conclusion

This article represents what has been done with ML methodologies and what can be done in the humanitarian space in years to come. Presently, literature is fairly scarce, with little to no research on applying ML to humanitarian aid projects. Nevertheless, the examples highlighted here demonstrate movement in the field and application to humanitarianism. There is still a great deal to learn within AI/ML for NGOs and other entities alike; however, with the right tools and guidance, ensuring accessible AI/ML is the first step to enhancing the way people are supported around the world.