Teams are trying to understand where it genuinely improves experimentation programmes, where it saves time, and where it currently creates more promise than practical value.
There is no universal answer yet. The technology is evolving quickly, teams are still figuring out how to integrate it, and results are mixed.
But some patterns are emerging.
AI is now being used across almost the entire experimentation lifecycle, from research and opportunity identification through to experiment design, prioritisation, analysis and knowledge management.
There is an increasingly clear distinction between where AI is delivering tangible value and where human judgement remains difficult to replace.
Where AI stands across the experimentation lifecycle today
AI leads on the left. People lead on the right.
| Strongest AI carries the work |
Promising AI finds, people decide |
Mixed AI speeds up parts |
Human-led AI informs, people own |
| Knowledge management Repetitive work Design variations |
Research | Experiment build Results analysis |
Ideation Prioritisation |
Promising
Teams are connecting AI to customer, product and behavioural data to identify potential experimentation opportunities.
This can involve analytics platforms, research repositories, customer feedback, surveys and session recordings.
Instead of manually searching multiple sources, AI can help surface patterns and opportunities worth investigating.
But identifying an opportunity and knowing what to do with it are different problems.
Human-led
This is one area where I think humans are currently stronger than AI.
AI can generate ideas quickly, but speed and volume aren't necessarily the same as quality.
Some of the most interesting experimentation ideas come from experiences outside a company's existing data: an interaction pattern seen in another industry, something experienced while travelling, a competitor's approach, or an idea borrowed from an unrelated category.
Humans can combine those abstract experiences into something new.
There is also nuance in understanding customers that can be difficult for AI to reproduce, particularly around:
| Emotional motivations | Language and messaging |
| Cultural differences | Customer anxieties and objections |
| Brand history | The difference between what customers say they want and what actually motivates them |
This is particularly important when experimentation involves copy.
A technically correct message isn't necessarily persuasive.
Understanding why a particular phrase resonates with a particular audience can require cultural context and lived experience that isn't easily captured in a dataset.
AI can still help once a human establishes the initial direction, by expanding variations, challenging assumptions, exploring adjacent ideas and structuring hypotheses.
The more useful workflow may currently be:
| Human insight | → | AI expansion | → | Human judgement |
The human provides the conceptual leap. AI helps explore the territory around it.
Human-led
Prioritisation is another area where AI is being explored heavily, but results appear more mixed.
On paper, it seems straightforward: give AI your backlog and relevant data, then ask which experiments should run first.
But prioritisation isn't purely mathematical.
A product launch, strategic initiative, important customer segment, technical constraint, resource issue or seasonal event can all influence the decision. Much of that context may not exist in the underlying data.
AI can process the information it has, but it cannot reliably apply context it doesn't have.
Strongest
One of the most compelling applications is turning an experimentation repository into something genuinely useful.
Experimentation knowledge often becomes trapped in spreadsheets, decks, documents, dashboards and individual people's memories.
AI can make that information easier to structure, tag, search and retrieve.
A well-designed knowledge base can answer questions such as:
Have we tested this before?
What did we learn from similar experiments?
Which hypotheses have already been explored?
What happened with this audience?
What previous evidence should inform this hypothesis?
This creates something more valuable than a database of old experiments.
It creates organisational memory.
Strongest
AI is particularly effective when the task is repetitive and well defined.
Examples include:
These tasks matter, but they don't necessarily require the most human creativity. Reducing the time spent on them creates capacity for more valuable work.
Strongest
Another area showing significant promise is design.
Teams are connecting AI with tools such as Figma and established design systems to accelerate experiment variations. This works particularly well when strong foundations already exist. A good design system gives AI established components, patterns, brand rules, typography and reusable elements to work within.
Rather than starting each variation from scratch, designers can accelerate the journey from:
| Idea | → | Design concept | → | Experiment-ready variation |
The result isn't necessarily replacing the designer. It's increasing experimentation velocity.
Mixed
Taking the design and actually building the experiment is where the experience appears less consistent.
AI can generate code and accelerate implementation, particularly when the design and underlying system are well structured.
But experimentation involves more than generating code. Technical constraints, legacy systems, analytics, responsive behaviour, edge cases, tracking, QA and platform limitations can all introduce complexity.
AI can accelerate parts of the build process, but doesn't necessarily eliminate the engineering work required to make an experiment production-ready.
This will likely improve as models and tools become more capable.
Mixed
Post-experiment analysis is another area where AI is useful, but limitations remain. AI can summarise results, identify patterns, surface segments and structure reports.
But answering:
"Why did this experiment produce this result?"
is much harder.
Important context may sit outside the experiment data. Perhaps the brand recently changed its positioning, a competitor launched something new, market conditions changed, or the organisation previously tested a similar idea.
AI can only reason from the information available to it.
When interpretation becomes highly contextual or opinion-based, practitioner experience and business understanding become particularly valuable.
Across the experimentation lifecycle, a pattern emerges.
| AI is particularly suited to tasks that are | Humans remain particularly important for |
| Repetitive Structured Information-heavy Pattern-based Standardised |
Judgement Context Strategy Business knowledge Creativity Interpretation Cultural understanding Abstract thinking |
This boundary isn't permanent. AI models are improving rapidly, and today's limitations may not remain limitations forever.
There is a temptation to frame AI's impact as:
"AI will automate experimentation."
The more immediate opportunity is changing how practitioners spend their time.
If AI handles report creation, documentation, repository maintenance, tagging, summarisation, repetitive research, design production and parts of implementation, practitioners can spend more time:
The goal isn't to remove humans from experimentation. It's to remove the work that prevents them from doing their most valuable work.
Why do so many experiments come back inconclusive?
Our guide shows how to design experiments that reach a clear answer, and the foundations that help AI and experimentation work together.
Download the guideThe current state of AI and experimentation is evolving rapidly.
Each generation of models expands what is possible, particularly as they become better at reasoning, working with structured data, using tools and operating across systems. Some current limitations will therefore be temporary.
The more useful question for experimentation leaders isn't:
"Can AI automate experimentation?"
It's:
The strongest teams may not be the ones that automate the most. They may be the ones that understand where automation creates leverage and where human judgement remains the differentiator.
AI is already changing experimentation, but perhaps not in the way early hype suggested.
The most tangible benefits today appear to be reducing repetitive work, structuring information, maintaining organisational knowledge and accelerating design and certain production tasks.
Other areas, particularly ideation, prioritisation and interpreting why experiments succeeded or failed, remain more dependent on context, creativity and human judgement.
That balance will continue to shift as the technology improves.
But experimentation has always been about more than processing data.
It's about understanding people, understanding businesses and making decisions under uncertainty.
For now, those are still very human problems.