日本語
In earlier articles, I explored how AI is evolving from a tool we converse with into an agent capable of carrying out work. This article looks at what happened when that idea was tested during a real production incident.
The incident affected gifts.magnote.com, an e-commerce site we operate. A sudden surge in automated bot traffic placed excessive load on the server and eventually brought the site down.
The emergency response took approximately half a day. After the changes were implemented, the problematic traffic fell by roughly 70 percent compared with pre-mitigation levels.
However, the reduction itself is not the main point of this case study. The more important lesson was how we divided responsibilities between the AI agent and the human operator: how broadly the AI should investigate, and where human judgment should take over.
The incident: an outage before the cause was clear
At the outset, we knew only that the server was under excessive load and the site had stopped responding. The traffic spike was obvious, but its source, purpose, and request patterns were not.
We could have changed the server configuration based on assumptions and then monitored the results. That approach might have produced a faster initial response, but an incorrect diagnosis could also have introduced additional problems.
Instead, we decided to investigate what was actually happening before making changes.
Using Claude Code to analyze the server logs
We created a dedicated local environment for the investigation and used Claude Code to analyze the server logs.
At this volume, reviewing every log entry manually would have taken a significant amount of time. We therefore asked Claude Code to examine three areas:
- Whether traffic was concentrated during particular periods
- Whether the same request patterns were being repeated
- Whether any behavior differed significantly from normal traffic
The analysis quickly revealed two anomalies.
First, the site was receiving an abnormal number of add-to-cart and add-to-wishlist requests. Second, a large proportion of those requests originated from Singapore.
Unlike simply viewing a product page, adding an item to a cart or wishlist typically triggers session handling, database operations, and other server-side processing. When these requests are repeated at scale, they can place a substantial load on the server.
Cross-checking multiple data sources
We did not want to base the diagnosis on server logs alone. We also reviewed data from Google Analytics, Google Search Console, and Cloudflare, comparing request origins, request types, and timing across the different sources.
This was one of the most useful aspects of using AI during the investigation. Rather than showing it only one screen or one data source, we were able to examine related information across several services.
A conventional investigation might involve opening the server logs, Analytics, Search Console, and Cloudflare one after another, then manually comparing the numbers.
In this case, Claude Code analyzed the server logs while browser-based agents collected information from the relevant service dashboards. We then reviewed the findings together through an ongoing conversation with the AI.
Working across multiple sources in parallel significantly shortened the time required to identify the likely cause.
Blocking problematic traffic before it reached the server
Once the request patterns became clear, the next question was where to block them.
The site uses Cloudflare for DNS and CDN services. We therefore added screening rules at the Cloudflare layer so that problematic requests could be filtered before reaching the origin server.
The location of the block matters. If the server first receives a request and then rejects it, the rejection process still consumes server resources.
Filtering the traffic at the CDN layer reduces the number of requests that reach the server in the first place. Put simply, it is the difference between turning someone away at the front door and stopping them at the entrance to the property.
Other problems discovered during the investigation
The investigation also uncovered issues unrelated to the bot traffic.
The site had overlapping caching mechanisms, and some of the plugin settings were inconsistent. Caching is intended to improve response times and reduce server load. However, when multiple caching layers overlap or behave unexpectedly, they can make the site less stable and make incident diagnosis more difficult.
We therefore reorganized the caching configuration as part of the same effort.
The response ultimately involved three measures:
- Filtering suspicious external traffic at the CDN layer
- Correcting the site’s internal caching configuration
- Optimizing the requests passed from the CDN to the origin server
Together, these changes significantly reduced the volume of traffic reaching the server.
Results: roughly 70 percent less problematic traffic
The emergency response took approximately half a day. After the configuration changes, the bot-driven traffic fell by roughly 70 percent compared with pre-mitigation levels.
The site became stable again, but there was also an unexpected secondary benefit: the data in Google Analytics became much easier to interpret.
Previously, bot traffic had been mixed into the analytics data as noise, making it difficult to understand how actual customers were using the site. Once the unnecessary traffic was reduced, genuine user behavior became more visible.
The original objective was to reduce server load. Improving the quality of the analytics data was an additional benefit.
For e-commerce operators, this is an important point. Bot traffic does not only affect infrastructure. It can also distort the data used to make business and marketing decisions.
Looking beyond the immediate fix
Once we had created an environment in which the AI could examine the server, Cloudflare, Analytics, and Search Console together, it seemed wasteful to stop after restoring the site.
We therefore used the same workflow to review the site’s broader configuration.
That review identified a problem with the Google Merchant Center integration. Some product data was missing entirely. We also found discrepancies between product information maintained in internal spreadsheets and the records registered on the e-commerce site.
We reviewed which attributes were registered for each product, what information was missing, and whether data was flowing correctly between the e-commerce site and Google Merchant Center.
This made it clear that we needed a more systematic way to manage product information rather than correcting individual discrepancies as they appeared.
We are now building a product information management system, or PIM, to reconcile data across our spreadsheets, the e-commerce site, and Google Merchant Center. The goal is to create a single source of truth for product information.
That project will be covered in a future article.
Where AI support ends and human judgment begins
The main lesson from this incident is that AI can deliver more value when it is asked to investigate the surrounding system, rather than simply perform one isolated task.
If we had asked only for a particular setting to be corrected, that setting might have been fixed. However, the overlapping caching configuration and the missing product data might have remained undiscovered.
Incident response should not stop at removing the visible symptom. It should also examine the conditions that allowed the problem to occur or made its impact worse.
At the same time, not every decision should be delegated to AI. The human operator must still decide what the AI should investigate, which proposed changes are appropriate, and how much authority the agent should be given.
Determining that boundary is one of the most important practical considerations when using AI agents in production environments.
Use the tool—do not let the tool use you
Since working on semiconductor design for the PlayStation 2, I have repeatedly shared one principle with the teams I have worked with:
Tools are meant to be used; we must not let them use us.
No matter how capable a tool becomes, allowing it to dictate our thinking leads to complacency. We should not accept its output without question. We must continue asking whether the answer is correct, whether the assumptions are valid, and whether a better approach exists.
AI can perform a much wider range of tasks than previous generations of tools. That makes human judgment more important, not less.
Let the AI investigate broadly, but do not delegate the final decision.
This incident reinforced that principle. AI accelerated the analysis and helped us examine areas that might otherwise have been overlooked. However, deciding what to change, what not to change, and where to stop remained a human responsibility.
The next article examines the complete rebuild of a B2B ordering site originally developed in WordPress four to five years ago. The most important part of that project was not asking Claude Code to start writing code immediately, but spending enough time discussing and clarifying the requirements before implementation.
Read the next article: Rebuilding a B2B Ordering Platform with Next.js and Supabase
Shinya Fujimoto | Founder / Chief Strategist, Silicon Valley Japan Lab
Based in the United States for more than 25 years, Shinya draws on his experience as both an engineer and a business leader to examine how Silicon Valley technology trends can be applied to management and business development in Japanese companies.