• Author: Peter Vnuk
AI agents are rapidly moving from technology demonstrations into everyday business practice. Their main benefit lies in their ability to combine multiple steps into a single workflow. First, they retrieve the necessary information and scan relevant documents, then they use an available tool or internal system based on the instructions, and finally, they prepare an output for human review or approval.
Compared to a standard chatbot, an agent gets much closer to the actual work done within a company. It can help customer support find the right answer in documentation, prepare meeting briefs for sales representatives, and reduce manual switching between multiple applications for administrative tasks. The closer such a tool gets to a real workflow, the more its technical and operational design matters.
The initial pilot can be set up relatively quickly, provided the company has a well-chosen scenario, a limited group of users, and an environment where they can safely test data (even dummy or synthetic data) and typical queries. When transitioning into daily operations, the demo becomes a service that people rely on. The agent must respond predictably even under heavy loads, operate according to individual user permissions, and leave a sufficient audit trail for the team to troubleshoot any incorrect or unexpected answers later on.
You will learn:
The pilot phase is designed to show whether the agent solves a specific problem better than the existing process. In companies, this often starts with an internal knowledge base, customer support, or sales preparation, as these areas involve many repetitive queries and allow for easy measurement of time savings. In this stage of the project, the company primarily assesses whether the new tool actually helps employees and whether its answers hold up during daily work.
Technical requirements are addressed continuously, as even a pilot should work with realistic data, documents, and tasks. At this stage, it pays to keep a narrow focus: first verify the value of a specific scenario, and only then determine what infrastructure will be needed for wider deployment based on the results. A well-prepared pilot therefore tests a smaller, clearly defined process rather than a broad mandate like "let's roll out an AI assistant for everyone".
Production operations bring higher responsibility and higher user expectations. The agent gradually transitions from a testing tool into a widely available service that must handle multiple concurrent requests, work with up-to-date data, and maintain predictable performance even during peak times. There is also increased pressure on permission management, as the agent must only answer using sources that the specific user actually has access to.
Area | Pilot Phase | Production Operations Goal | Verify value and usability | Ensure a stable service for daily work Users | Small team, selected roles | Multiple teams or wider company deployment Data | Limited sample of documents | Up-to-date internal data and controlled access Performance | Headroom for testing | Response time, concurrency, and scalability Risk | Mistakes are for learning | Mistakes have a clear resolution process Operations | Simpler oversight | Monitoring, logging, versioning, accountability
As the user base grows, so does the way people perceive the system. They will forgive an experiment for a slower response or occasional inaccuracy, but for a daily tool, they expect a stable service they can rely on. This is precisely where it becomes clear whether the agent was designed as an isolated demo or as an integral part of business operations.
i
A pilot is meant to demonstrate the value of a specific scenario. Production must ensure that the same scenario holds up under daily use, heavier loads, and clearly defined accountability.
When planning an AI agent, discussions often focus on the language model, as it is the most visible part of the solution. In an enterprise deployment, however, the architecture surrounding it is what truly matters. The model interprets instructions and generates responses, the application layer manages the workflow, and connectors serve as secure links to company resources. Through these, the agent accesses documents, databases, or internal tools without having unrestricted access to everything.
In the final solution, the individual layers complement each other based on their role in the overall process. Identity management determines which data a specific user can access, while logging helps trace the path of a request retrospectively. The user interface, on the other hand, determines whether people adopt the tool into their daily work or start bypassing it. For a simple prototype, these components can be very lightweight, but in enterprise production, they form the foundation of reliability and security.
This is clearly visible in RAG (Retrieval-Augmented Generation) scenarios, i.e., working with an internal knowledge base. The agent first retrieves relevant sections of company documents and only then compiles an answer from them. In practice, this might involve a combination of internal guidelines, contract templates, and technical documentation, where manual searching would take significantly longer.
This increases the demands on infrastructure, as documents must be prepared for search before use. The system splits them into meaningful chunks, converts their semantic meaning into a format suitable for search, and stores them so that the agent can quickly find passages relevant to a specific query. While GPUs primarily drive inference speed, the database, storage, CPU, and network layer determine how quickly the entire journey from query to response takes place.
i
Bottlenecks are often found outside the graphics card and can occur during document retrieval or file access. For more complex agents, latency from external APIs, long context windows, and a higher number of concurrent requests add to the delay.
With the transition to production, the first challenge to emerge is request concurrency. In a pilot, a few people from a single team work with the agent, but a production deployment might serve sales, support, administration, and management all at the same time. For infrastructure sizing, the number of concurrent requests and the duration of individual tasks are therefore critical factors.
More complex tasks place a completely different load on the system compared to a simple summary of a single document. An agent might first search for information across multiple sources, compare it with internal policies, request data from another company system, and only then prepare a draft response for a customer. In such a scenario, it is crucial to monitor the entire workflow, as errors can occur during retrieval, tool calls, data interpretation, or the final response generation.
As the agent's role in business processes grows, the boundaries of responsibility also shift. While a pilot allows for fine-tuning and occasional inaccuracies, a production deployment requires clearly defined guardrails. In practice, this mainly involves distinguishing between when the agent is merely preparing a draft and when its output directly impacts a customer, an order, or an internal decision. In these situations, it must be clear beforehand whether the agent can proceed autonomously or if the result requires human approval first.
A full deployment therefore requires a dedicated IT team or an integration partner. They must understand the company environment well enough to ensure the agent works with the correct data, respects permissions, and fits into existing workflows. While infrastructure provides the technical foundation, the agent itself must become an integrated part of daily operations, not an isolated tool operating on the sidelines.
For initial testing, a powerful workstation or a small AI server is usually the most practical choice. The team can quickly test different models and work with real documents without immediately designing the target infrastructure for the entire organisation. In this phase, RAM, fast NVMe storage, and a GPU with sufficient VRAM for the chosen model and working context play a major role.
In production, the infrastructure must include a buffer, as the system needs to handle peaks, a higher number of concurrent queries, and gradual growth. Alongside raw performance, factors like serviceability, backups, long-term scalability, and the ability to distribute loads if a single server is no longer sufficient become essential.
| Scenario | Typical Solution | What to Focus On |
|---|---|---|
| Development and initial testing | Powerful workstation | VRAM, RAM, fast SSD, easy environment management |
| Internal pilot | Workstation or smaller AI server | Real data, response time measurement, quality testing |
| Team deployment | AI server with performance headroom | Concurrent queries, monitoring, access rights |
| Production operations | Server infrastructure, multiple GPUs, or hybrid model | Scaling, availability, backups, network, operational oversight |
| Sensitive data | On-premises or private infrastructure | Data control, auditability, security policy |
Configuration choices should be based on the specific workload. An internal assistant for dozens of employees has different requirements than an agent working with extensive documentation and long context windows. It is best to test real queries and measure response times before deciding on the final infrastructure.
i
Once the pilot demonstrates consistent usage, it is worth calculating the operational economics alongside the technical design. For stable workloads, a company can compare cloud, on-premises infrastructure, and hybrid models. The calculation should factor in monthly costs, hardware utilisation, expected growth, and data handling requirements.
Placeholder for TCO calculator
With AI agents, performance is difficult to assess based on the graphics card model name or a single configuration parameter. VRAM capacity has a major impact, as it must accommodate the model, the context, and some runtime overhead. As soon as the agent works with longer inputs or a higher number of concurrent requests, memory requirements grow very rapidly.
System memory and storage primarily affect data handling. In RAG scenarios, the system continuously processes documents and stores their representations for later retrieval. Fast storage helps when working with document databases and temporary files, whereas a bottleneck in the infrastructure can slow down the entire chain even if the model itself runs on a powerful GPU.
Model quantisation can reduce memory requirements and enable operation on more affordable hardware. For simple internal queries, this can be a practical solution, but for specialised, legal, or financial outputs, quality must be closely monitored. A faster response has limited value if the system begins to struggle with details or becomes less stable over repeated use.
How large a model do you actually need, how long will the inputs be, how many people will use the system concurrently, and what response time is acceptable for work? Without these answers, choosing hardware is a guessing game.
A production agent requires oversight just like any other business application, but with a greater need to track individual steps. A single user request can trigger document retrieval, internal system calls, result interpretation, and response generation. If the output does not meet expectations, the company must be able to trace where the issue occurred.
At the technical level, response time and throughput are monitored closely, as these are most noticeable to users. In operations, however, it is equally important to know GPU utilisation, memory consumption, and whether errors are occurring in the model, connectors, or document retrieval. For more complex agents, trace logs—records of individual steps—are highly valuable.
The financial aspect of oversight is just as important as the technical side, because without ongoing metrics, a company cannot see what running the agent actually costs. In the cloud, costs vary based on model usage and the volume of processed requests, while with on-premises infrastructure, the key factor is how well the purchased hardware is utilised and what headroom remains for peaks. Only with this data can you determine whether the system needs more capacity, better optimisation, or a simpler operational model.
In a production deployment, a company agent often works with information that should only be accessible to specific roles or teams. For example, if it draws from contract or pricing documents, it must follow the same rules as an employee working with them manually. A flaw in permissions is not just a technical detail, but a direct security risk.
For knowledge assistants, the basic principle is simple: the agent should not answer based on a document that the user themselves could not open. Therefore, identity and access management must be part of the design from the start, rather than an afterthought following the pilot. In practice, it also matters whether the system can retrospectively show which source an answer was based on and who had permission to access it at that moment.
The situation is even more sensitive for agents capable of triggering actions. Preparing a response to a customer or suggesting a change to an order can happen automatically, but actually sending it or making changes to a company system often requires human confirmation. A production agent therefore needs a clearly defined boundary between recommendation, draft, and actual execution of an action.
A pilot should start with a process that involves repetitive work and where results can be reasonably measured. In practice, this is often customer support, as it deals with similar queries and relies on existing documentation. An internal knowledge base or sales preparation can work equally well, provided it is clear which part of the work the agent is meant to speed up and how success will be defined.
Only after clarifying the specific scenario does the choice of model, database, and hardware come into play. In a pilot, it is more important to see how the agent behaves in a real working context than to immediately design infrastructure for the entire company. That is why it pays to start with a smaller, well-defined use case that can be tested on real queries and gradually expanded.
i
You Might Also Be Interested In
The test suite should include more than just sample questions designed to make the agent look good. In the first step, anonymised or synthetic data may suffice, but the pilot should work with representative company documents as soon as possible. Only then will it become clear how the system handles incomplete instructions, older document versions, inconsistent terminology, or situations where it should admit uncertainty rather than generate a convincing but incorrect answer.
Even during the pilot, it is worth measuring response speed alongside quality and the ability to retrieve the correct source. It is equally important to monitor whether people actually use the tool or bypass it after the first try. If the pilot succeeds, the production design must address the expected load, data handling rules, security, and how the agent will be updated over time to meet company needs.
AI agents can speed up how companies work with information and repetitive tasks, but the true value only becomes apparent in daily operations. It is crucial that employees actually use the tool, that answers hold up even with less precise queries, and that the infrastructure handles peaks without a noticeable drop in response times. The pilot phase helps verify the value of a specific scenario. Production operations, however, require more robust infrastructure, access control, monitoring, change testing, and clear ownership of the entire system. A well-designed agent starts with a specific scenario, proceeds to a realistic test, and only then moves to infrastructure decisions.
When choosing the technical foundation, the most important factors are understanding the workload, the data, and the expected traffic. A development team, an internal pilot, and a production deployment for multiple departments all require different configurations. A company that identifies these differences early will avoid both underpowered solutions and unnecessarily expensive hardware purchases without a clear use case.
The pilot phase is designed to show whether the agent solves a specific problem better than the existing process. Production operations bring higher responsibility and higher user expectations – the agent transitions from a testing tool into a widely available service that must handle multiple concurrent requests, work with up-to-date data, and maintain predictable performance even during peak times.
With AI agents, performance is difficult to assess based on the graphics card model name or a single configuration parameter. Bottlenecks are often found outside the GPU and can occur during document retrieval or file access. The database, storage, CPU, and network layer all affect how quickly the entire journey from query to response takes place.
Preparing a response to a customer or suggesting a change to an order can happen automatically, but actually sending it or making changes to a company system often requires human confirmation. A production agent therefore needs a clearly defined boundary between recommendation, draft, and actual execution of an action.
A single user request can trigger document retrieval, internal system calls, result interpretation, and response generation. If the output does not meet expectations, the company must be able to trace where the issue occurred. Furthermore, without ongoing metrics, a company cannot see what running the agent actually costs.
RAG (Retrieval-Augmented Generation) is a scenario where the agent first retrieves relevant sections of company documents and only then compiles an answer from them. Documents must be prepared for search before use – the system splits them into meaningful chunks, converts their semantic meaning into a format suitable for search, and stores them for quick retrieval. This increases demands on storage, databases, and the network layer.
Peter Vnuk
Technologie jsou pro mě práce i zábava – nejvíc se věnuji smartphonům, notebookům, audiotechnice, umělé inteligenci a všemu hi-tech. Rád recenzuji novinky, sleduji futuristické trendy a odhaduji další vývoj technologií. Fascinuje mě sci-fi a vize budoucího světa, které často inspirují i reálný technologický pokrok. Profesionálně se věnuji také videohrám a hernímu průmyslu. Když zrovna nepracuji, rád si odpočinu u dobré hry, kvalitního piva nebo tvorbou technologických memes na Facebooku.
