In this article, RedCore shares five lessons from developing and implementing internal HR AI tools, explaining why effective automation requires more than prompts, from clean data and role-specific outputs to meaningful results, clear business logic and human oversight.
Choose a model, connect the data, write a few prompts — and get a tool that speeds up your processes 100 times over and frees your specialists from monkey jobs in the blink of an eye. Sounds perfect, doesn’t it? Except that, in practice, things work a little differently.
This becomes particularly noticeable when AI stops being an experiment and becomes part of daily work. At RedCore, automation is one of the priority areas: with a business group of nearly 5,000 specialists working with brands, reducing routine tasks can free up hundreds of working hours.
“We are constantly looking for tasks that technology can take over. But our experience in developing and implementing our own AI tools has shown that there is quite a long journey between ‘this can be automated’ and ‘this actually works conveniently, efficiently, and safely every day,'” says Iuliia Shkut, Head of HR Operations at RedCore.
This is particularly evident in two RedCore projects that address completely different HR challenges. One is an internal ChatBot that helps specialists find the information they need among more than 10,000 corporate documents. The other is AI Feedback Assistant, which works with the results of performance reviews and probation period assessments, generating separate summaries for Business Partners, managers, and specialists.
The tasks are different, but as these products evolved, the team encountered surprisingly similar problems. And most interestingly, each new problem became less and less likely to be solved by “just another prompt.”
At RedCore, the team shared its experience of developing internal HR AI tools and explained what they had to teach AI — and what they had to learn themselves along the way.
Lesson 1. AI knows many languages. But does it speak your company’s language?
At RedCore, the first challenge was ordinary corporate slang. ChatBot started receiving a flood of queries containing expressions like “sikliv” instead of Sick Leave, BP instead of Business Partner, and so on. At one point, this whole mix of Cyrillic and Latin characters, abbreviations, and typos simply resulted in “Error 404” appearing on the screen.
To teach the system to handle real queries, a focus group of specialists from different departments was involved in testing.
“Everyone played a part — tech specialists, designers, recruiters, Business Partners. They started asking questions the way they do in real life, and that was exactly what helped us train ChatBot,” Iuliia explains.
With Feedback Assistant, the same problem appeared from a different angle. Not in the queries, but in the input data. For example, if a manager leaves feedback along the lines of “everything is fine, handles tasks well,” the Assistant simply cannot determine what exactly the specialist is good at, what results support this assessment, or what they should develop further. The output ends up being an equally generic summary.
“We had to teach both managers and specialists how to provide quality feedback — because they are the primary sources of information,” Iuliia explains.
The rule is simple: before expecting AI to produce structured output, you need to bring order to the input. Otherwise, the system simply won’t work properly. “If your documents are a mess, the necessary information is difficult to find, and the same things are called by different names, the very first hours of using the tool will reveal it,” says Iuliia.
Lesson 2. One size doesn’t fit all
The next challenge for both products was user context. Initially, ChatBot distinguished between only two roles: specialist and manager. But it quickly became clear that this wasn’t enough.
“It’s important to understand in which role a person is asking a particular question, because that determines what information and which document they should receive,” Iuliia explains.
As a result, RedCore taught ChatBot to recognize additional roles, including recruiter and Business Partner. Depending on the user’s position, the system determines their access level and selects only the information and documents they are authorized to access.
Feedback Assistant works on the same principle. The system analyzes the same dataset but adapts the output to the person who will be using it.
“For a BP, what matters is analytics they can work with further. For a manager, it provides a structured summary of information already contained in the review materials, which can be used as supporting context for conversations with the specialist. For a specialist, it’s a clear picture of their own results. There is no universal solution here, and there shouldn’t be,” Iuliia explains.
As a result, working with reports at RedCore, which previously took around one and a half to two hours, has been reduced to approximately seven minutes.
Lesson 3. Completing a task ≠ Solving a problem
For ChatBot, the first victory was learning to find the right information. But immediately afterward, another question emerged: what had actually changed for the user?
If, after receiving an answer, the user still needs to open Confluence, find the relevant section, understand the instructions, and complete the entire process independently, the problem hasn’t really been solved.
“ChatBot has evolved from ‘find’ to ‘answer,’ and now to ‘do.’ For example, at RedCore, a specialist no longer needs to switch to another system to request vacation: ChatBot checks the available leave balance, creates the request, and sends it to the manager for approval,” Iuliia explains.
With Feedback Assistant, this boundary appeared between generating text and providing a useful conclusion. Generating a summary turned out to be relatively easy. Teaching the system to identify what matters most and provide specific information that could be used further was much more difficult.
In one of its early versions, the Assistant could produce something like: “The specialist confidently demonstrates excellent results that could potentially be further improved during the next evaluation period.”
“And so what? What exactly needs improvement? Which results? How are we supposed to work with this? It sounds nice, it’s technically correct — and practically useless,” Iuliia recalls her reaction.
That is why, in this case too, the first working result was not the final one.
“For an AI product, ‘it works’ is not the finish line. Real-world use constantly raises the bar for what should be considered a good result,” says Iuliia.
Lesson 4. AI can understand everything. Except what matters most
The most revealing case happened with Feedback Assistant during the completion of a probation period.
“The manager had already decided not to continue working with the specialist. This was even recorded in the original report. AI correctly analyzed the information — and then suggested a six-month development plan for the specialist,” Iuliia recalls.
Meantime, the Business Partner remains the final reviewer and may amend, disregard, or override any AI-generated output. Employment-related decisions are made by authorized human decision-makers.
For the model, this was logical: if there are areas for development, it should suggest ways to improve them. For the business, it was pointless because the decision had already been made. And fixing this scenario turned out to be more difficult than expected. Even after additional clarifications, the Assistant continued looking for ways to “save” the specialist.
“AI needs to understand not only the facts, but also their hierarchy: which signals can be taken into account when making a recommendation, and which ones completely change what happens next. And the closer we allow AI to move from analysis to action, the more important these rules become. We automate everything, but we automate wisely,” Iuliia concludes.
Eventually, Feedback Assistant was taught to consider not only the ratings and feedback themselves, but also the final review status. Now, further recommendations are generated with the decision already made in mind.
Lesson 5. AI must know not only what to do, but also where to stop
At the same time, naturally, the Business Partner remains the final reviewer in Feedback Assistant.
“The model simply doesn’t have access to the same information as a human. The system doesn’t know, for example, that the team has been working under an unusually heavy workload for the past few months. It doesn’t know that negative feedback may have appeared after a recent conflict. It doesn’t know the full history of the relationship between the specialist and the team.” says Iuliia.
The goal of automation isn’t to remove humans from the process at any cost, but to understand which parts of the process a machine genuinely performs better and where its authority should end.
According to Iuliia, despite the automation of HR processes, the role of the Business Partner at RedCore remains strong precisely because BPs work with managers and teams in a real-life context that the system doesn’t have.
Both RedCore products continue to evolve, even though they are already used daily. And perhaps this best describes the main lesson AI teaches all of us: you cannot configure it once and consider it finished. You have to keep teaching it — and learning alongside it.
Implementing AI effectively requires human involvement, while the development process itself quickly highlights things that may previously have gone unnoticed: gaps in data, documentation, business logic, and internal processes. And by teaching AI, companies inevitably learn to understand themselves better.
Want to work with a strong team? Send your CV to RedCore. https://link.redcore.group/4hBbLuR
