Revealing real workflows: a data-driven journey in HR process improvement

In the dynamic world of Human Resources, particularly within large organisations, managing complex processes like onboarding and recruiting can quickly become a significant challenge. This article delves into a transformative experiment conducted within the HR department of a 3,000-person social cooperative in Italy. The core innovation was the application of data mining to reveal and analyse the reality of operational workflows, an approach that proved crucial in an environment struggling to maintain predictability and efficiency. Despite the availability of legacy information systems, their underutilization and a reliance on manual processes—such as using Excel files for bulk onboardings—made it very difficult to gain a clear understanding or provide reliable delivery predictions to the business.

The initial landscape: unpredictability and manual overload

The organisation, which is frequently involved in public tenders, was under great pressure to onboard and offboard large numbers of employees quickly. This led to a situation where the HR department’s workflows were difficult to manage and legacy information systems were only partially utilised. For example, manual Excel files were the norm for bulk onboarding.

Early efforts focused on gaining control:

• Workflow mapping: the first step involved visually mapping the existing HR workflows in a “low tech, high touch” manner.
• Manual measurement: key steps were identified for measurement, and data was manually collected in an Excel file. Initial samples revealed that onboarding lead times scattered wildly from 1 to 96 days, with no discernible pattern. This made it impossible for HR to provide reliable delivery promises to the business.
• Bottleneck identification: analysis of data quickly pointed to the contract signature step as a major bottleneck, mirroring the overall process’s pattern. This step, involving remote digital signatures, was dramatically improved by addressing underlying issues.
• Improved predictability: after fixing the bottleneck, predictability significantly improved, with over 91% of onboardings delivered within eight days.
• Evolving with Kanban: to further mature the system, the team adopted an electronic Kanban board, to be able to implement more Kanban practices and automatically collect metrics. The same approach was also successfully extended to the Recruiting workflow.

Over the course of a year, the Kanban method yielded impressive results, achieving 97% of onboardings within six days and 82% of recruitments within ten days. The full story of these initial achievements can be read in the Kanban in HR case study on the Kanban+ portal.

Nevertheless, despite the improvements, a persistent challenge remained: the department continued to collect data on a sample basis rather than constantly. They were reluctant to fully adopt the new Kanban tool due to the perceived additional overhead of using a new tool alongside their existing legacy systems.

The core problem: a labyrinth of disparate systems

The HR department’s operations were spread across a highly heterogeneous and scattered set of legacy systems. These included:

• A Microsoft Form feeding into an Excel file.
• A dedicated recruiting application.
• An onboarding application.
• An HR payroll application.
• An external regional employment information system (crucial for legal compliance and the definition of done).

While some systems had overnight batch integrations, there was no unified view of the entire end-to-end workflow. Attempting to collect comprehensive data manually from these systems was a difficult, as each system exported data differently.

The experiment: a data lake to the rescue

Recognizing the need for comprehensive, continuous measurement, an experiment was launched using Algorilla, a knowledge discovery platform. This platform, originally developed to enable IT executives to gain control over corporate IT architectures, triggered a valuable insight: a ‘gold mine of data’ already existed within the logs and timestamps of the legacy systems that could be exploited to evolve the Kanban system.

Algorilla functions as a data lake system, capable of collecting heterogeneous data sources, combining them into an analyzable format and displaying them on dashboards. The premise was simple yet revolutionary: if the system could reveal in real time what was truly happening within complex IT infrastructures, it could do the same for business processes.

The proof of concept involved feeding data from all five disparate HR systems into Algorilla. The platform was designed to:

• Ingest data from various formats, including Excel files, database exports, and even PDF receipts.
• Combine and analyze these diverse data points to reconstruct the real workflow.
• In the future, automated agents could directly collect data from databases without manual exports.

Revealing the reality: key outcomes

The implementation delivered unprecedented clarity and insights:

• Comprehensive data analysis: for the first time, the HR department could analyze all historical data, not just samples, providing an accurate picture of how workflows were really working.
• End-to-end visibility: the platform enabled analysis of the entire recruiting-to-onboarding workflow, as well as detailed insights into individual process steps.
• Real-time monitoring: workflows were visualized with real-time Work-In-Progress (WIP) counts per step and average step durations. Dashboards included the typical Kanban metrics such as Throughput, Lead Time Distribution and Cumulative Flow Diagrams.
• Anomaly detection: the system helped identify outliers and unusual situations, such as what was nicknamed Speedy Gonzales’ hire, which was completed in minutes, suggesting retrospective data entry to catch up with forgotten system updates.
• Workflow correction: data analysis even corrected misinterpretations of the workflow itself. For example, the data revealed that payroll registration occurred before regional system registration, a sequence previously not fully understood.

A game changer for organisations bound by legacy systems

This approach can prove particularly valuable for organisations that rely on legacy systems. It enables them to analyse and enhance their processes without incurring the additional overhead associated with maintaining a separate Kanban system tool. As it works with existing data, it is perfectly aligned with the principle of ‘start with what you do now‘.

Planned future enhancements to the platform include the ability to display policies and flow efficiency on the dashboard, as well as the option to set up alerts for infringements of WIP limits. This will further embed Kanban practices and empower organisations to optimise their operations.

In essence, the experiment demonstrated that by strategically collecting and analyzing existing data from disparate legacy systems, organisations can uncover the true reality of their workflows, identify hidden inefficiencies, and make data-driven decisions. They can then leverage such information to expedite the evolutionary development of their Kanban system to achieve significant workflow improvements in a shorter timeframe.