Business Case: Big Data Management for comparison portal Billiger.de
Hadoop implementation during operation
Project requirements/tasks
The website "billiger.de", a brand of solute GmbH, is one of the largest and most clicked online price comparison portals in Germany. This platform helps consumers to find low-priced offers at online shops. Detailed product information makes it easier to decide between millions of offers from thousands of shops. These offers are updated several times a day, ensuring transparency and an overview of the online market for retailers and consumers. The amount of data to be processed is correspondingly high.
Hadoop is a free framework which, thanks to its interfaces and scalability, can be used flexibly, cost-effectively, efficiently and future-proof, especially with polystructured mass data. Internet giants such as Google, Yahoo and Facebook have long shown that Hadoop technology is currently the best suited to handle such data volumes.
solute created an appropriate system on a "greenfield site" and quickly realized that external consulting was needed due to the complexity and scope of the project.
© Billiger.de
Price comparison portals such as Billiger.de require a modern infrastructure in order to serve the different channels and to be able to specifically evaluate the diverse data.
This is our contribution
Hadoop specialist KYBEIDOS began integrating the Hadoop system into the existing IT architecture of "billiger.de" after a brief requirements analysis and architecture consulting - and this during ongoing operations. Thanks to the system configuration tool "Puppet" all installations and configurations could be carried out fully automatically. The decisive factors were status monitoring, the detection of failures and the analysis and correction of errors. Furthermore, the configuration was adapted to specific use cases. Penthao DI was used for data enrichment and processing. In addition, Kafka, Flume and HBase, a scalable, simple database for managing very large amounts of data within a Hadoop cluster, was used.
The Hadoop system was then upgraded to the latest version and connected to the Microsoft Windows user management system. Four months after the start of the project, KYBEIDOS had successfully fulfilled all requirements and at the same time provided flexible support for the BI department.
KYBEIDOS implemented our requirements professionally and advised all project partners very competently. I was also impressed by how flexibly and fairly KYBEIDOS reacted to changing circumstances.
Gunter Bach, Director IT solute GmbH │ „billiger.de“
Benefits at a glance
-
Integration of the Hadoop system into the existing IT architecture
-
Upgrade your Hadoop system to the latest version
-
Connection of the new Hadoop system to the Windows user administration
-
Status monitoring, error analysis and correction
-
Maintenance of ongoing operations
-
Technical consulting for the IT, business and BI department
The project at a glance
Branche:
-
E-Commerce
-
Comparison portal
-
Medium-sized companies
Project partners:
-
Management
-
Specialist-Department
-
IT-Department
-
BI-Department
Technologies:
- Hadoop
- Kafka
- Puppet
- HBase
- Flume
- Pentaho DI
- Flume
Bedarf:
-
Integration eines Hadoop-Systems in die vorhandene Infrastruktur
Kybeidos Beitrag:
- Beratung (techn./fachl.)
- Tool-Evaluierung
- Systemüberführung
- Systemanbindung
- Upgrade
- Operating
- Monitoring
- Support