Automated Data Pipelines and BI Solutions for Telecom

Client and Business Goals:

Our client is a multinational telecommunications conglomerate, commonly known as Axiata, with extensive operations in Asia. Axiata operates in Malaysia, Nepal, Cambodia, and Bangladesh. Its core business is to provide high-quality telecommunications services. And as any telecommunications provider, Axiata runs a hundred thousand cell towers throughout the Asia region. Setting up the cell tower is not a one-time process; instead, it’s a continuous process of updating hardware and software, changing aerial directions, and making adjustments due to weather conditions and maintenance requirements. Every change to the cell tower settings impacts a list of quality parameters. Tracking changes and analyzing which change leads to specific improvement are crucial parts of the operations department.

Product Description:

The product our client asked for was: 

1. A fully automated data pipeline running 24/7 without any additional maintenance. 
2. Reading the proprietary data from the hardware.
3. Loading data into the data warehouse which is suitable for further analytical queries and storing big data. 
4. Using a set of dashboards by the operations department to simplify management decisions.
5. Using a set of alerts to notify when the data contain anomalies.

Tech Stack:

Telnet, Apache Nifi, ClickHouse, Grafana, and Java 8

Challenges:

The business challenge as it was described by the client: 

‘There was an issue with data processing. Some platforms were really hard to integrate with and extract data. Also, we didn’t have dashboards and thereby couldn’t analyze the data and reveal potential problems beforehand’.

After further analysis, we unveiled the most important business problem which was 

‘Simplify decision-making for the operations department based on real-time telecom data rather than on guesses’.

There were a few technical challenges as well: 

1. Reading the telecom data over the proprietary Huawei telnet protocol. Since there were no ready-made solutions for this, we created a telnet reader from scratch and integrated it into data pipelines. 
2. Another challenge was to create the data pipeline using Apache Nifi using exactly-once semantics, avoiding duplicates and storing results in the column-oriented ClickHouse database.
3. And finally, we created a business intelligence solution using Grafana to let the operations department watch the changes in real time without making any special analytical queries. 

Solution:

We built the ETL platform to process the data every day without any outages. The operations team uses the data and dashboards daily to monitor the network state.

The solution we developed was a real time data pipeline consisting of three main parts:

• Apache Nifi

• ClickHouse

• Grafana

Business Results:

Broscorp delivered the following business results: 

1. Management can precisely understand which change in the hardware setup improves or affects the quality and performance.
2. Management can track the changes in real time on the all-in-one dashboard without looking for information from different sources.
3. The system is fully automated and self-manageable; no additional supervision is required.
4. The system is scalable, so it’s not going to break over time when the data volume increases significantly.

Learn more about “benefits of using business intelligence for telecommunications”

Broscorp Comes to the Rescue

Big data is changing how we do business, creating a demand for top data experts. Hence, data engineers help us handle those large amounts of information and get value from data. But unfortunately, experienced data engineers are challenging to find. 

Broscrop’s data experts would be glad to help you deal with your data and make maximum use of them. Our company gained vast experience in creating efficient, performant, and scalable ETL systems that solve real business problems in different areas such as telecom, finance, and blockchain. Please feel free to contact us if you want to make your business data-driven rather than guess-driven.