Data Warehouse & Business Intelligence

Description

  • Role: Technical lead & Programmer

  • Country: South Africa

  • 2012 - 2013

The objective of this project was to address the pain points of the legacy database infrastructure namely performance problems while loading & querying data, backups, maintenance, lack of disaster recovery, and the continuous manual intervention by the development team to keep the system running.

My manager & I was tasked to create a new department to address these problems. My initial responsibilities were to implement the chosen technology stack, implement processes within the team, understand the payment and financial data, train personnel, and programming.

My achievements, experience gain, and tasks during this project

  • Worked with large datasets. Two primary database were created, one hosting 6TB of data and the other 4TB.
  • Data loading optimisation (ETL data from various sources; binary files from mainframe, ISO8583 data dumps, csv, fixed file formats, excel, etc.)
  • Developed custom SQL DLLs and SSIS components for the ETL process
  • Designed processes and procedures that allow automation of SQL jobs and failures to be handled gracefully
  • Query and reporting optimisation and performance analysis
  • Assisted web development team to create a portal to view data and custom reports
  • Designed software to automate operational processes for loading and extracting data
  • Wore the DBA hat while the department was created. Implement automated backup procedures, maintenance of data (archive, rebuild indexes, etc.), implementing security permissions
  • Team leader to 10 personnel ranging from junior to intermediate. I was responsible for training and skills developments
  • Worked closely with the systems architect on hardware infrastructure for the system
  • Worked closely with the network department on the network architecture, PCI DSS rules, DR and fail over, backups, firewall access, access control, etc.
  • Responsible for assisting financial auditors with data extracts and queries, and I.T auditors for ensuring governance (change control, access control, etc.)

Technology