Driving Adoption and Ensuring Compliance in a Cloud Native Data Ecosystem for a Major Financial Institution

Authority piece: could be a quote, a video, an infographic, or a featured case study. Each piece will have its own pre-designed template for consistency. Replace with correct type template.

Challenge

A Top 10 Financial Institution was facing several challenges in achieving full adoption of its cloud native data ecosystem and constituent platforms. Major end-to-end enterprise-grade solutions were in place for divisions to use; however, teams were slow to migrate to the new enterprise patterns. This meant many teams within the company had to self-manage their own Apache Kafkas, which in turn created a problem with security as it was uncertain who had access and if they were secure. This further created cybersecurity issues, as loose datasets were a liability for compliance.

Solution

Our team at Infinitive worked on migration and adoption solutions for the financial institution’s data streaming platform to assist the enterprise as well as several lines of business and corporate divisions.

One major area was the client’s application log files, centrally managed by their Cyber division. Our team first requested logs of teams who needed their datasets registered. From there, a parser was created, and the raw schema was registered in the unified dataset registration platform.

Once the data was registered, our team was able to route it to the correct datasets. Automation was created in tech stacks such as Jenkins, AWS Lambda, ELK, Python, Terraform, and Ansible. All the pipelines were put in place for the ingestion, transformation, and registration of the datasets. This resulted in onboarding petabytes worth of data every month via the approved enterprise pattern.

Outcome

After our project’s completion, the financial institution was able to audit every dataset within the unified dataset platform. This included information such as where the data came from, who created it, how to contact them, how much data there was, and the cost to store the data. The unified dataset platform became more secure, further saving costs by avoiding additional penalties and fines. It also allowed for scalability and usability, resulting in the financial institution getting more value out of their data.

Allowed for the automatic provisioning of ingestion sinks and storage upon successful registrations, which can then be utilized within code/data pipelines
Tracked all sensitive data across the organization; customer data, social security numbers, geographical requirements (GDPR, etc), credit card info, etc
Allowed for peace of mind for all data use cases, including for audit

Our team was able to strengthen the financial institution’s data ecosystem by increasing adoption of preferred enterprise patterns, thus allowing for easy access to their data while also making it more secure and scalable. Ultimately, Infinitive enabled the institution to get more value out of their data and better comply with regulations.