CreditXpert's Journey to a Modern Cloud Data Solution

Quick Synopsis

CreditXpert’s products help individuals get better insights to their credit history and understand how actions can improve their score. CreditXpert offers cutting edge products which lead to better credit and the ultimate goal of homeownership.

Archetype first partnered with CreditXpert in a short data & strategy assessment project aimed at identifying gaps in the current vs. desired long-term state of the business. As part of the roadmap, Archetype recommended technology solutions to better integrate disparate data sources that would also provide efficiencies in their data wrangling processes. One of the biggest areas of improvement was in reporting. For reporting, their most important asset is log-based product information received from client endpoints. CreditXpert relied heavily on SAS processing and reporting which was time consuming, lacked quality control and visibility, and was not quickly repeatable nor reportable in a central way. Also, the R&D teams utilized data from this reporting which was hard to analyze due to the format and compilation complexities. The finance team also had reporting challenges with accounting data managed by a 3rd party, it was accessed only in specific reports sent monthly to CreditXpert. There were a lot opportunities for cross-functional teams to utilize other teams data but data remained constrained.

CreditXpert worked with Archetype to select and procure the appropriate data warehouse tools to efficiently ingest, transform and report their required subject areas. CreditXpert was already using Amazon Web Serivces (AWS) which was the selected cloud provider. Fivetran was recommended for data ingestion, Snowflake as the data warehouse, and Looker was selected as the preferred BI reporting tool. DBT Cloud is a tool used for data transformations within the Snowflake schemas. The solution was implemented in an agile way using Jira and documentation was added to a Confluence wiki which was easy to navigate and included a data governance initiative as well.

Problem

CreditXpert has been growing as a company and expanding on their technology solutions for each of their clients. There were many reporting related challenges that were addressed as part of the initial assessment.

  • The client’s usage of their products results in very large logs that CreditXpert needs to parse and map/transform every month. While their scripts handle the log parsing, the data volume is heavy and hard to analyze
  • The log files could be archived but there was no central location to review the data after the files were parsed and saved. Going back to analyze a historical month required re-processing
  • The product data captured in the operational logs was used for client billing and the duration of parsing/saving would often be delayed or not follow any traditional schedule
  • Product data could be more useful to other areas of the company, like marketing, but the typical reporting mechanisms only allow for narrow views
  • Comparing company data to 3rd party or market data could only be accomplished at a very high level
  • Financial data such as the GL was not able to be frequently used and analyzed
  • Standard operating procedures regarding the log files were not well documented and circulated. Knowledge was siloed to specific team members in most functional areas

Solution

Archetype and CreditXpert scoped out a modern cloud data solution that addressed the internal technology requirements, business requirements and budget/timeline. Together we embarked on a partnership to create new code, new processes, and new reports. The project addressed the following areas to improve the culture of data at CreditXpert:

  • The first initiative focused on data requirements and governance:
    • The team used Jira boards in an agile manner to organize tasks and drive productivity
    • Documentation was a major focus where all inventories, metrics, dimensions, and new operating procedures were kept in an easy-to-maintain Confluence wiki site
    • The broader project team met with executive leadership weekly to provide updates and discuss issues and decisions
  • Second was data staging and automation within the data warehouse
    • Fivetran was used to ingest flat files and other manual loads
    • Pre-processing using Amazon lambda functions and S3 buckets were automated and tested
    • Log file loading and parsing was completely re-written to minimize unnecessary code and provide efficiencies wherever possible
    • Frequency of loading data was set to as often as possible (now set to daily when prior loads were monthly/weekly)
    • Increased exception reporting and automated data checks cuts down on bad or errant data
    • Snowflakes staging, ODS and data warehouse layers updated daily with clean, single version of truth data
  • Third was the reporting initiative which used Looker as the technology
    • Report catalog converted to Looker and tested with CreditXpert resources
    • Multiple functional groups trained on new reporting tool by Archetype and a train the trainer approach
    • Executive team trained on benefits and use cases for BI reporting solution
  • Managed services were set up to help with enhancements and stabilization while the newly hired BI Lead was initiated to the role
    • There was an enhancement added to connect Quickbooks directly to Snowflake via Fivetran’s QuickBooks online connector once the Quickbooks project went live (right after the original project)

Result

There were many successes garnered from the initial implementation and on-going improvements leading to high degrees of optimization. The initiatives above have allowed CreditXpert to realize the following benefits:

  • A single platform for BI reporting encompassing functional areas: operations, finance, sales and marketing, management and leadership alike
  • The same type of monthly and quarterly reports that took hours thru SAS processing now take minutes in the new platform
  • Dependencies on 3rd party vendors for financial reporting have been eliminated and now take minutes to produce
  • New data sources implemented after the initial implementation can sync into new environment via Fivetran and Snowflake
  • A modern cloud data infrastructure built to last years which requires very little IT infrastructure support
  • A secure data warehouse platform
  • Well documented requirements, design and knowledge transfer documentation that is easy to access, read, and edit for all employees and contractors. Low on-boarding time required
  • Development framework is well documented and followed to promote code changes and keep versioning in tact. Code testing within DBT Cloud and promotion requirements ensure a far less buggy or errant data structure
  • Business needs have been constantly changing but the agile approach, governance mentality and modern tools allow for reporting to catch up quickly