Ep098: From BI to Gen AI: A CTO's Journey Through Data Evolution

Ash Pembroke, Portfolio CTO of Caylent, self-identifies as a "recovering data scientist."

Caylent is an AWS native services company.

Data quality remains an issue despite Gen AI.

Contrasts legalism versus mysticism in data quality.

Legalism: accurate data when applications need it.

Mysticism: insights that help decision-making.

Traditional data foundations approach is being challenged weekly.

Gen AI developments force rethinking of solution architectures.

Teams share solutions through giant Slack threads.

Example: Vector databases questioned after model context protocol.

Still do traditional data assessments, but stay flexible.

Integration and data processing constantly get abstracted.

Data strategy equals architecture strategy equals business strategy.

Traditional approach: standardize data across engineering teams.

New approach: allow business users to innovate.

Bring valuable techniques back to the organization.

Case study: North Sea wind turbine alerts.

Initially seen as data quality issue, revealed new predictive failure signal.

Gen AI enables local experimentation by business users.

Blurring lines between enterprise enablement and software building.

BrainBox AI case study: energy optimization across buildings.

Architecture decisions impact ability to scale products.

Work with business edges rather than looking for patterns.

Gen AI can process information from these working groups.

Think about data as a product, not asset.

Redimensionalize dependencies across your organization.

Now's a good time to attack data quality.

New tools help visualize complexity across organizations.