Summary
Databricks is experiencing rapid growth, expanding its array of data lakehouse offerings and diving into generative AI, primarily driven by their recent acquisition of MosaicML. Their approach has shifted from emphasizing their data lakehouse’s capacity to handle all enterprise data, structured and unstructured, at scale, to a broader identity as a comprehensive data and AI platform, now incorporating generative AI capabilities.Details
Infinitive attended the Databricks Data and AI World Tour held in New York City on November 8, 2023. This was one of many such conferences being conducted across the world. New announcements were not expected, and none were provided. However, the evolution of Databricks’ philosophy and messaging was on full display.Keynote
Michael Hartman, Databricks SVP of Regulated Industries led off the keynote. Hartman laid out the big ideas behind Databricks’ philosophy.
Key point #1 — Databricks sees itself as the open-source friendly option.
Describing today’s world as a period of immense technological change, Hartman urged the audience to remember prior periods of extreme change. The tradeoffs between short-term gain and long-term benefits figured prominently. Hartman cautioned against companies “painting themselves into a corner” by accepting vendor lock-in to maximize near term benefits.
Key point #2 — Databricks believes it is the leader in democratizing data and AI within an enterprise.
Hartman then moved on to a theme that would be repeated throughout the day – the overarching goal for businesses in this era should be to democratize both data and AI.
Key point #3 — Databricks is an integrated platform with the Unity Catalog as the foundation for that integration.
Hartman yielded to Arsalan Tavakoli, SVP of Field Engineering, as Tavakoli reiterated Databricks’ core mission: to democratize data and AI. He elaborated on how Databricks’ integrated product sets allow for that democratization, highlighting the significance of the Unity Catalog ensuring effective governance of data and AI processes.
Stan Guzik, Chief Technology & Innovation Officer at S&P Global Commodity Insights, took the stage to describe his company’s remarkable 6 year long digital transformation journey. At the beginning of their journey, their operations were predominately manual, compounded by a diverse array of technologies stemming from multiple acquisitions. By leveraging Databricks extensively, they streamlined their processes, utilizing the platform for data ingest, processing, analytics/AI, and delivery. Today, they supply data to their customers in a largely automated manner.
Key point #4 — Databricks sees tailored LLMs as they key to providing value for enterprises.
Tavakoli retook the stage and discussed the challenges of using an “off the shelf” large language model (LLM). He used the example of an internal Databricks business question – “How many DBUs were there in Europe last quarter?” He pointed out the potential pitfalls of relying on a generic LLM, highlighting the unique challenges Databricks faced, such as its distinct definition of DBU, the combination of Latin America with Europe and adherence to a non-calendar quarter system. His point was that the future of LLMs in enterprises requires a base LLM that has been fine tuned for the specific enterprise.
Following Tavakoli’s insightful remarks, Hartman described Databricks’ strategic move into the generative AI market through the acquisition of MosaicML, a development announced in June 2023. Their philosophy was summed up by the statement, “Your AI built on your data.” Beyond that, Hartman held forth that Databricks, facilitated by MosaicML technology, would allow customers to own their models providing control, privacy, and cost advantages.Top of Form
Tavakoli finished by defining Databricks’ clean room strategy – allowing data sharing with no exfiltration of your data from your control.
Analysis
Databricks is at an inflection point as it moves from a successful startup to a scale technology provider. They are staying true to their open-source roots and advertising that approach as a means for customers to avoid vendor lock-in. While that is true to an extent, lock in will still occur when customers implement the tools supplied by Databricks. They have aggressive plans to move from a product company to a platform company and seem to be well on their way to making that transition.
Given their history, philosophy, and accomplishments to date, Infinitive is confident that Databricks will succeed in their aspiration to become a scaled data and AI platform company.
