2. Synthetic Data Generation

To support both our machine learning model and the RAG (Retrieval-Augmented Generation) chatbot, we needed a sufficient volume of labeled and diverse data. However, the limited number of fictitious postpartum case forms made it challenging to build robust models and retrieval systems. To address this, we developed a synthetic data generator that produces statistically relevant data aligned with the patterns observed in the original samples. Our approach involved the following:

• Controlled Randomization: Key fields such as program year, baby and mother dates of birth, case worker names, and referral statuses were randomly generated within logical boundaries to preserve real-world coherence.
• Text Variation: We injected controlled variability into the notes and concerns fields using predefined templates and randomized phrases.
• Batch Configuration: We used Databricks Jobs to orchestrate the synthetic data generation process at scale. These jobs allowed us to configure the number of records to generate per run, automate scheduling, and monitor execution all within the Databricks environment.
• Storage as Delta Table: Generated data was saved directly to a Delta table within Databricks, ensuring compatibility with downstream ML pipelines and RAG indexing workflows.

This synthetic data generation allowed us to simulate realistic case distributions, stress-test our ML risk level classifier, and fine-tune the chatbot’s retrieval capabilities—all without exposing sensitive real-world data.

3. ML Risk Level Classification

To help case workers and supervisors quickly identify families needing urgent attention, we categorized the sample forms into three risk levels: High, Medium, and Low. These categories were based on a combination of the structured fields found within each form. Several key indicators contributed to the risk classification, including:

• Signs of postpartum depression
• Repeated missed appointments
• Health flags from the baby’s postnatal records

We developed a supervised machine learning model to learn from these indicators and automatically classify new forms into the appropriate risk category. The model was trained using both the original fictitious data and the generated synthetic data, ensuring it could generalize across diverse cases and language patterns. This automated classification capability allows teams to:

• Quickly filter and surface high-risk cases for immediate review
• Reduce manual review time by highlighting only the most critical forms
• Support downstream analytics and dashboards by tagging records with consistent, model-driven risk levels

By embedding this ML model directly into our Databricks pipeline, we streamlined the process of triaging large volumes of form data—enabling faster intervention and better resource prioritization.

4. RAG Chatbot

To enable fast, natural-language access to the valuable information embedded in case forms, especially the free-text notes, we built a Retrieval-Augmented Generation (RAG) chatbot using Databrick’s Mosaic AI. This chatbot allows users to ask questions like “Which mothers showed signs of distress?” or “List high-risk cases without follow-up notes” and get answers in real time. The chatbot architecture leverages several integrated Databricks features:

• Indexing with Databricks Vector Search: After preprocessing and chunking the form data, particularly the case worker notes, we used the Databricks BGE embedding model to convert each text chunk into semantic vectors. These embeddings were then stored in a Mosaic AI Vector Search index, enabling efficient similarity-based retrieval. This setup allowed the RAG chatbot to retrieve the most relevant context for each user query with high accuracy and low latency.
• LLM-powered Responses with DBRX: For generating answers, we used Databricks’ DBRX model, a powerful large language model designed for enterprise-grade performance and cost efficiency. This model synthesizes the retrieved chunks and crafts contextual answers grounded in the source material.
• Model Logging with MLflow: The RAG pipeline was packaged and logged using MLflow, enabling version control, metadata tracking, and reproducibility.
• Model Serving via Unity Catalog: The chatbot was deployed as a registered model in Unity Catalog, allowing secure, scalable serving across the organization with role-based access controls.

This RAG-powered chatbot not only made it easier for case workers and analysts to query the forms, but also significantly reduced the time spent manually reviewing historical cases. It added a conversational layer on top of our structured and unstructured data—bringing new levels of accessibility and efficiency to the case management process.

5. AI/BI and Dashboards

To monitor trends, outcomes, and key metrics from caseworker forms, we developed interactive dashboards using Databricks SQL. These dashboards enable us to quickly identify high-risk cases and provide stakeholders with built-in filters to explore specific areas of interest. We incorporated a variety of key performance indicators and dynamic visualizations to reveal trends over time and surface actionable, data-driven insights. By leveraging Databricks SQL, we streamlined our reporting process and empowered stakeholders to make informed decisions with real-time data The Dashboards leverage several integrated Databricks features:

• Dynamic graphs and charts that update in real time based on selected filters and data sources.
• Row and column-level security to anonymize sensitive data such as names and personally identifiable information (PII).
• Granular permissions that allow users to access all or specific subsets of data based on their role or department.
• SQL queries and custom visualizations tailored to stakeholder needs for quick insights and trend detection.
• Interactive filters that enable stakeholders to drill down into specific areas of interest and explore different dimensions of the data.
• Automated refreshes of data, ensuring up-to-date information is always at hand for decision-making.