Denys Linkov on Micro Metrics for LLM System Evaluation

Micrometrics and Business Value

• A micrometric is a specific metric defined to measure a problem seen in production or something that is foreseen to happen, and it is used to move some kind of business metric, which is different from broader metrics like accuracy and data science metrics such as F1 and Rouge 42s.
• The idea of micrometrics is to optimize for business value rather than premature optimization for broader metrics, which may not reflect the user experience 53s.
• An example of a micrometric is measuring how often a large language model switches languages, which was an issue that caused user upset, and implementing a retry mechanism to fix the issue 1m28s.
• The retry mechanism was able to fix 99% of the language switching issues, and it was a simple solution that was tracked and measured 1m48s.
• The use of micrometrics can help identify fundamental flaws within models, such as the language switching issue, which was not caused by a prompt template update 2m8s.
• Different industries and domains will have different micrometrics, and it's up to the domain experts to define what's actually happening and what metrics to use 2m36s.
• As a platform provider, one can only guess and see the problems that are encountered from customer complaints or interactions, but domain experts should learn to define their own metrics 2m43s.

Accuracy Metrics and Model Evaluation

• There is a difference between overall accuracy metrics and micrometrics, and the choice of large language model may still depend on the specific use case 3m4s.
• Evaluating models is challenging, and accuracy can be measured in different ways, such as exact match or Rouge, and different models may perform better in different use cases 3m18s.
• Evaluating the accuracy of Large Language Models (LLMs) is challenging due to flaws in various metrics, and it's essential to consider it as an approximation rather than an absolute measure 3m27s.
• Human feedback and labeling can also be inconsistent, with low overlap between expert labelers and average people, making it difficult to achieve an agreeable answer 3m41s.
• The use of LLMs can be attributed to laziness in defining good training and evaluation sets, but it's crucial to move back to defining these sets and knowing exactly what to look for 4m10s.

Defining Training and Evaluation Sets

• Defining training and evaluation sets involves understanding the desired metrics, which can be granular and trackable through complex systems like retrieval and generation pipelines 4m19s.
• In retrieval pipelines, metrics can include relevancy of retrieved documents, while generation pipelines involve more complex metrics like the BLEU score, which can be challenging to measure, especially for customer-facing agents 5m26s.
• Measuring the value or accuracy of compound answers that require multiple sources can be particularly difficult, as seen in datasets like Multihop QA 5m53s.
• Generation metrics can include specific requirements, such as mentioning the user's name, using a particular greeting, or adhering to brand voices, which can become defining factors and micrometrics 6m27s.

Metrics and Brand Voice

• Defining metrics and priorities is not a one-time exercise but a continuous process that involves updating the system to adapt to evolving information and data drift 5m13s.
• A 100-list of brand voice characteristics can be used to measure the response of a language model (LM), with specific guidelines for companies that can be quantitatively measured, such as searching for certain keywords 6m53s.
• Companies often struggle with defining a brand response, as there isn't a well-defined metric for qualities like kindness, requiring user or human evaluation to determine statistical significance 7m0s.
• Programmatic guidelines can be used to measure specific aspects of a brand's voice, such as how to handle situations where a customer orders a product not available at the store 7m37s.
• In a multimodal world, the response to complex scenarios could involve escalating to a more expensive model or connecting the customer to a human 8m21s.

Prompt Engineering and Optimization

• A balance between short-term and long-term improvements can be achieved by using techniques like prompt engineering, which is still an immature field that requires more rigor and measurement 8m32s.
• Auto-optimization frameworks like DPy can be used to define a training set, test set, and optimizer to find a good set of prompts, and assertions can be used to validate the correctness of a prompt 9m1s.
• The development of prompt engineering is expected to continue evolving, with a focus on building rigor and using multiple responses to evaluate the effectiveness of a prompt 9m22s.

User Experience and Design Patterns

• In end-user applications, presenting multiple responses to the user can be useful in specific contexts, such as content evaluation, where the user is guiding the model and doing human preference 9m43s.
• The design patterns for generative AI are still in the early stages, and more research is needed to develop effective user experience (UX) patterns 10m5s.
• Many current chatbots and language models still use methods from 20 years ago, and consistency across answers is a concern, especially when upgrading or switching between models, as it can cause confusion for customers who receive different responses to the same question 10m16s.
• The amount of context used can affect the model's performance, and intentional choices must be made about what factors to consider, such as time of day or user behavior, to improve the customer relationship 10m34s.

Model Upgrades and Performance

• Model upgrades require evaluation to mitigate regressions, and a test suite is essential for tracking changes in model performance 11m7s.
• Upgrading from one model version to another can result in significant changes in performance, even if the model is the same, as seen in the case of upgrading from ChatGPT's original version to the November version, which resulted in a 10% drop in accuracy 11m13s.
• Model providers often do not provide transparency about changes made to the model or updated metrics, making it necessary for developers to conduct their own evaluations 11m38s.
• Having multiple models can lead to conflicts, as different versions may perform better on certain tasks or have specific benefits, such as cost savings, requiring careful evaluation and migration strategies 11m58s.
• Large language models can be complex and non-deterministic, making it challenging to understand the impact of changes, even when using traditional model training methods 12m48s.

Micrometrics and Content Moderation

• Micrometrics can be useful for evaluating model performance, and a crawl-walk-run approach can be used to start with macrometrics and gradually move to more specific metrics 13m18s.
• Specific metrics can be used to evaluate LLM systems, such as measuring retrieval versus generation in RAG, and these micro metrics can be used to do more specific things 13m34s.
• Policy on content moderation is a common metric, where the goal is to determine how to respond when a user says something inappropriate, and this can be measured by tracking how many bad questions the user asks or how many out-of-domain questions they ask 13m42s.
• Different industries have different trade-offs between false positives and false negatives, and measuring how often these occur can be an important metric 14m30s.

Voiceflow Platform Overview

• Voiceflow is an AI orchestration platform that helps customers build out different workflows for tasks such as customer support and lead generation, and they focus on team collaboration, control, and hosted solutions 14m39s.
• Voiceflow's platform allows users to define business logic and build out workflows using a low-code approach, and they also provide features such as content moderation and event logging 15m13s.
• The platform allows users to instrument the system in different ways, such as using different prompts or API calls, and this can be used to track milestones that users go through in a workflow 16m29s.
• The hosted aspect of the platform allows users to build prototypes quickly and launch to production, and this can be useful for tasks such as building a new bank account opening workflow 16m0s.
• The platform provides useful analytics, such as tracking where users drop off in a workflow, and this can be used to improve the overall user experience 16m51s.

Custom Functions and Multimodal Models

• Custom functions and components can be written and reused for specific business requirements, allowing developers to build tailored user experiences 16m58s.
• Multimodal models are emerging, but they are not replacing existing tools; instead, they provide another option for people to use, and their application depends on the specific scenario 17m30s.
• A platform can be used to orchestrate different models and tools, such as choosing a model to process user-uploaded receipts and then verifying the information against an ERP or API-based system 17m59s.
• The platform allows for balancing general knowledge from AI systems with the need to program specific workflows and constrain users to a particular process 18m19s.
• The platform provides flexibility, enabling users to build differently and create custom workflows, from simple user input loops to more complex, strictly defined requirements 18m36s.
• In certain industries, such as banking, specific legal policies or terms and conditions must be output verbatim, requiring a more controlled workflow and less interpretation by large language models 19m9s.

AI Product Development and Iteration

• Many people start by building a basic product and then refine it based on user feedback and production data, treating AI as a product that requires ongoing development and improvement 19m43s.
• Launching an AI-powered product is not a one-time event; it requires ongoing evaluation and adaptation to changing business needs and user behaviors 20m1s.
• The technology landscape is constantly evolving, and it's essential to update and adapt, just like software, using Agile Release processes to ensure living artifacts rather than static ones 20m9s.
• Building experiences can be interesting, especially when working with a large number of users, including 4 million free users and 60 enterprises, each building different things 20m29s.
• Integrating with other APIs can still be a challenge, and having a natural language interface might make it easier, but tool use is still immature for what is needed 20m42s.
• Defining business processes and making specific API calls can be more efficient than using natural language queries, which can be too vague 20m59s.

LinkedIn Courses and Learning Preferences

• Creating LinkedIn courses on various topics, including prompt engineering, AI pricing, and grounding techniques, has been a successful experience, with a wide range of attendees 21m42s.
• The process of creating courses on LinkedIn involves working with a content manager to define priorities and interests, creating a table of contents, writing the content, and collaborating with a producer to bring the course to life 22m25s.
• The courses attract a variety of people, including those who are curious about AI, with some having more advanced experience, but the platform is not yet known for expert-curated courses 22m44s.
• The platform is available to educational providers, universities, libraries, and companies, which use it as their Learning Management System (LMS), resulting in a diverse range of attendees 23m6s.
• The best-performing courses tend to be introductory ones, such as "Intro to GP4," as people are looking for a basic understanding of the AI field 23m25s.
• Individuals have different learning preferences, with some opting for courses over self-directed learning through resources like Jet open.com 23m37s.

Conclusion

• The conversation is concluded with appreciation for the guest's participation in the podcast and their presence at Kon San Francisco 23m42s.