Susan Shu Chang on Bridging Foundational Machine Learning and Generative AI

Introduction to Machine Learning for Software Engineers

• The AI and Machine Learning for Software Engineers track at CuCon San Francisco covered various topics in foundational machine learning, including recommender systems, reinforcement learning, and machine learning in production 51s.
• The track aimed to provide engineers with concrete examples and insights into how machine learning works behind the scenes, as many engineers are working with companies that use machine learning or have been adjacent to it 2m36s.
• The track included talks from companies such as Netflix, which discussed how to get better recommendations and tailor them to specific people, and Meta, which presented an idea on using long-term interaction as a reinforcement learning problem 2m53s.
• The importance of machine learning in production was also highlighted, as having a fancy machine learning model is not enough if it cannot be run in production and serve users 1m52s.

Getting Started with Machine Learning

• Knowledge graphs and large language models were also discussed, and how they can be merged to build a better experience 2m11s.
• For those looking to get into machine learning, PyTorch and TensorFlow are still recommended as valid tools to play around with and get started, with many tutorials available online to build an initial end-to-end app 3m31s.
• Building a simple, lightweight web app is essential to connect machine learning models to the UI, which is familiar to software engineers, and is a common practice in the industry 3m51s.

Challenges and Considerations in Machine Learning

• Deploying machine learning models into production can be challenging, and common pitfalls include finding the right problem, communicating with management, and finding the right solutions 4m23s.
• As a principal data scientist, it's crucial to determine what is overkill and what is practical, and sometimes starting with rules-based approaches is more effective than machine learning models 4m38s.
• Rules-based approaches provide a baseline to compare machine learning models, and if the model performs worse, it may indicate that machine learning is not the best solution or needs significant improvement 4m50s.
• Focusing on business logic and rules in the early stages is essential to avoid wasting resources and ensure a good return on investment 5m17s.

Working with AI and Machine Learning Tools and APIs

• With the availability of tools and APIs like GT4, it's now more open for everyone to work with AI and machine learning, even without prior knowledge of how the models work 5m43s.
• However, having prior knowledge of AI and machine learning can still be beneficial, especially when it comes to fine-tuning and evaluating models, or when working with complex scenarios that require opening the "black box" 5m53s.
• Engineers can build a lot of things without knowing the exact internals of machine learning models, and tools like Open AI and Anthropics can be integrated into products using API keys 6m24s.
• The example of building an AI assistant using Open AI's API key demonstrates how engineers can build an MVP completely by themselves, and then collaborate with experts for improvements and fine-tuning 6m59s.
• Building a minimum viable product (MVP) is easy, but debugging and tweaking it can be difficult unless one understands how the tools and techniques work, which is where knowing the fundamentals is important 8m7s.
• Not understanding what's happening under the hood can set people up for failure, especially when working with complex concepts like embeddings and generative AI 7m49s.

Elastic Common Schema and AI Assistant Development

• The Elastic Common Schema allows for the creation of supervised models that can be trained on data without having access to the customer's data, as the schema ensures that the data will be in a consistent format 9m22s.
• The Elastic team ships these models downstream to customers, who can then use them in their environment without any involvement from the team 9m39s.
• The team also works on the Aumen AI assistant piece, which involves building out AI assistants and conducting tasks like evaluation, research, and R&D to add new functionality 9m56s.
• To prove that a new feature or functionality works, the team conducts R&D projects, such as using large language models for session summarization, and evaluates the results 10m25s.
• One example of a successful R&D project involved using large language models to summarize logs ingested into Elastic Search, which was able to understand the user's actions and commands from the logs 11m6s.
• The team used real logs, pre-processed the data, and fed it to a large language model to see if it could summarize the user's session, which was a successful proof of concept 11m31s.
• To improve the performance of language models, it is essential to have a large context window, and the team had to narrow down the content being fed into the models by extracting relevant fields and consolidating repeated data 11m45s.
• The team used GPT-40 and Gemini 1, which had larger context windows, but found that it didn't necessarily mean better performance, and they had to go back and extract relevant fields 11m57s.
• To evaluate the performance of the language model, the team used subject matter expertise, having a security researcher summarize the relevant information in natural language, and compared it to the response given by the large language model 12m56s.
• The project is still in the R&D phase and has not been merged into the capability, but the team may revisit it in the future, and the lessons learned from this project have been applied to subsequent projects 13m37s.

Building MVPs and Prototypes with Streamlit

• The team uses a template for evaluation, which has been found to be repeatable, and the AI tools and methods are constantly evolving, making it challenging to know what the next big thing is or how to adapt the database to make search work faster or better 14m17s.
• For people starting with machine learning, it is recommended to start with tools like Streamlit, which provides a front-end for Python and allows users to create interactive chatbots with a few lines of code 14m57s.
• Streamlit provides a way to create a website with a drop-down for basic responses, built-in templates, and an input box, making it easy to create MVPs (Minimum Viable Products) 15m10s.
• Industry professionals in machine learning use tools like Streamlit to quickly spin up prototypes and demonstrate ideas to others, such as product people, as it is more tangible and convincing than showing JSON output or requiring others to run a Python script with specific dependencies 15m42s.
• Streamlit is a useful tool for learning and starting small projects, but not for building large-scale end-to-end projects 16m24s.

Other Recommended Tools and Building Simple Apps

• Other recommended tools for learning include Django, Fast API, and Sanic, which allow users to build simple apps and understand machine learning inputs and outputs 17m2s.
• Building a simple app with these tools helps learners understand how machine learning fits into the entire infrastructure and can be used to create a Python function as a service or API 17m25s.

Machine Learning Interviews Book and Time Management

• A book titled "Machine Learning Interviews" was written to help new graduates or career transitioners prepare for machine learning interviews, and it came about after the author taught a well-received online course on the topic for O'Reilly 17m45s.
• The book proposal was initially declined, but the author eventually created a proposal that was selected by the publisher, and the book took a year to write in the author's spare time 18m40s.
• Effective time management is crucial, with a recommended 3-5 hours of work per week, and a preference for a slow and steady approach to avoid stress and crunch time 19m23s.

Skills and Preparation for Machine Learning Jobs

• The skills required for a machine learning job are highly dependent on the specific job and company, and it's essential to prioritize the skills mentioned in the job description 20m1s.
• A general machine learning life cycle and workflow can be broken down into stages such as raw data, data pipelines, model development, and model deployment, and jobs will correspond to one or more of these responsibilities 20m11s.
• Job seekers should not try to prepare for every possible topic, but instead focus on the specific areas mentioned in the job description and prioritize those questions 21m13s.
• The book covers topics such as how to select what topics to prioritize, main machine learning algorithms, general Ops skills and tooling, and the responsibilities of various roles 21m27s.

Evolving Field and Staying Up-to-Date

• Experienced machine learning professionals may still find the book helpful, even if they are already working in the field, as they may need to transition between different roles or areas of focus 21m57s.
• The machine learning field is rapidly evolving, making it challenging to learn the necessary skills, and it's essential to stay up-to-date with the latest developments 22m42s.

Common Mistakes in Machine Learning Interviews

• One common mistake made by job seekers is focusing too much on specific tooling, which may not add significant value to the interview process 23m11s.
• When interviewing candidates for machine learning positions, it's essential to focus on generic skills rather than specific tooling, as this can limit the pool of potential candidates unnecessarily 24m22s.

Setting Up Effective Machine Learning Interviews

• Companies setting up machine learning interviews should first identify the skills they want to hire for, such as data engineering, model training, or machine learning operations (MLOps) 24m55s.
• The interview structure should be tailored to the specific role, asking questions relevant to the primary responsibilities of the position, whether it's data engineering, model training, or MLOps 25m21s.
• A common mistake is not defining the role and its place in the company, leading to a disorganized interview process and a search for a "unicorn" candidate without a clear understanding of where they fit in the company 25m51s.
• When hiring for more senior positions, it's essential to assess not only technical skills but also the ability to mentor, lead projects, and have insight into other responsibilities, such as MLOps 26m33s.
• Senior candidates should demonstrate a broader understanding of the machine learning pipeline, including MLOps, and interview questions should aim to capture this breadth of skills 27m6s.
• The interview process should be designed to evaluate the candidate's ability to contribute to the specific role and the company, rather than just focusing on narrow technical skills 26m16s.

T-Shaped Skills and Specialization

• Having a T-shaped skill set, with a broad understanding of various aspects of machine learning and a deep understanding of a specific area, is beneficial for scaling up in a career, as it improves communication skills and the ability to work end-to-end 27m23s.
• Specialization is becoming more common in larger companies, allowing individuals to focus on specific areas, such as model training, but having a broader understanding of the entire process can still be beneficial 27m57s.
• Knowing a bit more about the deployment process and what comes before and after can differentiate a candidate from others with the same skills 28m16s.

Company Needs and Candidate Fit

• Companies with a clear vision of what they want to hire may reject candidates who interview well but don't fit their specific needs, highlighting the importance of understanding the company's requirements 28m40s.
• The ideal candidate for a role can change over time, depending on the company's current needs, making it essential for candidates to align their skills with the company's requirements 29m10s.
• Luck can play a role in the interview process, as the fit between the candidate's skills and the company's needs may not be immediately clear, especially in machine learning jobs 29m57s.
• The lack of specialization in machine learning jobs can make it challenging to determine the best fit for a role, but having a broad understanding of various aspects of machine learning can be beneficial 30m6s.