adds documents

documents/lecture-01.md

@@ -0,0 +1,563 @@
+---
+description: Introduction to planning, developing, and shipping ML-powered products.
+---
+
+# Lecture 1: Course Vision and When to Use ML
+
+<div align="center">
+<iframe width="720" height="405" src="https://www.youtube-nocookie.com/embed/-Iob-FW5jVM?list=PL1T8fO7ArWleMMI8KPJ_5D5XSlovTW_Ur" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+Lecture by [Josh Tobin](https://twitter.com/josh_tobin_).
+Notes by [James Le](https://twitter.com/le_james94) and [Vishnu Rachakonda](https://www.linkedin.com/in/vrachakonda/).<br />
+Published August 8, 2022.
+[Download slides](https://drive.google.com/file/d/18EVuJpnJ9z5Pz7oRYcgax_IzRVhbuAMC/view?usp=sharing).
+
+## 1 - Course Vision
+
+### History of FSDL
+
+**Full Stack Deep Learning (FSDL) is the course and community for people
+who are building products that are powered by machine learning (ML).**
+It's an exciting time to talk about ML-powered products because ML is
+rapidly becoming a mainstream technology - as you can see in startup
+funding, job postings, and continued investments of large companies.
+
+FSDL was originally started in 2018 when the most exciting ML-powered
+products were built by the biggest companies. However, the broader
+narrative in the field was that very few companies could get value out
+of this technology.
+
+Now in 2022, there's a proliferation of powerful products that are
+powered by ML. The narrative has shifted as well: There's
+standardization that has emerged around the tech stack - with
+transformers and NLP starting to seep their way into more use cases, as
+well as practices around how to apply ML technologies in the world. One
+of the biggest changes in the field in the past four years has been the
+emergence of the term **MLOps**.
+
+In addition to the
+field being more mature and research continuing to progress, a big
+reason for this rapid change is that **the training of models is starting to become
+commoditized**.
+
+-   With tools like [HuggingFace](https://huggingface.co), you can deploy a state-of-the-art NLP
+or CV model in one or two lines of code.
+
+-   AutoML is starting to work for a lot of applications.
+
+-   Companies like [OpenAI](https://openai.com/api/) are starting to provide models as a service where you
+don't even have to download open-source packages to use them. You
+can make a network call to get predictions from a state-of-the-art
+model.
+
+-   Many libraries are starting to standardize around frameworks like [Keras](https://keras.io/) and [PyTorch
+Lightning](https://www.pytorchlightning.ai/).
+
+### AI Progress
+
+The history of ML is characterized by stratospheric rises and meteoric falls of the public
+perception of the technology. These were driven by a few different AI
+winters that happened over the history of the field - where the
+technology didn't live up to its hype. If you project forward a few
+years, what will happen to ML?
+
+![](./media/image6.png)
+
+
+*Source: [5 Things You Should Know About
+AI](https://www.cambridgewireless.co.uk/media/uploads/resources/AI%20Group/AIMobility-11.05.17-Cambridge_Consultants-Monty_Barlow.pdf)
+(Cambridge Consultants, May 2017)*
+
+Here are the major categories of possible outcomes and our guess about their likelihoods:
+
+1.  A true AI winter, where people
+become skeptical about AI as a technology.
+We think this is less likely.
+
+2.  A slightly more likely outcome is that the overall luster of the
+technology starts to wear off, but specific applications are
+getting a ton of value out of it.
+
+3.  The upside outcome for the field is that AI continues to accelerate
+rapidly and becomes pervasive and incredibly effective.
+
+Our conjecture is that: **The way we, as a field, avoid an AI winter is
+by translating research progress into real-world products.** That's how
+we avoid repeating what has happened in the past.
+
+### ML-Powered Products Require a Different Process
+
+Building ML-powered products requires a fundamentally different process
+in many ways than developing ML models in an academic setting.
+
+![](./media/image7.png)
+
+
+In academia, you build **"flat-earth" ML** - selecting a problem,
+collecting data, cleaning and labeling the data, iterating on model
+development until you have a model that performs well on the dataset
+collected, evaluating that model, and writing a report at the end.
+
+![](./media/image5.png)
+
+
+But ML-powered products require **an outer loop** where after you deploy
+the model into production, you measure how that model performs when it
+interacts with real users. Then, you use real-world data to
+improve your model, setting up a data flywheel that enables
+continual improvement.
+
+### This Course
+
+![](./media/image2.png)
+
+
+This class is about the unique aspects you need to know beyond training
+models to build great ML-powered products. Here are some concrete goals
+for us:
+
+1.  Teaching you **generalist skills** and an understanding of the
+**components of ML-powered products** (and ML projects more
+generally).
+
+2.  Teaching you **enough MLOps to get things done**.
+
+3.  Sharing **best practices** and **explaining the motivation** behind them.
+
+4.  Learning things that might **help you with job interviews** for ML engineering roles.
+
+5.  **Forming a community** to learn together and from each other.
+
+We do NOT try to:
+
+1.  Teach you ML or software engineering from scratch.
+
+2.  Cover the whole breadth of deep learning techniques.
+
+3.  Make you an expert in any single aspect of ML.
+
+4.  Do research in deep learning.
+
+5.  Cover the full spectrum of MLOps.
+
+If you feel rusty on your pre-requisites but want to get started with
+FSDL, here are our recommendations to get up to speed with the
+fundamentals:
+
+-   Andrew Ng's [Machine Learning Coursera
+course](https://www.coursera.org/collections/machine-learning)
+
+-   Google's [crash course on Machine
+Learning](https://developers.google.com/machine-learning/crash-course)
+
+-   MIT's [The Missing
+Semester](https://missing.csail.mit.edu/) on software
+engineering
+
+### ML-Powered Products vs MLOps
+
+MLOps, as a discipline, has emerged in just the last few years. It is
+about practices for deploying, maintaining, and operating ML systems
+that generate ML models in production. A lot of MLOps is about:
+
+-   How do we put together an infrastructure that allows us to build
+models in a repeatable and governable way?
+
+-   How can we run ML systems in a potentially high-scale production
+setting?
+
+-   How can we collaborate on these systems as a team?
+
+![](./media/image1.png)
+
+
+ML-powered product building is a distinct but overlapping discipline. A lot of
+what it takes to build a great ML-powered product goes beyond the
+infrastructure side of ML systems. It focuses on how to fit ML into the
+context of the product or the application that you're building.
+
+Other topics in the scope of the ML product discipline include:
+
+-   How do you understand how your users are interacting with your
+model?
+
+-   How do you build a team or an organization that can work together
+effectively on ML systems?
+
+-   How do you do product management in the context of ML?
+
+-   What are the best practices for designing products that use ML as
+part of them?
+
+This class focuses on teaching you end-to-end what it takes to get a
+product out in the world that uses ML and will cover aspects of MLOps
+that are most critical in order to do that.
+
+### Chapter Summary
+
+1.  **ML-powered products are going mainstream** thanks to the
+democratization of modeling.
+
+2.  However, building **great ML-powered products requires a different
+process** from building models.
+
+3.  Full-Stack Deep Learning is **here to help**!
+
+## 2 - When To Use Machine Learning
+
+### When to Use ML At All
+
+**ML projects have a higher failure rate than software projects in
+general**. One reason that's worth acknowledging is that for many
+applications, ML is fundamentally still research. Therefore, we
+shouldn't aim for 100% success.
+
+Additionally, many ML projects are
+doomed to fail even before they are undertaken due to a variety of
+reasons:
+
+1.  They are technically infeasible or poorly scoped.
+
+2.  They never make the leap to a production environment.
+
+3.  The broader organization is not all on the same page about what
+would be considered success criteria for them.
+
+4.  They solve the problem that you set out to solve but do not solve a
+big enough problem to be worth their complexity.
+
+The bar for your ML projects should be that **their value must outweigh
+not just the cost of developing them but also the additional complexity
+that these ML systems introduce to your software** (as introduced in the
+classic paper "[The High-Interest Credit Card of Technical
+Debt](https://research.google/pubs/pub43146/)").
+
+In brief,
+ML systems erode the boundaries between other systems, rely on expensive
+data dependencies, are commonly plagued by system design anti-patterns,
+and are subject to the instability of the external world.
+
+Before starting an ML project, ask yourself:
+
+1.  **Are you ready to use ML?** More specifically, do you have a
+product? Are you collecting data and storing it in a sane way? Do
+you have the right people?
+
+2.  **Do you really need ML to solve this problem?** More specifically,
+do you need to solve the problem at all? Have you tried using
+rules or simple statistics to solve the problem?
+
+3.  **Is it ethical to use ML to solve this problem?** We have a
+[whole lecture about ethics](../lecture-9-ethics/)!
+
+### How to Pick Problems to Solve with ML
+
+Just like any other project prioritization, you want to look for use
+cases that have **high impact** and **low cost**:
+
+1.  **High-impact problems** are likely to be those that address friction in
+your product, complex parts of your pipeline, places where cheap
+prediction is valuable, and generally what other people in your
+industry are doing.
+
+2.  **Low-cost projects** are those with available data, where bad
+predictions are not too harmful.
+
+![](./media/image11.png)
+
+
+#### High-Impact Projects
+
+Here are some heuristics that you can use to find high-impact ML
+projects:
+
+1.  **Find problems that ML takes from economically infeasible to feasible**.
+A good resource here is the book "[Prediction Machines:
+The Simple Economics of
+AI](https://www.amazon.com/Prediction-Machines-Economics-Artificial-Intelligence/dp/1633695670)."
+The book's central thesis is that AI reduces the cost of
+prediction, which is central to decision-making. Therefore, look
+for projects where making prediction cheaper will have a huge impact.
+
+2.  **Think about what your product needs**.
+[This article from the ML team at Spotify](https://spotify.design/article/three-principles-for-designing-ml-powered-products)
+talks about the three principles for designing Discover Weekly,
+one of Spotify's most powerful and popular ML-powered features.
+
+3.  **Think about the types of problems that ML is particularly good at**.
+One common class of problem that is overlooked is
+["Software 2.0"](https://karpathy.medium.com/software-2-0-a64152b37c35),
+as coined by Andrej Kaparthy. Essentially, if you have a part of your
+system that is complex and manually defined, then that's
+potentially a good candidate to be automated with ML.
+
+4.  **Look at what other people in the industry are doing**.
+Generally, you can read papers and blog posts from both Big Tech and top
+earlier-stage companies.
+
+#### Low-Cost Projects
+
+![](./media/image12.png)
+
+
+There are three main drivers for how much a project will cost:
+
+1.  **Data availability**: How hard is it to acquire data? How expensive
+is data labeling? How much data will be needed? How stable is the
+data? What data security requirements do you have?
+
+2.  **Accuracy requirement**: How costly are wrong predictions? How
+frequently does the system need to be right to be useful? What are
+the ethical implications of your model making wrong predictions?
+It is noteworthy that **ML project costs tend to scale
+super-linearly in the accuracy requirement**.
+
+3.  **Problem difficulty**: Is the problem well-defined enough to be
+solved with ML? Is there good published work on similar problems?
+How much compute does it take to solve the problem? **Generally,
+it's hard to reason about what's feasible in ML**.
+
+#### What's Hard in ML?
+
+![](./media/image8.png)
+
+
+Here are the three types of hard problems:
+
+1.  **Output is complex**: The model predictions are ambiguous or in a
+high-dimensional structure.
+
+2.  **Reliability is required**: ML systems tend to fail in unexpected
+ways, so anywhere you need high precision or high robustness is
+going to be more difficult to solve with ML.
+
+3.  **Generalization is required**: These problems tend to be more in
+the research domain. They can deal with out-of-distribution data
+or do tasks such as reasoning, planning, or understanding
+causality.
+
+#### ML Feasibility Assessment
+
+This is a quick checklist you can use to assess the feasibility of your
+ML projects:
+
+1.  Make sure that you actually need ML.
+
+2.  Put in the work upfront to define success criteria with all of the
+stakeholders.
+
+3.  Consider the ethics of using ML.
+
+4.  Do a literature review.
+
+5.  Try to rapidly build a labeled benchmark dataset.
+
+6.  Build a "minimum" viable model using manual rules or simple
+heuristics.
+
+7.  Answer this question again: "Are you sure that you need ML at all?"
+
+### Not All ML Projects Should Be Planned The Same Way
+
+Not all ML projects have the same characteristics; therefore, they
+shouldn't be planned the same way. Understanding different archetypes of
+ML projects can help select the right approach.
+
+#### ML Product Archetypes
+
+The three archetypes offered here are defined by how they interact with
+real-world use cases:
+
+1.  **Software 2.0 use cases**: Broadly speaking, this means taking
+something that software or a product does in an automated fashion
+today and augmenting its automation with machine learning. An
+example of this would be improving code completion in the IDE
+(like [Github
+Copilot](https://github.com/features/copilot)).
+
+2.  **Human-in-the-loop systems:** Machine learning can be applied for
+tasks where automation is not currently deployed - but where
+humans could have their judgment or efficiency augmented. Simply
+put, helping humans do their jobs better by complementing them
+with ML-based tools. An example of this would be turning sketches
+into slides, a process will usually involve humans approving the
+output of a machine learning model that made the slides.
+
+3.  **Autonomous systems:** Systems that apply machine learning to
+augment existing or implement new processes without human input.
+An example of this would be full self-driving, where there is no
+opportunity for a driver to intervene in the functioning of the
+car.
+
+For each archetype, some key considerations inform how you should go
+about planning projects.
+
+![](./media/image10.png)
+
+
+1.  In the case of Software 2.0 projects, you should focus more on
+understanding **how impactful the performance of the new model
+is**. Is the model truly much better? How can the performance
+continue to increase across iterations?
+
+2.  In the case of human-in-the-loop systems, consider more **the
+context of the human user and what their needs might be**. How
+good does the system actually have to be to improve the life of a
+human reviewing its output? In some cases, a model that does even
+10% better with accuracy (nominally a small increase) might have
+outsize impacts on human users in the loop.
+
+3.  For autonomous systems, focus heavily on t**he failure rate and its
+consequences**. When there is no opportunity for human
+intervention, as is the case with autonomous systems, failures
+need to be carefully monitored to ensure outsize harm doesn't
+occur. Self-driving cars are an excellent example of an autonomous
+system where failure rates are carefully monitored.
+
+#### Data Flywheels
+
+As you build a software 2.0 project, strongly consider the concept of
+the **data flywheel**. For certain ML projects, as you improve your
+model, your product will get better and more users will engage with the
+product, thereby generating more data for the model to get even better.
+It's a classic virtuous cycle and truly the gold standard for ML
+projects.
+
+![](./media/image4.png)
+
+
+As you consider implementing data flywheels, remember to know the answer
+to these three questions:
+
+1.  **Do you have a data loop?** To build a data flywheel, you crucially
+need to be able to get labeled data from users in a scalable
+fashion. This helps increase access to high-quality data and
+define a data loop.
+
+2.  **Can you turn more data into a better model?** This somewhat falls
+onto you as the modeling expert, but it may also not be the case
+that more data leads to significantly better performance. Make
+sure you can actually translate data scale into better model
+performance.
+
+3.  **Does better model performance lead to better product use?** You
+need to verify that improvements with models are actually tied to
+users enjoying the product more and benefiting from it!
+
+#### Impact and Feasibility of ML Product Archetypes
+
+Let's visit our impact vs. feasibility matrix. Our three product
+archetypes differ across the spectrum.
+
+![](./media/image9.png)
+
+
+This is a pretty intuitive evaluation you can apply to all your ML
+projects: **If it's harder to build (like autonomous systems), it's
+likely to have a greater impact**! There are ways, however, to change
+this matrix in the context of specific projects.
+
+1.  For **Software 2.0**, data flywheels can magnify impact by allowing
+models to get much better and increase customer delight over time.
+
+2.  For **human-in-the-loop systems**, you can increase feasibility by
+leveraging good product design. Thoughtful design can help reduce
+expectations and accuracy requirements. Alternatively, a "good
+enough" mindset that prioritizes incremental delivery over time
+can make such systems more feasible.
+
+3.  For **autonomous systems**, leveraging humans in the loop can make
+development more feasible by adding guardrails and reducing the
+potential impact of failures.
+
+### Just Get Started!
+
+With all this discussion about archetypes and impact matrices, don't
+forget the most important component of engineering: **actually
+building**! Dive in and get started. Start solving problems and iterate
+on solutions.
+
+One common area practitioners trip up in is **tool fetishization.** As
+MLOps and production ML have flourished, so too has the number of tools
+and platforms that address various aspects of the ML process. You don't
+need to be perfect with your tooling before driving value from machine
+learning. Just because Google and Uber are doing things in a very
+structured, at-scale way doesn't mean you need to as well!
+
+In this course, we will primarily focus on how to set things up the
+right way to do machine learning in production without overcomplicating
+it. This is an ML products-focused class, not an MLOps class! Check out
+this talk by Jacopo Tagliabue describing [MLOps at Reasonable
+Scale](https://www.youtube.com/watch?v=Ndxpo4PeEms) for a
+great exposition of this mindset.
+
+### Chapter Summary
+
+1.  ML adds complexity. Consider whether you really need it.
+
+2.  Make sure what you're working on is high impact, or else it might
+get killed.
+
+## 3 - Lifecycle
+
+ML adds complexity to projects and isn't always a value driver. Once you
+know, however, that it's the right addition to your project, what does
+the actual lifecycle look like? What steps do we embark upon as we
+execute?
+
+In this course, the common running example we use is of **a pose
+estimation problem**. We'll use this as a case study to demonstrate the
+lifecycle and illustrate various points about ML-powered products.
+
+![](./media/image13.png)
+
+
+Here's a graphic that visualizes the lifecycle of ML projects:
+
+![](./media/image3.png)
+
+
+It provides a very helpful structure. Watch from 48:00 to 54:00 to dive
+deeper into how this lifecycle occurs in the context of a real machine
+learning problem around pose estimation that Josh worked on at OpenAI.
+
+Let's comment on some specific nuances:
+
+-   **Machine learning projects tend to be very iterative**. Each of
+these phases can feed back into any of the phases that go before
+it, as you learn more about the problem that you're working on.
+
+    -   For example, you might realize that "Actually, it's way too
+hard for us to get data in order to solve this problem!" or
+"It's really difficult for us to label the pose of these
+objects in 3D space".
+
+    -   A solution might actually be to go back a step in the lifecycle
+and set up the problem differently. For example, what if it
+were cheaper to annotate per pixel?
+
+    -   This could repeat itself multiple times as you progress through
+a project. It's a normal and expected part of the machine
+learning product development process.
+
+-   In addition to iteration during execution, there's also
+cross-project "platform" work that matters! **Hiring and
+infrastructure development are crucial to the long-term health of
+your project**.
+
+-   Going through this lifecycle and winning each step is what we'll
+cover in this class!
+
+## Lecture Summary
+
+In summary, here's what we covered in this lecture:
+
+1.  ML is NOT a cure-all. It's a complex technology that needs to be
+used thoughtfully.
+
+2.  You DON'T need a perfect setup to get going. Start building and
+iterate!
+
+3.  The lifecycle of machine learning is purposefully iterative and
+circuitous. We'll learn how to master this process together!

documents/lecture-01.srt

@@ -0,0 +1,352 @@
+1
+00:00:00,320 --> 00:00:33,120
+hey everyone welcome to the 2022 edition of full stack deep learning i'm josh tobin one of the instructors and i'm really excited about this version of the class because we've made a bunch of improvements and i think it comes at a really interesting time to be talking about some of these topics let's dive in today we're going to cover a few things first we'll talk about why this course exists and what you might hope to take away from it then we'll talk about the first question you should ask when you're starting a new ml project which is should we be using ml for this at all and then we'll talk through the high level overview of what the life cycle of a typical ml project might look like which will also give you a conceptual
+
+2
+00:00:31,840 --> 00:01:08,880
+outline for some of the things we'll talk about in this class really what is full stack deep learning about we aim to be the course and community for people who are building products that are powered by machine learning and i think it's a really exciting time to be talking about mlpowered products because machine learning is rapidly becoming a mainstream technology and you can see this in startup funding in job postings as well as in the continued investment of large companies in this technology i think it's particularly interesting to think about how this has changed since 2018 when we started teaching the class in 2018 a lot of the most exciting ml powered products were built by the biggest companies you had self-driving
+
+3
+00:01:06,880 --> 00:01:44,079
+cars that were starting to show promise you had systems like translation from big companies like google that were really starting to hit the market in a way that was actually effective but the broader narrative in the field was that very few companies were able to get value out of this technology and even on the research side right now gpt3 is becoming a mainstream technology but in 2018 gpt-1 was one of the state-of-the-art examples of language models and you know if you look at what it actually took to build a system like this it was the code and the standardization around it was still not there like these technologies were still hard to apply now on the other hand there's a much wider range of really
+
+4
+00:01:42,240 --> 00:02:20,000
+powerful products that are powered by machine learning dolly 2 is i think a great example image generation technology more on the consumer side tick tock is a really powerful example but it's not just massive companies now that are able to build machine learning powered products dscript is an application that we at full stack deep learning use all the time in fact we'll probably use it to edit this video that i'm recording right now and startups are also building things like email generation so there's a proliferation of machine learning powered products and the narrative has shifted i think a little bit as well which is that before these technologies were really hard to apply but now there's standardization
+
+5
+00:02:17,840 --> 00:02:55,519
+that's emerging both around the technology stack transformers and nlp starting to seep their way into more and more use cases as well as the practices around how to actually apply these technologies in the world one of the biggest changes in the field in the past four years has been the emergence of this term called ml ops which we'll talk a lot about in this class and so if you ask yourself why like why is this changed so rapidly i think in addition to the field just maturing and research continuing to progress i think one of the biggest reasons is that the training of models is starting to become commoditized we showed a couple of slides ago how complicated code for gpt-1 was now using something like
+
+6
+00:02:53,840 --> 00:03:28,480
+hugging face you can deploy a state-of-the-art nlp model or computer vision model in one or two lines of code on top of that automl is starting to actually work for a lot of applications i think four years ago we were pretty skeptical about it now i think it's a really good starting point for a lot of problems that you might want to solve and companies are starting to provide models really as a service where you don't even have to download open source package to use it you can just make a network call and you can have predictions from a state-of-the-art model and on the software side a lot of frameworks are starting to standardize around things like keras and pytorch lightning so a lot of the like spaghetti
+
+7
+00:03:27,200 --> 00:04:07,680
+code that you had to write to build these systems just isn't necessary anymore and so i think if you project forward a few years what's going to happen in ml i think the history of the ml is characterized by rise and fall of the public perception of the technology these were driven by a few different ai winters that happened over the history of the field where the technology didn't live up to its height live up to its promise and people became skeptical about it what's going to happen in the future of the field i think what a lot of people think is that this time is different we have real applications of machine learning that are generating a lot of value in the world and so the prospect of a true ai winter where
+
+8
+00:04:05,680 --> 00:04:46,560
+people become skeptical about ai as a technology maybe feels less likely but it's still possible a slightly more likely outcome is that the overall luster of the technology starts to wear off but certain applications are getting a ton of value out of this technology and then i think you know the upside outcome for the field is that ai continues to accelerate really rapidly and it becomes pervasive and incredibly effective and i think that's also what a lot of people believe to happen and so what i would conjecture is that the way that we as a field avoid an ai winter is by not just making progress in research but also making sure that that progress is translated to actual real world products that's how we avoid repeating
+
+9
+00:04:45,199 --> 00:05:20,639
+what's happened in the past that's caused the field to lose some of its luster but the challenge that presents is that building ml powered products requires a fundamentally different process in many ways than building the types of ml systems you create in academia the sort of process that you might use to develop a model in an academic setting i would call flat earth machine learning flat earth ml this is a process that will probably be familiar to many people you start by selecting a problem you collect some data to use to solve the problem you clean and label that data you iterate on developing a model until you have a model that performs well on the data set that you collected and then you evaluate that
+
+10
+00:05:19,120 --> 00:05:56,639
+model and if the model performs well according to your metrics then you write a report produce a jupiter notebook a paper or some slides and then you're done but in the real world the challenge is that if you deploy that model in production it's not going to perform well in the real world for long necessarily right and so ml powered products require this outer loop where you deploy the model into production you measure how that model is performing when it interacts with real users you use the real world data to build a data flywheel and then you continue this as part of an outer loop some people believe that the earth isn't round just because you can't see the outer loop in the ml system doesn't mean it's not
+
+11
+00:05:54,160 --> 00:06:32,080
+there and so this course is really about how to do this process of building ml-powered products and so what we won't cover as much is the theory and the math and the sort of computer science behind deep learning or and machine learning more broadly there are many great courses that you can check out to learn those materials we also will talk a little bit about training models and some of the practical aspects of that but this isn't meant to be your first course in training machine learning models again there's many great classes for that as well but what this class is about is the unique aspects that you need to know beyond just training models to build great ml powered products so our goals in the class are to teach you
+
+12
+00:06:30,080 --> 00:07:09,120
+a generalist skill set that you can use to build an ammo powered product and an understanding of how the different pieces of ml power products fit together we will also teach you a little bit about this concept of ml ops but this is not an ml ops class our goal is to teach you enough ml ops to get things done but not to cover the full depth of ml ops as a topic we'll also share certain best practices from what we've seen to work in the real world and try to explain some of the motivations behind them and if you're on the job market or if you're thinking about transitioning into a role in machine learning we also aim to teach you some things that might help you with ml engineering job interviews and then lastly in practice i think what we've
+
+13
+00:07:07,199 --> 00:07:42,560
+found to be maybe the most powerful part of this is forming a community that you can use to learn from your peers about what works in the real world and what doesn't we as instructors have solved many problems with ml but there's a very good chance that we haven't solved one that's like the one that you're working on but in the broader full stack deep learning community i would bet that there probably is someone who's worked on something similar and so we hope that this can be a place where folks come together to learn from each other as well as just learning from us now there are some things that we are explicitly not trying to do with this class we're not trying to teach you machine learning or software engineering from scratch if
+
+14
+00:07:40,800 --> 00:08:15,919
+you are coming at this class and you know you have an academic background in ml but you've never written production code before or you're a software engineer but i've never taken an ml class before you can follow along with this class but i would highly recommend taking these prerequisites before you dive into the material here because you'll i think get a lot more out of the class once you've learned the fundamentals of each of these fields we're also not aiming to cover the full breadth of deep learning techniques or machine learning techniques more broadly we'll talk about a lot of the techniques that are used in practice but the chances are that we won't talk about the specific model that you use for your use
+
+15
+00:08:14,240 --> 00:08:52,800
+case it's not the goal here we're also not trying to make you an expert in any single aspect of machine learning we have a project and a set of labs that are associated with this course that will allow you to spend some time working on a particular application of machine learning but there there isn't a focus on becoming an expert in computer vision or nlp or any other single branch of machine learning and we're also not aiming to help you do research in deep learning or any other ml field and similarly ml ops is this broad topic that involves everything from infrastructure and tooling to organizational practices and we're not aiming to be comprehensive here the goal of this class is to show you end to end
+
+16
+00:08:51,279 --> 00:09:28,640
+what it takes to build an ml-powered product and give you pointers to the different pieces of the field that you'll potentially need to go deeper on to solve the particular problem that you're working on so if you are feeling rusty on your prerequisites but want to get started with the class anyway here are some recommendations for classes on ml and software engineering that i'd recommend checking out if you want to remind yourself of some of the fundamentals i mentioned this distinction between ml power products and ml ops a little bit and so i wanted to dive into that a little bit more ml ops is this discipline that's emerged in the last couple of years really that is about practices for deploying and
+
+17
+00:09:26,880 --> 00:10:06,240
+maintaining and operating machine learning models and the systems that generate these machine learning models in production and so a lot of ml ops is about how do we put together the infrastructure that will allow us to build models in a repeatable and governable way how we're able to do this at scale how we're able to collaborate on these systems as a team and how we're able to really run these machine learning systems in a potentially high scale production setting super important topic if your goal is to make ml work in the real world and there's a lot of overlap with what we're covering in this class but we see mlpowered products as kind of a distinct but overlapping discipline because a lot of what it
+
+18
+00:10:04,800 --> 00:10:45,120
+takes to build a great ml powered product goes beyond the infrastructure side and the sort of repeatability and automation side of machine learning systems and it also focuses on how to fit machine learning into the context of product or the application that you're building so other topics that are in scope of this mlpowered product discipline are things like how do you understand how your users are interacting with your model and what type of model they need how do you build a team or an organization that can work together effectively on machine learning systems how do you do product management in the context of ml what are some of the best practices for designing products that use ml as part of them things like data labeling capturing
+
+19
+00:10:42,640 --> 00:11:24,240
+feedback from users etc and so this class is really focused on teaching you end to end what it takes to get a product out in the world that uses ml and we'll cover the aspects of mlaps that are most critical to understand in order to do that a little bit about us as instructors i'm josh tobin i'm co-founder and ceo of machine learning infrastructure startup called gantry previously i was a research scientist at openai and did my machine learning phd at berkeley and charles and sergey are my wonderful co-instructors who you'll be hearing from in the coming weeks on the history of full stack deep learning so we started out as a boot camp in 2018 sergey and i as well as my grad school advisor and our close collaborator peter
+
+20
+00:11:22,240 --> 00:12:00,720
+abeel had this collective realization that a lot of what we had been discovering about making ml work in the real world wasn't really well covered in other courses and we didn't really know if other people would be interested in this topic so we put it together as a one-time weekend long boot camp we got started to get good feedback on that and so it grew from there and we put the class online for the first time in 2021 and here we are so the way that this class was developed was a lot of this is from our personal experience our study and reading of materials in the field we also did a bunch of interviews with practitioners from this list of companies and at this point like a much longer list as well so we're constantly
+
+21
+00:11:58,880 --> 00:12:32,880
+out there talking to folks who are doing this who are building ml powered products and trying to fold their perspectives into what we teach in this class some logistics before we dive into the rest of the material for today first is if you're part of the synchronous cohort all of the communication for that cohort is going to happen on discord so if you're not on discord already then please reach out to us instructors and we'll make sure to get you on that if you're not on discord or if you're not checking it regularly there's a high likelihood that you're going to miss some of the value of the synchronous course we will have a course project again for folks who are participating in the synchronous option which we'll share
+
+22
+00:12:31,040 --> 00:13:11,839
+more details about on discord in the coming weeks and there's also i think one of the most valuable parts of this class is the labs which have undergone like a big revamp this time around i want to talk a little bit more about what we're covering there so the problem that we're going to be working on the labs is creating an application that allows you to take a picture of a handwritten page of text and then transcribe that into some actual text and so imagine that you have this web application where you can take a picture of your handwriting and then at the end you get the text that comes out of it and so the way this is going to work is we're going to build a web backend that allows you to send web requests decodes those images and sends
+
+23
+00:13:09,279 --> 00:13:47,920
+them to a prediction model an ocr model that will develop that will transcribe those into the text itself and those models are going to be generated by a model training system that will also show you how to build in the class and the architecture that we'll use will look something like this we'll use state-of-the-art tools that we think balance being able to really build a system like this in a principled way without adding too much complexity to what you're doing all right so just to summarize this section machine learning powered products are going mainstream and in large part this is due to the fact that it's just much much easier to build machine learning models today than it was even four or five years ago and
+
+24
+00:13:46,079 --> 00:14:21,839
+so i think the challenge ahead is given that we're able to create these models pretty easily how do we actually use the models to build great products and that's a lot of what we'll talk about in this class and i think the sort of fundamental challenge is that there's not only different tools that you need in order to build great products but also different processes and mindsets as well and that's what we're really aiming to do here in fsdl so looking forward to covering some of this material and hopefully helping create the next generation of ml powered products the next topic i want to dive into is when to use machine learning at all like what problems is this technology useful for solving and so the key points that we're
+
+25
+00:14:20,320 --> 00:14:55,920
+going to cover here are the first is that machine learning introduces a lot of complexity and so you really shouldn't do it before you're ready to do it and you should think about exhausting your other options before you introduce this to your stack on the flip side that doesn't mean that you need to a perfect infrastructure to get started and then we'll talk a little bit about what types of projects tend to lend themselves to being good applications of machine learning and we'll talk about how to know whether projects are feasible and whether they'll have an impact on your organization but to start out with when should you use machine learning at all so i think the first thing that's really critical to know
+
+26
+00:14:54,079 --> 00:15:32,000
+here is that machine learning projects have a higher failure rate than software products in general the statistic that you'll see most often floated around in blog posts or vendor pitches is that 87 percent this very precise number of machine learning projects fail i think it's also worth noting that 73 of all statistics are made up on the spot so and this one in particular i think is a little bit questionable whether this is actually a valid statistic or not anecdotally i would say that from what i've seen it's probably more like 25 it's still a very high number still a very high failure rate but maybe not the 90-ish percent that people are quoting the question you might ask is why is that the case right why is there such a
+
+27
+00:15:30,639 --> 00:16:12,639
+high failure rate for machine learning projects you know one reason that's worth acknowledging is that for a lot of applications machine learning is fundamentally still research so 100 success rate probably shouldn't be the target that we're aiming for but i do think that many machine learning projects are doomed to fail maybe even before they are undertaken and i think there's a few reasons that this can happen so oftentimes machine learning projects are technically infeasible or they're just scoped poorly and there's just too much of a lift to even get the first version of the model developed and that leads to projects failing because they just take too long to see any value another common failure mode that's becoming less and less
+
+28
+00:16:10,079 --> 00:16:50,720
+common is that a team that's really effective at developing a model may not be the right team to deploy that model into production and so there's this friction after the model is developed where you know the model maybe looks promising in a jupiter notebook but it never makes the leap to prod so hopefully you'll take things away from this class that will help you avoid being in this category another really common issue that i've seen is when you as a broader organization are not all on the same page about what we would consider to be successful here and so i've seen a lot of machine learning projects fail because you have a model that you think works pretty well and you actually know how to deploy into
+
+29
+00:16:49,120 --> 00:17:27,120
+production but the rest of the organization can't get comfortable with the fact that this is actually going to be running and serving predictions to users so how do we know when we're ready to deploy and then maybe the most frustrating of all these failure modes is when you actually have your model work well and it solves the problem that you set out to solve but it doesn't solve a big enough problem and so the organization decides hey this isn't worth the additional complexity that it's going to take to make this part of our stack you know i think this is a point i want to double click on which is that really i think the bar for your machine learning project should be that the value of the project must outweigh not just the cost of
+
+30
+00:17:25,679 --> 00:18:07,440
+developing it but the additional complexity that machine learning systems introduce into your software and machine learning introduces a lot of complexity to your software so this is kind of a quick summary of a classic paper that i would recommend reading which is the high interest credit card of technical debt paper the thesis of this paper is that machine learning as a technology tends to introduce technical debt at a much higher rate than most other software and the reasons that the authors point to are one an erosion of boundary between systems so machine learning systems often have the property for example that the predictions that they make influence other systems that they interact with if you recommend a user a particular type
+
+31
+00:18:05,120 --> 00:18:44,799
+of content that changes their behavior and so that makes it hard to isolate machine learning as a component in your system it also relies on expensive data dependencies so if your machine learning system relies on a feature that's generated by another part of your system then those types of dependencies the authors found can be very expensive to maintain it's also very common for machine learning systems to be developed with design anti-patterns somewhat avoidable but in practice very common and the systems are subject to the instability of the external world if your user's behavior changes that can dramatically affect the performance of your machine learning models in a way that doesn't typically happen with
+
+32
+00:18:42,240 --> 00:19:21,200
+traditional software so the upshot is before you start a new ml project you should ask yourself are we ready to use ml at all do we really need this technology to solve this problem and is it actually ethical to use ml to solve this problem to know if you're ready to use ml some of the questions you might ask are do we have a product at all do we have something that we can use to collect the data to know whether this is actually working are we already collecting that data and storing it in the same way if you're not currently doing data collection then it's going to be difficult to build your first ml system and do we have the team that will allow us to do this knowing whether you need ml to solve a problem i think the first question that
+
+33
+00:19:19,760 --> 00:19:58,880
+you should ask yourself is do we need to solve this problem at all or are we just inventing a reason to use ml because we're excited about the technology have we tried using rules or simple statistics to solve the problem with some exceptions i think usually the first version of a system that you deploy that will eventually use ml should be a simple rule based or statistics-based system because a lot of times you can get 80 of the benefit of your complex ml system with some simple rules now there's some exceptions to this if the system is an nlp system or a computer vision system where rules just typically don't perform very well but as a general rule i think if you haven't at least thought about whether you can use
+
+34
+00:19:57,360 --> 00:20:33,760
+a rule-based system to achieve the same outcome then maybe you're not ready to use ml yet and lastly is it ethical i won't dive into the details here because we'll have a whole lecture on this later in the course next thing i want to talk about is if we feel like we're ready to use ml in our organization how do we know if the problem that we're working on is a good fit to solving it with machine learning the sort of tl dr here is you want to look for like any other project prioritization you want to look for use cases that have high impact and low cost and so we'll talk about different heuristics that you can use to determine whether this application of machine learning is likely to be high impact and low cost and so we'll talk about
+
+35
+00:20:32,559 --> 00:21:10,240
+heuristics like friction in your products complex parts of your pipeline places where it's valuable to reduce the cost of prediction and looking at what other people in your industry are doing which is a very underrated technique for picking problems to work on and then we'll also talk about some heuristics for for assessing whether a machine learning project is going to be feasible from a cost perspective overall prioritization framework that we're going to look at here is projects that you want to select are ones that are feasible so they're low cost and they're high impact let's start with the high impact side of things so what are some mental models you can use to find high impact ml projects and these are some of the ones that we'll
+
+36
+00:21:08,000 --> 00:21:49,520
+cover so starting with a book called the economics of ai and so the question this book asks is what problems does machine learning make economically feasible to solve that were maybe not feasible to solve in the past and so the sort of core observation in this book is that really at a fundamental level what ai does is it reduces the cost of prediction before maybe you needed a person and that person would take five minutes to create a prediction it's very expensive it's very operationally complex ai can do that in a fraction of a second for the cost of essentially running your machine or running your gpu cheap prediction means that there's going to be predictions that are happening in more places even in problems whereas too
+
+37
+00:21:47,200 --> 00:22:27,360
+expensive to do before and so the upshot of this mental model for project selection is think about projects where cheap prediction will have a huge business impact like where would you hire a bunch of people to make predictions that it isn't feasible to do now um the next mental model i want to talk about for selecting high impact projects is just thinking about what is your product need and so i really like this article called three principles for designing ml-powered products from spotify and in this article they talked about the principles that they used to develop the discover weekly feature which i think is like one of the most powerful features of spotify and you know really the way they thought about it is this reduces
+
+38
+00:22:25,840 --> 00:23:06,080
+friction for our users reduces the friction of chasing everything down yourself and just brings you everything in a neat little package and so this is something that really makes their product a lot better and so that's another kind of easy way to come up with ideas for machine learning projects another angle to think about is what are types of problems that machine learning is particularly good at and one exploration of this mental model is an article called software 2.0 from andre carpathi which is also definitely worth a read and the kind of main thesis of this article is that machine learning is really useful when you can take a really complex part of your existing software system so a really messy stack of
+
+39
+00:23:04,400 --> 00:23:46,799
+handwritten rules and replace that with machine learning replace that with gradient descent and so if you have a part of your system that is complex manually defined rules then that's potentially a really good candidate for automating with ml and then lastly i think it's worth just looking at what other people in your industry are doing with ml and there's a bunch of different resources that you can look at to try to figure out what other success stories with ml are i really like this article covering the spectrum of use cases of ml at netflix there are various industry reports this is a summary of one from algorithmia which kind of covers the spectrum of what people are using ml to do and more generally i think looking at
+
+40
+00:23:44,960 --> 00:24:21,919
+papers from the biggest technology companies tends to be a good source of what those companies are trying to build with ml and how they're doing it as well as earlier stage tech companies that are still pretty ml forward and those companies i think are more likely to write these insights in blog posts than they are in papers and so here's a list that i didn't compile but i think is really valuable of case studies of using machine learning in the real world that are worth going through if you're looking for inspiration of what are types of problems you can solve and how might you solve them okay so coming back to our prioritization framework we talked about some mental models for what ml projects might be high impact
+
+41
+00:24:20,960 --> 00:24:57,760
+and the next thing that we're going to talk about is how to assess the cost of a machine learning project that you're considering so the way i like to think about the cost of machine learning power projects is there's three main drivers for how much a project is gonna cost the first and most important is data availability so how easy is it for you to get the data that you're gonna need to solve this problem the second most important is the accuracy requirement that you have for the problem that you're solving and then also important is the sort of intrinsic difficulty of the problem that you're trying to solve so let's start by talking about data availability the kind of key questions that you might ask here to assess
+
+42
+00:24:56,080 --> 00:25:36,000
+whether data availability is going to be a bottleneck for your project is do we have this data already and if not how hard is it and how expensive is it going to be to acquire how expensive is it not just to acquire but also to label if your labelers are really expensive then getting enough data to solve the problem really well might be difficult how much data will we need in total this can be difficult to assess a priori but if you have some way of guessing whether it's 5 000 or 10 000 or 100 000 data points this is an important input and then how stable is the data so if you're working on a problem where you don't really expect the underlying data to change that much over time then the project is going to be a lot more feasible than if
+
+43
+00:25:33,919 --> 00:26:15,200
+the data that you need changes on a day-to-day basis so data availability is probably the most important cost driver for a lot of machine learning powered projects because data just tends to be expensive and this is slightly less true outside of the deep learning realm it's particularly true in deep learning where you often require manual labeling but it also is true in a lot of other ml applications where data collection is expensive and lastly on data bill availability is what data security requirements do you have if you're able to collect data from your users and use that to retrain your model then that bodes well for the overall cost of the project if on the other hand you're not even able to look at the data that your
+
+44
+00:26:12,960 --> 00:26:53,760
+users are generating then that's just going to make the project more expensive because it's going to be harder to debug and harder to build a data flywheel moving on to the accuracy requirement the kinds of questions you might ask here are how expensive is it when you make a wrong prediction on one extreme you might have something like a self-driving car where a wrong prediction is extremely expensive because the prospect of that is really terrible on the other extreme is something like let's say potentially a recommender system where if a user sees a bad recommendation once it's probably not really going to be that bad maybe it affects their user experience over time and maybe and causes them to churn but certainly not
+
+45
+00:26:52,080 --> 00:27:30,320
+as bad as a wrong prediction in a self-driving car you also need to ask yourself how frequently does the system actually need to be right to be useful i like to think of systems like dolly 2 which is an image generation system as like a positive example of this where you can if you're just using dolly 2 as a creative supplement you can generate thousands and thousands of images and select the one that you like best for your use case so the system doesn't need to be right more than like once every n times in order to actually get value from it as a user on the other hand if the system needs to be 100 reliable like never ever make a wrong prediction in order for it to be useful then it's just going to be more expensive to build
+
+46
+00:27:28,640 --> 00:28:06,480
+these systems and then what are the ethical implications of your model making wrong predictions is like an important question to consider as well and then lastly on the problem difficulty questions to ask yourself are is this problem well defined enough to solve with ml are other people working on similar things doesn't necessarily need to be the exact same problem but if it's a sort of a brand new problem that no one's ever solved with mlv4 that's going to introduce a lot of technical risk another thing that's worth looking at if you're looking at other work on similar problems is how much compute did it actually take them to solve this problem and it's worth looking at that both on the training side as well as on
+
+47
+00:28:04,559 --> 00:28:42,240
+the inference side because if it's feasible to train your model but it takes five seconds to make a prediction then for some applications that will be good enough and some for some it won't and then i think like maybe the weakest heuristic here but still potentially a useful one is can a human do this problem at all if a human can solve the problem then that's a decent indication that a machine learning system might be able to solve it as well but not a perfect indication as we'll come back to so i want to double click on this accuracy requirement why is this such an important driver of the cost of machine learning projects the fundamental reason is that in my observation the project cost tends to scale like super linearly
+
+48
+00:28:40,000 --> 00:29:25,120
+in your accuracy requirement so as a very rough rule of thumb every time that you add an additional nine to your required accuracy so moving from 99.9 to 99.99 accuracy might lead to something like a 10x increase in your project costs because you might expect to need at least 10 times as much data if not more in order to actually solve the problem to that degree of accuracy required but also you might need a bunch of additional infrastructure monitoring support in order to ensure that the model is actually performing that accurately next thing i'm going to double click on is the problem difficulty so how do we know which problems are difficult for machine learning systems to solve the first point i want to make here is this is
+
+49
+00:29:23,039 --> 00:30:08,960
+like i think like a classically hard problem to really answer confidently and so i really like this comic for two reasons the first is because it gets at this core property of machine learning systems which is that it's not always intuitive which problems will be easy for a computer to solve and which ones will be hard for a computer to solve in 2010 doing gis lookup was super easy and detecting whether a photo was a bird was like a research team in five years level of difficulty so not super intuitive as someone maybe outside of the field the second reason i like this comic is because it also points to the sort of second challenge in assessing feasibility in ml which is that this field just moves so fast that if you're not keeping up with
+
+50
+00:30:07,279 --> 00:30:50,320
+what's going on in the state of the art then your understanding of what's feasible will be stale very quickly building an application to detect whether a photo is of a bird is no longer a research team in five years problem it's like a api call and 15 minutes type problem so take everything i say here with the grain of salt because the feasibility of ml projects is notoriously difficult to predict another example here is in the late 90s the new york times when they were talking about sort of ai systems beating humans at chess predicted that it might be a hundred years before a computer beats human echo or even longer and you know less than 20 years later machine learning systems from deep mind beat the best humans in
+
+51
+00:30:48,399 --> 00:31:22,000
+the world that go these predictions are notoriously difficult to make but that being said i think it's still worth talking about and so one heuristic that you'll hear for what's feasible to do with machine learning is this heuristic from andrew ing which is that anything that a normal person can do in less than one second we can automate with ai i think this is actually not a great heuristic for what's feasible to do with ai but you'll hear it a lot so i wanted to talk about it anyway there's some examples of where this is true right so recognizing the content of images understanding speech potentially translating speech maybe grasping objects with a robot and things like that are things that you could point to
+
+52
+00:31:20,320 --> 00:31:57,600
+as evidence for andrew's statement being correct but i think there's some really obvious counter examples as well machine learning systems are still no good at things that a lot of people are really good at like understanding human humor or sarcasm complex in-hand manipulation of objects generalizing to brand new scenarios that they've never seen before this is a heuristic that you'll see it's not one that i would recommend using seriously to assess whether your project is feasible or not there's a few things that we can say are definitely still hard in machine learning i kept a couple of things in these slides that we talked about being really difficult in machine learning when we started teaching the class in
+
+53
+00:31:55,120 --> 00:32:32,640
+2018 that i think i would no longer consider to be super difficult anymore unsupervised learning being one of them but reinforcement learning problems still tend to be not very feasible to solve for real world use cases although there are some use cases where with tons of data and compute reinforcement learning can be used to solve real world problems within the context of supervised learning there are also still problems that are hard so things like question answering a lot of progress over the last few years still these systems aren't perfect text summarization video prediction building 3d models another example of one that i think i would use to say is really difficult but with nerf and all the sort
+
+54
+00:32:30,640 --> 00:33:07,679
+of derivatives of that i think is more feasible than ever real world speech recognition so outside of the context of a clean data set in a noisy room can we recognize what people are saying resisting adversarial examples doing math although there's been a lot of progress on this problem as well over the last few months solving world war problems or bond guard problems this is an example by the way of a bomb card problem it's a visual analogy type problem so this is kind of a laundry list of some things that are still difficult even in supervised learning and so can we reason about this what types of problems are still difficult to do so i think one type is where not the input to the model itself but the prediction that the model is
+
+55
+00:33:06,000 --> 00:33:47,760
+making the output of the model where that is like a complex or high dimensional structure or where it's ambiguous right so for example 3d reconstruction the 3d model that you're outputting is very high dimensional and so that makes it difficult to do for ml video prediction not only high dimensional but also ambiguous just because you know what happened in the video for the last five seconds there's still maybe infinite possibilities for what the video might look like going forward so it's ambiguous and it's high dimensional which makes it very difficult to do with ml dialog systems again very ambiguous very open-ended very difficult to do with ml and uh open-ended recommender systems so a second category of problems that are
+
+56
+00:33:46,080 --> 00:34:26,720
+still difficult to do with ml are problems where you really need the system to be reliable machine learning systems tend to fail in all kinds of unexpected and hard to reason about ways so anywhere where you need really high precision or robustness is gonna be more difficult to solve using machine learning so failing safely out of distribution for example is still a difficult problem in ml robustness to adversarial attacks is still a difficult problem in ml and even things that are easier to do with low precision like estimating the position and rotation of an object in 3d space can be very difficult to do if you have a high precision requirement the last category of problems i'll point to here is problems where you need the
+
+57
+00:34:24,560 --> 00:35:04,720
+system to be able to generalize well to data that it's never seen before this can be data that's out of distribution it can be where your system needs to do something that looks like reasoning or planning or understanding of causality these problems tend to be more in the research domain today i would say one example is in the self-driving car world dealing with edge cases very difficult challenge in that field but also control problems in self-driving cars you know those stacks are incorporating more and more ml into them whereas the computer vision and perception part of self-driving cars adopted machine learning pretty early the control piece was using more traditional methods for much longer and then places where you have a small
+
+58
+00:35:02,800 --> 00:35:40,400
+amount of data again like if you're considering machine learning broadly small data is often possible but especially in the context of deep learning small data still presents a lot of challenges summing this up like how should you try to assess whether your machine learning project is feasible or not first question you should ask is do we really need to solve this problem with ml at all i would recommend putting in the work up front to define what is the success criteria that we need and doing this with everyone that needs to sign up on the project in the end not just the ml team let's avoid being an ml team that works on problems in isolation and then has those projects killed because no one actually really needed to solve
+
+59
+00:35:38,000 --> 00:36:13,440
+this problem or because the value of the solution is not worth the complexity that it adds to your product then you should consider the ethics of using ml to solve this problem and we'll talk more about this towards the end of the course in the ethics lecture then it's worth doing a literature review to make sure that there are examples of people working on similar problems trying to rapidly build a benchmark data set that's labeled so you can start to get some sense of whether your model's performing well or not then and only then building a minimum viable model so this is potentially even just manual rules or simple linear regression deploying this into production if it's feasible to do so or at least running
+
+60
+00:36:11,520 --> 00:36:50,640
+this on your existing problem so you have a baseline and then lastly it's worth just restating making sure that you once you've built this minimum viable model that may not even use ml just really asking yourself the question of whether this is good enough for now or whether it's worth putting in the additional effort to turn this into a complex ml system the next point i want to make here is that not all ml projects really have the same characteristics and so should be and so you shouldn't think about planning all ml projects in the same way i want to talk about some archetypes of different types of ml projects and the implications that they have for the feasibility of the projects and how you might run the projects
+
+61
+00:36:49,040 --> 00:37:28,560
+effectively and so the three archetypes i want to talk to are defined by how they interact with real world users and so the first archetype is software 2.0 use cases and so i would define this as taking something that software does today so an existing part of your product that you have let's say and doing it better more accurately or more efficiently with ml it's taking a part of your product that's already automated or already partially automated and adding more automation or more efficient automation using machine learning then the next archetype is human in the loop systems and so this is where you take something that is not currently automated in your system but it's something that humans are doing or
+
+62
+00:37:26,720 --> 00:38:07,839
+humans could be doing and helping them do that job better more efficiently or more accurate accurately by supplementing their judgment with ml based tools preventing them from needing to do the job on every single data point by giving them suggestions of what they can do so they can shortcut their process in a lot of places human loop systems are about making the humans that are ultimately making the decisions more efficient or more effective and then lastly autonomous systems and so these are systems where you take something that humans do today or maybe is just not being done at all today and fully automated with ml to the point where you actually don't need humans to do the judgment piece of it at all and so some
+
+63
+00:38:05,440 --> 00:38:47,200
+examples of software 2.0 are if you have an ide that has code completion can we do better code completion by using ml can we take a recommendation system that is initially using some simple rules and making it more customized can we take our video game ai that's using this rule-based system and make it much better by using machine learning some examples of human and loop systems would be building a product to turn hand-drawn sketches into slides you still have a human on the other end that's evaluating the quality of those sketches before they go in front of a customer or stakeholder so it's a human in the loop system but it's potentially saving a lot of time for that human email auto completion so if you use
+
+64
+00:38:45,359 --> 00:39:23,119
+gmail you've seen these email suggestions where it'll suggest sort of short responses to the email that you got i get to decide whether that email actually goes out to the world so it's not an automation system it's a human in the loop system or helping a radiologist do their job faster and then examples of autonomous systems are things like full self-driving right maybe there's not even a steering wheel in the car i can't interrupt the autonomous system and take over control of the car even if i wanted to or maybe it's not designed for me to do that very often fully automated customer support so if i go on a company's website and i interact with their customer support without even having the option of talking to an agent
+
+65
+00:39:21,280 --> 00:39:56,720
+or with them making it very difficult to talk to an agent that's an autonomous system or for example like fully automating website design so that to the point where people who are not design experts can just click a button and get a website designed for them and so i think some of the key questions that you need to ask before embarking on these projects are a little bit different depending on which archetype your project falls into so if you're working on a software 2.0 project then i think some of the questions you should be concerned about are how do you know that your models are actually performing improving performance over the baseline that you already have how confident are you that the type of performance improvement that
+
+66
+00:39:54,960 --> 00:40:32,560
+you might be able to get from ml is actually going to generate value for your business if it's just one percent better is that really worth the cost then do these performance improvements lead to what's called a data flywheel which i'll talk a little bit more about with human in the loop systems you might ask a different set of questions before you embark on the project like how good does the system actually need to be useful if the system you know is able to automate 10 of the work of the human that is ultimately making the decisions or producing the end product is that useful to them or does that just slow it slow them down how can you collect enough data to make it that good is it possible to actually build a data set
+
+67
+00:40:30,720 --> 00:41:08,400
+that is able to get you to that useful threshold for your system and for autonomous systems the types of questions you might ask are what is an acceptable failure rate for this system how many nines in your performance threshold do you need in order for this sort of not to cause harm in the world and how can you guarantee like how can you be really confident that one it won't exceed that failure rate and so this is something that in autonomous vehicles for example teams put a ton of effort into building the simulation and testing systems that they need to be confident that they won't exceed the failure rate that's except the very very low failure rate that's acceptable for those systems i want to double click on this data
+
+68
+00:41:06,160 --> 00:41:49,040
+flywheel concept for software 2.0 we talked about can we build a data flywheel that lead to better and better performance of the system and the way to think about a data flywheel is it's this virtuous cycle where as your model gets better you are able to use a better that better model to make a better product which allows you to acquire more users and as you have more users those users generate more data which you can use to build a better model and this creates this virtuous cycle and so the connections between each of these steps are also important in order for more users to allow you to collect more data you need to have a data loop where you need to have a way of automatically collecting data and deciding what data points to
+
+69
+00:41:46,960 --> 00:42:23,839
+label from your users or at least processes for doing these in order for more data to lead to a better model that's that's kind of on you as an ml practitioner right like you need to be able to translate more data more granular data more labels into a model that performs better for your users and then in order for the better model to lead to better users you need to be sure that better predictions are actually making your product better another point that i want to make on these project archetypes is i would sort of characterize them as having different trade-offs on this feasibility versus impact two by two that we talked about earlier software 2.0 projects since they're just taking something that you
+
+70
+00:42:22,480 --> 00:43:00,000
+already know you can automate and automating it better tend to be more feasible but since you already have an answer to the question that they're also answering they also tend to be lower impact on the other extreme autonomous systems tend to be very difficult to build because the accuracy requirements in general are quite high but the impact can be quite high as well because you're replacing something that literally doesn't exist and human in the loop systems tend to be somewhere in between where you can really like you can use this paradigm of machine learning products to build things that couldn't exist before but the impact is not quite as high because you still need people in the loop that are helping use their judgment
+
+71
+00:42:57,599 --> 00:43:40,400
+to complement the machine learning model there's ways that you can move these types of projects on the feasibility impact matrix to make them more likely to succeed so if you're working on a software 2.0 project you can make these projects have potentially higher impacts by implementing a data loop that allows you to build continual improvement data flywheel that we talked about before and potentially allows you to use the data that you're collecting from users interacting with this system to automate more tasks in the future so for example in the code completion ide example that we gave before you can you know if you're building something like github copilot then think about all the things that the data that you're collecting
+
+72
+00:43:38,560 --> 00:44:20,240
+from that could be useful for building in the future you can make human in the loop systems more feasible through good product design and we'll talk a little bit more about this in a future lecture but there's design paradigms in the product itself that can reduce the accuracy requirement for these types of systems and another way to make these projects more feasible is by adopting sort of a different mindset which is let's just make the system good enough and ship it into the real world so we can start the process of you know seeing how how real users interact with it and using the feedback that we get from our humans in the loop to make the model better and then lastly autonomous systems can be made more feasible by adding guard rails
+
+73
+00:44:18,240 --> 00:44:59,119
+or in some cases adding humans in the loop and so this is you can think of this as the approach to autonomous vehicles where you have safety drivers in the loop early on in the project or where you introduce tele operations so that a human can take control of the system if it looks like something is going wrong i think another point that is really important here is despite all this talk about what's feasible to do with ml the complexity that ml introduce is in your system i don't mean by any of this to say that you should do necessarily a huge amount of planning before you dive into using ml at all just make sure that the project that you're working on is the right project and then just dive in and get started and in particular i think a
+
+74
+00:44:57,359 --> 00:45:34,960
+failure mode that i'm seeing crop up more and more over the past couple of years that you should avoid is falling into the trap of tool fetishization so one of the great things that's happened in ml over the past couple of years is the rise of this ml ops discipline and alongside of that has been proliferation of different tools that are available on the market to help with different parts of the ml process and one thing that i've noticed that this has caused for a lot of folks is this sort of general feeling that you really need to have perfect tools before you get started you don't need perfect tools to get started and you also don't need a perfect model and in particular just because google or uber is doing
+
+75
+00:45:33,599 --> 00:46:12,400
+something like just because they have you know a feature store as part of their stack or they serve models in a particular way doesn't mean that you need to have that as well and so a lot of what we'll try to do in this class is talk about what's the middle ground be between doing things in the right way from a production perspective but not introducing too much complexity early on into your project so that's one of the reasons why fsdl is a class about building ml powered products in a practical way and not in mlaps class that's focused on what is the state of the art in the best possible infrastructure that you can use and um a talk and blog posts and associated set of things on this concept that i really
+
+76
+00:46:09,520 --> 00:46:54,960
+like is this ml offset reasonable scale push by some of the folks from kovio and the sort of central thesis of ml offs at reasonable scale is you're not google you probably have a finite compute budget not entire cloud you probably have a limited number of folks on your team you probably have not an infinite budget to spend on this and you probably have a limited amount of data as well and so those differences between what you have and what uber has or what google has have implications for what the right stack is for the problems that you're solving and so it's worth thinking about these cases separately and so if you're interested in what one company did and recommends for an ml stack that isn't designed to
+
+77
+00:46:52,000 --> 00:47:31,200
+scale to becoming uber scale then i recommend checking out this talk to summarize what we've covered so far machine learning is an incredibly powerful technology but it does add a lot of complexity and so before you embark on a machine learning project you should make sure that you're thinking carefully about whether you really need ml to solve the problem that you're solving and whether the problem is actually worth solving at all given the complexity that this adds and so let's avoid being ml teams that have their projects get killed because we're working on things that don't really matter to the business that we're a part of all right and the last topic i want to cover today is once you've sort of made this decision to embark on an ml
+
+78
+00:47:29,520 --> 00:48:07,599
+project what are the different steps that you're going to go through in order to actually execute on that project and this will also give you an outline for some of the other things you can expect from the class so the running case study that we'll use here is a modified version of a problem that i worked on when i was at open ai which is pose estimation our goal is to build a system that runs on a robot that takes the camera feed from that robot and uses it to estimate the position in 3d space and the orientation the rotation of each of the objects in the scene so that we can use those for downstream tasks and in particular so we can use them to feed into a separate model which will be used to tell the robot how it
+
+79
+00:48:06,000 --> 00:48:40,000
+actually can grasp the different objects in the scene machine learning projects start like any other project in a planning and project setup phase and so what the types of activities we'd be doing in this phase when we're working on this pose estimation project are things like deciding to work on post-estimation at all determining whether how much this is going to cost what resources we need to allocate to it considering the ethical implications and things like this right a lot of what we've been talking about so far in this lecture once we plan the project then we'll move into a data collection and labeling phase and so for pose estimation what this might look like is collecting the corpus of objects that
+
+80
+00:48:38,640 --> 00:49:18,559
+we're going to train our model on setting up our sensors like our cameras to capture our information about those objects actually capturing those objects and somehow figuring out how to annotate these images that we're capturing with ground truth like the pose of the of the objects in those images one point i want to make about the life cycle of mbl projects is that this is not like a straightforward path machine learning projects tend to be very iterative and each of these phases can feed back into any of the phases before as you learn more about the problem that you're working on so for example you might realize that actually it's way too hard for us to get data in order to solve this problem or it's really difficult for us to label
+
+81
+00:49:16,079 --> 00:49:54,559
+the pose of these objects in 3d space but what we can do is it's actually much cheaper for us to annotate like per pixel segmentation so can we reformulate the problem in a way that allows us to to use what we've learned about data collection and labeling to plan a better project once you have some data to work on then you enter the sort of training and debugging phase and so what we might do here is we might implement a baseline for our model not using like a complex neural network but just using some opencv functions and then once we have that working we might find a state-of-the-art model and reproduce it debug our implementation and iterate on our model run some hyper parameter sweeps until it performs well
+
+82
+00:49:52,720 --> 00:50:29,599
+on our task this can feed back into the data collection and labeling phase because we might realize that you know we actually need more data in order to solve this problem or we might also realize that there's something flawed in the process that we've been using to label the data that we're using data labeling process might need to be revisited but we can also loop all the way back to the project planning phase because we might realize that actually this task is a lot harder than we thought or the requirements that we specified at the planning phase trade off with each other so we need to revisit which are most important so for example like maybe we thought that we had an accuracy requirement of estimating the pose of these objects to
+
+83
+00:50:26,960 --> 00:51:06,720
+one tenth of one centimeter and we also had an a latency requirement for inference in our models of 1 100th of a second to run on robotic hardware and we might realize that hey you know we can get this really really tight accuracy requirement or we can have really fast inference but it's very difficult to do both so is it possible to relax one of those assumptions once you've trained a model that works pretty well offline for your task then your goal is going to be to deploy that model test it in the real world and then use that information to figure out where to go next for the purpose of this project that might look like piloting the grasping system in the lab so before we roll it out to actual users can we
+
+84
+00:51:04,880 --> 00:51:42,319
+test it in a realistic scenario and we might also do things like writing tests to prevent regressions and evaluate for bias in the model and then eventually rolling this out into production and monitoring it and continually improving it from there and so we can feed back here into the training and debugging stage because oftentimes what we'll find is that the model that worked really well for our offline data set once it gets into the real world it doesn't actually work as well as we thought whether that's because the accuracy requirement that we had for the model was wrong like we actually needed it to be more accurate than we thought or maybe the metric that we're looking at the accuracy is not actually the metric
+
+85
+00:51:39,760 --> 00:52:17,920
+that really matters for success at the downstream task that we're trying to solve because that could cause us to revisit the training phase we also could loop back to the data collection and labeling phase because common problem that we might find in the real world is that there's some mismatch between the training data that we collected and the data that we actually saw when we went out and tested this we could use what we learned from that to go collect more data or mine for hard cases like mine for the failure cases that we found in production and then finally as i alluded to before we could loop all the way back to the project planning phase because we realized that the metric that we picked doesn't really drive the downstream
+
+86
+00:52:15,920 --> 00:52:51,599
+behavior that we desired just because the grasp model is accurate doesn't mean that the robot will actually be able to successfully grasp the object so we might need to use a different metric to really solve this task or we might realize that the performance in the real world isn't that great and so we maybe need to add additional requirements to our model as well maybe it just needs to be faster to in order to run on a real robot so these are kind of like what i think of as the activities that you do in any particular machine learning project that you undertake but there's also some sort of cross project things that you need in order to be successful which we'll talk about in the class as well you need to be able to work on
+
+87
+00:52:49,920 --> 00:53:25,040
+these problems together as a team and you need to have the right infrastructure and tooling to make these processes more repeatable and these are topics that we'll cover as well so this is like a broad conceptual outline of the different topics that we'll talk about in this class and so to wrap up for today what we covered is machine learning is a complex technology and so you should use it because you need it or because you think it'll generate a lot of value but it's not a cure-all it doesn't solve every problem it won't automate every single thing that you wanted to automate so let's pick projects that are going to be valuable but in spite of this you don't need a perfect setup to get started and let's
+
+88
+00:53:23,440 --> 00:53:34,760
+spend the rest of this course walking through the project lifecycle and learning about each of these stages and how we can how we can use them to build great ml powered products
+

documents/lecture-02.md

@@ -0,0 +1,563 @@
+---
+description: Software engineering, Deep learning frameworks, Distributed training, GPUs, and Experiment Management.
+---
+
+# Lecture 2: Development Infrastructure & Tooling
+
+<div align="center">
+<iframe width="720" height="405" src="https://www.youtube-nocookie.com/embed/BPYOsDCZbno?list=PL1T8fO7ArWleMMI8KPJ_5D5XSlovTW_Ur" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+Lecture by [Sergey Karayev](https://twitter.com/sergeykarayev).
+Notes by [James Le](https://twitter.com/le_james94) and [Vishnu Rachakonda](https://www.linkedin.com/in/vrachakonda/).<br />
+Published August 15, 2022.
+[Download slides](https://drive.google.com/open?id=16pEG5GesO4_UAWiD5jrIReMGzoyn165M).
+
+## 1 - Introduction
+
+The **dream** of ML development is that given a project spec and some
+sample data, you get a continually improving prediction system deployed
+at scale.
+
+The **reality** is starkly different:
+
+-   You have to collect, aggregate, process, clean, label, and version
+the data.
+
+-   You have to find the model architecture and their pre-trained
+weights and then write and debug the model code.
+
+-   You run training experiments and review the results, which will be
+fed back into the process of trying out new architectures and
+debugging more code.
+
+-   You can now deploy the model.
+
+-   After model deployment, you have to monitor model predictions and
+close the data flywheel loop. Basically, your users generate fresh
+data for you, which needs to be added to the training set.
+
+![](./media/image3.png)
+
+
+This reality has roughly three components: data, development, and
+deployment. The tooling infrastructure landscape for them is large, so
+we'll have three lectures to cover it all. **This lecture focuses on the
+development component**.
+
+## 2 - Software Engineering
+
+![](./media/image7.png)
+
+
+### Language
+
+For your choice of **programming language**, Python is the clear winner
+in scientific and data computing because of all the libraries that have
+been developed. There have been some contenders like Julia and C/C++,
+but Python has really won out.
+
+### Editors
+
+To write Python code, you need an **editor**. You have many options,
+such as Vim, Emacs, Jupyter Notebook/Lab, VS Code, PyCharm, etc.
+
+-   We recommend [VS Code](https://code.visualstudio.com/)
+because of its nice features such as built-in git version control,
+documentation peeking, remote projects opening, linters and type
+hints to catch bugs, etc.
+
+-   Many practitioners develop in [Jupyter
+Notebooks](https://jupyter.org/), which is great as
+the "first draft" of a data science project. You have to put in
+little thought before you start coding and seeing the immediate
+output. However, notebooks have a variety of problems: primitive
+editor, out-of-order execution artifacts, and challenges to
+version and test them. A counterpoint to these problems is the
+[nbdev package](https://nbdev.fast.ai/) that lets
+you write and test code all in one notebook environment.
+
+-   We recommend you use **VS Code with built-in support for
+notebooks** - where you can write code in modules imported into
+notebooks. It also enables awesome debugging.
+
+If you want to build something more interactive,
+[Streamlit](https://streamlit.io/) is an excellent choice.
+It lets you decorate Python code, get interactive applets, and publish
+them on the web to share with the world.
+
+![](./media/image10.png)
+
+
+For setting up the Python environment, we recommend you see [how we did
+it in the
+lab.](https://github.com/full-stack-deep-learning/conda-piptools)
+
+## 3 - Deep Learning Frameworks
+
+![](./media/image15.png)
+
+
+Deep learning is not a lot of code with a matrix math library like
+Numpy. But when you have to deploy your code onto CUDA for GPU-powered
+deep learning, you want to consider deep learning frameworks as you
+might be writing weird layer types, optimizers, data interfaces, etc.
+
+### Frameworks
+
+There are various frameworks, such as PyTorch, TensorFlow, and Jax. They
+are all similar in that you first define your model by running Python
+code and then collect an optimized execution graph for different
+deployment patterns (CPU, GPU, TPU, mobile).
+
+1.  We prefer PyTorch because [it is absolutely
+dominant](https://www.assemblyai.com/blog/pytorch-vs-tensorflow-in-2022/)
+by measures such as the number of models, the number of papers,
+and the number of competition winners. For instance, about [77%
+of 2021 ML competition winners used
+PyTorch](https://blog.mlcontests.com/p/winning-at-competitive-ml-in-2022?s=w).
+
+2.  With TensorFlow, you have TensorFlow.js (that lets you run deep
+learning models in your browser) and Keras (an unmatched developer
+experience for easy model development).
+
+3.  Jax is a meta-framework for deep learning.
+
+![](./media/image12.png)
+
+
+[PyTorch](https://pytorch.org/) has excellent developer
+experience and is production-ready and even faster with TorchScript.
+There is a great distributed training ecosystem. There are libraries for
+vision, audio, etc. There are also mobile deployment targets.
+
+[PyTorch Lightning](https://www.pytorchlightning.ai/)
+provides a nice structure for organizing your training code, optimizer
+code, evaluation code, data loaders, etc. With that structure, you can
+run your code on any hardware. There are nice features such as
+performance and bottleneck profiler, model checkpointing, 16-bit
+precision, and distributed training libraries.
+
+Another possibility is [FastAI
+software](https://www.fast.ai/), which is developed
+alongside the fast.ai course. It provides many advanced tricks such as
+data augmentations, better initializations, learning rate schedulers,
+etc. It has a modular structure with low-level API, mid-level API,
+high-level API, and specific applications. The main problem with FastAI
+is that its code style is quite different from mainstream Python.
+
+At FSDL, we prefer PyTorch because of its strong ecosystem, but
+[TensorFlow](https://www.tensorflow.org/) is still
+perfectly good. If you have a specific reason to prefer it, you are
+still going to have a good time.
+
+[Jax](https://github.com/google/jax) is a more recent
+project from Google that is not specific to deep learning. It provides
+general vectorization, auto-differentiation, and compilation to GPU/TPU
+code. For deep learning, there are separate frameworks like
+[Flax](https://github.com/google/flax) and
+[Haiku](https://github.com/deepmind/dm-haiku). You should
+only use Jax for a specific need.
+
+### Meta-Frameworks and Model Zoos
+
+Most of the time, you will start with at least a model architecture that
+someone has developed or published. You will use a specific architecture
+(trained on specific data with pre-trained weights) on a model hub.
+
+-   [ONNX](https://onnx.ai/) is an open standard for
+saving deep learning models and lets you convert from one type of
+format to another. It can work well but can also run into some
+edge cases.
+
+-   [HuggingFace](https://huggingface.co/) has become an
+absolutely stellar repository of models. It started with NLP tasks
+but has then expanded into all kinds of tasks (audio
+classification, image classification, object detection, etc.).
+There are 60,000 pre-trained models for all these tasks. There is
+a Transformers library that works with PyTorch, TensorFlow, and
+Jax. There are 7,500 datasets uploaded by people. There's also a
+community aspect to it with a Q&A forum.
+
+-   [TIMM](https://github.com/rwightman/pytorch-image-models)
+is a collection of state-of-the-art computer vision models and
+related code that looks cool.
+
+## 4 - Distributed Training
+
+![](./media/image9.png)
+
+
+Let's say we have multiple machines represented by little squares above
+(with multiple GPUs in each machine). You are sending batches of data to
+be processed by a model with parameters. The data batch can fit on a
+single GPU or not. The model parameters can fit on a single GPU or not.
+
+The best case is that both your data batch and model parameters fit on a
+single GPU. That's called **trivial parallelism**. You can either launch
+more independent experiments on other GPUs/machines or increase the
+batch size until it no longer fits on one GPU.
+
+### Data Parallelism
+
+If your model still fits on a single GPU, but your data no longer does,
+you have to try out **data parallelism** - which lets you distribute a
+single batch of data across GPUs and average gradients that are computed
+by the model across GPUs. A lot of model development work is cross-GPU,
+so you want to ensure that GPUs have fast interconnects.
+
+If you are using a server card, expect [a linear
+speedup](https://lambdalabs.com/blog/best-gpu-2022-sofar/)
+in training time. If you are using a consumer card, expect [a sublinear
+speedup](https://lambdalabs.com/blog/titan-v-deep-learning-benchmarks/)
+instead.
+
+Data parallelism is implemented in PyTorch with the robust
+[DistributedDataParallel
+library](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html).
+[Horovod](https://github.com/horovod/horovod) is another
+3rd-party library option. PyTorch Lightning makes it dead simple to use
+either of these two libraries - where [speedup seems to be the
+same](https://www.reddit.com/r/MachineLearning/comments/hmgr9g/d_pytorch_distributeddataparallel_and_horovod/).
+
+A more advanced scenario is that you can't even fit your model on a
+single GPU. You have to spread the model over multiple GPUs. There are
+three solutions to this.
+
+### Sharded Data-Parallelism
+
+Sharded data parallelism starts with the question: What exactly takes up
+GPU memory?
+
+-   The **model parameters** include the floats that make up our model
+layers.
+
+-   The **gradients** are needed to do back-propagation.
+
+-   The **optimizer states** include statistics about the gradients
+
+-   Finally, you have to send a **batch of data** for model development.
+
+![](./media/image5.png)
+
+Sharding is a concept from databases where if you have one source of
+data, you actually break it into shards of data that live across your
+distributed system. Microsoft implemented an approach called
+[ZeRO](https://arxiv.org/pdf/1910.02054.pdf) that shards
+the optimizer states, the gradients, and the model parameters. **This
+results in an insane order of magnitude reduction in memory use, which
+means your batch size can be 10x bigger.** You should [watch the video
+in this
+article](https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/)
+to see how model parameters are passed around GPUs as computation
+proceeds.
+
+Sharded data-parallelism is implemented by Microsoft's
+[DeepSpeed](https://github.com/microsoft/DeepSpeed)
+library and Facebook's
+[FairScale](https://github.com/facebookresearch/fairscale)
+library, as well as natively by PyTorch. In PyTorch, it's called
+[Fully-Sharded
+DataParallel](https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/).
+With PyTorch Lightning, you can try it for a massive memory reduction
+without changing the model code.
+
+This same ZeRO principle can also be applied to a single GPU. You can
+train a 13B-parameter model on a single V100 (32GB) GPU. Fairscale
+implements this (called
+[CPU-offloading](https://fairscale.readthedocs.io/en/stable/deep_dive/offload.html)).
+
+### Pipelined Model-Parallelism
+
+**Model parallelism means that you can put each layer of your model on
+each GPU**. It is trivial to implement natively but results in only one
+GPU being active at a time. Libraries like DeepSpeed and FairScale make
+it better by pipelining computation so that the GPUs are fully utilized.
+You need to tune the amount of pipelining on the batch size to the exact
+degree of how you will split up the model on the GPU.
+
+### Tensor-Parallelism
+
+Tensor parallelism is another approach, which observes that there is
+nothing special about matrix multiplication that requires the whole
+matrix to be on one GPU. **You can distribute the matrix over multiple
+GPUs**. NVIDIA published [the Megatron-LM
+repo](https://github.com/NVIDIA/Megatron-LM), which does
+this for the Transformer model.
+
+You can actually use all of the three techniques mentioned above if you
+really want to scale a huge model (like a GPT-3 sized language model).
+Read [this article on the technology behind BLOOM
+training](https://huggingface.co/blog/bloom-megatron-deepspeed)
+for a taste.
+
+![](./media/image6.png)
+
+
+In conclusion:
+
+-   If your model and data fit on one GPU, that's awesome.
+
+-   If they do not, and you want to speed up training, try
+DistributedDataParallel.
+
+-   If the model still doesn't fit, try ZeRO-3 or Full-Sharded Data
+Parallel.
+
+For more resources to speed up model training, look at [this list
+compiled by DeepSpeed](https://www.deepspeed.ai/training/),
+[MosaicML](https://www.mosaicml.com), and
+[FFCV](https://ffcv.io).
+
+## 5 - Compute
+
+![](./media/image14.png)
+
+
+**Compute** is the next essential ingredient to developing machine
+learning models and products.
+
+The compute-intensiveness of models has grown tremendously over the last
+ten years, as the below charts from
+[OpenAI](https://openai.com/blog/ai-and-compute/) and
+[HuggingFace](https://huggingface.co/blog/large-language-models)
+show.
+
+![](./media/image1.png)
+
+
+Recent developments, including models like
+[GPT-3](https://openai.com/blog/gpt-3-apps/), have
+accelerated this trend. These models are extremely large and require a
+large number of petaflops to train.
+
+### GPUs
+
+**To effectively train deep learning models**, **GPUs are required.**
+NVIDIA has been the superior choice for GPU vendors, though Google has
+introduced TPUs (Tensor Processing Units) that are effective but are
+only available via Google Cloud. There are three primary considerations
+when choosing GPUs:
+
+1.  How much data fits on the GPU?
+
+2.  How fast can the GPU crunch through data? To evaluate this, is your
+data 16-bit or 32-bit? The latter is more resource intensive.
+
+3.  How fast can you communicate between the CPU and the GPU and between
+GPUs?
+
+Looking at recent NVIDIA GPUs, it becomes clear that a new
+high-performing architecture is introduced every few years. There's a
+difference between these chips, which are licensed for personal use as
+opposed to corporate use; businesses should only use **server**
+**cards**.
+
+![](./media/image8.png)
+
+
+Two key factors in evaluating GPUs are **RAM** and **Tensor TFlops**.
+The more RAM, the better the GPU contains large models and datasets.
+Tensor TFlops are special tensor cores that NVIDIA includes specifically
+for deep learning operations and can handle more intensive
+mixed-precision operations. **A tip**: leveraging 16-bit training can
+effectively double your RAM capacity!
+
+While these theoretical benchmarks are useful, how do GPUs perform
+practically? Lambda Labs offers [the best benchmarks
+here](https://lambdalabs.com/gpu-benchmarks). Their results
+show that the most recent server-grade NVIDIA GPU (A100) is more than
+2.5 times faster than the classic V100 GPU. RTX chips also outperform
+the V100. [AIME is also another source of GPU
+benchmarks](https://www.aime.info/en/blog/deep-learning-gpu-benchmarks-2021/).
+
+Cloud services such as Microsoft Azure, Google Cloud Platform, and
+Amazon Web Services are the default place to buy access to GPUs. Startup
+cloud providers like
+[Paperspace](https://www.paperspace.com/),
+[CoreWeave](https://www.coreweave.com/), and [Lambda
+Labs](https://lambdalabs.com/) also offer such services.
+
+### TPUs
+
+Let's briefly discuss TPUs. There are four generations of TPUs, and the
+most recent v4 is the fastest possible accelerator for deep learning. V4
+TPUs are not generally available yet, but **TPUs generally excel at
+scaling to larger and model sizes**. The below charts compare TPUs to
+the fastest A100 NVIDIA chip.
+
+![](./media/image11.png)
+
+
+It can be overwhelming to compare the cost of cloud access to GPUs, so
+[we made a tool that solves this
+problem](https://fullstackdeeplearning.com/cloud-gpus/)!
+Feel free to contribute to [our repository of Cloud GPU cost
+metrics](https://github.com/full-stack-deep-learning/website/).
+The tool has all kinds of nifty features like enabling filters for only
+the most recent chip models, etc.
+
+If we [combine the cost metrics with performance
+metrics](https://github.com/full-stack-deep-learning/website/blob/main/docs/cloud-gpus/benchmark-analysis.ipynb),
+we find that **the most expensive per hour chips are not the most
+expensive per experiment!** Case in point: running the same Transformers
+experiment on 4 V100s costs \$1750 over 72 hours, whereas the same
+experiment on 4 A100s costs \$250 over only 8 hours. Think carefully
+about cost and performance based on the model you're trying to train.
+
+Some helpful heuristics here are:
+
+1.  Use the most expensive per-hour GPU in the least expensive cloud.
+
+2.  Startups (e.g., Paperspace) tend to be cheaper than major cloud
+providers.
+
+### On-Prem vs. Cloud
+
+For **on-prem** use cases, you can build your own pretty easily or opt
+for a pre-built computer from a company like NVIDIA. You can build a
+good, quiet PC with 128 GB RAM and 2 RTX 3909s for about \$7000 and set
+it up in a day. Going beyond this can start to get far more expensive
+and complicated. Lambda Labs offers a \$60,000 machine with 8 A100s
+(super fast!). Tim Dettmers offers a great (slightly outdated)
+perspective on building a machine
+[here](https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/).
+
+Some tips on on-prem vs. cloud use:
+
+-   It can be useful to have your own GPU machine to shift your mindset
+from minimizing cost to maximizing utility.
+
+-   To truly scale-out experiments, you should probably just use the
+most expensive machines in the least expensive cloud.
+
+-   TPUs are worth experimenting with for large-scale training, given
+their performance.
+
+-   Lambda Labs is a sponsor, and we highly encourage looking at them
+for on-prem and cloud GPU use!
+
+## 6 - Resource Management
+
+![](./media/image2.png)
+
+
+Now that we've talked about raw compute, let's talk about options for
+**how to manage our compute resources**. Let's say we want to manage a
+set of experiments. Broadly speaking, we'll need hardware in the form of
+GPUs, software requirements (e.g., PyTorch version), and data to train
+on.
+
+### Solutions
+
+Leveraging best practices for specifying dependencies (e.g., Poetry,
+conda, pip-tools) makes the process of spinning up such experiments
+quick and easy on a single machine.
+
+If, however, you have a cluster of machines to run experiments on,
+[SLURM](https://slurm.schedmd.com/documentation.html) is
+the tried and true solution for workload management that is still widely
+used.
+
+For more portability, [Docker](https://www.docker.com/) is
+a way to package up an entire dependency stack into a lighter-than-a-VM
+package. [Kubernetes](https://kubernetes.io/) is the most
+popular way to run many Docker containers on top of a cluster. The OSS
+[Kubeflow](https://www.kubeflow.org/) project helps manage
+ML projects that rely on Kubernetes.
+
+These projects are useful, but they may not be the easiest or best
+choice. They're great if you already have a cluster up and running, but
+**how do you actually set up a cluster or compute platform?**
+
+*Before proceeding, FSDL prefers open source and/or transparently priced
+products. We discuss tools that fall into these categories, not SaaS
+with opaque pricing.*
+
+### Tools
+
+For practitioners all in on AWS, [AWS
+Sagemaker](https://aws.amazon.com/sagemaker/) offers a
+convenient end-to-end solution for building machine learning models,
+from labeling data to deploying models. Sagemaker has a ton of
+AWS-specific configuration, which can be a turnoff, but it brings a lot
+of easy-to-use old school algorithms for training and allows you to BYO
+algorithms as well. They're also increasing support for PyTorch, though
+the markup for PyTorch is about 15-20% more expensive.
+
+[Anyscale](https://www.anyscale.com/) is a company created
+by the makers of the Berkeley OSS project
+[Ray](https://github.com/ray-project/ray). Anyscale
+recently launched [Ray
+Train](https://docs.ray.io/en/latest/train/train.html),
+which they claim is faster than Sagemaker with a similar value
+proposition. Anyscale makes it really easy to provision a compute
+cluster, but it's considerably more expensive than alternatives.
+
+[Grid.ai](https://www.grid.ai/) is created by the PyTorch
+Lightning creators. Grid allows you to specify what compute parameters
+to use easily with "grid run" followed by the types of compute and
+options you want. You can use their instances or AWS under the hood.
+Grid has an uncertain future, as its future compatibility with Lightning
+(given their rebrand) has not been clarified.
+
+There are several non-ML options for spinning up compute too! Writing
+your own scripts, using various libraries, or even Kubernetes are all
+options. This route is harder.
+
+[Determined.AI](https://determined.ai/) is an OSS solution
+for managing on-prem and cloud clusters. They offer cluster management,
+distributed training, and more. It's pretty easy to use and is in active
+development.
+
+With all this said, **there is still room to improve the ease of
+experience for launching training on many cloud providers**.
+
+## 7 - Experiment and Model Management
+
+![](./media/image4.png)
+
+
+In contrast to compute, **experiment management is quite close to being
+solved**. Experiment management refers to tools and processes that help
+us keep track of code, model parameters, and data sets that are iterated
+on during the model development lifecycle. Such tools are essential to
+effective model development. There are several solutions here:
+
+-   [TensorBoard](https://www.tensorflow.org/tensorboard):
+A non-exclusive Google solution effective at one-off experiment
+tracking. It is difficult to manage many experiments.
+
+-   [MLflow](https://mlflow.org/): A non-exclusive
+Databricks project that includes model packaging and more, in
+addition to experiment management. It must be self-hosted.
+
+-   [Weights and Biases](https://wandb.ai/site): An
+easy-to-use solution that is free for personal and academic projects! Logging
+starts simply with an "experiment config" command.
+
+-   Other options include [Neptune
+AI](https://neptune.ai/), [Comet
+ML](https://www.comet.ml/), and [Determined
+AI](https://determined.ai/), all of which have solid
+experiment tracking options.
+
+Many of these platforms also offer **intelligent hyperparameter
+optimization**, which allows us to control the cost of searching for the
+right parameters for a model. For example, Weights and Biases has a
+product called [Sweeps](https://wandb.ai/site/sweeps) that
+helps with hyperparameter optimization. It's best to have it as part of
+your regular ML training tool; there's no need for a dedicated tool.
+
+## 8 - "All-In-One"
+
+![](./media/image13.png)
+
+
+There are machine learning infrastructure solutions that offer
+everything\--training, experiment tracking, scaling out, deployment,
+etc. These "all-in-one" platforms simplify things but don't come cheap!
+Examples include [Gradient by
+Paperspace](https://www.paperspace.com/gradient), [Domino
+Data Lab](https://www.dominodatalab.com/), [AWS
+Sagemaker](https://aws.amazon.com/sagemaker/), etc.

documents/lecture-02.srt

@@ -0,0 +1,256 @@
+1
+00:00:00,399 --> 00:00:49,360
+hi everyone welcome to week two of full stack deep learning 2022. today we have a lecture on development infrastructure and tooling my name is sergey and i have my assistant mishka right here so just diving right in the dream of machine learning development is that you provide a project spec identify birds maybe some sample data here's what the birds look like here's what i want to see and then you get a continually improving prediction system and it's deployed at scale but the reality is that it's not just some sample data you really have to find the data aggregated process it clean it label it then you have to find the model architecture potentially the pre-trained weights then you still have to look at the model code
+
+2
+00:00:46,480 --> 00:01:32,640
+probably edit it debug it run training experiments review the results that's going to feed back into maybe trying a new architecture debugging some more code and then when that's done you can actually deploy the model and then after you deploy it you have to monitor the predictions and then you close the data flywheel loop basically your user is generating fresh data for you that that you then have to add to your data set so this reality has roughly kind of three components and we divided into data and read this development in yellow and deployment in green and there are a lot of tools like the infrastructure landscape is pretty large so we have three lectures to cover all of it and today we're going to concentrate on
+
+3
+00:01:30,240 --> 00:02:16,239
+the development part the middle part which is probably what you're familiar with from previous courses most of what you do is model development we actually want to start even a little bit before that and talk about software engineering you know it starts with maybe the programming language and for machine learning it's pretty clear it has to be python and the reason is because of all the libraries that have been developed for it it's just the winner in scientific and data computing there have been some contenders so julia is actually the the ju in jupiter jupiter notebooks to write python code you need an editor you can be old school and use vim or emacs a lot of people just write in jupyter notebooks or jupyter lab which
+
+4
+00:02:14,000 --> 00:03:06,400
+also gives you a code editor window vs code is a very popular text editor python specific code editor pycharm is is really good as well at fsdl we recommend vs code it has a lot of nice stuff it hasn't built you know in addition to the nice editing features it has built-in git version control so you can see your commit you can actually stage line by line you can look at documentation as you write your code you can open projects remotely so like the window i'm showing here is actually on a remote machine that i've sshed into you can lend code as you write and if you haven't seen linters before it's basically this idea that if there are code style rules that you want to follow like a certain number of spaces
+
+5
+00:03:04,959 --> 00:03:49,280
+for indentation whatever you decide you want to do gotta you should just codify it so that you don't ever have to think about it or manually put that in your tools just do it for you and you've run something that just looks at your code all the time you can do a little bit of static analysis so for example there's two commas in a row it's not going to run in this file or potentially you're using a variable that never got defined and in addition python now has type hints so you can actually say you know this variable is supposed to be an integer and then if you use it as an argument to a function that expects expect to float a static type checker can catch and tell you about it before you actually run it so we set
+
+6
+00:03:46,799 --> 00:04:35,040
+that all up in the lab by the way and you will see how that works it's a very nice part of the lab a lot of people develop in jupiter notebooks and they're really fundamental to data science and i think for good reason i think it's a great kind of first draft of a project you just open up this notebook and you start coding there's very little thought that you have to put in before you start coding and start seeing immediate output so that kind of like fast feedback cycle that's really great and jeremy howard is a great practitioner so if you watch the fast ai course videos you'll see him use them to their full extent they do have problems though for example the editor that you use in the notebook is pretty primitive right
+
+7
+00:04:32,960 --> 00:05:23,039
+there's no refactoring support there's no maybe peaking of the documentation there's no copilot which i have now got used to in vs code there's out of order execution artifact so if you've run the cells in a different order you might not get the same result as if you ran them all in line it's hard to version them you either strip out the output of each cell in which case you lose some of the benefit because sometimes you want to save the artifact that you produced in the notebook or the file is pretty large and keeps changing and it's hard to test because it's just not very amenable to like the unit testing frameworks and and and best practices that people have built up counterpoint to everything i just said
+
+8
+00:05:20,400 --> 00:06:10,560
+is that you can kind of fix all of that and that's what jeremy howard is trying to do with nbdev which is this package that lets you write documentation your code and test for the code all in a notebook the full site deep learning recommendation is go ahead and use notebooks actually use the vs code built-in notebook support so i actually don't i'm not in the browser ever i'm just in in my vs code but i'm coding in a notebook style but also i usually write code in a module that then gets imported into a notebook and with this live reload extension it's quite nice because when you change code in the module and rerun the notebook that it gets the updated code and also you have nice things like you
+
+9
+00:06:08,960 --> 00:06:55,520
+have a terminal you can look at files and so on and by the way it enables really awesome debugging so if you want to debug some code you can put a breakpoint here on the right you see the little red dot and then i'm about to launch the cell with the debug cell command and it'll drop me in into the debugger at that break point and so this is just really nice without leaving the editor i'm able to to do a lot notebooks are great sometimes you want something a little more interactive maybe something you can share with the world and streamlit has come along and let you just decorate python codes you write a python script you decorate it with widgets and data loaders and stuff and you can get interactive applets
+
+10
+00:06:53,599 --> 00:07:45,120
+where people can let's say a variable can be controlled by a slider and everything just gets rerun very efficiently and then when you're happy with your applet you can publish it to the web and just share that streamlet address with your audience it's really quite great for setting up the python environment it can actually be pretty tricky so for deep learning usually you have a gpu and the gpu needs cuda libraries and python has a version and then each of the requirements that you use like pytorch or numpy have their own specific version also some requirements are for production like torch but some are only for development for example black is a code styling tool where my pi is a static analysis tool and it'd be nice to
+
+11
+00:07:41,440 --> 00:08:34,080
+just separate the two so we can achieve all these desired things by specifying python and cuda versions in environment.yaml file and use conda to install the python and the cuda version that we specified but then all the other requirements we specify in with basically just very minimal constraints so we say like torch version greater than 1.7 or maybe no constraints like numpy any version and then we use this tool called pip tools that will analyze the constraints we gave and the constraints they might have for each other and find a mutually compatible version of all the requirements and then locks it so that when you come back to the project you have exactly the versions of everything you used
+
+12
+00:08:32,640 --> 00:09:25,040
+and we can also just use a make file to simplify this now we do this in lab so you'll see this in lab and on that note please go through labs one through three they're already out and starts with an overview of what the labs are going to be about then pi torch lightning and pytorch and then we go through cnns transformers and we see a lot of the structure that i've been talking about so that is it for software engineering and the next thing i want to talk about are specifically deep learning frameworks and distributed training so why do we need frameworks well deep learning is actually not a lot of code if you have a matrix math library like numpy now fast.ai course does this pretty brilliantly they they basically have you
+
+13
+00:09:23,120 --> 00:10:12,560
+build your own deep learning library and and you see how very little code it is but when you have to deploy stuff onto cuda for gpu power deep learning and when you have to consider that you might be writing weird layers that have to you have to figure out the differentiation of the layers that you write that can get to be just a lot to maintain and so and then also there's all the layer types that have been published in the literature like the convolutional layers there's all the different optimizers so there's just a lot of code and for that you really need a framework so which framework should you use right well i think josh answered this you know pretty concisely about a year ago and you said jax is for researchers pi
+
+14
+00:10:10,480 --> 00:11:00,880
+torches for engineers and tensorflows for boomers so pytorch is the full stack deep learning choice but seriously though you know both pytorch and tensorflow and jaxx they all are similar you define a deep learning model by running python code writing and running python code and then what you get is an optimized execution graph that can target cpus gpus tpus mobile deployments now the reason you might prefer pytorch is because it just basically is absolutely dominant right so if you look at the number of models trained models that are shared on hugging face which is like the largest model zoo we'll talk about it in a few minutes you know there's models that are both pi torch and tensorflow there's some models
+
+15
+00:10:59,600 --> 00:11:50,240
+on jacks there's some models for tensorflow only there's a lot of models that are just for pi torch if you track paper submissions to academic conferences it's about 75 plus percent pi torch implementations of these research papers and my face is blocking the stat but it's something like 75 percent of machine learning competition winners used pytorch in 2022 now tensorflow is kind of cool tensorflow.js in particular lets you run deep learning models in your browser and pytorch doesn't have that and then keras as a development experience is i think pretty unmatched for just stacking together layers easily training the model and then there's jax which you might have heard about so jack's you know the main thing is you
+
+16
+00:11:48,800 --> 00:12:37,200
+need a meta framework for deep learning we'll talk about in a second but pytorch that's the pick excellent dev experience it's people used to say well maybe it's a little slow but it really is production ready even as is but you can make it even faster by compiling your model with a torch script there's a great distributed training ecosystem there's libraries for vision audio 3d data you know etc there's mobile deployment targets and with pytorch lightning which is what we use in labs have a nice structure for how to kind of where do you put your actual model code where you put your optimizer code where do you put your training code your evaluation code how should the data loaders look like and and then what you get is if you just
+
+17
+00:12:34,959 --> 00:13:26,800
+kind of structure your code as pytorch lightning expects it you can run your code on cpu or gpu or any number of gpus or tpus with just you know a few characters change in your code there's a performance profiler there's model checkpointing there's 16-bit precision there's distributed training libraries it's just all very nice to use now another possibility is fast ai software which is developed alongside the fastai cores and it provides a lot of advanced tricks like data augmentations better weight initializations learning grade schedulers it has this kind of modular structure where there's data blocks and learners and then even vision text tabular applications the main problem with it that i see is
+
+18
+00:13:24,399 --> 00:14:20,560
+the code style is quite different and in general it's it's a bit different than than mainstream pie torch it can be very powerful if you go in on it at fsdl we recommend pytorch lightning tensorflow is not just for boomers right fsdl prefers pi torch because we think it's a stronger ecosystem but tensorflow is still perfectly good and if you have a specific reason to prefer it such as that's what your employer uses you're gonna have a good time it still makes sense it's not bad jax is a recent a more recent project from google which is really not specific to deep learning it's about just general vectorization of all kinds of code and also auto differentiation of all kinds of code including your physics simulations
+
+19
+00:14:19,040 --> 00:15:03,440
+stuff like that and then whatever you can express in jax gets compiled to gpu or tpu code and super fast for deep learning there are separate frameworks like flax or haiku and you know here at fsdl we say use it if you have a specific need maybe you're doing research on something kind of weird that's fine or you know potentially you're working at google you're not allowed to use pytorch that could make it a pretty good reason to use jacks there's also this notion of meta frameworks and model zoos that i want to cover so model zooz is the idea that sure you can just start with blank pi torch but most of the time you're going to start with at least a model architecture that someone's developed and published
+
+20
+00:15:02,320 --> 00:15:49,519
+and a lot of the time you're going to start with actually a pre-trained model meaning someone trained the architecture on specific data they got weights that they then saved and uploaded to a hub and you can download and actually start not from scratch but from a pre-trained model onyx is this idea that deep learning models are all about the same right like we know what an mlp type of layer is we know what a cnn type of layer is and it doesn't matter if it's written in pytorch or tensorflow or cafe whatever it's written in we should be able to actually port it between the different code bases because the real thing that we're that we care about are the weights and the weights are just numbers right so onyx is this format that lets you
+
+21
+00:15:47,920 --> 00:16:39,279
+convert from pytorch to tensorflow and vice versa and it can work super well it can also not work super well you can run into some edge cases so if it's something that you need to do then definitely worth a try but it's not necessarily going to work for all types of models hugging face has become an absolutely stellar repository of models starting with nlp but have since expanded to all kinds of tasks audio classification image classification object detection there's sixty thousand pre-trained models for all these tasks there is a specific library of transformers that works with pytorch tensorflow jacks also 7.5 000 data sets that people have uploaded there's also a lot more to it it's worth checking out you can host your model for
+
+22
+00:16:36,720 --> 00:17:31,679
+inference and there's there's community aspects to it so it's a great resource another great resource specifically for vision is called tim state of the art computer vision models can be found on tim just search tim github next up let's talk about distributed training so the scenarios are we have multiple machines represented by little squares here with multiple gpus on each machine and you are sending batches of data to be processed by a model that has parameters right and the data batch can fit on a single gpu or potentially not fit on a single gpu and the model parameters can fit in a single gpu or potentially not fit in a single gpu so let's say the best case the easiest case is your batch of data fits on a single gpu
+
+23
+00:17:30,320 --> 00:18:21,280
+your model parameters fit on a single gpu and that's really called trivial parallelism you can launch independent experiments on other gpus so maybe do a hyper parameter search or potentially you increase your batch size until it can no longer fit on one gpu and then you have to figure something else out and but then yeah what you have to then figure out is okay well my model still fits on a single gpu but my data no longer fits on a single gpu so now i have to go and do something different and what that different thing is usually is data parallelism it lets you distribute a single batch of data across gpus and then average gradients that are computed by the model across all the gpus so it's the same model on each gpu but
+
+24
+00:18:18,880 --> 00:19:10,960
+different batches of data because a lot of this work is cross gpu we have to make sure that the gpus have fast interconnect right so gpu is connected usually through a pci interface to the computer but it and so if there's no other connection then all the data has to flow through the pci bus all the time it's possible that there is a faster interconnect like nv link between the gpus and then the data can leave the pci bus alone and just go straight across the the fast interconnect and the speed up you can expect is if you are using server cards like a100s a6000s you know v100s it's basically a linear speed up for data parallelism which is really cool if you're using consumer cards like 2080s or 3080s we'll talk about it a
+
+25
+00:19:08,720 --> 00:19:59,919
+little further down then unfortunately it's going to be a sublinear speed up so maybe if you have four gpus it'll be like a 3x speed up if you have a gpus maybe a 5x speed up and that's due to the the fact that the consumer cards don't have as fast of an interconnect so data parallelism is implemented in pi torch in the distributed data parallel library there's also a third-party library called horovod and you can use either one super simply using pytorch lightning you basically say what's your strategy if you don't say anything then it's single gpu but if your strategy is ddp then it uses the python distributed data parallel if you use strategy horovod then it uses horivon it seems like the speedup's basically
+
+26
+00:19:58,160 --> 00:20:48,640
+about the same there's no real reason to use horowat over distributed data parallel but it might make it easier for a specific case that you might have so it's good to know about but the first thing to try is just distributed data parallel now we come to a more advanced scenario which is now we can't even fit our model our model is so large it has billions of parameters it doesn't actually fit on a single gpu so we have to spread the model not just the data over multiple gpus and there's three solutions to this so sharded data parallelism starts with the question what exactly is in the gpu memory what is taking up the gpu memory so okay we have the model parameters the floats that make up our actual
+
+27
+00:20:47,360 --> 00:21:40,400
+layers we have the gradients we need to know about the gradients because that's what we average to do our backdrop but we also have optimizer states and that's actually a lot of data for the atom optimizer that's probably the most often used optimizer today it has to be statistics about the gradients basically and in addition if you're doing kind of float 16 training then your model parameters gradients might be float 16 but the optimizer will keep a copy of them as float32 as well so it can be a lot more data and then plus of course you send your batch of data so all of this has to fit on a gpu but does it actually have to fit on every gpu is the question so the baseline that we have is yeah let's send all of this stuff to each gpu
+
+28
+00:21:37,840 --> 00:22:33,440
+and that might take up like 129 gigabytes of data in this in this example this is from the paper called zero optimization storage training trillion parameter models okay so what if we shard the optimizer states sharding is a concept from databases where if you have one source of data you actually break it up into shards of data such that across your distributed system each part of your each node only sees a shard a single shard of the data so here the first thing we can try is we can shard the optimizer states each gpu doesn't have to have all the optimizer state it just has to have its little shard of it we can do the same for gradients and that's called zero two and then pretty crazily we can also do it for the
+
+29
+00:22:31,520 --> 00:23:19,840
+model parameters themselves and that's called zero three and that can result in a pretty insane order of magnitude reduction in memory use which means that your batch size can be 10 times bigger i recommend watching this helpful video that i have linked but you literally pass around the model params between the gpus as computation is proceeding so here we see four gpus four chunks of data entering the gpus and what happened is gpu zero had the model parameters for that first part of the model and it communicated these parameters to the other three gpus and then they did their computation and once they were complete with that computation the other gpus can actually delete the parameters for those first
+
+30
+00:23:18,559 --> 00:24:05,440
+four layers and then gpu one has the parameters for the next four layers and it broadcasts them to the other three gpus who are now able to do the next four layers of computation and that's just in the forward pass then you do the same with gradients and optimizer states in the backward pass this is a lot to implement thankfully we don't have to do it it's implemented by the deep speed library from microsoft and the fair scale library from facebook and recently actually also implemented natively by pytorch so in pytorch it's called fully sharded data parallel instead of zero three and with pytorch lightning you can actually try sharded ddp with just a tiny bit of a change try it see if you see a massive memory
+
+31
+00:24:04,400 --> 00:24:54,880
+reduction that can correspond to a speed up in your training now the same idea the zero three principle right is that the gpu only needs the model frames it needs in the moment for the computation it's doing at this moment the same principle can be applied to just a single gpu you can get a 13 billion parameters onto the gpu and you can train a 13 billion parameter model on a single v100 which doesn't even fit it natively and fair scale also implements this and calls it cpu offloading there's a couple more solutions model parallelism take your model your model let's say has three layers and you have three gpus you can put each layer on a gpu right and in pytorch you can just implement it very trivially but the
+
+32
+00:24:52,960 --> 00:25:41,840
+problem is that only one gpu will be active at a given time so the trick here is that and once again implemented by libraries like deep speed and fair scale they make it better so they pipeline this kind of computation so that gpus are mostly fully utilized although you need to tune the amount of pipelining on the batch size and exactly how you're going to split up the model into the gpus so this isn't as much of fire and forget solution like like sharded data parallel and another solution is tensor parallelism which basically is observing that there's nothing special about a matrix multiplication that requires the whole matrix to be on one gpu you can distribute the matrix over gpus so megatron lm is a repository from
+
+33
+00:25:39,279 --> 00:26:34,960
+nvidia which did this for the transformer model and is widely used so you can actually use all of these if you really need to scale and the model that really needs to scale is a gpt3 three-sized language model such as bloom which recently finished training so they used zero data parallelism tensor parallelism pipeline parallelism in addition to some other stuff and they called it 3d parallelism but they also write that since they started their endeavor the the zero stage three performance has dramatically improved and if they were to start over again today maybe they would just do sharded data parallel and that would just be enough so in conclusion you know if your model and data fits on one gpu that's awesome
+
+34
+00:26:32,799 --> 00:27:21,919
+if it doesn't or you want to speed up training then you can distribute over gpus with distributed data parallel if the model still doesn't fit you should try zero three or fully shared data parallel there's other ways to speed up there's 16 bit training there's maybe some special you know fast kernels for different types of layers like transformers you can maybe try sparse attention instead of normal dense attention so there's other things that these libraries like deep speed and fair skill implement that you can try and there's even more tricks that you could try for example for nlp there's this position encoding step you can use something called alibi which scales to basically all length of sequences
+
+35
+00:27:20,480 --> 00:28:09,600
+so you can actually train on shorter sequences and use this trick called sequence length warm up where you train on shorter sequences and then you increase the size and because you're using alibi it should not mess up your position and then for vision you can also use a size warm up by progressively increasing the size of the image you can use special optimizers and these tricks are implemented by a library called mosaic ml composer and they report some pretty cool speed ups and it's pretty easy to implement and they also have a cool web tool i'm a fan of these things that basically lets you see the efficient frontier for training models time versus cost kind of fun to play around with this mosaic ml explorer
+
+36
+00:28:08,000 --> 00:29:01,039
+there's also some research libraries like ffcv which actually try to optimize the data flow there are some simple tricks you can maybe do that speed it up a lot these things will probably find their way into mainstream pie torch eventually but it's worth giving this a try especially if again you're training on vision models the next thing we're going to talk about is compute that we need for deep learning i'm sure you've seen plots like this from open ai this is up through 2019 showing on a log scale just how many times the compute needs for the top performing models have grown and this goes even further into 2022 with the large language models like gpt3 they're just incredibly large and required an incredible amount of
+
+37
+00:28:58,720 --> 00:29:53,279
+pedoflops to train so basically nvidia is the only choice for deep learning gpus and recently google tpus have been made available in the gcp cloud and they're also very nice and the three main factors that we need to think about when it comes to gpus are how much data can you transfer to the gpu then how fast can you crunch through that data and that actually depends on whether the data is 32-bit or 16-bit and then how fast can you communicate between the cpu and the gpu and between gpus we can look at some landmark nvidia gpus so the first thing we might notice is that there's a basically a new architecture every year every couple of years it went from kepler with the k80 and k40 cards in 2014
+
+38
+00:29:51,520 --> 00:30:44,480
+up through ampere from 2020 on some cards are for server use some cards are for consumer use if you're doing stuff for business you're only supposed to use the server cards the ram that the gpu has allows you to fit a large model and a meaningful batch of data on the gpu so the more ram the better these are this is like kind of how much data can you crunch through in a unit time and there's also i have a column for tensor t flops are special tensor cores that nvidia specifically intends for deep learning operations which are mixed precision float32 and float16 these are much higher than just straight 32-bit teraflops if you use 16-bit training you effectively double or so your rain capacity we looked at the teraflops these are
+
+39
+00:30:42,720 --> 00:31:46,960
+theoretical numbers but how do they actually benchmark lame the labs is probably the best source of benchmark data and here they show relative to the v100 single gpu how do the different gpus compare so one thing we might notice is the a100 which is the most recent gpu that's the server grade is over 2.5 faster than v100 you'll notice there's a couple of different a100s the pcie versus sxm4 refers to how fast you can get the data onto the gpu and the 40 gig versus 80 gig refers to how much data can fit on the gpu also recently there's rtx a 4000 5000 6000 and so on cards and the a40 and these are all better than the v100 another source of benchmarks is aime they show you time for resnet50 model to go through
+
+40
+00:31:44,240 --> 00:32:42,720
+1.4 images in imagenet the configuration of four a100s versus four v100s is three times faster in in flow 32 and only one and a half times faster in float 16. there's a lot more stuff you can notice but that's what i wanted to highlight and we could buy some of these gpus we could also use them in the cloud so amazon web services google cloud platform microsoft azure are all the heavyweight cloud providers google cloud platform out of the three is special because it also has tpus and the startup cloud providers are lame the labs paper space core weave data crunch jarvis and others so briefly about tpus so there's four versions of them four generations the tpu v4 are the most recent ones and they're
+
+41
+00:32:40,480 --> 00:33:36,960
+just the fastest possible accelerator for deep learning this graphic shows speed ups over a100 which is the fastest nvidia accelerator but the v4s are not quite in general availability yet the v3s are still super fast and they excel at scaling so if you use if you have to train such a large model that you use multiple nodes multiple and all the cores in the tpu then this can be quite fast each tpu has 128 gigs of ram so there's a lot of different clouds and it's a little bit overwhelming to actually compare prices so we built a tool for cloud comparison cloud gpu comparison so we have aws gcp azure lambda labs paper space jarvis labs data crunch and we solicit pull requests so if you know another one like core weave
+
+42
+00:33:35,519 --> 00:34:34,639
+make a pull request to this csv file and then what you can do is you can filter so for example i want to see only the latest generation gpus i want to see only four or eight gpu machines and then maybe particularly i actually don't even want to see the i want to see only the a100s so let's only select the a100s so that narrows it down right so if we want to use that that narrows it down and furthermore maybe i only want to use the 80 gig versions so that narrows it down further and then we can sort by per gpu price or the total price and we can see the properties of the machines right so we know the gpu ram but how many virtual cpus and how much machine ram do these different providers supply to us now let's combine this cost
+
+43
+00:34:33,679 --> 00:35:33,119
+data with benchmark data and what we find is that something that's expensive per hour is not necessarily expensive per experiment using lambda labs benchmarking data if you use the forex v100 machine which is the cheapest per hour and you run an experiment using a transformers model that takes 72 hours it'll cost 1750 to run but if you use the 8x a100 machine it will only take eight hours to run and it'll actually only cost 250 and there's a similar story if you use confnet instead of transformer models less dramatic but still we find that the 8 by a100 machine is both the fastest and the cheapest so that's a little counter-intuitive so i was looking for more benchmarks so here is mosaic ml which i mentioned
+
+44
+00:35:30,960 --> 00:36:30,000
+earlier they're benchmarking the resnet 50 and this is on aws what they find is the 8x a100 machine is one and a half times faster and 15 cheaper than 8x v100 so this is a confident experiment and here's a transformer experiment ept2 model so the 8x a100 machine is twice as fast and 25 cheaper than the adax v100 machine and it's actually three times faster and 30 cheaper than the 8x t4 machine which is a touring generation gpu a good heuristic is use the most expensive per hour gpu which is probably going to be a 4x or 8x a100 in the least expensive cloud and from playing with that cloud gpu table you can convince yourself that the startups are much cheaper than the big boys so here i'm filtering by a100
+
+45
+00:36:26,960 --> 00:37:22,560
+and the per gpu cost on lambda labs is only one dollar and 10 cents per hour and on gcp azure and and aws it's at least you know 3.67 cents but what if you don't want to use the cloud there's two options you could build your own which is i would say easy or you can buy pre-built which is definitely even easier lambda labs builds them and nvidia builds them and then just pc builders like super micro and stuff like that build them you can build a pretty quiet pc with with a lot of ram and let's say you know two 390s or 2080 ti's or something that would maybe be five to eight thousand dollars it take you a day to build it and set it up maybe it's a rite of passage for deep learning practitioners
+
+46
+00:37:20,480 --> 00:38:15,680
+now if you want to go beyond four or 2000 series like 20 80s or two 3000 series like 30 90s that can be painful just because there's a lot of power that they consume and they get hot so pre-built can be better here's a 12 000 machine with two a5000s which each have 24 gigs ram it's going to be incredibly fast or maybe you want 8 gpus now this one is going to be loud you're going to have to put it in some kind of special facility like a colo and actually lame the labs can can stored in their colo for you it'd be maybe sixty thousand dollars for eight a six thousands which is a really really fast server lame the labs also provides actionable advice for selecting specific gpus there is a well known article from tim detmers
+
+47
+00:38:13,119 --> 00:38:58,320
+that is now slightly out of date because there's no ampere cards but it's still good he talks about more than just gpus but also about what cpu to get the ram the recommendations that that i want to give is i think it's it's useful to have your own gpu machine just to shift your mindset from minimizing cost of running in the cloud to maximizing utility of having something that you already paid for and just maximizing how much use you get out of it but to scale out experiments you probably need to enter the cloud and you should use the most expensive machines in the least expensive cloud tpus are worth experimenting with if you're doing large scale training lameda labs is a sponsor of the full-stack deep
+
+48
+00:38:56,800 --> 00:39:45,520
+learning projects that our students are doing this year it's actually an excellent choice for both buying a machine for yourself and it's the least expensive cloud for a100s now that we've talked about compute we can talk about how to manage it so we want to do is we want to launch an experiment or a set of experiments each experiment is going to need a machinery machines with gpu or gpus in the machine it's going to need some kind of setup like a python version cuda version nvidia drivers python requirements like a specific version of pytorch and then it needs a source of data so we could do this manually we could use a workload manager like slurm we could use docker and kubernetes or we could use some software specialized
+
+49
+00:39:43,920 --> 00:40:35,520
+for machine learning if you follow best practices for specifying dependencies like content pip tools that we covered earlier then all you have to do is log into the machine launch an experiment right activate your environment launch the experiment say how many gpus it needs if you however have a cluster of machines then you need to do some more advanced which is probably going to be slurm which is an old-school solution to workload management that's still that's still widely used this is actually a job from the big science effort to train the gpt3 size language model so they have 24 nodes with 64 cpus and 8 gpus on each node slurm is the way that they launched it on their cluster docker is a way to package up an entire
+
+50
+00:40:34,079 --> 00:41:22,560
+dependency stack in in something that's lighter than a full-on virtual machine nvidia docker is also something you'll have to install which let's use gpus and we'll actually use this in lab so we'll talk more about it later kubernetes has kind of emerged as as the best way the most popular way to run many docker containers on top of a cluster cube flow specifically is a project for machine learning both of these are google originated open source projects but they're not controlled by google anymore so with kubeflow you can spawn and manage jupiter notebooks you can manage multi-step workflows it interfaces with pytorch and tensorflow and you can run it on top of google cloud platform or aws or azure or on your own
+
+51
+00:41:20,800 --> 00:42:10,400
+cluster and it can be useful but it's a lot so it could be the right choice for you but we think it probably won't be slarm and kubeflow they make sense if you already have a cluster up and running but how do you even get a cluster up and running in the first place and before we proceed i try not to mention software as a service that doesn't show pricing i find that you know when you go to the website and it says call us or whatever contact us for a demo that's not the right fit for the fsdl community we like to use open source ideally but if it's not open source then at least something that's transparently priced aws sagemaker is a solution you've probably heard about if you've used amazon web services
+
+52
+00:42:08,160 --> 00:42:57,040
+and it's really a set of solutions it's everything from labeling data to launching notebooks to training to deploying your models and even to monitoring them and notebooks are a central paradigm they call it sagemaker studio and sagemaker could totally make sense to adopt if you're already using aws for everything if you're not already using aws for everything it's not such a silver bullet that it's worth adopting necessarily but if you are it's definitely worth a look so for training specifically they have some basically pre-built algorithms and they're quite they're quite old-school but you can also connect any other algorithm yourself it's a little more it's a little more complicated and right away you have to configure a
+
+53
+00:42:55,119 --> 00:43:48,880
+lot of i am you know roles and and security groups and stuff like that it might be overwhelming if all you're trying to do is train a machine learning model that said they do have increasing support for pytorch now notice if you're using sagemaker to launch your python training you're going to be paying about a 15 to 20 markup so there's special sagemaker instances that correspond to normal aws gpu instances but it's more expensive they do have support for using spot instances and so that could make it worth it any scale is a company from the makers of ray which is an open source project from berkeley and recently they released ray train which they claim is faster than sagemaker so the same idea basically lets you
+
+54
+00:43:46,800 --> 00:44:40,560
+scale out your training to many nodes with many gpus but does it faster and it has better spot instance support where if a spot instance gets killed during training it recovers from it intelligently and any scale any scale a software is a service that makes it you know really simple to provision compute with one line of code you can launch a cluster of any size that ease of use comes at a significant markup to amazon web services grid ai is makers of py torch lightning and the the tagline is seamlessly trained hundreds of machine learning models on the cloud with zero code changes if you have some kind of main dot pi method that's going to run your training and that can run on your laptop or on on
+
+55
+00:44:38,400 --> 00:45:27,760
+some local machine you can just scale it out to a grid of instances by prefacing it with grid run and then just saying what kind of instance type how many gpus should i use spot instances and so on and you can also you can use their instances or you can use aws under the hood and then it shows you all the experiments you're running and so on now i'm not totally sure about the long term plans for grid.ai because the makers of python's lightning are also rebranding as lightning.i which has its own pricing so i'm i'm just not totally sure but it's if it sticks around it looks like a really cool solution there's also non-machine learning specific solutions like you don't need sagemaker to provision compute on aws
+
+56
+00:45:25,440 --> 00:46:11,200
+you could just do it in a number of ways that people have been doing you know provisioning aws instances and then uniting them into a cluster you can write your own scripts you can use kubernetes you can use some libraries for spot instances but there's nothing you know we can really recommend that's super easy to use determined ai is a machine learning specific open source solution that lets you manage a cluster either on prem or in the cloud it's cluster management distributed training experiment tracking hyper parameter search a lot of extra stuff it was a startup also from berkeley it got acquired by hp but it's still an active development it's really easy to use you just install determined get a
+
+57
+00:46:09,839 --> 00:46:59,920
+cluster up and running you can also launch it on aws or gcp that said i feel like a truly simple solution to launching training on many cloud instances still doesn't exist so this is an area where i think there's room for a better solution and that cannot be said about experiment management and model management because i think there's great solutions there so what experiment management refers to is you know as we run machine learning experiments we we can lose track of which code parameters data set generated which model when we run multiple experiments that's even more difficult we need to like start making a spreadsheet of all the experiments we ran and the results and so on tensorboard is a solution from google
+
+58
+00:46:58,079 --> 00:47:48,640
+that's not exclusive to tensorflow it gives you this nice set of pages that lets you track your loss and see where your model saved and it's a great solution for single experiments it does get unwieldy to manage many experiments as you get into dozens of experiments ml flow tracking is a solution that is open source it's from data bricks but it's not exclusive to data breaks it's not only for experiment management it's also for model packaging and stuff like that but they do have a robust solution for experiment management you do have to host it yourself weights and biases is a really popular super easy to use solution that is free for public projects and paid for private projects they show you
+
+59
+00:47:46,240 --> 00:48:33,119
+all the experiments you've ever run slice them dice however you want for each experiment they record would you log like your loss but also stuff about your system like how utilized your gpu is which is pretty important to track and you basically just initialize it with your experiment config and then you log anything you want including images and we're actually going to see this in lab 4 which is this week they also have some other stuff like you can host reports and tables is a recent product that lets you slice and dice your data and predictions in really cool ways determine.ai also has an experiment tracking solution which is also perfectly good and there's other solutions too like neptune and comet
+
+60
+00:48:30,160 --> 00:49:22,240
+and a number of others really often we actually want to programmatically launch experiments by doing something that's called hyper parameter optimization so maybe we want to search over learning rates so as we launch our training we don't want to commit to a specific learning rate we basically want to search over learning rates from you know point zero zero zero one to point one it'd be even more awesome if like this was done intelligently where if multiple runs are proceeding in in parallel the ones that aren't going as well as others get stopped early and we get to search over more of the potential hyperparameter space weights and biases has a solution to this that's very pragmatic and easy to
+
+61
+00:49:19,599 --> 00:50:10,720
+use it's called sweeps the way this works is you basically add a yaml file to your project that specifies the parameters you want to search over and how you want to do the search so here on the right you'll see we're using this hyperband algorithm which is a state-of-the-art hyper-parameter optimization algorithm and then you launch agent on whatever machines you control the agent will pull the sweep server for a set of parameters run an experiment report results poll the server for more parameters and keep doing that and there's other solutions this is pretty table stakes kind of thing so sagemaker has hyperparameter search determined ai has hyperparameter search i think of it as just it's a part of your training harness so
+
+62
+00:50:09,680 --> 00:51:02,480
+if you're already using weights and biases just use sweeps from weights and biases if you're already using determine just use hyperparameter search from determined it's not worth using some specialized software for this and lastly there are all-in-one solutions that cover everything from data to development to deployment a single system for everything for development usually a notebook interface scaling a training experiment to many machines provisioning the compute for you tracking experiments versioning models but also deploying models and monitoring performance managing data of really all-in-one each maker is the you know the prototypical solution here but there's some other ones like gradients from paper space so look at
+
+63
+00:51:00,960 --> 00:51:48,240
+look at these features notebooks experiments data sets models and inference or domino data labs you can provision compute you can track the experiments you can deploy a model via a rest api you can monitor the predictions that the the api makes and you can publish little data applets kind of like streamlit you can also monitor spend and you see all the projects in one place domino's meant more for kind of non-deep learning machine learning but i just wanted to show it because it's a nice set of the all-in-one functionality so these all-in-one solutions could be good but before deciding we want to go in on one of them let's wait to learn more about data management and deployment in the weeks ahead
+
+64
+00:51:46,480 --> 00:51:52,960
+and that is it for development infrastructure and tooling thank you
+

documents/lecture-03.md

@@ -0,0 +1,597 @@
+---
+description: Principles for testing software, tools for testing Python code, practices for debugging models and testing ML
+---
+
+# Lecture 3: Troubleshooting & Testing
+
+<div align="center">
+<iframe width="720" height="405" src="https://www.youtube-nocookie.com/embed/RLemHNAO5Lw?list=PL1T8fO7ArWleMMI8KPJ_5D5XSlovTW_Ur" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+Lecture by [Charles Frye](https://twitter.com/charles_irl).<br />
+Notes by [James Le](https://twitter.com/le_james94) and [Vishnu Rachakonda](https://www.linkedin.com/in/vrachakonda/).<br />
+Published August 22, 2022.
+[Download slides](https://fsdl.me/2022-lecture-03-slides).
+
+## 1 - Testing Software
+
+1.  The general approach is that tests will help us ship faster with
+fewer bugs, but they won't catch all of our bugs.
+
+2.  That means we will use testing tools but won't try to achieve 100%
+coverage.
+
+3.  Similarly, we will use linting tools to improve the development
+experience but leave escape valves rather than pedantically
+following our style guides.
+
+4.  Finally, we'll discuss tools for automating these workflows.
+
+### 1.1 - Tests Help Us Ship Faster. They Don't Catch All Bugs
+
+![](./media/image1.png)
+
+**Tests are code we write that are designed to fail intelligibly when
+our other code has bugs**. These tests can help catch some bugs before
+they are merged into the main product, but they can't catch all bugs.
+The main reason is that test suites are not certificates of correctness.
+In some formal systems, tests can be proof of code correctness. But we
+are writing in Python (a loosely goosey language), so all bets are off
+in terms of code correctness.
+
+[Nelson Elhage](https://twitter.com/nelhage?lang=en)
+framed test suites more like classifiers. The classification problem is:
+does this commit have a bug, or is it okay? The classifier output is
+whether the tests pass or fail. We can then **treat test suites as a
+"prediction" of whether there is a bug**, which suggests a different way
+of designing our test suites.
+
+When designing classifiers, we need to trade off detection and false
+alarms. **If we try to catch all possible bugs, we can inadvertently
+introduce false alarms**. The classic signature of a false alarm is a
+failed test - followed by a commit that fixes the test rather than the
+code.
+
+To avoid introducing too many false alarms, it's useful to ask yourself
+two questions before adding a test:
+
+1.  Which real bugs will this test catch?
+
+2.  Which false alarms will this test raise?
+
+If you can think of more examples for the second question than the first
+one, maybe you should reconsider whether you need this test.
+
+One caveat is that: **in some settings, correctness is important**.
+Examples include medical diagnostics/intervention, self-driving
+vehicles, and banking/finance. A pattern immediately arises here: If you
+are operating in a high-stakes situation where errors have consequences
+for people's lives and livelihoods, even if it's not regulated yet, it
+might be regulated soon. These are examples of **low-feasibility,
+high-impact ML projects** discussed in the first lecture.
+
+![](./media/image19.png)
+
+
+### 1.2 - Use Testing Tools, But Don't Chase Coverage
+
+-   *[Pytest](https://docs.pytest.org/)* is the standard
+tool for testing Python code. It has a Pythonic implementation and
+powerful features such as creating separate suites, sharing
+resources across tests, and running parametrized variations of
+tests.
+
+-   Pure text docs can't be checked for correctness automatically, so
+they are hard to maintain or trust. Python has a nice module,
+[*[doctests]*](https://docs.python.org/3/library/doctest.html),
+for checking code in the documentation and preventing rot.
+
+-   Notebooks help connect rich media (charts, images, and web pages)
+with code execution. A cheap and dirty solution to test notebooks
+is adding some *asserts* and using *nbformat* to run the
+notebooks.
+
+![](./media/image17.png)
+
+
+Once you start adding different types of tests and your codebase grows,
+you will want coverage tools for recording which code is checked or
+"covered" by tests. Typically, this is done in lines of code, but some
+tools can be more fine-grained. We recommend
+[Codecov](https://about.codecov.io/), which generates nice
+visualizations you can use to drill down and get a high-level overview
+of the current state of your testing. Codecov helps you understand your
+tests and can be incorporated into your testing. You can say you want to
+reject commits not only where tests fail, but also where test coverage
+goes down below a certain threshold.
+
+However, we recommend against that. Personal experience, interviews, and
+published research suggest that only a small fraction of the tests you
+write will generate most of your value. **The right tactic,
+engineering-wise, is to expand the limited engineering effort we have on
+the highest-impact tests and ensure that those are super high quality**.
+If you set a coverage target, you will instead write tests in order to
+meet that coverage target (regardless of their quality). You end up
+spending more effort to write tests and deal with their low quality.
+
+![](./media/image16.png)
+
+
+### 1.3 - Use Linting Tools, But Leave Escape Valves
+
+**Clean code is of uniform and standard style**.
+
+1.  Uniform style helps avoid spending engineering time on arguments
+over style in pull requests and code review. It also helps improve
+the utility of our version control by cutting down on noisy
+components of diffs and reducing their size. Both benefits make it
+easier for humans to visually parse the diffs in our version
+control system and make it easier to build automation around them.
+
+2.  Standard style makes it easier to accept contributions for an
+open-source repository and onboard new team members for a
+closed-source system.
+
+![](./media/image18.png)
+
+
+One aspect of consistent style is consistent code formatting (with
+things like whitespace). The standard tool for that in Python is
+[the] *[black]* [Python
+formatter](https://github.com/psf/black). It's a very
+opinionated tool with a fairly narrow scope in terms of style. It
+focuses on things that can be fully automated and can be nicely
+integrated into your editor and automated workflows.
+
+For non-automatable aspects of style (like missing docstrings), we
+recommend [*[flake8]*](https://flake8.pycqa.org/). It comes
+with many extensions and plugins such as docstring completeness, type
+hinting, security, and common bugs.
+
+ML codebases often have both Python code and shell scripts in them.
+Shell scripts are powerful, but they also have a lot of sharp edges.
+*[shellcheck](https://www.shellcheck.net/)* knows all the
+weird behaviors of bash that often cause errors and issues that aren't
+immediately obvious. It also provides explanations for why it's raising
+a warning or an error. It's very fast to run and can be easily
+incorporated into your editor.
+
+![](./media/image6.png)
+
+
+One caveat to this is: **pedantic enforcement of style is obnoxious.**
+To avoid frustration with code style and linting, we recommend:
+
+1.  Filtering rules down to the minimal style that achieves the goals we
+set out (sticking with standards, avoiding arguments, keeping
+version control history clean, etc.)
+
+2.  Having an "opt-in" application of rules and gradually growing
+coverage over time - which is especially important for existing
+codebases (which may have thousands of lines of code that we need
+to be fixed).
+
+### 1.4 - Always Be Automating
+
+**To make the best use of testing and linting practices, you want to
+automate these tasks and connect to your cloud version control system
+(VCS)**. Connecting to the VCS state reduces friction when trying to
+reproduce or understand errors. Furthermore, running things outside of
+developer environments means that you can run tests automatically in
+parallel to other development work.
+
+Popular, open-source repositories are the best place to learn about
+automation best practices. For instance, the PyTorch Github library has
+tons of automated workflows built into the repo - such as workflows that
+automatically run on every push and pull.
+
+![](./media/image15.png)
+
+
+The tool that PyTorch uses (and that we recommend) is [GitHub
+Actions](https://docs.github.com/en/actions), which ties
+automation directly to VCS. It is powerful, flexible, performant, and
+easy to use. It gets great documentation, can be used with a YAML file,
+and is embraced by the open-source community. There are other options
+such as [pre-commit.ci](https://pre-commit.ci/),
+[CircleCI](https://circleci.com/), and
+[Jenkins](https://www.jenkins.io/); but GitHub Actions
+seems to have won the hearts and minds in the open-source community in
+the last few years.
+
+To keep your version control history as clean as possible, you want to
+be able to run tests and linters locally before committing. We recommend
+*[pre-commit](https://github.com/pre-commit/pre-commit)*
+to enforce hygiene checks. You can use it to run formatting, linting,
+etc. on every commit and keep the total runtime to a few seconds.
+*pre-commit* is easy to run locally and easy to automate with GitHub
+Actions.
+
+**Automation to ensure the quality and integrity of our software is a
+productivity enhancer.** That's broader than just CI/CD. Automation
+helps you avoid context switching, surfaces issues early, is a force
+multiplier for small teams, and is better documented by default.
+
+One caveat is that: **automation requires really knowing your tools.**
+Knowing Docker well enough to use it is not the same as knowing Docker
+well enough to automate it. Bad automation, like bad tests, takes more
+time than it saves. Organizationally, that makes automation a good task
+for senior engineers who have knowledge of these tools, have ownership
+over code, and can make these decisions around automation.
+
+### Summary
+
+1.  Automate tasks with GitHub Actions to reduce friction.
+
+2.  Use the standard Python toolkit for testing and cleaning your
+projects.
+
+3.  Choose testing and linting practices with the 80/20 principle,
+shipping velocity, and usability/developer experience in mind.
+
+## 2 - Testing ML Systems
+
+1.  Testing ML is hard, but not impossible.
+
+2.  We should stick with the low-hanging fruit to start.
+
+3.  Test your code in production, but don't release bad code.
+
+### 2.1 - Testing ML Is Hard, But Not Impossible
+
+Software engineering is where many testing practices have been
+developed. In software engineering, we compile source code into
+programs. In machine learning, training compiles data into a model.
+These components are harder to test:
+
+1.  Data is heavier and more inscrutable than source code.
+
+2.  Training is more complex and less well-defined.
+
+3.  Models have worse tools for debugging and inspection than compiled
+programs.
+
+In this section, we will focus primarily on "smoke" tests. These tests
+are easy to implement and still effective. They are among the 20% of
+tests that get us 80% of the value.
+
+### 2.2 - Use Expectation Testing on Data
+
+**We test our data by checking basic properties**. We express our
+expectations about the data, which might be things like there are no
+nulls in this column or the completion date is after the start date.
+With expectation testing, you will start small with only a few
+properties and grow them slowly. You only want to test things that are
+worth raising alarms and sending notifications to others.
+
+![](./media/image14.png)
+
+
+We recommend
+[*[great_expectations]*](https://greatexpectations.io/) for
+data testing. It automatically generates documentation and quality
+reports for your data, in addition to built-in logging and alerting
+designed for expectation testing. To get started, check out [this
+MadeWithML tutorial on
+great_expectations](https://github.com/GokuMohandas/testing-ml).
+
+![](./media/image13.png)
+
+To move forward, you want to stay as close to the data as possible:
+
+1.  A common pattern is that there's a benchmark dataset with
+annotations (in academia) or an external annotation team (in the
+industry). A lot of the detailed information about that data can
+be extracted by simply looking at it.
+
+2.  One way for data to get internalized into the organization is that
+at the start of the project, model developers annotate data ad-hoc
+(especially if you don't have the budget for an external
+annotation team).
+
+3.  However, if the model developers at the start of the project move on
+and more developers get onboarded, that knowledge is diluted. A
+better solution is an internal annotation team that has a regular
+information flow with the model developers is a better solution.
+
+4.  The best practice ([recommended by Shreya
+Shankar](https://twitter.com/sh_reya/status/1521903046392877056))
+is t**o have a regular on-call rotation where model developers
+annotate data themselves**. Ideally, these are fresh data so that
+all members of the team who are developing models know about the
+data and build intuition/expertise in the data.
+
+### 2.3 - Use Memorization Testing on Training
+
+**Memorization is the simplest form of learning**. Deep neural networks
+are very good at memorizing data, so checking whether your model can
+memorize a very small fraction of the full data set is a great smoke
+test for training. If a model can\'t memorize, then something is clearly
+very wrong!
+
+Only really gross issues with training will show up with this test. For
+example, your gradients may not be calculated correctly, you have a
+numerical issue, or your labels have been shuffled; serious issues like
+these. Subtle bugs in your model or your data are not going to show up.
+A way to catch smaller bugs is to include the length of run time in your
+test coverage. It's a good way to detect if smaller issues are making it
+harder for your model to learn. If the number of epochs it takes to
+reach an expected performance suddenly goes up, it may be due to a
+training bug. PyTorch Lightning has an "*overfit_batches*" feature that
+can help with this.
+
+**Make sure to tune memorization tests to run quickly, so you can
+regularly run them**. If they are under 10 minutes or some short
+threshold, they can be run every PR or code change to better catch
+breaking changes. A couple of ideas for speeding up these tests are
+below:
+
+![](./media/image3.png)
+
+Overall, these ideas lead to memorization tests that implement model
+training on different time scale and allow you to mock out scenarios.
+
+A solid, if expensive idea for testing training is to **rerun old
+training jobs with new code**. It's not something that can be run
+frequently, but doing so can yield lessons about what unexpected changes
+might have happened in your training pipeline. The main drawback is the
+potential expense of running these tests. CI platforms like
+[CircleCI](https://circleci.com/) charge a great deal for
+GPUs, while others like Github Actions don't offer access to the
+relevant machines easily.
+
+The best option for testing training is to **regularly run training with
+new data that's coming in from production**. This is still expensive,
+but it is directly related to improvements in model development, not
+just testing for breakages. Setting this up requires **a data flywheel**
+similar to what we talked about in Lecture 1. Further tooling needed to
+achieve will be discussed down the line.
+
+### 2.4 - Adapt Regression Testing for Models
+
+**Models are effectively functions**. They have inputs and produce
+outputs like any other function in code. So, why not test them like
+functions with regression testing? For specific inputs, we can check to
+see whether the model consistently returns the same outputs. This is
+best done with simpler models like classification models. It's harder to
+maintain such tests with more complex models. However, even in a more
+complex model scenario, regression testing can be useful for comparing
+changes from training to production.
+
+![](./media/image11.png)
+
+
+A more sophisticated approach to testing for ML models is to **use loss
+values and model metrics to build documented test suites out of your
+data**. Consider this similar to [the test-driven
+development](https://en.wikipedia.org/wiki/Test-driven_development)
+(TDD) code writing paradigm. The test that is written before your code
+in TDD is akin to your model's loss performance; both represent the gap
+between where your code needs to be and where it is. Over time, as we
+improve the loss metric, our model is getting closer to passing "the
+test" we've imposed on it. The gradient descent we use to improve the
+model can be considered a TDD approach to machine learning models!
+
+![](./media/image9.png)
+
+
+While gradient descent is somewhat like TDD, it's not *exactly* the same
+because simply reviewing metrics doesn't tell us how to resolve model
+failures (the way traditional software tests do).
+
+To fill in this gap, **start by [looking at the data points that have
+the highest loss](https://arxiv.org/abs/1912.05283)**. Flag
+them for a test suite composed of "hard" examples. Doing this provides
+two advantages: it helps find where the model can be improved, and it
+can also help find errors in the data itself (i.e. poor labels).
+
+As you examine these failures, you can aggregate types of failures into
+named suites. For example in a self-driving car use case, you could have
+a "night time" suite and a "reflection" suite. **Building these test
+suites can be considered the machine learning version of regression
+testing**, where you take bugs that you\'ve observed in production and
+add them to your test suite to make sure that they don\'t come up again.
+
+![](./media/image8.png)
+
+The method can be quite manual, but there are some options for speeding
+it up. Partnering with the annotation team at your company can help make
+developing these tests a lot faster. Another approach is to use a method
+called [Domino](https://arxiv.org/abs/2203.14960) that
+uses foundation models to find errors. Additionally, for testing NLP
+models, use the
+[CheckList](https://arxiv.org/abs/2005.04118) approach.
+
+### 2.5 - Test in Production, But Don't YOLO
+
+It's crucial to test in true production settings. This is especially
+true for machine learning models, because data is an important component
+of both the production and the development environments. It's difficult
+to ensure that both are very close to one another.
+
+**The best way to solve the training and production difference is to
+test in production**.
+
+Testing in production isn't sufficient on its own. Rather, testing in
+production allows us to develop tooling and infrastructure that allows
+us to resolve production errors quickly (which are often quite
+expensive). It reduces pressure on other kinds of testing, but does not
+replace them.
+
+![](./media/image7.png)
+
+
+We will cover in detail the tooling needed for production monitoring and
+continual learning of ML systems in a future lecture.
+
+### 2.6 - ML Test Score
+
+So far, we have discussed writing "smoke" tests for ML: expectation
+tests for data, memorization tests for training, and regression tests
+for models.
+
+**As your code base and team mature, adopt a more full-fledged approach
+to testing ML systems like the approach identified in the [ML Test
+Score](https://research.google/pubs/pub46555/) paper**. The
+ML Test Score is a rubric that evolved out of machine learning efforts
+at Google. It's a strict rubric for ML test quality that covers data,
+models, training, infrastructure, and production monitoring. It overlaps
+with, but goes beyond some of the recommendations we've offered.
+
+![](./media/image2.png)
+
+It's rather expensive, but worth it for high stakes use cases that need
+to be really well-engineered! To be really clear, this rubric is
+*really* strict. Even our Text Recognizer system we've designed so far
+misses a few categories. Use the ML Test Score as inspiration to develop
+the right testing approach that works for your team's resources and
+needs.
+
+![](./media/image5.png)
+
+## 3 - Troubleshooting Models
+
+**Tests help us figure out something is wrong, but troubleshooting is
+required to actually fix broken ML systems**. Models often require the
+most troubleshooting, and in this section we'll cover a three step
+approach to troubleshooting them.
+
+1.  "Make it run" by avoiding common errors.
+
+2.  "Make it fast" by profiling and removing bottlenecks.
+
+3.  "Make it right" by scaling model/data and sticking with proven
+architectures.
+
+### 3.1 - Make It Run
+
+This is the easiest step for models; only a small portion of bugs cause
+the kind of loud failures that prevent a model from running at all.
+Watch out for these bugs in advance and save yourself the trouble of
+models that don't run.
+
+The first type of bugs that prevent models from running at all are
+**shape errors.** When the shape of the tensors don't match for the
+operations run on them, models can't be trained or run. Prevent these
+errors by keeping notes on the expected size of tensors, annotate the
+sizes in the code, and even step through your model code with a debugger
+to check tensor size as you go.
+
+![](./media/image10.png)
+
+
+The second type of bugs is out of **memory errors**. This occurs when
+you try to push a tensor to a GPU that is too large to fit. PyTorch
+Lightning has good tools to prevent this. Make sure you're using the
+lowest precision your training can tolerate; a good default is 16 bit
+precision. Another common reason for this is trying to run a model on
+too much data or too large a batch size. Use the autoscale batch size
+feature in PyTorch Lightning to pick the right size batch. You can use
+gradient accumulation if these batch sizes get too small. If neither of
+these options work, you can look into manual techniques like tensor
+parallelism and gradient checkpoints.
+
+**Numerical errors** also cause machine learning failures. This is when
+NaNs or infinite values show up in tensors. These issues most commonly
+appear first in the gradient and then cascade through the model. PyTorch
+Lightning has a good tool for tracking and logging gradient norms. A
+good tip to check whether these issues are caused by precision issues is
+to switch to Python 64 bit floats and see if that causes these issues to
+go away. Normalization layers tend to cause these issues, generally
+speaking. So watch out for how you do normalization!
+
+### 3.2 - Make It Fast
+
+![](./media/image4.png)
+
+Once you can run a model, you'll want it to run fast. This can be tricky
+because the performance of DNN training code is very counterintuitive.
+For example, transformers can actually spend more time in the MLP layer
+than the attention layer. Similarly, trivial components like loading
+data can soak up performance.
+
+To solve these issues, the primary solution is to **roll up your sleeves
+and profile your code**. You can often find pretty easy Python changes
+that yield big results. Read these two tutorials by
+[Charles](https://wandb.ai/wandb/trace/reports/A-Public-Dissection-of-a-PyTorch-Training-Step--Vmlldzo5MDE3NjU?galleryTag=&utm_source=fully_connected&utm_medium=blog&utm_campaign=using+the+pytorch+profiler+with+w%26b)
+and [Horace](https://horace.io/brrr_intro.html) for more
+details.
+
+### 3.3 - Make It Right
+
+After you make it run fast, make the model right. Unlike traditional
+software, machine learning models never are truly perfect. Production
+performance is never perfect. As such, it might be more appropriate to
+say "make it as right as needed".
+
+Knowing this, making the model run and run fast allows us to make the
+model right through applying **scale.** To achieve performance benefits,
+scaling a model or its data are generally fruitful and achievable
+routes. It's a lot easier to scale a fast model. [Research from OpenAI
+and other institutions](https://arxiv.org/abs/2001.08361)
+is showing that benefits from scale can be rigorously measured and
+predicted across compute budget, dataset size, and parameter count.
+
+![](./media/image12.png)
+
+If you can't afford to scale yourself, consider finetuning a model
+trained at scale for your task.
+
+So far, all of the advice given has been model and task-agnostic.
+Anything more detailed has to be specific to the model and the relevant
+task. Stick close to working architectures and hyperparameters from
+places like HuggingFace, and try not to reinvent the wheel!
+
+## 4 - Resources
+
+Here are some helpful resources that discuss this topic.
+
+### Tweeters
+
+1.  [Julia Evans](https://twitter.com/b0rk)
+
+2.  [Charity Majors](https://twitter.com/mipsytipsy)
+
+3.  [Nelson Elhage](https://twitter.com/nelhage)
+
+4.  [kipply](https://twitter.com/kipperrii)
+
+5.  [Horace He](https://twitter.com/cHHillee)
+
+6.  [Andrej Karpathy](https://twitter.com/karpathy)
+
+7.  [Chip Huyen](https://twitter.com/chipro)
+
+8.  [Jeremy Howard](https://twitter.com/jeremyphoward)
+
+9.  [Ross Wightman](https://twitter.com/wightmanr)
+
+### Templates
+
+1.  [Lightning Hydra
+Template](https://github.com/ashleve/lightning-hydra-template)
+
+2.  [NN Template](https://github.com/grok-ai/nn-template)
+
+3.  [Generic Deep Learning Project
+Template](https://github.com/sudomaze/deep-learning-project-template)
+
+### Texts
+
+1.  [Reliable ML Systems
+talk](https://www.usenix.org/conference/opml20/presentation/papasian)
+
+2.  ["ML Test Score"
+paper](https://research.google/pubs/pub46555/)
+
+3.  ["Attack of the Cosmic
+Rays!"](https://blogs.oracle.com/linux/post/attack-of-the-cosmic-rays)
+
+4.  ["Computers can be
+understood"](https://blog.nelhage.com/post/computers-can-be-understood/)
+
+5.  ["Systems that defy detailed
+understanding"](https://blog.nelhage.com/post/systems-that-defy-understanding/)
+
+6.  [Testing section from MadeWithML course on
+MLOps](https://madewithml.com/courses/mlops/testing/)

documents/lecture-03.srt

@@ -0,0 +1,244 @@
+1
+00:00:00,080 --> 00:00:42,480
+hey folks welcome to the third lecture of full stack deep learning 2022 i'm charles frye today i'll be talking about troubleshooting and testing a high level outline of what we're going to cover today we'll talk about testing software in general and the sort of standard tools and practices that you can use to de-risk shipping software quickly then we'll move on to special considerations for testing machine learning systems the specific techniques and approaches that work best there and then lastly we'll go through what you do when your models are failing their tests how you troubleshoot your model first let's cover concepts for testing software so the general approach that we're gonna take is that tests are gonna help us ship faster with
+
+2
+00:00:40,800 --> 00:01:26,159
+fewer bugs but they aren't gonna catch all of our bugs and that means though we're going to use testing tools we aren't going to try and achieve 100 coverage similarly we're going to use linting tools to try and improve the development experience but leave escape valves rather than pedantically just following our style guides lastly we'll talk about tools for automating these workflows so first why are we testing it all tests can help us ship faster even if they aren't catching all bugs before they go into production so as a reminder what even our tests tests are code we write that's designed to fail in an intelligible way when our other code has bugs so for example this little test for the text recognizer from the full stack
+
+3
+00:01:24,320 --> 00:02:06,240
+deep learning code base checks whether the output of the text recognizer on a particular input is the same as what was expected and raises an error if it's not and these kinds of tests can help catch some bugs before they're merged into main or shipped into production but they can't catch all bugs and one reason why is that test suites in the tools we're using are not certificates of correctness in some formal systems tests like those can actually be used to prove that code is correct but we aren't working in one of those systems like agda the fear improving language or idris ii we're writing in python and so really it's a loosey-goosey language and all bets are off in terms of code correctness so if test suites aren't
+
+4
+00:02:05,439 --> 00:02:49,599
+like certificates of correctness then what are they like i like this framing from nelson el haga who's at anthropic ai who says that we should think of test suites as being more like classifiers and so to bring our intuition from working with classification algorithms and machine learning so the classification problem is does this commit have a bug or is it okay and what are classifier outputs is that the test pass or the test failed so our tests are our classifier of code and so you should think of that as a prediction of whether there's a bug this kind of frame shift suggests a different way of designing our test suites when we design classifiers we know that we need to trade off detection and false alarms
+
+5
+00:02:47,519 --> 00:03:34,159
+lots of people are thinking about detection when they're designing their test suites they're trying to make sure that they will catch all of the possible bugs but in doing so we can inadvertently introduce false alarms so the classic signature of a false alarm is a failed test that's followed by a commit that fixes the test rather than the code so that's an example from the full stack deep learning code base so in order to avoid introducing too many false alarms it's useful to ask yourself two questions before adding a test so the first question is which real bugs will this test catch what are some actual ways that the world might change around this code or that somebody might introduce a change to to some part of
+
+6
+00:03:32,159 --> 00:04:16,160
+the code base that this test will catch once you've listed a couple of those then ask yourself what are some false alarms that this test might raise what are some ways that the world around the test or the code could change that's still valid in good code but now this test will fail and if you can think of more examples for the latter case than the former then maybe you should reconsider whether you really need this test one caveat to this in some settings it actually is really important that you have a super high degree of confidence in the correctness of your code so this screenshot is from a deep learning diagnostic tool for cardiac ultrasounds by caption health that i worked on in an internship in that project we had a ton
+
+7
+00:04:14,159 --> 00:05:00,240
+of concern about the correctness of the model the confidence people had in the model and regulators expected to see that kind of information so there's other cases where this level of correctness is needed self-driving cars is one example you also see this in banking and finance there are a couple of patterns that immediately arise here one is the presence of regulators uh and more generally high stakes if you're operating in a high-stakes situation where errors have consequences for people's lives and livelihoods even if it's not regulated yet it might be regulated soon and in particular these are also all examples of those autonomous systems that class of low feasibility high impact machine learning project that we talked about in the
+
+8
+00:04:58,320 --> 00:05:44,240
+first lecture this is one of the reasons for their low feasibility is because correctness becomes really important for these kinds of autonomous systems so what does this mindset mean for how we approach testing and quality assurance for our code it means that we're going to use testing tools but we don't want to aim for complete coverage of our code so in terms of tools pi test is the standard tool for testing python code it is a very pythonic implementation and interface and it has also a ton of powerful features like marks for creating separate suites of tests sharing resources across tests and running tests in a variety of parameterized variations in addition to writing the kinds of separate test suites that are standard in lots of
+
+9
+00:05:42,080 --> 00:06:29,440
+languages in python there's a nice built-in tool called doctest for testing the code inside of our docstrings and this helps make sure that our docs strings don't get out of sync with our code which builds trust in the content of those docs strings and makes them easier to maintain doc tests are really nice but there are some limits they're framed around code snippets that could be run in a terminal and so they can only display what can be easily displayed in a terminal notebooks on the other hand can display things like rich media charts images and web pages also interleaved with code execution and text so for example with our data processing code we have some notebooks that that have charts and images in them that explain choices in
+
+10
+00:06:27,520 --> 00:07:08,960
+that data processing trouble is notebooks are hard to test we use a cheap and dirty solution we make sure that our notebooks run and to end then we add some assert statements and we use nb format to run the notebooks and flag when they sail so once you start adding lots of different types of tests and as your code base grows you're going to want to have tooling for recording what kind of code is actually being checked or covered by the tests typically this is done in terms of lines of code but some tools can be a little bit more finer grained the tool that we recommend for this is called codecov it generates a lot of really nice visualizations that you can use to drill down or get a high level overview of the current state of
+
+11
+00:07:07,360 --> 00:07:53,039
+your testing this is a great tool for helping you understand your testing and its state it can be incorporated into your testing effectively saying i'm going to reject commits not only where tests fail but also where test coverage goes down below some value or by a certain amount but we actually recommend against that personal experience interviews and even some published research suggests that only a small fraction of the tests that you write are going to generate the majority of your value and so the right tactic engineering wise is to expend the limited engineering effort that we have on the highest impact tests and making sure those are super high quality but if you set a coverage target then you're
+
+12
+00:07:51,280 --> 00:08:36,560
+instead going to write tests in order to meet that coverage target regardless of their quality so you end up spending more effort both to write the tests and then to maintain and deal with their low quality in addition to checking that our code is correct we're going to also want to check that our code is clean with linting tools but with the caveat that we always want to make sure that there are escape valves from these tools when we say the code is clean what we mean is that it's of a uniform style and of a standard style so uniform style helps avoid spending engineering time on arguments over style in pull requests and code review it also helps improve the utility of our version control system by cutting down on
+
+13
+00:08:33,360 --> 00:09:20,959
+unnecessary noisy components of dips and reducing their size both of these things will make it easier for humans to visually parse the dips in our version control system and make it easier to build automation around them and then we also generally want to adopt a standard style in whatever community it is that we are writing our code if you're an open source repository this is going to make it easier to accept contributions and even if you're working on a closed source team if your new team members are familiar with this style that's standard in the community they'll be faster to onboard one aspect of consistent style is consistent formatting of code with things like white space the standard tool for that in python is the black
+
+14
+00:09:18,320 --> 00:10:04,399
+python formatter it's a very opinionated tool but it has a fairly narrow scope in terms of style it focuses on things that can be fully automated so you can see it not only detects deviations from style but also implements the fix so that's really nice integrated into your editor integrate it into automated workflows and avoid engineers having to implement these things themselves for non-automatable aspects of style the tool we recommend is flake 8. non-automatable aspects of style are things like missing doc strings we don't have good enough automation tools to reliably generate doc strings for code automatically so these are going to require engineers to intervene in order to fix them one of the best things about flakegate is that
+
+15
+00:10:02,000 --> 00:10:47,680
+it comes with tons of extensions and plugins so you can check things like doctrine style and completeness like type hinting and even for security issues and common bugs all via flake8 extensions so those cover your python code ml code bases often have both python code and shell scripts in them shell scripts are really powerful but they also have a lot of sharp edges so shell check knows about all these kinds of weird behaviors of bash that often cause errors and issues that aren't immediately obvious and it also provides explanations for why it's raising a warning or an error it's a very fast to run tool so you can incorporate it into your editor and because it includes explanations you can often resolve the
+
+16
+00:10:45,279 --> 00:11:33,600
+issue without having to go to google or stack overflow and switch contexts out of your editing environment so these tools are great and a uniform style is important but really pedantically enforcing style can be self-defeating so i searched for the word slaykate on github and found over a hundred thousand commits mentioning placating these kinds of automated style enforcement tools and all these commits sort of drip frustration from engineers who are spending time on this that they wish that they were not so to avoid frustration with code style and linting we recommend filtering your rules down to the minimal style that achieves the goals that we set of sticking with standards and of avoiding arguments and
+
+17
+00:11:30,640 --> 00:12:12,880
+of keeping version control history clean another suggestion is to have an opt-in rather than an opt-out application of rules so by default many of these rules may not be applied to all files in the code base but you can opt in and add a particular rule to a particular file and then you can sort of grow this coverage over time and avoid these kinds of frustrations this is especially important for applying these kinds of style recommendations to existing code bases which may have thousands of lines of code that need to be fixed in order to make best use of these testing and linting practices you're going to want to embrace automation as much as possible in your development workflows for the things we talked about already
+
+18
+00:12:11,760 --> 00:12:58,240
+with testing and linting you're going to want to automate these and connect them to your cloud version control system and run these tasks in the cloud or otherwise outside of development environments so connecting diversion control state reduces friction when trying to reproduce or understand errors and running things outside of developer environments means that you can run these tests in parallel to other development work so you can kick off tests that might take 10 or 20 minutes and spend that time responding to slack messages or moving on to other work one of the best places to learn about best practices for automation are popular open source repos so i checked out pytorch's github repository and found
+
+19
+00:12:55,040 --> 00:13:42,800
+that they had tons and tons of automated workflows built into the repository they also followed what i think are some really nice practices like they had some workflows that are automatically running on every push and pull and these are mostly code related tasks that run for less than 10 minutes so that's things like linting and maybe some of the quicker tests other tasks that aren't directly code related but maybe do things like check dependencies and any code-related tasks that take more than 10 minutes to run are run on a schedule so we can see that for example closing stale pull requests is done on a schedule because it's not code related pytorch also runs a periodic suite of tests that takes hours to run you don't
+
+20
+00:13:40,639 --> 00:14:23,920
+want to run that every time that you push or pull so the tool that they use and that we recommend is github actions this ties your automation entirely directly to your version control system and that has tons of benefits also github actions is really powerful it's really flexible there's a generous free tier it's performant and on top of all this it's really easy to use it's got really great documentation configuring github actions is done just using a yaml file and because of all these features it's been embraced by the open source community which has contributed lots and lots of github actions that maybe already automate the workflow that you're interested in that's why we recommend github actions there are other
+
+21
+00:14:21,360 --> 00:15:07,120
+options precommit.ci circleci and jenkins all great choices all automation tools that i've seen work but github actions seems to have won hearts and minds in the open source community in the last couple years so that makes sure that these tests and lints are being run in code before it's shipped or before it's merged into main but part of our goal was to keep our version control history as clean as possible so we want to be able to run these locally as well and before committing and so for that we recommend a tool called pre-commit which can run all kinds of different tools and automations automatically before commits so it's extremely flexible and can run lots of stuff you will want to keep the total run time to just a few seconds or
+
+22
+00:15:05,440 --> 00:15:46,160
+you'll discourage engineers from committing which can lead to work getting lost pre-commit super easy to run locally in part because it separates out the environment for these linting tools from the rest of the development environment which avoids a bunch of really annoying tooling and system administration headaches they're also super easy to automate with github actions automation to ensure the quality and integrity of our software is a huge productivity enhancer that's broader than just ci cd which is how you might which is how you might hear tools like github actions referred to automation helps you avoid context switching if a task is being run fully automatically then you don't have to switch context and remember the
+
+23
+00:15:44,160 --> 00:16:26,880
+command line arguments that you need in order to run your tool it services issues more quickly than if these things were being run manually it's a huge force multiplier for small teams that can't just throw engineer hours at problems and it's better documented than manual processes the script or artifact that you're using to automate a process serves as documentation for how a process is done if somebody wants to do it manually the one caveat is that fully embracing automation requires really knowing your tools well knowing docker well enough to use it is not the same as knowing docker well enough to automate it and bad automation like bad tests can take more time away than it saves so organizationally that actually makes
+
+24
+00:16:24,720 --> 00:17:08,559
+automation a really good task for senior engineers who have knowledge of these tools have ownership over code and can make these kinds of decisions around automation perhaps with junior engineer mentees to actually write the implementations so in summary automate tasks with github actions to reduce friction in development and move more quickly use the standard python tool cat for testing and cleaning your projects and choose in that toolkit the testing and linting practices with the 80 20 principle for tests with shipping velocity and with usability and developer experience in mind now that we've covered general ideas for testing software systems let's talk about the specifics that we need for testing machine learning systems the key point
+
+25
+00:17:06,880 --> 00:17:53,039
+in this section is that testing email is difficult but if we adapt ml specific coding practices and focus on low hanging fruit to start then we can test our ml code and then additionally testing machine learning means testing in production but testing in production doesn't mean that you can just release bad code and let god sort it out so why is testing machine learning hard so software engineering is where a lot of testing practices have been developed and in software engineering we compile source code into programs so we write source code and a compiler turns that into a program that can take inputs and return outputs in machine learning training compiles in a sense data into a model and all of these components are
+
+26
+00:17:51,520 --> 00:18:46,240
+harder to test in the machine learning case than in the software engineering case data is heavier and more inscrutable than source code training is more complex less well-defined and less mature than compilation and models have worse tools for debugging and inspection than compiled programs so this means that ml is the dark souls of software testing it's a notoriously difficult video game but just because something is difficult doesn't mean that it's impossible in the latest souls game elden ring a player named let me solo her defeated one of the hardest bosses in the game wearing nothing but a jar on their head if testing machine learning code is the dark souls of software testing then with practice and with the
+
+27
+00:18:44,080 --> 00:19:28,400
+right techniques you can become the let me solo her of software testing and so in our recommendations in this section we're going to focus mostly on what are sometimes called smoke tests which let you know when something is on fire and help you resolve that issue so these tests are easy to implement but they are still very effective so they're among the 20 percent of tests that get us 80 of the value for data the kind of smoke testing we recommend is expectation testing so we test our data by checking basic properties we express our expectations about the data which might be things like there are no nulls in this column the completion date is after the start date and so with these you're going to want to start small checking
+
+28
+00:19:26,160 --> 00:20:13,679
+only a few properties and grow them slowly and only test things that are worth raising alarms over worth sending people notifications worth bringing people in to try and resolve them so you might be tempted to say oh these are human heights they should be between four and eight feet but actually there are people between the heights of two and ten feet so loosening these expectations to avoid false positives is an important way to make them more useful so you can even say that i should be not negative and less than 30 feet and that will catch somebody maybe accidentally entering a height in inches but it doesn't express strong expectations about the statistical distribution of heights you could try and build something for expectation
+
+29
+00:20:11,200 --> 00:20:52,480
+testing with a tool like pie test but there's enough specifics and there's good enough tools that it's worth reaching for something else so the tool we recommend is great expectations in part because great expectation automatically generates documentation for your data and quality reports and has built-in logging and learning designed for expectation testing so we are going to go through this in the lab we'll go through a lot of the other tools that we've talked about in the lab this week so if you want to check out great expectations we recommend the made with ml tutorial on great expectations by gogumontis loose expectation testing is a really uh is a great start for testing your data pipeline what do you
+
+30
+00:20:50,720 --> 00:21:33,600
+do as you move forward from that the number one recommendation that i have is to stay as close to your data as possible so from top to bottom we have data annotation setups going from furthest away from the model development team to closest one common pattern is that there's some benchmark data set with annotations that you're using uh which is super common in academia or there's an external annotation team which is very common in industry and in that case a lot of the detailed information about the data that you can learn by looking at it and using it yourself are going to be internalized into the organization so one way that that sometimes does get internalized is that at the start of the project some
+
+31
+00:21:31,280 --> 00:22:16,080
+data will get annotated ad hoc by model developers especially if you're not using some external benchmark data set or you don't yet have budget for an external annotation team and that's an improvement but if the model developers who around at the start of the project move on and as more developers get onboarded that knowledge is diluted better than that is an internal annotation team that has regular information flow whether that's stand-ups and syncs or exchange of documentation that information flows to the model developers but probably the best practice and one that i saw recommended by shreya shankar on twitter is to have a regular on-call rotation where model developers annotate data themselves ideally fresh data so that
+
+32
+00:22:13,919 --> 00:22:59,840
+all members of the team who are developing models know about the data and develop intuition and expertise in the data for testing our training code we're going to use memorization testing so memorization is the simplest form of learning steep neural networks are very good at memorizing data and so checking whether your model can memorize a very small fraction of the full data set is a great smoke test for trading and if a model can't memorize then something is clearly very wrong only really gross issues with training are going to show up with this test so your gradients aren't being calculated correctly you have a numerical issue your labels have been shuffled and subtle bugs in your model or your data
+
+33
+00:22:57,760 --> 00:23:38,240
+are not going to show up in this but you can improve the coverage of this test by including the run time in the test because regressions there can reveal bugs that just checking whether you can eventually memorize a small data set wouldn't reveal so if you're including the wall time that can catch performance regressions but also if you're including the number of steps or epochs required to hit some criterion value of the loss then you can catch some of these small issues that make learning harder but not impossible there's a nice feature of pytorch lighting overfit batches that can quickly implement this memorization test and if you design them correctly you can incorporate these tests into end-to-end model deployment testing to
+
+34
+00:23:36,880 --> 00:24:14,880
+check to make sure that the data that the model memorized in training is also something it can correctly respond to in production with these memorization tests you're going to want to tune them to run quickly so that you can run them as often as possible if you can get them to under 10 minutes you might run them on every pull request or on every push so this is something that we worked on in updating the course for 2022 so the simplest way to speed these jobs up is to simply buy faster machines but if you're already on the fastest machines possible or you don't have budget then you start by reducing the size of the data set that the model is memorizing down to the batch size that you want to use in training once you reduce the
+
+35
+00:24:13,120 --> 00:24:52,559
+batch size below what's in training you're starting to step further and further away from the training process that you're trying to test and so going down this list we're getting further and further from the thing that we're actually testing but allowing our tests to run more quickly the next step that can really speed up a memorization test is to turn off regular regularization which is meant to reduce overfitting and memorization is a form of overfitting so that means turning off dropout turning off augmentation you can also reduce the model size without reducing the architecture so reduce the number of layers reduce the width of layers while keeping all of those components in place and if that's not enough you can remove
+
+36
+00:24:50,799 --> 00:25:31,679
+some of the most expensive components and in the end you should end up with a tier of memorization tests which are more or less close to how you actually train models in production that you can run on different time scales one recommendation you'll see moving forward and trying to move past just smoke testing by checking for memorization is to rerun old training jobs with new code so this is never something that you're going to be able to run on every push probably not nightly either if you're looking at training jobs that run for multiple days and the fact that this takes a long time to run is one of the reasons why it's going to be really expensive no matter how you do it for example if you use if you're gonna be
+
+37
+00:25:29,360 --> 00:26:10,720
+doing this with circle ci uh you'll need gpu runners to execute your training jobs but those are only available in the enterprise level plan which is twenty four thousand dollars a year at a bare minimum i've seen some very large bills for running gpus in circleci github actions on the other hand does not have gpu runners available so you'll need to host them yourself though it is on the roadmap to add gpu runners to github actions and that means that you're probably going to maybe double your training spend maybe you're adding an extra machine to rerun your training jobs or maybe you're adding to your cloud budget to pay for more cloud machines to run these jobs and all this expenditure here is only on testing code
+
+38
+00:26:09,120 --> 00:26:52,159
+it doesn't have any connection to the actual models that we're trying to ship the best thing to do is to test training by regularly running training with new data that's coming in from production this is still going to be expensive because you're going to be running more training than you were previously but now that training spend is going to model development not code testing so it's easier to justify having this set up requires the data flywheel that we talked about in lecture one and that requires production monitoring tooling and all kinds of other things that we'll talk about in the monitoring and continual learning lecture lastly for testing our models we're going to adapt regression testing at a very base level
+
+39
+00:26:49,440 --> 00:27:31,279
+models are effectively functions so we can test them like functions they take in inputs they produce outputs we can write down what those outputs should be for specific inputs and then test them this is easiest for classification and other tasks with simple output if you have really complex output like for example our text recognizer that returns tests then these tests can often become really flaky and hard to maintain but even for those kinds of outputs you can use these tests to check for differences between how the model is behaving in training and how it's behaving in production the better approach is still relatively straight forward is to use the values of the loss and your metrics to help build documented regression test
+
+40
+00:27:29,120 --> 00:28:10,880
+suites out of your data out of the data you're using in training and the data you see in production the framing that i like to bring to this comes from test driven development so test driven development is a paradigm for testing that says first you write the test and that test fails because you haven't written any of the code that it's testing and then you write code until you pass the test this is straightforward or incorporate into testing our models because in some sense we're already doing it think of the loss as like a fuzzy test signal rather than simply failing or not failing the loss tells us how badly a test was failed so how badly did we miss the expected output on this particular input and so
+
+41
+00:28:09,360 --> 00:28:51,919
+just like in test driven development that's a test that's written before that code writing process and during training our model is changing and it changes in order to do better on the tests that we're providing and the model stops changing once it passes the test so in some sense gradient descent is already test driven development and maybe that is an explanation for the carpathi quote that gradient descent writes better code than me but just because gradient scent is test-driven development doesn't mean that we're done testing our models because what's missing here is that the loss and other metrics are telling us that we're failing but they aren't giving us actionable insights or a way to resolve that failure the simplest and
+
+42
+00:28:49,679 --> 00:29:37,600
+most generic example is to find data points with the highest loss in your validation and test set or coming from production and put them in a suite labeled hard but note that the problem isn't always going to be with the model searching for high loss examples does reveal issues about what your model is learning but it also reveals issues in your data like bad labels so this doesn't just test models it also tests data and then we want to aggregate individual failures that we observe into named suites of specific types of failure so this is an example from a self-driving car task of detecting pedestrians cases where pedestrians were not detected it's much easier to incorporate this into your workflows if you already have a
+
+43
+00:29:36,240 --> 00:30:13,840
+connection between your model development team and your annotation team reviewing these examples here we can see that what seems to be the same type of failure is occurring more than once in two examples there's a pedestrian who's not visible because they're covered by shadows in two examples there are reflections off of the windshield that are making it harder to see the pedestrian and then some of the examples come from night scenes so we can collect these up create a data set with that label and treat these as test suites to drive model development decisions and so this is kind of like the machine learning version of a type of testing called regression testing where you take bugs that you've observed
+
+44
+00:30:12,159 --> 00:30:56,320
+in production and add them to your test suite to make sure that they don't come up again so the process that i described is very manual but there's some hope that this process might be automated in the near future a recent paper described a method called domino that uses much much larger cross-modal embedding models so foundation models to understand what kinds of errors a smaller model like a specific model designed just to detect birds or pedestrians what kinds of mistakes is it making on images as your models get more mature and you understand their behavior and the data that they're operating on better you can start to test more features of your models so for more ways to test models with an emphasis on nlp see the
+
+45
+00:30:54,480 --> 00:31:38,399
+checklist paper that talks about different ways to do behavioral testing of models in addition to testing data training and models in our development environment we're also going to want to test in production and the reason why is that production environments differ from development environments this is something that is true for complex software systems outside of machine learning so charity majors of honeycomb has been a big proponent of testing and production on these grounds and this is especially true for machine learning models because data is an important component of both the production and the development environments and it's very difficult to ensure that those two things are close to each other and so
+
+46
+00:31:36,080 --> 00:32:18,720
+the solution in this case is to run our tests in production but testing in production doesn't mean only testing in production testing in production means monitoring production for errors and fixing them quickly as chip win the author of designing machine learning systems pointed out this means building infrastructure and tooling so that errors in production are quickly fixed doing this safely and effectively and ergonomically requires tooling to monitor production and a lot of that tooling is fairly new especially tooling that can handle the particular type of production monitoring that we need in machine learning we'll cover it along with monitoring and continual learning in that lecture so in summary we
+
+47
+00:32:16,720 --> 00:33:03,440
+recommend focusing on some of the low-hanging fruit when testing ml and sticking to tests that can alert you alert you to when the system is on fire so that means expectation tests of simple properties of data memorization tests for training and data-based regression tests for models but what about as your code base and your team matures one really nice rubric for organizing the testing of a really mature ml code base is the ml test score so the ml test score came out of google research and it's this really strict rubric for ml test quality so it includes tests for data models training infrastructure and production monitoring and it overlaps with but goes beyond some of the recommendations that we've
+
+48
+00:33:00,799 --> 00:33:48,559
+given already maintaining and automating all of these tests is really expensive but it can be worth it for a really high stakes or large scale machine learning system so we didn't use the ml test score to design the text recognizer code base but we can check what we implemented against it some of the recommendations in the machine learning test score didn't end up being relevant for our model for example some of the data tests are organized around tabular data for traditional machine learning rather than for deep learning but there's still lots of really great suggestions in the ml test score so you might be surprised to see we're only hitting a few of these criteria in each category but that's a function of how
+
+49
+00:33:46,240 --> 00:34:33,040
+strict this testing rubric is so they also provide some data on how teams doing ml at google did on this rubric and if we compare ourselves to that standard the text recognizer is about in the middle which is not so bad for a team not working with google scale resources tests alert us to the presence of bugs but in order to resolve them we'll need to do some troubleshooting and one of the components of the machine learning pipeline that's going to need the most troubleshooting and which is going to require very specialized approaches is troubleshooting models so the key idea in this section is to take a three-step approach to troubleshooting your model first make it run by avoiding the common kinds of errors that can
+
+50
+00:34:30,159 --> 00:35:18,240
+cause crashes shape issues out of memory errors and numerical problems then make your model fast by profiling it and removing any bottlenecks then lastly make the model write improve its performance on test metrics by scaling out the model and the data and sticking with proven architectures first how do we make a model run luckily this step is actually relatively easy in that only a small portion of bugs in machine learning cause the kind of loud failure that we're tackling here so there's shape errors out of memory errors and numerical errors shape errors occur when the shapes of tensors don't match the shapes expected by the operations applied to them so while you're writing your pytorch code it's a good idea to
+
+51
+00:35:15,200 --> 00:35:58,640
+keep notes on what you expect the shapes of your tensors to be to annotate those in the code as we do in the full stack deep learning code base and to even step through this code in a debugger checking the shapes as you go another one of the most common errors in deep learning is out of memory when you when you try and push a tensor to the gpu that's too large to fit on it something of a right of passage for deep learning engineering luckily pytorch lightning has a bunch of really nice tools built into this first make sure you're using the lowest precision that your training can tolerate a good default is half precision floats or 16-bit floats a common culprit is that you're trying to run your model on too much data at once
+
+52
+00:35:57,040 --> 00:36:40,400
+on too large of a batch so you can use the auto scale batch size feature in pi torch lightning to pick a batch size that uses as much gpu memory as you have but no more and if that batch size is too small to get you stable gradients that can be used for training you can use gradient accumulation across batches also easily within lightning to get the same gradients that you would have gotten if you calculated on a much larger batch if none of those work and you're already operating on gpus with the maximum amount of ram then you'll have to look into manual techniques like tensor parallelism and gradient checkpointing another cause of crashes for machine learning models is numerical errors when tensors end up with nands or
+
+53
+00:36:38,240 --> 00:37:23,119
+infinite values in them most commonly these numerical issues appear first in the gradient the gradients explode or shrink to zero and then the values of parameters or activations become infinite or nan so you can observe some of these gradient spikes occurring in some of the experiments for the dolly mini project that have been publicly posted pi torch lightning comes with a nice tool for tracking gradient norms and logging them so that you can see them and correlate them with the appearance of nance and infinities and crashes in your training a nice debugging step to check what the cause might be whether the cause is due to precision issues or due to a more fundamental numerical issue is to switch to double precision floats the default
+
+54
+00:37:20,640 --> 00:38:02,960
+floating point size in python 64-bit floats and see if that causes these issues to go away if it doesn't then that means that there's some kind of issue with your numerical code and you'll want to find a numerically stable implementation to base your work off of or apply error analysis techniques and one of the most common causes of these kinds of numerical errors are the normalization layers like batch norm and layer norm that's what's involved in these gradient spikes in dolly mini so you'll want to make sure to check carefully that you're using normalization in the way that's been found to work for the types of data and architectures that you're using once your battle can actually run end to end and calculate gradients correctly the
+
+55
+00:38:00,000 --> 00:38:47,040
+next step is to make it go fast this could be tricky because the performance of deep neural network training code is very counter-intuitive for example with typical hyper-parameter choices transformer layers spend more time on the plain old mlp component than they do on the intention component and as we saw in lecture two for popular optimizers just keeping track of the optimizer state actually uses more gpu memory than any of the other things you might expect would take up that memory like model parameters or data and then furthermore without careful parallelization what seem like fairly trivial components like loading data can end up dwarfing what would seem like the actual performance bottlenecks like the
+
+56
+00:38:45,200 --> 00:39:31,440
+forwards and backwards passes and parameter updates the only solution here is to kind of roll up your sleeves and get your hands dirty and actually profile your code so we'll see this in the lab but the good news is that you can often find relatively low hanging fruit to speed up training like making changes just in the regular python code and not in not any component of the model and lastly once you've got a model that can run and that runs quickly it's time to make the model correct by reducing its loss on tester production data the normal recommendation for software engineering is make it run make it right make it fast so why is make it right last in this case and the reason why is that machine learning models are
+
+57
+00:39:29,359 --> 00:40:12,480
+always wrong production performance is never perfect and if we think of non-zero loss as a partial test failure for our models then our tests are always at least partially failing so it's never really possible to truly make it right and then the other reason that we want to put performance first is they can kind of solve all your problems with model correctness with scale so if your model is over fitting to the training data and your production loss is way higher then you can scale up your data if your model is underfitting and you're you can't get the training loss to go down as much as you'd like then scale up your model if you're have distribution shift which means that your training and validation loss are both low but your
+
+58
+00:40:09,920 --> 00:40:57,680
+production or test loss is really high then just scale up both folks at openai and elsewhere have done work demonstrating that the performance benefits from scale can be very rigorously measured and predicted across compute budget data set size and parameter count generating these kinds of scaling law charts is an important component of openai's workflows for deciding how to build models and how to run training but scaling costs money so what do you do if you can't afford the level of scale required to reach the performance that you want in that case you're going to want to fine-tune or make use of a model trained at scale for your tasks this is something we'll talk about in the building on foundation models lecture all the other advice
+
+59
+00:40:55,200 --> 00:41:40,800
+around addressing overfitting addressing underfitting resolving distribution shift is going to be model and task specific and it's going to be hard to know what is going to work without trying it so this is just a selection of some of the advice i've seen given or been given about improving model performance and they're mutually exclusive in many cases because they're so tied to the particular task and model and data that they're being applied to so the easiest way to resolve this is to stick as close as possible to working architectures and hyper parameter choices that you can get from places like the hugging face hub or papers with code and in fact this is really how these hyperparameter choices and architectures arise it's via a slow
+
+60
+00:41:38,560 --> 00:42:22,480
+evolutionary process of people building on techniques and hyperparameter choices that work rather than people designing things entirely from scratch so that brings us to the end of the troubleshooting and testing lecture we covered the general approach to testing software both tools and practices that you can use to ship more safely more quickly then we covered the specific things that you need in order to test ml systems data sets training procedures and models both the most basic tests that you should implement at the beginning and then how to grow those into more sophisticated more robust tests and then lastly we considered the workflows and techniques that you need to troubleshoot model performance so
+
+61
+00:42:20,160 --> 00:42:42,359
+we'll see more on all these topics in the lab for this week if you'd like to learn more about any of these topics check out the slides online for a list of recommended twitter follows project templates and medium to long form text resources to learn more about troubleshooting and testing that's all for this lecture thanks for listening and happy testing
+

documents/lecture-04.md

@@ -0,0 +1,421 @@
+---
+description: Sourcing, storing, exploring, processing, labeling, and versioning data for deep learning.
+---
+
+# Lecture 4: Data Management
+
+<div align="center">
+<iframe width="720" height="405" src="https://www.youtube-nocookie.com/embed/Jlm4oqW41vY?list=PL1T8fO7ArWleMMI8KPJ_5D5XSlovTW_Ur" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+Lecture by [Sergey Karayev](https://sergeykarayev.com).<br />
+Notes by [James Le](https://twitter.com/le_james94) and [Vishnu Rachakonda](https://www.linkedin.com/in/vrachakonda/).<br />
+Published August 29, 2022.
+[Download slides](https://fsdl.me/2022-lecture-04-slides).
+
+## 1 - Introduction
+
+One thing people don't quite get as they enter the field of ML is how
+much of it deals with data - putting together datasets, exploring the
+data, wrangling the data, etc. The key points of this lecture are:
+
+1.  Spend 10x as much time exploring the data as you would like to.
+
+2.  Fixing, adding, and augmenting the data is usually the best way to
+improve performance.
+
+3.  Keep it all simple!
+
+## 2 - Data Sources
+
+![](./media/image9.png)
+
+There are many possibilities for the sources of data. You might have
+images, text files, logs, or database records. In deep learning, you
+need to get that data into a local filesystem disk next to a GPU. **How
+you send data from the sources to training is different for each
+project**.
+
+-   With images, you can simply download them from S3.
+
+-   With text files, you need to process them in some distributed way,
+analyze the data, select a subset, and put that on a local
+machine.
+
+-   With logs and database records, you can use a data lake to aggregate
+and process the data.
+
+![](./media/image2.png)
+
+
+The basics will be the same - a filesystem, object storage, and
+databases.
+
+### Filesystem
+
+The **filesystem** is a fundamental abstraction. Its fundamental unit is
+a file - which can be text or binary, is not versioned, and is easily
+overwritten. The filesystem is usually on a disk connected to your
+machine - physically connected on-prem, attached in the cloud, or even
+distributed.
+
+The first thing to know about discs is that their speed and bandwidth
+range - from hard discs to solid-state discs. There are two orders of
+magnitude differences between the slowest (SATA SSD) and the fastest
+(NVMe SSD) discs. Below are some latency numbers you should know, with
+the human-scale numbers in parentheses:
+
+![](./media/image12.png)
+
+
+What formats should the data be stored on the local disc?
+
+-   If you work with binary data like images and audio, just use the
+standard formats like JPEG or MP3 that it comes in.
+
+-   If you work with metadata (like labels), tabular data, or text data,
+then compressed JSON or text files are just fine. Alternatively,
+Parquet is a table format that is fast, compact, and widely used.
+
+### Object Storage
+
+The **object storage** is an API over the filesystem. Its fundamental
+unit is an object, usually in a binary format (an image, a sound file, a
+text file, etc.). We can build versioning or redundancy into the object
+storage service. It is not as fast as the local filesystem, but it can b
+fast enough within the cloud.
+
+### Databases
+
+**Databases** are persistent, fast, and scalable storage and retrieval
+of structured data systems. A helpful mental model for this is: all the
+data that the databases hold is actually in the computer\'s RAM, but the
+database software ensures that if the computer gets turned off,
+everything is safely persisted to disk. If too much data is in the RAM,
+it scales out to disk in a performant way.
+
+You should not store binary data in the database but the object-store
+URLs instead. [Postgres](https://www.postgresql.org/) is
+the right choice most of the time. It is an open-source database that
+supports unstructured JSON and queries over that JSON.
+[SQLite](https://www.sqlite.org/) is also perfectly good
+for small projects.
+
+Most coding projects that deal with collections of objects that
+reference each other will eventually implement a crappy database. Using
+a database from the beginning with likely save you time. In fact, most
+MLOps tools are databases at their core (e.g.,
+[W&B](https://wandb.ai/site) is a database of experiments,
+[HuggingFace Hub](https://huggingface.co/models) is a
+database of models, and [Label
+Studio](https://labelstud.io/) is a database of labels).
+
+![](./media/image11.png)
+
+
+**Data warehouses** are stores for online analytical processing (OLAP),
+as opposed to databases being the data stores for online transaction
+processing (OLTP). You get data into the data warehouse through a
+process called **ETL (Extract-Transform-Load)**: Given a number of data
+sources, you extract the data, transform it into a uniform schema, and
+load it into the data warehouse. From the warehouse, you can run
+business intelligence queries. The difference between OLAP and OLTP is
+that: OLAPs are column-oriented, while OLTPs are row-oriented.
+
+![](./media/image13.png)
+
+
+**Data lakes** are unstructured aggregations of data from multiple
+sources. The main difference between them and data warehouses is that
+data lakes use ELT (Extract-Load-Transform) process: dumping all the
+data in and transforming them for specific needs later.
+
+**The big trend is unifying both data lake and data warehouse, so that
+structured data and unstructured data can live together**. The two big
+platforms for this are
+[Snowflake](https://www.snowflake.com/) and
+[Databricks](https://www.databricks.com/). If you are
+really into this stuff, "[Designing Data-Intensive
+Applications](https://dataintensive.net/)" is a great book
+that walks through it from first principles.
+
+## 3 - Data Exploration
+
+![](./media/image4.png)
+
+To explore the data, you must speak its language, mostly SQL and,
+increasingly, DataFrame. **SQL** is the standard interface for
+structured data, which has existed for decades. **Pandas** is the main
+DataFrame in the Python ecosystem that lets you do SQL-like things. Our
+advice is to become fluent in both to interact with both transactional
+databases and analytical warehouses and lakes.
+
+[Pandas](https://pandas.pydata.org/) is the workhorse of
+Python data science. You can try [DASK
+DataFrame](https://examples.dask.org/dataframe.html) to
+parallelize Pandas operations over cores and
+[RAPIDS](https://rapids.ai/) to do Pandas operations on
+GPUs.
+
+## 4 - Data Processing
+
+![](./media/image8.png)
+
+Talking about data processing, it's useful to have a motivational
+example. Let's say we have to train a photo popularity predictor every
+night. For each photo, the training data must include:
+
+1.  Metadata (such as posting time, title, and location) that sits in
+the database.
+
+2.  Some features of the user (such as how many times they logged in
+today) that are needed to be computed from logs.
+
+3.  Outputs of photo classifiers (such as content and style) that are
+needed to run the classifiers.
+
+Our ultimate task is to train the photo predictor model, but we need to
+output data from the database, compute the logs, and run classifiers to
+output their predictions. As a result, we have **task dependencies**.
+Some tasks can't start until others are finished, so finishing a task
+should kick off its dependencies.
+
+Ideally, dependencies are not always files but also programs and
+databases. We should be able to spread this work over many machines and
+execute many dependency graphs all at once.
+
+![](./media/image7.png)
+
+
+-   [Airflow](https://airflow.apache.org/) is a standard
+scheduler for Python, where it's possible to specify the DAG
+(directed acyclic graph) of tasks using Python code. The operator
+in that graph can be SQL operations or Python functions.
+
+-   To distribute these jobs, the workflow manager has a queue for the
+tasks and manages the workers that pull from them. It will restart
+jobs if they fail and ping you when the jobs are finished.
+
+-   [Prefect](https://www.prefect.io/) and
+[Dagster](https://dagster.io/) are contenders to
+improve and replace Airflow in the long run.
+
+The primary advice here is not to **over-engineer things**. You can get
+machines with many CPU cores and a lot of RAM nowadays. For example,
+UNIX has powerful parallelism, streaming, and highly optimized tools.
+
+## 5 - Feature Store
+
+![](./media/image3.png)
+
+Let's say your data processing generates artifacts you need for
+training. How do you make sure that, in production, the trained model
+sees the same processing taking place (which happened during training)?
+How do you avoid recomputation during retraining?
+
+**Feature stores** are a solution to this (that you may not need!).
+
+-   The first mention of feature stores came from [this Uber blog post
+describing their ML platform,
+Michelangelo](https://eng.uber.com/michelangelo-machine-learning-platform/).
+They had an offline training process and an online prediction
+process, so they built an internal feature store for both
+processes to be in sync.
+
+-   [Tecton](https://www.tecton.ai/) is the leading SaaS
+solution to feature store.
+
+-   [Feast](https://feast.dev/) is a common open-source
+option.
+
+-   [Featureform](https://www.featureform.com/) is a
+relatively new option.
+
+## 6 - Datasets
+
+![](./media/image1.png)
+
+What about datasets specifically made for machine learning?
+
+[HuggingFace
+Datasets](https://huggingface.co/docs/datasets) is a great
+source of machine learning-ready data. There are 8000+ datasets covering
+a wide variety of tasks, like computer vision, NLP, etc. The Github-Code
+dataset on HuggingFace is a good example of how these datasets are
+well-suited for ML applications. Github-Code can be streamed, is in the
+modern Apache Parquet format, and doesn't require you to download 1TB+
+of data in order to properly work with it. Another sample dataset is
+RedCaps, which consists of 12M image-text pairs from Reddit.
+
+![](./media/image15.png)
+
+
+Another interesting dataset solution for machine learning is
+[Activeloop](https://www.activeloop.ai/). This tool is
+particularly well equipped to work with data and explore samples without
+needing to download it.
+
+## 7 - Data Labeling
+
+![](./media/image10.png)
+
+### No Labeling Required
+
+The first thing to talk about when it comes to labeling data
+is...**maybe we don\'t have to label data?** There are a couple of
+options here we will cover.
+
+**Self-supervised learning** is a very important idea that allows you to
+avoid painstakingly labeling all of your data. You can use parts of your
+data to label other parts of your data. This is very common in NLP right
+now. This is further covered in the foundation model lecture. The long
+and short of it is that models can have elements of their data masked
+(e.g., the end of a sentence can be omitted), and models can use earlier
+parts of the data to predict the masked parts (e.g., I can learn from
+the beginning of the sentence and predict the end). This can even be
+used across modalities (e.g., computer vision *and* text), as [OpenAI
+CLIP](https://github.com/openai/CLIP) demonstrates.
+
+![](./media/image14.png)
+
+
+**Image data augmentation** is an almost compulsory technique to adopt,
+especially for vision tasks. Frameworks like
+[torchvision](https://github.com/pytorch/vision) help with
+this. In data augmentation, samples are modified (e.g., brightened)
+without actually changing their core "meaning." Interestingly,
+augmentation can actually replace labels.
+[SimCLR](https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html)
+is a model that demonstrates this - where its learning objective is to
+maximize agreement between augmented views of the same image and
+minimize agreement between different images.
+
+For other forms of data, there are a couple of augmentation tricks that
+can be applied. You can delete some cells in tabular data to simulate
+missing data. In text, there aren't established techniques, but ideas
+include changing the order of words or deleting words. In speech, you
+could change the speed, insert pauses, etc.
+
+**Synthetic data** is an underrated idea. You can synthesize data based
+on your knowledge of the label. For example, you can [create
+receipts](https://github.com/amoffat/metabrite-receipt-tests)
+if your need is to learn how to recognize receipts from images. This can
+get very sophisticated and deep, so tread carefully.
+
+You can also get creative and ask your users to label data for you.
+Google Photos, as any user of the app knows, regularly gets users to
+label images about where people in photos are the same or different.
+
+![](./media/image16.png)
+
+
+This is an example of the data flywheel. Improving the data allows the
+user to improve the model, which in turn makes their product experience
+better.
+
+### Labeling Solutions
+
+These are all great options for avoiding labeling data. However,
+**you'll usually have to label some data to get started.**
+
+Labeling has standard annotation features, like bounding boxes, that
+help capture information properly. Training annotators properly is more
+important than the particular kind of annotation. Standardizing how
+annotators approach a complex, opinable task is crucial. Labeling
+guidelines can help capture the exact right label from an annotator.
+Quality assurance is key to ensuring annotation and labeling are
+happening properly.
+
+There are a few options for sourcing labor for annotations:
+
+1.  Full-service data labeling vendors offer end-to-end labeling
+solutions.
+
+2.  You can hire and train annotators yourself.
+
+3.  You can crowdsource annotation on a platform like Mechanical Turk.
+
+**Full-service companies offer a great solution that abstracts the need
+to build software, manage labor, and perform quality checks**. It makes
+sense to use one. Before settling on one, make sure to dedicate time to
+vet several. Additionally, label some gold standard data yourself to
+understand the data yourself and to evaluate contenders. Take calls with
+several contenders, ask for work samples on your data, and compare them
+to your own labeling performance.
+
+-   [Scale AI](https://scale.com/) is the dominant data
+labeling solution. It offers an API that allows you to spin up
+tasks.
+
+-   Additional contenders include
+[Labelbox](https://labelbox.com/) and
+[Supervisely](https://supervise.ly/).
+
+-   [LabelStudio](https://labelstud.io/) is an open-source
+solution for performing annotation yourself, with a companion
+enterprise version. It has a great set of features that allow you
+to design your interface and even plug-in models for active
+learning!
+
+-   [Diffgram](https://diffgram.com/) is a competitor to
+Label Studio.
+
+-   Recent offerings, like
+[Aquarium](https://www.aquariumlearning.com/) and
+[Scale Nucleus](https://scale.com/nucleus), have
+started to help concentrate labeling efforts on parts of the
+dataset that are most troublesome for models.
+
+-   [Snorkel](https://snorkel.ai/) is a dataset management
+and labeling platform that uses weak supervision, which is a
+similar concept. You can leverage composable rules (e.g., all
+sentences that have the term "amazing" are positive sentiments)
+that allow you to quickly label data faster than if you were to
+treat every data point the same.
+
+In conclusion, try to avoid labeling using techniques like
+self-supervised learning. If you can't, use labeling software and
+eventually outsource the work to the right vendor. If you can't afford
+vendors, consider hiring part-time work rather than crowdsourcing the
+work to ensure quality.
+
+## 8 - Data Versioning
+
+![](./media/image6.png)
+
+Data versioning comes with a spectrum of approaches:
+
+1.  Level 0 is bad. In this case, data just lives on some file system.
+In these cases, the issue arises because the models are
+unversioned since their data is unversioned. Models are part code,
+part data. This will lead to the consequence of being unable to
+get back to a previous level of performance if need be.
+
+2.  You can prevent this event with Level 1, where you snapshot your
+data each time you train. This somewhat works but is far from
+ideal.
+
+3.  In Level 2, data is versioned like code, as a commingled asset with
+versioned code. You can use a system like
+[git-lfs](https://git-lfs.github.com/) that allows
+you to store large data assets alongside code. This works really
+well!
+
+4.  Level 3 involves specialized solutions for working with large data
+files, but this may not be needed unless you have a very specific
+need (i.e., uniquely large or compliance-heavy files).
+
+![](./media/image5.png)
+
+[DVC](https://dvc.org/) is a great tool for this. DVC
+helps upload your data asset to a remote storage location every time you
+commit changes to the data file or trigger a commit; it functions like a
+fancier git-lfs. It adds features like lineage for data and model
+artifacts, allowing you to recreate pipelines.
+
+Several techniques are associated with privacy-controlled data, like
+[federated
+learning](https://blog.ml.cmu.edu/2019/11/12/federated-learning-challenges-methods-and-future-directions/),
+differential privacy, and learning on encrypted data. These techniques
+are still in research, so they aren't quite ready for an FSDL
+recommendation.

documents/lecture-04.srt

@@ -0,0 +1,160 @@
+1
+00:00:00,080 --> 00:00:45,920
+hey everyone welcome to week four of full stack deep learning my name is sergey i have my assistant mishka right here there she is and today we're going to be talking about data management one of the things that people don't quite get as they enter the field of machine learning is just how much of it is actually just dealing with data putting together data sets looking at data munching data it's like half of the problem and it's more than half of the job for a lot of people but at the same time it's not something that people want to do the key points of this presentation are going to be that you should do a lot of it you should spend about 10 times as much time exploring the data as you would
+
+2
+00:00:44,879 --> 00:01:36,240
+like to and let it really just flow through you and usually the best way to improve performance of your model is going to be fixing your data set adding to the data set or maybe augmenting your data as you train and the last key point is keep it all simple you might be overwhelmed especially if you haven't been exposed to a lot of this stuff before there's a lot of words and terminology in different companies you don't have to do any of it and in fact you might benefit if you keep it as simple as possible that said we're going to be talking about this area of the ammo ops landscape and we'll start with the sources of data there's many possibilities for the sources of data you might have images you might have text files you might have
+
+3
+00:01:34,880 --> 00:02:22,319
+maybe logs database records but in deep learning you're going to have to get that data onto some kind of local file system disk right next to a gpu so you can send data and train and how exactly you're going to do that is different for every project different for every company so maybe you're training on images and you simply download the images that's all it's going to be from s3 or maybe you have a bunch of text that you need to process in some distributed way then analyze the data select the subset of it put that on the local machine or maybe you have a nice process with a data lake that ingests logs and database records and then from that you can aggregate and process it so that's always going to be different
+
+4
+00:02:20,480 --> 00:03:13,440
+but the basics are always going to be the same and they concern the file system object storage and databases so the file system is the fundamental abstraction and the fundamental unit of it is a file which can be a text file or a binary file it's not versioned and it can be easily overwritten or deleted and usually this is the file system is on a disk that's connected to your machine may be physically connected or maybe attached in the cloud or maybe it's even the distributed file system although that's less common now and we'll be talking about directly connected disks the first thing to know about disks is that the speed of them and the bandwidth of them is a quite quite a range from hard disks which are
+
+5
+00:03:11,200 --> 00:04:05,120
+usually spinning magnetic disks to solid-state disks which can be connected through the sata protocol or the nvme protocol and there's two orders of magnitude difference between the slowest which is like sata spinning disks and the fastest which are nvme solid state disks and making these slides i realized okay i'm showing you that but there's also some other latency numbers you should know so there's a famous document that you might have seen on the internet originally credited to jeff dean who i think credited peter norvig from google but i added human scale numbers in parens so here's how it's going to go so if you access the l1 l2 maybe even l3 cache of the cpu it's a very limited store of data but
+
+6
+00:04:03,599 --> 00:04:59,280
+it's incredibly fast it only takes a name a second to access and in human scale you might think of it as taking a second and then accessing ram is the next fastest thing and it's about 100 times slower but it's still incredibly fast and then that's just kind of finding something in ram but reading a whole megabyte sequentially from ram is now 250 microseconds which if the cache access took a second now it's taken two and a half days to read a megabyte from ram and if you're reading a megabyte from a sata connected ssd drive now you're talking about weeks so it's one and a half weeks and if you're reading a one one megabit of data from a spinning disk now we're talking about months and finally if you're sending a packet
+
+7
+00:04:57,120 --> 00:05:51,120
+of data from california across the ocean to europe and then back we're talking about years on a human scale in a 150 millisecond on the absolute scale and if gpu timing info i'd love to include it here so please just send it over to full stack so what format should data be stored on the local disk if it's binary data like images or audio just use the standard formats like jpegs or mp3 that it comes in they're already compressed you can't really do better than that for the metadata like labels or tabular data or text data compress json or text files just fine or parquet is a table format that's fast it's compressed by default as it's written and read that's compact and it's very widely used now let's talk about
+
+8
+00:05:48,960 --> 00:06:46,000
+object storage i think of it as an api over the file system where the fundamental unit is now an object and it's usually binary so it's maybe an image or a sound file but it could also be a text we can build in versioning or redundancy into the object storage service so instead of a file that can easily be overridden and isn't versioned we can say that an object whenever i update it it's actually just updating the version of it s3 is the fundame is the most common example and it's not as fast as local file system but it's fast enough especially if you're staying within the cloud databases are persistent fast and scalable storage and retrieval of structured data systems the metal model that i like to use is
+
+9
+00:06:45,039 --> 00:07:38,960
+that all the data that the database holds is actually in the ram of the computer but the database software ensures that if the computer gets turned off everything is safely persisted to disk and if it actually is too much data for ram it scales out to disk but still in a very performant way do not store binary data in the database you should store the object store urls to the binary data in the database instead postgres is the right choice it's an open source database and most of the time it's what you should use for example it supports unstructured json and queries over that unstructured json but sqlite is perfectly good for small projects it's a self-contained binary every language has an interface to it
+
+10
+00:07:35,919 --> 00:08:24,879
+even your browser has it and i want to stress that you should probably be using a database most coding projects like anything that deals with collections of objects that reference each other like maybe you're dealing with snippets of text that come from documents and documents of authors and maybe authors have companies or something like that this is very common and that code base will probably implement some kind of database and you can save yourself time and gain performance if you just use the database from the beginning and many mo ops tools specifically are at their core databases like weights and biases is a database of experiments hugging phase model hub is a database of models label studio which we'll talk about is a
+
+11
+00:08:22,960 --> 00:09:22,320
+database of labels plus obviously user interfaces for generating the labels and uploading the models and stuff like that but coming from an academic background i think it's important to fully appreciate databases data warehouses are stores for online analytical processing as opposed to databases which are data stores for online transaction processing and the difference i'll cover in a second but the way you get data into data warehouses is another acronym called etl extract transform load so maybe you have a number of data sources here it's like files database otp database and some sources in the cloud you'll extract data transform it into a uniform schema and then load it into the data warehouse and then from the
+
+12
+00:09:20,160 --> 00:10:15,839
+warehouse we can run business intelligence queries we know that it's archived and so what's the difference between olaps and otps like why are they different software platforms instead of just using postgres for everything so the difference is all laps for analytical processing are usually column oriented which lets you do queries what's the mean length of the text of comments over the last 30 days and it lets them be more compact because if you're storing the column you can compress that whole column in storage and oltps are usually row oriented and those are for queries select all the comments for this given user data lakes are unstructured aggregation of data from multiple sources so the main difference to data
+
+13
+00:10:12,399 --> 00:11:10,720
+warehouses is that instead of extract transform load its extract load into the lake and then transform later and the trend is unifying both so both unstructured and structured data should be able to live together the big two platforms for this our snowflake and databricks and if you're interested in this stuff this is a really great book that walks through the stuff from first principles that i think you will enjoy now that we have our data stored if we would like to explore it we have to speak the language of data and the language of data is mostly sql and increasingly it's also data frames sql is the standard interface for structured data it's existed for decades it's not going away it's worth
+
+14
+00:11:08,480 --> 00:12:01,200
+being able to at least read and it's well worth being able to write and for python pandas is the main data frame solution which basically lets you do sql-like things but in code without actually writing sql our advice is to become fluent in both this is how you interact with both transactional databases and analytical warehouses and lakes pandas is really the workhorse of python data science i'm sure you've seen it i just wanted to give you some tips if pandas are slow on something it's worth trying das data frames have the same interface but they paralyze operations over many cores and even over multiple machines if you set that up and something else that's worth trying if you have gpus available is rapids and
+
+15
+00:11:59,600 --> 00:12:55,040
+video rapids lets you do a subset of what pandas can do but on gpus so significantly faster for a lot of types of data so talking about data processing it's useful to have a motivational example so let's say we have to train a photo popularity predictor every night and for each photo training data must include maybe metadata about the photos such as the posting time the title that the user gave the location was taken maybe some features of the user and then maybe outputs of classifiers of the photo for content maybe style so the metadata is going to be in the database the features we might have to compute from logs and the photo classifications we're going to need to run those classifiers so we have dependencies our ultimate
+
+16
+00:12:52,959 --> 00:13:50,800
+task is to train the photopredictor model but to do we need to output data from database compute stuff from logs and run classifiers to output their predictions what we'd like is to define what we have to do and as things finish they should kick off their dependencies and everything should ideally not only have not only be files but programs and databases we should be able to spread this work over many machines and we're not the only ones running this job or this isn't the only job that's running on these machines how do we actually schedule multiple such jobs airflow is a pretty standard solution for python where it's possible to specify the acyclical graph of tasks using python code and the operators in that graph can be
+
+17
+00:13:48,320 --> 00:14:52,320
+sql operations or actually python functions and other plugins for airflow and to distribute these jobs the workflow manager has a queue has workers that report to it will restart jobs if they fail and will ping you when the jobs are done prefect is another is another solution that's been to improve over air flow it's more modern and dagster is another contender for the airflow replacement the main piece of advice here is don't over engineer this you can get machines with many cpu cores and a ton of ram nowadays and unix itself has powerful parallelism streaming tools that are highly optimized and this is a little bit of a contrived example from a decade ago but hadoop was all the rage in 2014 it was a distributed data processing
+
+18
+00:14:51,120 --> 00:15:43,279
+framework and so to run some kind of job that just aggregated a bunch of text files and computed some statistics over them the author spanned set up a hadoop job and it took 26 minutes to run but just writing a simple unix command that reads all the files grabs for the string sorts it and gives you the unique things was only 70 seconds and part of the reason is that this is all actually happening in parallel so it's making use of your cores pretty efficiently and you can make even more efficient use of them with the parallel command or here it's an argument to x-args and that's not to say that you should do everything just in unix but it is to say that just because the solution exists doesn't mean that it's right for you it
+
+19
+00:15:41,680 --> 00:16:39,120
+might be the case that you can just run your stuff in a single python script on your 32 core pc feature stores you might have heard about the situation that they deal with is all the data processing we we're doing is generating artifacts that we'll need for training time so how do we ensure that in production the model that was trained sees data where the same processing took place as it as as happened during training time and also when we retrain how do we avoid recomputing things that we don't need to recompute so feature store is our solution to this that you may not need the first mention i saw feature stores were was in this blog post from uber describing their machine learning platform michelangelo
+
+20
+00:16:36,560 --> 00:17:43,520
+and so they had offline training process and an online prediction process and they had feature stores for both that had to be in sync tecton is probably the leading sas solution to a feature storage for open source solutions feast is a common one and i recently came across feature form that looks pretty good as well so this is something you need check it out if it's not something you need don't feel like you have to use it in summary binary data like images sound files maybe compressed text store is object metadata about the data like labels or user activity with object should be stored in the database don't be afraid of sql but also know if you're using data frames there are accelerated solutions to them
+
+21
+00:17:41,200 --> 00:18:35,200
+if dealing with stuff like logs and other sources of data that are disparate it's worth setting up a data lake to aggregate all of it in one place you should have a repeatable process to aggregate the data you need for training which might involve stuff like airflow and depending on the expense and complexity of processing a feature store could be useful at training time the data that you need should be copied over to a file system on a really fast local drive and then you should optimize gpu transfer so what about specifically data sets for machine learning training hugging phase data sets is a great hub of data there's over 8 000 data sets revision nlp speech etc so i wanted to take a look at a few
+
+22
+00:18:33,360 --> 00:19:28,320
+example data sets here's one called github code it's over a terabyte of text 115 million code files the hugging face library the datasets library allows you to stream it so you don't have to download the terabyte of data in order to see some examples of it and the underlying format of the data is parquet tables so there's thousands of parquet tables each about half a gig that you can download piece by piece another example data set is called red caps pretty recently released 12 million image text pairs from reddit the images don't come with the data you need to download the images yourself make sure as you download it's multithreaded they give you example code and the underlying format then of the
+
+23
+00:19:26,080 --> 00:20:21,600
+database are the images you download plus json files that have the labels or the text that came with the images so the real foundational format of the data is just the json files and there's just urls in those files to the objects that you can then download here's another example data set common voice from wikipedia 14 000 hours of speech in 87 languages the format is mp3 files plus text files with the transcription of what the person's saying there's another interesting data set solution called active loop where you can also explore data stream data to your local machine and even transform data without saving it locally it look it has a pretty cool viewer of the data so here's looking at microsoft
+
+24
+00:20:18,159 --> 00:21:14,159
+coco computer vision data set and in order to get it onto your local machine it's a simple hub.load the next thing we should talk about is labeling and the first thing to talk about when it comes to labeling is maybe we don't have to label data self-supervised learning is a very important idea that you can use parts of your data to label other parts of your data so in natural language this is super common right now and we'll talk more about this in the foundational models lecture but given a sentence i can mask the last part of the sentence and to use the first part of the sentence to predict how it's going to end but i can also mask the middle of the sentence and use the whole sentence to predict the middle or i can even mask
+
+25
+00:21:12,640 --> 00:22:04,640
+the beginning of the sentence and use the completion of the sentence to predict the beginning in vision you can extract patches and then predict the relationship of the patches to each other and you can even do it across modalities so openai clip which we'll talk about in a couple of weeks is trained in this contrastive way where a number of images and the number of text captions are given to the model and the learning objective is to minimize the distance between the image and the text that it came with and to maximize the distance between the image and the other texts the and when i say between the image and the text the embedding of the image and the embedding of the texts and this led to great results this is
+
+26
+00:22:02,640 --> 00:22:56,960
+one of the best vision models for all kinds of tasks right now data augmentation is something that must be done for training vision models there's frameworks that provide including torch vision that provide you functions to do this it's changing the brightness of the data the contrast cropping it skewing it flipping it all kinds of transformations that basically don't change the meaning of the image but change the pixels of the image this is usually done in parallel to gpu training on the cpu and interestingly the augmentation can actually replace labels so there's a paper called simclear where the learning objective is to extract different views of an image and maximize the agreement or the similarity of the
+
+27
+00:22:55,679 --> 00:23:50,000
+embeddings of the views of the same image and minimize the agreement between the views of the different images so without labels and just with data augmentation and a clever learning objective they were able to learn a model that performs very well for even supervised tasks for non-vision data augmentation if you're dealing with tabular data you could delete some of the table cells to simulate what it would be like to have missing data for text i'm not aware of like really well established techniques but you could maybe delete words replace words with synonyms change the order of things and for speech you could change the speed of the file you could insert pauses you could remove some stuff you
+
+28
+00:23:47,039 --> 00:24:36,000
+can add audio effects like echo you can strip out certain frequency bands synthetic data is also something where the labels would basically be given to you for free because you use the label to generate the data so you know the label and it's still somewhat of an underrated idea that's often worth starting with we certainly do this in the lab but it can get really deep right so you can even use 3d rendering engines to generate very realistic vision data where you know exactly the label of everything in the image and this was done for receipts in this project that i link here you can also ask your users if you have users to label data for you i love how google photos does this they always ask me is this the same or different person
+
+29
+00:24:34,480 --> 00:25:29,919
+and this is sometimes called the data flywheel right where i'm incentivized to answer because it helps me experience the product but it helps google train their models as well because i'm constantly generating data but usually you might have to label some data as well and data labeling always has some standard set of features there's bounding boxes key points or part of speech tagging for text there's classes there's captions what's important is training the annotators so whoever will be doing the annotation make sure that they have a complete rulebook of how they should be doing it because there's reasonable ways to interpret the task so here's some examples like if i'm only seeing the head of the fox should i label only
+
+30
+00:25:27,360 --> 00:26:21,360
+the head or should i label the inferred location of the entire fox behind the rock it's unclear and quality assurance is something that's going to be key to annotation efforts because different people are just differently able to uh adhere to the rules where do you get people to annotate you can work with full-service data labeling companies you can hire your own annotators probably part-time and maybe promote the most the most able ones to quality control or you could potentially crowdsource this was popular in the past with mechanical turk the full service companies provide you the software stack the labor to do it and quality assurance and it probably makes sense to use them so how do you pick one you should at
+
+31
+00:26:18,480 --> 00:27:12,880
+first label some data yourself to make sure that you understand the task and you have a gold standard that you can evaluate companies on then you should probably take calls with several of the companies or just try them out if they let you try it out online get a work sample and then look at how the work sample agrees with your own gold standard and then see how the price of the annotation compares scale dot ai is probably the dominant data labeling solution today and they take an api approach to this where it's you create tasks for them and then receive results and there are many other annotations like label box supervisedly and there's just a million more label studio is an open source solution that you can run yourself
+
+32
+00:27:11,440 --> 00:28:05,600
+there's an enterprise edition for managed hosting but there's an open source edition that you can just run in the docker container on your own machine we're going to use it in the lab and it has a lot of different interfaces for text images you can create your own interfaces you can even plug in models and do active learning for annotation diff gram is something i've come across but i haven't used it personally they claim to be better than label studio and it looks pretty good an interesting feature that that i've seen some software offerings have is evaluate your current model on your data and then explore how it performed such that you can easily select subsets of data for further labeling or potentially
+
+33
+00:28:03,360 --> 00:28:56,320
+find mistakes in your labeling and just understand how your model is performing on the data there's aquarium learning and scale nucleus are both solutions to this that you can check out snorkel you might have heard about and it's using this idea of weak supervision where if you have a lot of data to label some of it is probably really easy to label if you're labeling sentiment of text and if they're using the word wonderful then it's probably positive so if you can create a rule that says if the text contains the word wonderful just apply the positive label to it and you create a number of these labeling functions and then the software intelligently composes them and it could be a really fast way to to
+
+34
+00:28:54,159 --> 00:29:44,640
+go through a bunch of data there's the open source project of snorkel and there's the commercial platform and i recently came across rubrics which is a very similar idea that's fully open source so in conclusion for labeling first think about how you can do self-supervised learning and avoid labeling if you need to label which you probably will need to do use labeling software and really get to know your data by labeling it yourself for a while after you've done that you can write out detailed rules and then outsource to a full service company otherwise if you don't want to outsource you can't afford it you should probably hire some part-time contractors and not try to crowdsource because crowdsourcing is a lot of
+
+35
+00:29:42,480 --> 00:30:35,120
+quality assurance overhead it's a lot better to just find a good person who can trust to do the job and just have them label lastly in today's lecture we can talk about versioning i like to think of data versioning as a spectrum where the level zero is unversioned and level three is specialized data versioning solution so label level one level zero is bad okay where you have data that just lives on the file system or is on s3 or in a database and it's not version so you train a model you deploy the model and the problem is when you deploy the model what you're deploying is partly the code but partly the data that generated the weights right and if the data is not versioned then your model is in effect not
+
+36
+00:30:33,039 --> 00:31:21,679
+versioned and so what will probably happen is that your performance will degrade at some point and you won't be able to get back to a previous level of high performance so you can solve this with level one each time you train you just take a snapshot of your data and you store it somewhere so this kind of works because you'll be able to get back to that performance by retraining but it'd be nicer if i could just version the data as easily as code not through some separate process and that's where we arrive at level two where we just we version data exactly in the same way as reversion code so let's say we're having a data set of audio files and text transcriptions so we're going to upload the audio files
+
+37
+00:31:19,679 --> 00:32:12,559
+to s3 that's probably where they were to begin with and the labels for the files we can just store in a parquet file or a json file where it's going to be the s3 url and the transcription of it now even this metadata file can get pretty big it's a lot of text but you can use git lfs which stands for large file storage and we can just add them and the git add will version the data file exactly the same as your version your code file and this can totally work you do not need to definitely go to level three would be using a specialized solution for versioning data and this usually helps you store large files directly and it could totally make sense but just don't assume that you need it right away if you can get away with just
+
+38
+00:32:11,039 --> 00:33:08,240
+get lfs that would be the fstl recommendation if it's starting to break then the leading solution for level three versioning is dvc and there's a table comparing the different versioning solutions like pachyderm but this table is biased towards dvc because it's by a solution that's github for dbc called dags hub and the way dvc works is you set it up you add your data file and then the most basic thing it does is it can upload to s3 or google cloud storage or whatever some other network storage whatever you set up every time you commit it'll upload your data somewhere and it'll make sure it's versioned so it's like a replacement for git lfs but you can go further and you can also record the lineage of
+
+39
+00:33:06,000 --> 00:34:05,519
+the data so how exactly was this data generated how does this model artifact get generated so you can use dvc run to mark that and then use dvc to recreate the pipelines the last thing i want to say is we get a lot of questions at fstl about privacy sensitive data and this is still a research area there's no kind of off-the-shelf solution we can really recommend federated learning is a research area that refers to training a global model from data on local devices without the the model training process having access to the local data so it's there's a federated server that has the model and it sends what to do to local models and then it syncs back the models and differential privacy is another term
+
+40
+00:34:02,640 --> 00:34:52,200
+this is for aggregating data such that even though you have the data it's aggregated in such a way that you can't identify the individual points so it should be safe to train on sensitive data because you won't actually be able to understand the individual points of it and another topic that is in the same vein is learning on encrypted data so can i have data that's encrypted that i can't decrypt but can i still do machine learning on it in a way that generates useful models and these three things are all research areas and i'm not aware of like really good off-the-shelf solutions for them unfortunately that concludes our lecture on data management thank you
+

documents/lecture-05.md

@@ -0,0 +1,788 @@
+---
+description: How to turn an ML model into an ML-powered product
+---
+
+# Lecture 5: Deployment
+
+<div align="center">
+<iframe width="720" height="405" src="https://www.youtube-nocookie.com/embed/W3hKjXg7fXM?list=PL1T8fO7ArWleMMI8KPJ_5D5XSlovTW_Ur" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+Lecture by [Josh Tobin](https://twitter.com/josh_tobin_).<br />
+Notes by [James Le](https://twitter.com/le_james94) and [Vishnu Rachakonda](https://www.linkedin.com/in/vrachakonda/).<br />
+Published September 5, 2022.
+[Download slides](https://fsdl.me/2022-lecture-05-slides).
+
+## Introduction
+
+![](./media/image21.png)
+
+Deploying models is a critical part of making your models good, to begin
+with. When you only evaluate the model offline, it's easy to miss the
+more subtle flaws that the model has, where it doesn't actually solve
+the problem that your users need it to solve. Oftentimes, when we deploy
+a model for the first time, only then do we really see whether that
+model is actually doing a good job or not. Unfortunately, for many data
+scientists and ML engineers, model deployment is an afterthought
+relative to other techniques we have covered.
+
+Much like other parts of the ML lifecycle, we'll focus on deploying a
+minimum viable model as early as possible, which entails **keeping it
+simple and adding complexity later**. Here is the process that this
+lecture covers:
+
+-   Build a prototype
+
+-   Separate your model and UI
+
+-   Learn the tricks to scale
+
+-   Consider moving your model to the edge when you really need to go
+fast
+
+## 1 - Build a Prototype To Interact With
+
+There are many great tools for building model prototypes.
+[HuggingFace](https://huggingface.co/) has some tools
+built into its playground. They have also recently acquired a startup
+called [Gradio](https://gradio.app/), which makes it easy
+to wrap a small UI around the model.
+[Streamlit](https://streamlit.io/) is another good option
+with a bit more flexibility.
+
+![](./media/image19.png)
+
+
+Here are some best practices for prototype deployment:
+
+1.  **Have a basic UI**: The goal at this stage is to play around with
+the model and collect feedback from other folks. Gradio and
+Streamlit are your friends here - often as easy as adding a couple
+of lines of code to create a simple interface for the model.
+
+2.  **Put it behind a web URL**: An URL is easier to share. Furthermore,
+you will start thinking about the tradeoffs you'll be making when
+dealing with more complex deployment schemes. There are cloud
+versions of [Streamlit](https://streamlit.io/cloud)
+and [HuggingFace](https://huggingface.co/) for this.
+
+3.  **Do not stress it too much**: You should not take more than a day
+to build a prototype.
+
+A model prototype won't be your end solution to deploy. Firstly, a
+prototype has limited frontend flexibility, so eventually, you want to
+be able to build a fully custom UI for the model. Secondly, a prototype
+does not scale to many concurrent requests. Once you start having users,
+you'll hit the scaling limits quickly.
+
+![](./media/image18.png)
+
+
+Above is an abstract diagram of how your application might look. The
+**client** is your user's device that interacts with your application.
+This device can be a browser, a vehicle, or a mobile phone. This device
+calls over a network to a **server**. The server talks to a **database**
+(where data is stored), used to power the application.
+
+![](./media/image6.png)
+
+
+There are different ways of structuring your application to fit an ML
+model inside. The prototype approach mentioned in the beginning fits
+into the **model-in-service** approach - where your hosted web server
+has a packaged version of the model sitting inside it. This pattern has
+pros and cons.
+
+The biggest pro is that if you are doing something complex, you get to
+reuse your existing infrastructure. It does not require you as a model
+developer to set up new things from scratch.
+
+However, there is a number of pronounced cons:
+
+1.  **Your web server may be written in a different language**, so
+getting your model into that language can be difficult.
+
+2.  **Models may change more frequently than server code** (especially
+early in the lifecycle of building your model). If you have a
+well-established application and a nascent model, you do not want
+to redeploy the entire application every time that you make an
+update to the model (sometimes multiple updates per day).
+
+3.  If you have a large model to run inference on, you'll have to load
+that model on your web server. **Large models can eat into the
+resources for your web server**. That might affect the user
+experience for people using that web server, even if they are not
+interacting with the model.
+
+4.  **Server hardware is generally not optimized for ML workloads**. In
+particular, you rarely will have a GPU on these devices.
+
+5.  **Your model and application may have different scaling
+properties**, so you might want to be able to scale them
+differently.
+
+## 2 - Separate Your Model From Your UI
+
+### 2.1 - Batch Prediction
+
+![](./media/image8.png)
+
+
+The first pattern to pull your model from your UI is called **batch
+prediction**. You get new data in and run your model on each data point.
+Then, you save the results of each model inference into a database. This
+can work well under some circumstances. For example, if there are not a
+lot of potential inputs to the model, you can re-run your model on some
+frequency (every hour, every day, or every week). You can have
+reasonably fresh predictions to return to those users that are stored in
+your database. Examples of these problems include the early stages of
+building recommender systems and internal-facing tools like marketing
+automation.
+
+To run models on a schedule, you can leverage the data processing and
+workflow tools mentioned in our previous lecture on data management. You
+need to re-run data processing, load the model, run predictions, and
+store those predictions in your database. This is exactly a **Directed
+Acyclic Graph workflow of data operations** that tools like
+[Dagster](https://dagster.io/),
+[Airflow](https://airflow.apache.org/), or
+[Prefect](https://www.prefect.io/) are designed to solve.
+It's worth noting that there are also tools like
+[Metaflow](https://metaflow.org/) that are designed more
+for ML or data science use cases that might be potentially even an
+easier way to get started.
+
+Let's visit the pros and cons of this batch prediction pattern. Starting
+with the pros:
+
+1.  Batch prediction is **simple to implement** since it reuses existing
+batch processing tools that you may already be using for training
+your model.
+
+2.  It **scales very easily** because databases have been engineered for
+decades for such a purpose.
+
+3.  Even though it looks like a simple pattern, it has been **used in
+production by large-scale production systems for years**. This is
+a tried-and-true pattern you can run and be confident that it'll
+work well.
+
+4.  It is **fast to retrieve the prediction** since the database is
+designed for the end application to interact with.
+
+Switching to the cons:
+
+1.  Batch prediction **doesn't scale to complex input types**. For
+instance, if the universe of inputs is too large to enumerate
+every single time you need to update your predictions, this won't
+work.
+
+2.  **Users won't be getting the most up-to-date predictions from your
+model**. If the feature that goes into your model changes every
+hour, minute, or subsecond, but you only run your batch prediction
+job every day, the predictions your users see might be slightly
+stale.
+
+3.  **Models frequently become "stale."** If your batch jobs fail for
+some reason, it can be hard to detect these problems.
+
+### 2.2 - Model-as-Service
+
+The second pattern is called **model-as-service**: we run the model
+online as its own service. The service is going to interact with the
+backend or the client itself by making requests to the model service and
+receiving responses back.
+
+![](./media/image16.png)
+
+
+The pros of this pattern are:
+
+1.  **Dependability** - model bugs are less likely to crash the web
+application.
+
+2.  **Scalability** - you can choose optimal hardware for the model and
+scale it appropriately.
+
+3.  **Flexibility** - you can easily reuse a model across multiple
+applications.
+
+The cons of this pattern are:
+
+1.  Since this is a separate service, you add a network call when your
+server or client interacts with the model. That can **add
+latency** to your application.
+
+2.  It also **adds infrastructural complexity** because you are on the
+hook for hosting and managing a separate service.
+
+Even with these cons, **the model-as-service pattern is still a sweet
+spot for most ML-powered products** since you really need to be able to
+scale independently of the application in most complex use cases. We'll
+walk through the basic components of building your model service -
+including REST APIs, dependency management, performance optimization,
+horizontal scaling, rollout, and managed options.
+
+#### REST APIs
+
+**Rest APIs** serve predictions in response to canonically-formatted
+HTTP requests. There are other alternative protocols to interact with a
+service that you host on your infrastructures, such as
+[GRPC](https://grpc.io/) (used in TensorFlow Serving) and
+[GraphQL](https://graphql.org/) (common in web development
+but not terribly relevant to model services).
+
+![](./media/image3.png)
+
+
+Unfortunately, there is currently no standard for formatting requests
+and responses for REST API calls.
+
+1.  [Google Cloud](https://cloud.google.com/) expects a
+batch of inputs structured as a list called "instances" (with keys
+and values).
+
+2.  [Azure](https://azure.microsoft.com/en-us/) expects a
+list of things called "data", where the data structure itself
+depends on what your model architecture is.
+
+3.  [AWS Sagemaker](https://aws.amazon.com/sagemaker/)
+expects instances that are formatted differently than they are in
+Google Cloud.
+
+Our aspiration for the future is to move toward **a standard interface
+for making REST API calls for ML services**. Since the types of data
+that you might send to these services are constrained, we should be able
+to develop a standard as an industry.
+
+#### Dependency Management
+
+Model predictions depend on **code**, **model weights**, and
+**dependencies**. In order for your model to make a correct prediction,
+all of these dependencies need to be present on your web server.
+Unfortunately, dependencies are a notorious cause of trouble as it is
+hard to ensure consistency between your development environment and your
+server. It is also hard to update since even changing a TensorFlow
+version can change your model.
+
+At a high level, there are two strategies for managing dependencies:
+
+1.  **Constrain the dependencies for your model** by saving your model
+in an agnostic format that can be run anywhere.
+
+2.  **Use containers** to constrain the entire inference program.
+
+![](./media/image11.png)
+
+
+##### Constraining Model Dependencies
+
+The primary way to constrain the dependencies of just your model is
+through a library called [ONNX](https://onnx.ai/) - the
+Open Neural Network Exchange. The goal of ONNX is to be **an
+interoperability standard for ML models**. The promise is that you can
+define a neural network in any language and run it consistently
+anywhere. The reality is that since the underlying libraries used to
+build these models change quickly, there are often bugs in the
+translation layer, which creates even more problems to solve for you.
+Additionally, ONNX doesn't deal with non-library code such as feature
+transformations.
+
+##### Containers
+
+To understand how to manage dependencies with containers, we need to
+understand [the differences between Docker and Virtual
+Machines](https://medium.freecodecamp.org/a-beginner-friendly-introduction-to-containers-vms-and-docker-79a9e3e119b),
+how Docker images are built via Docker files and constructed via layers,
+the ecosystem around Docker, and specific wrappers around Docker that
+you can use for ML.
+
+![](./media/image10.png)
+
+
+In a **virtual machine**, you package up the entire operating system
+(OS) as well as the libraries and applications that are built on top of
+that OS. A virtual machine tends to be very heavyweight because the OS
+itself has a lot of code and is expensive to run. A **container** such
+as Docker removes that need by packaging the libraries and applications
+together. A Docker engine that runs on top of your OS knows how to
+virtualize the OS and run the libraries/applications.
+
+By virtue of being **lightweight**, Docker is used differently than how
+Virtual Machines were used. A common pattern is to spin up [a new
+Docker container](https://www.docker.com/what-container)
+for every discrete task. For example, a web application might have four
+containers: a web server, a database, a job queue, and a worker. These
+containers are run together as part of an orchestration system.
+
+![](./media/image15.png)
+
+
+Docker containers are created from [Docker
+files](https://docs.docker.com/engine/reference/builder/).
+Each Docker file runs a sequence of steps to define the environment
+where you will run your code. Docker also allows you to build, store,
+and pull Docker containers from a Docker Hub that is hosted on some
+other servers or your cloud. You can experiment with a code environment
+that is on your local machine but will be identical to the environment
+you deploy on your server.
+
+Docker is separated into [three different
+components](https://docs.docker.com/engine/docker-overview):
+
+1.  The **client** is where you'll be running on your laptop to build an
+image from a Dockerfile that you define locally using some
+commands.
+
+2.  These commands are executed by a **Docker Host**, which can run on
+either your laptop or your server (with more storage or more
+performance).
+
+3.  That Docker Host talks to a **registry** - which is where all the
+containers you might want to access are stored.
+
+![](./media/image1.png)
+
+
+With this separation of concerns, you are not limited by the amount of
+compute and storage you have on your laptop to build, pull, and run
+Docker images. You are also not limited by what you have access to on
+your Docker Host to decide which images to run.
+
+In fact, there is a powerful ecosystem of Docker images that are
+available on different public Docker Hubs. You can easily find these
+images, modify them, and contribute them back to the Hubs. It's easy to
+store private images in the same place as well. Because of this
+community and the lightweight nature of Docker, it has become
+[incredibly popular in recent
+years](https://www.docker.com/what-container#/package_software)
+and is ubiquitous at this point.
+
+There is a bit of a learning curve to Docker. For ML, there are a few
+open-source packages designed to simplify this:
+[Cog](https://github.com/replicate/cog),
+[BentoML](https://github.com/bentoml/BentoML), and
+[Truss](https://github.com/trussworks). They are built by
+different model hosting providers that are designed to work well with
+their model hosting service but also just package your model and all of
+its dependencies in a standard Docker container format.
+
+![](./media/image12.png)
+
+These packages have **two primary components**: The first one is a
+standard way of defining your prediction service. The second one is a
+YAML file that defines the other dependencies and package versions that
+will go into the Docker container running on your laptop or remotely.
+
+If you want to have the advantages of using Docker for making your ML
+models reproducible but do not want to go through the learning curve of
+learning Docker, it's worth checking out these three libraries.
+
+#### Performance Optimization
+
+!!! info "What about performance _monitoring_?"
+    In this section, we focus on ways to improve the performance of your
+    models, but we spend less time on how exactly that performance is monitored,
+    which is a challenge in its own right.
+
+    Luckily, one of the
+    [student projects](../project-showcase/) for the 2022 cohort,
+    [Full Stack Stable Diffusion](../project-showcase/#full-stack-stable-diffusion),
+    took up that challenge and combined
+    [NVIDIA's Triton Inference Server](https://developer.nvidia.com/nvidia-triton-inference-server),
+    the [Prometheus monitoring tool](https://en.wikipedia.org/wiki/Prometheus_(software)),
+    and
+    the [Grafana analytics dashboarding tool](https://en.wikipedia.org/wiki/Grafana)
+    to monitor a robust, scalable, and observable deployment of Stable Diffusion models.
+
+    Check out the repo on GitHub
+    [here](https://github.com/okanlv/fsdl-full-stack-stable-diffusion-2022)
+    if you want to see a worked example of a fully-monitored DL-powered application.
+
+To make model inference on your machine more efficient, we need to
+discuss GPU, concurrency, model distillation, quantization, caching,
+batching, sharing the GPU, and libraries that automate these tasks for
+you.
+
+##### GPU or no GPU?
+
+There are some advantages to hosting your model on a GPU:
+
+1.  It's probably the same hardware you train your model on, to begin
+with. That can eliminate any lost-in-translation issues.
+
+2.  As your model gets big and your techniques get advanced, your
+traffic gets large. GPUs provide high throughput to deal with
+that.
+
+However, GPUs introduce a lot of complexity:
+
+1.  They are more complex to set up.
+
+2.  They are more expensive.
+
+As a result, **just because your model is trained on a GPU does not mean
+that you need to actually host it on a GPU in order for it to work**. In
+the early version of your model, hosting it on a CPU should suffice. In
+fact, it's possible to get high throughput from CPU inference at a low
+cost by using some other techniques.
+
+##### Concurrency
+
+With **concurrency**, multiple copies of the model run in parallel on
+different CPUs or cores on a single host machine. To do this, you need
+to be careful about thread tuning. There's [a great Roblox
+presentation](https://www.youtube.com/watch?v=Nw77sEAn_Js)
+on how they scaled BERT to serve a billion daily requests, just using
+CPUs.
+
+##### Model Distillation
+
+With **model distillation**, once you have a large model that you've
+trained, you can train a smaller model that imitates the behavior of
+your larger one. This entails taking the knowledge that your larger
+model learned and compressing that knowledge into a much smaller model
+that you may not have trained to the same degree of performance from
+scratch. There are several model distillation techniques pointed out in
+[this blog
+post](https://heartbeat.comet.ml/research-guide-model-distillation-techniques-for-deep-learning-4a100801c0eb).
+They can be finicky to do by yourself and are infrequently used in
+practice. An exception is distilled versions of popular models (such as
+[DistilBERT](https://huggingface.co/docs/transformers/model_doc/distilbert)).
+
+##### Quantization
+
+With **quantization**, you execute some or potentially all of the
+operations in your model in a lower fidelity representation of the
+numbers that you are doing the math. These representations can be 16-bit
+floating point numbers or 8-bit integers. This introduces some tradeoffs
+with accuracy, but it's worth making these tradeoffs because the
+accuracy you lose is limited relative to the performance you gain.
+
+The recommended path is to use built-in quantization methods in
+[PyTorch](https://pytorch.org/blog/introduction-to-quantization-on-pytorch/)
+and TensorFlow. More specifically, [HuggingFace
+Optimum](https://huggingface.co/docs/optimum) is a good
+choice if you have already been using HuggingFace's pre-trained models.
+You can also run **quantization-aware training**, which often results in
+higher accuracy.
+
+![](./media/image5.png)
+
+
+##### Caching
+
+With **caching**, you realize that for some ML models, some inputs are
+more common than others. Instead of always calling the model every time
+a user makes a request, let's store the common requests in a cache.
+Then, let's check that cache before running an expensive operation.
+Caching techniques can get fancy, but the basic way of doing this is to
+use [functools library in
+Python](https://docs.python.org/3/library/functools.html).
+
+![](./media/image2.png)
+
+
+##### Batching
+
+With **batching**, you take advantage of the fact that ML models often
+achieve a higher throughput when doing prediction in parallel,
+especially in a GPU. To accomplish this, you need to gather predictions
+until you have a batch, run those predictions, and return them to your
+user. You want to tune the batch size that deals optimally with the
+latency-throughput tradeoff. You also need to have a way to shortcut the
+process if latency becomes too long. Batching is complicated to
+implement, so you probably do not want to implement this yourself.
+
+##### Sharing the GPU
+
+Your model may not take up all of the GPU memory with your inference
+batch size. **Why don't you run multiple models on the same GPU?** This
+is a place where you want to use a model serving solution that supports
+GPU sharing out of the box.
+
+##### Libraries
+
+There are offerings from TensorFlow, PyTorch, and third-party tools from
+NVIDIA and Anyscale. NVIDIA's choice is probably the most powerful but
+can be difficult to get started with. Starting with Anyscale's [Ray
+Serve](https://docs.ray.io/en/latest/serve/index.html) may
+be an easier way to get started.
+
+![](./media/image20.png)
+
+
+#### Horizontal Scaling
+
+If you're going to scale up to a large number of users interacting with
+your model, it's not going to be enough to get the most efficiency out
+of one server. At some point, you'll need to scale horizontally to have
+traffic going to multiple copies of your model running on different
+servers. This is called **horizontal scaling**. This technique involves
+taking traffic that would usually go to a single machine and splits
+across multiple machines.
+
+Each machine has a copy of the service, and a tool called a load
+balancer distributes traffic to each machine. In practice, there are two
+ways to do this: with either **container orchestration** (e.g.
+Kubernetes) or **serverless** (e.g. AWS Lambda).
+
+##### Container Orchestration
+
+In container orchestration, we use
+[Kubernetes](https://kubernetes.io/) to help manage
+containerized applications (in Docker containers, for example) and run
+them across machines.
+
+![](./media/image14.png)
+
+
+Kubernetes is quite interesting, but it's probably overkilled to learn
+too much about it if your only goal is to deploy machine learning
+models. There are a number of frameworks that make it easiest to deploy
+ML models with Kubernetes, including
+[Kubeflow](https://www.kubeflow.org/),
+[Seldon](https://www.seldon.io/), etc.
+
+##### Serverless
+
+If Kubernetes isn't the path for you (e.g. you don't want to have to
+worry about infrastructure at all), serverless is another option for
+deploying models. In this paradigm, app code and dependencies are
+packaged into .zip files or Docker containers with a single entry point
+function, which is a single function (e.g. *model.predict()*) that will
+be run repeatedly. This package is then deployed to a service like [AWS
+Lambda](https://aws.amazon.com/lambda/), which almost
+totally manages the infrastructure required to run the code based on the
+input. Scaling to thousands of requests and across multiple machines is
+taken care of by these services. In return, you pay for the compute time
+that you consume.
+
+Since model services tend to run discretely and not continuously (like a
+web server), serverless is a great fit for machine learning deployment.
+
+![](./media/image7.png)
+
+
+**Start with serverless!** It's well worth the time saved in managing
+infrastructure and dealing with associated challenges. There are still
+some problems you should be aware of though.
+
+1.  First, the size of the actual deployment package that can be sent to
+a serverless service tends to be limited, which makes large models
+impossible to run.
+
+2.  Second, there is also a cold start problem. If there is no traffic
+being sent to the service in question, the service will "wind
+down" to zero compute use, at which point it takes time to start
+again. This lag in starting up upon the first request to the
+serverless service is known as the "cold start" time. This can
+take seconds or even minutes.
+
+3.  Third, it can be hard to actually build solid software engineering
+concepts, like pipelines, with serverless. Pipelines enable rapid
+iteration, while serverless offerings often do not have the tools
+to support rapid, automated changes to code of the kind pipelines
+are designed to do.
+
+4.  Fourth, state management and deployment tooling are related
+challenges here.
+
+5.  Finally, most serverless functions are CPU only and have limited
+execution time. If you need GPUs for inference, serverless might
+not be for you quite yet. There are, however, new offerings like
+[Banana](https://www.banana.dev/) and
+[Pipeline](https://www.pipeline.ai/) that are
+seeking to solve this problem of serverless GPU inference!
+
+#### Model Rollouts
+
+If serving is how you turn a model into something that can respond to
+requests, rollouts are how you manage and update these services. To be
+able to make updates effectively, you should be able to do the
+following:
+
+1.  **Roll out gradually**: You may want to incrementally send traffic
+to a new model rather than the entirety.
+
+2.  **Roll back instantly**: You may want to immediately pull back a
+model that is performing poorly.
+
+3.  **Split traffic between versions**: You may want to test differences
+between models and therefore send some traffic to each.
+
+4.  **Deploy pipelines of models**: Finally, you may want to have entire
+pipeline flows that ensure the delivery of a model
+
+Building these capabilities in a reasonably challenging infrastructure
+problem that is beyond the scope of this course. In short, managed
+services are a good option for this that we'll now discuss!
+
+#### Managed Options
+
+All of the major cloud providers offer their managed service options for
+model deployment. There are a number of startups offering solutions as
+well, like BentoML or Banana.
+
+![](./media/image9.png)
+
+The most popular managed service is [AWS
+Sagemaker](https://aws.amazon.com/sagemaker/). Working with
+Sagemaker is easier if your model is already in a common format like a
+Huggingface class or a SciKit-Learn model. Sagemaker has convenient
+wrappers for such scenarios. Sagemaker once had a reputation for being a
+difficult service to work with, but this is much less the case for the
+clear-cut use case of model inference. Sagemaker, however, does have
+real drawbacks around ease of use for custom models and around cost. In
+fact, Sagemaker instances tend to be 50-100% more expensive than EC2.
+
+### 2.3 - Takeaways
+
+To summarize this section, remember the following:
+
+1.  You *probably* don't need GPU inference, which is hard to access and
+maintain. Scaling CPUs horizontally or using serverless can
+compensate.
+
+2.  Serverless is probably the way to go!
+
+3.  Sagemaker is a great way to get started for the AWS user, but it can
+get quite expensive.
+
+4.  Don't try to do your own GPU inference; use existing tools like
+TFServing or Triton to save time.
+
+5.  Watch out for new startups focused on GPU inference.
+
+## 3 - Move to the Edge?
+
+Let's now consider the case of moving models out of web service and all
+the way to the "edge", or wholly on-device. Some reasons you may need to
+consider this include a lack of reliable internet access for users or
+strict data security requirements.
+
+If such hard and fast requirements aren't in place, you'll need to take
+into account the tradeoff between accuracy and latency and how this can
+affect the end-user experience. Put simply, **if you have exhausted all
+options to reduce model prediction time (a component of latency),
+consider edge deployment**.
+
+![](./media/image4.png)
+
+
+Edge deployment adds considerable complexity, so it should be considered
+carefully before being selected as an option. In edge prediction, model
+weights are directly loaded on our client device after being sent via a
+server (shown above), and the model is loaded and interacted with
+directly on the device.
+
+This approach has compelling pros and cons:
+
+1.  Some pros to particularly call out are the latency advantages that
+come without the need for a network and the ability to scale for
+"free," or the simple fact that you don't need to worry about the
+challenges of running a web service if all inference is done
+locally.
+
+2.  Some specific cons to call out are the often limited hardware and
+software resources available to run machine learning models on
+edge, as well as the challenge of updating models since users
+control this process more than you do as the model author.
+
+### 3.1 - Frameworks
+
+Picking the right framework to do edge deployment depends both on how
+you train your model and what the target device you want to deploy it on
+is.
+
+-   [TensorRT](https://developer.nvidia.com/tensorrt): If
+you're deploying to NVIDIA, this is the choice to go with.
+
+-   [MLKit](https://developers.google.com/ml-kit) and
+[CoreML](https://developer.apple.com/documentation/coreml)**:**
+For phone-based deployment on either Android **or** iPhone, go
+with MLKit for the former and CoreML for the latter.
+
+-   [PyTorch Mobile](https://pytorch.org/mobile)**:** For
+compatibility with both iOS and Android, use PyTorch Mobile.
+
+-   [TFLite](https://www.tensorflow.org/lite): A great
+choice for using TensorFlow in a variety of settings, not just on
+a phone or a common device.
+
+-   [TensorFlow JS](https://www.tensorflow.org/js)**:**
+The preferred framework for deploying machine learning in the
+browser.
+
+-   [Apache TVM](https://tvm.apache.org/): A library
+agnostic, target device agnostic option. This is the choice for
+anyone trying to deploy to as diverse a number of settings as
+possible.
+
+Keep paying attention to this space! There are a lot of startups like
+[MLIR](https://mlir.llvm.org/),
+[OctoML](https://octoml.ai/),
+[TinyML](https://www.tinyml.org/), and
+[Modular](https://www.modular.com/) that are aiming to
+solve some of these problems.
+
+### 3.2 - Efficiency
+
+No software can help run edge-deployed models that are simply too large;
+**model efficiency** is important for edge deployment! We previously
+discussed quantization and distillation as options for model efficiency.
+However, there are also network architectures specifically designed to
+work better in edge settings like
+[MobileNets](https://medium.com/@yu4u/why-mobilenet-and-its-variants-e-g-shufflenet-are-fast-1c7048b9618d).
+MobileNets replace the more expensive computations typical of server-run
+models with simpler computations and achieve acceptable performance
+oftentimes.
+
+![](./media/image17.png)
+
+
+MobileNets are a great tool for model deployments and are a great case
+study in model efficiency. Another similarly great case study is
+[DistillBERT](https://medium.com/huggingface/distilbert-8cf3380435b5).
+
+![](./media/image13.png)
+
+### 3.3 - Mindsets
+
+As we wrap up this lecture, keep in mind the following mindsets as you
+consider edge deployment:
+
+1.  **Start with the edge requirement, not the architecture choice**.
+It's easy to pick a high-performing model architecture, only to
+then find it impossible to run on the edge device. Avoid this
+scenario at all costs! Tricks like quantization can account for up
+to 10x improvement, but not much more.
+
+2.  **Once you have a model that works on the edge, you can iterate
+locally without too much additional re-deployment.** In this case,
+make sure to add metrics around the model size and edge
+performance to your experiment tracking.
+
+3.  **Treat tuning the model as an additional risk and test
+accordingly.** With the immaturity of edge deployment frameworks,
+it's crucial to be especially careful when testing your model on
+the exact hardware you'll be deploying on.
+
+4.  **Make sure to have fallbacks!** Models are finicky and prone to
+unpredictable behavior. In edge cases, it's especially important
+to have easily available fallback options for models that aren't
+working.
+
+### 3.4 - Conclusion
+
+To summarize this section:
+
+1.  Web deployment is easier, so use edge deployment only if you need
+to.
+
+2.  Choose your framework to match the available hardware and
+corresponding mobile frameworks, or try Apache TVM to be more
+flexible.
+
+3.  Start considering hardware constraints at the beginning of the
+project and choose architectures accordingly.

documents/lecture-05.srt

@@ -0,0 +1,396 @@
+1
+00:00:00,240 --> 00:00:32,000
+hey everybody welcome back this week we're going to talk about deploying models into production so we're talking about this part of the life cycle and why do we spend a whole week on this maybe the answer is obvious right which is if you want to build a machine learning powered product you need some way of getting your model into production but i think there's a more subtle reason as well which is that i think of deploying models as a really critical part of making your models good to begin with the reason for that is when you only evaluate your model offline it's really easy to miss some of the more subtle flaws that model has where it doesn't actually solve the problem that your users needed to solve
+
+2
+00:00:30,320 --> 00:01:07,040
+oftentimes when we deploy a model for the first time only then do we really see whether that model is actually doing a good job or not but unfortunately for a lot of data scientists and ml engineers model deployment is kind of an afterthought relative to some of the other techniques that you've learned and so the goal of this lecture is to cover different ways of deploying models into production and we're not going to be able to go in depth in all of them because it's it's a broad and deep topic worthy probably of a course itself and i'm not personally an expert in it but what we will do is we'll cover like a couple of happy paths that will take you to getting your first model in production for most use cases and then
+
+3
+00:01:05,680 --> 00:01:41,119
+we'll give you a tour of some of the other techniques that you might need to learn about if you want to do something that is outside of that normal 80 so to summarize it's really important to get your model into production because only there do you see if it actually works if it actually solves the task that you set out to solve the technique that we're going to emphasize that you use for this is much like what we use in other parts of the life cycle and it's focused on like getting an mvp out early deploy early deploy a minimum viable model as early as possible and deploy often we're also going to emphasize keeping it simple and adding to bluxy later and so we'll start we'll walk through this the following process starting with building
+
+4
+00:01:39,280 --> 00:02:10,959
+a prototype then we'll talk about how to separate your model in your ui which is sort of one of the first things that you'll need to do to make a more complex ui or to scale then we'll talk about some of the tricks that you need to do in order to scale your model up to serve many users and then finally we'll talk about more advanced techniques that you might use when you need your model to be really fast which often means moving it from a web server to the edge so the first thing that we'll talk about is how to build the first prototype of your production model and the goal here is just something that you can play around with yourself and share with your friends luckily unlike when we first taught this class there's many great
+
+5
+00:02:08,879 --> 00:02:46,160
+tools for building prototypes of models hugging face has some tools built into their playground they've also recently acquired a company called gradio which we'll be using in the lab for the course which makes it very easy to wrap a small user interface around the model and then streamlit is also a great tool for doing this streamlight gives you a little bit more flexibility than something like radio or hugging face spaces at the cost of just needing to put a little bit more thought into how to pull all the pieces together in your ui but it's still very easy to use a few best practices to think about when you're deploying the prototype model first i would encourage you to have a basic ui for the model not
+
+6
+00:02:44,720 --> 00:03:20,480
+just to have an api and the reason for that is you know the goal at this stage is to play around with the model get feedback on the model both yourself and also from your friends or your co-workers or whoever else you're talking with this project about gradio and streamlight are really your friends here gradio really it's often as easy as adding a couple of lines of code to create a simple interface for a model streamlit is a little bit more ambitious in that it's a tool that allows you to build pretty complex uis just using python so it'll be familiar interfaces for you if you're a python developer but will require a little bit more thought about how you want to structure things but still very easy next best practice
+
+7
+00:03:18,800 --> 00:03:51,040
+is don't just run this on your laptop it's actually worth at this stage putting it behind a web url why is that important one it's easier to share right so part of the goal here is to collect feedback from other folks but it also starts to get you thinking about some of the trade-offs that you'll be making when you do a more complex deployment how much latency does this model actually have luckily there are cloud versions of both streamlit and hub and face which are which make this very easy so there's at this point in time not a lot of excuse not to just put this behind a simple url so you can share with people and then the last tip here is just don't stress too much at this stage again this is a prototype this is
+
+8
+00:03:49,360 --> 00:04:20,880
+something that should take you not more than like maybe a day if you're doing it for the first time but if you're building many of these models maybe it even just takes you a couple hours we've talked about this first step which is buildings prototype and next i want to talk about why is this not going to work like why is this not going to be the end solution that you use to deploy your model so where will this fail the first big thing is with any of these tools that we discussed you're going to have limited flexibility in terms of how you build the user interface for your model and extremely gives you more flexibility there than gradio but still relatively limited flexibility and so eventually you're gonna want to be able to build a
+
+9
+00:04:19,040 --> 00:04:53,520
+fully custom ui for the model and then secondly these systems tend not to scale very well to many concurrent requests so if it's just you or you and a couple friends playing around the model that's probably fine but once you start to have users you'll hit the scaling limits of these pretty quickly and this is a good segue to talk about at a high level different ways you can structure your machine learning power application in particular where the model fits into that application so we'll start with an abstract diagram of how your application might look there's a few different components to this on the left we have a client and the client is essentially your user and that's the device that they're using to interact with the
+
+10
+00:04:52,160 --> 00:05:28,320
+application that you built so it could be a browser it could be a vehicle whatever that device is that they're interacting with then that device will make calls over a network to a server that server is typically if you're building a web app where most of your code is running that server will talk to a database where there's data stored that's used for powering the application and there's different ways of structuring this application to fit a machine learning model inside the prototype approach that we just described mostly fits into this model in service approach where the web server that you're hosting actually just has a packaged version of the model sitting inside of it when you write a streamled script for a gradioscript part of that
+
+11
+00:05:26,560 --> 00:06:02,000
+script will be to load the model and so that script will be building your ui as well as running the model at the same time so this pattern like all patterns has pros and cons the biggest pro i think is one it's really easy if you're using one of these prototype development tools but two even if you are doing something a little bit more complicated like you're reusing your web infrastructure for the app that your company is building you get to reuse a lot of existing infrastructure so it doesn't require you as a model developer to set up a lot of new things just to try your model out and that's really great but there are a number of pretty pronounced cons to this as well the first is that your web server in many
+
+12
+00:05:59,840 --> 00:06:34,080
+cases like once you get beyond this streamlight and gradio type example might be written in a different language than your model like it might be written in in ruby or in javascript or something like that getting your model into that language can be difficult the second reason is that oftentimes especially in early in the life cycle of building your model your model might be changing more frequently than your server code so if you have a relatively well established application but a model that you're still building you might not want to have to redeploy the entire application every single time that you make an update to the model which might be every day or even multiple times a day the third con of this approach is that it
+
+13
+00:06:31,360 --> 00:07:06,240
+doesn't scale very well with model size so if you have a really large model that you're trying to run inference on you'll have to load that on your web server and so that's going to start to eat into the resources of that web server and might affect the user experience for people using that web server even if they're not interacting with the model or that's not the primary thing that they're doing in that web application because all of the resources from that web server are being directed to making this model run the fourth reason is that server hardware the hardware that you're probably running your web application or your mobile application on is generally not optimized very well for machine learning workloads and so in particular
+
+14
+00:07:04,240 --> 00:07:38,720
+you're very rarely going to have a gpu on these devices that may or may not be a deal breaker which we'll come back to later in the lecture and the last con is that your model itself and the application that's part of might have very different scaling properties and you might want to be able to scale them differently so for example if you're running a very lightweight ui then it might not take a lot of resources or a lot of thought to scale it to many users but if your model itself is really complicated or very large you might need to get into some of the advanced techniques in this lecture and host these models on gpus to get them to scale you don't want to necessarily have to bring all of that complexity to your
+
+15
+00:07:37,120 --> 00:08:13,840
+web server it's important when there's different scaling properties to be able to separate these concerns as part of the application that you're building so that brings us to the second step which is pulling your model out of the ui and there's a couple of different ways that we can do this and we'll talk about two different patterns here the first is to pull your model out of the ui and have it interact directly with the database this is called batch prediction so how does this work periodically you will get new data in and you'll run your model on each of those data points then you'll save the results of that model inference into a database this can work really well in some circumstances so for example if there's just not a lot of
+
+16
+00:08:11,599 --> 00:08:50,240
+potential inputs to the model if you have one prediction per user or one prediction per customer or something along those lines then you can rerun your model on some frequency like every hour or every day or every week and you can have reasonably fresh predictions to return to those users just stored in your database so examples of types of problems where this can work well are you know in the early stages of building out a recommender system in some cases for doing more internal facing use cases like marketing automation if for example you want to give each of your marketing leads a score that tells your marketing your sales team how much effort to put into closing those leads then you'll have this finite universe of leads that
+
+17
+00:08:48,800 --> 00:09:23,680
+needs a prediction for the model so you can just run a model prediction on every single possible lead store that in a database and then let your users interact with it from there how can you actually do this how do you actually run the model on the schedule the data processing and workflow tools that we talked about in the previous lecture also work really well here what you'll need to do is you'll need to re-run your data pre-processing you'll then need to load the model run the predictions and store the predictions in the database that you're using for your application and so this is exactly a directed acyclic graph a workflow of data operations that tools like dagster airflow or prefix are designed to solve
+
+18
+00:09:22,240 --> 00:09:58,720
+it's worth noting here that there's also tools like metaflow that are designed more for a machine learning or data science use case that might be potentially even an easier way to get started so what are the pros and cons of this pattern of running your model offline and putting the predictions in a database the biggest pro is that this is just really easy to implement right it's reusing these existing batch processing tools that you may already be using for trading your model and it doesn't require you to host any type of new web server to get those predictions to users you can just put the predictions in the database that your product is already using it also scales very easily because databases themselves are designed and
+
+19
+00:09:57,040 --> 00:10:35,120
+have been engineered for decades to scale really easily it's also you know it seems like a simple pattern but it's used in production by very large scale production systems by large companies and it has been for years often times this is for things like recommender systems this is a tried and true pattern that you can run and become pretty confident that it'll work well and then it's also relatively low latency because the database itself is designed for the end application to interact with so latency was a concern that the database designers were able to solve for us there's also some very pronounced cons to this approach and the most important one is that it just doesn't work for every type of model if you have complex
+
+20
+00:10:33,360 --> 00:11:08,880
+inputs to your model if the universe of inputs is too large to enumerate every single time you need to update your predictions then this just isn't going to work second con is that your users are not going to be getting the most up-to-date predictions from your model if the features going into your model let's say change every hour or every minute or some second but you only run this batch prediction job every day then the predictions your users see might be slightly stale think about this in the context of a recommender system if you're only running the predictions of the recommender system every day then those recommendations that you serve to your users won't take into account all of the contacts that those users have
+
+21
+00:11:07,040 --> 00:11:42,720
+provided you in between those predictions so the movies that they watch today the tv shows that they watch today those won't be taken into account in at least the machine learning part of their recommendations but there's you know there's other algorithmic ways to make sure that you don't do things like show users the same movie twice and the final con here is that models frequently can become stale so if your batch job fails for some reason there's a timeout in one of your data pre-processing steps and the new predictions don't get dumped into the database these types of things can make this problem of not getting up-to-date predictions worse and worse and they can be very hard to detect although there's tools for data quality
+
+22
+00:11:41,200 --> 00:12:17,920
+that can really help detect them the next pattern that we're going to talk about is rather than running the model offline and putting the predictions in a database instead let's run the model online as its own service the service is going to interact with the backend or the client itself by making requests to this model service sending hey what is the prediction for this particular input and receiving responses back the model says that the prediction for this input is this particular value the pros of this approach are it's dependable if you have a bug in your model if your model is running directly in the web server then that can crash your entire application but hosting this as an independent service in your application
+
+23
+00:12:16,160 --> 00:12:51,680
+means that's less likely second it's more scalable so you can choose what is the best hardware what is the best infrastructure setup for the model itself and scale that as you need to without needing to worry about how that affects the rest of your application third it's really flexible if you stand up a model service for a particular model you can reuse that service in other applications or other parts of your application very easily concierge since this is a separate service you add a network call when your server or your client interacts with the model it has to make a request and receive a response over the network so that can add some latency to your application it adds infrastructure complexity relative to
+
+24
+00:12:50,560 --> 00:13:29,680
+the other techniques that we've talked about before because now you're on the hook for hosting and managing a separate service just to host your model this is i think really the challenge for a lot of ml teams is that hey i'm good at training models i'm not sure how to run a web service however i do think this is the sweet spot for most ml powered products because the cons of the other approaches are just too great you really need to be able to scale models independently of the application itself in most complex use cases and for a lot of interesting uses of ml we don't have a finite universe of inputs to the model that we can just enumerate every day we really need to be able to have our users send us whatever requests that they want
+
+25
+00:13:27,440 --> 00:13:59,680
+to get and receive a customized response back in this next section we'll talk through the basics of how to build your model service there's a few components to this we will talk about rest apis which are the language that your service will use to interact with the rest of your application we'll talk about dependency management so how to deal with these pesky versions of pi torch or tensorflow that you might need to be upgrading and we'll talk about performance optimization so how to make this run fast and scale well and then we'll talk about rollout so how to get the next version of your model into production once you're ready to deploy it and then finally we'll once we've covered sort of the technical
+
+26
+00:13:58,560 --> 00:14:40,399
+considerations that you'll need to think about we'll talk about managed options that solve a lot of these technical problems for you first let's talk about rest apis what are rest apis rest apis serve predictions in response to canonically formatted http requests there's other alternative protocols to rest for interacting with a service that you host on your infrastructure probably the most common one that you'll see in ml is grpc which is used in a lot of google products like tensorflow serving graphql is another really commonly used protocol in web development that is not terribly relevant for building model services so what does a rest api look like you may have seen examples of this before but when you are sending data to
+
+27
+00:14:37,199 --> 00:15:20,000
+a web url that's formatted as json blog oftentimes this is a rest request this is an example of what it might look like to interact with the rest api in this example we are sending some data to this url which is where the rest api is hosted api.fullstackdeeplearning.com and we're using the post method which is one of the parts of the rest standard that tells the server how it's going to interact with the data that we're sending and then we're sending this json blob of data that represents the inputs to the model that we want to receive a prediction from so one question you might ask is there any standard for how to format the inputs that we send to the model and unfortunately there isn't really any standard yet here are a few
+
+28
+00:15:16,959 --> 00:16:01,279
+examples from rest apis for model services hosted in the major clouds and we'll see some differences here between how they expect the inputs to the model to be formatted for example in google cloud they expect a batch of inputs that is structured as a list of what they call instances each of which has values and a key in azure they expect a list of things called data where the data structure itself depends on what your model architecture is and in sagemaker they also expect instances but these instances are formatted differently than they are in google cloud so one thing i would love to see in the future is moving toward a standard interface for making rest api calls for machine learning services since the types of
+
+29
+00:16:00,079 --> 00:16:35,839
+data that you might send to these services is pretty constrained we should be able to develop a standard as an industry the next topic we'll cover is dependency management model predictions depend not only on the weights of the model that you're running the prediction on but also on the code that's used to turn those weights into the prediction including things like pre-processing and the dependencies the specific library versions that you need in order to run the function that you called and in order for your model to make a correct prediction all of these dependencies need to be present on your web server unfortunately dependencies are a notorious cause of trouble in web applications in general and in
+
+30
+00:16:34,000 --> 00:17:10,799
+particular in machine learning web services the reason for that is a few things one they're very hard to make consistent between your development environment and your server how do you make sure that the server is running the exact same version of tensorflow pytorch scikit-learn numpy whatever other libraries you depend on as your jupyter notebook was when you train those models the second is that they're hard to update if you update dependencies in one environment you need to update them in all environments and in machine learning in particular since a lot of these libraries are moving so quickly small changes in something like a tensorflow version can change the behavior of your model so it's important to be like
+
+31
+00:17:09,439 --> 00:17:47,360
+particularly careful about these versions in ml at a high level there's two strategies that will cover for managing dependencies the first is to constrain the dependencies for just your model to save your model in a format that is agnostic that can be run anywhere and then the second is to wrap your entire inference program your entire predict function for your model into what's called a container so let's talk about how to constrain the dependencies of just your model the primary way that people do this today is through this library called onyx the open neural network exchange and the goal of onyx is to be an interoperability standard for machine learning models what they want you to be able to do is to define a neural network
+
+32
+00:17:45,280 --> 00:18:24,960
+in any language and run it consistently anywhere no matter what inference framework you're using hardware you're using etc that's the promise the reality is that since the underlying libraries used to build these models are currently changing so quickly there's often bugs in this translation layer and in many cases this can create more problems than it actually solves for you and the other sort of open problem here is this doesn't really deal with non-library code in many cases in ml things like feature transformations image transformations you might do as part of your tensorflow or your pi torch graph but you might also just do as a python function that wraps those things and these open neural network standards like
+
+33
+00:18:22,960 --> 00:18:57,440
+onyx don't really have a great story for how to handle pre-processing that brings us to a second strategy for managing dependencies which is containers how can you manage dependencies with containers like docker so we'll cover a few things here we'll talk about the differences between docker and general virtual machines which you might have covered in a computer science class we'll talk about how docker images are built via docker files and constructed via layers we'll talk a little bit about the ecosystem around docker and then we'll talk about specific wrappers around docker that you can use for machine learning the first thing to know about docker is how it differs from virtual machines which is an older technique for
+
+34
+00:18:55,600 --> 00:19:33,440
+packaging up dependencies in a virtual machine you essentially package up the entire operating system as well as all the libraries and applications that are built on top of that operating system so it tends to be very heavy weight because the operating system is itself just a lot of code and expensive to run the improvement that docker made is by removing the need to package up the operating system alongside the application instead you have the libraries and applications packaged up together in something called a container and then you have a docker engine that runs on top of your the operating system on your laptop or on your server that knows how to to virtualize the os and run your bins and libraries and
+
+35
+00:19:32,080 --> 00:20:08,559
+applications on top of it so we just learned that docker is much more lightweight than the typical virtual machine and by virtue of being lightweight it is used very differently than vms were used in particular a common pattern is to spin up a new docker container for every single discrete task that's part of your application so for example if you're building a web application you wouldn't just have a single docker container like you might if you were using a virtual machine instead you might have four you might have one for the web server itself one for the database one for job queue and one for your worker since each one of these parts of your application serves a different function it has different library dependencies and maybe
+
+36
+00:20:07,200 --> 00:20:43,039
+in the future you might need to scale it differently each one of them goes into its own container and those containers are are run together as part of an orchestration system which we'll talk about in a second how do you actually create a docker container docker containers are created from docker files this is what a docker file looks like it runs a sequence of steps to define the environment that you're going to run your code in so in this case it is importing another container that has some pre-packaged dependencies for running python 2.7 hopefully you're not running python 2.7 but if you were you could build a docker container that uses it using this from command at the top and then doing other things like adding
+
+37
+00:20:41,440 --> 00:21:21,120
+data from your local machine hip installing packages exposing ports and running your actual application you can build these docker containers on your laptop and store them there if you want to when you're doing development but one of the really powerful things about docker is it also allows you to build store and pull docker containers from a docker hub that's hosted on some other server on docker servers or on your cloud provider for example the way that you would run a docker container typically is by using this docker run command so what that will do is in this case it will find this container on the right called gordon slash getting started part two and it'll try to run that container but if you're connected
+
+38
+00:21:19,360 --> 00:21:59,280
+to a docker hub and you don't have that docker image locally then what it'll do is it'll automatically pull it from the docker hub that you're connected to the server that your docker engine is connected to it'll download that docker container and it will run it on your local machine so you can experiment with that code environment that's going to be identical to the one that you deploy on your server and in a little bit more detail docker is separated into three different components the first is the client this is what you'll be running on your laptop to build an image from a docker file that you define locally to pull an image that you want to run some code in on your laptop to run a command inside of an image those commands are
+
+39
+00:21:56,799 --> 00:22:35,440
+actually executed by a docker host which is often run on your laptop but it doesn't have to be it can also be run on a server if you want more storage or more performance and then that docker host talks to a registry which is where all of the containers that you might want to access are stored this separation of concerns is one of the things that makes docker really powerful because you're not limited by the amount of compute and storage you have on your laptop to build pull and run docker images and you're not limited by what you have access to on your docker host to decide which images to run in fact there's a really powerful ecosystem of docker images that are available on different public docker hubs you can
+
+40
+00:22:33,360 --> 00:23:07,600
+easily find these images modify them and contribute them back and have the full power of all the people on the internet that are building docker files and docker images there might just be one that already solves your use case out of the box it's easy to store private images in the same place as well so because of this community and lightweight nature of docker it's become incredibly popular in recent years and is pretty much ubiquitous at this point so if you're thinking about packaging dependencies for deployment this is probably the tool that you're going to want to use docker is not as hard to get started with as it sounds you'll need to read some documentation and play around with docker files a little bit to get a
+
+41
+00:23:05,679 --> 00:23:40,240
+feel for how they work and how they fit together you oftentimes won't need to build your own docker image at all because of docker hubs and you can just pull one that already works for your use case when you're getting started that being said there is a bit of a learning curve to docker isn't there some way that we can simplify this if we're working on machine learning and there's a number of different open source packages that are designed to do exactly that one is called cog another is called bento ml and a third is called truss and these are all built by different model hosting providers that are designed to work well with their model hosting service but also just package your model and all of its dependencies in a
+
+42
+00:23:38,720 --> 00:24:14,960
+standard docker container format so you could run it anywhere that you want to and the way that these systems tend to work is there's two components the first is there's a standard way of defining your prediction service so your like model.predict function how do you wrap that in a way that this service understands so in cog it's this base predictor class that you see on the bottom left in truss it's dependent on the model library that you're using like you see on the right hand side that's the first thing is how do you actually package up this model.predict function and then the second thing is a yaml file which sort of defines the other dependencies and package versions that are going to go into this docker
+
+43
+00:24:13,360 --> 00:24:47,520
+container that will be run on your laptop or remotely and so this this sort of a simplified version of the steps that you would put into your docker build command but at the end of the day it packages up in the standard format so you can deploy it anywhere so if you want to have some of the advantages of using docker for making your machine learning models reproducible and deploying them but you don't want to actually go through the learning curve of learning docker or you just want something that's a little bit more automated for machine learning use cases then it's worth checking out these three libraries the next topic we'll discuss is performance optimization so how do we make models go bur how do we make them
+
+44
+00:24:45,919 --> 00:25:22,559
+go fast and there's a few questions that we'll need to answer here first is should we use a gpu to do inference or not we'll talk about concurrency model distillation quantization caching batching sharing the gpu and then finally libraries that automate a lot of these things for you so the spirit of this is going to be sort of a whirlwind tour through some of the major techniques of making your models go faster and we'll try to give you pointers where you can go to learn more about each of these topics the first question you might ask is should you host your model on a gpu or on a cpu there's some advantages to hosting your model on a gpu the first is that it's probably the same hardware that you train your model on to begin with so
+
+45
+00:25:20,640 --> 00:25:59,919
+that can eliminate some loss and translation type moments the second big con is that as your model gets really big and as your techniques get relatively advanced your traffic gets very large this is usually how you can get the sort of maximum throughput like the most number of users that are simultaneously hitting your model is by hosting the model on a gpu but gpus introduce a lot of complexity as well they're more complex to set up because they're not as well trodden the path for hosting web services as cpus are and they're often almost always actually more expensive so i think one point that's worth emphasizing here since it's a common misconception i see all the time is just because your model was trained on a gpu does not mean that you
+
+46
+00:25:57,760 --> 00:26:36,080
+need to actually host it on a gpu in order for it to work so consider very carefully whether you really need a gpu at all or whether you're better off especially for an early version of your model just hosting it on a cpu in fact it's possible to get very high throughput just from cpu inference at relatively low cost by using some other techniques and so one of the main ones here is concurrency concurrency means on a single host machine not just having a single copy of the model running but having multiple copies of the model running in parallel on different cpus or different cpu cores how can you actually do this the main technique that you need to be careful about here is thread tuning so making sure that in torch it
+
+47
+00:26:34,400 --> 00:27:10,320
+knows which threads you need to use in order to actually run the model otherwise the different torch models are going to be competing for threads on your machine there's a great blog post from roblox about how they scaled up bert to serve a billion daily requests just using cpus and they found this to be much easier and much more cost effective than using gpus cpus can be very effective for scaling up to high throughput as well you don't necessarily need gpus to do that the next technique that we'll cover is model distillation what is model distillation model distillation means once you have your model that you've trained maybe a very large or very expensive model that does very well at the task that you want to
+
+48
+00:27:08,000 --> 00:27:44,399
+solve you can train a smaller model that tries to imitate the behavior of your larger one and so this generally is a way of taking the knowledge that your larger model learned and compressing that knowledge into a much smaller model that maybe you couldn't have trained to the same degree of performance from scratch but once you have that larger model it's able to imitate it so how does this work i'll just point you to this blog post that covers several techniques for how you can do this it's worth noting that this can be tricky to do on your own and is i would say relatively infrequently done in practice in production a big exception to that is oftentimes there are distilled versions of popular models the stilbert is a
+
+49
+00:27:42,399 --> 00:28:21,120
+great example of this that are pre-trained for you that you can use for very limited performance trade-off the next technique that we're going to cover is quantization what is it this means that rather than taking all of the matrix multiplication math that you do when you make a prediction with your model and doing that all in the sort of full precision 64 or 32-bit floating point numbers that your model weights might be stored in instead you execute some of those operations or potentially all of them in a lower fidelity representation of the numbers that you're doing the math with and so these can be 16-bit floating point numbers or even in some cases 8-bit integers this introduces some trade-offs with accuracy
+
+50
+00:28:19,279 --> 00:28:55,600
+but oftentimes this is a trade-off that's worth making because the accuracy you lose is pretty limited relative to the performance that you gain how can you do this the recommended path is to use the built-in methods in pytorch and hugging face and tensorflow lite rather than trying to roll this on your own and it's also worth starting to think about this even when you're training your model because techniques called quantization aware training can result in higher accuracy with quantized models than just naively training your model and then running quantization after the fact i want to call out one tool in particular for doing this which is relatively new optimum library from uh hugging face which just makes this very
+
+51
+00:28:53,840 --> 00:29:31,840
+easy and so if you're already using hugging face models there's a little downside to trying this out next we'll talk about caching what is caching for some machine learning models if you look at the patterns of the inputs that users are requesting that model to make predictions on there's some inputs that are much more common than others so rather than asking the model to make those predictions from scratch every single time users make those requests first let's store the common requests in a cache and then let's check that cache before we actually run this expensive operation of running a forward pass on our neural network how can you do this there's a huge depth of techniques that you can use for intelligent caching but
+
+52
+00:29:29,760 --> 00:30:07,679
+there's also a very basic way to do this using func tools library in python and so this looks like it's just adding a wrapper to your model.predict code that will essentially check the cache to see if this input is stored there and return the sort of cached prediction if it's there otherwise run the function itself and this is also one of the techniques used in the roblox blog post that i highlighted before for scaling this up to a billion requests per day the pretty important part of their approach so for some use cases you can get a lot of lift just by simple caching the next technique that we'll talk about is batching so what is the idea behind batching well typically when you run inference on a machine learning model
+
+53
+00:30:05,919 --> 00:30:46,240
+unlike in training you are running it with bat shy as equals one so you have one request come in from a user and then you respond with the prediction for that request and the fact that we are running a prediction on a single request is part of why generally speaking gpus are not necessarily that much more efficient than cpus for running inference what batching does is it takes advantage of the fact that gpus can achieve much higher throughput much higher number of concurrent predictions when they do that prediction in parallel on a batch of inputs rather than on a single input at a time how does this work you have individual predictions coming in from users i want a prediction for this input i want a prediction for this input so
+
+54
+00:30:44,799 --> 00:31:23,360
+you'll need to gather these inputs together until you have a batch of a sufficient size and then you'll run a prediction on that batch and then split the batch into the predictions that correspond to the individual requests and return those to the individual users so there's a couple of pretty tricky things here one is you'll need to tune this batch size in order to trade off between getting the most throughput from your model which generally requires a larger batch size and reducing the inference latency for your users because if you need to wait too long in order to gather enough predictions to fit into that batch then your users are gonna pay the cost of that they're gonna be the ones waiting for that response to come
+
+55
+00:31:21,840 --> 00:31:57,440
+back so you need to tune the batch size to trade off between those two considerations you'll also need some way to shortcut this process if latency becomes too long so let's say that you have a lull in traffic and normally it takes you a tenth of a second to gather your 128 inputs that you're going to put into a bash but now all of a sudden it's taking a full second to get all those inputs that can be a really bad user experience if they just have to wait for other users to make predictions in order to see their response back so you'll want some way of shortcutting this process of gathering all these data points together if the latency is becoming too long for your user experience so hopefully it's clear from
+
+56
+00:31:55,600 --> 00:32:32,559
+this that this is pretty complicated to implement and it's probably not something that you want to implement on your own but luckily it's built into a lot of the libraries for doing model hosting on gpus which we'll talk about in a little bit the next technique that we'll talk about is sharing the gpu between models what does this mean your model may not necessarily fully utilize your gpu for inference and this might be because your batch size is too small or because there's too much other delay in the system when you're waiting for requests so why not just have multiple models if you have multiple model services running on the same view how can you do this this is generally pretty hard and so this is also a place where
+
+57
+00:32:30,799 --> 00:33:08,080
+you'll want to run an out-of-the-box model serving solution that solves this problem for you so we talked about how in gpu inference if you want to make that work well there's a number of things like sharing the gpu between models and intelligently batching the inputs to the models to trade off between latency and throughput that you probably don't want to implement yourself luckily there's a number of libraries that will solve some of these gpu hosting problems for you there's offerings from tensorflow which is pretty well baked into a lot of google cloud's products and pytorch as well as third-party tools from nvidia and any scale and ray nvidia's is probably the most powerful and is the one that i often see from companies that are trying
+
+58
+00:33:06,399 --> 00:33:43,200
+to do very high throughput model serving but can also often be difficult to get started with starting with ray serve or the one that's specific to your neural net library is maybe an easier way to get started if you want to experiment with this all right we've talked about how to make your model go faster and how to optimize the performance of the model on a single server but if you're going to scale up to a large number of users interacting with your model it's not going to be enough to get the most efficiency out of one server at some point you'll need to scale horizontally to have traffic going to multiple copies of your model running on different servers so what is horizontal scaling if you have too much traffic for a single
+
+59
+00:33:41,360 --> 00:34:18,079
+machine you're going to take that stream of traffic that's coming in and you're going to split it among multiple machines how can you actually achieve this each machine that you're running your model on will have its own separate copy of your service and then you'll route traffic between these different copies using a tool called a load balancer in practice there's two common methods of doing this one is container orchestration which is a sort of set of techniques and technologies kubernetes being the most popular for managing a large number of different containers that are running as part of one application on your infrastructure and then a second common method especially in machine learning is serverless so
+
+60
+00:34:16,800 --> 00:34:52,320
+we'll talk about each of these let's start with container orchestration when we talked about docker we talked about how docker is different than typical deployment and typical virtual machines because rather than running a separate copy of the operating system for every virtual machine or program that you want to run instead you run docker on your server and then docker is able to manage these lightweight virtual machines that run each of the parts of your application that you want to run so when you deploy docker typically what you'll do is you'll run a docker host on a server and then you'll have a bunch of containers that the docker host is responsible for managing and running on that server but when you want to scale
+
+61
+00:34:50,960 --> 00:35:28,240
+out horizontally so when you want to have multiple copies of your application running on different servers then you'll need a different tool in order to coordinate between all of these different machines and docker images the most common one is called kubernetes kubernetes works together with very closely with docker to build and run containerized distributed applications kubernetes helps you remove the sort of constraint that all of the containers are running on the same machine kubernetes itself is a super interesting topic that is worth reading about if you're interested in distributed computing and infrastructure and scaling things up but for machine learning deployment if your only goal is to deploy ml models it's probably overkill
+
+62
+00:35:26,720 --> 00:36:01,520
+to learn a ton about kubernetes there's a number of frameworks that are built on top of kubernetes that make it easier to use for deploying models the most commonly used ones in practice tend to be kubeflow serving and selden but even if you use one of these libraries on top of kubernetes for container orchestration you're still going to be responsible for doing a lot of the infrastructure management yourself and serverless functions are an alternative that remove a lot of the need for infrastructure management and are very well suited for machine learning models the way these work is you package up your app code and your dependencies into a docker container and that docker container needs to have a single entry
+
+63
+00:36:00,079 --> 00:36:36,079
+point function like one function that you're going to run over and over again in that container so for example in machine learning this is most often going to be your model.predict function then you deploy that container to a service service like aws lambda or the equivalence in google or azure clouds and that service is responsible for running that predict function inside of that container for you over and over and over again and takes care of everything else scaling load balancing all these other considerations that if you're horizontally scaling a server would be your problem to solve on top of that there's a different pricing model so if you're running a web server then you control that whole web server and so you
+
+64
+00:36:34,400 --> 00:37:08,560
+pay for all the time that it's running 24 hours a day but with serverless you only pay for the time that these servers are actually being used to run your model you know if your model is only serving predictions or serving most of its predictions eight hours a day let's say then you're not paying for the other 16 hours where it's not serving any predictions because of all these things serverless tends to be very well suited to building model services especially if you are not an infrastructure expert and you want a quick way to get started so we recommend this as a starting point for once you get past your prototype application so the genius idea here is your servers can't actually go down if you don't have any we're doing
+
+65
+00:37:06,960 --> 00:37:41,280
+serverless serverless is not without its cons one of the bigger challenges that has gotten easier recently but is still often a challenge in practice is that the packages that you can deploy with these serverless applications tend to be limited in size so if you have an absolutely massive model you might run into those limits there's also a cold start problem what this means is serverless is designed to scale all the way down to zero so if you're not receiving any traffic if you're not receiving any requests for your model then you're not going to pay which is one of the big advantages of serverless but the problem is when you get that first request after the serverless function has been cold for a while it
+
+66
+00:37:39,520 --> 00:38:15,440
+takes a while to start up it can be seconds or even minutes to get that first prediction back once you've gotten that first prediction back it's faster to get subsequent predictions back but it's still worth being aware of this limitation another challenge practical challenge is that many of these server these serverless services are not well designed for building pipelines and models so if you have a complicated chaining of logic to produce your prediction then it might be difficult to implement that in a server-less context there's little little or no state management available in serverless functions so for example if caching is really important for your application it can be difficult to build that caching
+
+67
+00:38:13,920 --> 00:38:50,320
+in if you're deploying your model in serverless and there's often limited deployment tooling as well so rolling out new versions of the serverless function there's often not all the tooling that you'd want to make that really easy and then finally these serverless functions today are cpu only and they have limited execution time of you know a few seconds or a few minutes so if you truly need gpus for imprints then serverless is not going to be your answer but i don't think that limitation is going to be true forever in fact i think we might be pretty close to serverless gpus there's already a couple of startups that are claiming to offer serverless gpu for inference and so if you want to do inference on gpus but you
+
+68
+00:38:48,560 --> 00:39:24,880
+don't want to manage gpu machines yourself i would recommend checking out these two options from these two young startups the next topic that we'll cover in building a model service is rollouts so what do you need to think about in terms of rolling out new models if serving is how you turn your machine learning model into something that can respond to requests that lives on a web server that anyone or anyone that you want to can send a request to and get a prediction back then rollouts are how you manage and update these services so if you have a new version of a model or if you want to split traffic between two different versions to run an a b test how do you actually do that from an infrastructure perspective you probably
+
+69
+00:39:23,440 --> 00:39:56,960
+want to have the ability to do a few different things so one is to roll out new versions gradually what that means is when you have version n plus one of your model and you want to replace version n with it it's sometimes helpful to be able to rather than just instantly switching over all the traffic to n plus one instead start by sending one percent of your traffic to n plus one and then ten percent and then 50 and then once you're confident that it's working well then switch all of your traffic over to it so you'll want to be able to roll out new versions gradually on the flip side you'll want to be able to roll back to an old version instantly so if you detect a problem with the new version of the model that you deployed hey on this
+
+70
+00:39:55,200 --> 00:40:28,720
+10 of traffic that i'm sending to the new model users are not responding well to it or it's sending a bunch of errors you'll want to be able to instantly revert to sending all of your traffic to the older version of the model you want to be able to split traffic between versions a sort of a prerequisite for doing these things as well as running an av test you also want to be able to deploy pipelines of models or deploy models in a way such that they can shadow the prediction traffic they can look at the same inputs as your main model and produce predictions that don't get sent back to users so that you can test whether the predictions look reasonable before you start to show them to users this is just kind of like a
+
+71
+00:40:26,800 --> 00:41:06,720
+quick flavor of some of the things that you might want to solve for in a way of doing model rollouts this is a challenging infrastructure problem so it's beyond the scope of this lecture in this class really if you're using a managed option which we'll come to in a bit or you have infrastructure that's provided for you by your team it may take care of this for you already but if not then looking into a managed option might be a good idea so manage options take care of a lot of the scaling and roll out challenges that you'd otherwise face if you host models yourself even on something like aws lambda there's a few different categories of options here the cloud providers all provide their own sort of managed options as well as in
+
+72
+00:41:04,560 --> 00:41:42,079
+most of the end-to-end ml platforms so if you're already using one of these cloud providers or end-to-end ml platforms pretty heavily it's worth checking out their offering to see if that works for you and there's also a number of startups that have offerings here so there's a couple that are i would say more focused on developer experience like bento ml and cortex so if you find sagemaker really difficult to use or you just hate the developer experience for it it might be worth checking one of those out cortex recently was acquired by databricks so it might also start to be incorporated more into their offerings then there's startups that are have offerings that are more have good ease of use but are also really focused on performance
+
+73
+00:41:39,760 --> 00:42:17,440
+banana is a sort of popular upcoming example of that to give you a feel of what these manage options look like i want to double click on sagemaker which is probably the most popular managed offering the happy path in sagemaker is if your model is already in a digestible format a hugging face model or a scikit-learn model or something like that and in those cases deploying the sagemaker is pretty easy so you will instead of using like kind of a base hugging face class you'll instead use this sagemaker wrapper for the hogging face class and then call fit like you normally would that can also be run on the cloud and then to deploy it you just will call the dot deploy method of this hugging face wrapper and you'll specify
+
+74
+00:42:15,920 --> 00:42:52,319
+how many instances you want this to run on as well as how beefy you need the hardware to be to run it then you can just call predictor.predicts using some input data and it'll run that prediction on the cloud for you in order to return your response back you know i would say in the past sagemaker had a reputation for being difficult to use if you're just doing inference i don't think that reputation is that warranted i think it's actually like pretty easy to use and in many cases is a very good choice for deploying models because it has a lot of easy wrappers to prevent you from needing to build your own docker containers or things like that and it offers options for both deploying model to a dedicated web server like you see
+
+75
+00:42:50,960 --> 00:43:29,119
+in this example as well as to a serverless instance the main trade-offs with using sagemaker are one is you want to do something more complicated than standard huggy face or psychic learn model you'll again still need to deploy a container and the interface for deploying a container is maybe not as user friendly or straightforward as you might like it to be interestingly as of yesterday it was quite a bit more expensive for employing models to dedicated instances than raw ec2 but maybe not so much more expensive than serverless if you're going to go serverless anyway and you're willing to pay 20 overhead to have something that is a better experience for deploying most machine learning models then sagemaker is worth checking out if
+
+76
+00:43:26,960 --> 00:44:07,839
+you're already on amazon take aways from building a model service first you probably don't need to do gpu inference and if you're doing cpu inference then oftentimes scaling horizontally to more servers or even just using serverless is the simplest option is often times enough serverless is probably the recommended option to go with if you can get away with cpus and it's especially helpful if your traffic is spiky so if you have more users in the morning or if you only send your model predictions at night or if your traffic is low volume where you wouldn't max out a full beefy web server anyway sagemaker is increasingly a perfectly good way to get started if you're on aws can get expensive once you've gotten to the
+
+77
+00:44:06,400 --> 00:44:42,720
+point where that cost really starts to matter then you can consider other options if you do decide to go down the route of doing gpu inference then don't try to roll your own gpu inference instead it's worth investing in using a tool like tensorflow serving or triton because these will end up saving you time and leading to better performance in the end and lastly i think it's worth keeping an eye on the startups in this space for on-demand gpu inference because i think that could change the equation of whether gpu inference is really worth it for machine learning models the next topic that we'll cover is moving your model out of a web server entirely and pushing it to the edge so pushing it to where your users are when
+
+78
+00:44:41,520 --> 00:45:15,520
+should you actually start thinking about this sometimes it's just obvious let's say that you uh your users have no reliable internet connection they're driving a self-driving car in the desert or if you have very strict data security or privacy requirements if you're building on an apple device and you can't send the data that you need you need to make the predictions back to a web server otherwise if you don't have those strict requirements the trade-off that you'll need to consider is both the accuracy of your model and the latency of your user receiving a response from that model affect the thing that we ultimately care about which is building a good end user experience latency has a couple of different components to it one
+
+79
+00:45:13,599 --> 00:45:50,240
+component to it is the amount of time it takes the model to make the prediction itself but the other component is the network round trip so how long it takes for the user's request to get to your model service and how long it takes for the prediction to get back to the client device that your user is running on and so if you have exhausted your options for reducing the amount of time that it takes for them all to make a prediction or if your requirements are just so strict that there's no way for you to get within your latency sla by just reducing the amount of time it takes for the model to make prediction then it's worth considering moving to the edge even if you have you know reliable internet connection and don't have very
+
+80
+00:45:48,640 --> 00:46:23,119
+strict data security and privacy requirements but it's worth noting that moving to the edge adds a lot of complexity that isn't present in web development so think carefully about whether you really need this this is the model that we're considering in edge prediction where the model itself is running on the client device as opposed to running on the server or in its own service the way this works is you'll send the waste to the client device and then the client will load the model and interact with it directly there's a number of pros and cons to this approach the biggest pro is that this is the lowest latency way that you can build machine learning powered products and latency is often a pretty important
+
+81
+00:46:21,440 --> 00:46:56,240
+driver of user experience it doesn't require an internet connection so if you're building robots or other types of devices that you want to run ml on this can be a very good option it's great with data security because the data that needs to make the prediction never needs to leave the user's device and in some sense you get scale for free right because rather than needing to think about hey how do i scale up my web service to serve the needs of all my users each of those users will bring their own hardware that will be used to run the model's predictions so you don't need to think as much about how to scale up and down the resources you need for running model inference there's some pretty pronounced cons to this approach
+
+82
+00:46:54,640 --> 00:47:32,079
+as well first of all on these edge devices you generally have very limited hardware resources available so if you're used to running every single one of your model predictions on beefy modern agpu machine you're going to be in for a bit of a shock when it comes to trying to get your model to work on the devices that you needed to work on the tools that you use to do this to make models run on limited hardware are less full featured and in many cases harder to use and more error in bug prone than the neural network libraries that you might be used to working with in tensorflow and pi torch since you need to send updated model weights to the device it can be very difficult to update models in web deployment you have
+
+83
+00:47:30,480 --> 00:48:05,520
+full control over what version of the model is deployed and so there's a bug you can roll out a fix very quickly but on the edge you need to think a lot more carefully about your strategy for updating the version of the model that your users are running on their devices because they may not always be able to get the latest model and then lastly when things do go wrong so if your if your model has is making errors or mistakes it can be very difficult to detect those errors and fix them and debug them because you don't have the raw data that's going through your models available to you as a model developer since it's all on the device of your user next we're gonna give a lightning tour of the different frameworks that you can use for doing
+
+84
+00:48:04,000 --> 00:48:40,960
+edge deployment and the right framework to pick depends both on how you train your model and what the target device you want to deploy it on is so we're not going to aim to go particularly deep on any of these options but really just to give you sort of a broad picture of what are the options you can consider as you're making this decision so we'll split this up mostly by what device you're deploying to so simplest answer is if you're deploying to an nvidia device then the right answer is probably tensor rt so whether that's like a gpu like the one you train your model on or one of the nvidia's devices that's more specially designed to deploy on the edge tensorrt tends to be a go-to option there if instead
+
+85
+00:48:38,720 --> 00:49:23,359
+you're deploying not to an nvidia device but to a phone then both android and apple have libraries for deploying neural networks on their particular os's which are good options if you know that you're only going to be deploying to an apple device or to an android device but if you're using pytorch and you want to be able to deploy both on ios and on android then you can look into pytorch mobile which compiles pi torch down into something that can be run on either of those operating systems similarly tensorflow lite aims to make tensorflow work on different mobile os's as well as well as other edge devices that are neither mobile devices nor nvidia devices if you're deploying not to a nvidia device not to a phone and not to
+
+86
+00:49:21,839 --> 00:49:58,800
+some other edge device that you might consider but deploying to the browser for reasons of performance or scalability or data privacy then tensorflow.js is probably the main example to look at here i'm not aware of a good option for deploying pytorch to the browser and then lastly you know you might be thinking why is there such a large universe of options like i need to follow this complicated decision tree to pick something that depends on the way i train my model the target device i'm deploying it to there aren't even good ways of filling in some of the cells in that graph like how do you run a pi torch model on an edge device that is not a phone for example it's maybe not super clear in that case it might be
+
+87
+00:49:56,720 --> 00:50:33,680
+worth looking into this library called apache tvm apache tvm aims to be a library agnostic and target agnostic tool for compiling your model down into something that can run anywhere the idea is build your model anywhere run it anywhere patrick tvm has some adoption but is i would say at this point still pretty far from being a standard in the industry but it's an option that's worth looking into if you need to make your models work on many different types of devices and then lastly i would say pay attention to this space i think this is another sort of pretty active area for development for machine learning startups in particular there's a startup around patchy tvm called octoml which is worth looking into and there's a new
+
+88
+00:50:32,000 --> 00:51:11,839
+startup that's built by the developers of lower level library called mlir called modular which is also aiming to solve potentially some of the problems around edge deployment as well as tinyml which is a project out of google we talked about the frameworks that you can use to actually run your model on the edge but those are only going to go so far if your model is way too huge to actually put it on the edge at all and so we need ways of creating more efficient models in a previous section we talked about quantization and distillation both of those techniques are pretty helpful for designing these types of models but there's also model architectures that are specifically designed to work well on mobile or edge
+
+89
+00:51:09,760 --> 00:51:48,559
+devices and the operative example here is mobile nets the idea of mobile nets is to take some of the expensive operations in a typical comp net like convolutional layers with larger filter sizes and replace them with cheaper operations like one by one convolutions and so it's worth checking out this mobilenet paper if you want to learn a little bit more about how mobile networks and maybe draw inspiration for how to design a mobile-friendly architecture for your problem mobile desks in particular are a very good tool for a mobile deployment they tend to not have a huge trade-off in terms of accuracy relative to larger models but they are much much smaller and easier to fit on edge devices another case study
+
+90
+00:51:46,720 --> 00:52:23,200
+that i recommend checking out is looking into distilbert distilbert is an example of model distillation that works really well to get a smaller version of bert that removes some of the more expensive operations and uses model distillation to have a model that's not much less performant than bert but takes up much less space and runs faster so to wrap up our discussion on edge deployment i want to talk a little bit about some of the sort of key mindsets for edge deployment that i've learned from talking to a bunch of practitioners who have a lot more experience than i do in deploying machine learning models on the edge the first is there's a temptation i think to finding the perfect model architecture first and then figuring out how to make
+
+91
+00:52:21,119 --> 00:52:59,599
+it work on your device and oftentimes if you're pulling on a web server you can make this work because you always have the option to scale up horizontally and so if you have a huge model it might be expensive to run but you can still make it work but on the edge practitioners believe that the best thing to do is to choose your architecture with your target hardware in mind so you should not be considering architectures that have no way of working on your device and kind of a rule of thumb is you might be able to make up for a factor of let's say an order of magnitude 2 to 10x in terms of inference time or model size through some combination of distillation quantization and other tricks but usually you're not going to get much
+
+92
+00:52:58,000 --> 00:53:33,920
+more than a 10x improvement so if your model is 100 times too large or too slow to run in your target context then you probably shouldn't even consider that architecture the next mindset is once you have one version of the model that works on your edge device you can iterate locally without needing to necessarily test all the changes that you make on that device which is really helpful because deploying and testing on the edge itself is tricky and potentially expensive but you can iterate locally once the version that you're iterating on does work as long as you only gradually add to the size of the model or the latency of the model and one thing that practitioners recommended doing that is i think a step
+
+93
+00:53:32,960 --> 00:54:09,599
+that's worth taking if you're going to do this is to add metrics or add tests for model size and latency so that if you're iterating locally and you get a little bit carried away and you double the size of your model or triple the size of your model you'll at least have a test that reminds you like hey you probably need to double check to make sure that this model is actually going to run on the device that we needed to run on another mindset that i learned from practitioners of edge supplement is to treat tuning the model for your device as an additional risk in the model deployment life cycle and test it accordingly so for example always test your models on production hardware before actually deploying them to
+
+94
+00:54:07,839 --> 00:54:45,359
+production hardware now this may seem obvious but it's not the easiest thing to do in practice and so some folks that are newer to edge deployment will skip this step the reason why this is important is because since these edge deployment libraries are immature there can often be minor differences in the way that the neural network works on your edge device versus how it works on your training device or on your laptop so it's important to run the prediction function of your model on that edge device on some benchmark data set to test both the latency as well as the accuracy of the model on that particular hardware before you deploy it otherwise the differences in how your model works on that hardware versus how it works in
+
+95
+00:54:43,280 --> 00:55:24,240
+your development environment can lead to unforeseen errors or unforeseen degradations and accuracy of your deployed model then lastly since machinery models in general can be really finicky it's a good idea to build fallback mechanisms into the application in case the model fails or you accidentally roll out a bad version of the model or the model is running too slow to solve the task for your user and these fallback mechanisms can look like earlier versions of your model much simpler or smaller models that you know are going to be reliable and run in the amount of time you need them to run in or even just like rule-based functions where if your model is taking too long to make a prediction or is erroring out
+
+96
+00:55:22,799 --> 00:55:58,000
+or something you still have something that is going to return a response to your end user so to wrap up our discussion of edge deployment first thing to remind you of is web deployment is truly much easier than edge fluid so only use edge deployment if you really need to second you'll need to choose a framework to do edge deployment and the way that you'll do this is by matching the library that you use to build your neural network and the available hardware picking the corresponding edge deployment framework that matches those two constraints if you want to be more flexible like if you want your model to be able to work on multiple devices it's worth considering something like apache tvm third start considering the
+
+97
+00:55:56,480 --> 00:56:29,440
+additional constraints that you'll get from edge deployment at the beginning of your project don't wait until you've invested three months into building the perfect model to think about whether that model is actually going to be able to run on the edge instead make sure that those constraints for your edge deployment are taken into consideration from day one and choose your architectures and your training methodologies accordingly to wrap up our discussion of deploying machine learning models fully models is a necessary step of building a machine learning power product but it's also a really useful one for making your models better because only in real life do you get to see how your model actually works on the
+
+98
+00:56:27,839 --> 00:57:03,040
+task that we really care about so the mindsets that we encourage you to have here are deploy early and deploy often so you can start collecting that feedback from the real world as quickly as possible keep it simple and add complexity only as you need to because this deployment is a can be a rabbit hole and there's a lot of complexity to deal with here so make sure that you really need that complexity so start by building a prototype then once you need to start to scale it up then separate your model from your ui by either doing bath predictions or building a model service then once the like sort of naive way that you've deployed your model stops scaling then you can either learn the tricks to scale or use a managed
+
+99
+00:57:00,559 --> 00:57:29,839
+service or a cloud provider option to handle a lot of that scaling for you and then lastly if you really need to be able to operate your model on a device that doesn't have consistent access to the internet if you have very hard data security requirements or if you really really really want to go fast then consider moving your model to the edge but be aware that's going to add a lot of complexity and force you to deal with some less mature tools when you want to do that that wraps up our lecture on deployment and we'll see you next week
+

documents/lecture-06.md

@@ -0,0 +1,809 @@
+---
+description: How to continuously improve models in production
+---
+
+# Lecture 6: Continual Learning
+
+<div align="center">
+<iframe width="720" height="405" src="https://www.youtube-nocookie.com/embed/nra0Tt3a-Oc?list=PL1T8fO7ArWleMMI8KPJ_5D5XSlovTW_Ur" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+Lecture by [Josh Tobin](https://twitter.com/josh_tobin_).
+Notes by [James Le](https://twitter.com/le_james94) and [Vishnu Rachakonda](https://www.linkedin.com/in/vrachakonda/).<br />
+Published September 12, 2022.
+[Download slides](https://fsdl.me/2022-lecture-06-slides).
+
+## 1 - Overview
+
+The core justification for continual learning is that, unlike in
+academia, we never deal with static data distributions in the real
+world. The implication is that: **if you want to use ML in production
+and build ML-powered products, you need to think about your goal of
+building a continual learning system, not just a static model**.
+
+Recalling the data flywheel that we've described in this class before:
+as you get more users, those users bring more data. You can use the data
+to make a better model. A better model helps you attract even more users
+and build a better model over time. Andrej Karpathy described the most
+optimistic version of it as "[Operation
+Vacation](https://www.youtube.com/watch?v=hx7BXih7zx8)" -
+if we make our continual learning system good enough, it'll get better
+on its own over time, and ML engineers can just go on vacation.
+
+![](./media/image6.png)
+
+The reality is quite different. Initially, we gather, clean, and label
+some data. We train a model on that data. Then we evaluate the model and
+loop back to training the model to improve it based on our evaluations.
+Finally, we get a minimum viable model and deploy it into production.
+
+![](./media/image1.png)
+
+The problem begins after we deploy the model: we generally don't have a
+great way of measuring how our models are actually performing in
+production. Often, we just spot-check some predictions to see if they
+are doing what they are supposed to do. If it seems to work, then it's
+great. We move on to work on other things.
+
+![](./media/image8.png)
+
+Unfortunately, the ML engineer is probably not the one who discovers the
+problems, to begin with. Some business user or product manager gets
+complaints from users about a dipping metric, which leads to an
+investigation. This already costs the company money because the product
+and business teams must investigate the problem.
+
+![](./media/image12.png)
+
+Eventually, they point back to the ML engineer and the model he is
+responsible for. At this point, we are stuck on doing ad-hoc analyses
+because we don't know what caused the model failure. Eventually, we can
+run a bunch of SQL queries and paste together some Jupyter notebooks to
+figure out what the problem is. If we are lucky, we can run an A/B test.
+If the test looks good, we'll deploy it into production. Then, we are
+back to where we started - **not getting ongoing feedback about how the
+model is doing in production**.
+
+The upshot is that **continual learning is the least well-understood
+part of the production ML lifecycle**. Very few companies are doing this
+in production today. This lecture focuses on how to improve different
+steps of the continual learning process, pointers to learn about each
+step, and recommendations for doing it pragmatically and adopting it
+gradually.
+
+## 2 - How to Think About Continual Learning
+
+Our opinionated view about continual learning is **training a sequence
+of models that can adapt to a continuous stream of data that comes into
+production.** You can think about continual learning as an outer loop in
+your training process. On one end of the loop is your application, which
+consists of a model and some other code that users interact with that
+application by submitting requests, getting predictions back, and
+submitting feedback about how well the model did at providing that
+prediction.
+
+The continual learning loop starts with **logging**, which is how we get
+all the data into the loop. Then we have **data curation**, **triggers**
+for the retraining process, **dataset formation** to pick the data to
+retrain on, the **training** process itself, and **offline testing** to
+validate whether the retrained model is good enough to go into
+production. After the model is deployed, we have **online testing**, and
+that brings the next version of the model into production, where we can
+start the loop all over.
+
+Each of these stages passes the output to the next step. Output is
+defined by a set of rules. These rules combine to form our **retraining
+strategy**. Let's discuss what the retraining strategy looks like for
+each stage:
+
+![](./media/image7.png)
+
+
+At the **logging** stage, the key question answered by the retraining
+strategy is **what data should we store**? At the end of this stage, we
+have an "infinite stream" of potentially unlabeled data coming from
+production and can be used for downstream analysis.
+
+![](./media/image3.png)
+
+
+At the **curation** stage, the key rules we need to define are **what
+data from that infinite stream will we prioritize for labeling and
+potential retraining?** At the end of this stage, we have a reservoir of
+candidate training points that have labels and are fully ready to be fed
+back into a training process.
+
+![](./media/image5.png)
+
+
+At the **retraining trigger** stage, the key question is **when should
+we retrain?** The output of this stage is a signal to kick off a
+retraining job.
+
+![](./media/image2.png)
+
+
+At the **dataset formation** stage, the key rules we need to define are
+**from this entire reservoir of data, what specific subset of that data
+are we using to train on for this particular training job?** The output
+of this stage is a view into that reservoir or training data that
+specifies the exact data points to be used for the training job.
+
+![](./media/image22.png)
+
+
+At the **offline testing** stage, the key rule we need to define is
+**what "good enough" looks like for all stakeholders.** The output of
+this stage is equivalent to a "pull request" report card for your model
+with a clear sign-off process. Once you are signed off, the new model
+will roll out into production.
+
+![](./media/image21.png)
+
+
+Finally, at the **deployment and online testing** stage, the key rule to
+define is **how do we know if this deployment was successful?** The
+output of this stage is a signal to roll this model out fully to all of
+your users.
+
+In an idealized world, from an ML engineer's perspective, once the model
+is deployed, the first version of the model is to not retrain the model
+directly. Instead, we want the model to sit on top of the retraining
+strategy and try to improve that strategy over time. Rather than
+training models daily, we look at metrics about how well the strategy is
+working and how well it's solving the task of improving our model over
+time in response to changes in the world. The input that we provide is
+by tuning the strategy to do a better job of solving that task.
+
+For most ML engineers, our jobs don't feel like that at a high level.
+**Our retraining strategy is just retraining models whenever we feel
+like it**. We can get good results from ad-hoc retraining, but when you
+start getting consistent results and no one is actively working on the
+model day to day anymore, then it's worth starting to add some
+automation. Alternatively, if you find yourself needing to retrain the
+model more than once a week (or even more frequently than that) to deal
+with changing results in the real world, then it's worth investing in
+automation just to save yourself.
+
+## 3 - Periodic Retraining
+
+The first baseline retraining strategy that you should consider after
+you move on from ad-hoc is just **periodic retraining**:
+
+1.  At the logging stage, we simply log everything.
+
+2.  At the curation stage, we sample uniformly at random from the data
+that we've logged up until we get the maximum number of data
+points that we are able to handle. Then we label them using some
+automated tools.
+
+3.  Our retraining trigger will just be periodic.
+
+4.  We train once a week, but we do it on the last month's data, for
+example.
+
+5.  Then we compute the test set accuracy after each training, set a
+threshold on that, or more likely manual review the results each
+time, and spot-check some of the predictions.
+
+6.  When we deploy the model, we do spot evaluations of that deployed
+model on a few individual predictions to make sure things look
+healthy.
+
+![](./media/image17.png)
+
+
+Periodic retraining won't work in every circumstance. There are several
+failure modes:
+
+1.  The first category is that you have more data than you can log or
+label. If you have a **high volume** of data, you might need to be
+more careful about what data to sample and enrich, particularly if
+that data comes from **a long-tail distribution** - where you have
+edge cases that your model needs to perform well on, but those
+edge cases might not be caught by just doing standard uniform
+sampling. Or if that data is expensive to label like in a
+**human-in-the-loop** scenario - where you need custom labeling
+rules or labeling is a part of the product. In either of those
+cases, you need to be more careful about what subset of your data
+you log and enrich to be used down the road.
+
+2.  The second category has to do with **managing the cost of
+retraining**. If your model is expensive to retrain, retraining it
+periodically is not going to be the most cost-efficient way to go,
+especially if you do it on a rolling window of data every single
+time. You will leave a lot of performance on the table by not
+retraining more frequently. You can partially solve this by
+increasing the retraining frequency, but this will increase the
+costs even further.
+
+3.  The final failure mode is situations where you have **a high cost of
+bad predictions**. Every time you retrain your model, it
+introduces risk, which comes from the fact that the data you're
+training the model on might be bad in some way. It might be
+corrupted, might have been attacked by an adversary, or might not
+be representative anymore of all the cases that your model needs
+to perform well on. The more frequently you retrain and the more
+sensitive you are to model failures, the more thoughtful you need
+to be about careful model evaluation such that you are not unduly
+taking on too much risk from frequent retraining.
+
+## 4 - Iterating On Your Retraining Strategy
+
+The main takeaway from this section is that **we will use monitoring and
+observability to determine what changes we want to make to our
+retraining strategy**.
+
+1.  We'll do that by monitoring just the metrics that actually that
+matter and using all other metrics for debugging.
+
+2.  When we debug an issue with our model, that will lead to potentially
+retraining our model. But more broadly than that, we can think of
+it as a change to the retraining strategy - changing our
+retraining triggers, our offline tests, our sampling strategies,
+the metrics for observability, etc.
+
+3.  As we get more confident in our monitoring, we can introduce more
+automation to our system.
+
+There are no real standards or best practices on model monitoring yet.
+The main principles we'll follow are: (1) We'll focus on monitoring what
+matters and what breaks empirically; and (2) We'll compute other signals
+too but use them for observability and debugging.
+
+![](./media/image13.png)
+
+
+What does it mean to monitor a model in production? We think about it
+as: You have some metric to assess the model quality (i.e, accuracy) and
+a time series of how that metric changes over time. The question you try
+to answer is: **Is this bad or okay?** Do you need to pay attention to
+this degradation or not?
+
+The questions we'll need to answer are:
+
+1.  What metrics should we be looking at when we are monitoring?
+
+2.  How can we tell if those metrics are bad and warrant an
+intervention?
+
+3.  What are the tools that help us with this process?
+
+### What Metrics to Monitor
+
+Choosing the right metric to monitor is probably the most important part
+of this process. Below you can find different types of metrics ranked in
+order of how valuable they are.
+
+![](./media/image11.png)
+
+
+#### Outcomes and Feedback From Users
+
+The most valuable one to look at is **outcome data or feedback from your
+users**. Unfortunately, there are no one-size-fits-all ways to do this
+because it depends a lot on the specifics of the product you are
+building. This is more of a product management question of how to design
+your product in a way that you can capture feedback from your users as
+part of the product experience.
+
+#### Model Performance Metrics
+
+The next most valuable signal to look at is **model performance
+metrics**. These are offline metrics such as accuracy. This is less
+useful than user feedback because of loss mismatch. A common experience
+many ML practitioners have is that improving model performance leads to
+the same or worse outcome. There's very little excuse for not doing
+this. To some degree, you can label some production data each day by
+setting up an on-call rotation or throwing a labeling party. These
+practices will give you some sense of how your model performance trends
+over time.
+
+![](./media/image10.png)
+
+
+#### Proxy Metrics
+
+The next best thing to look at is **proxy metrics**, which are
+correlated with bad model performance. These are mostly domain-specific.
+For example, if you are building text generation with a language model,
+two examples would be repetitive and toxic outputs. If you are building
+a recommendation system, an example would be the share of personalized
+responses. **Edge cases** can be good proxy metrics. If there are
+certain problems you know that you have with your model, if those
+increase in prevalence, that might mean your model is not doing very
+well.
+
+There's an academic direction that aims at being able to take any metric
+you care about and approximate it on previously unseen data. How well do
+we think our model is doing on this new data? Which would make these
+proxy metrics a lot more practically useful? There are a number of
+different approaches here: from training an auxiliary model to predict
+how well your main model might do on this offline data, to using
+heuristics and human-in-the-loop methods.
+
+![](./media/image20.png)
+
+
+An unfortunate result from this literature is that it's not possible to
+have a single method you use in all circumstances to approximate how
+your model is doing on out-of-distribution data. Let's say you are
+looking at the input data to predict how the model will perform on those
+input points. Then the label distribution changes. As a result, you
+won't be able to take into account that change in your approximate
+metric.
+
+#### Data Quality
+
+The next signal to look at is **data quality.** [Data quality
+testing](https://lakefs.io/data-quality-testing/) is a set
+of rules you apply to measure the quality of your data. This deals with
+questions such as: How well does a piece of information reflect reality?
+Does it fulfill your expectations of what's comprehensive? Is your
+information available when you need it? Some common examples include
+checking whether the data has the right schema, the data is in the
+expected range, and the number of records is not anomalous.
+
+![](./media/image19.png)
+
+This is useful because data problems tend to be the most common issue
+with ML models in practice. In [a Google
+report](https://www.usenix.org/conference/opml20/presentation/papasian)
+which covered 15 years of different pipeline outages with a particular
+ML model, most of the outages that happened with that model were
+distributed systems problems, commonly data problems.
+
+#### Distribution Drift
+
+##### Why Measure Distribution Drift?
+
+Your model's performance is only guaranteed on **data sampled from the
+same distribution** as it was trained on. This can have a huge impact in
+practice. A recent example includes changes in model behavior during the
+pandemic. A bug in the retraining pipeline caused the recommendations
+not to be updated for new users, leading to millions of dollars in
+revenue lost.
+
+##### Types of Distribution Drift
+
+Distribution drift manifests itself in different ways in the wild:
+
+1.  **Instantaneous drift** happens when a model is deployed in a new
+domain, a bug is introduced in the pre-processing pipeline, or a
+big external shift like COVID occurs.
+
+2.  **Gradual drift** happens when users\' preferences change or new
+concepts get introduced to the corpus over time.
+
+3.  **Periodic drift** happens when users' preferences are seasonal or
+people in different time zones use your model differently.
+
+4.  **Temporary drift** happens when a malicious user attacks your
+model, a new user tries your product and churns, or someone uses
+your product in an unintended way.
+
+##### How to Measure It?
+
+How to tell if your distribution is drifted?
+
+1.  Your first **select a window of "good" data to serve as a
+reference**. To select that reference, you can use a fixed window
+of production data you believe to be healthy. [Some
+papers](https://arxiv.org/abs/1908.04240) advocate
+for using a sliding window of production data. In practice, most
+of the time you probably should use your validation data as the
+reference.
+
+2.  Once you have that reference data, you **select a new window of
+production data to measure your distribution distance on**. This
+is not a super principled approach and tends to be
+problem-dependent. A pragmatic solution is to pick one or several
+window sizes with a reasonable amount of data and slide them.
+
+3.  Finally, once you have your reference window and production window,
+you **compare the windows using a distribution distance metric**.
+
+##### What Metrics To Use?
+
+Let's start by considering the one-dimensional case, where you have a
+particular feature that is one-dimensional and can compute a density of
+that feature on your reference/production windows. You want some metric
+that approximates the distance between these two distributions.
+
+![](./media/image9.png)
+
+
+There are a few options here:
+
+1.  The commonly recommended ones are the KL divergence and the KS test.
+But they are actually bad choices.
+
+2.  Sometimes-better options would be (1) infinity norm or 1-norm of the
+diff between probabilities for each category, and (2)
+Earth-mover's distance (a more statistically principled approach).
+
+Check out [this Gantry blog
+post](https://gantry.io/blog/youre-probably-monitoring-your-models-wrong/)
+to learn more about why the commonly recommended metrics are not so good
+and the other ones are better.
+
+##### Dealing with High-Dimensional Data
+
+In the real world for most models, we have potentially many input
+features or even unstructured data that is very high-dimensional. How do
+we deal with detecting distribution drift in those cases?
+
+1.  You can measure **drift on all of the features independently**: If
+you have a lot of features, you will hit [the multiple hypothesis
+testing
+problem](https://multithreaded.stitchfix.com/blog/2015/10/15/multiple-hypothesis-testing/).
+Furthermore, this doesn't capture cross-correlation.
+
+2.  You can measure **drift on only the important features**: Generally
+speaking, it's a lot more useful to measure drift on the outputs
+of the model than the inputs. You can also [rank the importance
+of your input
+features](https://christophm.github.io/interpretable-ml-book/feature-importance.html)
+and measure drift on the most important ones.
+
+3.  You can look at **metrics that natively compute or approximate the
+distribution distance between high-dimensional distributions**:
+The two that are worth checking out are [maximum mean
+discrepancy](https://jmlr.csail.mit.edu/papers/v13/gretton12a.html)
+and [approximate Earth-mover's
+distance](https://arxiv.org/abs/1904.05877). The
+caveat here is that they are pretty hard to interpret.
+
+![](./media/image14.png)
+
+A more principled way to measure distribution drift for high-dimensional
+inputs to the model is to use **projections**. The idea of a projection
+is that:
+
+1.  You first take some high-dimensional input to the model and run that
+through a function.
+
+2.  Each data point your model makes a prediction on gets tagged by this
+projection function. The goal of this projection function is to
+reduce the dimensionality of that input.
+
+3.  Once you've reduced the dimensionality, you can do drift detection
+on that lower-dimensional representation of the high-dimensional
+data.
+
+This approach works for any kind of data, no matter what the
+dimensionality is or what the data type is. It's also highly flexible.
+There are different types of projections that can be useful:
+**analytical projections** (e.g., mean pixel value, length of sentence,
+or any other function), **random projections** (e.g., linear), and
+**statistical projections** (e.g., autoencoder or other density models,
+T-SNE).
+
+##### Cons of Looking at Distribution Drift
+
+![](./media/image18.png)
+
+**Models are designed to be robust to some degree of distribution
+drift**. The figure on the left above shows a toy example to demonstrate
+this point. We have a classifier that's trained to predict two classes.
+We've induced a synthetic distribution shift to shift the red points on
+the top left to bottom. These two distributions are extremely different,
+but the model performs equally well on the training data and the
+production data. In other words, knowing the distribution shift doesn't
+tell you how the model has reacted to that shift.
+
+The figure on the right is a research project that used data generated
+from a physics simulator to solve problems on real-world robots. The
+training data was highly out of distribution (low-fidelity, random
+images). However, by training on this set of training data, the model
+was able to generalize to real-world scenarios on the test data.
+
+Beyond the theoretical limitations of measuring distribution drift, this
+is just hard to do in practice. You have to window size correctly. You
+have to keep all this data around. You need to choose metrics. You need
+to define projections to make your data lower-dimensional.
+
+#### System Metrics
+
+The last thing to consider looking at is your standard **system
+metrics** such as CPU utilization, GPU memory usage, etc. These don't
+tell you anything about how your model is actually performing, but they
+can tell you when something is going wrong.
+
+#### Practical Recommendations
+
+We also want to look at how hard it is to compute the aforementioned
+stages in practice. As seen below, the Y-axis shows the **value** of
+each signal and the X-axis shows the **feasibility** of measuring each
+signal.
+
+1.  Measuring outcomes or feedback has pretty wide variability in terms
+of how feasible it is to do, as it depends on how your product is
+set up.
+
+2.  Measuring model performance tends to be the least feasible thing to
+do because it involves collecting some labels.
+
+3.  Proxy metrics are easier to compute because they don't involve
+labels.
+
+4.  System metrics and data quality metrics are highly feasible because
+you have off-the-shelf tools for them.
+
+![](./media/image15.png)
+
+
+Here are our practical recommendations:
+
+1.  Basic data quality checks are zero-regret, especially if you are
+retraining your model.
+
+2.  Get some way to measure feedback, model performance, or proxy
+metrics, even if it's hacky or not scalable.
+
+3.  If your model produces low-dimensional outputs, monitoring those for
+distribution shifts is also a good idea.
+
+4.  As you evolve your system, practice the **observability** mindset.
+
+While you can think of monitoring as measuring the known unknowns (e.g.,
+setting alerts on a few key metrics), [observability is measuring
+unknown
+unknowns](https://www.honeycomb.io/blog/observability-a-manifesto/)
+(e.g., having the power to ask arbitrary questions about your system
+when it breaks). An observability mindset means two implications:
+
+1.  You should keep around the context or raw data that makes up the
+metrics that you are computing since you want to be able to drill
+all the way down to potentially the data points themselves that
+make up the degraded metric.
+
+2.  You can go crazy with measurement by defining a lot of different
+metrics. You shouldn't necessarily set alerts on each of those
+since you don't want too many alerts. Drift is a great example
+since it is useful for debugging but less so for monitoring.
+
+Finally, it's important to **go beyond aggregate metrics**. If your
+model is 99% accurate in aggregate but only 50% accurate for your most
+important user, is it still "good"? The way to deal with this is by
+flagging important subgroups or cohorts of data and alerting on
+important metrics across them. Some examples are categories you don't
+want to be biased against, "important" categories of users, and
+categories you might expect to perform differently on (languages,
+regions, etc.).
+
+### How To Tell If Those Metrics are "Bad"
+
+We don't recommend statistical tests (e.g., KS-Test) because they try to
+return a p-value for the likelihood that the data distributions are not
+the same. When you have a lot of data, you will get very small p-values
+for small shifts. This is not what we actually care about since models
+are robust to a small number of distribution shifts.
+
+Better options than statistical tests include fixed rules, specific
+ranges, predicted ranges, and unsupervised detection of new patterns.
+[This article on dynamic data
+testing](https://blog.anomalo.com/dynamic-data-testing-f831435dba90?gi=fb4db0e2ecb4)
+has the details.
+
+![](./media/image16.png)
+
+### Tools for Monitoring
+
+The first category is **system monitoring** tools, a premature category
+with different companies in it
+([Datadog](https://www.datadoghq.com/),
+[Honeycomb](https://www.honeycomb.io/), [New
+Relic](https://newrelic.com/), [Amazon
+CloudWatch](https://aws.amazon.com/cloudwatch/), etc.).
+They help you detect problems with any software system, not just ML
+models. They provide functionality for setting alarms when things go
+wrong. Most cloud providers have decent monitoring solutions, but if you
+want something better, you can look at monitoring-specific tools to
+monitor anything.
+
+This raises the question of whether we should just use these system
+monitoring tools to monitor ML metrics as well. [This blog
+post](https://www.shreya-shankar.com/rethinking-ml-monitoring-3/)
+explains that it's feasible but highly painful due to many technical
+reasons. Thus, it's better to use ML-specific tools.
+
+Two popular open-source monitoring tools are
+[EvidentlyAI](https://github.com/evidentlyai) and
+[whylogs](https://github.com/whylabs/whylogs).
+
+-   Both are similar in that you provide them with samples of data and
+they produce a nice report that tells you where their distribution
+shifts are.
+
+-   The big limitation of both is that they don't solve the data
+infrastructure and the scale problem. You still need to be able to
+get all that data into a place where you can analyze it with these
+tools.
+
+-   The main difference between them is that whylogs is more focused on
+gathering data from the edge by aggregating the data into
+statistical profiles at inference time. You don't need to
+transport all the data from your inference devices back to your
+cloud.
+
+![](./media/image4.png)
+
+Lastly, there are a bunch of different SaaS vendors for ML monitoring
+and observability: [Gantry](https://gantry.io/),
+[Aporia](https://www.aporia.com/),
+[Superwise](https://superwise.ai/),
+[Arize](https://arize.com/),
+[Fiddler](https://www.fiddler.ai/),
+[Arthur](https://arthur.ai/), etc.
+
+
+## 5 - Retraining Strategy
+
+We’ve talked about monitoring and observability, which allow you to identify issues with your continual learning system. Now, we’ll talk about how we will fix the various stages of the continual learning process based on what we learn from monitoring and observability.
+
+
+### Logging
+
+The first stage of the continual learning loop is **logging**. As a reminder, the goal of logging is to get data from your model to a place where you can analyze it. The key question to answer here is: “**what data should I actually log?**”
+
+For most of us, the best answer is just to log all of the data. Storage is cheap. It's better to have data than not to have it. There are, however, some situations where you can't do that. For example, if you have too much traffic going through your model to the point where it's too expensive to log all of it, or if you have data privacy concerns, or if you're running your model at the edge, you simply may not be able to able to log all your data. 
+
+In these situations, there are two approaches that you can take. The first approach is **profiling**. With profiling, rather than sending all the data back to your cloud and then using that to monitor, you instead compute **statistical profiles** of your data on the edge that describe the data distribution that you're seeing. This is great from a data security perspective because it doesn't require you to send all the data back home. It minimizes your storage cost. Finally, you don't miss things that happen in the tails, which is an issue for the next approach. That'll describe the place to use. This approach is best used for security-critical applications. Computing statistical profiles is a pretty interesting topic in computer science and data summarization that is worth checking out if you’re interested in this approach. 
+
+![alt_text](./media/image22.png "image_tooltip")
+
+
+The other approach is **sampling**. With sampling, you'll just take certain data points and send those back to your monitoring and logging system. The advantage of sampling is that it has minimal impact on your inference resources. You don't have to actually spend the computational budget to compute profiles. You also get to have access to the raw data for debugging and retraining, albeit a smaller amount. This is the approach we recommend for any other kind of application.
+
+
+### Curation
+
+The next step in the continual learning loop is **curation**. The goal of curation is to take the infinite stream of production data, which is potentially unlabeled, and turn it into a finite reservoir of enriched data suitable for training. Here, we must answer, “**what data should be enriched?**”
+
+You could **sample and enrich data randomly**, but that may not prove helpful to your model. Importantly, you miss rare classes or events. A better approach can be to perform **stratified subsampling**, wherein you sample specific proportions of individuals from various subpopulations (e.g. race). The most advanced strategy for picking data to enrich is to **curate data points** that are somehow interesting for the purpose of improving your model. 
+
+There are a few different ways of doing this: **user-driven curation loops** via feedback loops, **manual curation** via error analysis, and **automatic curation** via active learning. 
+
+User-driven curation is a great approach that is easy to implement, assuming you have a clear way of gathering user feedback. If your user churns, clicks thumbs down, or performs some other similar activity on the model’s output, you have an easy way of understanding data that could be enriched for future training jobs.
+
+![alt_text](./media/image23.png "image_tooltip")
+
+If you don't have user feedback, or if you need even more ways of gathering interesting data from your system, the second most effective way is by doing **manual error analysis**. In this approach, we look at the errors that our model is making, reason about the different types of failure modes that we're seeing, and try to write functions or rules that help capture these error modes. We'll use those functions to gather more data that might represent those error cases. Some examples of these function-based approaches are **similarity-based curation**, which uses nearest neighbors, and **projection-based curation**, wherein we train a new function or model to recognize key data points.
+
+The last way to curate data is to do so automatically using a class of algorithms called **[active learning](https://lilianweng.github.io/posts/2022-02-20-active-learning/)**. The way active learning works is that, given a large amount of unlabeled data, we will try to determine which data points would improve model performance the most (if you were to label those data points next and train on them). These algorithms define **a sampling strategy**, rank all of your unlabeled examples using **a scoring function** that defines the sampling strategy, and mark the data points with the highest scores for future labeling. 
+
+There are a number of different scoring function approaches that are shown below.
+
+
+
+1. **Most uncertain**: sample low-confidence and high-entropy predictions or predictions that an ensemble disagrees on.
+2. **Highest predicted loss**: train a separate model that predicts loss on unlabeled points, then sample the highest predicted loss.
+3. **Most different from labels**: train a model to distinguish labeled and unlabeled data, then sample the easiest to distinguish.
+4. **Most representative**: choose points such that no data is too far away from anything we sampled.
+5. **Big impact on training**: choose points such that the expected gradient is large or points where the model changes its mind the most about its prediction during training.
+
+Uncertainty scoring tends to be the most commonly used method since it is simple and easy to implement. 
+
+You might have noticed that there's a lot of similarity between some of the ways that we do data curation and the way that we do monitoring. That's no coincidence--**monitoring and data curation are two sides of the same coin!** They're both interested in solving the problem of finding data points where the model may not be performing well or where we're uncertain about how the model is performing on those data points.
+
+![alt_text](./media/image24.png "image_tooltip")
+
+Some examples of people practically applying data curation are OpenAI’s DALL-E 2, which uses [active learning and manual curation](https://openai.com/blog/dall-e-2-pre-training-mitigations/), Tesla, which uses [feedback loops and manual curation](https://www.youtube.com/watch?v=hx7BXih7zx8), and Cruise, which uses feedback loops.
+
+Some tools that help with data curation are [Scale Nucleus](https://scale.com/nucleus), [Aquarium](https://www.aquariumlearning.com/), and [Gantry](https://gantry.io/). 
+
+To summarize then, here are our final set of recommendations for applying data curation.
+
+
+
+1. Random sampling is a fine starting point. If you want to avoid bias or have rare classes, do stratified sampling instead.
+2. If you have a feedback loop, then user-driven curation is a no-brainer. If not, confidence-based active learning is easy to implement.
+3. As your model performance increases, you’ll have to look harder for challenging training points. Manual techniques are unavoidable and should be embraced. Know your data!
+
+
+### Retraining Triggers
+
+After we've curated our infinite stream of unlabeled data down to a reservoir of labeled data that's ready to potentially train on, the next thing that we'll need to decide is “**what trigger are we gonna use to retrain?**”
+
+The main takeaway here is that moving to automated retraining is **not** always necessary. In many cases, just manually retraining is good enough. It can save you time and lead to better model performance. It's worth understanding when it makes sense to actually make the harder move to automated retraining.
+
+The main prerequisite for moving to automated retraining is being able to reproduce model performance when retraining in a fairly automated fashion. If you're able to do that and you are not really working on the model actively, it's probably worth implementing some automated retraining. As a rule of thumb, if you’re retraining the model more than once a month, automated retraining may make sense.
+
+When it's time to move to automated training, the main recommendation is to just keep it simple and **retrain periodically**, e.g. once a week. The main question though is, how do you pick the right training schedule? The recommendation here is to:
+
+
+
+1. Apply measurement to figure out a reasonable retraining schedule.
+2. Plot your model performance and degradation over time.
+3. Compare how retraining the model at various intervals would have resulted in improvements to its performance.
+
+As seen below, the area between the curves represents the opportunity cost, so always remember to balance the upside of retraining with the operational costs of retraining.
+
+![alt_text](./media/image25.png "image_tooltip")
+
+This is a great area for future academic research! More specifically, we can look at ways to automate determining the optimal retraining strategy based on performance decay, sensitivity to performance, operational costs, and retraining costs.
+
+An additional option for retraining, rather than time-based intervals, is **performance triggers** (e.g. retrain when the model accuracy dips below 90%). This helps react more quickly to unexpected changes and is more cost-optimal, but requires very good instrumentation to process these signals along with operational complexity.
+
+An idea that probably won't be relevant but is worth thinking about is **online learning**. In this paradigm, you train on every single data point as it comes in. It's not very commonly used in practice. 
+
+A version of this idea that is used fairly frequently in practice is **online adaptation**. This method operates not at the level of retraining the whole model itself but rather on the level of adapting the policy that sits on top of the model. What is a policy you ask? A policy is the set of rules that takes the raw prediction that the model made, like the score or the raw output of the model, and turns it into the output the user sees. In online adaptation, we use algorithms like multi-armed bandits to tune these policies. If your data changes very frequently, it is worth looking into this method.
+
+
+### Dataset Formation
+
+Imagine we've fired off a trigger to start a new training job. The next question we need to answer is, among all of the labeled data in our reservoir of data, **what specific data points should we train on for this particular new training job?**
+
+We have four options here. Most of the time in deep learning, we'll just use the first option and **train on all the data that we have available** to us. Remember to keep your data version controlled and your curation rules consistent.
+
+![alt_text](./media/image26.png "image_tooltip")
+
+If you have too much data to do that, you can use recency as a heuristic for a second option and **train on only a sliding window of the most recent data** (if recency is important) or **sample a smaller portion** (if recency isn’t). In the latter case, compare the aggregate statistics between the old and new windows to ensure there aren’t any bugs. It’s also important in both cases to compare the old and new datasets as they may not be related in straightforward ways. 
+
+![alt_text](./media/image27.png "image_tooltip")
+
+A useful third option is **online batch selection**, which can be used when recency doesn’t quite matter. In this method, we leverage label-aware selection functions to choose which items in mini-batches to train on. 
+
+![alt_text](./media/image28.png "image_tooltip")
+
+A more difficult fourth option that isn’t quite recommended is **continual fine-tuning**. Rather than retraining from scratch every single time, you train your existing model on just new data. The reason why you might wanna do this primarily is because it's much more cost-effective. The paper below shares some findings from GrubHub, where they found a 45x cost improvement by doing this technique relative to sliding windows.
+
+![alt_text](./media/image29.png "image_tooltip")
+
+The big challenge here is that unless you're very careful, it's easy for the model to forget what it learned in the past. The upshot is that you need to have mature evaluation practices to be very careful that your model is performing well on all the types of data that it needs to perform well on.
+
+
+### Offline Testing
+
+After the previous steps, we now have a new candidate model that we think is ready to go into production. The next step is to test that model. The goal of this stage is to produce a report that our team can sign off on that answers the question of whether this new model is good enough or whether it's better than the old model. The key question here is, “**what should go into that report?**”
+
+This is a place where there's not a whole lot of standardization, but the recommendation we have here is to compare your current model with the previous version of the model on all of the metrics that you care about, all of the subsets of data that you've flagged are important, and all the edge cases you’ve defined. Remember to adjust the comparison to account for any sampling bias.
+
+Below is a sample comparison report. Note how the validation set is broken out into concrete subgroups. Note also how there are specific validation sets assigned to common error cases.
+
+![alt_text](./media/image30.png "image_tooltip")
+
+In continual learning, evaluation sets are dynamically refined just as much as training sets are. Here are some guidelines for how to manage evaluation sets in a continual learning system:
+
+
+
+1. As you curate new data, add some of it to your evaluation sets. For example, if you change how you do sampling, add that newly sampled data to your evaluation set. Or if you encounter a new edge case, create a test case for it.
+2. Corollary 1: you should version control your evaluation sets as well.
+3. Corollary 2: if your data changes quickly, always hold out the most recent data for evaluation.
+
+Once you have the testing basics in place, a more advanced option that you can look into here is **expectation testing**. Expectation tests work by taking pairs of examples where you know the relationship between the two. These tests help a lot with understanding the generalizability of models.
+
+![alt_text](./media/image31.png "image_tooltip")
+
+Just like how data curation is highly analogous to monitoring, so is offline testing. We want to observe our metrics, not just in aggregate but also across all of our important subsets of data and across all of our edge cases. One difference between these two is that **you will have different metrics available in offline testing and online testing**. For example, you’re much more likely to have labels offline. Online, you’re much more likely to have feedback. We look forward to more research that can predict online metrics from offline ones.
+
+
+### Online Testing
+
+Much of this we covered in the last lecture, so we’ll keep it brief! Use shadow mode and A/B tests, roll out models gradually, and roll back models if you see issues during rollout. 
+
+
+## 6 - The Continual Improvement Workflow
+
+To tie it all together, we’ll conclude with an example. Monitoring and continual learning are two sides of the same coin. We should be using the signals that we monitor to very directly change our retraining strategy. This section describes the future state that comes as a result of investing in the steps laid out previously. 
+
+Start with a place to store and version your strategy. The components of your continual learning strategy should include the following:
+
+
+
+* Inputs, predictions, user feedback, and labels.
+* Metric definitions for monitoring, observability, and offline testing.
+* Projection definitions for monitoring and manual data curation.
+* Subgroups and cohorts of interest for monitoring and offline testing.
+* Data curation logic.
+* Datasets for training and evaluation.
+* Model comparison reports.
+
+Walk through this example to understand how changes to the retraining strategy occur as issues surface in our machine learning system.
+
+![alt_text](./media/image32.png "image_tooltip")
+
+## 7 - Takeaways
+
+To summarize, continual learning is a nascent, poorly understood topic that is worth continuing to pay attention to. Watch this space! In this lecture, we focused on all the steps and techniques that allow you to use retraining effectively. As MLEs, leverage monitoring to strategically improve your model. Always start simple, and get better!

documents/lecture-06.srt

@@ -0,0 +1,440 @@
+1
+00:00:00,080 --> 00:00:38,960
+hi everybody welcome back to full stack deep learning this week we're going to talk about continual learning which is in my opinion one of the most exciting topics that we cover in this class continual learning describes the process of iterating on your models once they're in production so using your production data to retrain your models for two purposes first to adapt your model to any changes in the real world that happen after you train your model and second to use data from the real world to just improve your model in general so let's dive in the sort of core justification for continual learning is that unlike in academia in the real world we never deal with static data distributions and so the implication of
+
+2
+00:00:36,719 --> 00:01:14,640
+that is if you want to use ml in production if you want to build a good machine learning powered product you need to think about your goal as building a continual learning system not just building a static model so i think how we all hope this would work is the data flywheel that we've described in this class before so as you get more users those users bring more data you can use the data to make better model that better model helps you attract even more users and build a better model over time and the most automated version of this the most optimistic version of it was described by andre karpathy as operation vacation if we make our continual learning system good enough then it'll just get better on its own
+
+3
+00:01:12,880 --> 00:01:47,920
+over time and we as machine learning engineers can just go on vacation and when we come back the model will be better but the reality of this is actually quite different i think it starts out okay so we gather some data we clean and label that data we train a model on the data then we evaluate the model we loop back to training the model to make it better based on the evaluations that we made and finally we get to the point where we're done we have a minimum viable model and we're ready to ship it into production and so we deploy it the problem begins after we deploy it which is that we generally don't really have a great way of measuring how our models are actually performing in production so often what
+
+4
+00:01:46,159 --> 00:02:21,440
+we'll do is we'll just spot check some predictions to see if it looks like it's doing what it's supposed to be doing and if it seems to be working then that's great we probably move on and work on some other project that is until the first problem pops up and now unfortunately i as a machine learning engineer and probably not the one that discovers that problem to begin with it's probably you know some business user or some pm that realizes that hey we're getting complaints from a user or we're having a metric that's dipped and this leads to an investigation this is already costing the company money because the product and the business team are having to investigate this problem eventually they are able to
+
+5
+00:02:19,440 --> 00:02:53,280
+point this back to me and to the model that i am responsible for and at this point you know i'm kind of stuck doing some ad hoc analyses because i don't really know what the cause of the model of the failure of the model is maybe i haven't even looked at this model for a few weeks or a few months you know maybe eventually i'm able to like run a bunch of sql queries you know paste together some jupiter notebooks and figure out what i think the problem is so i'll retrain the model i'll redeploy it and if we're lucky we can run an a b test and if that a b test looks good then we'll deploy it into production and we're sort of back where we started not getting ongoing feedback about how the model is really doing in production the
+
+6
+00:02:51,519 --> 00:03:29,040
+upshot of all this is that continual learning is really the least well understood part of the production machine learning lifecycle and very few companies are actually doing this well in production today and so this lecture in some ways is going to feel a little bit different than some of the other lectures a big part of the focus of this lecture is going to be being opinionated about how we think you should think about the structure of continual learning problems this is you know some of what we say here will be sort of well understood industry best practices and some of it will be sort of our view on what we think this should look like i'm going to throw a lot of information at you about each of the different steps of the
+
+7
+00:03:27,120 --> 00:04:06,000
+continual learning process how to think about improving how you do these parts once you have your first model in production and like always we'll provide some recommendations for how to do this pragmatically and how to adopt it gradually so first i want to give sort of an opinionated take on how i think you should think about continual learning so i'll define continual learning as training a sequence of models that is able to adapt to a continuous stream of data that's coming in in production you can think about continual learning as an outer loop on your training process on one end of the loop is your application which consists of a model as well as some other code users interact with that application by submitting
+
+8
+00:04:03,840 --> 00:04:41,440
+requests getting predictions back and then submitting feedback about how well the model did at providing that prediction the continual learning loop starts with logging which is how we get all the data into the loop then we have data curation triggers for doing the retraining process data set formation to pick the data to actually retrain on and we have the training process itself then we have offline testing which is how we validate whether the retrained model is good enough to go into production after it's deployed we have online testing and then that brings the next version of the model into production where we can start the loop all over again each of these stages passes an output to the next step and the way that output is defined is by
+
+9
+00:04:39,040 --> 00:05:17,360
+using a set of rules and all these rules together roll up into something called a retraining strategy next we'll talk about what the retraining strategy defines for each stage and what the output looks like so the logging stage the key question that's answered by the retraining strategy is what data should we actually store and at the end of this we have an infinite stream of potentially unlabeled data that's coming from production and is able to be used for downstream analysis at the curation stage the key rules that we need to define are what data from that infinite stream are we going to prioritize for labeling and potential retraining and at the end of the stage we'll have a reservoir of a finite number of
+
+10
+00:05:15,919 --> 00:05:52,880
+candidate training points that have labels and are fully ready to be fed back into a training process at the retraining trigger stage the key question to answer is when should we actually retrain how do we know when it's time to hit the retrain button and the output of the stage is a signal to kick off a retraining job at the data set formation stage the key rules we need to define are from among this entire reservoir of data what specific subset of that data are we actually going to train on for this particular training job you can think of the output of this as a view into that reservoir of training data that specifies the exact data points that are going to go into this training job at the offline testing
+
+11
+00:05:50,960 --> 00:06:28,800
+stage the key rules that we need to define are what is good enough look like for all of our stakeholders how are we going to agree that this model is ready to be deployed and the output of the stage looks like something like the equivalent of a pull request a report card for your model that has a clear sign-off process that once you're signed off the new model will roll out into prod and then finally at the deployment online testing stage the key rules that we need to find are how do we actually know if this deployment was successful and the output of the stage will be the signal to actually roll this model out fully to all of your users in an idealized world the way i think we should think of our role as machine
+
+12
+00:06:26,720 --> 00:07:07,360
+learning engineers once we've deployed the first version of the model is not to retrain the model directly but it's to sit on top of the retraining strategy and babysit that strategy and try to improve the strategy itself over time so rather than training models day-to-day we're looking at metrics about how well the strategy is working how well it's solving the task of improving our model over time in response to changes to the world and the input that we provide is by tuning the strategy by changing the rules that make up the strategy to help the strategy do a better job of solving that task that's a description of the goal state of our role as an ml engineer in the real world today for most of us our job doesn't really feel like this at
+
+13
+00:07:05,520 --> 00:07:39,759
+a high level because for most of us our retraining strategy is just retraining models whenever we feel like it and that's not actually as bad as it seems you can get really good results from ad hoc retraining but when you start to be able to get really consistent results when you retrain models and you're not really working on the model day-to-day anymore then it's worth starting to add some automation alternatively if you find yourself needing to retrain the model more than you know once a week or even more frequently than that to deal with changing results in the real world then it's also worth investing in automation just to save yourself time the first baseline retraining strategy that you should consider after you move
+
+14
+00:07:38,240 --> 00:08:18,479
+on from ad hoc is just periodic retraining and this is what you'll end up doing in most cases in the near term so let's describe this periodic retraining strategy so at the logging stage we'll simply log everything a curation will sample uniformly at random from the data that we've logged up until we get the max number of data points that we're able to handle we're able to label or we're able to train on and then we'll label them using some automated tool our retraining trigger will just be periodic so we'll train once a week but we'll do it on the last month's data for example and then we will compute the test set accuracy after each training set a threshold on that or more likely manually review the results each time and spot check some of
+
+15
+00:08:16,400 --> 00:08:51,600
+the predictions and then when we deploy the model we'll do spot evaluations of that deployed model on a few individual predictions just to make sure things look healthy and we'll move on this baseline looks something like what most companies do for automated retraining in the real world retraining periodically is a pretty good baseline and in fact it's what i would suggest doing when you're ready to start doing automated retraining but it's not going to work in every circumstance so let's talk about some of the failure modes the first category of failure modes has to do with when you have more data than you're able to log or able to label if you have a high volume of data you might need to be more careful about what data you sample
+
+16
+00:08:48,640 --> 00:09:30,160
+and enrich particularly if either that data comes from a long tail distribution where you have edge cases that your model needs to perform well on but those edge cases might not be caught by just doing standard uniform random sampling or if that data is expensive to label like in a human in the loop scenario where you need custom labeling rules or labeling is part of the product in either of those cases long tail distribution or human in the loop setup you probably need to be more careful about what subset of your data that you log and enrich to be used down the road second category of where this might fail has to do with managing the cost of retraining if your model is really expensive to retrain then
+
+17
+00:09:28,720 --> 00:10:01,600
+retraining it periodically is probably not going to be the most cost efficient way to go especially if you do it on a rolling window of data every single time let's say that you retrain your model every week but your data actually changes a lot every single day you're going to be leaving a lot of performance on the table by not retraining more frequently you could increase the frequency and retrain say every few hours but this is going to increase costs even further the final failure mode is situations where you have a high cost of bad predictions one thing that you should think about is every single time you retrain your model it introduces risk that risk comes from the fact that the data that you're training
+
+18
+00:10:00,320 --> 00:10:38,480
+the model on might be bad in some way it might be corrupted it might have been attacked by an adversary or it might just not be representative anymore of all the cases that your model needs to perform well on so the more frequently you retrain and the more sensitive you are to failures of the model the more thoughtful you need to be about how do we make sure that we're carefully evaluating this model such that we're not unduly taking on risk too much risk from retraining frequently when you're ready to move on from periodic retraining it's time to start iterating on your strategy and this is the part of the lecture where we're going to cover a grab box of tools that you can use to help figure out how to iterate on your strategy and what
+
+19
+00:10:36,959 --> 00:11:10,320
+changes the strategy to make you don't need to be familiar in depth with every single one of these tools but i'm hoping to give you a bunch of pointers here that you can use when it's time to start thinking about how to make your model better so the main takeaway from this section is going to be we're going to use monitoring and observability as a way of determining what changes we want to make to our retraining strategy and we're going to do that by monitoring just the metrics that actually matter the most important ones for us to care about and then using all of their metrics and information for debugging when we debug an issue with our model that's going to lead to potentially retraining our model but more broadly
+
+20
+00:11:09,040 --> 00:11:45,760
+than that we can think of it as a change to the retraining strategy like changing our retraining triggers changing our offline tests our sampling strategies the metrics we use for observability etc and then lastly another principle for iterating on your strategy is as you get more confident in your monitoring as you get more confident that you'll be able to catch issues with your model if they occur then you can start to introduce more automation into your system so do things manually at first and then as you get more confident in your monitoring start to automate them let's talk about how to monitor and debug models in production so that we can figure out how to improve our retraining strategy the tldr here is like many parts of this
+
+21
+00:11:43,839 --> 00:12:18,160
+lecture there's no real standards or best practices here yet and there's also a lot of bad advice out there the main principles that we're gonna follow here are we're gonna focus on monitoring things that really matter and also things that tend to break empirically and we're going to also compute all the other signals that you might have heard of data drift all these other sorts of things but we're primarily going to use those for debugging and observability what does it mean to monitor a model in production the way i think about it is you have some metric that you're using to assess the quality of your model like your accuracy let's say then you have a time series of how that metric changes over time and the question that you're
+
+22
+00:12:16,240 --> 00:12:51,839
+trying to answer is is this bad or is this okay do i need to pay attention to this degradation or do i not need to pay attention so the questions that we'll need to answer are what metrics should we be looking at when we're doing monitoring how can we tell if those metrics are bad and it warrants an intervention and then lastly we'll talk about some of the tools that are out there to help you with this process choosing the right metric to monitor is probably the most important part of this process and here are the different types of metrics or signals that you can look at ranked in order of how valuable they are if you're able to get them the most valuable thing that you can look at is outcome data or feedback from your users
+
+23
+00:12:50,160 --> 00:13:26,079
+if you're able to get access to this signal then this is by far the most important thing to look at unfortunately there's no one-size-fits-all way to do this because it just depends a lot on the specifics of the product that you're building for example if you're building a recommender system then you might measure feedback based on did the user click on the recommendation or not but if you're building a self-driving car that's not really a useful or even feasible signal to gather so you might instead gather data on whether the user intervened and grabbed the wheel to take over autopilot from the car and this is really more of a product design or product management question of how can you actually design your product in such
+
+24
+00:13:24,480 --> 00:13:59,279
+a way that it that you're able to capture feedback from your users as part of that product experience and so we'll come back and talk a little bit more about this in the ml product management lecture the next most valuable signal to look at if you can get it is model performance metrics these are your offline model metrics things like accuracy the reason why this is less useful than user feedback is because of loss mismatch so i think a common experience that many ml practitioners have is you spend let's say a month trying to make your accuracy one or two percentage points better and then you deploy the new version of the model and it turns out that your users don't care they react just the same way they did
+
+25
+00:13:57,360 --> 00:14:35,120
+before or even worse to that new theoretically better version of the model there's often very little excuse for not doing this at least to some degree you can just label some production data each day it doesn't have to be a ton you can do this by setting up an on-call rotation or just throwing a labeling party each day where you spend 30 minutes with your teammates you know labeling 10 or 20 data points each even just like that small amount will start to give you some sense of how your model's performance is trending over time if you're not able to measure your actual model performance metrics then the next best thing to look at are proxy metrics proxy metrics are metrics that are just correlated with bad model performance
+
+26
+00:14:33,199 --> 00:15:07,279
+these are mostly domain specific so for example if you're building text generation with a language model then two examples here would be repetitive outputs and toxic outputs if you're building a recommender system then an example would be the share of personalized responses if you're seeing fewer personalized responses then that's probably an indication that your model is doing something bad if you're looking for ideas for proxy metrics edge cases can be good proxy metrics if there's certain problems that you know that you have with your model if those increase in prevalence then that might mean that your model's not doing very well that's the practical side of proxy metrics today they're very domain specific
+
+27
+00:15:06,000 --> 00:15:44,720
+either you're going to have good proxy metrics or you're not but i don't think it has to be that way there's an academic direction i'm really excited about that is aimed at being able to take any metric that you care about like your accuracy and approximate it on previously unseen data so how well do we think our model is doing on this new data which would make these proxy metrics a lot more practically useful there's a number of different approaches here ranging from training an auxiliary model to predict how well your main model might do on these on this offline data to heuristics to human loop methods and so it's worth checking these out if you're interested in seeing how people might do this in two or three years one unfortunate
+
+28
+00:15:43,519 --> 00:16:21,440
+result from this literature though that's worth pointing out is that it's probably not going to be possible to have a single method that you use in all circumstances to approximate how your model is doing on out of distribution data so the way to think about that is let's say that you have you're looking at the input data to predict how the model is going to perform on those input points and then the label distribution changes if you're only looking at the input points then how would you be able to take into account that label distribution change in your approximate metric but there's more theoretical rounding for this result as well all right back to our more pragmatic scheduled programming the next signal that you can look at is data
+
+29
+00:16:19,519 --> 00:16:59,680
+quality and data quality testing is just a set of rules that you can apply to measure the quality of your data this is dealing with questions like how well does the data reflect reality um how comprehensive is it and how consistent is it over time some common examples of data quality testing include checking whether the data has the right schema whether the values in each of the columns are in the range that you'd expect that you have enough columns that you don't have too much missing data simple rules like that the reason why this is useful is because data problems tend to be the most common issue with machine learning models in practice so this is a report from google where they covered 15 years of different pipeline outages
+
+30
+00:16:57,600 --> 00:17:37,840
+with a particular machine learning model and their main finding was that most of the outages that happened with that model did not really a lot to do with ml at all they were often distributed systems problems or also really commonly there were data problems one example that they give is a common type of failure where a data pipeline lost the permissions to read the data source that it depended on and so was starting to fail so these types of data issues are often what will cause models to fail spectacularly in production the next most helpful signal to look at is distribution drift even though distribution drift is a less useful signal than say user feedback it's still really important to be able to measure whether your data
+
+31
+00:17:35,840 --> 00:18:17,679
+distributions change so why is that well your model's performance is only guaranteed if the data that it's evaluated on is sampled from the same distribution as it was trained on and this can have a huge impact in practice recent examples include total change in model behavior during the pandemic as words like corona took on new meeting or bugs and retraining pipelines that cause millions of dollars of losses for companies because they led to changing data distributions distribution drift manifests itself in different ways in the wild there's a few different types that you might see so you might have an instantaneous drift like when a model is deployed in a new domain or a bug is introduced in a re in a pre-processing
+
+32
+00:18:15,039 --> 00:18:53,120
+pipeline or some big external shift like covid you could have a gradual drift like if user preferences change over time or new concepts keep getting added to your corpus you could have periodic drifts like if your user preferences are seasonal or you could have a temporary drift like if a malicious user attacks your model and each of these different types of drifts might need to be detected in slightly different ways so how do you tell if your distribution is drifted the approach we're going to take here is we're going to first select a window of good data that's going to serve as a reference going forward how do you select that reference well you can use a fixed window of production data that you believe to be healthy so
+
+33
+00:18:51,200 --> 00:19:30,080
+if you think that your model was really healthy at the beginning of the month you can use that as your reference window some papers advocate for sliding this window of production data to use as your reference but in practice most of the time what most people do is they'll use something like their validation data as their reference once you have that reference data then you'll select your new window of production data to measure your distribution distance on there isn't really a super principal approach for how to select the window of data to measure drift on and it tends to be pretty problem specific so a pragmatic solution that what a lot of people do is they'll just pick one window size or even they'll just pick a few window
+
+34
+00:19:27,679 --> 00:20:08,160
+sizes with some reasonable amount of data so that's not too noisy and then they'll just slide those windows and lastly once you have your reference window and your production window then you'll compare these two windows using a distribution distance metric so what metrics should you use let's start by considering the one-dimensional case where you have a particular feature that is one-dimensional and you are able to compute a density of that feature on your reference window and your production window then the way to think about this problem is you're going to have some metric that approximates the distance between these two distributions there's a few options here the ones that are commonly recommended are the kl divergence and
+
+35
+00:20:06,160 --> 00:20:39,679
+the ks test unfortunately those are commonly recommended but they're also bad choices sometimes better options would be things like using the infinity norm or the one norm which are what google advocates for using or the earth mover's distance which is a bit more of a statistically principled approach and i'm not going to go into details of these metrics here but check out the blog post at the bottom if you want to learn more about why the commonly recommended ones are not so good and the other ones are better so that's the one-dimensional case if you just have a single input feature that you're trying to measure distribution distance on but in the real world for most models we have potentially many input features or
+
+36
+00:20:37,840 --> 00:21:13,919
+even unstructured data that is very high dimensional so how do we deal with detecting distribution drift in those cases one thing you could consider doing is just measuring drifts on all of the features independently problem that you'll run into there is if you have a lot of features you're going to hit the multiple hypothesis testing problem and secondly this doesn't capture cross correlation so if so if you have two features and the distributions of each of those features stay the same but the correlation between the features changed then that wouldn't be captured using this type of system another common thing to do would be to measure drift only on the most important features one heuristic here is that generally
+
+37
+00:21:11,919 --> 00:21:49,039
+speaking it's a lot more useful to measure drift on the outputs of the model than the inputs the reason for that is because inputs change all the time your model tends to be robust to some degree of distribution shift of the inputs but if the outputs change then that might be more indicative that there's a problem and also outputs tend to be for most machine learning models tend to be lower dimensional so it's a little bit easier to monitor you can also rank the importance of your input features and measure drift on the most important ones you can do this just heuristically using the ones that you think are important or you can compute some notion of feature importance and use that to rank the features that you want to monitor lastly
+
+38
+00:21:47,440 --> 00:22:27,679
+there are metrics that you can look at that natively compute or approximate the distribution distance between high dimensional distributions and the two that are most worth checking out there are the maximum mean discrepancy and the approximate earth mover's distance the caveat here is that these are pretty hard to interpret so if you have a maximum mean discrepancy alert that's triggered that doesn't really tell you much about where to look for the potential failure that caused that distribution drift a more principled way in my opinion to measure distribution drift for high dimensional inputs to the model is to use projections the idea of a projection is you take some high dimensional input to the model or output
+
+39
+00:22:25,440 --> 00:23:08,240
+an image or text or just a really large feature vector and then you run that through a function so each data point that your model makes a prediction on gets tagged by this projection function and the goal of the projection function is to reduce the dimensionality of that input then once you've reduced the dimensionality you can do your drift detection on that lower dimensional representation of the high dimensional data and the great thing about this approach is that it works for any kind of data whether it's images or text or anything else no matter what the dimensionality is or what the data type is and it's highly flexible there's many different types of projections that can be useful you can define analytical projections that are
+
+40
+00:23:05,039 --> 00:23:46,880
+just functions of your input data and so these are things like looking at the mean pixel value of an image or the length of a sentence that's an input to the model or any other function that you can think of analytical projections are highly customizable they're highly interpretable and can often detect problems in practice if you don't want to use your domain knowledge to craft projections by writing analytical functions then you can also just do generic projections like random projections or statistical projections like running each of your inputs through an auto encoder something like that this is my recommendation for detecting drift for high dimensional and unstructured data and it's worth also just taking note of
+
+41
+00:23:44,799 --> 00:24:24,720
+this concept of projections because we're going to see this concept pop up in a few other places as we discuss other aspects of continual learning distribution drift is an important signal to look at when you're monitoring your models and in fact it's what a lot of people think of when they think about model monitoring so why do we rank it so low on the list let's talk about the cons of looking at distribution drift i think the big one is that models are designed to be robust to some degree of distribution drift the figure on the left shows sort of a toy example to demonstrate this point which is we have a classifier that's trained to predict two classes and we've induced a synthetic distribution shift just
+
+42
+00:24:22,880 --> 00:24:59,039
+shifting these points from the red ones on the top left to the bottom ones on the bottom right these two distributions are extremely different the marginal distributions in the chart on the bottom and then chart on the right-hand side have very large distance between the distributions but the model performs actually equally well on the training data as it does on the production data because the shift is just shifted directly along the classifier boundary so that's kind of a toy example that demonstrates that you know distribution shift is not really the thing that we care about when we're monitoring our models because just knowing that the distribution has changed doesn't tell us how the models has reacted to that
+
+43
+00:24:57,440 --> 00:25:35,520
+distribution change and then another example that's worth illustrating is some of my research when i was in grad school was using data that was generated from a physics simulator to solve problems on real world robots and the data that we used was highly out of distribution for the test case that we cared about the data looked like these kind of very low fidelity random images like on the left and we found that by training on a huge variety of these low fidelity random images our model was able to actually generalize to real world scenario like the one on the right so huge distribution shifts intuitively between the data the model was trained on and the data it was evaluated on but it was able to perform well on both
+
+44
+00:25:33,760 --> 00:26:15,039
+beyond the theoretical limitations of measuring distribution drift this can also just be hard to do in practice you have to pick window sizes correctly you have to keep all this data around you need to choose metrics you need to define projections to make your data lower dimensional so it's not a super reliable signal to look at and so that's why we advocate for looking at ones that are more correlated with the thing that actually matters the last thing you should consider looking at is your standard system metrics like cpu utilization or how much gpu memory your model is taking up things like that so those don't really tell you anything about how your model is actually performing but they can tell you when something is going
+
+45
+00:26:12,880 --> 00:26:49,840
+wrong okay so this is a ranking of all the different types of metrics or signals that you could look at if you're able to compute them but to give you a more concrete recommendation here we also have to talk about how hard it is to compute these different signals in practice we'll put the sort of value of each of these types of signals on the y-axis and on the x-axis we'll talk about the feasibility like how easy is it to actually measure these things measuring outcomes or feedback has pretty wide variability in terms of how feasible it is to do depends a lot on how your product is set up and the type of problem that you're working on measuring model performance tends to be the least feasible thing to do because
+
+46
+00:26:47,360 --> 00:27:26,799
+it does involve collecting some labels and so things like proxy metrics are a little bit easier to compute because they don't involve labels whereas system metrics and data quality metrics are highly feasible because there's you know great off-the-shelf libraries and tools that you can use for them and they don't involve doing anything sort of special from a machine learning perspective so the practical recommendation here is getting basic data quality checks is effectively zero regret especially if you are in the phase where you're retraining your model pretty frequently because data quality issues are one of the most common causes of bad model performance in practice and they're very easy to implement the next
+
+47
+00:27:24,000 --> 00:28:02,000
+recommendation is get some way of measuring feedback or model performance or if you really can't do either of those things than a proxy metric even if that way of measuring model performance is hacky or not scalable this is the most important signal to look at and is really the only thing that will be able to reliably tell you if your model is doing what it's supposed to be doing or not doing what it's supposed to be doing and then if your model is producing low dimensional outputs like if you're doing binary classification or something like that then monitoring the output distribution the score distribution also tends to be pretty useful and pretty easy to do and then lastly as you evolve your system like once you have these
+
+48
+00:28:00,000 --> 00:28:45,120
+basics in place and you're iterating on your model and you're trying to get more confident about evaluation i would encourage you to adopt a mindset about metrics that you compute that's borrowed from the concept of observability so what is the observability mindset we can think about monitoring as measuring the known unknowns so if there's four or five or ten metrics that we know that we care about accuracy latency user feedback the monitoring approach would be to measure each of those signals we might set alerts on even just a few of those key metrics on the other hand observability is about measuring the unknown unknowns it's about having the power to be able to ask arbitrary questions about your system when it
+
+49
+00:28:42,640 --> 00:29:19,520
+breaks for example how does my accuracy break out across all of the different regions that i've been considering what is my distribution drift for each of my features not signals that you would necessarily set alerts on because you don't have any reason to believe that these signals are things that are going to cause problems in the future but when you're in the mode of debugging being able to look at these things is really helpful and if you choose to adopt the observability mindset which i would highly encourage you to do especially in machine learning because it's just very very critical to be able to answer arbitrary questions to debug what's going on with your model then there's a few implications first
+
+50
+00:29:17,919 --> 00:29:55,760
+you should really keep around the context or the raw data that makes up the metrics that you're computing because you're gonna want to be able to drill all the way down to potentially the data points themselves that make up the metric that has degraded it's also as a side note helpful to keep around the raw data to begin with for things like retraining the second implication is that you can kind of go crazy with measurement you can define lots of different metrics on anything that you can think of that might potentially go wrong in the future but you shouldn't necessarily set alerts on each of those or at least not very or at least not very precise alerts because you don't want to have the problem of getting too
+
+51
+00:29:54,399 --> 00:30:31,120
+many alerts you want to be able to use these signals for the purpose of debugging when something is going wrong drift is a great example of this it's very useful for debugging because let's say that your accuracy was lower yesterday than it was the rest of the month well one way that you might debug that is by trying to see if there's any input fields or projections that look different that distinguish yesterday from the rest of the month those might be indicators of what is going wrong with your model and the last piece of advice i have on model monitoring and observability is it's very important to go beyond aggregate metrics let's say that your model is 99 accurate and let's say that's really good but for one
+
+52
+00:30:29,120 --> 00:31:06,799
+particular user who happens to be your most important user it's only 50 accurate can we really still consider that mobs to be good and so the way to deal with this is by flagging important subgroups or cohorts of data and being able to slice and dice performance along those cohorts and potentially even set alerts on those cohorts some examples of this are categories of users that you don't want your model to be biased against or categories of users that are particularly important for your business or just ones where you might expect your model to perform differently on them like if you're rolled out in a bunch of different regions or a bunch of different languages it might be helpful to look at how your performance breaks
+
+53
+00:31:04,799 --> 00:31:44,399
+out across those regions or languages all right that was a deep dive in different metrics that you can look at for the purpose of monitoring the next question that we'll talk about is how to tell if those metrics are good or bad there's a few different options for doing this that you'll see recommended one that i don't recommend and i alluded to this a little bit before is two sample statistical tests like aks test the reason why i don't recommend this is because if you think about what these two sample tests are actually doing they're trying to return a p-value for the likelihood that this data and this data are not coming from the same distribution and when you have a lot of data that just means that even really tiny shifts
+
+54
+00:31:42,399 --> 00:32:21,679
+in the distribution will get very very small p values because even if the distributions are only a tiny bit different if you have a ton of samples you'll be able to very confidently say that those are different distributions but that's not actually what we care about since models are robust to small amounts of distribution shift better options than statistical tests include the following you can have fixed rules like there should never be any null values in this column you can have specific ranges so your accuracy should always be between 90 and 95 there can be predicted ranges so the accuracy is within what an off-the-shelf anomaly detector thinks is reasonable or there's also unsupervised detection of
+
+55
+00:32:19,600 --> 00:32:55,200
+just new patterns in this signal and the most commonly used ones in practice are the first two fixed rules and specified ranges but predicted ranges via anomaly detection can also be really useful especially if there's some seasonality in your data the last topic i want to cover on model monitoring is the different tools that are available for monitoring your models the first category is system monitoring tools so this is a pretty mature category with a bunch of different companies in it and these are tools that help you detect problems with any software system not just machine learning models and they provide functionality for setting alarms when things go wrong and most of the cloud providers have pretty decent
+
+56
+00:32:53,679 --> 00:33:31,440
+solutions here but if you want something better you can look at one of the observability or monitoring specific tools like honeycomb or datadog you can monitor pretty much anything in these systems and so it kind of raises the question of whether we should just use systems like this for monitoring machine learning metrics as well there's a great blog post on exactly this topic that i recommend reading if you're interested in learning about why this is feasible but pretty painful thing to do and so maybe it's better to use something that's ml specific here in terms of ml specific tools there's some open source tools the two most popular ones are evidently ai and y logs and these are both similar in that you provide them
+
+57
+00:33:29,120 --> 00:34:07,200
+with samples of data and they produce a nice report that tells you where is their distribution shifts how have your model metrics changed etc the big limitation of these tools is that they don't solve the data infrastructure and the scale problem for you you still need to be able to get all that data into a place where you can analyze it with these tools and in practice that ends up being one of the hardest parts about this problem the main difference between these tools is that why logs is a little bit more focused on gathering data from the edge and the way they do that is by aggregating the data into statistical profiles at inference time itself so you don't need to transport all the data from your inference devices back to your
+
+58
+00:34:05,600 --> 00:34:44,960
+cloud which in some cases can be very helpful and lastly there's a bunch of different sas vendors for ml monitoring and observability my startup gantry has some functionality around this and there's a bunch of other options as well all right so we've talked about model monitoring and observability and the goal of monitoring and observability in the context of continual learning is to give you the signals that you need to figure out what's going wrong with your continual learning system and how you can change the strategy in order to influence that outcome next we're going to talk about for each of the stages in the continual learning loop what are the different ways that you might be able to go beyond the basics and
+
+59
+00:34:43,520 --> 00:35:19,040
+use what we learned from monitoring and observability to improve those stages the first stage of the continual learning loop is logging as a reminder the goal of logging is to get data from your model to a place where you can analyze it and the key question to answer is what data should i actually log for most of us the best answer is just to log all of your data storage is cheap and it's better to have data than not have it but there's some situations where you can't do that for example if you have just too much traffic going through your model to the point where it's too expensive to log all of it um if you have data privacy concerns if you're not actually allowed to look at your users data or if you're running
+
+60
+00:35:16,880 --> 00:35:56,400
+your model at the edge and it's too expensive to get all that data back because you don't have enough network bandwidth if you can't log all of your data there's two things that you can do the first is profiling the idea of profiling is that rather than sending all the data back to your cloud and then using that to do monitoring or observability or retraining instead you can compute statistical profiles of your data on the edge that describe the data distribution that you're seeing so the nice thing about this is it's great from a data security perspective because it doesn't require you to send all the data back home it minimizes your storage cost and lastly you don't miss things that happen in the tails which is an issue for the
+
+61
+00:35:55,040 --> 00:36:30,560
+next approach that we'll describe the place to use this really is primarily for security critical applications the other approach is sampling in sampling you'll just take certain data points and send those back home the advantage of sampling is that it has minimal impact on your inference resources so you don't have to actually spend the computational budget to compute profiles and you get to have access to the raw data for debugging and retraining and so this is what we recommend doing for pretty much every other application should describe in a little bit more detail how statistical profiles work because it's kind of interesting let's say that you have a stream of data that's coming in from two classes cat and dog and you
+
+62
+00:36:28,800 --> 00:37:13,200
+want to be able to estimate what is the distribution of cat and dog over time without looking at all of the raw data so for example maybe in the past you saw three examples of a dog and two examples of a cat a statistical profile that you can store that summarizes this data is just a histogram so the histogram says we saw three examples of a dog and two a cat and over time as more and more examples stream in rather than actually storing those data we can just increment the histogram and keep track of how many total examples of each category that we've seen over time and so like a neat fact of statistics is that for a lot of the statistics that you might be interested in looking at quantiles means accuracy other statistics you can
+
+63
+00:37:10,720 --> 00:37:49,280
+compute you can approximate those statistics pretty accurately by using statistical profiles called sketches that have minimal size so if you're interested in going on a tangent and learning more about an interesting topic in computer science that's one i'd recommend checking out next step in the continual learning loop is curation to remind you the goal of curation is to take your infinite stream of production data which is potentially unlabeled and turn this into a finite reservoir of data that has all the enrichments that it needs like labels to train your model on the key question that we need to answer here is similar to the one that we need to answer when we're sampling data at log time which is what data
+
+64
+00:37:47,280 --> 00:38:25,760
+should we select for enrichment the most basic strategy for doing this is just sampling data randomly but especially as your model gets better most of the data that you see in production might not actually be that helpful for improving your model and if you do this you could miss rare classes or events like if you have an event that happens you know one time in every 10 000 examples in production but you are trying to improve your model on it then you might not sample any examples of that at all if you just sample randomly a way to improve on random sampling is to do what's called stratified sampling the idea here is to sample specific proportions of data points from various subpopulations so common ways that you might stratify for
+
+65
+00:38:23,200 --> 00:39:05,520
+sampling in ml could be sampling to get a balance among classes or sampling to get a balance among categories that you don't want your model to be biased against like gender lastly the most advanced and interesting strategy for picking data to enrich is to curate data points that are somehow interesting for the purpose of improving your model and there's a few different ways of doing this that we'll cover the first is to have this notion of interesting data be driven by your users which will come from user feedback and feedback loops the second is to determine what is interesting data yourself by defining error cases or edge cases and then the third is to let an algorithm define this for you and this is a category of techniques known as
+
+66
+00:39:04,079 --> 00:39:40,800
+active learning if you already have a feedback loop or a way of gathering feedback from your users in your machine learning system which you really should if you can then this is probably the easiest and potentially also the most effective way to pick interesting data for the purpose of curation and the way this works is you'll pick data based on signals that come from your users that they didn't like your prediction so this could be the user churned after interacting with your model it could be that they filed a support ticket about a particular prediction the model made it could be that they you know click the thumbs down button that you put in your products that they changed the label that your model produced for them or
+
+67
+00:39:38,880 --> 00:40:19,599
+that they intervened with an automatic system like they grab the wheel of their autopilot system if you don't have user feedback or if you need even more ways of gathering interesting data from your system then probably the second most effective way of doing this is by doing manual error analysis the way this works is we will look at the errors that our model is making we will reason about the different types of failure modes that we're seeing we'll try to write functions or rules that help capture these error modes and then we'll use those functions to gather more data that might represent those error cases two sub-categories of how to do this one is what i would call similarity-based curation and the way this works is if
+
+68
+00:40:17,359 --> 00:40:57,119
+you have some data that represents your errors or data that you think might be an error then you can pick an individual data point or a handful of data points and run a nearest neighbor similarity search algorithm to find the data points in your stream that are the closest to the one that your model is maybe making a mistake on the second way of doing this which is potentially more powerful but a little bit harder to do is called projection based curation the way this works is rather than just picking an example and grabbing the nearest neighbors of that example instead we are going to find an error case like the one on the bottom right where there's a person crossing the street with a bicycle and then we're gonna write a
+
+69
+00:40:54,400 --> 00:41:35,599
+function that attempts to detect that error case and this could just be trading a simple neural network or it could be just writing some heuristics the advantage of doing similarity-based curation is that it's really easy and fast right like you just have to click on a few examples and you'll be able to get things that are similar to those examples this is beginning to be widely used in practice thanks to the explosion of vector search databases on the market it's relatively easy to do this and what this is particularly good for is events that are rare they don't occur very often in your data set but they're pretty easy to detect like if you had a problem with your self-driving car where you have llamas crossing the road a
+
+70
+00:41:33,760 --> 00:42:10,480
+similarity search-based algorithm would probably do a reasonably good job of detecting other llamas in your training set on the other hand projection-based curation requires some domain knowledge because it requires you to think a little bit more about what is the particular error case that you're seeing here and write a function to detect it but it's good for more subtle error modes where a similarity search algorithm might be too coarse-screened it might find examples that look similar on the surface to the one that you are detecting but don't actually cause your model to fail the last way to curate data is to do so automatically using a class of algorithms called active learning the way active learning works
+
+71
+00:42:08,240 --> 00:42:42,640
+is given a large amount of unlabeled data what we're going to try to do is determine which data points would improve model performance the most if you were to label those data points next and train on them and the way that these algorithms work is by defining a sampling strategy or a query strategy and then you rank all of your unlabeled examples using a scoring function that defines that strategy and take the ones with the highest scores and send them off to be labeled i'll give you a quick tour of some of the different types of scoring functions that are out there and if you want to learn more about this then i'd recommend the blog post linked on the bottom you have scoring functions that sample data points that the model
+
+72
+00:42:40,560 --> 00:43:16,800
+is very unconfident about you have scoring functions that are defined by trying to predict what is the error that the model would make on this data point if we had a label for it you have scoring functions that are designed to detect data that doesn't look anything like the data that you've already trained on so can we distinguish these data points from our training data if so maybe those are the ones that we should sample and label we have scoring functions that are designed to take a huge data set of points and boil it down to the small number of data points that are most representative of that distribution lastly there's scoring functions that are designed to detect data points that if we train on them we
+
+73
+00:43:15,119 --> 00:43:49,839
+think would have a big impact on training so where they would have a large expected gradient or would tend to cause the model to change its mind so that's just a quick tour of different types of scoring functions that you might implement uncertainty based scoring tends to be the one that i see the most in practice largely because it's very simple to implement and tends to produce pretty decent results but it's worth diving a little bit deeper into this if you do decide to go down this route if you're paying close attention you might have noticed that there's a lot of similarity between some of the ways that we do data curation the way that we pick interesting data points and the way that we do monitoring i
+
+74
+00:43:47,359 --> 00:44:25,839
+think that's no coincidence monitoring and data curation are two sides of the same coin they're both interested in solving the problem of finding data points where the model may not be performing well or where we're uncertain about how the model is performing on those data points so for example user driven curation is kind of another side of the same coin of monitoring user feedback metrics both of these things look at the same metrics stratified sampling is a lot like doing subgroup or cohort analysis making sure that we're getting enough data points from subgroups that are important or making sure that our metrics are not degrading on those subgroups projections are used in both data curation and monitoring to
+
+75
+00:44:23,599 --> 00:45:05,200
+take high dimensional data and break them down into distributions that we think are interesting for some purpose and then in active learning some of the techniques also have mirrors in monitoring like predicting the loss on an unlabeled data point or using the model's uncertainty on that data point next let's talk about some case studies of how data curation is done in practice the first one is a blog post on how openai trained dolly2 to detect malicious inputs to the model there's two techniques that they used here they used active learn learning using uncertainty sampling to reduce the false positives for the model and then they did a manual curation actually they did it kind of an automated way but they did
+
+76
+00:45:02,319 --> 00:45:41,599
+similarity search to find similar examples to the ones that the model was not performing well on the next example from tesla this is a talk i love from andre carpathi about how they build a data flywheel of tesla and they use two techniques here one is feedback loops so gathering information about when users intervene with the autopilot and then the second is manual curation via projections for edge case detection and so this is super cool because they actually have infrastructure that allows ml engineers when they discover a new edge case to write an edge case detector function and then actually deploy that on the fleet that edge case detector not only helps them curate data but it also helps them decide which data to sample
+
+77
+00:45:40,000 --> 00:46:17,280
+which is really powerful the last case study i want to talk about is from cruz they also have this concept of building a continual learning machine and the main way they do that is through feedback loops that's kind of a quick tour of what some people actually use to build these data curation systems in practice there's a few tools that are emerging to help with data curation scale nucleus and aquarium are relatively similar tools that are focused on computer vision and they're especially good at nearest neighbor based sampling at my startup gantry we're also working on some tools to help with this across a wide variety of different applications concrete recommendations on data curation random sampling is probably a fine starting
+
+78
+00:46:14,400 --> 00:46:51,359
+point for most use cases but if you have a need to avoid bias or if there's rare classes in your data set you probably should start even with stratified sampling or at the very least introduce that pretty soon after you start sampling if you have a feedback loop as part of your machine learning system and i hope you're taking away from this that how helpful it is to have these feedback loops then user-driven curation is kind of a no-brainer this is definitely something that you should be doing and is probably going to be the thing that is the most effective in the early days of improving your model if you don't have a feedback loop then using confidence-based active learning is a next best bet because it's pretty easy
+
+79
+00:46:49,599 --> 00:47:25,680
+to implement and works okay in practice and then finally as your model performance increases you're gonna have to look harder and harder for these challenging training points at the end of the day if you want to squeeze the maximum performance out of your model there's no avoiding manually looking at your your data and trying to find interesting failure modes there's no substitute for knowing your data after we've curated our infinite stream of unlabeled data down to a reservoir of labeled data that's ready to potentially train on the next thing that we'll need to decide is what trigger are we going to use to retrain and the main takeaway here is that moving to automated retraining is not always necessary in many cases just
+
+80
+00:47:23,839 --> 00:47:59,520
+manually refraining is good enough but it can save you time and lead to better better model performance so it's worth understanding when it makes sense to actually make that move the main prerequisite for moving to automated retraining is just being able to reproduce model performance when retraining in a fairly automated fashion so if you're able to do that and you are not really working on this model very actively anymore then it's probably worth implementing some automated retraining if you just find yourself retraining this model super frequently then it'll probably save you time to implement this earlier when it's time to move to automated training the main recommendation is just keep it simple and retrain periodically like once a
+
+81
+00:47:57,280 --> 00:48:32,319
+week rerun training on that schedule the main question though is how do you pick that training schedule so what i recommend doing here is doing a little bit of like measurement to figure out what is a reasonable retraining schedule you can plot your model performance over time and then compare to how the model would have performed if you had retrained on different frequencies you can just make basic assumptions here like if you retrain you'll be able to reach the same level of accuracy and what you're going to be doing here is looking at these different retraining schedules and looking at the area between these curves like on the chart on the top right the area between these two curves is your opportunity cost in
+
+82
+00:48:30,880 --> 00:49:09,920
+terms of like how much model performance you're leaving on the table by not retraining more frequently and then once you have a number of these different opportunity costs for different retraining frequencies you can plot those opportunity costs and then you can sort of run the ad hoc exercise of trying to balance you know where is the rate trade-off point for us between the performance gain that we get from retraining more frequently and the cost of that retraining which is both the cost of running the retraining itself as well as the operational cost that we'll introduce by needing to evaluate that model more frequently that's what i'd recommend doing in practice a request i have for research is i think it'd be great i think it's
+
+83
+00:49:08,240 --> 00:49:43,839
+very feasible to have a technique that would automatically determine the optimal retraining strategy based on how performance tends to decay how sensitive you are to that performance decay your operational costs and your retraining costs so i think you know eventually we won't need to do the manual data analysis every single time to determine this retraining frequency if you're more advanced then the other thing you can consider doing is retraining based on performance triggers this looks like setting triggers on metrics like accuracy and only retraining when that accuracy dips below a predefined threshold some big advantages to doing this are you can react a lot more quickly to unexpected changes that
+
+84
+00:49:42,160 --> 00:50:16,240
+happen in between your normal training schedule it's more cost optimal because you can skip a retraining if it wouldn't actually improve your model's performance but the big cons here are that since you don't know in advance when you're gonna be retraining you need to have good instrumentation and measurement in place to make sure that when you do retrain you're doing it for the right reasons and that the new model is actually doing well these techniques i think also don't have a lot of good theoretical justification and so if you are the type of person that wants to understand you know why theoretically this should work really well i don't think you're going to find that today and probably the most important con is
+
+85
+00:50:14,720 --> 00:50:50,800
+that this adds a lot of operational complexity because instead of just knowing like hey at 8 am i know my retraining is going live and so i can check in on that instead this retraining could happen at any time so your whole system needs to be able to handle that and that just introduces a lot of new infrastructure that you'll need to build and then lastly an idea that probably won't be relevant to most of you but is worth thinking about because i think it's it could be really powerful in the future is online learning where you train on every single data point as it comes in it's not very commonly used in practice but one sort of relaxation of this idea that is used fairly frequently in practice is online adaptation the way
+
+86
+00:50:48,160 --> 00:51:28,319
+online adaptation works is it operates not the level of retraining the whole model itself but it operates on the level of adapting the policy that sits on top of the model what is a policy a policy is the set of rules that takes the raw prediction that the model made like the score or the raw output of the model and then turns that into the actual thing that the user sees so like a classification threshold is an example of a policy or if you have many different versions of your model that you're ensembling what are the weights of those ensembles or even which version of the model is this particular request going to be routed to in online adaptation rather than retraining the model on each new data point as it comes
+
+87
+00:51:26,000 --> 00:52:07,359
+in instead we use an algorithm like multi-arm bandits to tune the weights of this policy online as more data comes in so if your data changes really frequently in practice or you are have a hard time training your model frequently enough to adapt to it then online adaptation is definitely worth looking into next we've fired off a trigger to start a training job and the next question we need to answer is among all of the labeled data in our reservoir of data which specific data points should we train on for this particular training job most of the time in deep learning we'll just train on all the data that we have available to us but if you have too much data to do that then depending on whether recency of data is
+
+88
+00:52:05,680 --> 00:52:43,599
+an important signal to determine whether that data is useful you'll either slide a window to make sure that you're looking at the most recent data therefore in many cases the most useful data or we'll use techniques like sampling or online batch selection if not and a more advanced technique to be aware of that is hard to execute in practice today is continual fine-tuning we'll talk about that as well so the first option is just to train on all available data so you have a data set that you'll keep track of that your last model was trained on then over time between your last training and your next training you'll have a bunch of new data come in you'll curate some of that data then you'll just take all that data
+
+89
+00:52:42,160 --> 00:53:20,240
+you'll add it to the data set and you'll train the new model on the combined data set so the keys here are you need to keep this data version controlled so that you know which data was added to each training iteration and it's also important if you want to be able to evaluate the model properly to keep track of the rules that you use to curate that new data so if you're sampling in a way that's not uniform from your distribution you should keep track of the rules that you use to sample so that you can determine where that data actually came from second option is to bias your sampling toward more recent data by using a sliding window the way this works is at each point when you train your model you look
+
+90
+00:53:17,119 --> 00:53:55,760
+backward and you gather a window of data that leads up to the current moment and then at your next training you slide that window forward and so there might be a lot of overlap potentially between these two data sets but you have all the new data or like a lot of the new data and you get rid of the oldest data in order to form the new data set couple key things to do here are it's really helpful to look at the different statistics between the old and new data sets to catch bugs like if you have a large change in a particular distribution of one of the columns that might be indicative of a new bug that's been introduced and one challenge that you'll find here is just comparing the old and the new versions of the models
+
+91
+00:53:52,960 --> 00:54:30,960
+since they are not trained on data that is related in a very straightforward way if you're working in a setting where you need to sample data you can't train on all of your data but there isn't any reason to believe that recent data is much better than older data then you can sample data from your reservoir using a variety of techniques the most promising of which is called online batch selection normally if we were doing stochastic gradient descent then what we do is we would sample mini batches on every single training iteration until we run out of data or until we run out of compute budget in online batch selection instead what we do is before each training step we sample a larger batch like much larger than the mini batch
+
+92
+00:54:29,359 --> 00:55:08,480
+that we ultimately want to train on we rank each of the items in the mini batch according to a label aware selection function and then we take the top n items according to that function and train on those the paper on the right describes a label aware selection function called the reducible holdout loss selection function that performs pretty well on some relatively large data sets and so if you're going to look into this technique this is probably where i would start the last option that we'll discuss which is not recommended to do today is continual fine-tuning the way this works is rather than retraining from scratch every single time instead just only train your existing model on just new data the reason why you might
+
+93
+00:55:06,880 --> 00:55:41,839
+want to do this primarily is because it's much more cost effective the paper on the right shares some findings from grubhub where they found a 45x cost improvement by doing this technique relative to sliding windows but the big challenge here is that unless you're very careful it's easy for the model to forget what it learned in the past so the upshot is that you need to have pretty mature evaluation to be able to be very careful that your model is performing well on all the types of data that it needs to perform well on before it's worth implementing something like this so now we've triggered a retraining we have selected the data points that are going to go into the training job we've trained our model you know run our
+
+94
+00:55:39,839 --> 00:56:13,839
+hyperparameter sweeps if we want to and we have a new candidate model that we think is ready to go into production the next step is to test that model the goal of this stage is to produce a report that our team can sign off on that answers the question of whether this new model is good enough or whether it's better than the old model and the key question here is what should go into that report again this is a place where there's not a whole lot of standardization but the recommendation we have here is to compare your current model with the previous version of the model on all the following all the metrics that you care about all of the slices or subsets of data that you've flagged is important all of the edge
+
+95
+00:56:11,520 --> 00:56:47,760
+cases that you've defined and in a way that's adjusted to account for any sample and bias that you might have introduced by your curation strategy an example of what such a report could look like is the following across the top we have all of our metrics in this case accuracy precision and recall and then all on the left are all of the data sets and slices that we're looking at so the things to notice here are we have our main validation set which is like what most people use for evaluating models but rather than just looking at that those numbers in the aggregate we also break it out across a couple of different categories in this case the age of the user and the age of the account that belongs to that user and
+
+96
+00:56:45,680 --> 00:57:25,200
+then below the main validation set we also have more specific validation sets that correspond to particular error cases that we know have given our model trouble or a previous version of our model trouble in the past these could be like just particular edge cases that you've found in the past like maybe your model handles examples of poor grammar very poorly or it doesn't know what some gen z slang terms mean like these are examples of failure modes you've found for your model in the past that get rolled into data sets to test the next version of your model in continual learning just like how training sets are dynamic and change over time evaluation sets are dynamic as well as you curate new data you should add some of it to
+
+97
+00:57:23,440 --> 00:57:58,400
+your training sets but also add some of it to your evaluation sets for example if you change how you do sampling you might want to add some of that newly sampled data to your eval set as well to make sure that your eval set represents that new sampling strategy or if you discover a new edge case instead of only adding that edge case to the training set it's worth holding out some examples of that edge case as a particular unit test to be part of that offline evaluation suite two corollaries to note of the fact that evaluation sets are dynamic the first is that you should also version control your evaluation sets just like you do your training sets the second is that if your data is evolving really quickly then part of the
+
+98
+00:57:56,559 --> 00:58:36,079
+data that you hold out should always be the most recent data the data from you know the past day or the past hour or whatever it is to make sure that your model is generalizing well to new data once you have the basics in place a more advanced thing that you can look into here that i think is pretty promising is the idea of expectation tests the way that expectation tests work are you take pairs of examples where you know the relationship so let's say that you're doing sentiment analysis and you have a sentence that says my brother is good if you make the positive word in that sentence more positive and instead say my brother is great then you would expect your sentiment classifier to become even more positive about that sentence these types
+
+99
+00:58:33,200 --> 00:59:16,400
+of tests have been explored in nlp as well as recommendation systems and they're really good for testing whether your model generalizes in predictable ways and so they give you more granular information than just aggregate performance metrics about how your model does on previously unseen data one observation to make here is that just like how data curation is highly analogous to monitoring so is offline testing just like in monitoring we want to observe our metrics not just in aggregate but also across all of our important subsets of data and across all of our edge cases one difference between these two is that you will in general have different metrics available in offline testing and online testing for
+
+100
+00:59:13,440 --> 00:59:51,520
+example you are much more likely to have labels available offline in fact you always have labels available offline because that is uh how you're going to train your model but online you're much more likely to have feedback and so even though these two ideas are highly analogous and should share a lot of metrics and definitions of subsets and things like that one point of friction that you that will occur between online monitoring and offline testing is that the metrics are a little bit different so one direction for research that i think would be really exciting to see more of is using offline metrics like accuracy to predict online metrics like user engagement and then lastly once we've tested our candidate model offline
+
+101
+00:59:49,359 --> 01:00:28,319
+it's time to deploy it and evaluate it online so we talked about this last time so i don't want to reiterate too much but as a reminder if you have the infrastructural capability to do so then you should do things like first running your model in shadow mode before you um actually roll it out to real users then running an a b test to make sure that users are responding to it better than they did the old model then once you have a successful av test rolling it out to all of your users but doing so gradually and then finally if you see issues during that rollout just to roll it back to the old version of the model and try to figure out what went wrong so we talked about the different stages of continual learning from
+
+102
+01:00:25,280 --> 01:01:07,440
+logging data to curating it to triggering retraining testing the model and rolling out to production and we also talked about monitoring and observability which is about giving you a set of rules that you can use to tell whether your retraining strategy needs to change and we observed that in a bunch of different places the fundamental elements that you study in monitoring like projections and user feedback and model uncertainty are also useful for different parts of the continual learning process and that's no coincidence i see monitoring and continual learning as two sides of the same coin we should be using the signals that we monitor to very directly change our retraining strategy so the last thing i want to do is just try to
+
+103
+01:01:05,839 --> 01:01:42,160
+make this a little bit more concrete by walking through an example of a workflow that you might have from detecting an issue in your model to altering the strategy this section describes more of a feature state until you've invested pretty heavily in infrastructure it's going to be hard to make it feel as seamless as this in practice but i wanted to mention it anyway because i think it provides like a nice end state for what we should aspire to in our continual learning workflows the thing you would need to have in place before you're able to actually execute what i'm going to describe next is a place to store and version all of the elements of your strategy which include metric definitions for both online and offline
+
+104
+01:01:40,319 --> 01:02:20,240
+testing performance thresholds for those metrics definitions of any of the projections that you want to use for monitoring and also for data curation subgroups or cohorts that you think are particularly important to break out your metrics along the logic that defines how you do data curation whether it's sampling rules or anything else and then finally the specific data sets that you use for each different run of your training or evaluation our example continue improvement loop starts with an alert and in this case that alert might be our user feedback got worse today and so our job is now to figure out what's going on so the next thing we'll use is some of our observability tools to investigate what's going on here and we
+
+105
+01:02:18,720 --> 01:02:56,079
+might you know run some subgroup analyses and look at some raw data and figure out that the problem is really mostly isolated to new users the next thing that we might do is do error analysis so look at those new users and the data points that they're sending us and try to reason about why those data points are performing worse and what we might discover is something like our model was trained assuming that people were going to write emails but now users are submitting a bunch of text that has things that aren't normally found in emails like emojis and that's causing our model problems so here's where we might make the first change to our retraining strategy we could define new users as a cohort of interest because we
+
+106
+01:02:54,400 --> 01:03:30,160
+never want performance to decline on new users again without getting an alert about that then we could define a new projection that helps us detect data that has emojis and add that projection to our observability metrics so that anytime in the future if we want as part of an investigation to see how our performance differs between users that are submitting emojis and ones that are not we can always do that without needing to rewrite the projection next we might search our reservoir for historical examples that contain emojis so that we can use them to make our model better and then adjust our strategy by adding that subset of data as a new test case so now whenever we test the model going forward we'll
+
+107
+01:03:27,839 --> 01:04:03,920
+always see how it performs on data with emojis in addition to adding emoji examples to as a test case we would also curate them and add them back into our training set and do a retraining then once we have the new model that's trained we'll get this new model comparison report which will include also the new cohort that we defined as part of this process and the new emoji edge case data set that we defined and then finally if we're doing manual deployment we can just deploy that model and that completes the continual improvement loop so to wrap up what do i want you to take away from this continual learning is a complicated rapidly evolving and poorly understood topic so this is an area to pay attention to if you're interested in
+
+108
+01:04:02,640 --> 01:04:39,599
+seeing how the cutting edge of production machine learning is evolving and the main takeaway from this lecture is we broke down the concept of a retraining strategy which consists of a number of different pieces definitions of metrics subgroups of interest projections that help you break down and analyze high dimensional data performance thresholds for your metrics logic for curating new data sets and the specific data sets that you're going to use for retraining and evaluation at a high level the way that we can think about our role as machine learning engineers once we've deployed the first version of the model is to use rules that we define as part of our observability and monitoring suite to iterate on the strategy for many of you
+
+109
+01:04:37,920 --> 01:05:13,280
+in the near term this won't feel that different from just using that data to retrain the model however you'd like to but i think thinking of this as a strategy that you can tune at a higher level is a productive way of understanding it as you move towards more and more automated retraining lastly just like every other aspect of the ml life cycle that we talked about in this course our main recommendation here is to start simple and add complexity later in the context of continual learning what that means is it's okay to retrain your models manually to start as you get more advanced you might want to automate retraining and you also might want to think more intelligently about how you sample data to make sure that you're
+
+110
+01:05:11,119 --> 01:05:19,200
+getting the data that is most useful for improving your model going forward that's all for this week see you next time
+

documents/lecture-07.md

@@ -0,0 +1,285 @@
+---
+description: Building on Transformers, GPT-3, CLIP, StableDiffusion, and other Large Models.
+---
+
+# Lecture 7: Foundation Models
+
+<div align="center">
+<iframe width="720" height="405" src="https://www.youtube-nocookie.com/embed/Rm11UeGwGgk?list=PL1T8fO7ArWleMMI8KPJ_5D5XSlovTW_Ur" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+Lecture by [Sergey Karayev](https://twitter.com/sergeykarayev).
+Notes by [James Le](https://twitter.com/le_james94) and [Vishnu Rachakonda](https://www.linkedin.com/in/vrachakonda/).<br />
+Published September 19, 2022.
+[Download slides](https://fsdl.me/2022-lecture-07-slides).
+
+Foundation models are very large models trained on very large datasets that
+can be used for multiple downstream tasks.
+
+We’ll talk about fine-tuning, Transformers, large language models, prompt engineering, other applications of large models, and vision and text-based models like CLIP and image generation.
+
+![alt_text](media/image-1.png "image_tooltip")
+
+## 1 - Fine-Tuning
+
+Traditional ML uses a lot of data and a large model, which takes a long time. But if you have a small amount of data, you can use **transfer learning** to benefit from the training on a lot of data. You basically use the same model that you have pre-trained, add a few layers, and unlock some weights.
+
+We have been doing this in computer vision since 2014. Usually, you train a model on ImageNet, keep most of the layers, and replace the top three or so layers with newly learned weights. Model Zoos are full of these models like AlexNet, ResNet, etc. in both TensorFlow and PyTorch.
+
+In NLP, pre-training was initially limited only to the first step: word embeddings. The input to a language model is words. One way you can encode them to be a vector (instead of a word) is **one-hot encoding**. Given a large matrix of words, you can make an embedding matrix and embed each word into a real-valued vector space. This new matrix is down to the dimension on the order of a thousand magnitude. Maybe those dimensions correspond to some semantic notion.
+
+![alt_text](media/image-2.png "image_tooltip")
+
+
+[Word2Vec](https://jalammar.github.io/illustrated-word2vec/) trained a model like this in 2013. It looked at which words frequently co-occur together. The learning objective was to maximize cosine similarity between their embeddings. It could do cool demos of vector math on these embeddings. For example, when you embed the words “king,” “man,” and “woman,” you can do vector math to get a vector that is close to the word “queen” in this embedding space.
+
+It’s useful to see more context to embed words correctly because words can play different roles in the sentence (depending on their context). If you do this, you’ll improve accuracy on all downstream tasks. In 2018, a number of models such as ELMO and ULMFit [published pre-trained LSTM-based models that set state-of-the-art results on most NLP tasks](https://ruder.io/nlp-imagenet/).
+
+But if you look at the model zoos today, you won’t see any LSTMs. You’ll only see Transformers everywhere. What are they?
+
+
+## 2 - Transformers
+
+Transformers come from a paper called “[Attention Is All You Need](https://arxiv.org/abs/1706.03762)” in 2017, which introduced a groundbreaking architecture that sets state-of-the-art results on translation first and a bunch of NLP tasks later.
+
+![alt_text](media/image-3.png "image_tooltip")
+
+
+It has a decoder and an encoder. For simplicity, let’s take a look at the encoder. The interesting components here are self-attention, positional encoding, and layer normalization.
+
+
+### Self-Attention
+
+![alt_text](media/image-4.png "image_tooltip")
+
+
+Basic self-attention follows: Given an input sequence of vectors x of size t, we will produce an output sequence of tensors of size t. Each tensor is a weighted sum of the input sequence. The weight here is just a dot product of the input vectors. All we have to do is to make that weighted vector sum to 1. We can represent it visually, as seen below. The input is a sentence in English, while the output is a translation in French.
+
+![alt_text](media/image-5.png "image_tooltip")
+
+
+So far, there are no learned weights and no sequence order. Let’s learn some weights!* If we look at the input vectors, we use them in three ways: as **queries** to compare two other input vectors, as **keys** to compare them to input vectors and produce the corresponding output vector, and as **values **to sum up all the input vectors and produce the output vector.
+* We can process each input vector with three different matrices to fulfill these roles of query, key, and value. We will have three weighted matrices, and everything else remains the same. If we learn these matrices, we learn attention.
+* It’s called **multi-head attention **because we learn different sets of weighted matrices simultaneously, but we implement them as just a single matrix.
+
+So far, we have learned the query, key, and value. Now we need to introduce some notion of order to the sequence by encoding each vector with its position. This is called **positional encoding**.
+
+
+### Positional Encoding
+
+![alt_text](media/image-6.png "image_tooltip")
+
+
+Let’s say we have an input sequence of words
+
+]* The first step is to embed the words into a dense, real-valued word embedding. This part can be learned.
+* However, there is no order to that embedding. Thus, we will add another embedding that only encodes the position.
+* In brief, the first embedding encodes only the content, while the second embedding encodes only the position. If you add them, you now have information about both the content and the position.
+
+
+### Layer Normalization
+
+![alt_text](media/image-7.png "image_tooltip")
+
+
+Neural network layers work best when the input vectors have uniform mean and standard deviation in each dimension. As activations flow through the network, the means and standard deviations get blown out by the weight matrices. [Layer normalization](https://arxiv.org/pdf/1803.08494.pdf) is a hack to re-normalize every activation to where we want them between each layer.
+
+That’s it! All the amazing results you’ll see from now on are just increasingly large Transformers with dozens of layers, dozens of heads within each layer, large embedding dimensions, etc. The fundamentals are the same. It’s just the Transformer model.
+
+[Anthropic](https://www.anthropic.com/) has been publishing great work lately to investigate why Transformers work so well. Check out these publications:
+
+1. [A Mathematical Framework for Transformer Circuits](https://transformer-circuits.pub/2021/framework/index.html)
+2. [In-Context Learning and Induction Heads](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html)
+3. [Toy Models of Superposition](https://transformer-circuits.pub/2022/toy_model/index.html)
+
+
+## 3 - Large Language Models
+
+
+### Models
+
+GPT and [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) came out in 2018 and 2019, respectively. The name means “generative pre-trained Transformers.” They are decoder-only models and use masked self-attention. This means: At a poi that at the output sequence, you can only attend to two input sequence vectors that came before that point in the sequence.
+
+![alt_text](media/image-8.png "image_tooltip")
+
+
+These models were trained on 8 million web pages. The largest model has 1.5 billion parameters. The task that GPT-2 was trained on is predicting the next word in all of this text on the web. They found that it works increasingly well with an increasing number of parameters.
+
+![alt_text](media/image-9.png "image_tooltip")
+
+
+[BERT](https://arxiv.org/abs/1810.04805) came out around the same time as Bidirectional Encoder Representations for Transformers. It is encoder-only and does not do attention masking. It has 110 million parameters. During training, BERT masks out random words in a sequence and has to predict whatever the masked word is.
+
+![alt_text](media/image-10.png "image_tooltip")
+
+
+[T5 (Text-to-Text Transformer)](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) came out in 2020. The input and output are both text strings, so you can specify the task that the model supposes to be doing. T5 has an encoder-decoder architecture. It was trained on the C4 dataset (Colossal Clean Crawled Corpus), which is 100x larger than Wikipedia. It has around 10 billion parameters. You can download [the open-sourced model](https://github.com/google-research/text-to-text-transfer-transformer#released-model-checkpoints) and run it on your machine.
+
+[GPT-3](https://openai.com/blog/gpt-3-apps/) was one of the state-of-the-art models in 2020. It was 100x larger than GPT/GPT-2 with 175 billion parameters. Because of its size, GPT-3 exhibits unprecedented capabilities of few-shot and zero-shot learning. As seen in the graph below, the more examples you give the model, the better its performance is. The larger the model is, the better its performance is. If a larger model was trained, it would be even better.
+
+![alt_text](media/image-11.png "image_tooltip")
+
+
+OpenAI also released [Instruct-GPT](https://openai.com/blog/instruction-following/) earlier this year. It had humans rank different GPT-3 outputs and used reinforcement learning to fine-tune the model. Instruct-GPT was much better at following instructions. OpenAI has put this model, titled ‘text-davinci-002,’ in their API. It is unclear how big the model is. It could be ~10x smaller than GPT-3.
+
+![alt_text](media/image-12.png "image_tooltip")
+
+
+DeepMind released [RETRO (Retrieval-Enhanced Transformers)](https://arxiv.org/pdf/2112.04426.pdf) in 2021. Instead of learning language and memorizing facts in the model’s parameters, why don’t we just learn the language in parameters and retrieve facts from a large database of internal text? To implement RETRO, they encode a bunch of sentences with BERT and store them in a huge database with more than 1 trillion tokens. At inference time, they fetch matching sentences and attend to them. This is a powerful idea because RETRO is connected to an always updated database of facts.
+
+![alt_text](media/image-13.png "image_tooltip")
+
+
+DeepMind released another model called [Chinchilla](https://gpt3demo.com/apps/chinchilla-deepmind) in 2022, which observed the scaling laws of language models. They [trained over 400 language models](https://arxiv.org/pdf/2203.15556.pdf) from 70 million to 16 billion parameters on 5 billion to 500 billion tokens. They then derived formulas for optimal model and training set size, given a fixed compute budget. They found that most large language models are “undertrained,” meaning they haven’t seen enough data.
+
+![alt_text](media/image-14.png "image_tooltip")
+
+
+To prove this, they trained a large model called [Gopher](https://gpt3demo.com/apps/deepmind-gopher) with 280 billion parameters and 300 billion tokens. With Chincilla, they reduced the number of parameters to 70 billion and used four times as much data (1.4 trillion tokens). Chinchilla not only matched Gopher’s performance but actually exceeded it. Check out [this LessWrong post](https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications) if you want to read about people’s opinions on it.
+
+
+### Vendors
+
+OpenAI offers four model sizes: Ada, Babbage, Curie, and Davinci. [Each has a different price](https://openai.com/api/pricing/) and different capabilities. Most of the impressive GPT-3 results on the Internet came from Davinci. These correspond to 350M, 1.3B, 6.7B, and 175B parameters. You can also fine-tune models for an extra cost. The quota you get when you sign up is pretty small, but you can raise it over time. You have to apply for review before going into production.
+
+There are some alternatives to OpenAI:
+
+1. [Cohere AI](https://cohere.ai/) has similar models for [similar prices](https://cohere.ai/pricing).
+2. [AI21](https://www.ai21.com/) also has some large models.
+3. There are also open-source large language models, such as [Eleuther GPT-NeoX](https://www.eleuther.ai/projects/gpt-neox/) (20B parameters), [Facebook OPT-175B](https://ai.facebook.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/) (175B parameters), and [BLOOM from BigScience](https://bigscience.huggingface.co/blog/bloom) (176B parameters). If you want to use one of these open-source models but do not have to be responsible for deploying it, you can use [HuggingFace’s inference API](https://huggingface.co/inference-api).
+
+
+## 4 - Prompt Engineering
+
+GPT-3 and other large language models are mostly alien technologies. It’s unclear how they exactly work. People are finding out how they work by playing with them. We will cover some notable examples below. Note that if you play around with them long enough, you are likely to discover something new.
+
+GPT-3 is surprisingly bad at reversing words due to **tokenization**: It doesn’t see letters and words as humans do. Instead, it sees “tokens,” which are chunks of characters. Furthermore, it gets confused with long-ish sequences. Finally, it has trouble merging characters. For it to work, you have to teach GPT-3 the algorithm to use to get around its limitations. Take a look at [this example from Peter Welinder](https://twitter.com/npew/status/1525900849888866307).
+
+![alt_text](media/image-15.jpg "image_tooltip")
+
+
+Another crazy prompt engineering is “Let’s Think Step By Step.” This comes from a paper called “[Large Language Models are Zero-Shot Reasoners](https://arxiv.org/pdf/2205.11916.pdf).” Simply adding “Let’s Think Step By Step” into the prompt increases the accuracy of GPT-3 on one math problem dataset from 17% to 78% and another math problem dataset from 10% to 40%.
+
+![alt_text](media/image-16.png "image_tooltip")
+
+
+Another unintuitive thing is that the context length of GPT is long. You can give it a **long instruction** and it can return the desired output. [This example](https://twitter.com/goodside/status/1557381916109701120) shows how GPT can output a CSV file and write the Python code as stated. You can also use **formatting tricks **to reduce the training cost, as you can do multiple tasks per call. Take a look at [this example](https://twitter.com/goodside/status/1561569870822653952) for inspiration.
+
+We have to be careful since our models might get pwnage or possessed. User input in the prompt may instruct the model to do something naughty. This input can even reveal your prompt to [prompt injection attacks](https://simonwillison.net/2022/Sep/12/prompt-injection/) and [possess your AI](https://twitter.com/goodside/status/1564112369806151680). This actually works in GPT-3-powered production apps.
+
+![alt_text](media/image-17.png "image_tooltip")
+
+
+Further work is needed before putting GPT-3-powered apps into production. There are some tools for prompt engineering such as [PromptSource](https://github.com/bigscience-workshop/promptsource) and [OpenPrompt](https://github.com/thunlp/OpenPrompt), but we definitely need better tools.
+
+
+## 5 - Other Applications
+
+
+### Code
+
+![alt_text](media/image-18.png "image_tooltip")
+
+
+One notable application of large foundation models is **code generation**. With a 40- billion-parameter Transformer model pre-trained on all the Github code it could find, [DeepMind Alphacode](https://www.deepmind.com/blog/competitive-programming-with-alphacode) was able to achieve an above-average score on the Codeforce competition. To do this, they used a model to generate a large set of potential solutions and another model to winnow down the options by actually executing them. 
+
+The general idea to highlight from this is **filtering the outputs of a model**. You can have a separate model that does filtering, or you can have some kind of verification + validation process. This can really significantly boost accuracy. OpenAI demonstrates impressive results on [different math word problems](https://openai.com/blog/grade-school-math/), as seen below.
+
+![alt_text](media/image-19.png "image_tooltip")
+
+
+Code generation has moved into products of late, like [Github Copilot](https://github.blog/2022-06-21-github-copilot-is-generally-available-to-all-developers/). We highly recommend trying it out! Another option for a similar tool is [replit’s new tool for coding](https://blog.replit.com/ai). 
+
+We’re just getting started with the applications of foundation models to the programming workflow. In fact, things are about to start getting really wild. [A recent paper](https://arxiv.org/pdf/2207.14502.pdf) showed that a large language model that generated its own synthetic puzzles to learn to code could improve significantly. **Models are teaching themselves to get better!**
+
+![alt_text](media/image-20.png "image_tooltip")
+
+
+Playing around with systems like GPT-3 and their ability to generate code can feel quite remarkable! Check out some fun experiments Sergey ran ([here](https://twitter.com/sergeykarayev/status/1569377881440276481) and [here](https://twitter.com/sergeykarayev/status/1570848080941154304)).
+
+![alt_text](media/image-21.jpg "image_tooltip")
+
+### Semantic Search
+
+**Semantic search** is another interesting application area. If you have texts like words, sentences, paragraphs, or even whole documents, you can embed that text with large language models to get vectors. If you have queries in sentences or paragraphs, you can also embed them in the same way. With this function, you can generate embeddings and easily find semantic overlap by examining the cosine similarity between embedding vectors.
+
+![alt_text](media/image-22.png "image_tooltip")
+
+
+Implementing this semantic search is hard. Computations on large, dense vectors with float data types are intensive. Companies like Google and Facebook that use this approach have developed libraries like [FAISS](https://towardsdatascience.com/using-faiss-to-search-in-multidimensional-spaces-ccc80fcbf949) and [ScaNN](https://cloud.google.com/blog/topics/developers-practitioners/find-anything-blazingly-fast-googles-vector-search-technology) to solve the challenges of implementing semantic search. 
+
+Some open-source options for this include [Haystack from DeepSet](https://www.deepset.ai/haystack) and [Jina.AI](https://github.com/jina-ai/jina). Other vendor options include [Pinecone](https://www.pinecone.io/), [Weaviate](https://weaviate.io/), [Milvus](https://milvus.io/), [Qdrant](https://qdrant.tech/), [Google Vector AI Matching Engine](https://cloud.google.com/vertex-ai/docs/matching-engine/overview), etc.
+
+
+### Going Cross-Modal
+
+Newer models are bridging the gap between data modalities (e.g. using both vision and text). One such model is [the Flamingo model](https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/tackling-multiple-tasks-with-a-single-visual-language-model/flamingo.pdf), which uses a special model component called a **perceiver resampler** (an attention module that translates images into fixed-length sequences of tokens).
+
+![alt_text](media/image-23.png "image_tooltip")
+
+
+Another paper about [Socratic Models](https://socraticmodels.github.io/) was recently published. The author trained several large models (a vision model, a language model, and an audio model) that are able to interface with each other using language prompts to perform new tasks.
+
+Finally, the concept of “Foundation Models” came from the paper “[On the Opportunities and Risks of Foundation Models](https://arxiv.org/abs/2108.07258)” by researchers at Stanford Institute for Human-Centered AI. We think “Large Language Models” or “Large Neural Networks” might be more useful terms.
+
+
+## 6 - CLIP and Image Generation
+
+Now, let's talk about some of the most exciting applications of this kind of model: in vision!
+
+In a 2021 OpenAI paper called “[Learning transferrable visual models from natural language supervision](https://arxiv.org/abs/2103.00020)”, **CLIP (Contrastive Language–Image Pre-training)** was introduced. In this paper, the authors encode text via Transforms, encode images via ResNets or Visual Transformers, and apply contrastive training to train the model. Contrastive training matches correct image and text pairs using cosine similarity. The code for this is tremendously simple!
+
+![alt_text](media/image-24.png "image_tooltip")
+
+
+With this powerful trained model, you can map images and text using embeddings, even on unseen data. There are two ways of doing this. One is to use a “linear probe” by training a simple logistic regression model on top of the features CLIP outputs after performing inference. Otherwise, you can use a “zero-shot” technique that encodes all the text labels and compares them to the encoded image. Zero-shot tends to be better, but not always.
+
+Since OpenAI CLIP was released in an open-source format, there have been many attempts to improve it, including [the OpenCLIP model](https://github.com/mlfoundations/open_clip), which actually outperforms CLIP.
+
+To clarify, CLIP doesn’t go directly from image to text or vice versa. It uses embeddings. This  embedding space, however, is super helpful for actually performing searches across modalities. This goes back to our section on vector search. There are so many cool projects that have come out of these efforts! (like [this](https://rom1504.github.io/clip-retrieval/) and [this](https://github.com/haltakov/natural-language-image-search))
+
+To help develop mental models for these operations, consider how to actual perform **image captioning** (image -> text) and image generation (text -> image). There are two great examples of this written in [the ClipCap paper](https://arxiv.org/pdf/2111.09734.pdf). At a high level, image captioning is performed through training a separate model to mediate between a frozen CLIP, which generates a series of word embeddings, and a frozen GPT-2, which takes these word embeddings and generates texts. 
+
+The intermediate model is a Transformer model that gets better at modeling images and captions.
+
+![alt_text](media/image-25.png "image_tooltip")
+
+
+In **image generation**, the most well-known approach is taken by [DALL-E 2 or unCLIP](https://cdn.openai.com/papers/dall-e-2.pdf). In this method, two additional components are introduced to a CLIP system, a prior that maps from text embedding to image embeddings and a decoder that maps from image embedding to image. The prior exists to solve the problem that many text captions can accurately work for an image. 
+
+![alt_text](media/image-26.png "image_tooltip")
+
+
+In DALL-E 2’s case, they use an approach for the prior called **a diffusion model**. [Diffusion models](https://towardsdatascience.com/diffusion-models-made-easy-8414298ce4da) are trained to denoise data effectively through training on incrementally noisy data.
+
+![alt_text](media/image-27.png "image_tooltip")
+
+
+In DALL-E 2, the diffusion method is applied to the **prior** model, which trains its denoising approach on a sequence of encoded text, CLIP text embedding, the diffusion timestamp, and the noised CLIP embedding, all so it can predict the un-noised CLIP image embedding. In doing so, it helps us bridge the gap between the raw text caption to the model, which can be infinitely complicated and “noisy”, and the CLIP image embedding space.
+
+![alt_text](media/image-28.png "image_tooltip")
+
+
+The **decoder** helps us go from the prior’s output of an image embedding to an image. This is a much simpler approach for us to understand. We apply a U-Net structure to a diffusion training process that is able to ultimately “de-noise” the input image embedding and output an image.
+
+![alt_text](media/image-29.png "image_tooltip")
+
+
+The results of this model are incredible! You can even generate images and merge images using CLIP embeddings. There are all kinds of funky ways of playing with the embeddings to create various image outputs.
+
+![alt_text](media/image-30.png "image_tooltip")
+
+
+Other models of interest are Parti and StableDiffusion.
+
+* Google published [Parti](https://parti.research.google/) soon after DALL-E 2. Parti uses a VQGAN model instead of a diffusion model, where the image is represented as a sequence of high-dimensional tokens).
+* [StableDiffusion](https://stability.ai/blog/stable-diffusion-public-release) has been released publicly, so definitely [check it out](https://github.com/CompVis/latent-diffusion)! It uses a “latent diffusion” model, which diffuses the image in a low-dimensional latent space and decodes the image back into a pixel space.
+
+![alt_text](media/image-31.png "image_tooltip")
+
+
+There has been an absolute explosion of these applications. Check out these examples on [image-to-image](https://twitter.com/DiffusionPics/status/1568219366097039361/), [video generation](https://twitter.com/jakedowns/status/1568343105212129280), and [photoshop plugins](https://www.reddit.com/r/StableDiffusion/comments/wyduk1/). The sky is the limit.
+
+Prompting these models is interesting and can get pretty involved. Someday this may even be tool and code-based. You can learn from other people on [Lexica](https://lexica.art/) and [promptoMANIA](https://promptomania.com/).
+
+It’s truly a remarkable time to be involved with AI models as they scale to new heights.

documents/lecture-08.md

@@ -0,0 +1,713 @@
+---
+description: Building ML-powered products and the teams who create them
+---
+
+# Lecture 8: ML Teams and Project Management
+
+<div align="center">
+<iframe width="720" height="405" src="https://www.youtube-nocookie.com/embed/a54xH6nT4Sw?list=PL1T8fO7ArWleMMI8KPJ_5D5XSlovTW_Ur" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+Lecture by [Josh Tobin](https://twitter.com/josh_tobin_).
+Notes by [James Le](https://twitter.com/le_james94) and [Vishnu Rachakonda](https://www.linkedin.com/in/vrachakonda/).<br />
+Published September 26, 2022.
+[Download slides](https://fsdl.me/2022-lecture-08-slides).
+
+## 0 - Why is this hard?
+
+Building any product is hard:
+
+-   You have to hire great people.
+
+-   You have to manage and develop those people.
+
+-   You have to manage your team's output and make sure your vectors are
+aligned.
+
+-   You have to make good long-term technical choices and manage
+technical debt.
+
+-   You have to manage expectations from leadership.
+
+-   You have to define and communicate requirements with stakeholders.
+
+Machine Learning (ML) adds complexity to that process:
+
+-   ML talent is expensive and scarce.
+
+-   ML teams have a diverse set of roles.
+
+-   Projects have unclear timelines and high uncertainty.
+
+-   The field is moving fast, and ML is the "[high-interest credit card
+of technical
+debt](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf)."
+
+-   Leadership often doesn't understand ML.
+
+-   ML products fail in ways that are hard for laypeople to understand.
+
+In this lecture, we'll talk about:
+
+1.  ML-related **roles** and their required skills.
+
+2.  How to **hire** ML engineers (and how to get hired).
+
+3.  How ML teams are **organized** and fit into the broader
+organization.
+
+4.  How to **manage** an ML team and ML products.
+
+5.  **Design** considerations for ML products.
+
+## 1 - Roles
+
+### Common Roles
+
+Let's look at the most common ML roles and the skills they require:
+
+-   The **ML Product Manager** works with the ML team, other business
+functions, the end-users, and the data owners. This person designs
+docs, creates wireframes, and develops a plan to prioritize and
+execute ML projects.
+
+-   The **MLOps/ML Platform Engineer** builds the infrastructure to make
+models easier and more scalable to deploy. This person handles the
+ML infrastructure that runs the deployed ML product using
+platforms like AWS, GCP, Kafka, and other ML tooling vendors.
+
+-   The **ML Engineer** trains and deploys prediction models. This
+person uses tools like TensorFlow and Docker to work with
+prediction systems running on real data in production.
+
+-   The **ML Researcher** trains prediction models, often those that are
+forward-looking or not production-critical. This person uses
+libraries like TensorFlow and PyTorch on notebook environments to
+build models and reports describing their experiments.
+
+-   The **Data Scientist** is a blanket term used to describe all of the
+roles above. In some organizations, this role entails answering
+business questions via analytics. This person can work with
+wide-ranging tools from SQL and Excel to Pandas and Scikit-Learn.
+
+![](./media/image9.png)
+
+### Skills Required
+
+What skills are needed for these roles? The chart below displays a nice
+visual - where the horizontal axis is the level of ML expertise and the
+size of the bubble is the level of communication and technical writing
+(the bigger, the better).
+
+![](./media/image4.png)
+
+-   The **MLOps** is primarily a software engineering role, which often
+comes from a standard software engineering pipeline.
+
+-   The **ML Engineer** requires a rare mix of ML and Software
+Engineering skills. This person is either an engineer with
+significant self-teaching OR a science/engineering Ph.D. who works
+as a traditional software engineer after graduate school.
+
+-   The **ML Researcher** is an ML expert who usually has an MS or Ph.D.
+degree in Computer Science or Statistics or finishes an industrial
+fellowship program.
+
+-   The **ML Product Manager** is just like a traditional Product
+Manager but with a deep knowledge of the ML development process
+and mindset.
+
+-   The **Data Scientist** role constitutes a wide range of backgrounds,
+from undergraduate to Ph.D. students.
+
+There is an important distinction between a task ML engineer and a
+platform ML engineer, coined by Shreya Shankar in [this blog
+post](https://www.shreya-shankar.com/phd-year-one/):
+
+1.  **Task ML engineers** are responsible for maintaining specific ML
+pipelines. They only focus on ensuring that these ML models are
+healthy and updated frequently. They are often overburdened.
+
+2.  **Platform ML engineers** help task ML engineers automate tedious
+parts of their jobs. They are called MLOps/ML Platform engineers
+in our parlance.
+
+## 2 - Hiring
+
+### The AI Talent Gap
+
+In 2018 (when we started FSDL), the AI talent gap was the main story.
+There were so few people who understood this technology, so the biggest
+block for organizations was that they couldn't find people who were good
+at ML.
+
+In 2022, the AI talent gap persists. But it tends to be less of a
+blocker than it used to be because we have had four years of folks
+switching careers into ML and software engineers emerging from
+undergraduate with at least a couple of ML classes under their belts.
+
+The gap tends to be in folks that understand more than just the
+underlying technology but also have experience in seeing how ML fails
+and how to make ML successful when it's deployed. That's the reality of
+how difficult it is to hire ML folks today, especially those with
+**production experience**.
+
+### Sourcing
+
+Because of this shallow talent pool and the skyrocketing demand, hiring
+for ML positions is pretty hard. Typical ML roles come in the following
+structure:
+
+-   ML Adjacent roles: ML product manager, DevOps, Data Engineer
+
+-   Core ML Roles: ML Engineer, ML Research/ML Scientist
+
+-   Business analytics roles: Data Scientist
+
+For ML-adjacent roles, traditional ML knowledge is less important, as
+demonstrated interest, conversational understanding, and experience can
+help these professionals play an impactful role on ML teams. Let's focus
+on how to hire for **the core ML roles**.
+
+![](./media/image6.png)
+
+
+While there's no perfect way to **hire ML engineers**, there's
+definitely a wrong way to hire them, with extensive job descriptions
+that demand only the best qualifications (seen above). Certainly, there
+are many good examples of this bad practice floating around.
+
+-   Rather than this unrealistic process, consider hiring for software
+engineering skills, an interest in ML, and a desire to learn. You
+can always train people in the art and science of ML, especially
+when they come with strong software engineering fundamentals.
+
+-   Another option is to consider adding junior talent, as many recent
+grads come out with good ML knowledge nowadays.
+
+-   Finally, and most importantly, be more specific about what you need
+the position and professional to do. It's impossible to find one
+person that can do everything from full-fledged DevOps to
+algorithm development.
+
+To **hire ML researchers**, here are our tips:
+
+-   Evaluate the quality of publications, over the quantity, with an eye
+toward the originality of the ideas, the execution, etc.
+
+-   Prioritize researchers that focus on important problems instead of
+trendy problems.
+
+-   Experience outside academia is also a positive, as these researchers
+may be able to transition to industry more effectively.
+
+-   Finally, keep an open mind about research talent and consider
+talented people without PhDs or from adjacent fields like physics,
+statistics, etc.
+
+To find quality candidates for these roles, here are some ideas for
+sourcing:
+
+-   Use standard sources like LinkedIn, recruiters, on-campus
+recruiting, etc.
+
+-   Monitor arXiv and top conferences and flag the first authors of
+papers you like.
+
+-   Look for good implementations of papers you like.
+
+-   Attend ML research conferences (NeurIPS, ICML, ICLR).
+
+![](./media/image7.png)
+
+As you seek to recruit, stay on top of what professionals want and make
+an effort to position your company accordingly. ML practitioners want to
+be empowered to do great work with interesting data. Building a culture
+of learning and impact can help recruit the best talent to your team.
+Additionally, sell sell sell! Talent needs to know how good your team is
+and how meaningful the mission can be.
+
+### Interviewing
+
+As you interview candidates for ML roles, try to **validate your
+hypotheses of their strengths while testing a minimum bar on weaker
+aspects**. For example, ensure ML researchers can think creatively about
+new ML problems while ensuring they meet a baseline for code quality.
+It's essential to test ML knowledge and software engineering skills for
+all industry professionals, though the relative strengths can vary.
+
+The actual ML interview process is much less well-defined than software
+engineering interviews, though it is modeled off of it. Some helpful
+inclusions are projects or exercises that test the ability to work with
+ML-specific code, like take-home ML projects. Chip Huyen's
+"[Introduction to ML Interviews
+Book](https://huyenchip.com/ml-interviews-book/)" is a
+great resource.
+
+### Finding A Job
+
+To find an ML job, you can take a look at the following sources:
+
+-   Standard sources such as LinkedIn, recruiters, on-campus recruiting,
+etc.
+
+-   ML research conferences (NeurIPS, ICLR, ICML).
+
+-   Apply directly (remember, there's a talent gap!).
+
+Standing out for competitive roles can be tricky! Here are some tips (in
+increasing order of impressiveness) that you can apply to differentiate
+yourself:
+
+1.  Exhibit ML interest (e.g., conference attendance, online course
+certificates, etc.).
+
+2.  Build software engineering skills (e.g., at a well-known software
+company).
+
+3.  Show you have a broad knowledge of ML (e.g., write blog posts
+synthesizing a research area).
+
+4.  Demonstrate ability to get ML projects done (e.g., create side
+projects, re-implement papers).
+
+5.  Prove you can think creatively in ML (e.g., win Kaggle competitions,
+publish papers).
+
+## 3 - Organizations
+
+### Organization Archetypes
+
+There exists not yet a consensus on the right way to structure an ML
+team. Still, a few best practices are contingent upon different
+organization archetypes and their ML maturity level. First, let's see
+what the different ML organization archetypes are.
+
+**Archetype 1 - Nascent and Ad-Hoc ML**
+
+-   These are organizations where no one is doing ML, or ML is done on
+an ad-hoc basis. Obviously, there is little ML expertise in-house.
+
+-   They are either small-to-medium businesses or less
+technology-forward large companies in industries like education or
+logistics.
+
+-   There is often low-hanging fruit for ML.
+
+-   But there is little support for ML projects, and it's challenging to
+hire and retain good talent.
+
+**Archetype 2 - ML R&D**
+
+-   These are organizations in which ML efforts are centered in the R&D
+arm of the organization. They often hire ML researchers and
+doctorate students with experience publishing papers.
+
+-   They are larger companies in sectors such as oil and gas,
+manufacturing, or telecommunications.
+
+-   They can hire experienced researchers and work on long-term business
+priorities to get big wins.
+
+-   However, it is very difficult to get quality data. Most often, this
+type of research work rarely translates into actual business
+value, so usually, the amount of investment remains small.
+
+**Archetype 3 - ML Embedded Into Business and Product Teams**
+
+-   These are organizations where certain product teams or business
+units have ML expertise alongside their software or analytics
+talent. These ML individuals report up to the team's
+engineering/tech lead.
+
+-   They are either software companies or financial services companies.
+
+-   ML improvements are likely to lead to business value. Furthermore,
+there is a tight feedback cycle between idea iteration and product
+improvement.
+
+-   Unfortunately, it is still very hard to hire and develop top talent,
+and access to data and compute resources can lag. There are also
+potential conflicts between ML project cycles and engineering
+management, so long-term ML projects can be hard to justify.
+
+**Archetype 4 - Independent ML Function**
+
+-   These are organizations in which the ML division reports directly to
+senior leadership. The ML Product Managers work with Researchers
+and Engineers to build ML into client-facing products. They can
+sometimes publish long-term research.
+
+-   They are often large financial services companies.
+
+-   Talent density allows them to hire and train top practitioners.
+Senior leaders can marshal data and compute resources. This gives
+the organizations to invest in tooling, practices, and culture
+around ML development.
+
+-   A disadvantage is that model handoffs to different business lines
+can be challenging since users need the buy-in to ML benefits and
+get educated on the model use. Also, feedback cycles can be slow.
+
+**Archetype 5 - ML-First Organizations**
+
+-   These are organizations in which the CEO invests in ML, and there
+are experts across the business focusing on quick wins. The ML
+division works on challenging and long-term projects.
+
+-   They are large tech companies and ML-focused startups.
+
+-   They have the best data access (data thinking permeates the
+organization), the most attractive recruiting funnel (challenging
+ML problems tends to attract top talent), and the easiest
+deployment procedure (product teams understand ML well enough).
+
+-   This type of organization archetype is hard to implement in practice
+since it is culturally difficult to embed ML thinking everywhere.
+
+### Team Structure Design Choices
+
+Depending on the above archetype that your organization resembles, you
+can make the appropriate design choices, which broadly speaking follow
+these three categories:
+
+1.  **Software Engineer vs. Research**: To what extent is the ML team
+responsible for building or integrating with software? How
+important are Software Engineering skills on the team?
+
+2.  **Data Ownership**: How much control does the ML team have over data
+collection, warehousing, labeling, and pipelining?
+
+3.  **Model Ownership**: Is the ML team responsible for deploying models
+into production? Who maintains the deployed models?
+
+Below are our design suggestions:
+
+If your organization focuses on **ML R&D**:
+
+-   Research is most definitely prioritized over Software Engineering
+skills. Because of this, there would potentially be a lack of
+collaboration between these two groups.
+
+-   ML team has no control over the data and typically will not have
+data engineers to support them.
+
+-   ML models are rarely deployed into production.
+
+If your organization has **ML embedded into the product**:
+
+-   Software Engineering skills will be prioritized over Research
+skills. Often, the researchers would need strong engineering
+skills since everyone would be expected to product-ionize his/her
+models.
+
+-   ML teams generally do not own data production and data management.
+They will need to work with data engineers to build data
+pipelines.
+
+-   ML engineers totally own the models that they deploy into
+production.
+
+If your organization has **an independent ML division**:
+
+-   Each team has a potent mix of engineering and research skills;
+therefore, they work closely together within teams.
+
+-   ML team has a voice in data governance discussions, as well as a
+robust data engineering function.
+
+-   ML team hands-off models to users but is still responsible for
+maintaining them.
+
+If your organization is **ML-First**:
+
+-   Different teams are more or less research-oriented, but in general,
+research teams collaborate closely with engineering teams.
+
+-   ML team often owns the company-wide data infrastructure.
+
+-   ML team hands the models to users, who are responsible for operating
+and maintaining them.
+
+The picture below neatly sums up these suggestions:
+
+![](./media/image12.png)
+
+## 4 - Managing
+
+### Managing ML Teams Is Challenging
+
+The process of actually managing an ML team is quite challenging for
+four reasons:
+
+1.  **Engineering Estimation:** It's hard to know how easy or hard an ML
+project is in advance. As you explore the data and experiment with
+different models, there is enormous scope for new learnings about
+the problem that materially impact the timeline. Furthermore,
+knowing what methods will work is often impossible. This makes it
+hard to say upfront how long or how much work may go into an ML
+project.
+
+2.  **Nonlinear Progress:** As the chart below from a [blog
+post](https://medium.com/@l2k/why-are-machine-learning-projects-so-hard-to-manage-8e9b9cf49641)
+by Lukas Biewald (CEO of [Weights and
+Biases](https://wandb.ai/site)) shows, progress on ML
+projects is unpredictable over time, even when the effort expended
+grows considerably. It's very common for projects to stall for
+extended periods of time.
+
+![](./media/image1.png)
+
+3.  **Cultural gaps:** The relative culture of engineering and research
+professionals is very different. Research tends to favor novel,
+creative ideas, while engineering prefers tried and true methods
+that work. As a result, ML teams often experience a clash of
+cultures, which can turn toxic if not appropriately managed. A
+core challenge of running ML teams is addressing the cultural
+barriers between ML and software engineering so that teams can
+harmoniously experiment and deliver ML products.
+
+4.  **Leadership Deficits**: It's common to see a lack of detailed
+understanding of ML at senior levels of management in many
+companies. As a result, expressing feasibility and setting the
+right expectations for ML projects, especially high-priority ones,
+can be hard.
+
+### How To Manage ML Teams Better
+
+Managing ML teams is hardly a solved problem, but you can take steps to
+improve the process.
+
+**Plan probabilistically**
+
+Many engineering projects are managed in a waterfall fashion, with the
+sequential tasks defined up front clearly. Instead of forcing this
+method of engineering management on difficult ML projects, try assigning
+a likelihood of success to different tasks to better capture the
+experimental process inherent to ML engineering. As these tasks progress
+or stall, rapidly re-evaluate your task ordering to better match what is
+working. Having this sense of both (1) **how likely a task is to
+succeed** and (2) **how important it is** makes project planning
+considerably more realistic.
+
+![](./media/image10.png)
+
+
+**Have a portfolio of approaches**
+
+Embrace multiple ideas and approaches to solve crucial research
+challenges that gate production ML. Don't make your plan dependent on
+one approach working!
+
+**Measure inputs, not results**
+
+As you work through several approaches in your portfolio, do not overly
+emphasize whose ideas ultimately work as a reflection of contribution
+quality. This can negatively impact team members' creativity, as they
+focus more on trying to find only what they currently think could work,
+rather than experimenting in a high-quality fashion (which is ultimately
+what leads to ML success).
+
+**Have researchers and engineers work together**
+
+The collaboration between engineering and research is essential for
+quality ML products to get into production. Emphasize collaboration
+across the groups and professionals!
+
+**Get quick wins**
+
+Taking this approach makes it more likely that your ML project will
+succeed in the long term. It allows you to demonstrate progress to your
+leadership more effectively and clearly.
+
+**Educate leadership on uncertainty**
+
+This can be hard, as leadership is ultimately accountable for addressing
+blind spots and understanding timeline risk. There are things you can
+do, however, to help improve leadership's knowledge about ML timelines.
+
+-   Avoid building hype around narrow progress metrics material only to
+the ML team (e.g., "*We improved F1 score by 0.2 and have achieved
+awesome performance!*").
+
+-   Instead, be realistic, communicate risk, and emphasize real product
+impact (e.g., "Our model improvements should increase the number
+of conversions by 10%, though we must continue to validate its
+performance on additional demographic factors.)
+
+-   Sharing resources like [this a16z primer](https://a16z.com/2016/06/10/ai-deep-learning-machines/),
+[this class from Prof. Pieter
+Abbeel](https://executive.berkeley.edu/programs/artificial-intelligence),
+and [this Google's People + AI
+guidebook](https://pair.withgoogle.com/guidebook) can
+increase awareness of your company's leadership.
+
+### ML PMs are well-positioned to educate the organization
+
+There are two types of ML product managers.
+
+1.  **Task PMs**: These are the more common form of ML PM. They are
+generally specialized into a specific product area (e.g. trust and
+safety) and have a strong understanding of the particular use
+case.
+
+2.  **Platform PMs**: These are a newer form of PMs. They have a broader
+mandate to ensure that the ML team (generally centralized in this
+context) is highest leverage. They manage workflow and priorities
+for this centralized team. To support this, they tend to have a
+broad understanding of ML themselves. These PMs are critical for
+educating the rest of the company about ML and ensuring that teams
+trust the output of models.
+
+Both types of PMs are crucial for ML success. Platform PMs tend to have
+a particularly powerful role to play in pushing an organization's
+adoption of machine learning and making it successful.
+
+### What is "Agile" for ML?
+
+There are two options similar to what Agile is for software development
+in the ML context. They are shown below:
+
+![](./media/image2.png)
+
+
+They are both structured, data-science native approaches to project
+management. You can use them to provide standardization for project
+stages, roles, and artifacts.
+
+**TDSP** tends to be more structured and is a strong alternative to the
+Agile methodology. **CRISP-DM** is somewhat higher level and does not
+provide as structured a project management workflow. If you genuinely
+have a large-scale coordination problem, you can try these frameworks,
+but don't otherwise. They can slow you down since they are more oriented
+around "traditional" data science and not machine learning.
+
+## 5 - Design
+
+Let's talk about how to actually design machine learning products now.
+The biggest challenge with designing such products often isn't
+implementing them; it's **bridging the gap between users' inflated
+expectations and the reality**.
+
+Users often expect extremely sophisticated systems capable of solving
+many more problems than they actually can.
+
+![](./media/image11.png)
+
+In reality, machine learning systems are more like dogs that are trained
+to do a special task; weird little guys with a penchant for distraction
+and an inability to do much more than they are explicitly told.
+
+![](./media/image13.png)
+
+All this leads to a big gap between what can be done and what users
+expect!
+
+### The Keys to Good ML Product Design
+
+In practice, **good ML product design bridges users expectations and
+reality**. If you can help users understand the benefits and limitations
+of the model, they tend to be more satisfied. Furthermore, always have
+backup plans for model failures! Over-automating systems tends to be a
+recipe for unhappy users. Finally, building in feedback loops can really
+increase satisfaction over time.
+
+There are a couple ways to **explain the benefits and limitations** of
+an ML system to users.
+
+-   Focus on the problems it solves, not the fact that the system is
+"AI-powered".
+
+-   If you make the system feel "human-like" (unconstrained input,
+human-like responses), expect users to treat it as human-like.
+
+-   Furthermore, seek to include guardrails or prescriptive interfaces
+over open-ended, human-like experiences. A good example of the
+former approach is [Amazon
+Alexa](https://alexa.amazon.com/), which has specific
+prompts that its ML system responds to.
+
+![](./media/image5.png)
+
+
+**Handling failures** is a key part of keeping ML systems users happy.
+There's nothing worse than a "smart" system that conks out when you do
+something slightly unexpected. Having built-in solutions to solve for
+automation issues is extremely important. One approach is letting users
+be involved to correct improper responses. Another is to focus on the
+notion of "model confidence" and only offer responses when the threshold
+is met. A good example of a handling failure approach is how Facebook
+recommends photo tags for users, but doesn't go so far as to autoassign.
+
+### Types of User Feedback
+
+How can you collect feedback from users in a way that avoids these
+issues? There are different types of user feedback and how they help
+with model improvement.
+
+![](./media/image3.png)
+
+
+Let's go across this chart.
+
+1.  The simplest form of feedback is **indirect implicit feedback**. For
+example, did the user churn from the product? That tells you
+immediately how the user felt about the system without them giving
+a clear signal themselves.
+
+2.  Another form is **direct implicit feedback**, which involves the
+user "taking the next step". For example, in an automated user
+onboarding flow, did the user click through into ensuing steps?
+This is trickier to implement, but can be useful for future
+training iterations.
+
+3.  The next type of feedback is **binary explicit feedback**, wherein
+users are specifically asked (e.g. via thumbs up/down buttons) how
+they feel about the model performance.
+
+4.  You can make this more sophisticated and add **categorical explicit
+feedback**, which allows users to sort their feedback into various
+types.
+
+5.  To really get a sense of how users feel, consider offering **free
+text feedback**. This is tricky to use for model training and can
+be involved for users, but it's very useful to highlight the
+highest friction predictions.
+
+6.  The gold standard, of course, are **model corrections**; they are
+free labels!
+
+Whenever building explicit feedback into ML systems, avoid relying on
+users' altruism and be clear about why they should engage in the
+feedback. Instead, build positive feedback loops by allowing users to
+experience the benefits of their feedback quickly.
+
+**Great ML product experiences are designed from scratch**. ML is a very
+specific technology with clear advantages and drawbacks. Design needs to
+be thoughtfully executed around these products. It's especially
+important to allow users to interact safely with ML products that may
+fail in unexpected ways. Always try to find ways to build in feedback
+loops to make the ML product better over time.
+
+There are tons of resources that can help you get started with this
+emerging field.
+
+-   [Google's People + AI
+Guidebook](https://pair.withgoogle.com/guidebook)
+
+-   [Guidelines for Human-AI
+Interaction](https://dl.acm.org/doi/abs/10.1145/3290605.3300233)
+
+-   [Agency Plus Automation: Designing AI into Interactive
+Systems](http://idl.cs.washington.edu/files/2019-AgencyPlusAutomation-PNAS.pdf)
+
+-   [Designing Collaborative
+AI](https://medium.com/@Ben_Reinhardt/designing-collaborative-ai-5c1e8dbc8810)
+
+In conclusion, we talked through a number of adjacent considerations to
+building ML systems and products. In short, you ship the team as much
+you do the code; be thoughtful about how you hire, manage, and structure
+ML teams as much as ML products!
+
+![](./media/image8.png)
+

documents/lecture-08.srt

@@ -0,0 +1,416 @@
+1
+00:00:00,179 --> 00:00:40,500
+hey everybody welcome back this week we're going to talk about something a little bit different than we do most weeks most weeks we talk about specific technical aspects of building machine learning powered products but this week we're going to focus on some of the organizational things that you need to do in order to work together on ml-powered products as part of an interdisciplinary team so the the reality of building ml Power Products is that building any product well is really difficult you have to figure out how to hire grade people you need to be able to manage those people and get the best out of them you need to make sure that your team is all working together towards a shared goal you need to make good
+
+2
+00:00:38,399 --> 00:01:20,280
+long-term technical choices manage technical debt over time you need to make sure that you're managing expectations not just of your own team but also of leadership of your organization and you need to be able to make sure that you're working well within the confines of the requirements of the rest of the org that you're understanding those requirements well and communicating back to your progress to the rest of the organization against those requirements but machine learning adds even more additional complexity to this machine learning Talent tends to be very scarce and expensive to attract machine learning teams are not just a single role but today they tend to be pretty interdisciplinary which makes
+
+3
+00:01:18,659 --> 00:02:00,600
+managing them an even bigger challenge machine learning projects often have unclear timelines and there's a high degree of uncertainty to those timelines machine learning itself is moving super fast and machine learning as we've covered before you can think of as like the high interest credit card of technical debt so keeping up with making good long-term decisions and not incurring too much technical debt is especially difficult in ml unlike traditional software ml is so new that in most organizations leadership tends not to be that well educated in it they might not understand some of the core differences between ML and other technology that you're working with machine learning products tend to fail in ways that are really hard for Lay
+
+4
+00:01:58,799 --> 00:02:36,360
+people to understand and so that makes it very difficult to help the rest of the stakeholders in your organization understand what they could really expect from the technology that you're building and what is realistic for us to achieve so throughout the rest rest of this lecture we're going to kind of touch on some of these themes and cover different aspects of this problem of working together to build ml Power Products as an organization so here are the pieces that we're going to cover we're going to talk about different roles that are involved in building ml products we're going to talk about some of the unique aspects involved in hiring ml Talent we're going to talk about organization of teams and how the ml team tends to
+
+5
+00:02:34,739 --> 00:03:16,140
+fit into the rest of the org and some of the pros and cons of different ways of setting that up we'll talk about managing ml teams and ml product management and then lastly we'll talk about some of the design considerations for how to design a product that is well suited to having a good ml model that backs it so let's dive in and talk about rules the most common ml rules that you might hear of are things like ml product manager ml Ops or ml platform or ml info teams machine learning Engineers machine learning researchers or ml scientists data scientists so there's a bunch of different roles here and one kind of obvious question is what's the difference between all these different things so let's break down the job
+
+6
+00:03:14,400 --> 00:03:57,900
+function that each of these roles plays within the context of building the ml product starting with the ml product manager their goal is to work with the NL team the business team the users and any other stakeholders to prioritize projects and make sure that they're being executed well to meet the requirements of the rest of the organization so what they produce is things like design docs wireframes and work plans and they're using tools like jira like notion to help sort of organize the work of the rest of the team the ml Ops or ml platform team are focused on building the infrastructure needed in order to make models easier to deploy more scalable or generally reduce the workload of individual contributors
+
+7
+00:03:56,159 --> 00:04:38,160
+who are working on different ml models the output of what they build is some infrastructure some shared tools that can be used across the ml teams in your company and they're working with tools like AWS like Kafka or other data infrastructure tools and potentially working with ML infrastructure vendors as well to sort of bring best in breed from traditional data and software tools and this new category of ml vendors that are providing like mlops tools together to create this sort of best solution for the specific problems that your company is trying to solve then we have the ml engineer the ml engineer is kind of a catch-all role and the way that I like to think of their responsibilities is they're the person who is responsible
+
+8
+00:04:36,000 --> 00:05:15,660
+for training and deploying and maintaining the prediction model that powers the mlpark product they're not just the person who is you know solely training the model and then handing it off to someone else but they're also responsible for deploying it and then maintaining it once it's in production and so they need to know Technologies like tensorflow for training models but also like Docker for packaging models and making sure that they run on production infrastructure the next role is the ml researcher so this is a role that exists in some organizations that the responsibility stops after the model has been trained and so oftentimes these models are either handed off to some other team to productionize or these
+
+9
+00:05:13,800 --> 00:05:54,660
+folks are focused on building models that are not yet production critical or forward-looking maybe they're prototyping some use cases that might be useful down the line for the organization and their work product is a trained model and oftentimes it's a report or a code repo that describes what this model does how to use it and how to reproduce their results so they're working with ML trading tools and also prototyping tools like jupyter notebooks to produce a version of a model that just needs to work once to sort of show that the thing that they're trying to do is possible and then lastly we get the data scientist data scientist is kind of a patch-all term for potentially any of the things above in some organizations data science is quite
+
+10
+00:05:53,220 --> 00:06:31,319
+distinct from what we've been thinking of as a machine learning role in this class and these are folks in some organizations that are responsible for answering business questions using analytics so in some organizations a data scientists is you know the same as an ml researcher or an ml engineer and other organizations data science is a distinct function that is responsible for answering business questions using data the ml work is the responsibility of an ml team so the next thing we'll talk about is what are the different skills that you actually need to be successful in these roles we're going to plot this on a two by two on the x-axis is the amount of skill that you need in machine learning like how much ml do you
+
+11
+00:06:28,979 --> 00:07:07,500
+really need to know on the y-axis is the software engineering skill needed and then the size of the bubble is a requirement on communication or technical writing how good do you have to be at communicating your ideas to other people so starting with ML Ops or ml platform teams this is really primarily a software engineering role and oftentimes where these folks will come into the organization is through their you know traditional software engineering or data engineering hiring pipeline or even moving over from a data engineering role in another part of the organization another common pattern for how organizations find ml Ops or ml platform Engineers is they Source them from mles at their organization it's
+
+12
+00:07:05,580 --> 00:07:43,740
+oftentimes like an ml engineer who used to just work on one model and then got frustrated by the lack of tooling so decided to move into more of a platform role the ml engineer since this is someone who is required to understand the models deeply and also be able to productionize them this tends to be a rare mix of ml skills and software engineering skills so there's sort of two paths that I typically see for folks becoming ml Engineers oftentimes these are software Engineers who have a pretty significant amount of self-teaching or on the other hand maybe they are someone who's trained in machine learning traditionally like they have a science or engineering PhD but then they switch careers into software engineering after
+
+13
+00:07:41,460 --> 00:08:25,220
+grad school or after undergrad and then later decided to fuse those two skill sets ml researchers these are your ml experts so this is kind of the only role on this list that I would say it's pretty typical still to see a graduate degree or another path to these roles are these industrial Fellowship programs like Google brain residency that are explicitly designed to train people without a PhD in in this distinct skill of research since data science is kind of like a catch-all term for a bunch of different roles in different organizations it also admits a variety of different backgrounds and oftentimes these are undergrads who went to a data science specific program or their science phds who are making the
+
+14
+00:08:23,039 --> 00:09:05,220
+transition into industry and then lastly mlpms oftentimes these folks come from a traditional product management background but they do need to have a deep understanding of the specifics of the ml development process and that can come from having you know work closely with ML teams for a long time having just a really strong independent interest in ml or oftentimes what I see is folks who are you know former data scientists or ml Engineers who make the switch into PM it can be really effective at pming ml projects because they have a deep understand in the technology one other distinction that I think is worth covering when talking about the variety of different roles in ml organizations is the distinction between a task ml
+
+15
+00:09:02,880 --> 00:09:44,100
+engineer and a platform ml engineer this is a distinction that was coined by Shreya Shankar in blog post that's linked below and the distinction is that some ml Engineers are really responsible for like one ml pipeline or maybe a handful of ml pipelines that they're assigned to and so they're the ones that are day in and day out responsible for making sure that this model is healthy making sure that it's being updated frequently and that any failures are sort of being accounted for these folks are often like very overburdened this can be a very sort of expansive role because they have to be training models and deploying them and understanding where they break since mlgiers are often spread so thin some ml Engineers end up
+
+16
+00:09:41,100 --> 00:10:22,260
+taking on a role that looks more like a ml platform team or ml Ops Team where they work across teams to help ml Engineers automate tedious parts of their jobs and so we in our parlance this is called an ml platform engineer or ml Ops engineer but you'll also hear this referred to as an ml engineer or a platform ml engineer so we've talked a little bit about what are some of the different roles in the process of building ml Power Products now let's talk about hiring so how to think about hiring ml Specialists and if you are an ml specialist looking for a job how to think about making yourself more attractive as a job candidate so a few different things that we'll cover here the first is the AI Talent gap which is
+
+17
+00:10:20,339 --> 00:10:59,519
+sort of the reality of ml hiring these days and we'll talk about how to source for ML Engineers if you're hiring folks we'll talk about interviewing and then lastly we'll talk about finding a job four years ago when we started teaching full stack deep learning the AI Talent Gap was the main story in many cases for what teams found Difficult about building with ML there was just so so few people that understood this technology that the biggest thing blocking a lot of organizations was just they couldn't find people who are good at machine learning four years later the AI Talent Gap persists and there's still you know news stories every few months that are being written about how difficult it is for companies to find ml
+
+18
+00:10:57,120 --> 00:11:38,279
+talent but my observation day to day in the field is that it tends to be less of a blocker than it used to be because you know we've had four years of folks switching careers into ML and four years of you know software Engineers emerging from undergrad with at least a couple of ml classes in many cases under their belts so there's more and more people now that are capable of doing ml but there's still a gap and in particular that Gap tends to be in folks that understand more than just the underlying technology but also have experience in seeing how seeing how it fails and how to make it successful when it's deployed so that's the reality of how difficult it is to hire machine learning folks today especially those who have
+
+19
+00:11:36,360 --> 00:12:14,459
+production experience so if you are hiring ml folks how should you think about finding people if you're hiring ml product managers or ml platform or ml Ops Engineers the main skill set that you need to look for is still the sort of core underlying skill set for those roles so product management or data engineering or platform Engineering in general but it is critical to find folks who have experience at least interacting with teams that are building production ml systems because I think one sort of failure mode that I've seen relatively frequently especially for ML platform teams is if you just bring in folks with pure software engineering background a lot of times it's difficult for them to understand the user requirements well
+
+20
+00:12:12,660 --> 00:12:52,920
+enough in order to engineer things that actually solve the user's problems users here being the task mles who are you know the ones who are going to be using the infrastructure data that we'll focus for the rest of the section mostly on these two roles ml engineer and ml scientist so there's a right and a wrong way to hire ml engineers and the wrong way oftentimes looks maybe something like this so you see a job description for the Unicorn machine learning engineer the duties for this person are they need to keep up with seed of the art they need to implement new models from scratches that come out they need a deep understanding of the underlying mathematics and ability to invent new models for new tasks as it arises they
+
+21
+00:12:51,480 --> 00:13:30,060
+need to also be able to build tooling and infrastructure for the ml team because ml teams need tooling to do their jobs they need to be able to build data pipelines as well because without data ml is nothing they need to deploy these models and monitor them in production because without deploying models you're not actually solving a problem so in order to fulfill all these duties you need these requirements as this unicorn mle role you of course need a PhD you need at least four years of tensorflow experience four years as a software engineer you need to have Publications and nurips or other top ml conferences experience building large-scale distributed systems and so when you add all this up hopefully it's becoming
+
+22
+00:13:28,500 --> 00:14:09,720
+clear why this is the wrong way to hire ml Engineers there's just not really very many people that fit this description today if any and so the implication is the right way to hire ml Engineers is to be very very specific about what you actually need from these folks and in most cases the right answer is to primarily hire for software engineering skills not ml skills you do need folks that have at least a background in ml and a desire to learn ML and you can teach people how to do ml if they have a strong interest in it they know the basics and they're really strong in the software engineering side another approach instead of hiring for software engineering skills and training people in the on the ml side is to go
+
+23
+00:14:07,380 --> 00:14:46,680
+more Junior most undergrads in computer science these days graduate with ML experience and so these are folks that have traditional computer science training and some theoretical ml understanding so they have sort of the seeds of being good at both ML and software engineering but maybe not a lot of experience in either one and then the third way that you can do this more effectively is to be more specific about what you really really need for not the ml engineering function in general but for this particular role right so not every ml engineer needs to be a devops expert to be successful not every ml engineer needs to be able to implement new papers from scratch to be successful either for many of the MLS years that
+
+24
+00:14:45,240 --> 00:15:25,440
+you're hiring what they really need to do is something along the lines of taking a model that is you know pretty established as something that works while pulling it off the shelf or training it using a pretty robust library and then being able to deploy that model into production so focus on hiring people to have those skills not these aspirational skills that you don't actually really need for your company NeXT let's talk about a couple things I've found to be important for hiring ml researchers the first is a lot of folks when they're hiring ml researchers they look first at the number of Publications they have in top conferences I think it's really critical to focus entirely on the Quant the quality of public
+
+25
+00:15:23,160 --> 00:16:01,019
+locations not the quantity and this unfortunately requires a little bit of judgment about what high quality research looks like but hopefully there's someone on your team that can provide that judgment it's more interesting to me to find machine learning researchers who have you know one or two Publications that you think are really creative or very applicable to the field that you're working in or have really really strong promising results then to find someone who's you know published 20 papers but each of them are just sort of an incremental Improvement to the state of the art if you're working in the context of a company where you're trying to build a product and you're hiring researchers then I think another really important
+
+26
+00:15:59,339 --> 00:16:34,079
+thing to filter for is looking for researchers who have an eye for working on problems that really matter a lot of researchers maybe through no fault of their own just because of the incentives and Academia focus on problems that are trendy if everyone else is publishing about reinforcement learning then they'll publish about reinforcement learning if everyone else is publishing about generative models then they'll make an incremental improvements to generative models to get them a publication but what you really want to to look for I think is folks that have an independent sense of what problems are important to work on because in the context of your company no one's going to be telling these folks like hey this
+
+27
+00:16:33,300 --> 00:17:08,400
+is what everyone's going to be publishing about this year oftentimes experience outside of Academia can be a good proxy for this but it's not really necessary it's just sort of one signal to look at if you already have a research team established then it's worth considering hiring talented people from adjacent Fields hiring from physics or statistics or math at open AI they did this with to really strong effects they would look for sort of folks that were really technically talented but didn't have a lot of ml expertise and they would train them in them out this works a lot better if you do have experienced researchers who can provide mentorship and guidance for folks I probably wouldn't hire like a first
+
+28
+00:17:06,780 --> 00:17:47,280
+researcher that doesn't have ml experience and then it's also worth remembering that especially these days you really don't need a PhD to do ml research many undergrads have a lot of experience doing ml research and graduates of some of these industrial Fellowship programs like Googles or Facebooks or open AIS have learned the basics of how to do research regardless of whether they have a PhD so that's how to think about evaluating candidates for ML engineering or ml research roles the next thing I want to talk about is how to actually find those candidates so your standard sources like LinkedIn or recruiters or on campus recruiting all work but another thing that can be really effective if you want to go
+
+29
+00:17:44,280 --> 00:18:26,580
+deeper is every time there's a new dump of papers on archive or every year at nurips and other top conferences just keep an eye on what you think are the most exciting papers and flag mostly the first authors of those papers because those are the ones that tend to be doing most of the work and are generally more recruitable because they tend to be more Junior in their careers Beyond looking at papers you can also do something similar for good re-implementations of papers that like so if you are you know looking at some hot new paper and a week later there's a re-implementation of that paper that has high quality code and hits the main results then chances are whoever wrote that implementation is probably pretty good and so they could
+
+30
+00:18:24,960 --> 00:19:01,919
+be worth recruiting you can do a lot of this in person now that ml research conferences are back in person or you can just reach out to folks that you are interested in talking to over the Internet since there's a talent shortage in ml it's not enough just to know how to find good ml candidates and evaluate them you also need to know how to think about attracting them to your company I want to talk a little bit about from what I've seen what a lot of ml practitioners are interested in the roles they take and then talk about ways that you can make your company Stand Out along those axes so one thing a lot of ml practitioners want is to work with Cutting Edge tools and techniques to be working with latest state of the art
+
+31
+00:19:00,419 --> 00:19:36,780
+research another thing is to build knowledge in an exciting field to like a more exciting branch of ml or application of ml working with excellent people probably pretty consistent across many technical domains but certainly true in ml working on interesting data sets this is kind of one unique thing in ml since the work that you can do is constrained in many cases the data sets that you have access to being able to offer unique data sets can be pretty powerful probably again true in general but I've noticed for a lot of ml folks in particular it's important for them to feel like they're doing work that really matters so how do you stand out on these axes you can work on Research oriented projects even if the sort of mandate of
+
+32
+00:19:35,580 --> 00:20:12,780
+your team is primarily to help your company doing some research work that you can publicize and that you could point to as being indicative of working on The Cutting Edge open source libraries things like that can really help attract top candidates if you want to emphasize the ability of folks to sort of build skills and knowledge in an exciting field you can build a team culture around learning so you can host reading groups in your company you can organize learning days which is something that we did at open AI where we would dedicate back then a day per week just to be focused on learning new things but you can do it less frequently than that professional development budgets conference budgets things like
+
+33
+00:20:11,340 --> 00:20:50,820
+this that you can emphasize and this is probably especially valuable if your strategy is to hire more Junior folks or more software engineering oriented folks and train them up in machine learning emphasize how much they'll be able to learn about MLA company one sort of hack to being able to hire good ml people is to have other good ml people on the team this is maybe easier said than done but one really high profile hire can help attract many many other people in the field and if you don't have the luxury of having someone high profile on your team you can help your existing team become more high profile by helping them publish blogs and papers so that other people start to know how talented your team actually is when you're attracting
+
+34
+00:20:48,240 --> 00:21:27,419
+ml candidates you can focus on sort of emphasizing the uniqueness of your data set in recruiting materials so if you have know the best data set for a particular subset of the legal field or the medical field emphasize how interesting that is to work with how much data you have and how unique it is that you have it and then lastly you know just like any other type of recruiting selling the mission of the company and the potential for ML to have an impact on that mission can be really effective next let's talk about ml interviews what I would recommend testing for if you are on the interviewer side of an ml interview is to try to hire for strengths and meet a minimum bar for everything else and this can help you avoid falling into the Trap
+
+35
+00:21:24,780 --> 00:22:01,380
+of looking for unicorn mles so some things that you can test are you want to validate your hypotheses of candidate strengths so if it's a researcher you want to make sure that they can think creatively about new ml problems and one way you can do this is to probe how thoughtful they were about previous projects if they're Engineers if they're mles then you want to make sure that they're great generalist software Engineers since that's sort of the core skill set in ml engineering and then you want to make sure they meet a minimum bar on weaker areas so for researchers I would advocate for only hiring researchers in Industry contexts who have at least the very basics in place about software engineering knowledge and
+
+36
+00:21:59,220 --> 00:22:33,299
+the ability to write like decent code if not you know really high quality production ready code because in context of working with a team other people are going to need to use their code and it's not something that everyone learns how to do when they're in grad school for ML for software Engineers you want to make sure that they at least meet a minimum bar on machine learning knowledge and this is really testing for like are they passionate about this field that they have put in the requisite effort to learn the basics of ml that's a good indication that they're going to learn ml quickly on the job if you're hiring them mostly for their software engineering skills so what do ml interviews actually consist of so this
+
+37
+00:22:31,320 --> 00:23:10,380
+is today much less well defined than your software engineering interviews some common types of Assessments that I've seen are your normal sort of background and culture fit interviews whiteboard coding interviews similar to you'd see in software engineering pair coding like in software engineering but some more ml specific ones include pair debugging where you and an interviewer will sit down and run some ml code and try to find Hey where's the bug in this code oftentimes this is ml specific code and the goal is to test for how well is this person able to find bugs in ml code since bugs tend to be where we spend most of our time in machine learning math puzzles are often common especially involving things like linear algebra
+
+38
+00:23:08,340 --> 00:23:46,080
+take-home projects other types of Assessments include applied ml questions so typically this will have the flavor of hey here's a problem that we're trying to solve with ML let's talk through the sort of high level pieces of how we'd solve it what type of algorithm we'd use what type of system them we need to build to support it another Common Assessment is probing the past projects that you've listed on your resume or listed as part of the interview process asking you about things you tried will work what didn't work and trying to assess what role you played in that project and how thoroughly you thought through the different alternative paths that you could have considered and then lastly ml Theory questions are also pretty common
+
+39
+00:23:43,919 --> 00:24:20,880
+in these interview type assessments that's sort of the universe of things that you might consider interviewing for if you're trying to hire ml folks or that you might expect to find on an ml interview if you are on the the other side and trying to interview for one of these jobs and the last thing I'll say on interviews is there's a great book from chipwin the introduction to machine learning interviews book which is available for free online which is especially useful I think if you're preparing to interview for machine learning roles speaking of which what else should you be doing if your goal is to find new job in machine learning the first question I typically hear is like where should I even look for ML jobs
+
+40
+00:24:19,080 --> 00:24:51,059
+your standard sources like LinkedIn and recruiters all work ml Research Conference references can also be a fantastic place just go up and talk to the folks that are standing around the booths at those conferences they tend to be you know looking for candidates and you can also just apply directly and this is sort of something that people tell you not to do for most roles but remember there's a talent Gap in machine learning so this can actually be more effective than you might think when you're applying what's the best way to think about how to stand out for these roles so I think like sort of a baseline thing is for many companies they really want to see that you're expressing some sort of interest in ml you've been
+
+41
+00:24:49,260 --> 00:25:29,460
+attending conferences you've been taking online courses you've been doing something to sort of put get your foot in the door for getting into the field better than that is being able to demonstrate that you have some software engineering skills again for many ml organizations hiring for software engineering is in many ways more important than hiring for ML skills if you can show that you have a broad knowledge of ml so writing blog posts that synthesize a particular research area or articulating a particular algorithm in a way that is that is new or creative or compelling can be a great way to stand out but even better than that is demonstrating an ability to you know ship ml projects and the best way to do this I think if you are not
+
+42
+00:25:27,600 --> 00:26:04,980
+working in ml full-time right now is through side projects these can be ideas of whatever you want to work on they can be paper re-implementation so they can be your project for this course and then probably if you really want to stand out maybe the most impressive thing that you can do is to prove that you can think creatively in ml right think Beyond just reproducing things that other people have done but be able to you know win kaggle competitions or publish papers and so this is definitely not necessary to get a job in ml but this will sort of put your resume at the top of the stack so we've talked about some of the different roles that are involved in building ml products and how to think about hiring for those roles or being
+
+43
+00:26:03,419 --> 00:26:43,320
+hired for those roles the next thing that we're going to talk about is how machine learning teams fit into the context of the rest of the organization since we're still in the relatively early days of adopting this technology there's no real consensus yet in terms of the best way to structure an ml team but what we'll cover today is taxonomy of some of the best practices for different security levels of organizations and how they think about structuring their ml teams and so we'll think about this as scaling a mountain from least mature ml team to most mature so the bottom of the mountain is the nascent or ad hoc ml archetype so what this looks like is you know your company has just started thinking about mL no
+
+44
+00:26:41,940 --> 00:27:20,460
+one's really doing it yet or maybe there's a little of it being done on an ad hoc basis by the analytics team or one of the product teams and most smaller medium businesses are at most in this category but some of the less technology for larger organizations still fall in this category as well so the great thing about being at this stage is that there's a ton of low hanging fruit often for ML to come in and help solve but the disadvantage if you're going to go in and work in an organization at this stage is that there's often little support available for ML projects you probably won't have any infrastructure that you can rely on and it can be difficult to hire and retain good talent plus leadership in the company may not really be bought
+
+45
+00:27:18,419 --> 00:27:59,940
+into how useful ml could be so that's some of the things to think about if you're going to go take a role role in one of these organizations once the company has decided hey this ml thing is something exciting something that we should invest in typically they'll move up to an ml r d stage so what this looks like is they'll have a specific team or specific like subset of their r d organization that's focused on machine learning they'll typically hire researchers or phds and these folks will be focused on building prototypes internally or potentially doing external facing research so some of the larger oil and gas companies manufacturing companies telecom companies were in the stage even just a few years ago although
+
+46
+00:27:58,260 --> 00:28:36,000
+they've in many cases moved on from it now if you're going to go work in one of these organizations one of the big advantages is you can get away with being less experienced on the research side and since the ml team isn't really going to be on the hook today for any sort of meaningful business outcomes another big Advantage is that these teams can work on long-term business priorities and they can focus on trying to get to what would be really big wins for the organization but the disadvantage to be aware of if you're thinking about joining a team at this stage or building a team at this stage is that oftentimes since the ml team is sort of siled off into an R D part of the organization or a separate team from
+
+47
+00:28:34,080 --> 00:29:11,580
+the different products initiatives it can be difficult for them to get the data that they need to solve the problems that they need to solve it's just not a priority in many cases for other parts of the business to give them the data and then probably the biggest disadvantage of this stage is that you know it doesn't usually work it doesn't usually translate to business value for the organization and so oftentimes ml teams kind of get stuck at this stage where they don't invest very much in ml and ml is kind of siled and so they don't see strong results and they can't really justify doubling down the next evolution of ml organizations oftentimes is embedding machine learning directly into business and product teams so what
+
+48
+00:29:09,900 --> 00:29:52,140
+this looks like is you'll have some product teams within the organization that have a handful of ml people side by side with their software or analytics teams and these ml teams will report up into the sort of engineering or Tech organizations directly instead of being in their own sort of reporting arm a lot of tech companies when they start adopting ml sort of pretty quickly get to this category because they're pretty agile software organizations and pretty Tech forward organizations anyway and a lot of the financial services company is tend towards this model as well the big sort of overwhelming advantage of this organizational model is that when these ml teams ship stuff successfully it almost always is able to translate
+
+49
+00:29:50,159 --> 00:30:27,419
+pretty directly to business value since the people that are doing ml sit side by side with the folks that are you know building the product or building the feature that the ml is going to be part of and this gives them a really tight feedback cycle between new ideas that they have for how to make the ml better how to make the product better with ml into actual results as part of the products the disadvantages of building ml this way are oftentimes it can be hard to hire and develop really really great ml people because great ml people often want to work with other great ml people it can also be difficult to get these ml folks access to the resources that they need to be really successful so that's the infrastructure they need
+
+50
+00:30:25,620 --> 00:31:03,360
+the data they need or the compute they need because they don't have sort of a central team that reports high up in the organization to ask for help and one other disadvantage of this model is that oftentimes this is where you see conflicts between the way that ml projects are run the sort of iterative process that is high risk and the way that the software teams that these ml folks are a part of are organized sometimes you'll see conflict between folks getting frustrated with the ml folks on their team for not shipping quickly or not being able to sort of commit to a timeline that they promised the next ml organization architect will cover is independent machine learning's function what this looks like is you'll
+
+51
+00:31:01,799 --> 00:31:42,960
+have a machine learning division of the company that reports up to senior leadership so they report to the CEO or the CTO or something along those lines this is what distinguishes it from the mlr D archetype where the ml team is often you know reporting to someone more Junior in the organization often a foreigner as sort of a smaller bet this is the organization making a big bet to investing in machine learning so oftentimes this is also the archetype where you'll start to see mlpms or platform nlpms that work with researchers and ml engineers and some of these other roles in order to deliver like a cross-functional product the big advantage of this model is access to resources so since you have a centralized ml team you can often hire
+
+52
+00:31:40,679 --> 00:32:18,960
+really really talented people and build a talent density in the organization and you can also train people more easily since you have more ml people sitting in a room together or in a zoom room together in some cases since you report to senior leadership you can also often like Marshal more resources in terms of data from the rest of the organization or budget for compute than you can in other archetypes and it makes it a lot easier when you have a centralized organization to invest in things like tooling and infrastructure and culture and best practices around developing ml in your organization the big disadvantage of this model is that it leads to handoffs and that can add friction to the process that you as an
+
+53
+00:32:16,980 --> 00:32:54,720
+ml team need to run in order to actually get your models into production and the last ml organization archetype the the end State the goal if you're trying to build ml the right way in your organization is to be an ml first organization so what this looks like is you have buy-in up and down the organization that ml is something that you as a company want to invest in you have an ml division that works on the most challenging long-term projects and invests in sort of centralized data and centralized infrastructure but you also have expertise in ml in every line of business that focuses on quick wins and working with the central ml division to sort of translate the ideas they have the implementations they make into
+
+54
+00:32:52,320 --> 00:33:30,480
+actual outcomes for the products that the company is building so you'll see this in the biggest tech companies like the Googles and Facebooks of the world as well as startups that were founded with ML as a core guiding principle for how they want to build the products and these days more and more you're starting to see other tech companies who began investing in ml four or five years ago start to become closer to this archetype there's mostly advantages to this model you have great access to data It's relatively easy to recruit and most importantly it's probably easiest in this archetype out of all them to get value out of ml because the products teams that you're working with understand machine learning and really
+
+55
+00:33:28,799 --> 00:34:07,740
+the only disadvantage of this model is that it's difficult and expensive and it takes a long time for organizations that weren't born with this mindset to adopt it because you have to recruit a lot of really good ml people and you need to culturally embed ml thinking into your organization the next thing that we'll talk about is some of the design choices you need to make if you're building an ml team we'll talk about how those depend on the archetype of the organization that you fit into the first question is software engineering versus research so to what extent is the mltm responsible for building software versus just training models the second question is data ownership so is the ml team also responsible for creating publishing data
+
+56
+00:34:06,240 --> 00:34:43,379
+or do they just consume that from other teams and the last thing is model ownership the ml team are they the ones that are going to productionize models or is that the responsibility of some other team in the mlr D archetype typically you'll prioritize research over software engineering skills and the MLT won't really have any ownership over the data or oftentimes even the skill sets to build data pipelines themselves and similarly they won't be responsible for deploying models either and in particular models will rarely make it into production so that won't really be a huge issue embedded ml teams typically they'll prioritize software engineering skills over research skills and all researchers if they even have
+
+57
+00:34:42,060 --> 00:35:21,359
+researchers will need to have strong software engineers skills because everyone's expected to deploy it ml teams still generally doesn't own data because they are working with data Engineers from the rest of the organizations to build data pipelines but since the expectation in these types of organizations is that everyone deploys typically ml Engineers will own maintenance of the models that they deploy in the ml function archetype typically the requirement will be that you'll need to have a team that has a strong mix of software engineering research and data skills so the team size here starts to become larger a minimum might be something like one data engineer one ml engineer potentially a platform engineer or a devops engineer
+
+58
+00:35:18,839 --> 00:35:57,420
+and potentially a PM but these teams are often working with a bunch of other functions so they can in many cases get much larger than that and you know in many cases in these organizations you'll have both software engineers and researchers working closely together within the context of a single team usually at this stage ml teams will start to have a voice in data governance discussions and they'll probably also have some strong internal data engineering functions as well and then since the ml team is centralized at this stage they'll hand off models to a user but in many cases they'll still be responsible for maintaining them although that line is blurry in a lot of organizations that run this model finally in ml first organizations
+
+59
+00:35:55,320 --> 00:36:32,700
+there's no real standardization around how teams are research oriented or not but research teams do tend to work pretty closely with software engineering teams to get things done in some cases the ml team is actually the one that owns company-wide data infrastructure because ml is such a central bet for the company that it makes sense for the ml team to make some of the sort of main decisions about how data will be organized then finally if the ml team is the one that actually built the model they'll typically hand it off to a user who since they have the basic ml skills and knowledge to do this they'll actually be the one to maintain the model and here's all this on one slide if you want to look at it all together
+
+60
+00:36:31,140 --> 00:37:06,720
+all right so we've talked about machine learning teams and organizations and how these come together and the next thing that we're going to talk about is team management and product management for machine learning so the first thing to know about product management and team management for ML is that it tends to be really challenging there's a few reasons for this the first is that it's hard to tell in advance how easy or hard something is going to be so this is an example from a blog post by Lucas B Walt where they ran a kaggle competition and in the first week of that kago competition they saw a huge increase in the accuracy of the best performing model they went from 35 to 70 accuracy within one week and they were thinking
+
+61
+00:37:04,859 --> 00:37:42,359
+this is great like we're gonna hit 95 accuracy and this contest is going to be a huge success but then if you zoom out and look at the entire course of the project over three months it turns out that most of that accuracy gain came in the first week and the improvements thereafter were just marginal and that's not because of a lack of effort the number of participating teams was still growing really rapidly over the course of that time so the upshot is it's really hard to tell in advance how easier or hard something is in ml and looking at signals like how quickly are we able to make progress on this project can be very misleading or related challenge is that progress on ML projects tends to be very non-linear so
+
+62
+00:37:40,260 --> 00:38:17,460
+it's very common for projects to stall for weeks or longer because the ideas that you're trying just don't work or because you hit some sort of unforeseen snag with not having the right data or something like that that causes you to really get stuck and on top of that in the earliest stages of doing the project it can be very difficult to plan and to tell how long the project is going to take because it's unclear what approach will actually work for training a model that's good enough to solve the problem and the upshot of all this is that estimating the timeline for a project when you're in the project planning phase can be very difficult in other words production ml is still somewhere between research and Engineering another
+
+63
+00:38:14,700 --> 00:38:54,540
+challenge for managing ml teams is that there's cultural gaps that exist between research and Engineering organizations these folks tend to come from different backgrounds they have different training they have different values goals and norms for example oftentimes you know stereotypically researchers care about novelty and about how exciting the approach is that they took to solve a problem whereas you know again stereotypically oftentimes software Engineers care about did we make the thing work and in more toxic cultures these two sides often can class and even if they don't Clash directly they might not really value each other as much as they should because both sides are often necessary to build the thing that you
+
+64
+00:38:53,099 --> 00:39:28,740
+want to build to make batteries worse when you're managing a team as part of an organization you're not just responsible for making sure the team does what they're supposed to do but you'll also have to manage up to help leadership understand your progress and what the Outlook is for the thing that you're building since ml is such a new technology many leaders and organizations even in good technology organizations don't really understand it so next I want to talk about some of the ways that you can manage machine learning projects better and the first approach that I'll talk about is doing project planning probabilistically so oftentimes when we think about project planning for software projects we think
+
+65
+00:39:26,520 --> 00:40:07,740
+about it as sort of a waterfall right where you have a set of tasks and you have a set of time estimates for those tasks and a set of dependencies for those tasks and you can plan these out one after another so if task G depends on tasks D and F then task G will happen once those are done if task D depends on C which depends on task a you'll start Task D after a and C are done Etc but in machine learning this can lead to frustration and badly estimated timelines because each of these projects has a higher chance of failure than it does in a typical software project what we ended up doing at open AI was doing project planning probabilistically so rather than assuming that like a particular task is going to take a
+
+66
+00:40:06,060 --> 00:40:45,540
+certain amount of time instead we assign probabilities to let the likelihood of completion of each of these tasks and potentially pursue alternate tasks that allow us to unlock the same dependency in parallel so in this example you know maybe task fee and task C are both alternative approaches to unlocking task D so we might do both of them at the same time and so if we realize all of a sudden that task C is not going to work and task B is taking longer than we expected then we can adjust the timeline appropriately and then we can start planning the next wave of tasks once we know how we're going to solve the prerequisite tasks that we needed a coral area of doing machine learning project planning probabilistically is
+
+67
+00:40:43,560 --> 00:41:20,700
+that you you shouldn't have any path critical projects that are fundamentally research research projects have a very very high rate of failure rather than just saying like this is how we're going to solve this problem instead you should be willing to try a variety of approaches to solve that problem that doesn't necessarily mean that you need to do them all in parallel but many good machine learning organizations do so one way to think about this is you know if you know that you need to build like a model that's never been built in your organization before you can have like a friendly competition of ideas if you have a culture that's built around working together as a team to get to the right answer and not just rewarding the
+
+68
+00:41:19,140 --> 00:41:55,079
+one person who solves the problem correctly another corollary to this idea that that many machine learning ideas can and will fail is that when you're doing Performance Management it's important not to get hung up on just who is the person whose ideas worked in the long term it's important for people to do things that work like over the course of you know many many months or years if nothing that you try works then that's maybe an indication that you're not trying the right things you're not executing effectively but on at any given project object like on a timeline of weeks or a quarter then the success measure that you should be looking at is how well you executed on the project not whether the project happened to be one
+
+69
+00:41:53,579 --> 00:42:29,820
+of the ones that worked one failure mode that I've seen in organizations that hire both researchers and Engineers is implicitly valuing one side more than the other so thinking engineering is more important than research which can lead to things getting stuck on the ml side because the ml side is not getting the resources or attention that they deserve or thinking that research is more important than engineering which can lead to creating ml innovations that are not actually useful so oftentimes the way around this is to have engineers and researchers work very closely together in fact like sometimes uncomfortably close together like working together on the same code base for the same project and understanding
+
+70
+00:42:28,320 --> 00:43:05,700
+that these folks bring different skill sets to the table another key to success I've seen is trying to get quick wins so rather than trying to build a perfect model and then deploy it trying to ship something quickly to demonstrate that this thing can work and then iterate on it over time and then the last thing that you need to do if you're in a position of being the product manager or the engineering manager for an ml team is to put more emphasis than you might think that you need on educating the rest of your organization on how ml Works diving into that a bit more if your organization is relatively new to adopting ml I'd be willing to bet that a lot of people in the organization don't understand one or more of these things
+
+71
+00:43:03,359 --> 00:43:43,260
+for us as like ml practitioners it can be really natural to think about where ml can and can't be used but for a lot of technologists or Business Leaders that are new to ml the uses of ml that are practical can be kind of counter-intuitive and so they might have ideas for ML projects that are feasible and they might miss ideas for ML projects that are pretty easy that don't fit their mental model of what ml can use another common point of friction in dealing with the rest of the organization is convincing the rest of the organization that the ml that you built actually works Business Leaders and folks from product teams typically the same metrics that convince us as ml practitioners that this model is useful
+
+72
+00:43:41,460 --> 00:44:18,780
+won't convince them like just looking at an F1 score or an accuracy score doesn't really tell them what they need to know about whether this model is really solving the task that it needs to solve for the business outcome that they're aiming for and one particular way that this presents itself pretty frequently is in Business Leaders and other stakeholders not really sort of wrapping their heads around the fact that ml is inherently probabilistic and that means that it will fail in production and so a lot of times where ml efforts get hung up is in the same stakeholders potentially that champion the project to begin with not really being able to get comfortable with the fact that once the model is out in the world it's you know
+
+73
+00:44:17,339 --> 00:44:57,780
+the users are going to start to see failures that it makes in almost all cases and the last common failure mode in working with the rest of the organization is the rest of the organization treating ml projects like other software projects and not realizing that they need to be managed differently than other software projects too and one particular way that I've seen this become a problem is when leadership gets frustrated at ml team because they're not able to really accurately convey how long projects are going to take to complete so educating leadership and other stakeholders on the probabilistic nature of ml projects is important to maintaining your sanity as an ml team if you want to share some resources with your execs that they can
+
+74
+00:44:55,500 --> 00:45:37,079
+use to learn more about how these projects play out in the practice of real organizations I would recommend Peter beale's AI strategy class from the business school at UC Berkeley and Google's people in AI guidebook which we'll be referring to a lot more in the rest of the lecture as well the last thing I'll say on educating the rest of the organization on ml is that mlpms I think play like one of the most critical roles in doing this effectively to illustrate this I'm going to make an analogy to the two types of ml engineers and describe two prototypal types of mlpms that I see in different organizations so on one hand we have our task mlpms these are like a PM that's responsible for a specific product or
+
+75
+00:45:35,280 --> 00:46:16,680
+specific product feature that heavily uses ml these folks will need to have a pretty specialized knowledge of ML and how it applies to the particular domain that they're working on so for example they might be the PM for the trust and safety product for your team or particular recommendation product for your team and these are probably the more common type of mlpms in Industry today but an emerging type of mlpm is the platform mlpm platform mlpms tend to start to make sense when you have a centralized ml team and that centralized ml team needs to play some role in educating the rest of the organization in terms of like what are productive uses of ml in all the products that the organization is building because these
+
+76
+00:46:14,640 --> 00:46:55,260
+folks are responsible for managing the workflow in and out of the ml team so helping filter out projects that aren't really high priority for the business or aren't good uses of ml helping proactively find projects that might have a big impact on the the product or the company by spending a lot of time with PMS from the rest of the organization and communicating those priorities to the ml team and outward to the rest of the organization this requires a broad knowledge of ml because a lot of what this role entails is trying to really understand where ml tan and should and shouldn't be applied in the context of all the things the organization is doing and one of the other critical roles that platform MLT
+
+77
+00:46:52,619 --> 00:47:30,720
+and PMs could play is spreading ml knowledge and culture throughout the rest of the organization not just going to PMs and business stakeholders from the other product functions and Gathering requirements from them but also helping educate them on what's possible to do with ML and helping them come up with ideas to use ml in their areas of responsibility that they find exciting so that they can over time really start to build their own intuition about what types of things they should be considering ml to be used for and then another really critical role that these platform mlpms can play is mitigating the risks of you know we've built a model but we can't convince the rest of the organization to actually use it by being really crisp
+
+78
+00:47:28,920 --> 00:48:07,440
+about what are the requirements that we actually need this model to fulfill and then proactively communicating with the other folks that need to be bought in about the model's performance to help them understand all the things that they'll need to understand about them also really trust its performance so platform mlpms are or I think a newer Trend in ml organizations but I think one that can have a big impact on the success of ml organizations when you're in this phase starting to build a centralized ml team or trans transition from a centralized ml team to becoming an ml first organization one question I get a lot about ml product management is what's the equivalent of agile or any of these established development
+
+79
+00:48:05,280 --> 00:48:46,980
+methodologies for software in ml is there something like that that we can just take off the shelf and apply and deliver successful ml products and the answer is there's a couple of emerging ml project management methodologies the first is Chris DM which is actually an older methodology but it was originally focused on Data Mining and has been subsequently applied to data science and ML and the second is the team data science process tdsp from Microsoft what these two things have in common is that they describe the stages of ml projects as sort of a loop where you start by trying to understand the problem that you're trying to solve acquiring data building a model evaluating it and then finally deploying it so the main reason
+
+80
+00:48:45,180 --> 00:49:24,960
+to use one of these methodologies would be if you really want standardization for what you call the different stages of the Project Life Cycle if you're choosing between these two tdsp tends to be a little bit more structured it provides like sort of more granular list of roles responsibilities templates that you can use to actually execute on this process crisp DM is a bit higher level so if you need an actual like granular project management framework then I would start by trying tdsp but I'll see more generally it's reasonable to use these if you truly have a large scale coordination problem if you're trying to get a large ml team working together successfully for the first time but I would otherwise recommend skipping these
+
+81
+00:49:23,280 --> 00:50:03,660
+because they're more focused on traditional data mining or data science processes and they'll probably slow you down so I would sort of exercise caution before implementing one of these methodologies in full the last thing I want to talk about is designing products that lend themselves well to being powered by Machine learning so I think the fundamental challenge in doing this is a gap between in what users expect when they're ended in AI powered products and what they actually get and so what users tend to think when they're given an AI powered product is you know their mental model is often human intelligence but better and in Silicon so they think it um has this knowledge of the world that it as achieved by
+
+82
+00:50:02,099 --> 00:50:41,099
+reading the whole internet oftentimes they think that this product knows me better than I know myself because it has all the data about me from every interaction I've ever had with software they think that AI Power Products learn from their mistakes and that they generalize to new problems right because it's intelligence it's able to learn from new examples to solve new tasks but I think a better mental model for what you actually get with an ml powered products is a dog that you train to solve a puzzle right so it's amazing that it can solve the puzzle and it's able to solve surprisingly hard puzzles but at the end of the day it's just a dog solving a puzzle and in particular dogs are weird little guys right they
+
+83
+00:50:39,000 --> 00:51:18,180
+tend to fail and strange and unexpected ways that you know we as people with like human intelligence might not expect they also get distracted easily right like if you take them outside they might not be able to solve the same problem that they're able to solve inside they don't generalize outside of a narrow domain The Stereotype is that you can't teach an old dog new tricks and in ml it's often hard to adapt general knowledge should new tasks or new contexts dogs are great at learning tricks but they can't do it if you don't give them treats and similarly machine Learning Systems don't tend to learn well without feedback or rewards in place to help understand where they're performing well and where they're not
+
+84
+00:51:16,020 --> 00:51:54,240
+performing well and lastly both dogs learning tricks and machine Learning Systems might misbehave if you leave them unattended the implication is that there's a big gap between users mental model for machine learning products and what they actually get from machine learning products so the upshot is that the goal of good ml product design is to bridge the user's expectation with reality and there's a few components to that the first is helping users understand what they're actually getting from the model and also its limitations the the second is that since failures are inevitable we need to be able to handle those failures gracefully which means not over relying on Automation and being able to fall back in many cases
+
+85
+00:51:52,260 --> 00:52:37,140
+too human in the loop and then the final goal of ml product design is to build in feedback loops that help us use data from our users to actually improve the system one of the best practices for ML product design is explaining the benefits and limitations of the system to users one way that you can do that is since users tend to have misconceptions about what AI can and can't do focus on what problem the product is actually solving for the user not on the fact that it's AI powered and similarly the more open-ended and human feeling you make the product experience like allowing users to enter any information that they want to or ask questions in whatever natural language that they want to the more they're going to treat it as
+
+86
+00:52:34,980 --> 00:53:15,240
+human-like and expose some of the failure modes that the system still has so one example of this was when Amazon Alexa was first released one of the sort of controversial decisions that they made was they limited it to a very specific set of prompts that you could say to it rather than having it be an open-ended language or dialogue system and that allowed them to really focus on training users to interact with the system in a way that it was likely to be able to understand and then finally the reality is that your model has limitations and so you should explain those limitations to users and consider actually just baking those limitations into the model as guardrails so not letting your users provide input to your
+
+87
+00:53:13,680 --> 00:53:53,880
+model that you know the model is not going to perform well on so that could be as simple as you know if your NLP system was designed to perform well on English text then detecting if users input text in some other language and you know either warning them or not allowing them to input text in a language where your model is not going to perform well the next best practice for ML product design is to not over rely on Automation and instead try to design where possible for a human in the loop automation is great but failed automation can be worse than automation at all so it's worth thinking about even if you know what the right answer is for your users how can you add low friction ways to let users confirm the model's
+
+88
+00:53:52,140 --> 00:54:28,920
+predictions so that they don't have a terrible experience when the model does something wrong and they have no way to fix it one example of this was back when Facebook had an auto tagging feature of you know recognizing your face and pictures and suggesting who the person was they didn't just assign the tag to the face even though they almost always knew exactly who that person was because it'd be a really bad experience if all of a sudden you were tagged in some picture of someone else instead they just add like simple yes no that lets you confirm that they in fact got the prediction that this is your face correctly in order to mitigate the effect of when the model inevitably does make some bad predictions there's a
+
+89
+00:54:27,720 --> 00:55:06,300
+couple of patterns that can help there the first is it's a really good idea to always bake in some way of letting users take control of the system like in a self-driving car to be able to grab the wheel and steer the car back on track if it makes a mistake and another pattern for mitigating the cost so bad predictions is looking at how confident the model is in its response and maybe being prudent about only showing responses to users that are pretty high confidence potentially falling back to a rules-based system or just telling the user that you don't have a good answer to that question the third best practice for ML product design is building in feedback loops with your users so let's talk about some of the different types
+
+90
+00:55:04,859 --> 00:55:42,000
+of feedback that you might collect from your users on the x-axis is how easy it is to use the feedback that you get in order to actually directly make your model better on the y-axis is how much friction does it add to your users to collect this feedback so roughly speaking you could think about like above this line on the middle of the chart is implicit feedback that you collect from your users without needing to change their behavior and on the right side of the chart are signals that you can train on directly without needing to have some human intervention the type of feedback that introduces the least friction to your user is just collecting indirect implicit feedback on how well the prediction is working for
+
+91
+00:55:40,079 --> 00:56:17,520
+them so these are signals about user behavior that tend to be a proxy for mobile performance like did the user churn or not these tend to be super easy to collect because they're often instrumented in your product already and they're really useful because they correspond to important outcomes for our products the challenge in using these is that it's often very difficult to tell whether the model is the cause because these are high level sort of business outcomes that may depend on many other things other than just your model's prediction so to get more directly useful signals from your users you can consider collecting direct implicit feedback where you collect signals from the products that measure how useful
+
+92
+00:56:15,240 --> 00:56:51,240
+this prediction is to the user directly rather than indirectly for example if you're giving the user a recommendation you can measure whether they clicked on the recommendation or if you're suggesting an email for them to send did they send that email or did they copy the suggestion so they can use it in some other application oftentimes these take the form of did the user take the next step in whatever process that they're running that they take the prediction you gave them and use it Downstream for whatever tasks they're trying to do the great thing about this type of feedback is that you can often train on directly because it gives you a signal about you know which predictions the model made that were actually good
+
+93
+00:56:48,900 --> 00:57:27,119
+at solving the task for the user but the challenge is that not every setup of your product lends itself to collecting this type of feedback so you may need to redesign your products in order to collect feedback like this next we'll move on to explicit types of user feedback explicit feedback is where you ask your user directly to provide feedback on the model's performance and the lowest friction way to do this for users tends to be to give them some sort of binary feedback mechanism which can be like a thumbs up or thumbs down button in your product this is pretty easy for users because it just requires them to like click one button and it can be a decent training signal there's some research and using signals like this in
+
+94
+00:57:24,660 --> 00:58:03,960
+order to guide the learning process of models to be more aligned with users preferences if you want a little bit more signal than just was this prediction good or bad you can also ask users to help you categorize the feedback that they're giving they could for example like flag certain predictions as incorrect or offensive or irrelevant or not useful to me you can even set this up as a second step in the process after binary feedback so users will still give you binary feedback even if they don't want to spend the time to categorize that feedback and these signals can be really useful for debugging but it's difficult to set things up in such a way that you can train on them directly another way you can get more granular feedback on Mall's
+
+95
+00:58:02,220 --> 00:58:38,040
+predictions is to have like some sort of free text input where users can tell you what they thought about in prediction this often manifests itself in support tickets or support requests for your model this requires a lot of work on the part of your users and it can be very difficult to use as a model developer because you have to parse through this like unstructured feedback about your model's predictions yet it tends to be quite useful sometimes in practice because since it's high friction to actually provide this kind of feedback the feedback that users do provide can be very high signal it can highlight in some cases like the highest friction predictions since users are willing to put in the time to complain about them
+
+96
+00:58:36,180 --> 00:59:19,200
+and then finally the gold standard for user feedback if it's possible to do in the context of your products and your user experience is is to have users correct the predictions that your model actually makes so if you can get users to label stuff for you directly then that's great then you're in a really good spot here and so one way to think about like where this can actually be feasible is if the thing that you're making a prediction for is useful to the user Downstream within the same product experience that you're building not is this useful for them to copy and use in a different app but is it useful for them to use within my app so one example of this is in product called great scope which Sergey built there is a model that
+
+97
+00:59:16,020 --> 00:59:59,700
+when students submit their exams it tries to match the handwritten name on the exam with the name of the student in the student registry now if the model doesn't really know who that student is if it's low confidence or if it gets the prediction wrong then the instructor can go in and re-categorize that to be the correct name that's really useful to them because they need to have the exam categorized to the correct student anyway but it's also very direct supervisory signal for the model so it's Best of Both Worlds whenever you're thinking about building explicit feedback into your products it's always worth keeping in mind that you know users are not always as altruistic as we might hope that they would be and so you
+
+98
+00:59:57,720 --> 01:00:35,460
+should also think about like how is it going to be worthwhile for users to actually spend the time to give us feedback on this the sort of most foolproof way of doing this is as we described before to gather feedback as part of an existing user workflow but if that's not possible if the goal of users providing the feedback is to make the model better then one way you can encourage them to do that is to make it explicit how the feedback will make their user experience better and generally speaking like the more explicit you can be here and the shorter the time interval is between when they give the feedback and when they actually see the product get better the more of a sort of positive feedback loops this
+
+99
+01:00:33,900 --> 01:01:14,040
+creates for that the more likely is that they're actually going to do it a good example here is to acknowledge user feedback and adjust automatically so so if your user provided you feedback saying hey I really like running up hills then sort of good response to that feedback might be great here's another hell that you can run up in 1.2 kilometers they see the results of that feedback immediately and it's very clear how it's being used to make the product experience better less good is the example to the right of that where the response to the feedback just says thank you for your feedback because I as a user when I give that feedback there's no way for me to know whether that feedback is actually making the product
+
+100
+01:01:12,000 --> 01:01:50,520
+experience better so it discourages me from getting more feedback in the future the main Takeaway on product design for machine learning is that great ml powered products and product experiences are not just you know take an existing product that works well in both and then on top of it they're actually designed from scratch with machine learning and the particularities of machine learning in mind and some reasons for that include that unlike what your users might think machine learning is not superhuman intelligence encoded in Silicon and so your product experience needs to help users understand that in the context of the particular problem that you are solving for them it also needs to help them interact safely with
+
+101
+01:01:48,420 --> 01:02:25,079
+this model that has failure modes via human in the loop and guard rails around the experience with interacting with that model and finally great ml products are powered by great feedback loops right because the perfect version of the model doesn't exist and certainly it doesn't exist in the first version of the model that you deployed and so one important thing to think about when you're designing your product is how can you help your users make the product experience better by collecting the right feedback from them this is a pretty young and underexplored topic and so here's a bunch of resources that I would recommend checking out if you want to learn more about this many of the examples that we used in the previous
+
+102
+01:02:23,579 --> 01:02:59,880
+slides are pulled from these resources and in particular the resource from Google in the top bullet point is really good if you want to understand the basics of this field so to wrap up this lecture we talk about a bunch of different topics related to how to build machine learning products as a team and the first is machine learning roles and the sort of takeaway here is that there's many different skills involved in production machine learning machine production ml is inherently interdisciplinary so there's an opportunity for lots of different skill sets to help contribute when you're building machine learning teams since there's a scarcity of talent especially talent that is good at both software engineering and machine learning it's
+
+103
+01:02:58,140 --> 01:03:32,339
+important to be specific about what you really need for these roles but paradoxically as an outsider it can be difficult to break into the field and the sort of main recommendation that we had for how to get around that is by using projects to build awareness of your thinking about machine learning the next thing that we talk about is how machine learning teams fit into the broader organization we covered a bunch of different archetypes for how that can work and we looked at how machine learning teams are becoming more Standalone and more interdisciplinary in how they function next we talk about managing ml teams and managing ml products managing ml teams is hard and there's no Silver Bullet here but one
+
+104
+01:03:30,900 --> 01:04:01,400
+sort of concrete thing that we looked at is probabilistic Project planning as a way to help alleviate some of the challenges of understanding how long it's going to take to finish machine learning projects and then finally we talk about product design in the context of of machine learning and the main takeaway there is that today's machine learning systems are not AGI right they're Limited in many ways and so it's important to make sure that your users understand that and that you can use the interaction that you build with your users to help mitigate those limitations so that's all for today and we'll see you next week
+

documents/lecture-09.md

@@ -0,0 +1,825 @@
+---
+description: Building ML for good while building good ML
+---
+
+# Lecture 9: Ethics
+
+<div align="center">
+<iframe width="720" height="405" src="https://www.youtube-nocookie.com/embed/7FQpbYTqjAA?list=PL1T8fO7ArWleMMI8KPJ_5D5XSlovTW_Ur" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+Lecture by [Charles Frye](https://twitter.com/charles_irl).
+Notes by [James Le](https://twitter.com/le_james94) and [Vishnu Rachakonda](https://www.linkedin.com/in/vrachakonda/).<br />
+Published October 03, 2022.
+[Download slides](https://fsdl.me/2022-lecture-09-slides).
+
+In this final lecture of FSDL 2022, we'll talk about ethics. After going
+through the context of what we mean by ethics, we'll go through three
+different areas where ethics come up:
+
+1.  **Tech Ethics**: ethics that anybody who works in the tech industry
+broadly needs to think about.
+
+2.  **ML Ethics**: what ethics has specifically meant for the ML
+industry.
+
+3.  **AI Ethics**: what ethics might mean in the future where true AGI
+exists.
+
+## 1 - Overview and Context
+
+All ethics lectures are wrong, but some are useful. They are more useful
+if we admit and state what our assumptions or biases are. We'll also
+talk about three general themes that come up often when ethical concerns
+are raised in tech/ML: alignment, trade-offs, and humility.
+
+![](./media/image17.png)
+
+In this lecture, we'll approach ethics on the basis of **concrete
+cases** - specific instances where people have raised concerns. We'll
+talk about **cases where people have taken actions that have led to
+claims and counter-claims of ethical or unethical behavior** - such as
+the use of automated weapons, the use of ML systems to make decisions
+like sentencing and bail, and the use of ML algorithms to generate art.
+In each case when criticism has been raised, part of that criticism has
+been that the technology is unethical.
+
+Approaching ethics in this way allows us to answer the question of "What
+is ethics?" by way of Ludwig Wittgenstein's quote: "*The meaning of a
+word is its use in the language*." We'll focus on times when people have
+used the word "ethics" to describe what they like or dislike about a
+specific technology.
+
+If you want to try it out for yourself, you should check out the game
+"[Something Something Soup
+Something](https://soup.gua-le-ni.com/)." In this browser
+game, you are presented with a bunch of dishes and have to decide
+whether they are soup or not soup, as well as whether they can be served
+to somebody who ordered soup. By playing a game like this, you'll
+discover (1) how difficult it is to come up with a concrete definition
+of soup and (2) how poorly your working definition of soup fits with any
+given soup theory.
+
+Because of this case-based approach, we won't be talking about ethical
+schools or "trolley" problems. Rather than considering [these
+hypothetical
+scenarios](https://www.currentaffairs.org/2017/11/the-trolley-problem-will-tell-you-nothing-useful-about-morality),
+we'll talk about concrete and specific examples from the past decade of
+work in our field and adjacent fields.
+
+![](./media/image19.png)
+
+If you want another point of view that emphasizes the trolley problems,
+you should check out [Sergey's lecture from the last edition of the
+course from
+2021](https://fullstackdeeplearning.com/spring2021/lecture-9/).
+It presented similar ideas from a different perspective and came to the
+same conclusion and some different conclusions.
+
+A useful theme from that lecture that we should all have in mind when we
+ponder ethical dilemmas is "What Is Water?" - which came up from [a
+famous commencement speech by David Foster
+Wallace](https://www.youtube.com/watch?v=PhhC_N6Bm_s). If
+we aren't thoughtful and paying attention, things that are very
+important can become background, assumptions, and invisible to us.
+
+The approach of **relying on prominent cases risks replicating social
+biases**. Some ethical claims are amplified and travel more because
+people (who are involved) have more resources and are better connected.
+Using these forms of case-based reasoning (where you explain your
+beliefs in concrete detail) can **hide the principles that are actually
+in operation**, making them disappear like water.
+
+But in the end, **so much of ethics is deeply personal** that we can't
+expect to have a perfect approach. We can just do the best we can and
+hopefully become better every day.
+
+## 2 - Themes
+
+We'll see three themes repeatedly coming up throughout this lecture:
+
+1.  **Alignment**: a conflict between what we want and what we get.
+
+2.  **Trade-Offs**: a conflict between what we want and what others
+want.
+
+3.  **Humility**: a response when we don't know what we want or how to
+get it.
+
+### Alignment
+
+The problem of **alignment** (where what we want and what we get differ)
+come up over and over again. A primary driver of this is called the
+**proxy problem** - in which we often optimize or maximize some proxies
+for the thing that we really care about. If the alignment (or loosely
+the correlation between that proxy and the thing we care about) is poor
+enough, then by trying to maximize that proxy, we can end up hurting the
+thing we originally cared about.
+
+![](./media/image16.png)
+
+There was [a recent
+paper](https://arxiv.org/abs/2102.03896) that did a
+mathematical analysis of this idea. You can see these kinds of proxy
+problems everywhere once you look for them.
+
+-   On the top right, we have a train and validation loss chart from one
+of the training runs for the FSDL text recognizer. The thing we
+can optimize is the training loss. That's what we can use to
+calculate gradients and improve the parameters of our network. But
+the thing we really care about is the performance of the network
+on data points that it has not seen (like the validation set, the
+test set, or data in production). If we optimize our training loss
+too much, we can actually cause our validation loss to go up.
+
+-   Similarly, there was [an interesting
+paper](https://openreview.net/forum?id=qrGKGZZvH0)
+suggesting that increasing your accuracy on classification tasks
+can actually result in a decrease in the utility of your
+embeddings in downstream tasks.
+
+-   You can find these proxy problems outside of ML as well. [This
+thread](https://skeptics.stackexchange.com/questions/22375/did-a-soviet-nail-factory-produce-useless-nails-to-improve-metrics)
+reveals an example where a factory that was making chemical
+machines (rather than creating a machine that was cheaper and
+better) chose not to adopt producing that machine because their
+output was measured in weight. So the thing that the planners
+actually cared about, economic efficiency and output, was not
+optimized because it was too difficult to measure.
+
+One reason why these kinds of proxy problems arise so frequently is due
+to issues of information. **The information that we are able to measure
+is not the information that we want**. At a higher level, we often don't
+know what it is that we truly needed. We may want the validation loss,
+but what we need is the loss in production or really the value our users
+will derive from this model.
+
+### Trade-Offs
+
+Even when we know what we want or what we need, we are likely to run
+into the second problem - **the tradeoff between stakeholders**. It is
+sometimes said that the need to negotiate tradeoffs is one of the
+reasons why engineers do not like thinking about some of these problems
+around ethics. That's not quite right because we do accept tradeoffs as
+a key component of engineering.
+
+-   In [this O'Reilly book on the fundamentals of software
+architecture](https://www.oreilly.com/library/view/fundamentals-of-software/9781492043447/),
+the first thing they state at the beginning is that **everything
+in software architecture is a tradeoff.**
+
+-   [This satirical O'Reilly
+book](https://www.reddit.com/r/orlybooks/comments/50meb5/it_depends/)
+says that every programming question has the answer: "It depends."
+
+![](./media/image20.png)
+
+
+The famous chart above compares the different convolutional networks on
+the basis of their accuracy and the number of operations to run them.
+Thinking about these tradeoffs between speed and correctness is exactly
+the thing we have to do all the time in our job as engineers.
+
+We can select the **Pareto Front** for the metrics we care about. A way
+to remember what a Pareto front is [this definition of a data scientist
+from Josh
+Wills](https://twitter.com/josh_wills/status/198093512149958656?lang=en):
+"Person who is better at statistics than any software engineer and
+better at software engineering than any statistician." The Pareto Front
+in the chart above includes the models that are more accurate than those
+with fewer FLOPs and use fewer FLOPs than those that are more accurate.
+
+A reason why engineers may dislike thinking about these problems is that
+**it's hard to identify and quantify these tradeoffs**. These are indeed
+proxy problems. Even further, once measured, where on that front do we
+fall? As engineers, we may develop expertise in knowing whether we want
+high accuracy or low latency, but we are not as comfortable deciding how
+many current orphans we want to trade for what amount of future health.
+This raises questions both in terms of measurement and decision-making
+that are outside of our expertise.
+
+### Humility
+
+The appropriate response is **humility** because most engineers do not
+explicitly train in these skills. Many engineers and managers in tech,
+in fact, constitutionally prefer optimizing single metrics that are not
+proxies. Therefore, when encountering a different kind of problem, it's
+important to bring a humble mindset, ask for help from experts, and
+recognize that the help you get might not be immediately obvious to what
+you are used to.
+
+Additionally, when intervening due to an ethical concern, it's important
+to remember this humility. It's easy to think that when you are on the
+good side, this humility is not necessary. But even trying to be helpful
+is a delicate and dangerous undertaking. We want to make sure that as we
+resolve ethical concerns, we come up with solutions that are not just
+parts of the problem.
+
+### User Orientation Undergirds Each Theme
+
+We can resolve all of these via **user orientation**.
+
+1.  By getting feedback from users, we maintain **alignment** between
+our system and our users.
+
+2.  When making **tradeoffs**, we should resolve them in consultation
+with users.
+
+3.  **Humility** means we actually listen to our users because we
+recognize we don't have the answers to all the questions.
+
+## 3 - Tech Ethics
+
+The tech industry can't afford to ignore ethics as public trust in tech
+declines. We need to learn from other nearby industries that have done a
+better job on professional ethics. We'll also touch on some contemporary
+topics.
+
+### Tech Industry's Ethical Crisis
+
+Throughout the past decade, the tech industry has been plagued by
+scandal - whether that's how tech companies interface with national
+governments at the largest scale or how tech systems are being used or
+manipulated by people who create disinformation or fake social media
+accounts that hack the YouTube recommendation system.
+
+As a result, distrust in tech companies has risen markedly in the last
+ten years. [This Public Affairs Pulse
+survey](https://pac.org/public-affairs-pulse-survey-2021)
+shows that in 2013, the tech industry was one of the industries with
+less trustworthiness on average. In 2021, it has rubbed elbows with
+famously more distrusted industries such as energy and pharmaceuticals.
+
+![](./media/image10.png)
+
+Politicians care quite a bit about public opinion polls. In the last few
+years, the fraction of people who believe that large tech companies
+should be more regulated has gone up a substantial amount. [Comparing
+it to 10 years ago, it's astronomically
+higher](https://news.gallup.com/poll/329666/views-big-tech-worsen-public-wants-regulation.aspx).
+So there will be a substantial impact on the tech industry due to this
+loss of public trust.
+
+We can learn from nearby fields: from the culture of professional ethics
+in engineering in Canada (by wearing [the Iron
+Ring](https://en.wikipedia.org/wiki/Iron_Ring)) to ethical
+standards for human subjects research ([Nuremberg
+Code](https://en.wikipedia.org/wiki/Nuremberg_Code), [1973
+National Research
+Act](https://en.wikipedia.org/wiki/National_Research_Act)).
+We are at the point where we need a professional code of ethics for
+software. Hopefully, many codes of ethics developed in different
+communities can compete with each other and merge into something that
+most of us can agree on. That can be incorporated into our education for
+new members of our field.
+
+Let's talk about two particular ethical concerns that arise in tech in
+general: carbon emissions and dark/user-hostile design patterns.
+
+### Tracking Carbon Emissions
+
+Because carbon emissions scale with cost, you only need to worry about
+them when the costs of what you are working on are very large. Then you
+won't be alone in making these decisions and can move a bit more
+deliberately to make these choices more thoughtfully.
+
+Anthropogenic climate change from carbon emissions raises ethical
+concerns - tradeoffs between the present and future generations. The
+other view is that this is an issue that arises from a classic alignment
+problem: many organizations are trying to maximize their profit, which
+is based on prices for goods that don't include externalities (such as
+environmental damage caused by carbon emissions, leading to increased
+temperatures and lactic change).
+
+![](./media/image8.png)
+
+The primary dimension along which we have to worry about carbon
+emissions is in **compute jobs that require power**. That power can
+result in carbon emissions. [This
+paper](https://aclanthology.org/P19-1355/) walks through
+how much carbon dioxide was emitted using typical US-based cloud
+infrastructure.
+
+-   The top headline shows that training a large Transformer model with
+neural architecture search produces as much carbon dioxide as five
+cars create during their lifetimes.
+
+-   It's important to remember that power is not free. On US-based cloud
+infrastructure, \$10 of cloud spent is roughly equal to \$1 of air
+travel costs. That's on the basis of something like the numbers
+and the chart indicating air travel across the US from New York to
+San Francisco.
+
+-   Just changing cloud regions can actually reduce your emissions quite
+a bit. There's [a factor of
+50x](https://www.youtube.com/watch?v=ftWlj4FBHTg)
+from regions with the most to least carbon-intensive power
+generation.
+
+The interest in this problem has led to new tools.
+[Codecarbon.io](https://codecarbon.io/) allows you to
+track power consumption and reduce carbon emissions from your computing.
+[ML CO2 Impact](https://mlco2.github.io/impact/) is
+oriented directly towards machine learning.
+
+### Deceptive Design and Dark Patterns
+
+The other ethical concern in tech is **deceptive design**. An
+unfortunate amount of deception is tolerated in some areas of software.
+As seen below, on the left is a nearly complete history of the way
+Google displays ads in its search engine results. It started off very
+clearly colored and separated out with bright colors from the rest of
+the results. Then about ten years ago, that colored background was
+removed and replaced with a tiny little colored snippet that said "Ad."
+Now, as of 2020, that small bit is no longer even colored. It is just
+bolded. This makes it difficult for users to know which content is being
+served to them because somebody paid for it (versus content served up
+organically).
+
+![](./media/image15.png)
+
+A number of **dark patterns** of deceptive design have emerged over the
+last ten years. You can read about them on the website called
+[deceptive.design](https://www.deceptive.design/). There's
+also a Twitter account called
+[\@darkpatterns](https://twitter.com/darkpatterns) that
+shares examples found in the wild.
+
+A practice in the tech industry that's on a very shaky ethical /legal
+ground is **growth hacking**. This entails a set of techniques for
+achieving rapid growth in user base or revenue for a product and has all
+the connotations you might expect from the name - with examples
+including LinkedIn and Hotmail.
+
+![](./media/image14.png)
+
+**ML can actually make this problem worse if we optimize short-term
+metrics**. These growth hacks and deceptive designs can often drive user
+and revenue growth in the short term but worsen user experience and draw
+down on goodwill towards the brand in a way that can erode the long-term
+value of customers. When we incorporate ML into the design of our
+products with A/B testing, we have to watch out to make sure that the
+metrics that we are optimizing do not encourage this kind of deception.
+
+These arise inside another alignment problem. One broadly-accepted
+justification for the private ownership of the means of production is
+that private enterprise delivers broad social value aligned by price
+signals and market focus. But these private enterprises optimize metrics
+that are, at best, a proxy for social value. There's the possibility of
+an alignment problem where **companies pursuing and maximizing their
+market capitalization can lead to net negative production of value**. If
+you spend time at the intersection of funding, leadership, and
+technology, you will encounter it.
+
+![](./media/image12.png)
+
+
+In the short term, you can **push for longer-term thinking within your
+organization** to allow for better alignment between metrics and goals
+and between goals and utility. You can also learn to recognize
+user-hostile designs and **advocate for user-centered design instead**.
+
+To wrap up this section on tech ethics:
+
+1.  The tech industry should learn from other disciplines if it wants to
+avoid a trust crisis.
+
+2.  We can start by educating ourselves about common deceptive or
+user-hostile practices in our industry.
+
+## 4 - ML Ethics
+
+The ethical concerns raised about ML have gone beyond just the ethical
+questions about other kinds of technology. We'll talk about common
+ethical questions in ML and lessons learned from Medical ML.
+
+### Why Not Just Tech Ethics?
+
+ML touches human lives more intimately than other technologies. Many ML
+methods, especially deep neural networks, make human-legible data into
+computer-legible data. Humans are more sensitive to errors and have more
+opinions about visual and text data than they do about the type of data
+manipulated by computers. As a result, there are more stakeholders with
+more concerns that need to be traded off in ML applications.
+
+Broadly speaking, ML involves being wrong pretty much all the time. Our
+models are statistical and include "randomness." Randomness is almost
+always an admission of ignorance. As we admit a certain degree of
+ignorance in our models, our models will be wrong and misunderstand
+situations that they are put into. It can be upsetting and even harmful
+to be misunderstood by our models.
+
+Against this backlash of greater interest or higher stakes, a number of
+common types of ethical concerns have coalesced in the last couple of
+years. There are somewhat established camps of answers to these
+questions, so you should at least know where you stand on the four core
+questions:
+
+1.  Is the model "fair"?
+
+2.  Is the system accountable?
+
+3.  Who owns the data?
+
+4.  Should the system be built at all?
+
+### Common Ethical Questions in ML
+
+#### Is The Model "Fair"?
+
+The classic case on this comes from criminal justice with [the COMPAS
+system](https://en.wikipedia.org/wiki/COMPAS_(software))
+for predicting whether a defendant will be arrested again before trial.
+If they are arrested again, that suggests they committed a crime during
+that time. This assesses a certain degree of risk for additional harm
+while the justice system decides what to do about a previous arrest and
+potential crime.
+
+The operationalization here was a 10-point re-arrest probability based
+on past data about this person, and they set a goal from the very
+beginning to be less biased than human judges. They operationalize that
+by calibrating these arrest probabilities across subgroups. Racial bias
+is a primary concern in the US criminal justice system, so they took
+care to make sure that these probabilities of re-arrest were calibrated
+for all racial groups.
+
+![](./media/image2.png)
+
+
+The system was deployed and used all around the US. It's proprietary and
+difficult to analyze. But using the Freedom of Information Act and
+coalescing together a bunch of records, [people at ProPublica were able
+to run their own analysis of this
+algorithm](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing).
+They determined that the model was not more or less wrong for one racial
+group or another. It tended to have more false positives for Black
+defendants and more false negatives for White defendants. So despite the
+creators of COMPAS taking into account bias from the beginning, they
+still ended up with an algorithm with this undesirable property of being
+more likely to falsely accuse Black defendants than White defendants.
+
+It turned out that some quick algebra revealed that some form of
+race-based bias is inevitable in this setting, as indicated [in this
+paper](https://arxiv.org/abs/1610.07524). There are a large
+number of fairness definitions that are mutually incompatible. [This
+tutorial by Arvind
+Narayanan](https://www.youtube.com/watch?v=jIXIuYdnyyk&ab_channel=ArvindNarayanan)
+is an excellent one to display them.
+
+It is noteworthy that **the impact of "unfairness" is not fixed**. The
+story is often presented as "no matter what, the journalists would have
+found something to complain about." But note that equalizing false
+positive rates and positive predictive value across groups would lead to
+a higher false negative rate for Black defendants relative to White
+defendants. In the context of American politics, that's not going to
+lead to complaints from the same people.
+
+![](./media/image6.png)
+
+
+This is the story about the necessity of confronting the tradeoffs that
+will inevitably come up. Researchers at Google made [a nice little
+tool](https://research.google.com/bigpicture/attacking-discrimination-in-ml/)
+where you can think through and make these tradeoffs for yourself. It's
+helpful for building intuition on these fairness metrics and what it
+means to pick one over the other.
+
+Events in this controversy kicked off a flurry of research on fairness.
+[The Fairness, Accountability, and Transparency
+conference](https://facctconference.org/) has been held for
+several years. There has been a ton of work on both **algorithmic-level
+approaches** on measuring and incorporating fairness metrics into
+training and **qualitative work** on designing systems that are more
+transparent and accountable.
+
+In the case of COMPAS, **re-arrest is not the same as recidivism**.
+Being rearrested requires that a police officer believes you committed a
+crime. Police officers are subject to their own biases and patterns of
+policing, which result in a far higher fraction of crimes being caught
+for some groups than for others. Our real goal, in terms of fairness and
+criminal justice, might be around reducing those kinds of unfair impacts
+and using past rearrest data that have these issues.
+
+#### Representation Matters for Model Fairness
+
+![](./media/image18.png)
+
+Unfortunately, it is easy to make ML-powered tech that fails for
+minoritized groups. For example, off-the-shelf computer vision tools
+often fail on darker sins (as illustrated in [this talk by Joy
+Buolamwini](https://www.ted.com/talks/joy_buolamwini_how_i_m_fighting_bias_in_algorithms)).
+This is not a new issue in technology, just a more salient one with ML.
+
+There has been a good amount of progress on this in the last five years.
+An example is [Google's Model
+Cards](https://modelcards.withgoogle.com/about) which show
+how well a model will perform on human subgroups of interest.
+HuggingFace has good integrations for creating these kinds of model
+cards.
+
+When you invite people for talks or hire people to join your
+organizations, you should work to reduce the bias of that discovery
+process by diversifying your network. Some good resources include
+[Black in AI](https://blackinai.github.io/#/), [Diversify
+Tech Job Board](https://www.diversifytech.co/job-board/),
+[Women in Data Science](https://www.widsconference.org/),
+and the [You Belong in AI
+podcast](https://anchor.fm/ucla-acm-ai). You can make
+professional connections via them to improve the representation of
+minoritized groups in the engineering, design, and product management
+process.
+
+#### Is The System Accountable?
+
+At a broader level than fairness, we should expect "accountability" from
+ML systems. Some societies and states, including the EU, consider "[the
+right to an explanation](https://arxiv.org/abs/1606.08813)"
+in the face of important judgments to be a part of human rights.
+
+In the GDPR act, there is [a section that enshrines
+accountability](https://www.consumerfinance.gov/rules-policy/regulations/1002/interp-9/#9-b-1-Interp-1).
+This isn't quite a totally new requirement; credit denials in the US
+have been required to be explained since 1974. People have a right to
+know what and why into making decisions for them!
+
+If you want to impose this "accountability" on a deep neural network and
+understand its selections, there are a number of methods that use the
+input-output gradient to explain the model. You can see a list of
+several methods in order of increasing performance below (from [this
+paper](https://arxiv.org/abs/1810.03292)). These approaches
+don't quite have strong theoretical underpinnings or a holistic
+explanation, and are not that robust as a result. A lot of these methods
+act primarily as edge detectors. The paper shows how even randomizing
+layers in a model does not materially change the interpretability output
+of GradCAM methods.
+
+![](./media/image11.png)
+
+
+As a result, introspecting DNNs effectively requires reverse engineering
+the system to really understand what is going on, largely thanks to
+efforts like [Distil](https://distil.pub/) and
+[Transfomer Circuits](https://transformer-circuits.pub/).
+
+Due to these technical challenges, machine learning systems are prone to
+unaccountability that impacts most those least able to understand and
+influence their outputs. Books such as [Automating
+Inequality](https://www.amazon.com/Automating-Inequality-High-Tech-Profile-Police/dp/1250074312)
+describe the impacts of these systems. In such a context, you should
+seek to question the purpose of model, involve those impacted by the
+decisions (either through direct human inputs or through other means),
+and ensure that equal attention is paid to benefits and harms of
+automation.
+
+#### Who Owns The Data?
+
+**Humans justifiably feel ownership of the data they creat**e, which is
+subsequently used to train machine learning models. Large datasets used
+to train models like GPT-3 are created by mining this data without the
+explicit involvement of those who create the data. Many people are not
+aware that this is both possible and legal. As technology has changed,
+what can be done with data has changed.
+
+[You can even verify if your data has been used to train models
+on](https://haveibeentrained.com/). Some of these images
+are potentially [obtained
+illegally](https://arstechnica.com/information-technology/2022/09/artist-finds-private-medical-record-photos-in-popular-ai-training-data-set/),
+as a result of sensitive data being posted openly without the recorded
+consent of the originator.
+
+![](./media/image5.png)
+
+
+Each of these controversies around image generation and illegal data has
+opened up a new frontier in **data governance**. Focus will be placed on
+ensuring new ML systems are sensitive to personal and professional
+concerns of those who generate the data ML systems are trained on.
+[Emad Mostaque](https://uk.linkedin.com/in/emostaque), CEO
+of [Stability AI](https://stability.ai/), has gone so far
+as to offer future opt out systems from systems similar to Stable
+Diffusion.
+
+Here are some practical tips: [Dataset
+cards](https://huggingface.co/docs/datasets/dataset_card)
+can be helpful in providing documentation in a similar fashion to model
+cards. There are also ethics lists, like [the deon ethic
+checklist](https://deon.drivendata.org/examples/) that
+helps design proper systems. Deon also has a helpful list of failure
+cases.
+
+#### Should This Be Built At All?
+
+The undercurrent behind this conversation is the justifiable question of
+whether some of these systems should be built at all, let alone in an
+ethical way.
+
+**ML-powered weaponry** is the canonical example here, which is already
+in use. The definition of these systems are blurry, as both systems old
+and new have had various autonomous capacities. This is difficult to get
+a sense of due to the secrecy associated with weapon systems.
+
+Some have argued that "autonomous weapons" have existed for hundreds of
+years, but even this does not mean that they are ethical. Mines are good
+examples of these systems. Movements like t[he Campaign Against Killer
+Robots](https://www.stopkillerrobots.org/about-us/) are
+trying to prevent the cycle we entered with mines - where we invented
+them, when we realized the incredible harm, and why we are trying to ban
+them. Why invent these at all?
+
+Let's wrap up this entire section with some closing questions that you
+should always have a thoughtful answer to as you build a machine
+learning system.
+
+1.  **Is the model "fair"?** Fairness is possible, but requires
+trade-offs.
+
+2.  **Is the system accountable?** Accountability is easier than
+interpretability.
+
+3.  **Who owns the data?** Answer this upfront. Changes are on the way.
+
+4.  **Should the system be built at all?** Repeatedly ask this and use
+it to narrow scope.
+
+### What Can We Learn from Medical ML
+
+*Note: The FSDL team would like to thank [Dr. Amir Ashraf
+Ganjouei](https://scholar.google.com/citations?user=pwLadpcAAAAJ)
+for his feedback on this section.*
+
+Interestingly, medicine can teach us a lot about how to apply machine
+learning in a responsible way. Fundamentally, this has led to a mismatch
+between how medicine works and how machine learning systems are built
+today.
+
+Let's start with a startling fact: **the machine learning response to
+COVID-19 was an abject failure**. In contrast, the biomedical response
+was a major triumph. For example, the vaccines were developed with
+tremendous speed and precision.
+
+![](./media/image9.png)
+
+Machine learning did not acquit itself well with the COVID-19 problem.
+Two reviews ([Roberts et al.,
+2021](https://www.nature.com/articles/s42256-021-00307-0)
+and [Wynants et al.,
+2020-2022](https://www.bmj.com/content/369/bmj.m1328))
+found that nearly all machine learning models were insufficiently
+documented, had little to no external validation, and did not follow
+model development best practices. A full 25% of the papers used a
+dataset incorrect for the task, which simply highlighted the difference
+between children and adults, not pneumonia and COVID.
+
+Medicine has a strong culture of ethics that professionals are
+integrated into from the point they start training. Medical
+professionals take the Hippocratic oath of practicing two things: either
+help or do not harm the patient. In contrast, the foremost belief
+associated with software development tends to be the infamous "Move fast
+and break things." While this approach works for harmless software like
+web apps, **it has serious implications for medicine and other more
+critical sectors**. Consider the example of a retinal implant that was
+simply deprecated by developers and left hundreds without sight [in
+this Statnews
+article](https://www.statnews.com/2022/08/10/implant-recipients-shouldnt-be-left-in-the-dark-when-device-company-moves-on/).
+
+![](./media/image4.png)
+
+**Researchers are drawing inspiration from medicine to develop similar
+standards for ML**.
+
+-   For example, clinical trial standards have been extended to ML.
+These standards were developed through extensive surveys,
+conferences, and consensus building (detailed in
+[these](https://www.nature.com/articles/s41591-020-1037-7)
+[papers](https://www.nature.com/articles/s41591-020-1034-x)).
+
+-   Progress is being made in understanding how this problem presents.
+[A recent
+study](https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2796833)
+found that while clinical activities are generally performed at a
+high compliance level, statistical and data issues tend to suffer
+low compliance.
+
+-   New approaches are developing [entire "auditing"
+procedures](https://www.thelancet.com/journals/landig/article/PIIS2589-7500(22)00003-6/fulltext)
+that exquisitely identify the activities required to effectively
+develop models.
+
+Like medicine, machine learning is intimately intertwined with people's
+lives. The most important question to ask is "Should this system be
+built at all?". Always ask yourselves this and understand the
+implications!
+
+## 5 - AI Ethics
+
+AI ethics are a frontier in both the technology and the ethics worlds.
+False claims and hype are the most pressing concerns, but other risks
+could present themselves soon.
+
+### AI Snake Oils
+
+**False claims outpace the performance of AI**. This poses a serious
+threat to adoption and satisfaction with AI systems long term.
+
+-   For example, if you call something "AutoPilot", people might truly
+assume it is fully autonomous, as happened in the below case of a
+Tesla user. This goes back to our discussion about how AI systems
+are more like funky dogs than truly human intelligent systems.
+
+-   Another example of this is [IBM's Watson
+system](https://www.ibm.com/ibm/history/ibm100/us/en/icons/watson/),
+which went from tackling the future of healthcare to being sold
+off for parts.
+
+![](./media/image13.png)
+
+These false claims tend to be amplified in the media. But this isn't
+confined to traditional media. Even Geoff Hinton, a godfather of modern
+machine learning, has been [a little too aggressive in his forecasts
+for AI
+performance](https://www.youtube.com/watch?v=2HMPRXstSvQ)!
+
+You can call this **"AI Snake Oil"** as Arvind Narayanan does in [his
+Substack](https://aisnakeoil.substack.com/) and
+[talk](https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf).
+
+Let's separate out where true progress has been made versus where
+progress is likely to be overstated. On some level, AI perception has
+seen tremendous progress, AI judgment has seen moderate progress, and AI
+prediction of social outcomes has seen not nearly as much progress.
+
+![](./media/image3.png)
+
+### Frontiers: AI Rights and X-Risk
+
+There's obvious rationale that should artificial sentient beings exist,
+tremendous ethical implications would be raised. Few people believe that
+we are truly on the precipice of sentient beings, but there is
+disagreement on how close we are.
+
+![](./media/image1.png)
+
+There's a different set of concerns around how to regard self-improving
+intelligent beings, for which there is already evidence. Large Language
+Models have been show to be able to improve themselves in a range of
+studies
+([here](https://openreview.net/forum?id=92gvk82DE-) and
+[here](https://arxiv.org/abs/2207.14502v1)).
+
+Failing to pursue this technology would lead to [a huge opportunity
+cost](https://nickbostrom.com/astronomical/waste) (as
+argued by Nick Bostrom)! There truly is a great opportunity in having
+such systems help us sold major problems and lead better lives. The key
+though, is that such technology should be developed in the **safest way
+possible,** not the fastest way.
+
+[The paperclip
+problem](https://www.lesswrong.com/tag/paperclip-maximizer)
+shows how the potential for misalignment between AI systems and humans
+could dramatically reduce human utility and even compromise our
+interests. Imagine a system designed to manufacture paperclips... could
+actually develop the intelligence to alter elements of society to favor
+paper clips?! This thought experiments illustrates how self-learning
+systems could truly change our world for the worse in a misaligned way.
+
+These ideas around existential risk are most associated with [the
+Effective Altruism community](https://www.eaglobal.org/).
+Check out resources like [Giving What We
+Can](https://www.givingwhatwecan.org/donate/organizations)
+and [80,000 Hours](https://80000hours.org/) if you're
+interested!
+
+## 6 - What Is To Be Done?
+
+This course can't end on a dour a note as existential risk. What can be
+done to mitigate these consequences and participate in developing truly
+ethical AI?
+
+1.  The first step is **to educate yourself on the topic**. There are
+many great books that give lengthy, useful treatment to this
+topic. We recommend [Automating
+Inequality](https://www.amazon.com/Automating-Inequality-High-Tech-Profile-Police/dp/1250074312),
+[Weapons of Math
+Destruction](https://www.amazon.com/Weapons-Math-Destruction-Increases-Inequality/dp/0553418815),
+and [The Alignment
+Problem](https://www.amazon.com/Alignment-Problem-Machine-Learning-Values/dp/0393635821).
+
+2.  After reading this, **consider how to prioritize your actions**.
+What do you want to impact? When do you want to do that? Place
+them in this two-by-two to get a sense of where their importance
+is.
+
+![](./media/image7.png)
+
+**Ethics cannot be purely negative**. We do good, and we want to
+*prevent* bad! Focus on the good you can do and be mindful of the harm
+you can prevent.
+
+Leading organizations like
+[DeepMind](https://www.deepmind.com/about/operating-principles)
+and [OpenAI](https://openai.com/charter/) are leading from
+the front. Fundamentally, building ML well aligns with building ML for
+good. All the leading organizations emphasize effective *and*
+responsible best practices for building ML powered practices. Keep all
+this in mind as you make the world a better place with your AI-powered
+products!

documents/lecture-09.srt

@@ -0,0 +1,488 @@
+1
+00:00:00,539 --> 00:00:45,660
+hey everyone welcome to the ninth and final lecture of full stack deep learning 2022. today we'll be talking about ethics after going through a little bit of context of what it is that we mean by ethics what I mean by ethics when I talk about it we'll go through three different areas where ethics comes up both Broad tech ethics ethics that anybody who works in the tech industry broadly needs to think about and care about what ethics has meant specifically for the machine learning industry what's happened in the last couple of years as ethical concerns have come to the Forefront and then finally what ethics might mean in a future where true artificial general intelligence exists so first let's do a little bit of
+
+2
+00:00:42,899 --> 00:01:33,000
+context setting even more so than other topics all lectures on ethics are wrong but some of them are useful and they're more useful if we admit and state what our assumptions or biases or approaches are before we dive into the material and then I'll also talk about three kind of General themes that I see coming up again and again when ethical concerns are raised in Tech and in machine learning themes of alignment themes of trade-off and the critical theme of humility so in this lecture I'm going to approach ethics on the basis of concrete cases specific instances where people have raised concerns so we'll talk about cases where people have taken actions that have led to claims and counter claims of ethical or unethical Behavior
+
+3
+00:01:29,040 --> 00:02:13,680
+the use of automated weapons the use of machine learning systems for making decisions like sentencing and bail and the use of machine learning algorithms to generate art in each case one criticism has been raised part of the criticism has been that the technology Awards impact is unethical so approaching ethics in this way allows me to give my favorite answer to the question of what is ethics which is to quote one of my favorite philosophers Ludwig wickenstein and say that the meaning of a word is its use in the language so we'll be focusing on times when people have used the word ethics to describe what they like or dislike about some piece of technology and this approach to definition is an interesting
+
+4
+00:02:11,940 --> 00:02:51,959
+one if you want to try it out for yourself you should check out the game something something soup something which is a browser game at the link in the bottom left of this slide in which you presented with a bunch of dishes and you have to decide whether they are soup or not soup whether they can be served to somebody who ordered soup and by playing a game like this you can discover both how difficult it is to really put your finger on a concrete definition of soup and how poorly maybe your working definition of soup fits with any given soup theory because of this sort of case-based approach we won't be talking about ethical schools and we won't be doing any trolley problems so this article here from current affairs asks
+
+5
+00:02:50,400 --> 00:03:34,800
+you to consider this particular example of a of an ethical dilemma where an asteroid containing all of the universe's top doctors who are working on a cure for all possible illnesses is hurtling towards the planet of Orphans and you can destroy the asteroid and save the orphans but if you do so the hope for a cure for all diseases will be lost forever and the question posed by the authors of this article is is this hypothetical useful at all for Illuminating any moral truths so rather than considering these hypothetical scenarios about trolley cars going down rails and fat men standing on Bridges we'll talk about concrete specific examples from the last 10 years of work in our field and adjacent Fields but
+
+6
+00:03:32,580 --> 00:04:11,040
+this isn't the only way of talking about or thinking about ethics it's the way that I think about it is the way that I prefer to talk about it is not the only one and it might not be the one that works for you so if you want another point of view and one that really emphasizes and loves trolley problems then you should check out sergey's lecture from the last edition of the course from 2021 it's a really delightful talk and presents some similar ideas from a very different perspective coming to some of the same conclusions and some different conclusions a useful theme team from that lecture that I think we should all have in mind when we're pondering ethical dilemmas and the related questions that they bring up is the
+
+7
+00:04:09,060 --> 00:04:52,080
+theme of what is water from last year's lecture so this is a famous little story from a commencement speech by David Foster Wallace where an older fish swing by two younger fish says morning boys how's the water and after he swims away one of the younger fish turns the other and says wait what the hell is water the idea is that if we aren't thoughtful if we aren't paying attention some things that are very important can become background can become assumption and can become invisible and so when I share these slides with Sergey he challenged me to answer this question for myself about how we were approaching ethics this time around and I'll say that this approach of relying on prominent cases risks replicating a lot of social biases
+
+8
+00:04:50,520 --> 00:05:33,060
+some people's ethical claims are Amplified and some fall on unhearing ears some stories travel more because the people involved have more resources and are better connected and using these forms of case-based reasoning where you explain your response or your beliefs in terms of these concrete specifics can end up hiding the principles that are actually in operation maybe you don't even realize that that's how you're making the decision maybe some of the true ethical principles that you're operating under can disappear like water to these fish so don't claim that the approach I'm taking here is perfect but in the end so much of Ethics is deeply personal that we can't expect to have a perfect approach we can just do the best
+
+9
+00:05:30,479 --> 00:06:18,120
+we can and hopefully better every day so we're gonna see three themes repeatedly come up throughout this talk two different forms of conflict that give rise to ethical disputes one when there is conflict between what we want and what we get and another when there is conflict between what we want and what others want and then finally a theme of maybe an appropriate response a response of humility when we don't know what we want or how to get it the problem of alignment where what we want and what we get differ we'll come up over and over again and one of the primary drivers of this is what you might call the proxy problem which is in the end we are often optimizing or maximizing some proxy of the thing that we really care about and
+
+10
+00:06:15,720 --> 00:06:52,080
+if the alignment or Loosely the correlation between that proxy and the thing that we actually care about is poor enough then by trying to maximize that proxy we can end up hurting the thing that we originally cared about there is a nice paper that came out just very recently doing a mathematical analysis of this idea that's actually been around for quite some time excuse you can see these kinds of proxy problems everywhere once you're looking for them on the top right I have a train and validation loss chart from one of the training runs for the full stack deep learning text recognizer the thing that we can actually optimize is the training loss that's what we can use to calculate gradients and improve the
+
+11
+00:06:50,639 --> 00:07:35,400
+parameters of our network but the thing that we really care about is the performance of the network on data points it hasn't seen like the validation set or the test set or data in production if we optimize our training lost too much then we can actually cause our validation loss to go up similarly there was an interesting paper that suggested that increasing your accuracy on classification tasks can actually result in a decrease in the utility of your embeddings in Downstream tasks and you can find these proxy problems outside of machine learning as well there's a famous story involving a Soviet Factory and nails that turned out to be false but in looking up a reference for it I was able to find an actual example where a factory that was
+
+12
+00:07:33,300 --> 00:08:18,120
+making chemical machines rather than creating a machine that was cheaper and better chose not to adopt producing that machine because their output was measured in weight so the thing that that the planners actually cared about economic efficiency and output was not what was being optimized for because it was too difficult to measure and one reason why these kinds of proxy problems arise so frequently is due to issues of information the information that we're able to measure is not the information that we want so the training loss is the information that we have but the information that we want is the validation loss but then at a higher level we often don't even know what it is that we truly need so we may want the
+
+13
+00:08:15,360 --> 00:09:03,660
+validation loss but what we need is the loss in production or really the value our users will derive from this model but even when we do know what it is that we want or what it is that we need we're likely to run into the second kind of problem the problem of a trade-off between stakeholders going back to our hypothetical example with the asteroid of doctors hurtling towards the planet of Orphans what makes this challenging is the need to determine a trade-off between the wants and needs of the people on the asteroid the wants and needs of the orphans on the planet and the wants and needs of future people who cannot be reached for comment and to weigh in on this concern is some sometimes said that this need to
+
+14
+00:09:02,040 --> 00:09:40,920
+negotiate trade offices one of the reasons why Engineers don't like thinking about some of these problems around ethics I don't think that's quite right because we do accept trade-offs as a key component of engineering there's this nice O'Reilly book on the fundamentals of software architecture the first thing that they State at the very beginning is that everything in software architecture is a trade-off and even this satirical oh really book says that every programming question has the answer it depends so we're comfortable negotiating trade-offs take for example this famous chart comparing the different convolutional networks on the basis of their accuracy and the number of operations that it takes to run them
+
+15
+00:09:38,580 --> 00:10:22,080
+thinking about these kinds of trade-offs between speed and correctness is exactly the sort of thing that we have to do all the time in our job as engineers and one part of it that is maybe easier is at least selecting What's called the Pareto front for the metrics that we care about my favorite way of remembering what a Pareto front is is this definition of a data scientist from Josh Wills which is a data scientist who's better at Stats than any software engineer and better at software engineering than any statistician so this Pareto front that I've drawn here is the models that have are more accurate than anybody who takes fewer flops and use fewer flops than anybody who is more accurate so I think rather than fundamentally being about
+
+16
+00:10:20,100 --> 00:11:02,640
+trade-offs one of the reasons why Engineers maybe dislike thinking about these problems is that it's really hard to identify the axes for a chart like the one that I just showed it's very hard to quantify these things and if we do quantify things like the utility or the rights of people involved in a problem we know that those quantifications are far away from what what they truly want to measure there's a proxy problem in fact but even further ones measured where on that front do we fall as Engineers we maybe develop an expertise in knowing whether we want high accuracy or low latency or computational load but we are not as comfortable deciding how many current orphans we want to trade for what amount
+
+17
+00:11:01,260 --> 00:11:45,120
+of future health so this raises questions both in terms of measurement and in terms of decision making that are outside of our expertise so the appropriate response here is humility because we don't explicitly train these skills the way that we do many of the other skills that are critical for our job and many folks engineers and managers in technology seem to kind of deepen their bones prefer optimizing single metrics making a number go up so there's no trade-offs to think about and those metrics are they're not proxies they're the exact same thing that you care about my goal within this company my objective for this quarter my North Star is user growth or lines of code and by God I'll make that go up so when we
+
+18
+00:11:43,380 --> 00:12:28,800
+encounter a different kind of problem it's important to bring a humble mindset a student mindset to the problems to ask for help to look for experts and to recognize that the help that you get and the experts that you find might not be immediately obviously which you want or what you're used to additionally one form of this that we'll see repeatedly is that when attempting to intervene because of an ethical concern it's important to remember this same humility it's easy to think when you are on the good side that this humility is not necessary but even trying to be helpful is a delicate and dangerous undertaking one of my favorite quotes from the systems Bible so we want to make sure as we resolve the ethical concerns that
+
+19
+00:12:26,220 --> 00:13:06,000
+people raise about our technology that we come up with solutions that are not just part of the problem so the way that I resolve all of these is through user orientation by getting feedback from users we maintain alignment between ourselves and the system that we're creating and the users that it's meant to serve and then when it's time to make trade-offs we should resolve them in consultation with users and in my opinion we should tilt the scales in their favor and away from the favor of other stakeholders including within our own organization and then humility is one of the reasons why we actually listen to users at all all because we are humble enough to recognize that we don't have the answers to all of these
+
+20
+00:13:03,720 --> 00:13:46,260
+questions all right with our context and our themes under our belt let's dive into some concrete cases and responses we'll start by considering ethics in the broader world of technology that machine learning fights itself in so the key thing that I want folks to take away from this section is that the tech industry cannot afford to ignore ethics as public trust in Tech declines we need to learn from other nearby industries that have done a better job on professional ethics and then we'll talk about some contemporary topics some that I find particularly interesting and important throughout the past decade the technology industry has been plagued by Scandal whether that's how technology companies interface with national
+
+21
+00:13:43,680 --> 00:14:30,839
+governments at the largest scale over to how technological systems are being used or manipulated by people creating disinformation or fake social media accounts or targeting children with automatically generated content that hacks the YouTube recommendation system and the impact effect of this has been that distrust in tech companies has risen markedly in the last 10 years so this is from the public affairs pulse survey just last year the tech industry went from being in 2013 one of the industries that the fewest people felt was less trustworthy than average to rubbing elbows with famously much distrusted Industries like energy and pharmaceuticals and the tech industry doesn't have to win elections so we
+
+22
+00:14:29,220 --> 00:15:16,079
+don't have to care about public polling as much as politicians but politicians care quite a bit about those public opinion polls and just in the last few years the fraction of people who believe that the large tech companies should be more regulated has gone up a substantial amount and comparing it to 10 years ago it's astronomically higher so there will be substantial impacts on our industry due to this loss of public trust so as machine learning engineers and researchers we can learn from nearby Fields so I'll talk about two of them one a nice little bit about the culture of professional ethics in Engineering in Canada and then a little bit about ethical standards for human subjects research so one of the worst
+
+23
+00:15:12,779 --> 00:15:57,899
+construction disasters in modern history was the collapse of the Quebec bridge in 1907. 75 people who were working on the bridge at the time were killed and a parliamentary inquiry placed the blame pretty much entirely on two engineers in response there was the development of some additional rituals that many Canadian Engineers take part in when they finish their education that are meant to impress upon them the weight of their responsibility so one component of this is a large iron ring which literally impresses that weight upon people and then another is an oath that people take a non-legally binding oath that includes saying that I will not hence forward suffer or pass or be privy to the passing of bad workmanship or
+
+24
+00:15:56,100 --> 00:16:37,800
+faulty material I think the software would look quite a bit different if software Engineers took an oath like this and took it seriously one other piece I wanted to point out is that it includes within it some built-in humility asking pardon ahead of time for the assured failures lots of machine learning is still in the research stage and so some people may say that oh well that's important for the people who are building stuff but I'm working on R D for fundamental technology so I don't have to worry about that but research is also subject to regulation so this is something I was required to learn because I did my PhD in a neuroscience Department that was funded by the National Institutes of Health which
+
+25
+00:16:34,500 --> 00:17:18,360
+mandates training in ethics and in the ethical conduct of research so these regulations for human subjects research date back to the 1940s when there were medical experiments on unwilling human subjects by totalitarian regimes this is still pretty much the Cornerstone for laws on human subjects research around the world through the Helsinki declaration which gets regularly updated in the US the Touchstone bit of regulation on this the 9 1973 research act requires among other things informed consent from people who are participating in research and there were two major revelations in the late 60s and early 70s that led to this legislation not dissimilar to the scandals that have plagued the technology industry recently one was the
+
+26
+00:17:16,740 --> 00:18:04,860
+infliction of hepatitis on mentally disabled children in New York in order to test hepatitis treatments and the other was the non-treatment of syphilis in black men at Tuskegee in order to study the progression of the disease in both cases these subjects did not provide informed consent and seemed to be selected for being unable to advocate for themselves or to get legal redress for the harms they were suffering and so if we are running experiments and those experiments involve humans evolve our users we are expected to adhere to the same principles and one of the famous instances of mismatch between the culture in our industry and the culture of human subjects research was was when some researchers at Facebook studied
+
+27
+00:18:02,760 --> 00:18:49,440
+emotional contagion by altering people's news feeds either adding more negative content or adding more positive content and they found a modest but robust effect that introducing more positive content caused people to post more positively when people found out about this they were very upset the authors noted that Facebook's data use policy includes that the user's data and interactions can be used for this but most people who were Facebook users and the editorial board of pnas where this was published did not see it that way so put together I think we are at the point where we need a professional code of ethics for software hopefully many codes of Ethics developed in different communities that can Bubble Up compete
+
+28
+00:18:48,000 --> 00:19:35,700
+with each other and merge to finally something that most of us or all of us can agree on and that is incorporated into our education and acculturation of new members into our field and into more aspects of how we build to close out this section I wanted to talk about some particular ethical concerns that arise in Tech in general first around carbon emissions and then second around dark patterns and user hostile designs the good news with carbon emissions is that because they scale with cost it's only something that you need to worry about when the costs of what you're building what you're working on are very large at which time you both won't be alone in making these decisions and you can move a bit more deliberately and make these
+
+29
+00:19:32,760 --> 00:20:19,440
+choices more thoughtfully so first what are the ethical concerns with carbon emissions anthropogenic climate change driven by CO2 emissions raises a classic trade-off which was dramatized in this episode of Harvey Birdman Attorney at Law in which George Jetson travels back from the future to sue the present for melting the ice caps and destroying his civilization so unfortunately we don't have future Generations present now to advocate for themselves the other view is that this is an issue that arises from a classic alignment problem which is many organizations are trying to maximize their profit that raw profit is based off of prices for goods that don't include externalities like the environmental damage caused by carbon
+
+30
+00:20:16,740 --> 00:21:06,960
+dioxide emissions leading to increased temperatures and climactic change so the primary Dimension along which we have to worry about carbon emissions is in compute jobs that require power that power has to be generated somehow and that can result in the emission of carbon and so there was a nice paper Linked In This slide that walked through how much carbon dioxide was emitted using typical us-based Cloud infrastructure and the top headline from this paper was that training a large Transformer model with neural architecture search produces as much carbon dioxide as five cars create during their lifetime so that sounds like quite a bit of carbon dioxide and it is in fact but it's important to remember that power is not free and so
+
+31
+00:21:05,160 --> 00:21:49,200
+there is a metric that we're quite used to tracking that is at least correlated with our carbon emissions our compute spend and if you look for the cost runs between one and three million dollars to run the neural architecture search that emitted five cars worth of CO2 and one to three million dollars is actually a bit more than it would cost to buy five cars and provide their fuel so the number that I like to use is that four us-based Cloud infrastructure like the US West one that many of us find ourselves in ten dollars of cloud spend is roughly equal to one dollar worth of air travel costs so that's on the basis of something like the numbers in the chart indicating air travel across the United States from New York to San
+
+32
+00:21:47,220 --> 00:22:28,799
+Francisco I've been taking care to always say us-based cloud infrastructure because just changing Cloud regions can actually reduce your emissions quite a bit there's actually a factor of nearly 50x from some of the some of the cloud regions that have have the most carbon intensive power generation like AP Southeast 2 and the regions that have the the least carbon intensive power like ca Central one that chart comes from a nice talk from hugging face that you can find on YouTube part of their course that talks a little bit more about that paper and about managing carbon emissions interest in this problem has led to some nice new tooling one code carbon dot IO allows you to track power consumption and therefore
+
+33
+00:22:26,820 --> 00:23:12,419
+CO2 emissions just like you would any of your other metrics and then there's also this mlco2 impact tool that's oriented a little bit more directly towards machine learning the other ethical concern in Tech that I wanted to bring up is deceptive design and how to recognize it an unfortunate amount of deception is tolerated in some areas of software the example on the left comes from an article by Narayanan at all that shows a fake countdown timer that claims that an offer will only be available for an hour but when it hits zero nothing the offer is still there there's also a possibly apocryphal example on the right here you may have seen these numbers next to products when online shopping saying that some number of people are currently
+
+34
+00:23:10,799 --> 00:23:57,059
+looking at this product this little snippet of JavaScript here produces a random number to put in that spot so that example on the right may not be real but because of real examples like the one on the left it strikes a chord with a lot of developers and Engineers there's a kind of slippery slope here that goes from being unclear or maybe not maximally upfront about something that is a source of friction or a negative user experience in your product and then in trying to remove that friction or sand that edge down you slowly find yourself being effectively deceptive to your users on the left is a nearly complete history of the way Google displays ads in its search engine results it started off very clearly
+
+35
+00:23:54,960 --> 00:24:38,039
+colored and separated out with a bright color from the rest of the results and then a about 10 years ago that colored background was removed and replaced with just a tiny little colored snippet that said add and now as of 2020 that small bit there is no longer even colored it's just bolded and so this makes it difficult for users to know which content is being served to them because somebody paid for them to see it versus being served up organically so a number of patterns of deceptive design also known as dark patterns have emerged over the last 10 years you can read about them on this website deceptive.design there's also a Twitter account at dark patterns where you can share examples that you find in the wild so some
+
+36
+00:24:36,360 --> 00:25:19,980
+examples that you might be familiar with are the roach motel named after a kind of insect trap where you can get into a situation very easily but then it's very hard to get out of it if you've ever attempted to cancel a gym membership or delete your Amazon account then you may have found yourself a roach in a motel another example is trick questions where forms intentionally make it difficult to choose the option that most use users want for example using negation in a non-standard way like check this box to not receive emails from our service one practice in our industry that's on very shaky ethical and legal ground is growth hacking which is a set of techniques for achieving really rapid growth in user
+
+37
+00:25:17,820 --> 00:26:00,659
+base or revenue for a product and has all the connotations you might expect from the name hack LinkedIn was famously very spammy when it first got started I'd like to add you to my Professional Network on LinkedIn became something of a meme and this was in part because LinkedIn made it very easy to unintentionally send LinkedIn invitations to every person you'd ever emailed they ended up actually having to pay out in a class action lawsuit because they were sending multiple follow-up emails when user only clicked to send an invitation once and the structure of their emails made it seem like they were being sent by the user rather than LinkedIn and the use of these growth hacks goes back to the very Inception of email Hotmail Market itself
+
+38
+00:25:58,200 --> 00:26:43,860
+in part by attacking on a signature to the bottom of every email that said PS I love you get your free email at Hotmail so this seemed like it was being sent by the actual user I grabbed a snippet from a top 10 growth hacks article that said that the personal sounding nature of the message and the fact that it came from a friend made this a very effective growth hack but it's fundamentally deceptive to add this to messages in such a way that it seems personal and to not tell users that this change is being made to the emails that they're sending so machine learning can actually make this problem worse if we are optimizing short-term metrics these growth acts and deceptive designs can often Drive user and revenue
+
+39
+00:26:41,460 --> 00:27:25,679
+growth in the short term but they do that by worsening user experience and drawing down on Goodwill towards the brand in a way that can erode the long-term value of customers when we incorporate machine learning into the design of our products with a B testing we have to watch out to make sure that the the metrics that we're optimizing don't encourage this kind of deception so consider these two examples on the right the top example is a very straightforwardly implemented and direct and easy to understand form for users to indicate whether they want to receive emails from the company and from its Affiliates in example B the wording of the first message has been changed so that it indicates that the first hitbox
+
+40
+00:27:23,520 --> 00:28:05,760
+should be checked to not receive emails while the second one should not be ticked in order to not receive emails and if you're a b testing these two designs against each other and your metric is the number of people who sign up to receive emails then it's highly likely that the system is going to select example B so taking care and setting up a b tests such that either they're tracking longer term metrics or things that correlate with them and that the variant generation system that generates all the different possible designs can't generate any designs that we would be unhappy with as we would hopefully be unhappy with the deceptive design in example B and I think it's also important to call out that this
+
+41
+00:28:03,840 --> 00:28:46,679
+problem arises inside of another alignment problem we were considering the case where the long-term value of customers and the company's interests were being harmed by these deceptive designs but unfortunately that's not always going to be the case the private Enterprises that build most technology these days are able to deliver Broad Social value to make the world a better place as they say but the way that they do that is generally by optimizing metrics that are at best a very weak proxy for that value that they're delivering like their market capitalization and so there's the possibility of an alignment problem where companies pursuing and maximizing their own profit and success can lead to net negative production of value and
+
+42
+00:28:44,880 --> 00:29:23,700
+this misalignment is something that if you spend time at the intersection of capital and funding leadership and Technology development you will encounter it so it's important to consider these questions ahead of time and come to your own position whether that's trade reading this as the price of doing business or the way the world Works seeking ways to improve this alignment or considering different ways to build technology but on the shorter term you can push for longer term thinking within your organization to allow for better alignment between the metrics that you're measuring and the goals that you're setting and between the goals that you're setting and what is overall good for our industry and for
+
+43
+00:29:21,960 --> 00:30:06,480
+the broader world and you can also learn to recognize these user hostile design patterns call them out when you see them and you can advocate for a More user-centered Design instead so to wrap up our section on ethics for Building Technology broadly we as an industry should learn from other disciplines if we want to avoid a trust crisis or if we want to avoid the crisis getting any worse and we can start by educating ourselves about the common user hostile practices in our industry and how to avoid them now that we've covered the kinds of ethical concerns and conflicts that come up when Building Technology in general let's talk about concerns that are specific to machine learning just in the past couple of years there have been
+
+44
+00:30:04,200 --> 00:30:42,480
+more and more ethical concerns raised about the uses of machine learning and this has gone beyond just the ethical questions that can get raised about other kinds of technology so we'll talk about some of the common ethical questions that have been raised repeatedly over the last couple of years and then we'll close out by talking about what we can learn from a particular sub-discipline of machine learning medical machine learning so the fundamental reason I think that ethics is different for machine learning and maybe more Salient is that machine learning touches human lives more intimately than a lot of other kinds of technology so many machine learning methods especially deep learning methods make human legible data into computer
+
+45
+00:30:40,320 --> 00:31:22,860
+legible data so we're working on things like computer vision on processing natural language and humans are more sensitive to errors in and have more opinions about this kind of data about images like this puppy than they do about the other kinds of data manipulated by computers like abstract syntax trees so because of of this there are more stakeholders with more concerns that need to be traded off in machine learning applications and then more broadly machine learning involves being wrong pretty much all the time there's the famous statement that all models are wrong though some are useful and I think the first part applies at least particularly strongly to machine learning our models are statistical and
+
+46
+00:31:20,760 --> 00:32:01,320
+include in them Randomness the way that we frame our problems the way that we frame our optimization in terms of cross entropies or divergences and Randomness is almost always an admission of ignorance even the quintessential examples of Randomness like random number generation in computers and the flipping of a coin are things that we know in fact are not random truly they are in fact predictable and if we knew the right things and had the right laws of physics and the right computational power then we could predict how a coin would land we could control it we could predict what the next number to come out of a random number generator would be whether it's pseudorandom or based on some kind of Hardware Randomness and so
+
+47
+00:31:59,760 --> 00:32:41,520
+we're admitting a certain degree of ignorance in our models and that means our models are going to be wrong and they're going to misunderstand situations that they are put into and it can be very upsetting and even harmful to be misunderstood by a machine learning model so against this backdrop of Greater interest or higher Stakes a number of common types of ethical concern have coalesced in the last couple of years and there are somewhat established camps of answers to these questions and you should at least know where it is you stand on these core questions so for four really important questions that you should be able to answer about about anything that you build with machine learning are is the model fair and what does that mean in
+
+48
+00:32:39,840 --> 00:33:20,039
+this situation is the system that you're building accountable who owns the data involved in this system and finally and perhaps most importantly an undergirding all of these questions is should this system be built at all so first is the model we're building Fair the classic case on this comes from Criminal Justice from the compass system for predicting before trial whether a defendant will be arrested again so if they're arrested again that's just they committed a crime during that time and so this is assessing a certain degree of risk for additional harm while the justice system is deciding what to do about a previous arrest and potential crime so the operationalization here was a 10-point rearrest probability based off of past
+
+49
+00:33:17,640 --> 00:34:07,799
+data about this person and they set a goal from the very beginning to be less biased than human judges so they operationalize that by calibrating these arrest probabilities and making sure that if say a person received a 2 2 on this scale they had a 20 chance of being arrested again and then critically that those probabilities were calibrated across subgroups so racial bias is one of the primary concerns around bias in criminal justice in the United States and so they took care to make sure that these probabilities of rearrest were calibrated for all racial groups the system was deployed in it is actually used all around the United States it's proprietary so it's difficult to analyze but using the Freedom of Information Act
+
+50
+00:34:05,519 --> 00:34:53,820
+and by colliding together a bunch of Records some people at propublica were able to run their own analysis of this algorithm and they determined that though this calibration that Compass claimed for arrest probabilities was there so the model was not more or less wrong for one racial group or another the way that the model tended to fail was different across racial groups so the model had more false positives for black defendants so saying that somebody was higher risk but then them not going on to reoffend and had more false negatives for white defendants so labeling them as low risk and then them going on to reoffend so despite North Point the creators of compass taking into account bias from the beginning
+
+51
+00:34:51,599 --> 00:35:31,260
+they ended up with an algorithm with this undesirable property of being more likely to effectively falsely accuse defendants who were black than defendants who were white this report touched off a ton of controversy and back and forth between propublica the creator of the article and North Point Craters of compass and also a bunch of research and it turned out that some quick algebra revealed that some form of race-based bias is inevitable in this setting so the things that we care about when we're building a binary classifier are relatively simple you can write down all of these metrics directly so we care about things like the false positive rate which means we've imprisoned somebody with no need the false negative
+
+52
+00:35:29,760 --> 00:36:16,380
+rate which means we missed an opportunity to event a situation that led to an arrest and then we also care about the positive predictive value which is this rearrest probability that Compass was calibrated on so because all of these metrics are related to each other and related to The Joint probability distribution of our model's labels and the actual ground truth if the probability of rearrest differs across groups then we have to have that some of these numbers are different across groups and that is a form of racial bias so the basic way that this argument works just involves rearranging these numbers and saying that if the numbers on the left side of this equation are different for Group 1 and group two then it can't possibly be the
+
+53
+00:36:14,280 --> 00:36:54,900
+case that all three of the numbers on the right hand side are the same for Group 1 and group two and I'm presenting this here as though it only impacts these specific binary classification metrics but there are are in fact a very large number of definitions of fairness which are mutually incompatible so there's a nice a really incredible tutorial by Arvin Narayanan who was also the first author on the dark patterns work on a bunch of these fairness definitions what they mean and why they're in commensurate so I can highly recommend that lecture so returning to our concrete case if the prevalence is differ across groups then one of our things that we're concerned with the false positive rate the false negative
+
+54
+00:36:53,160 --> 00:37:35,339
+rate or the positive predictive value will not be equal and that's something that people can point to and say that's unfair in the middle that positive predictive value was equalized across groups in compass that was what they really wanted to make sure was equal cross groups and because the probability of rearrest was larger for black defendants then either the false positive rate had to be bigger or the false negative rate had to be bigger for that group and there's an analysis in this cholachova 2017 paper that suggests that the usual way that this will work is that there will be a higher false positive rate for the group with a larger prevalence so the fact that there will be some form of unfairness that we
+
+55
+00:37:33,420 --> 00:38:14,280
+can't just say oh well all these metrics are the same across all groups and so everything has to be fair that fact is fixed but the impact of the unfairness of models is not fixed the story is often presented as oh well no matter what the journalists would have found something to complain about there's always critics and so you know you don't need to worry about fairness that much but I think it's important to note that the particular kind of unfairness that came about from this model from focusing on this positive predictive value led to a higher false positive rate more unnecessary imprisonment for black defendants the false positive rate and the positive predictive value were equalized across groups that would have
+
+56
+00:38:12,420 --> 00:38:54,119
+led to a higher false negative rate for black defendants relative to White defendants and in the context of American politics and concerns about racial inequity in the criminal justice system bias against white defendants is not going to lead to complaints from the same people and has a different relationship to the historical operation of the American justice system and so far from this being a story about the hopelessness of thinking about or caring about fairness this is a story about the necessity of confronting the trade-offs that are inevitably going to come up so some researchers that Google made a nice little tool where you can try thinking through and making these trade-offs for yourself it's a loan decision rather
+
+57
+00:38:51,900 --> 00:39:37,740
+than a criminal justice decision but it has a lot of the same properties you have a binary classifier you have different possible goals that you might set either maximizing the profit of the loaning entity or providing equal opportunity to the two groups and it's very helpful for building intuition on these fairness metrics and what it means to pay pick one over the other and these events in this controversy kicked off a real flurry of research on fairness and there's now been several years of this fairness accountability and transparity Conference fact there's tons of work on both algorithmic level approaches to try and measure these fairness metrics incorporate them into training and also more qualitative work on designing
+
+58
+00:39:36,180 --> 00:40:20,940
+systems that are more transparent and accountable so the compass example is really important for dramatizing these issues of fairness but I think it's very critical for this case and for many others to step back and ask whether this model should be built at all so this algorithm for scoring risk is proprietary and uninterpretable it doesn't give answers for why a person is higher risk or not and because it is closed Source there's no way to examine it it achieves an accuracy of about 65 which is quite High given that the marginal probability of reoffence is much lower than 50 but it's important to compare the baselines here pulling together a bunch of non-experts like you would on a jury has an accuracy of about
+
+59
+00:40:17,640 --> 00:41:01,140
+65 percent and creating a simple scoring system on the basis of how old the person is and how many prior arrests they have also has an accuracy of around 65 and it's much easier to feel comfortable with the system that says if you've been arrested twice then you have a higher risk of being arrested again and so you'll be imprisoned before trial then a system that just says oh well we ran the numbers and it looks like you have a high chance of committing a crime but even framing this problem in terms of who is likely to be rearrested is already potentially a mistake so a slightly different example of predicting failure to appear in court was tweeted out by Moritz heart who's one of the main researchers in this area choosing
+
+60
+00:40:59,520 --> 00:41:37,320
+to try to predict who will fail to appear in court treating this as something that is then a fact of the universe that this person is likely to fail to appear in court and then intervening on this and punishing them for that for that fact it's important to recognize why people fail to appear in court in general often it's because they don't have child care to cover for the care of their dependence while they're in court they don't have transportation their work schedule is inflexible or the core deployment schedule is inflexible or unreasonable it'd be better to implement steps to mitigate these issues and reduce the number of people who are likely to fail to appear in court for example by making it possible to join
+
+61
+00:41:35,579 --> 00:42:20,940
+Court remotely that's a far better approach for all involved than simply getting really really good at predicting Who currently fails to appear in court so it's important to remember that the things that we're measuring the things that we're predicting are not the be-all end-all in themselves the things that we care about are things like an effective and fair justice system and this comes up perhaps most acutely in the case of compass when we recognize that rearrest is not the same as recidivism it's not the same thing as committing more crimes being rearrested requires that a police officer believes that you committed a crime police officers are subject effect to their own biases and patterns of policing result in a far higher fraction
+
+62
+00:42:18,480 --> 00:43:04,440
+of crimes being caught for some groups than for others and so our real goal in terms of fairness and criminal justice might be around reducing those kinds of unfair impacts and using past rearrest data that we know has these issues to determine who is treated more harshly by the criminal justice system is likely to exacerbate these issues there's also a notion of model fairness that is broader than just models that make decisions about human beings so even if you're deciding a model that works on text or works on images you should consider which kinds of people your model works well for and in general representation both on engineering and management teams and in data sets really matters for this kind of model fairness so it's
+
+63
+00:43:02,640 --> 00:43:46,680
+unfortunately still very easy to make machine learning powered technology that fails for minoritized groups so for example off-the-shelf computer vision tools will often fail on darker skin so this is an example by Joy bull and weenie from MIT on how a computer vision based project that she was working on ran into difficulties because the face detection algorithm could not detect her face even though it could detect the faces of some of her friends with lighter skin and in fact she found that just putting on a white mask was enough to get the computer vision model to detect her face so this is unfortunately not a new issue in technology it's just a more Salient one with machine learning so one example is that hand soap
+
+64
+00:43:43,920 --> 00:44:32,640
+dispensers that use infrared to determine when to dispense soap will often work better for lighter skin than darker skin and issues around lighting and vision and skin tone go back to the foundation of Photography let alone computer vision the design of film of cameras and printing processes was oriented around primarily making lighter skin photograph well as in these so-called Shirley cards that were used by code DAC for calibration these resulted in much worse experiences for people with darker skin using these cameras there has been a good amount of work on this and progress since four or five years ago one example of the kind of tool that can help with this are these model cards this particular format
+
+65
+00:44:30,660 --> 00:45:14,760
+for talking about what a model can and cannot do that was published by a number of researchers including Margaret Mitchell and Timmy Gabriel it includes explicitly considering things like on which human subgroups of Interest many of them minoritized identities how well does the model perform hugging face has good Integrations for creating these kinds of model cards I think it's important to note that just solving these things by changing the data around or by calculating demographic information is not really an adequate response if the CEO of Kodak or their partner had been photographed poorly by those cameras then there's no chance that that issue would have been allowed to stay for decades so when you're
+
+66
+00:45:12,359 --> 00:45:57,839
+looking at inviting people for talks hiring people or joining organizations you should try to make sure that you have worked to reduce the bias of that Discovery process by diversifying your network and your input sources the diversify Tech job board is a really wonderful source for candidates and then there are also professional organizations inside of the ml World black and Ai and women in data science being two of the larger and more successful ones these are great places to get started to make the kinds of professional connections that can improve the representations of these minoritized groups in the engineering and design and product management process where these kinds of issues should be solved a lot of progress has
+
+67
+00:45:55,680 --> 00:46:44,040
+been made but these problems are still pretty difficult to solve an unbiased face detector might not be so challenging but unbiased image generation is still really difficult for example if you make an image generation model from internet scraped data without any safeguards in place then if you ask it to generate a picture of a CEO it will generate the stereotypical CEO a six foot or taller white man and this applies across a wide set of jobs and situations people can find themselves in and this led to a lot of criticism of early text damage generation models like Dolly and the solution that openai opted to this was to edit prompts that people put in if you did not fully specify what kind of person should be generated then
+
+68
+00:46:41,579 --> 00:47:21,300
+race and gender words would be added to the prompt with weights based on the world's population so people discovered this somewhat embarrassingly by writing prompts like a person holding a sign that says or pixel art of a person holding a text sign that says and then seeing that the appended words were then printed out by the model suffice it to say that this change did not make very many people very happy and indicates that more work needs to be done to de-bias image generation models at a broader level than just fairness we can also ask whether the system we're building is accountable to the people it's serving or acting upon and this is important because some people can consider explanation and accountability
+
+69
+00:47:19,560 --> 00:48:00,060
+in the face of important judgments to be human rights this is the right to an explanation in the European Union's general data protection regulation gdpr there is a subsection that mentions the right to obtain an explanation of a decision reached after automated assessment and the right to challenge that decision the legal status here is a little bit unclear there's a nice archive paper that talks about this a bit about what the right to an explanation might mean but what's more important for our purposes is just to know that there is an increasing chorus of people claiming that this is indeed a human right and it's not an entirely New Concept and it's not even really technology or automation specific as far
+
+70
+00:47:57,420 --> 00:48:40,800
+back as 1974 has been the law in the United States that If you deny credit to a person you must disclose the principal reasons for denying that credit application and in fact I found this interesting it's expected that you provide no more than four reasons why you denied them credit but the general idea that somebody as a right to know why something happened to them in certain cases is enshrined in some laws so what are we supposed to do if we use a deep neural network to decide whether somebody should be Advanced Credit or not so there are some off-the-shelf methods for introspecting deep neural networks that are all based off of input output gradients how would changing the pixels of this input image change the
+
+71
+00:48:38,579 --> 00:49:18,359
+class probabilities and the output so this captures a kind of local contribution but as you can see from the small image there it doesn't produce a very compelling map and there's no reason to think that just changing one pixel a tiny bit should really change the model's output that much one Improvement to that called Smooth grad is to add noise to the input and then average results kind of getting a sense for what the gradients look like in a general area around the input there isn't great theory on why that should give better explanations but people tend to find these explanations better and you can see in the smooth grad image on the left there that you can pick out the picture of a bird it seems like that is
+
+72
+00:49:16,440 --> 00:50:04,560
+giving a better explanation or an explanation that we like better for why this network is identifying that as a picture of a bird there's a bunch of kind of hacking methods like specific tricks you need when you're when you're using the relu activation there's some methods that are better for classification like grad cam one that is more popular integrated gradients takes the integral of the gradient along a path from some baseline to the final image and this method has a nice interpretation in terms of Cooperative Game Theory something called a shapley value that quantifies how much a particular collection of players in a game contributed to the final reward and adding noise to integrated gradients tends to produce really clean
+
+73
+00:50:02,460 --> 00:50:45,480
+explanations that people like but unfortunately these methods are generally not very robust their outputs tend to correlate pretty strongly in the case of images with just an edge detector there's built-in biases to convolutional networks and the architectures that we use that 10 and to emphasize certain features of images what this particular chart shows from this archive paper by Julius adebayo Moritz heart and others is that even as we randomize layers in the network going from left to right we are randomizing starting at the top of the network and then randomizing more layers going down even for popular methods like integrated gradients with smoothing or guided back propagation we can effectively randomize
+
+74
+00:50:43,800 --> 00:51:27,140
+a really large fraction of the network without changing the gross features of the explanation and resulting in an explanation that people would still accept and believe even though this network is now producing random output so in general introspecting deep neural networks and figuring out what's going inside them requires something that looks a lot more like a reverse engineering process that's still very much a research problem there's some great work on distill on reverse engineering primarily Vision networks and then some great work from anthropic AI recently on Transformer circuits that's reverse engineering large Lang language models and Chris Ola is the researcher who's done the most work here but it still is the sort of thing that
+
+75
+00:51:24,300 --> 00:52:06,839
+even getting a loose qualitative sense for how neural networks work and what they are doing in response to inputs is still the type of thing that takes a research team several years so Building A system that can explain why it took a particular decision is maybe not currently possible with deep neural networks but that doesn't mean that the systems that we build with them have to be unaccountable if somebody dislikes the decision that they get and the explanation that we give is well the neural network said you shouldn't get a loan and they challenge that it might be time to bring in a human in the loop to make that decision and building that in to the system so that it's an expected mode of operation and is considered an
+
+76
+00:52:03,900 --> 00:52:50,099
+important part of the feedback and the operation of the system is key to building an accountable system so this book automating inequality by Virginia Eubanks talks a little bit about the ways in which Technical Systems as their build today are very prone to this unaccountability where the people who are Indian most impacted by these systems some of the most critical stakeholders for these systems for example recipients of government assistance are unable to have their voices and their needs heard and taken into account in the operation of a system so this is perhaps the point at which you should ask when building a system with machine learning whether this should be built at all and particular to ask who benefits and who
+
+77
+00:52:47,579 --> 00:53:28,500
+is harmed by automating this task in addition to concerns around the behavior of models increasing concern has been pointed towards data and in particular who owns and who has rights to the data involved in the creation of machine Learning Systems it's important to remember that the training data that we use for our machine learning algorithms is almost always generated by humans and they generally feel some ownership over that data and we end up behaving a little bit like this comic on the right where they hand us some data that they made and then we say oh this is ours now I made this and in particular the large data sets you train the really large models that are pushing the frontiers of what is possible with machine learning
+
+78
+00:53:26,220 --> 00:54:09,359
+are produced by crawling the Internet by searching over all the images all the text posted on the internet and pulling large fractions of it down and many people are not aware that this is possible let alone legal and so to some extent any consent that they gave to their data being used was not informed and then additionally as technology has changed in the last decade and machine learning has gotten better what can be done with data has changed somebody uploading their art a decade ago certainly did not have on their radar the idea that they were giving consent to that art being used to create an algorithm that can mimic its style and you can in fact check whether an image of interest to you has been used to
+
+79
+00:54:06,240 --> 00:54:53,520
+train one of the large text image models specifically this have I been trained.com website will search through the Leon data set that is used to train the stable diffusion model for images that you upload so you can look to see if any pictures of you were incorporated into the data set and this goes further than just pictures that people might rather not have used in this way to actual data that has somehow been obtained illegally there's an Arts technical article a particular artist who was interested in this found that some of their medical photos which they did not consent to have uploaded to the internet somehow found their way into the lay on data set and so cleaning large web scraped data sets from this
+
+80
+00:54:51,540 --> 00:55:37,800
+kind of illegally obtained data is definitely going to be important as more attention is paid to these models as they are product eyes and monetized and more on people's radar even for data that is obtained legally saying well technically you did agree to this does not generally satisfy people remember the Facebook emotion research study technically some reading of the Facebook user data policy did support the way that they were running their experiment but many users disagreed many artists feel that creating an art generation tool that threatens their livelihoods and copies art down to the point of even faking watermarks and logos on images when told to recreate the style of an artist is an ethical use of that data
+
+81
+00:55:35,700 --> 00:56:23,400
+and it certainly is the case that creating a sort of parrot that can mimic somebody is something that a lot of people find concerning dealing with these issues around data governance is likely to be a new frontier imagine of stable diffusion has said that he's partnering with people to create mechanisms for artists to opt in or opt out of being included in training data sets for future versions of stable diffusion I found that noteworthy because mostacc has been very vocal in his defense of image generation technology and of what it can be used for but even he is interested in adjusting the way data is used there's also been work from Tech forward artists like Holly Hunter who was involved in the creation of have I been trained
+
+82
+00:56:20,520 --> 00:57:05,460
+around trying to incorporate AI systems into art in a way that empowers artists and compensates them rather than immiserating them just as we can create cards for models we can also create cards for data sets that describe how they were curated what the sources were and any other potential issues with the data and perhaps in the future even how to opt out of or be removed from a data set so this is an example from a hugging face as with model cards there's lots of good examples of data set cards on hugging face there's also a nice checklist the Dion ethics checklist that is mostly focused around data ethics but covers a lot of other ground they also have this nice list of examples for each question in their checklist of cases
+
+83
+00:57:02,760 --> 00:57:47,339
+where people have run into ethical or legal trouble by building an ml project that didn't satisfy a particular checklist item running underneath all of this has been this final most important question of whether this system should be built at all one particular use case that very frequently elicits this question is building ml-powered Weaponry ml powered Weaponry is already here it's already starting to be deployed in the world there are some remote controlled weapons that use computer vision for targeting deployed by the Israeli military in the West Bank using this smart shooter technology that's designed to in principle take normal weapons and add computer vision based targeting to them to make them into smart weapons
+
+84
+00:57:44,819 --> 00:58:29,220
+right now this deployed system shown on the left uses only sponge tipped bullets which are designed to be less lethal but they can still cause serious injury and according to the deployers in the pilot stage so it's a little unclear to what extent autonomous Weaponry is already here and being used because the definition is a little bit blurry so for example the hayrop Drone shown in the top left is a loitering munition a type of drone that can fly around hold its position for a while and then automatically destroy any radar system that locks onto it this type of drone was used in the nagorno-karabakh war between Armenia and Azerbaijan in 2021 but there's also older autonomous weapon systems the Phalanx c-whiz is designed
+
+85
+00:58:27,660 --> 00:59:10,740
+to automatically fire at Targets moving towards Naval vessels at very very high velocities so these are velocities they're usually only achieved by rocket Munitions not by manned craft and that system's been used since at least the first Gulf War in 1991. there was an analysis in 2017 by The Economist to try and look for how many systems with automated targeting there were and in particular how many of them could engage with targets without involving humans at all so that would be the last section of human out of the loop systems but given the general level of secrecy in some cases and hype and others around military technology it can be difficult to get a very clear sense and the blurriness of this definition has led
+
+86
+00:59:08,940 --> 00:59:50,339
+some to say that autonomous weapons are actually at least 100 years old for example anti-personnel mines that were used starting in the 30s and in World War II attempts to you detect whether a person has come close to them and then explode and in some sense that is an autonomous weapon and if we broaden our definition that far then maybe lots of different kinds of traps are some form of autonomous weapon but just because these weapons already exist and maybe even have been around for a century does not mean that designing ml-powered weapons is ethical anti-personnel mines in fact are the subject of a mind Ban Treaty that a very large number of countries have signed unfortunately not some of the countries with the largest
+
+87
+00:59:47,400 --> 01:00:26,880
+militaries in the world but that at least suggests that for one type of autonomous weapon that has caused a tremendous amount of collateral damage there's interest in Banning them and so perhaps rather than building these autonomous weapons so we can then ban them it would be better if we just didn't build them at all so the campaign to stop Killer Robots is a group to look into if this is something that's interesting to you it brings us to the end of our tour of the four common questions that people raise around the ethics of building an ml system I've provided some of my answers to these questions and some of the common answers to these questions but you should have thoughtful answers to these for the
+
+88
+01:00:25,619 --> 01:01:07,619
+individual projects that you work on first is the model fair I think it's generally possible but it requires trade-offs is the system accountable I think it's pretty challenging to make interpretable deep Learning Systems where interpretability allows an explanation for why a decision was made but making a system that's accountable where answers can be changed in response to user feedback or perhaps user lawsuit is possible you'll definitely want to answer the question of who owns the data up front and be on the lookout for changes especially to these large-scale internet scraped data sets and then lastly should this be built at all you'll want to ask this repeatedly throughout the life cycle of the technology I wanted to close this
+
+89
+01:01:04,920 --> 01:01:53,460
+section by talking about just how much the machine learning world can learn from medicine and from applications of machine learning to medical problems this is a field I've had a chance to work in and I've seen some of the best work on building with ML responsibly come from this field and fundamentally it's because of a mismatch between machine learning and medicine that impedance mismatch has led to a ton of learning so first we'll talk about the Fiasco that was machine learning and the covid-19 pandemic then briefly consider why medicine would have this big of a mismatch with machine learning and what the benefits of examining it closer might be and then lastly we'll talk about some concrete research on auditing
+
+90
+01:01:50,819 --> 01:02:34,680
+and Frameworks for building with ML that have come out of medicine first something that should be scary and embarrassing for people in machine learning medical researchers found that almost all machine learning research on covid-19 was effectively useless this is in the context of a biomedical response to covid-19 that was an absolute Triumph in the first year vaccinations prevented some tens of millions of deaths these vaccines were designed based on novel Technologies like lipid nanoparticles for delivering mRNA and even more traditional techniques like small molecule Therapeutics for example paxilavid the quality of research that was done was extremely high so on the right we have an inferred 3D structure
+
+91
+01:02:32,099 --> 01:03:24,900
+for a coronavirus protein in complex with the primary effective molecule in paxilavid allowing for a mechanistic understanding of how this drug was working at the atomic level and at this crucial time machine learning did not really acquit itself well so there were two reviews one in bmj and one in nature that reviewed a large set of prediction models for covid-19 either prognosis or diagnosis primarily prognosis in the case of the Winans at all paper in bmj or diagnosis on the basis of chest x-rays and CT scans and both of these reviews found that almost all of the papers were insufficiently documented did not follow best practices for developing models and did not have sufficient external validation testing
+
+92
+01:03:21,059 --> 01:04:06,780
+on external data to justify any wider use of these models even though many of them were provided as software or apis ready to be used in a clinical setting so the depth of the errors here is really very sobering a full quarter of the papers analyzed in the Roberts at all review used a pneumonia data set as a control group so the idea was we don't want our model just to detect whether people are sick or not just having having coveted patients and healthy patients might cause models that detect all pneumonias as covid so let's incorporate this pneumonia data set but they failed to mention and perhaps failed to notice that the pneumonia data set was all children all pediatric patients so the models that they were
+
+93
+01:04:04,799 --> 01:04:52,079
+training were very likely just detecting children versus adults because that would give them perfect performance on Pneumonia versus covid on that data set so it's a pretty egregious error of modeling and data set construction alongside bunch of other more subtle errors around proper validation and reporting of models and methods so I think one reason for the substantial difference in responses here is that medicine both in practice and in research has a very strong professional culture of Ethics that equips it to handle very very serious and difficult problems at least in the United States medical doctors still take the Hippocratic Oath parts of which date back all the way to Hippocrates one of the founding fathers of Greek medicine
+
+94
+01:04:49,559 --> 01:05:42,839
+and one of the core precepts of that oath is to do no harm meanwhile one of the core precepts of the Contemporary tech industry represented here by this ml generated Greek bust of Mark Zuckerberg is to move fast and break things with the implication that breaking things is not so bad and well that's probably the right approach for building lots of kinds of web applications and other software when this culture gets applied to things like medicine the results can be really ugly one particularly striking example of this was when a retinal implant that was used to restore sight to some blind people was deprecated by the vendor and so stopped working and there was no recourse for these patients because there is no other organization capable
+
+95
+01:05:40,020 --> 01:06:23,460
+of maintaining these devices the news here is not all bad for machine learning there are researchers who are working at the intersection of medicine and machine learning and developing and proposing solutions to some of these issues that I think might have broad applicability on building responsibly with machine learning first the clinical trial standards that are used for other medical devices and for pharmaceuticals have been extended to machine learning the spirit standard for Designing clinical trials and the consort standard for reporting results of clinical trials these have both been extended to include ml with Spirit Ai and consort AI two Links at the bottom of this slide for the details on the contents of both of
+
+96
+01:06:21,900 --> 01:07:07,380
+those standards one thing I wanted to highlight here was the process by which these standards were created and which is reported in those research articles which included an international survey with over a hundred participants and then a conference with 30 participants to come up with a final checklist and then a pilot use of it to determine how well it worked so the standard for producing standards in medicine is also quite high and something we could very much learn from in machine learning so because of that work and because people have pointed out these concerns progress is being made on doing better work in machine learning for medicine this recent paper in the Journal of the American Medical Association does a
+
+97
+01:07:05,400 --> 01:07:52,680
+review of clinical trials involving machine learning and finds that for many of the components of these clinical trial standards compliance and quality is very high incorporating clinical context state very clearly how the method will contribute to clinical care but there are definitely some places with poor compliance for example interestingly enough very few trials reported how low quality data was handled how data was assessed for quality and and how cases of poor quality data should be handled I think that's also something that the broader machine learning world could do a better job on and then also analysis of errors that models made which also shows up in medical research and clinical trials as analysis of Adverse Events this kind of
+
+98
+01:07:50,460 --> 01:08:34,199
+error analysis was not commonly done and this is something that in talking about testing and troubleshooting and in talking about model monitoring and continual learning we've tried to emphasize the importance of this kind of error analysis for building with ML there's also this really gorgeous pair of papers by Lauren Oakton Raynor and others in the Lancet that both developed and applied this algorithmic auditing framework for medical ml so this is something that is probably easier to incorporate into other ml workflows than is a full-on clinical trial approach but still has some of the same rigor incorporates checklists and tasks and defined artifacts that highlight what the problems are and what needs to be
+
+99
+01:08:31,980 --> 01:09:11,940
+tracked and shared while building a machine Learning System one particular component that I wanted to highlight and is here indicated in blue is that there's a big emphasis on failure modes and error analysis and what they call adversarial testing which is coming up with different kinds of inputs to put into the model to see how it performs so sort of like a behavioral check on the model these are all things that we've emphasized as part of how to build a model well there's lots of other components of this audit that the broader ml Community would do well to incorporate into their work there's a ton of really great work being done a lot of these papers are just within the last three or six months so I think it's
+
+100
+01:09:10,620 --> 01:09:51,779
+a pretty good idea to keep your finger on the pulse here so to speak in medical ml the Stanford in Institute for AI and medicine has a regular panel that gets posted on YouTube they also share a lot of great other kinds of content via Twitter and then a lot of the researchers who did some of the work that I shared Lauren Oakton Raynor Benjamin Khan are also active on Twitter along with other folks who've done great work that I didn't get time to talk about like Judy chichoya and Matt Lundgren closing out this section like medicine machine learning can be very intimately intertwined with people's lives and so ethics is really really Salient perhaps the most important ethical question to ask ourselves over
+
+101
+01:09:48,540 --> 01:10:33,360
+and over again is should this system be built at all what are the implications of building the system of automating this task or this work and it seems clear that if we don't regulate ourselves we will end up being regulated and so we should learn from older Industries like medicine rather than just assuming we can disrupt our way through so as our final section I want to talk about the ethics of artificial intelligence this is clearly a frontier both for the field of Ethics trying to think through these problems and for the technology communities that are building this I think that right now false claims and hype around artificial intelligence are the most pressing concern but we shouldn't sleep on some of the major
+
+102
+01:10:31,560 --> 01:11:18,540
+ethical issues that are potentially oncoming with AI so right now claims and Hyperbole and hype around artificial intelligence are outpacing capabilities even though those capabilities are also growing fast and this risks a kind of blowback so one way to summarize this is say that if you call something autopilot people are going to treat it like autopilot and then be upset or worse when that's not the case so famously there is an incident where somebody who believed that Tesla is lean and braking assistant system autopilot was really full self-driving was killed in a car crash in this gap between what people expect out of ml systems and what they actually get is something that Josh talked about in the project management
+
+103
+01:11:16,800 --> 01:12:02,219
+lecture so this is something that we're already having to incorporate into our engineering and our product design that people are overselling the capacities of ml systems in a way that gives users a bad idea of what is possible and this problem is very widespread even large and mature organizations like IBM can create products like Watson which was the capable question and answering system and then sell it as artificial intelligence and try to revolutionize or disrupt Fields like medicine and then end up falling far short of these extremely lofty goals they've set themselves and along the way they get at least the beginning journalistic coverage with pictures of robot hands reaching out to grab balls of light or
+
+104
+01:12:00,300 --> 01:12:44,400
+brains inside computers or computers inside brains so not only do companies oversell what their technology can do but these overstatements are repeated or Amplified by traditional and social media and this problem even extends to Academia there is a Infamous now case where Japan in 2017 said that Radiologists at that point were like Wiley Coyote already over the edge of the cliff and haven't realized that there's no ground underneath them and that people should stop training Radiologists now because within five years AKA now deep learning is going to be better than Radiologists some of the work in the intersection of medicine in ml that I presented was done by people who were in their Radiology training at
+
+105
+01:12:42,659 --> 01:13:27,840
+the time around the time this statement was made and were lucky that they continued training as Radiologists while also gaining ml expertise so that they could do the slow hard work of bringing deep learning and machine learning into Radiology this overall problem of overselling artificial intelligence you could call AI snake oil so that's the name of an upcoming book and a new sub stack by Arvin Narayanan are now very good friend and so this refers not just to people overselling the capabilities of large language models or predicting that we'll have artificial intelligence by Christmas but people who use this General Aura of hypanic segment around artificial intelligence to sell shoddy technology an example from this really
+
+106
+01:13:25,440 --> 01:14:12,960
+great set of slides linked here the tool Elevate that claims to be able to assess personality and job suitability from a 30 second video including identifying whether the person in the video is a change agent or not so the call here is to separate out the actual places where there's been rapid Improvement in what's possible with machine learning for example computer perception identifying the contents of images face recognition Orion in here even includes medical diagnosis from scans from places where there's not been as much progress and so the split that he proposes that I think is helpful is that most things that involve some form of human judgment like determining whether something is hate speech or what grade an essay should
+
+107
+01:14:10,080 --> 01:14:51,179
+receive these are on the borderline most forms of prediction especially around what he calls social outcomes so things like policing jobs Child Development these are places where there has not been substantial progress and where the risk of somebody essentially riding the coattails of gpt3 with some technique that doesn't perform any better than linear regression is at its highest so we don't have artificial intelligence yet but if we do synthesize intelligent agents a lot of thorny ethical questions are going to immediately arise so it's probably a good idea as a field and as individuals for us to think a little bit about these ahead of time so there's broad agreement that creating sentient intelligent beings would have ethical
+
+108
+01:14:49,020 --> 01:15:38,580
+implications just this past summer Google engineer Blake Lemoine became convinced that a large language model built by Google Lambda was in fact conscious and almost everyone agrees that that's not the case for these large language models but there's pretty big disagreement on how far away we are and perhaps most importantly this concern did cause a pretty big reaction both inside the field and in the popular press in my view it's a bit unfortunate that this conversation was started so early because it's so easy to dismiss this claim if it happens too many more times we might end up inured to these kinds of conversations in a boy who cried AI type situation there's also a different set of concerns around what
+
+109
+01:15:36,719 --> 01:16:19,679
+might happen with the creation of a self-improving artificial intelligence so there's already some hints in this direction for one the latest Nvidia GPU architecture Hopper incorporates a very large number of AI design circuits pictured here on the left the quality of the AI design circuits are superior this is also something that's been reported by the folks working on tpus at Google there's also cases in which large language models can be used to build better models for example large language models can teach themselves to program better and large language models can also use large language models at least as well as humans this suggests the possibility of virtuous Cycles in machine learning capabilities and
+
+110
+01:16:17,820 --> 01:17:03,300
+machine intelligence and failing to pursue this kind of very powerful technology comes with a very substantial opportunity cost this is something that's argued by the philosopher Nick Bostrom in a famous paper called astronomical waste that points out just given the size of the universe the amount of resources and the amount of time it will be around there's a huge cost in terms of potential good potential lives worth living that we leave on the table if we do not develop the Necessary Technology quickly but the primary lesson that's drawn in this paper is actually not that technology should be developed as quickly as possible but rather that it should be developed as safely as possible which is to say that the probability that this
+
+111
+01:17:00,960 --> 01:17:48,540
+imagined Galaxy or Universe spanning Utopia comes into being that probability should be maximized and so this concern around safety originating the work of Bostrom and others has become a central concern for people thinking about the ethical implications of artificial intelligence and so the concerns around self-improving intelligent systems that could end up being more intelligent than humans are nicely summarized in the parable of the paperclip maximizer also from Bostrom at least popularized in the book super intelligence so the idea here is a classic example of this proxy problem in alignment so we design an artificial intelligence system for building paper clips so it's designed to make sure that the paper clip producing
+
+112
+01:17:46,260 --> 01:18:27,900
+component of our economy runs as effectively as possible produces as many paper clips as it can and we incorporate self-improvement into it so that it becomes smarter and more capable over time at first it improves human utility as it introduces better industrial processes for paper clips but as it becomes more intelligent perhaps it finds a way to manipulate the legal system and manipulate politics to introduce a more favorable tax code for pay-per-clip related Industries and that starts to hurt overall human utility uh even as the number of paper clips created and the capacity of the paperclip maximizer increases and of course at the point when we have mandatory national service in the paperclip mines or that all matter in
+
+113
+01:18:26,400 --> 01:19:14,280
+the universe is converted to paper clips we've pretty clearly decreased human utility as this paperclip maximizer has maximized its objective and increased its own capacity so this still feels fairly far away and a lot of the speculations feel a lot more like science fiction than science fact but the stakes here are high enough that it is certainly worth having some people thinking about and working on it and many of the techniques can be applied to controlled and responsible deployment of less capable ml systems as a small aside these ideas around existential risk and super intelligences are often associated with the effective altruism Community which is concerned with the best ways to do the most good both with what you do
+
+114
+01:19:11,880 --> 01:19:54,480
+with your career one of the focuses is the 80 000 hours organization and also through charitable donations as a way to by donating to the highest impact Charities and non-profits have the largest positive impact on the world so there's a lot of very interesting ideas coming out of this community and it's particularly appealing to a lot of folks who work in technology and especially in machine learning so it's worth checking out so that brings us to the end of our planned agenda here after giving some context around what our approach to Ethics in this lecture would look like we talked about ethical concerns in three different fields first past and immediate concerns around the ethical development of Technology then up and
+
+115
+01:19:52,260 --> 01:20:42,540
+coming and near future concerns around building ethically with machine learning and then finally a taste of the ethical concerns we might face in a future where machine learning gives way to artificial intelligence with a reminder that we should make sure not to oversell our progress on that front so I got to the end of these slides and realized that this was the end of the course and felt that I couldn't leave it on uh dower and sad note of unusable medical algorithms and existential risk from Super intelligences so I wanted to close out with a bit of a more positive note on the things that we can do so I think the first and most obvious step is education a lot of these ideas around ethics are unfamiliar to people with a technical
+
+116
+01:20:40,560 --> 01:21:27,179
+background there's a lot of great longer form content that captures a lot of these ideas and can help you build your own knowledge of the history and context and eventually your own opinions on these topics I can highly recommend each of these books the alignment problem is a great place to get started it focuses pretty tightly on ML ethics and AI ethics it covers a lot of recent research and is very easily digestible for an ml audience you might also want to consider some of these books around more Tech ethics like weapons of math destruction by Kathy O'Neill and automating inequality by Virginia Eubanks from there you can prioritize things that you want to act on make your own two by two around things that have
+
+117
+01:21:24,420 --> 01:22:10,980
+impact now and can have very high impact for me I think that's things around deceptive design and dark patterns and around AI snake oil then there's also places where acting in the future might be very important and high impact for me I think that's things around ml Weaponry behind my head is existential risk from Super intelligences on super high impact but something that we can't act on right now and then all the things in between you can create your own two by two on these and then search around for organizations communities and people working on these problems to align yourself with and by way of a final goodbye as we're ending this class I want to call out that a lot of the discussion of Ethics in this lecture was
+
+118
+01:22:09,420 --> 01:22:53,040
+very negative because of the framing around cases where people raised ethical concerns but ethics is not and cannot be purely negative about avoiding doing bad things the work that we do in Building Technology with machine learning can do good in the world not just avoid doing harm we can reduce suffering so this diagram here from a neuroscience from a brain machine interface paper from 2012 is what got me into the field of machine learning in the first place it shows a tetraplegic woman who has learned to control a robot arm using only other thoughts by means of an electrode attached to her head and while the technical achievements in this paper were certainly very impressive the thing that made the strongest impression on me
+
+119
+01:22:50,460 --> 01:23:33,000
+reading this paper in college was the smile on the woman's face in the final panel if you've experienced this kind of limit Mobility either yourself or in someone close to you then you know that the joy even from something as simple as being able to feed yourself is very real we can also do good by increasing Joy not just reducing suffering despite the concerns that we talked about with text to image models there they're clearly being used to create Beauty and Joy or as Ted Underwood a digital Humanity scholar put it to explore a dimension of human culture that was accidentally created across the last five thousand years of captioning that's beautiful and it's something we should hold on to that's not to say that this happens
+
+120
+01:23:30,000 --> 01:24:17,040
+automatically by Building Technology the world automatically becomes better but leading organizations in our field are making proactive statements on this openai around long term safely around long-term safety and Broad distribution of the benefits of machine learning and artificial intelligence research Deep Mind stating which Technologies they won't pursue and making a clear statement of a gold a broadly benefit Humanity the final bit of really great news that I have is that the tools for building ml well that you've learned throughout this class align very well with building ml for good so we saw it with the medical machine learning around failure analysis and we can also see it in the principles for for responsible
+
+121
+01:24:15,060 --> 01:25:03,199
+development from these leading organizations Deep Mind mentioning accountability to people and Gathering feedback Google AI mentioning it as well and if you look closely at Google ai's list of recommended practices for responsible AI use multiple metrics to assess training and monitoring understand limitations use tests directly examine raw data Monitor and update your system after deployment these are exactly the same principles that we've been emphasizing in this course around building ml powered products the right way these techniques will also help you build machine learning that does what's right and so on that note I want to thank you for your time and your interest in this course and I wish you the best of luck as you
+
+122
+01:24:59,940 --> 01:25:03,199
+go out to build with ML
+