{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8ec2fef2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 4: Software Engineering Applied to LLMs\n",
    "* **Created by:** Eric Martinez\n",
    "* **For:** Software Engineering 2\n",
    "* **At:** University of Texas Rio-Grande Valley"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60fef658",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Quality and Performance Issues\n",
    "\n",
    "* Applications depend on external APIs which has issues with flakiness and pricing, how do we avoid hitting APIs in testing?\n",
    "* Responses may not be correct or accurate, how do we increase confidence in result?\n",
    "* Responses may be biased or unethical or unwanted output, how do we stop this type of output?\n",
    "* User requests could be unethical or unwanted input, how do we filter this type of input?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fc1b19a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Prototyping\n",
    "* Develop prompt prototypes early when working with customers or stakeholders, it is fast and cheap to test that the idea will work.\n",
    "* Test against realistic examples, early. Fail fast and iterate quickly.\n",
    "* Make a plan for how you will source dynamic data. If there is no path, the project is dead in the water."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2528a3c9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Testing\n",
    "* Unit test prompts using traditional methods to increase confidence.\n",
    "* Unit test your prompts using LLMs to increase confidence.\n",
    "* Write tests that handle API errors or bad output (malformed, incorrect, unethical).\n",
    "* Use 'mocking' in integration tests to avoid unnecessary calls to APIs, flakiness, and unwanted charges."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9cdafd2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Handling Bad Output\n",
    "* Develop 'retry' mechanisms when you get unwanted output.\n",
    "* Develop specific prompts for different 'retry' conditions. Include the context, what went wrong, and what needs to be fixed.\n",
    "* Consider adding logging to your app to keep track of how often your app gets bad output."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f7de0be",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Template Languages and Version Control\n",
    "* Consider writing your prompt templates in dynamic template languages like ERB, Handlebars, etc.\n",
    "* Keep prompt templates and prompts in version control in your app's repo.\n",
    "* Write tests for handling template engine errors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3987a54c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Prompt Injection/Leakage\n",
    "* User-facing prompts should be tested against prompt injection attacks\n",
    "* Validate input at the UI and LLM level\n",
    "* Consider using an LLM to check if an output is similar to the prompt\n",
    "* Have mechanisms for anomaly detection and incident response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0e0c388",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Security\n",
    "* **Do not:** store API keys in application code as strings, encrypted or not.\n",
    "* **Do not:** store API keys in compiled binaries distributed to users.\n",
    "* **Do not:** store API keys in metadeta files bundled with your application.\n",
    "* **Do:** create an intermediate web app (or API) with authentication/authorization that delegates requests to LLMs at run-time for use in front-end applications\n",
    "* **Do:** if your front-end application does not have user accounts, consider implementing guest or anonymous accounts and expiring or rotating keys\n",
    "* **Do:** when allowing LLMs to use tools, consider designing systems to pass-through user ids to tools so that they tools operate at the same level of access as the end-user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "181dd4ad",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}