Large language models (LLMs), especially the instruction-tuned ones, can generate human-like responses to chat prompts. Using Zeus for energy measurement, we created a leaderboard for LLM chat energy consumption.
More models will be added over time. Stay tuned!
""" return text def get_detail_text(self, detail_mode: bool) -> str: if detail_mode: text = """ **TPOT (Time Per Output Token)** is the time between each token generated by LLMs as part of their response. An average TPOT of 0.20 seconds roughly corresponds to a person reading at 240 words per minute and assuming one word is 1.3 tokens on average. You can tweak the TPOT slider to adjust the target average TPOT for the models. Each row corresponds to one model, given a constraint on the maximum average TPOT. If more than one GPU types were chosen, the row shows results from the GPU with the lowest energy consumption per request. Columns - **Model**: The name of the model. - **Params (B)**: Number of parameters in the model. - **GPU**: Name of the GPU model used for benchmarking. - **TP**: Tensor parallelism degree. - **PP**: Pipeline parallelism degree. (TP * PP is the total number of GPUs used.) - **Energy/req (J)**: Energy consumed per request in Joules. - **Avg TPOT (s)**: Average time per output token in seconds. - **Token tput (toks/s)**: Average number of tokens generated by the engine per second. - **Avg Output Tokens**: Average number of output tokens in the LLM's response. - **Avg BS**: Average batch size of the serving engine over time. - **Max BS**: Maximum batch size configuration of the serving engine. For more detailed information, please take a look at the **About** tab. """ else: text = """ Columns - **Model**: The name of the model. - **Parameters (Billions)**: Number of parameters in the model. This is the size of the model. - **GPU model**: Name of the GPU model used for benchmarking. - **Energy per response (Joules)**: Energy consumed for each LLM response in Joules. Checking "Show more technical details" above the table will reveal more detailed columns. Also, for more detailed information, please take a look at the **About** tab. """ return text class LLMCodeTableManager(LLMTableManager): """LLM table manager for coding tasks.""" def get_tab_name(self) -> str: return "LLM Code" def get_intro_text(self) -> str: text = """Large language models (LLMs) are also capable of generating code. Using Zeus for energy measurement, we created a leaderboard for the energy consumption of LLMs specifically trained for code generation.
More models will be added over time. Stay tuned!
""" return text def get_detail_text(self, detail_mode: bool) -> str: if detail_mode: text = """ **TPOT (Time Per Output Token)** is the time between each token generated by LLMs as part of their response. An average TPOT of 0.20 seconds roughly corresponds to a person reading at 240 words per minute and assuming one word is 1.3 tokens on average. You can tweak the TPOT slider to adjust the target average TPOT for the models. Each row corresponds to one model, given a constraint on the maximum average TPOT. If more than one GPU types were chosen, the row shows results from the GPU with the lowest energy consumption per request. Columns - **Model**: The name of the model. - **Params (B)**: Number of parameters in the model. - **GPU**: Name of the GPU model used for benchmarking. - **TP**: Tensor parallelism degree. - **PP**: Pipeline parallelism degree. (TP * PP is the total number of GPUs used.) - **Energy/req (J)**: Energy consumed per request in Joules. - **Avg TPOT (s)**: Average time per output token in seconds. - **Token tput (toks/s)**: Average number of tokens generated by the engine per second. - **Avg Output Tokens**: Average number of output tokens in the LLM's response. - **Avg BS**: Average batch size of the serving engine over time. - **Max BS**: Maximum batch size configuration of the serving engine. For more detailed information, please take a look at the **About** tab. """ else: text = """ Columns - **Model**: The name of the model. - **Parameters (Billions)**: Number of parameters in the model. This is the size of the model. - **GPU model**: Name of the GPU model used for benchmarking. - **Energy per response (Joules)**: Energy consumed for each LLM response in Joules. Checking "Show more technical details" above the table will reveal more detailed columns. Also, for more detailed information, please take a look at the **About** tab. """ return text class VLMChatTableManager(LLMTableManager): """VLM table manager for chat tasks.""" def get_tab_name(self) -> str: return "VLM Visual Chat" def get_intro_text(self) -> str: text = """Vision language models (VLMs) are large language models that can understand images along with text and generate human-like responses to chat prompts with images. Using Zeus for energy measurement, we created a leaderboard for VLM chat energy consumption.
More models will be added over time. Stay tuned!
""" return text def get_detail_text(self, detail_mode: bool) -> str: if detail_mode: text = """ **TPOT (Time Per Output Token)** is the time between each token generated by LLMs as part of their response. An average TPOT of 0.20 seconds roughly corresponds to a person reading at 240 words per minute and assuming one word is 1.3 tokens on average. You can tweak the TPOT slider to adjust the target average TPOT for the models. Each row corresponds to one model, given a constraint on the maximum average TPOT. If more than one GPU types were chosen, the row shows results from the GPU with the lowest energy consumption per request. Columns - **Model**: The name of the model. - **Params (B)**: Number of parameters in the model. - **GPU**: Name of the GPU model used for benchmarking. - **TP**: Tensor parallelism degree. - **PP**: Pipeline parallelism degree. (TP * PP is the total number of GPUs used.) - **Energy/req (J)**: Energy consumed per request in Joules. - **Avg TPOT (s)**: Average time per output token in seconds. - **Token tput (toks/s)**: Average number of tokens generated by the engine per second. - **Avg Output Tokens**: Average number of output tokens in the LLM's response. - **Avg BS**: Average batch size of the serving engine over time. - **Max BS**: Maximum batch size configuration of the serving engine. For more detailed information, please take a look at the **About** tab. """ else: text = """ Columns - **Model**: The name of the model. - **Parameters (Billions)**: Number of parameters in the model. This is the size of the model. - **GPU model**: Name of the GPU model used for benchmarking. - **Energy per response (Joules)**: Energy consumed for each LLM response in Joules. Checking "Show more technical details" above the table will reveal more detailed columns. Also, for more detailed information, please take a look at the **About** tab. """ return text class DiffusionTableManager(TableManager): def __init__(self, data_dir: str, task_name: str) -> None: """Load leaderboard data from files in `data_dir`. Under `data_dir`, there should be: - `models.json`: JSON file that maps huggingface model IDs to model info. Some models listed in this file may not have benchmark results. - `schema.yaml`: YAML file containing the schema of the benchmark. Then, benchmark data files are nested under `data_dir` according to the schema. One directory hierarchy for each choice in the schema and then two more -- the model's HuggingFace hub organization and the model name. """ super().__init__(data_dir) self.task_name = task_name if "to video" in task_name.lower(): self.energy_col = "Energy/video (J)" self.energy_col_readable = "Energy per video (Joules)" elif "to image" in task_name.lower(): self.energy_col = "Energy/image (J)" self.energy_col_readable = "Energy per image (Joules)" else: raise ValueError(f"Unknown task name: {task_name=}") # Read in the data into a Pandas DataFrame. # Important: The ordering `self.schema` determines the directory structure. self.schema = yaml.safe_load(open(self.data_dir / "schema.yaml")) models: dict[str, dict[str, Any]] = json.load( open(self.data_dir / "models.json") ) res_df = pd.DataFrame() for choice in itertools.product(*self.schema.values()): result_dir = self.data_dir / "/".join(choice) with contextlib.suppress(FileNotFoundError): for model_id, model_info in models.items(): for file in (result_dir / model_id).glob("*.json"): model_df = pd.DataFrame([json.load(open(file))]) # Sanity checks and standardization of schema values. assert model_df["Model"].iloc[0] == model_id for key, val in zip(self.schema.keys(), choice): assert ( str(val).lower() in str(model_df[key].iloc[0]).lower() ) model_df[key] = val # Format the model name as an HTML anchor. model_df["Model"] = self._wrap_model_name(model_info["url"], model_info["nickname"]) model_df["Total params"] = model_info["total_params"] model_df["Denoising params"] = model_info["denoising_params"] model_df["Resolution"] = model_info["resolution"] res_df = pd.concat([res_df, model_df]) if res_df.empty: raise ValueError( f"No benchmark JSON files were read from {self.data_dir=}." ) # Order columns columns = res_df.columns.to_list() cols_to_order = ["Model", "Denoising params", "Total params"] cols_to_order.extend(self.schema.keys()) columns = cols_to_order + [col for col in columns if col not in cols_to_order] res_df = res_df[columns] # Order rows res_df = res_df.sort_values(by=["Model", *self.schema.keys(), self.energy_col]) self.full_df = res_df.round(2) # We need to set the default view separately when `gr.State` is forked. self.set_filter_get_df(detail_mode=False) def get_benchmark_checkboxes(self) -> dict[str, list[str]]: return self.schema def get_all_models(self) -> list[str]: return self.full_df["Model"].apply(self._unwrap_model_name).unique().tolist() def set_filter_get_df(self, detail_mode: bool, *filters) -> pd.DataFrame: """Set the current set of filters and return the filtered DataFrame. Filters can either be completely empty, or be a concatenated list of choices from all checkboxes and all sliders. """ # If the filter is empty, we default to the first choice for each key. if not filters: checkboxes = [choices[:1] for choices in self.schema.values()] sliders = [slider[3] for slider in self.get_benchmark_sliders().values()] filters = checkboxes + sliders index = np.full(len(self.full_df), True) # Checkboxes for setup, choice in zip(self.schema, filters): index = index & self.full_df[setup].isin(choice) cur_df = self.full_df.loc[index] # Sliders (We just have Batch latency for now.) # For each `Model`, we want to first filter out rows whose `Batch latency (s)` is greater than the slider value. # Finally, only just leave the row whose `Energy/image (J)` or `Energy/video (J)` is the smallest. batch_latency = filters[-1] cur_df = ( cur_df .groupby("Model")[cur_df.columns] .apply( lambda x: x[x["Batch latency (s)"] <= batch_latency], include_groups=True, ) .sort_values(by=self.energy_col) .reset_index(drop=True) .groupby("Model") .head(1) ) if not detail_mode: core_columns = ["Model", "Denoising params", "GPU", "Resolution", "Frames", self.energy_col] readable_name_mapping = { "Denoising params": "Denoising parameters (Billions)", "GPU": "GPU model", self.energy_col: self.energy_col_readable, } for column in cur_df.columns: if column not in core_columns: cur_df = cur_df.drop(column, axis=1) cur_df = cur_df.rename(columns=readable_name_mapping) return cur_df class DiffusionT2ITableManager(DiffusionTableManager): """Diffusion table manager for text-to-image tasks.""" def get_tab_name(self) -> str: return "Diffusion Text to image" def get_intro_text(self) -> str: text = """Diffusion models generate images that align with input text prompts. Using Zeus for energy measurement, we created a leaderboard for the energy consumption of Diffusion text-to-image.
More models will be added over time. Stay tuned!
""" return text def get_detail_text(self, detail_mode: bool) -> str: if detail_mode: text = """ Each row corresponds to one model, given a constraint on the maximum computation time for the whole batch. If more than one GPU types were chosen, the row shows results from the GPU with the lowest energy consumption per image. Columns - **Model**: The name of the model. - **Denoising params**: Number of parameters in the denosing module (e.g., UNet, Transformer). - **Total params**: Total number of parameters in the model, including encoders and decoders. - **GPU**: Name of the GPU model used for benchmarking. - **Energy/image (J)**: Energy consumed per generated image in Joules. - **Batch latency (s)**: Time taken to generate a batch of images in seconds. - **Batch size**: Number of prompts/images in a batch. - **Denoising steps**: Number of denoising steps used for the diffusion model. - **Resolution**: Resolution of the generated image. For more detailed information, please take a look at the **About** tab. """ else: text = """ Columns - **Model**: The name of the model. - **Denoising parameters (Billions)**: Number of parameters in the diffusion model's (core) denoising module. This part of the model is run repetitively to generate gradually refine the image. - **GPU model**: Name of the GPU model used for benchmarking. - **Energy per image (Joules)**: Energy consumed for each generated image in Joules. - **Resolution**: Resolution of the generated image. Checking "Show more technical details" above the table will reveal more detailed columns. Also, for more detailed information, please take a look at the **About** tab. """ return text def get_benchmark_sliders(self) -> dict[str, tuple[float, float, float, float]]: return {"Batch latency (s)": (0.0, 60.0, 1.0, 10.0)} class DiffusionT2VTableManager(DiffusionTableManager): """Diffusion table manager for text-to-video tasks.""" def get_tab_name(self) -> str: return "Diffusion Text to video" def get_intro_text(self) -> str: text = """Diffusion models generate videos that align with input text prompts. Using Zeus for energy measurement, we created a leaderboard for the energy consumption of Diffusion text-to-video.
More models will be added over time. Stay tuned!
""" return text def get_detail_text(self, detail_mode: bool) -> str: if detail_mode: text = """ Each row corresponds to one model, given a constraint on the maximum computation time for the whole batch. If more than one GPU types were chosen, the row shows results from the GPU with the lowest energy consumption per video. Columns - **Model**: The name of the model. - **Denoising params**: Number of parameters in the denosing module (e.g., UNet, Transformer). - **Total params**: Total number of parameters in the model, including encoders and decoders. - **GPU**: Name of the GPU model used for benchmarking. - **Energy/video (J)**: Energy consumed per generated video in Joules. - **Batch latency (s)**: Time taken to generate a batch of videos in seconds. - **Batch size**: Number of prompts/videos in a batch. - **Denoising steps**: Number of denoising steps used for the diffusion model. - **Frames**: Number of frames in the generated video. - **Resolution**: Resolution of the generated video. For more detailed information, please take a look at the **About** tab. """ else: text = """ Columns - **Model**: The name of the model. - **Denoising parameters (Billions)**: Number of parameters in the diffusion model's (core) denoising module. This part of the model is run repetitively to generate gradually refine the video. - **GPU model**: Name of the GPU model used for benchmarking. - **Energy per video (Joules)**: Energy consumed for each generated image in Joules. - **Frames**: Number of frames in the generated video. - **Resolution**: Resolution of the generated video. Checking "Show more technical details" above the table will reveal more detailed columns. Also, for more detailed information, please take a look at the **About** tab. """ return text def get_benchmark_sliders(self) -> dict[str, tuple[float, float, float, float]]: return {"Batch latency (s)": (0.0, 60.0, 1.0, 10.0)} class DiffusionI2VTableManager(DiffusionTableManager): """Diffusion table manager for image-to-video tasks.""" def get_tab_name(self) -> str: return "Diffusion Image to video" def get_intro_text(self) -> str: text = """Diffusion models generate videos given an input image (and sometimes alongside with text). Using Zeus for energy measurement, we created a leaderboard for the energy consumption of Diffusion image-to-video.
More models will be added over time. Stay tuned!
""" return text def get_detail_text(self, detail_mode: bool) -> str: if detail_mode: text = """ Each row corresponds to one model, given a constraint on the maximum computation time for the whole batch. If more than one GPU types were chosen, the row shows results from the GPU with the lowest energy consumption per video. Columns - **Model**: The name of the model. - **Denoising params**: Number of parameters in the denosing module (e.g., UNet, Transformer). - **Total params**: Total number of parameters in the model, including encoders and decoders. - **GPU**: Name of the GPU model used for benchmarking. - **Energy/video (J)**: Energy consumed per generated video in Joules. - **Batch latency (s)**: Time taken to generate a batch of videos in seconds. - **Batch size**: Number of prompts/videos in a batch. - **Denoising steps**: Number of denoising steps used for the diffusion model. - **Frames**: Number of frames in the generated video. - **Resolution**: Resolution of the generated video. For more detailed information, please take a look at the **About** tab. """ else: text = """ Columns - **Model**: The name of the model. - **Denoising parameters (Billions)**: Number of parameters in the diffusion model's (core) denoising module. This part of the model is run repetitively to generate gradually refine the video. - **GPU model**: Name of the GPU model used for benchmarking. - **Energy per video (Joules)**: Energy consumed for each generated image in Joules. - **Frames**: Number of frames in the generated video. - **Resolution**: Resolution of the generated video. Checking "Show more technical details" above the table will reveal more detailed columns. Also, for more detailed information, please take a look at the **About** tab. """ return text def get_benchmark_sliders(self) -> dict[str, tuple[float, float, float, float]]: return {"Batch latency (s)": (0.0, 120.0, 1.0, 60.0)} class LegacyTableManager: def __init__(self, data_dir: str) -> None: """Load the legacy LLM leaderboard data from CSV files in data_dir. Inside `data_dir`, there should be: - `models.json`: a JSON file containing information about each model. - `schema.yaml`: a YAML file containing the schema of the benchmark. - `score.csv`: a CSV file containing the NLP evaluation metrics of each model. - `*_benchmark.csv`: CSV files containing the system benchmark results. Especially, the `*_benchmark.csv` files should be named after the parameters used in the benchmark. For example, for the CSV file that contains benchmarking results for A100 and the chat-concise task (see `schema.yaml`) for possible choices, the file should be named `A100_chat-concise_benchmark.csv`. """ # Load and merge CSV files. df = self._read_tables(data_dir) # Add the #params column. models = json.load(open(f"{data_dir}/models.json")) df["parameters"] = df["model"].apply(lambda x: models[x]["params"]) # Make the first column (model) an HTML anchor to the model's website. def format_model_link(model_name: str) -> str: url = models[model_name]["url"] nickname = models[model_name]["nickname"] return ( f'{nickname}' ) df["model"] = df["model"].apply(format_model_link) # Sort by our 'energy efficiency' score. df = df.sort_values(by="energy", ascending=True) # The full table where all the data are. self.full_df = df # Default view of the table is to only show the first options. self.set_filter_get_df() def _read_tables(self, data_dir: str) -> pd.DataFrame: """Read tables.""" df_score = pd.read_csv(f"{data_dir}/score.csv") with open(f"{data_dir}/schema.yaml") as file: self.schema: dict[str, list] = yaml.safe_load(file) res_df = pd.DataFrame() # Do a cartesian product of all the choices in the schema # and try to read the corresponding CSV files. for choice in itertools.product(*self.schema.values()): filepath = f"{data_dir}/{'_'.join(choice)}_benchmark.csv" with contextlib.suppress(FileNotFoundError): df = pd.read_csv(filepath) for key, val in zip(self.schema.keys(), choice): df.insert(1, key, val) res_df = pd.concat([res_df, df]) if res_df.empty: raise ValueError(f"No benchmark CSV files were read from {data_dir=}.") df = pd.merge(res_df, df_score, on=["model"]).round(2) # Order columns. columns = df.columns.to_list() cols_to_order = ["model"] cols_to_order.extend(self.schema.keys()) cols_to_order.append("energy") columns = cols_to_order + [col for col in columns if col not in cols_to_order] df = df[columns] # Delete rows with *any* NaN values. df = df.dropna() return df def _format_msg(self, text: str) -> str: """Formats into HTML that prints in Monospace font.""" return f"{text}"
def get_dropdown(self):
columns = self.full_df.columns.tolist()[1:]
return [
gr.Dropdown(choices=columns, value="parameters", label="X"),
gr.Dropdown(choices=columns, value="energy", label="Y"),
gr.Dropdown(choices=["None", *columns], label="Z (optional)"),
]
def update_dropdown(self):
columns = self.full_df.columns.tolist()[1:]
return [
gr.Dropdown.update(choices=columns),
gr.Dropdown.update(choices=columns),
gr.Dropdown.update(choices=["None", *columns]),
]
def set_filter_get_df(self, *filters) -> pd.DataFrame:
"""Set the current set of filters and return the filtered DataFrame."""
# If the filter is empty, we default to the first choice for each key.
if not filters:
filters = [choices[:1] for choices in self.schema.values()]
index = np.full(len(self.full_df), True)
for setup, choice in zip(self.schema, filters):
index = index & self.full_df[setup].isin(choice)
self.cur_df = self.full_df.loc[index]
self.cur_index = index
return self.cur_df
def get_intro_text(self) -> str:
"""Return the leaderboard's introduction text in HTML."""
return """
We used Zeus to benchmark various open source LLMs in terms of how much time and energy they consume for inference.
For more detailed information, please take a look at the About tab. Every benchmark is limited in some sense -- Before you interpret the results, please take a look at the Limitations section there, too.
""" # The global instance of the TableManager should only be used when # initializing components in the Gradio interface. If the global instance # is mutated while handling user sessions, the change will be reflected # in every user session. Instead, the instance provided by gr.State should # be used. global_ltbm = LegacyTableManager("data/legacy") global_tbms = [ LLMChatTableManager("data/llm_text_generation/chat", "Chat"), LLMCodeTableManager("data/llm_text_generation/code", "Code"), VLMChatTableManager("data/mllm_text_generation/chat", "Visual chat"), DiffusionT2ITableManager("data/diffusion/text-to-image", "Text to image"), DiffusionT2VTableManager("data/diffusion/text-to-video", "Text to video"), DiffusionI2VTableManager("data/diffusion/image-to-video", "Image to video"), ] # Custom JS. # XXX: This is a hack to make the model names clickable. # Ideally, we should set `datatype` in the constructor of `gr.DataFrame` to # `["markdown"] + ["number"] * (len(df.columns) - 1)` and format models names # as an HTML tag. However, because we also want to dynamically add new # columns to the table and Gradio < 4.0 does not support updating `datatype` with # `gr.DataFrame.update` yet, we need to manually walk into the DOM and replace # the innerHTML of the model name cells with dynamically interpreted HTML. # Desired feature tracked at https://github.com/gradio-app/gradio/issues/3732 dataframe_update_js = f""" function format_model_link() {{ // Iterate over the cells of the first column of the leaderboard table. var table_element = document.querySelectorAll(".tab-leaderboard"); for (var table of table_element) {{ for (let index = 1; index <= {len(global_ltbm.full_df) + sum(len(tbm.full_df) for tbm in global_tbms)}; index++) {{ // Get the cell from `table`. var cell = table.querySelector(`div > div > div > table > tbody > tr:nth-child(${{index}}) > td:nth-child(1) > div > span`); // var cell = document.querySelector( // `.tab-leaderboard > div > div > div > table > tbody > tr:nth-child(${{index}}) > td:nth-child(1) > div > span` // ); // If nothing was found, it likely means that now the visible table has less rows // than the full table. This happens when the user filters the table. In this case, // we should just return. if (cell == null) break; // This check exists to make this function idempotent. // Multiple changes to the Dataframe component may invoke this function, // multiple times to the same HTML table (e.g., adding and sorting cols). // Thus, we check whether we already formatted the model names by seeing // whether the child of the cell is a text node. If it is not, // it means we already parsed it into HTML, so we should just return. if (cell.firstChild.nodeType != 3) break; // Decode and interpret the innerHTML of the cell as HTML. var decoded_string = new DOMParser().parseFromString(cell.innerHTML, "text/html").documentElement.textContent; var temp = document.createElement("template"); temp.innerHTML = decoded_string; var model_anchor = temp.content.firstChild; // Replace the innerHTML of the cell with the interpreted HTML. cell.replaceChildren(model_anchor); }} }} // Return all arguments as is. return arguments }} """ # Custom CSS. custom_css = """ /* Make ML.ENERGY look like a clickable logo. */ .text-logo { color: #23d175 !important; text-decoration: none !important; } /* Make the submit button the same color as the logo. */ .btn-submit { background: #23d175 !important; color: white !important; border: 0 !important; } /* Center the plotly plot inside its container. */ .plotly > div { margin: auto !important; } /* Limit the width of the first column to 300 px. */ table td:first-child, table th:first-child { max-width: 300px; overflow: auto; white-space: nowrap; } /* Make tab buttons larger */ .tab-nav > button { font-size: 18px !important; } /* Color texts. */ .green-text { color: #23d175 !important; } .red-text { color: #ff3860 !important; } /* Flashing model name borders. */ @keyframes blink { 0%, 33%, 67%, 100% { border-color: transparent; } 17%, 50%, 83% { border-color: #23d175; } } /* Older browser compatibility */ @-webkit-keyframes blink { 0%, 33%, 67%, 100% { border-color: transparent; } 17%, 50%, 83% { border-color: #23d175; } } .model-name-text { border: 2px solid transparent; /* Transparent border initially */ animation: blink 3s ease-in-out 1; /* One complete cycle of animation, lasting 3 seconds */ -webkit-animation: blink 3s ease-in-out 1; /* Older browser compatibility */ } /* Grey out components when the Colosseum is down. */ .greyed-out { pointer-events: none; opacity: 0.4; } /* Make the Citation header larger */ #citation-header > div > span { font-size: 16px !important; } /* Align everything in tables to the right. */ /* Not the best solution, but at least makes the numbers align. */ .tab-leaderboard span { text-align: right; } """ # The app will not start without a controller address set. controller_addr = os.environ.get("COLOSSEUM_CONTROLLER_ADDR") if controller_addr is None: COLOSSEUM_UP = False COLOSSEUM_DOWN_MESSAGE = "