Upload folder using huggingface_hub

.gitignore

@@ -0,0 +1 @@
+**/__pycache__/

CONTRIBUTING.md

@@ -0,0 +1,106 @@
+*This guideline is very much a work-in-progress.*
+
+Contriubtions to `timm` for code, documentation, tests are more than welcome!
+
+There haven't been any formal guidelines to date so please bear with me, and feel free to add to this guide.
+
+# Coding style
+
+Code linting and auto-format (black) are not currently in place but open to consideration. In the meantime, the style to follow is (mostly) aligned with Google's guide: https://google.github.io/styleguide/pyguide.html. 
+
+A few specific differences from Google style (or black)
+1. Line length is 120 char. Going over is okay in some cases (e.g. I prefer not to break URL across lines).
+2. Hanging indents are always prefered, please avoid aligning arguments with closing brackets or braces.
+
+Example, from Google guide, but this is a NO here:
+```
+   # Aligned with opening delimiter.
+   foo = long_function_name(var_one, var_two,
+                            var_three, var_four)
+   meal = (spam,
+           beans)
+
+   # Aligned with opening delimiter in a dictionary.
+   foo = {
+       'long_dictionary_key': value1 +
+                              value2,
+       ...
+   }
+```
+This is YES:
+
+```
+   # 4-space hanging indent; nothing on first line,
+   # closing parenthesis on a new line.
+   foo = long_function_name(
+       var_one, var_two, var_three,
+       var_four
+   )
+   meal = (
+       spam,
+       beans,
+   )
+
+   # 4-space hanging indent in a dictionary.
+   foo = {
+       'long_dictionary_key':
+           long_dictionary_value,
+       ...
+   }
+```
+
+When there is descrepancy in a given source file (there are many origins for various bits of code and not all have been updated to what I consider current goal), please follow the style in a given file.
+
+In general, if you add new code, formatting it with black using the following options should result in a style that is compatible with the rest of the code base:
+
+```
+black --skip-string-normalization --line-length 120 <path-to-file>
+```
+
+Avoid formatting code that is unrelated to your PR though.
+
+PR with pure formatting / style fixes will be accepted but only in isolation from functional changes, best to ask before starting such a change.
+
+# Documentation
+
+As with code style, docstrings style based on the Google guide: guide: https://google.github.io/styleguide/pyguide.html
+
+The goal for the code is to eventually move to have all major functions and `__init__` methods use PEP484 type annotations.
+
+When type annotations are used for a function, as per the Google pyguide, they should **NOT** be duplicated in the docstrings, please leave annotations as the one source of truth re typing.
+
+There are a LOT of gaps in current documentation relative to the functionality in timm, please, document away!
+
+# Installation
+
+Create a Python virtual environment using Python 3.10. Inside the environment, install torch` and `torchvision` using the instructions matching your system as listed on the [PyTorch website](https://pytorch.org/).
+
+Then install the remaining dependencies:
+
+```
+python -m pip install -r requirements.txt
+python -m pip install -r requirements-dev.txt  # for testing
+python -m pip install -e .
+```
+
+## Unit tests
+
+Run the tests using:
+
+```
+pytest tests/
+```
+
+Since the whole test suite takes a lot of time to run locally (a few hours), you may want to select a subset of tests relating to the changes you made by using the `-k` option of [`pytest`](https://docs.pytest.org/en/7.1.x/example/markers.html#using-k-expr-to-select-tests-based-on-their-name). Moreover, running tests in parallel (in this example 4 processes) with the `-n` option may help:
+
+```
+pytest -k "substring-to-match" -n 4 tests/
+```
+
+## Building documentation
+
+Please refer to [this document](https://github.com/huggingface/pytorch-image-models/tree/main/hfdocs).
+
+# Questions
+
+If you have any questions about contribution, where / how to contribute, please ask in the [Discussions](https://github.com/huggingface/pytorch-image-models/discussions/categories/contributing) (there is a `Contributing` topic).

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "{}"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2019 Ross Wightman
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

MANIFEST.in

@@ -0,0 +1,3 @@
+include timm/models/_pruned/*.txt
+include timm/data/_info/*.txt
+include timm/data/_info/*.json

README.md

@@ -1,12 +1,785 @@
 ---
-title: Cs Mixer
-emoji: 📉
-colorFrom: red
-colorTo: green
+title: cs-mixer
+app_file: demo.py
 sdk: gradio
-sdk_version: 3.38.0
-app_file: app.py
-pinned: false
+sdk_version: 3.37.0
 ---
+# PyTorch Image Models
+- [Sponsors](#sponsors)
+- [What's New](#whats-new)
+- [Introduction](#introduction)
+- [Models](#models)
+- [Features](#features)
+- [Results](#results)
+- [Getting Started (Documentation)](#getting-started-documentation)
+- [Train, Validation, Inference Scripts](#train-validation-inference-scripts)
+- [Awesome PyTorch Resources](#awesome-pytorch-resources)
+- [Licenses](#licenses)
+- [Citing](#citing)
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+## Sponsors
+
+Thanks to the following for hardware support:
+* TPU Research Cloud (TRC) (https://sites.research.google/trc/about/)
+* Nvidia (https://www.nvidia.com/en-us/)
+
+And a big thanks to all GitHub sponsors who helped with some of my costs before I joined Hugging Face.
+
+## What's New
+
+❗Updates after Oct 10, 2022 are available in version >= 0.9❗
+* Many changes since the last 0.6.x stable releases. They were previewed in 0.8.x dev releases but not everyone transitioned.
+* `timm.models.layers` moved to `timm.layers`:
+  * `from timm.models.layers import name` will still work via deprecation mapping (but please transition to `timm.layers`).
+  * `import timm.models.layers.module` or `from timm.models.layers.module import name` needs to be changed now.
+* Builder, helper, non-model modules in `timm.models` have a `_` prefix added, ie `timm.models.helpers` -> `timm.models._helpers`, there are temporary deprecation mapping files but those will be removed.
+* All models now support `architecture.pretrained_tag` naming (ex `resnet50.rsb_a1`).
+  * The pretrained_tag is the specific weight variant (different head) for the architecture.
+  * Using only `architecture` defaults to the first weights in the default_cfgs for that model architecture.
+  * In adding pretrained tags, many model names that existed to differentiate were renamed to use the tag  (ex: `vit_base_patch16_224_in21k` -> `vit_base_patch16_224.augreg_in21k`). There are deprecation mappings for these.
+* A number of models had their checkpoints remaped to match architecture changes needed to better support `features_only=True`, there are `checkpoint_filter_fn` methods in any model module that was remapped. These can be passed to `timm.models.load_checkpoint(..., filter_fn=timm.models.swin_transformer_v2.checkpoint_filter_fn)` to remap your existing checkpoint.
+* The Hugging Face Hub (https://huggingface.co/timm) is now the primary source for `timm` weights. Model cards include link to papers, original source, license. 
+* Previous 0.6.x can be cloned from [0.6.x](https://github.com/rwightman/pytorch-image-models/tree/0.6.x) branch or installed via pip with version.
+
+### May 11, 2023
+* `timm` 0.9 released, transition from 0.8.xdev releases
+
+### May 10, 2023
+* Hugging Face Hub downloading is now default, 1132 models on https://huggingface.co/timm, 1163 weights in `timm`
+* DINOv2 vit feature backbone weights added thanks to [Leng Yue](https://github.com/leng-yue)
+* FB MAE vit feature backbone weights added
+* OpenCLIP DataComp-XL L/14 feat backbone weights added
+* MetaFormer (poolformer-v2, caformer, convformer, updated poolformer (v1)) w/ weights added by [Fredo Guan](https://github.com/fffffgggg54)
+* Experimental `get_intermediate_layers` function on vit/deit models for grabbing hidden states (inspired by DINO impl). This is WIP and may change significantly... feedback welcome.
+* Model creation throws error if `pretrained=True` and no weights exist (instead of continuing with random initialization)
+* Fix regression with inception / nasnet TF sourced weights with 1001 classes in original classifiers
+* bitsandbytes (https://github.com/TimDettmers/bitsandbytes) optimizers added to factory, use `bnb` prefix, ie `bnbadam8bit`
+* Misc cleanup and fixes
+* Final testing before switching to a 0.9 and bringing `timm` out of pre-release state
+
+### April 27, 2023
+* 97% of `timm` models uploaded to HF Hub and almost all updated to support multi-weight pretrained configs
+* Minor cleanup and refactoring of another batch of models as multi-weight added. More fused_attn (F.sdpa) and features_only support, and torchscript fixes.
+
+### April 21, 2023
+* Gradient accumulation support added to train script and tested (`--grad-accum-steps`), thanks [Taeksang Kim](https://github.com/voidbag)
+* More weights on HF Hub (cspnet, cait, volo, xcit, tresnet, hardcorenas, densenet, dpn, vovnet, xception_aligned)
+* Added `--head-init-scale` and `--head-init-bias` to train.py to scale classiifer head and set fixed bias for fine-tune
+* Remove all InplaceABN (`inplace_abn`) use, replaced use in tresnet with standard BatchNorm (modified weights accordingly). 
+
+### April 12, 2023
+* Add ONNX export script, validate script, helpers that I've had kicking around for along time. Tweak 'same' padding for better export w/ recent ONNX + pytorch.
+* Refactor dropout args for vit and vit-like models, separate drop_rate into `drop_rate` (classifier dropout), `proj_drop_rate` (block mlp / out projections), `pos_drop_rate` (position embedding drop), `attn_drop_rate` (attention dropout). Also add patch dropout (FLIP) to vit and eva models.
+* fused F.scaled_dot_product_attention support to more vit models, add env var (TIMM_FUSED_ATTN) to control, and config interface to enable/disable
+* Add EVA-CLIP backbones w/ image tower weights, all the way up to 4B param 'enormous' model, and 336x336 OpenAI ViT mode that was missed.
+
+### April 5, 2023
+* ALL ResNet models pushed to Hugging Face Hub with multi-weight support
+  * All past `timm` trained weights added with recipe based tags to differentiate
+  * All ResNet strikes back A1/A2/A3 (seed 0) and R50 example B/C1/C2/D weights available
+  * Add torchvision v2 recipe weights to existing torchvision originals
+  * See comparison table in https://huggingface.co/timm/seresnextaa101d_32x8d.sw_in12k_ft_in1k_288#model-comparison
+* New ImageNet-12k + ImageNet-1k fine-tunes available for a few anti-aliased ResNet models
+  * `resnetaa50d.sw_in12k_ft_in1k` - 81.7 @ 224, 82.6 @ 288
+  * `resnetaa101d.sw_in12k_ft_in1k` - 83.5 @ 224, 84.1 @ 288
+  * `seresnextaa101d_32x8d.sw_in12k_ft_in1k` - 86.0 @ 224, 86.5 @ 288 
+  * `seresnextaa101d_32x8d.sw_in12k_ft_in1k_288` - 86.5 @ 288, 86.7 @ 320
+
+### March 31, 2023
+* Add first ConvNext-XXLarge CLIP -> IN-1k fine-tune and IN-12k intermediate fine-tunes for convnext-base/large CLIP models.
+
+| model                                                                                                                |top1  |top5  |img_size|param_count|gmacs |macts |
+|----------------------------------------------------------------------------------------------------------------------|------|------|--------|-----------|------|------|
+| [convnext_xxlarge.clip_laion2b_soup_ft_in1k](https://huggingface.co/timm/convnext_xxlarge.clip_laion2b_soup_ft_in1k) |88.612|98.704|256     |846.47     |198.09|124.45|
+| convnext_large_mlp.clip_laion2b_soup_ft_in12k_in1k_384                                                               |88.312|98.578|384     |200.13     |101.11|126.74|
+| convnext_large_mlp.clip_laion2b_soup_ft_in12k_in1k_320                                                               |87.968|98.47 |320     |200.13     |70.21 |88.02 |
+| convnext_base.clip_laion2b_augreg_ft_in12k_in1k_384                                                                  |87.138|98.212|384     |88.59      |45.21 |84.49 |
+| convnext_base.clip_laion2b_augreg_ft_in12k_in1k                                                                      |86.344|97.97 |256     |88.59      |20.09 |37.55 |
+
+* Add EVA-02 MIM pretrained and fine-tuned weights, push to HF hub and update model cards for all EVA models. First model over 90% top-1 (99% top-5)! Check out the original code & weights at https://github.com/baaivision/EVA for more details on their work blending MIM, CLIP w/ many model, dataset, and train recipe tweaks.
+
+| model                                              |top1  |top5  |param_count|img_size|
+|----------------------------------------------------|------|------|-----------|--------|
+| [eva02_large_patch14_448.mim_m38m_ft_in22k_in1k](https://huggingface.co/timm/eva02_large_patch14_448.mim_m38m_ft_in1k) |90.054|99.042|305.08     |448     |
+| eva02_large_patch14_448.mim_in22k_ft_in22k_in1k    |89.946|99.01 |305.08     |448     |
+| eva_giant_patch14_560.m30m_ft_in22k_in1k           |89.792|98.992|1014.45    |560     |
+| eva02_large_patch14_448.mim_in22k_ft_in1k          |89.626|98.954|305.08     |448     |
+| eva02_large_patch14_448.mim_m38m_ft_in1k           |89.57 |98.918|305.08     |448     |
+| eva_giant_patch14_336.m30m_ft_in22k_in1k           |89.56 |98.956|1013.01    |336     |
+| eva_giant_patch14_336.clip_ft_in1k                 |89.466|98.82 |1013.01    |336     |
+| eva_large_patch14_336.in22k_ft_in22k_in1k          |89.214|98.854|304.53     |336     |
+| eva_giant_patch14_224.clip_ft_in1k                 |88.882|98.678|1012.56    |224     |
+| eva02_base_patch14_448.mim_in22k_ft_in22k_in1k     |88.692|98.722|87.12      |448     |
+| eva_large_patch14_336.in22k_ft_in1k                |88.652|98.722|304.53     |336     |
+| eva_large_patch14_196.in22k_ft_in22k_in1k          |88.592|98.656|304.14     |196     |
+| eva02_base_patch14_448.mim_in22k_ft_in1k           |88.23 |98.564|87.12      |448     |
+| eva_large_patch14_196.in22k_ft_in1k                |87.934|98.504|304.14     |196     |
+| eva02_small_patch14_336.mim_in22k_ft_in1k          |85.74 |97.614|22.13      |336     |
+| eva02_tiny_patch14_336.mim_in22k_ft_in1k           |80.658|95.524|5.76       |336     |
+
+* Multi-weight and HF hub for DeiT and MLP-Mixer based models
+
+### March 22, 2023
+* More weights pushed to HF hub along with multi-weight support, including: `regnet.py`, `rexnet.py`, `byobnet.py`, `resnetv2.py`, `swin_transformer.py`, `swin_transformer_v2.py`, `swin_transformer_v2_cr.py`
+* Swin Transformer models support feature extraction (NCHW feat maps for `swinv2_cr_*`, and NHWC for all others) and spatial embedding outputs.
+* FocalNet (from https://github.com/microsoft/FocalNet) models and weights added with significant refactoring, feature extraction, no fixed resolution / sizing constraint
+* RegNet weights increased with HF hub push, SWAG, SEER, and torchvision v2 weights. SEER is pretty poor wrt to performance for model size, but possibly useful.
+* More ImageNet-12k pretrained and 1k fine-tuned `timm` weights:
+  * `rexnetr_200.sw_in12k_ft_in1k` - 82.6 @ 224, 83.2 @ 288
+  * `rexnetr_300.sw_in12k_ft_in1k` - 84.0 @ 224, 84.5 @ 288
+  * `regnety_120.sw_in12k_ft_in1k` - 85.0 @ 224, 85.4 @ 288
+  * `regnety_160.lion_in12k_ft_in1k` - 85.6 @ 224, 86.0 @ 288
+  * `regnety_160.sw_in12k_ft_in1k` - 85.6 @ 224, 86.0 @ 288  (compare to SWAG PT + 1k FT this is same BUT much lower res, blows SEER FT away)
+* Model name deprecation + remapping functionality added (a milestone for bringing 0.8.x out of pre-release). Mappings being added...
+* Minor bug fixes and improvements.
+
+### Feb 26, 2023
+* Add ConvNeXt-XXLarge CLIP pretrained image tower weights for fine-tune & features (fine-tuning TBD) -- see [model card](https://huggingface.co/laion/CLIP-convnext_xxlarge-laion2B-s34B-b82K-augreg-soup)
+* Update `convnext_xxlarge` default LayerNorm eps to 1e-5 (for CLIP weights, improved stability)
+* 0.8.15dev0
+
+### Feb 20, 2023
+* Add 320x320 `convnext_large_mlp.clip_laion2b_ft_320` and `convnext_lage_mlp.clip_laion2b_ft_soup_320` CLIP image tower weights for features & fine-tune
+* 0.8.13dev0 pypi release for latest changes w/ move to huggingface org
+
+### Feb 16, 2023
+* `safetensor` checkpoint support added
+* Add ideas from 'Scaling Vision Transformers to 22 B. Params' (https://arxiv.org/abs/2302.05442) -- qk norm, RmsNorm, parallel block
+* Add F.scaled_dot_product_attention support (PyTorch 2.0 only) to `vit_*`, `vit_relpos*`, `coatnet` / `maxxvit` (to start)
+* Lion optimizer (w/ multi-tensor option) added (https://arxiv.org/abs/2302.06675)
+* gradient checkpointing works with `features_only=True`
+
+### Feb 7, 2023
+* New inference benchmark numbers added in [results](results/) folder.
+* Add convnext LAION CLIP trained weights and initial set of in1k fine-tunes
+  * `convnext_base.clip_laion2b_augreg_ft_in1k` - 86.2% @ 256x256
+  * `convnext_base.clip_laiona_augreg_ft_in1k_384` - 86.5% @ 384x384
+  * `convnext_large_mlp.clip_laion2b_augreg_ft_in1k` - 87.3% @ 256x256
+  * `convnext_large_mlp.clip_laion2b_augreg_ft_in1k_384` - 87.9% @ 384x384
+* Add DaViT models. Supports `features_only=True`. Adapted from https://github.com/dingmyu/davit by [Fredo](https://github.com/fffffgggg54).
+* Use a common NormMlpClassifierHead across MaxViT, ConvNeXt, DaViT
+* Add EfficientFormer-V2 model, update EfficientFormer, and refactor LeViT (closely related architectures). Weights on HF hub.
+  * New EfficientFormer-V2 arch, significant refactor from original at (https://github.com/snap-research/EfficientFormer). Supports `features_only=True`.
+  * Minor updates to EfficientFormer.
+  * Refactor LeViT models to stages, add `features_only=True` support to new `conv` variants, weight remap required.
+* Move ImageNet meta-data (synsets, indices) from `/results` to [`timm/data/_info`](timm/data/_info/).
+* Add ImageNetInfo / DatasetInfo classes to provide labelling for various ImageNet classifier layouts in `timm`
+  * Update `inference.py` to use, try: `python inference.py /folder/to/images --model convnext_small.in12k --label-type detail --topk 5`
+* Ready for 0.8.10 pypi pre-release (final testing).
+
+### Jan 20, 2023
+* Add two convnext 12k -> 1k fine-tunes at 384x384
+  * `convnext_tiny.in12k_ft_in1k_384` - 85.1 @ 384
+  * `convnext_small.in12k_ft_in1k_384` - 86.2 @ 384
+
+* Push all MaxxViT weights to HF hub, and add new ImageNet-12k -> 1k fine-tunes for `rw` base MaxViT and CoAtNet 1/2 models
+
+|model                                                                                                                   |top1 |top5 |samples / sec  |Params (M)     |GMAC  |Act (M)|
+|------------------------------------------------------------------------------------------------------------------------|----:|----:|--------------:|--------------:|-----:|------:|
+|[maxvit_xlarge_tf_512.in21k_ft_in1k](https://huggingface.co/timm/maxvit_xlarge_tf_512.in21k_ft_in1k)                    |88.53|98.64|          21.76|         475.77|534.14|1413.22|
+|[maxvit_xlarge_tf_384.in21k_ft_in1k](https://huggingface.co/timm/maxvit_xlarge_tf_384.in21k_ft_in1k)                    |88.32|98.54|          42.53|         475.32|292.78| 668.76|
+|[maxvit_base_tf_512.in21k_ft_in1k](https://huggingface.co/timm/maxvit_base_tf_512.in21k_ft_in1k)                        |88.20|98.53|          50.87|         119.88|138.02| 703.99|
+|[maxvit_large_tf_512.in21k_ft_in1k](https://huggingface.co/timm/maxvit_large_tf_512.in21k_ft_in1k)                      |88.04|98.40|          36.42|         212.33|244.75| 942.15|
+|[maxvit_large_tf_384.in21k_ft_in1k](https://huggingface.co/timm/maxvit_large_tf_384.in21k_ft_in1k)                      |87.98|98.56|          71.75|         212.03|132.55| 445.84|
+|[maxvit_base_tf_384.in21k_ft_in1k](https://huggingface.co/timm/maxvit_base_tf_384.in21k_ft_in1k)                        |87.92|98.54|         104.71|         119.65| 73.80| 332.90|
+|[maxvit_rmlp_base_rw_384.sw_in12k_ft_in1k](https://huggingface.co/timm/maxvit_rmlp_base_rw_384.sw_in12k_ft_in1k)        |87.81|98.37|         106.55|         116.14| 70.97| 318.95|
+|[maxxvitv2_rmlp_base_rw_384.sw_in12k_ft_in1k](https://huggingface.co/timm/maxxvitv2_rmlp_base_rw_384.sw_in12k_ft_in1k)  |87.47|98.37|         149.49|         116.09| 72.98| 213.74|
+|[coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k](https://huggingface.co/timm/coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k)            |87.39|98.31|         160.80|          73.88| 47.69| 209.43|
+|[maxvit_rmlp_base_rw_224.sw_in12k_ft_in1k](https://huggingface.co/timm/maxvit_rmlp_base_rw_224.sw_in12k_ft_in1k)        |86.89|98.02|         375.86|         116.14| 23.15|  92.64|
+|[maxxvitv2_rmlp_base_rw_224.sw_in12k_ft_in1k](https://huggingface.co/timm/maxxvitv2_rmlp_base_rw_224.sw_in12k_ft_in1k)  |86.64|98.02|         501.03|         116.09| 24.20|  62.77|
+|[maxvit_base_tf_512.in1k](https://huggingface.co/timm/maxvit_base_tf_512.in1k)                                          |86.60|97.92|          50.75|         119.88|138.02| 703.99|
+|[coatnet_2_rw_224.sw_in12k_ft_in1k](https://huggingface.co/timm/coatnet_2_rw_224.sw_in12k_ft_in1k)                      |86.57|97.89|         631.88|          73.87| 15.09|  49.22|
+|[maxvit_large_tf_512.in1k](https://huggingface.co/timm/maxvit_large_tf_512.in1k)                                        |86.52|97.88|          36.04|         212.33|244.75| 942.15|
+|[coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k](https://huggingface.co/timm/coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k)            |86.49|97.90|         620.58|          73.88| 15.18|  54.78|
+|[maxvit_base_tf_384.in1k](https://huggingface.co/timm/maxvit_base_tf_384.in1k)                                          |86.29|97.80|         101.09|         119.65| 73.80| 332.90|
+|[maxvit_large_tf_384.in1k](https://huggingface.co/timm/maxvit_large_tf_384.in1k)                                        |86.23|97.69|          70.56|         212.03|132.55| 445.84|
+|[maxvit_small_tf_512.in1k](https://huggingface.co/timm/maxvit_small_tf_512.in1k)                                        |86.10|97.76|          88.63|          69.13| 67.26| 383.77|
+|[maxvit_tiny_tf_512.in1k](https://huggingface.co/timm/maxvit_tiny_tf_512.in1k)                                          |85.67|97.58|         144.25|          31.05| 33.49| 257.59|
+|[maxvit_small_tf_384.in1k](https://huggingface.co/timm/maxvit_small_tf_384.in1k)                                        |85.54|97.46|         188.35|          69.02| 35.87| 183.65|
+|[maxvit_tiny_tf_384.in1k](https://huggingface.co/timm/maxvit_tiny_tf_384.in1k)                                          |85.11|97.38|         293.46|          30.98| 17.53| 123.42|
+|[maxvit_large_tf_224.in1k](https://huggingface.co/timm/maxvit_large_tf_224.in1k)                                        |84.93|96.97|         247.71|         211.79| 43.68| 127.35|
+|[coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k](https://huggingface.co/timm/coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k)          |84.90|96.96|        1025.45|          41.72|  8.11|  40.13|
+|[maxvit_base_tf_224.in1k](https://huggingface.co/timm/maxvit_base_tf_224.in1k)                                          |84.85|96.99|         358.25|         119.47| 24.04|  95.01|
+|[maxxvit_rmlp_small_rw_256.sw_in1k](https://huggingface.co/timm/maxxvit_rmlp_small_rw_256.sw_in1k)                      |84.63|97.06|         575.53|          66.01| 14.67|  58.38|
+|[coatnet_rmlp_2_rw_224.sw_in1k](https://huggingface.co/timm/coatnet_rmlp_2_rw_224.sw_in1k)                              |84.61|96.74|         625.81|          73.88| 15.18|  54.78|
+|[maxvit_rmlp_small_rw_224.sw_in1k](https://huggingface.co/timm/maxvit_rmlp_small_rw_224.sw_in1k)                        |84.49|96.76|         693.82|          64.90| 10.75|  49.30|
+|[maxvit_small_tf_224.in1k](https://huggingface.co/timm/maxvit_small_tf_224.in1k)                                        |84.43|96.83|         647.96|          68.93| 11.66|  53.17|
+|[maxvit_rmlp_tiny_rw_256.sw_in1k](https://huggingface.co/timm/maxvit_rmlp_tiny_rw_256.sw_in1k)                          |84.23|96.78|         807.21|          29.15|  6.77|  46.92|
+|[coatnet_1_rw_224.sw_in1k](https://huggingface.co/timm/coatnet_1_rw_224.sw_in1k)                                        |83.62|96.38|         989.59|          41.72|  8.04|  34.60|
+|[maxvit_tiny_rw_224.sw_in1k](https://huggingface.co/timm/maxvit_tiny_rw_224.sw_in1k)                                    |83.50|96.50|        1100.53|          29.06|  5.11|  33.11|
+|[maxvit_tiny_tf_224.in1k](https://huggingface.co/timm/maxvit_tiny_tf_224.in1k)                                          |83.41|96.59|        1004.94|          30.92|  5.60|  35.78|
+|[coatnet_rmlp_1_rw_224.sw_in1k](https://huggingface.co/timm/coatnet_rmlp_1_rw_224.sw_in1k)                              |83.36|96.45|        1093.03|          41.69|  7.85|  35.47|
+|[maxxvitv2_nano_rw_256.sw_in1k](https://huggingface.co/timm/maxxvitv2_nano_rw_256.sw_in1k)                              |83.11|96.33|        1276.88|          23.70|  6.26|  23.05|
+|[maxxvit_rmlp_nano_rw_256.sw_in1k](https://huggingface.co/timm/maxxvit_rmlp_nano_rw_256.sw_in1k)                        |83.03|96.34|        1341.24|          16.78|  4.37|  26.05|
+|[maxvit_rmlp_nano_rw_256.sw_in1k](https://huggingface.co/timm/maxvit_rmlp_nano_rw_256.sw_in1k)                          |82.96|96.26|        1283.24|          15.50|  4.47|  31.92|
+|[maxvit_nano_rw_256.sw_in1k](https://huggingface.co/timm/maxvit_nano_rw_256.sw_in1k)                                    |82.93|96.23|        1218.17|          15.45|  4.46|  30.28|
+|[coatnet_bn_0_rw_224.sw_in1k](https://huggingface.co/timm/coatnet_bn_0_rw_224.sw_in1k)                                  |82.39|96.19|        1600.14|          27.44|  4.67|  22.04|
+|[coatnet_0_rw_224.sw_in1k](https://huggingface.co/timm/coatnet_0_rw_224.sw_in1k)                                        |82.39|95.84|        1831.21|          27.44|  4.43|  18.73|
+|[coatnet_rmlp_nano_rw_224.sw_in1k](https://huggingface.co/timm/coatnet_rmlp_nano_rw_224.sw_in1k)                        |82.05|95.87|        2109.09|          15.15|  2.62|  20.34|
+|[coatnext_nano_rw_224.sw_in1k](https://huggingface.co/timm/coatnext_nano_rw_224.sw_in1k)                                |81.95|95.92|        2525.52|          14.70|  2.47|  12.80|
+|[coatnet_nano_rw_224.sw_in1k](https://huggingface.co/timm/coatnet_nano_rw_224.sw_in1k)                                  |81.70|95.64|        2344.52|          15.14|  2.41|  15.41|
+|[maxvit_rmlp_pico_rw_256.sw_in1k](https://huggingface.co/timm/maxvit_rmlp_pico_rw_256.sw_in1k)                          |80.53|95.21|        1594.71|           7.52|  1.85|  24.86|
+
+### Jan 11, 2023
+* Update ConvNeXt ImageNet-12k pretrain series w/ two new fine-tuned weights (and pre FT `.in12k` tags)
+  * `convnext_nano.in12k_ft_in1k` - 82.3 @ 224, 82.9 @ 288  (previously released)
+  * `convnext_tiny.in12k_ft_in1k` - 84.2 @ 224, 84.5 @ 288
+  * `convnext_small.in12k_ft_in1k` - 85.2 @ 224, 85.3 @ 288
+
+### Jan 6, 2023
+* Finally got around to adding `--model-kwargs` and `--opt-kwargs` to scripts to pass through rare args directly to model classes from cmd line
+  * `train.py /imagenet --model resnet50 --amp --model-kwargs output_stride=16 act_layer=silu`
+  * `train.py /imagenet --model vit_base_patch16_clip_224 --img-size 240 --amp --model-kwargs img_size=240 patch_size=12`
+* Cleanup some popular models to better support arg passthrough / merge with model configs, more to go.
+
+### Jan 5, 2023
+* ConvNeXt-V2 models and weights added to existing `convnext.py`
+  * Paper: [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](http://arxiv.org/abs/2301.00808)
+  * Reference impl: https://github.com/facebookresearch/ConvNeXt-V2 (NOTE: weights currently CC-BY-NC)
+
+### Dec 23, 2022 🎄☃
+* Add FlexiViT models and weights from https://github.com/google-research/big_vision (check out paper at https://arxiv.org/abs/2212.08013)
+  * NOTE currently resizing is static on model creation, on-the-fly dynamic / train patch size sampling is a WIP
+* Many more models updated to multi-weight and downloadable via HF hub now (convnext, efficientnet, mobilenet, vision_transformer*, beit)
+* More model pretrained tag and adjustments, some model names changed (working on deprecation translations, consider main branch DEV branch right now, use 0.6.x for stable use)
+* More ImageNet-12k (subset of 22k) pretrain models popping up:
+  * `efficientnet_b5.in12k_ft_in1k` - 85.9 @ 448x448
+  * `vit_medium_patch16_gap_384.in12k_ft_in1k` - 85.5 @ 384x384
+  * `vit_medium_patch16_gap_256.in12k_ft_in1k` - 84.5 @ 256x256
+  * `convnext_nano.in12k_ft_in1k` - 82.9 @ 288x288
+
+### Dec 8, 2022
+* Add 'EVA l' to `vision_transformer.py`, MAE style ViT-L/14 MIM pretrain w/ EVA-CLIP targets, FT on ImageNet-1k (w/ ImageNet-22k intermediate for some)
+  * original source: https://github.com/baaivision/EVA
+
+| model                                     | top1 | param_count |  gmac | macts | hub                                     |
+|:------------------------------------------|-----:|------------:|------:|------:|:----------------------------------------|
+| eva_large_patch14_336.in22k_ft_in22k_in1k | 89.2 |       304.5 | 191.1 | 270.2 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_large_patch14_336.in22k_ft_in1k       | 88.7 |       304.5 | 191.1 | 270.2 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_large_patch14_196.in22k_ft_in22k_in1k | 88.6 |       304.1 |  61.6 |  63.5 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_large_patch14_196.in22k_ft_in1k       | 87.9 |       304.1 |  61.6 |  63.5 | [link](https://huggingface.co/BAAI/EVA) |
+
+### Dec 6, 2022
+* Add 'EVA g', BEiT style ViT-g/14 model weights w/ both MIM pretrain and CLIP pretrain to `beit.py`.
+  * original source: https://github.com/baaivision/EVA
+  * paper: https://arxiv.org/abs/2211.07636
+
+| model                                    |   top1 |   param_count |   gmac |   macts | hub                                     |
+|:-----------------------------------------|-------:|--------------:|-------:|--------:|:----------------------------------------|
+| eva_giant_patch14_560.m30m_ft_in22k_in1k |   89.8 |        1014.4 | 1906.8 |  2577.2 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_giant_patch14_336.m30m_ft_in22k_in1k |   89.6 |        1013   |  620.6 |   550.7 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_giant_patch14_336.clip_ft_in1k       |   89.4 |        1013   |  620.6 |   550.7 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_giant_patch14_224.clip_ft_in1k       |   89.1 |        1012.6 |  267.2 |   192.6 | [link](https://huggingface.co/BAAI/EVA) |
+
+### Dec 5, 2022
+
+* Pre-release (`0.8.0dev0`) of multi-weight support (`model_arch.pretrained_tag`). Install with `pip install --pre timm`
+  * vision_transformer, maxvit, convnext are the first three model impl w/ support
+  * model names are changing with this (previous _21k, etc. fn will merge), still sorting out deprecation handling
+  * bugs are likely, but I need feedback so please try it out
+  * if stability is needed, please use 0.6.x pypi releases or clone from [0.6.x branch](https://github.com/rwightman/pytorch-image-models/tree/0.6.x)
+* Support for PyTorch 2.0 compile is added in train/validate/inference/benchmark, use `--torchcompile` argument
+* Inference script allows more control over output, select k for top-class index + prob json, csv or parquet output
+* Add a full set of fine-tuned CLIP image tower weights from both LAION-2B and original OpenAI CLIP models
+
+| model                                            |   top1 |   param_count |   gmac |   macts | hub                                                                                  |
+|:-------------------------------------------------|-------:|--------------:|-------:|--------:|:-------------------------------------------------------------------------------------|
+| vit_huge_patch14_clip_336.laion2b_ft_in12k_in1k  |   88.6 |         632.5 |  391   |   407.5 | [link](https://huggingface.co/timm/vit_huge_patch14_clip_336.laion2b_ft_in12k_in1k)  |
+| vit_large_patch14_clip_336.openai_ft_in12k_in1k  |   88.3 |         304.5 |  191.1 |   270.2 | [link](https://huggingface.co/timm/vit_large_patch14_clip_336.openai_ft_in12k_in1k)  |
+| vit_huge_patch14_clip_224.laion2b_ft_in12k_in1k  |   88.2 |         632   |  167.4 |   139.4 | [link](https://huggingface.co/timm/vit_huge_patch14_clip_224.laion2b_ft_in12k_in1k)  |
+| vit_large_patch14_clip_336.laion2b_ft_in12k_in1k |   88.2 |         304.5 |  191.1 |   270.2 | [link](https://huggingface.co/timm/vit_large_patch14_clip_336.laion2b_ft_in12k_in1k) |
+| vit_large_patch14_clip_224.openai_ft_in12k_in1k  |   88.2 |         304.2 |   81.1 |    88.8 | [link](https://huggingface.co/timm/vit_large_patch14_clip_224.openai_ft_in12k_in1k)  |
+| vit_large_patch14_clip_224.laion2b_ft_in12k_in1k |   87.9 |         304.2 |   81.1 |    88.8 | [link](https://huggingface.co/timm/vit_large_patch14_clip_224.laion2b_ft_in12k_in1k) |
+| vit_large_patch14_clip_224.openai_ft_in1k        |   87.9 |         304.2 |   81.1 |    88.8 | [link](https://huggingface.co/timm/vit_large_patch14_clip_224.openai_ft_in1k)        |
+| vit_large_patch14_clip_336.laion2b_ft_in1k       |   87.9 |         304.5 |  191.1 |   270.2 | [link](https://huggingface.co/timm/vit_large_patch14_clip_336.laion2b_ft_in1k)       |
+| vit_huge_patch14_clip_224.laion2b_ft_in1k        |   87.6 |         632   |  167.4 |   139.4 | [link](https://huggingface.co/timm/vit_huge_patch14_clip_224.laion2b_ft_in1k)        |
+| vit_large_patch14_clip_224.laion2b_ft_in1k       |   87.3 |         304.2 |   81.1 |    88.8 | [link](https://huggingface.co/timm/vit_large_patch14_clip_224.laion2b_ft_in1k)       |
+| vit_base_patch16_clip_384.laion2b_ft_in12k_in1k  |   87.2 |          86.9 |   55.5 |   101.6 | [link](https://huggingface.co/timm/vit_base_patch16_clip_384.laion2b_ft_in12k_in1k)  |
+| vit_base_patch16_clip_384.openai_ft_in12k_in1k   |   87   |          86.9 |   55.5 |   101.6 | [link](https://huggingface.co/timm/vit_base_patch16_clip_384.openai_ft_in12k_in1k)   |
+| vit_base_patch16_clip_384.laion2b_ft_in1k        |   86.6 |          86.9 |   55.5 |   101.6 | [link](https://huggingface.co/timm/vit_base_patch16_clip_384.laion2b_ft_in1k)        |
+| vit_base_patch16_clip_384.openai_ft_in1k         |   86.2 |          86.9 |   55.5 |   101.6 | [link](https://huggingface.co/timm/vit_base_patch16_clip_384.openai_ft_in1k)         |
+| vit_base_patch16_clip_224.laion2b_ft_in12k_in1k  |   86.2 |          86.6 |   17.6 |    23.9 | [link](https://huggingface.co/timm/vit_base_patch16_clip_224.laion2b_ft_in12k_in1k)  |
+| vit_base_patch16_clip_224.openai_ft_in12k_in1k   |   85.9 |          86.6 |   17.6 |    23.9 | [link](https://huggingface.co/timm/vit_base_patch16_clip_224.openai_ft_in12k_in1k)   |
+| vit_base_patch32_clip_448.laion2b_ft_in12k_in1k  |   85.8 |          88.3 |   17.9 |    23.9 | [link](https://huggingface.co/timm/vit_base_patch32_clip_448.laion2b_ft_in12k_in1k)  |
+| vit_base_patch16_clip_224.laion2b_ft_in1k        |   85.5 |          86.6 |   17.6 |    23.9 | [link](https://huggingface.co/timm/vit_base_patch16_clip_224.laion2b_ft_in1k)        |
+| vit_base_patch32_clip_384.laion2b_ft_in12k_in1k  |   85.4 |          88.3 |   13.1 |    16.5 | [link](https://huggingface.co/timm/vit_base_patch32_clip_384.laion2b_ft_in12k_in1k)  |
+| vit_base_patch16_clip_224.openai_ft_in1k         |   85.3 |          86.6 |   17.6 |    23.9 | [link](https://huggingface.co/timm/vit_base_patch16_clip_224.openai_ft_in1k)         |
+| vit_base_patch32_clip_384.openai_ft_in12k_in1k   |   85.2 |          88.3 |   13.1 |    16.5 | [link](https://huggingface.co/timm/vit_base_patch32_clip_384.openai_ft_in12k_in1k)   |
+| vit_base_patch32_clip_224.laion2b_ft_in12k_in1k  |   83.3 |          88.2 |    4.4 |     5   | [link](https://huggingface.co/timm/vit_base_patch32_clip_224.laion2b_ft_in12k_in1k)  |
+| vit_base_patch32_clip_224.laion2b_ft_in1k        |   82.6 |          88.2 |    4.4 |     5   | [link](https://huggingface.co/timm/vit_base_patch32_clip_224.laion2b_ft_in1k)        |
+| vit_base_patch32_clip_224.openai_ft_in1k         |   81.9 |          88.2 |    4.4 |     5   | [link](https://huggingface.co/timm/vit_base_patch32_clip_224.openai_ft_in1k)         |
+
+* Port of MaxViT Tensorflow Weights from official impl at https://github.com/google-research/maxvit
+  * There was larger than expected drops for the upscaled 384/512 in21k fine-tune weights, possible detail missing, but the 21k FT did seem sensitive to small preprocessing
+
+| model                              |   top1 |   param_count |   gmac |   macts | hub                                                                    |
+|:-----------------------------------|-------:|--------------:|-------:|--------:|:-----------------------------------------------------------------------|
+| maxvit_xlarge_tf_512.in21k_ft_in1k |   88.5 |         475.8 |  534.1 |  1413.2 | [link](https://huggingface.co/timm/maxvit_xlarge_tf_512.in21k_ft_in1k) |
+| maxvit_xlarge_tf_384.in21k_ft_in1k |   88.3 |         475.3 |  292.8 |   668.8 | [link](https://huggingface.co/timm/maxvit_xlarge_tf_384.in21k_ft_in1k) |
+| maxvit_base_tf_512.in21k_ft_in1k   |   88.2 |         119.9 |  138   |   704   | [link](https://huggingface.co/timm/maxvit_base_tf_512.in21k_ft_in1k)   |
+| maxvit_large_tf_512.in21k_ft_in1k  |   88   |         212.3 |  244.8 |   942.2 | [link](https://huggingface.co/timm/maxvit_large_tf_512.in21k_ft_in1k)  |
+| maxvit_large_tf_384.in21k_ft_in1k  |   88   |         212   |  132.6 |   445.8 | [link](https://huggingface.co/timm/maxvit_large_tf_384.in21k_ft_in1k)  |
+| maxvit_base_tf_384.in21k_ft_in1k   |   87.9 |         119.6 |   73.8 |   332.9 | [link](https://huggingface.co/timm/maxvit_base_tf_384.in21k_ft_in1k)   |
+| maxvit_base_tf_512.in1k            |   86.6 |         119.9 |  138   |   704   | [link](https://huggingface.co/timm/maxvit_base_tf_512.in1k)            |
+| maxvit_large_tf_512.in1k           |   86.5 |         212.3 |  244.8 |   942.2 | [link](https://huggingface.co/timm/maxvit_large_tf_512.in1k)           |
+| maxvit_base_tf_384.in1k            |   86.3 |         119.6 |   73.8 |   332.9 | [link](https://huggingface.co/timm/maxvit_base_tf_384.in1k)            |
+| maxvit_large_tf_384.in1k           |   86.2 |         212   |  132.6 |   445.8 | [link](https://huggingface.co/timm/maxvit_large_tf_384.in1k)           |
+| maxvit_small_tf_512.in1k           |   86.1 |          69.1 |   67.3 |   383.8 | [link](https://huggingface.co/timm/maxvit_small_tf_512.in1k)           |
+| maxvit_tiny_tf_512.in1k            |   85.7 |          31   |   33.5 |   257.6 | [link](https://huggingface.co/timm/maxvit_tiny_tf_512.in1k)            |
+| maxvit_small_tf_384.in1k           |   85.5 |          69   |   35.9 |   183.6 | [link](https://huggingface.co/timm/maxvit_small_tf_384.in1k)           |
+| maxvit_tiny_tf_384.in1k            |   85.1 |          31   |   17.5 |   123.4 | [link](https://huggingface.co/timm/maxvit_tiny_tf_384.in1k)            |
+| maxvit_large_tf_224.in1k           |   84.9 |         211.8 |   43.7 |   127.4 | [link](https://huggingface.co/timm/maxvit_large_tf_224.in1k)           |
+| maxvit_base_tf_224.in1k            |   84.9 |         119.5 |   24   |    95   | [link](https://huggingface.co/timm/maxvit_base_tf_224.in1k)            |
+| maxvit_small_tf_224.in1k           |   84.4 |          68.9 |   11.7 |    53.2 | [link](https://huggingface.co/timm/maxvit_small_tf_224.in1k)           |
+| maxvit_tiny_tf_224.in1k            |   83.4 |          30.9 |    5.6 |    35.8 | [link](https://huggingface.co/timm/maxvit_tiny_tf_224.in1k)            |
+
+### Oct 15, 2022
+* Train and validation script enhancements
+* Non-GPU (ie CPU) device support
+* SLURM compatibility for train script
+* HF datasets support (via ReaderHfds)
+* TFDS/WDS dataloading improvements (sample padding/wrap for distributed use fixed wrt sample count estimate)
+* in_chans !=3 support for scripts / loader
+* Adan optimizer
+* Can enable per-step LR scheduling via args
+* Dataset 'parsers' renamed to 'readers', more descriptive of purpose
+* AMP args changed, APEX via `--amp-impl apex`, bfloat16 supportedf via `--amp-dtype bfloat16`
+* main branch switched to 0.7.x version, 0.6x forked for stable release of weight only adds
+* master -> main branch rename
+
+### Oct 10, 2022
+* More weights in `maxxvit` series, incl first ConvNeXt block based `coatnext` and `maxxvit` experiments:
+  * `coatnext_nano_rw_224` - 82.0 @ 224 (G) -- (uses ConvNeXt conv block, no BatchNorm)
+  * `maxxvit_rmlp_nano_rw_256` - 83.0 @ 256, 83.7 @ 320  (G) (uses ConvNeXt conv block, no BN)
+  * `maxvit_rmlp_small_rw_224` - 84.5 @ 224, 85.1 @ 320 (G)
+  * `maxxvit_rmlp_small_rw_256` - 84.6 @ 256, 84.9 @ 288 (G) -- could be trained better, hparams need tuning (uses ConvNeXt block, no BN)
+  * `coatnet_rmlp_2_rw_224` - 84.6 @ 224, 85 @ 320  (T)
+  * NOTE: official MaxVit weights (in1k) have been released at https://github.com/google-research/maxvit -- some extra work is needed to port and adapt since my impl was created independently of theirs and has a few small differences + the whole TF same padding fun.
+
+### Sept 23, 2022
+* LAION-2B CLIP image towers supported as pretrained backbones for fine-tune or features (no classifier)
+  * vit_base_patch32_224_clip_laion2b
+  * vit_large_patch14_224_clip_laion2b
+  * vit_huge_patch14_224_clip_laion2b
+  * vit_giant_patch14_224_clip_laion2b
+
+### Sept 7, 2022
+* Hugging Face [`timm` docs](https://huggingface.co/docs/hub/timm) home now exists, look for more here in the future
+* Add BEiT-v2 weights for base and large 224x224 models from https://github.com/microsoft/unilm/tree/master/beit2
+* Add more weights in `maxxvit` series incl a `pico` (7.5M params, 1.9 GMACs), two `tiny` variants:
+  * `maxvit_rmlp_pico_rw_256` - 80.5 @ 256, 81.3 @ 320  (T)
+  * `maxvit_tiny_rw_224` - 83.5 @ 224 (G)
+  * `maxvit_rmlp_tiny_rw_256` - 84.2 @ 256, 84.8 @ 320 (T)
+
+### Aug 29, 2022
+* MaxVit window size scales with img_size by default. Add new RelPosMlp MaxViT weight that leverages this:
+  * `maxvit_rmlp_nano_rw_256` - 83.0 @ 256, 83.6 @ 320  (T)
+
+### Aug 26, 2022
+* CoAtNet (https://arxiv.org/abs/2106.04803) and MaxVit (https://arxiv.org/abs/2204.01697) `timm` original models
+  * both found in [`maxxvit.py`](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/maxxvit.py) model def, contains numerous experiments outside scope of original papers
+  * an unfinished Tensorflow version from MaxVit authors can be found https://github.com/google-research/maxvit
+* Initial CoAtNet and MaxVit timm pretrained weights (working on more):
+  * `coatnet_nano_rw_224` - 81.7 @ 224  (T)
+  * `coatnet_rmlp_nano_rw_224` - 82.0 @ 224, 82.8 @ 320 (T)
+  * `coatnet_0_rw_224` - 82.4  (T)  -- NOTE timm '0' coatnets have 2 more 3rd stage blocks
+  * `coatnet_bn_0_rw_224` - 82.4  (T)
+  * `maxvit_nano_rw_256` - 82.9 @ 256  (T)
+  * `coatnet_rmlp_1_rw_224` - 83.4 @ 224, 84 @ 320  (T)
+  * `coatnet_1_rw_224` - 83.6 @ 224 (G)
+  * (T) = TPU trained with `bits_and_tpu` branch training code, (G) = GPU trained
+* GCVit (weights adapted from https://github.com/NVlabs/GCVit, code 100% `timm` re-write for license purposes)
+* MViT-V2 (multi-scale vit, adapted from https://github.com/facebookresearch/mvit)
+* EfficientFormer (adapted from https://github.com/snap-research/EfficientFormer)
+* PyramidVisionTransformer-V2 (adapted from https://github.com/whai362/PVT)
+* 'Fast Norm' support for LayerNorm and GroupNorm that avoids float32 upcast w/ AMP (uses APEX LN if available for further boost)
+
+
+### Aug 15, 2022
+* ConvNeXt atto weights added
+  * `convnext_atto` - 75.7 @ 224, 77.0 @ 288
+  * `convnext_atto_ols` - 75.9  @ 224, 77.2 @ 288
+
+### Aug 5, 2022
+* More custom ConvNeXt smaller model defs with weights
+  * `convnext_femto` - 77.5 @ 224, 78.7 @ 288
+  * `convnext_femto_ols` - 77.9  @ 224, 78.9 @ 288
+  * `convnext_pico` - 79.5 @ 224, 80.4 @ 288
+  * `convnext_pico_ols` - 79.5 @ 224, 80.5 @ 288
+  * `convnext_nano_ols` - 80.9 @ 224, 81.6 @ 288
+* Updated EdgeNeXt to improve ONNX export, add new base variant and weights from original (https://github.com/mmaaz60/EdgeNeXt)
+
+### July 28, 2022
+* Add freshly minted DeiT-III Medium (width=512, depth=12, num_heads=8) model weights. Thanks [Hugo Touvron](https://github.com/TouvronHugo)!
+
+### July 27, 2022
+* All runtime benchmark and validation result csv files are finally up-to-date!
+* A few more weights & model defs added:
+  * `darknetaa53` -  79.8 @ 256, 80.5 @ 288
+  * `convnext_nano` - 80.8 @ 224, 81.5 @ 288
+  * `cs3sedarknet_l` - 81.2 @ 256, 81.8 @ 288
+  * `cs3darknet_x` - 81.8 @ 256, 82.2 @ 288
+  * `cs3sedarknet_x` - 82.2 @ 256, 82.7 @ 288
+  * `cs3edgenet_x` - 82.2 @ 256, 82.7 @ 288
+  * `cs3se_edgenet_x` - 82.8 @ 256, 83.5 @ 320
+* `cs3*` weights above all trained on TPU w/ `bits_and_tpu` branch. Thanks to TRC program!
+* Add output_stride=8 and 16 support to ConvNeXt (dilation)
+* deit3 models not being able to resize pos_emb fixed
+* Version 0.6.7 PyPi release (/w above bug fixes and new weighs since 0.6.5)
+
+### July 8, 2022
+More models, more fixes
+* Official research models (w/ weights) added:
+  * EdgeNeXt from (https://github.com/mmaaz60/EdgeNeXt)
+  * MobileViT-V2 from (https://github.com/apple/ml-cvnets)
+  * DeiT III (Revenge of the ViT) from (https://github.com/facebookresearch/deit)
+* My own models:
+  * Small `ResNet` defs added by request with 1 block repeats for both basic and bottleneck (resnet10 and resnet14)
+  * `CspNet` refactored with dataclass config, simplified CrossStage3 (`cs3`) option. These are closer to YOLO-v5+ backbone defs.
+  * More relative position vit fiddling. Two `srelpos` (shared relative position) models trained, and a medium w/ class token.
+  * Add an alternate downsample mode to EdgeNeXt and train a `small` model. Better than original small, but not their new USI trained weights.
+* My own model weight results (all ImageNet-1k training)
+  * `resnet10t` - 66.5 @ 176, 68.3 @ 224
+  * `resnet14t` - 71.3 @ 176, 72.3 @ 224
+  * `resnetaa50` - 80.6 @ 224 , 81.6 @ 288
+  * `darknet53` -  80.0 @ 256, 80.5 @ 288
+  * `cs3darknet_m` - 77.0 @ 256, 77.6 @ 288
+  * `cs3darknet_focus_m` - 76.7 @ 256, 77.3 @ 288
+  * `cs3darknet_l` - 80.4 @ 256, 80.9 @ 288
+  * `cs3darknet_focus_l` - 80.3 @ 256, 80.9 @ 288
+  * `vit_srelpos_small_patch16_224` - 81.1 @ 224, 82.1 @ 320
+  * `vit_srelpos_medium_patch16_224` - 82.3 @ 224, 83.1 @ 320
+  * `vit_relpos_small_patch16_cls_224` - 82.6 @ 224, 83.6 @ 320
+  * `edgnext_small_rw` - 79.6 @ 224, 80.4 @ 320
+* `cs3`, `darknet`, and `vit_*relpos` weights above all trained on TPU thanks to TRC program! Rest trained on overheating GPUs.
+* Hugging Face Hub support fixes verified, demo notebook TBA
+* Pretrained weights / configs can be loaded externally (ie from local disk) w/ support for head adaptation.
+* Add support to change image extensions scanned by `timm` datasets/readers. See (https://github.com/rwightman/pytorch-image-models/pull/1274#issuecomment-1178303103)
+* Default ConvNeXt LayerNorm impl to use `F.layer_norm(x.permute(0, 2, 3, 1), ...).permute(0, 3, 1, 2)` via `LayerNorm2d` in all cases.
+  * a bit slower than previous custom impl on some hardware (ie Ampere w/ CL), but overall fewer regressions across wider HW / PyTorch version ranges.
+  * previous impl exists as `LayerNormExp2d` in `models/layers/norm.py`
+* Numerous bug fixes
+* Currently testing for imminent PyPi 0.6.x release
+* LeViT pretraining of larger models still a WIP, they don't train well / easily without distillation. Time to add distill support (finally)?
+* ImageNet-22k weight training + finetune ongoing, work on multi-weight support (slowly) chugging along (there are a LOT of weights, sigh) ...
+
+### May 13, 2022
+* Official Swin-V2 models and weights added from (https://github.com/microsoft/Swin-Transformer). Cleaned up to support torchscript.
+* Some refactoring for existing `timm` Swin-V2-CR impl, will likely do a bit more to bring parts closer to official and decide whether to merge some aspects.
+* More Vision Transformer relative position / residual post-norm experiments (all trained on TPU thanks to TRC program)
+  * `vit_relpos_small_patch16_224` - 81.5 @ 224, 82.5 @ 320 -- rel pos, layer scale, no class token, avg pool
+  * `vit_relpos_medium_patch16_rpn_224` - 82.3 @ 224, 83.1 @ 320 -- rel pos + res-post-norm, no class token, avg pool
+  * `vit_relpos_medium_patch16_224` - 82.5 @ 224, 83.3 @ 320 -- rel pos, layer scale, no class token, avg pool
+  * `vit_relpos_base_patch16_gapcls_224` - 82.8 @ 224, 83.9 @ 320 -- rel pos, layer scale, class token, avg pool (by mistake)
+* Bring 512 dim, 8-head 'medium' ViT model variant back to life (after using in a pre DeiT 'small' model for first ViT impl back in 2020)
+* Add ViT relative position support for switching btw existing impl and some additions in official Swin-V2 impl for future trials
+* Sequencer2D impl (https://arxiv.org/abs/2205.01972), added via PR from author (https://github.com/okojoalg)
+
+### May 2, 2022
+* Vision Transformer experiments adding Relative Position (Swin-V2 log-coord) (`vision_transformer_relpos.py`) and Residual Post-Norm branches (from Swin-V2) (`vision_transformer*.py`)
+  * `vit_relpos_base_patch32_plus_rpn_256` - 79.5 @ 256, 80.6 @ 320 -- rel pos + extended width + res-post-norm, no class token, avg pool
+  * `vit_relpos_base_patch16_224` - 82.5 @ 224, 83.6 @ 320 -- rel pos, layer scale, no class token, avg pool
+  * `vit_base_patch16_rpn_224` - 82.3 @ 224 -- rel pos + res-post-norm, no class token, avg pool
+* Vision Transformer refactor to remove representation layer that was only used in initial vit and rarely used since with newer pretrain (ie `How to Train Your ViT`)
+* `vit_*` models support removal of class token, use of global average pool, use of fc_norm (ala beit, mae).
+
+### April 22, 2022
+* `timm` models are now officially supported in [fast.ai](https://www.fast.ai/)! Just in time for the new Practical Deep Learning course. `timmdocs` documentation link updated to [timm.fast.ai](http://timm.fast.ai/).
+* Two more model weights added in the TPU trained [series](https://github.com/rwightman/pytorch-image-models/releases/tag/v0.1-tpu-weights). Some In22k pretrain still in progress.
+  * `seresnext101d_32x8d` - 83.69 @ 224, 84.35 @ 288
+  * `seresnextaa101d_32x8d` (anti-aliased w/ AvgPool2d) - 83.85 @ 224, 84.57 @ 288
+
+### March 23, 2022
+* Add `ParallelBlock` and `LayerScale` option to base vit models to support model configs in [Three things everyone should know about ViT](https://arxiv.org/abs/2203.09795)
+* `convnext_tiny_hnf` (head norm first) weights trained with (close to) A2 recipe, 82.2% top-1, could do better with more epochs.
+
+### March 21, 2022
+* Merge `norm_norm_norm`. **IMPORTANT** this update for a coming 0.6.x release will likely de-stabilize the master branch for a while. Branch [`0.5.x`](https://github.com/rwightman/pytorch-image-models/tree/0.5.x) or a previous 0.5.x release can be used if stability is required.
+* Significant weights update (all TPU trained) as described in this [release](https://github.com/rwightman/pytorch-image-models/releases/tag/v0.1-tpu-weights)
+  * `regnety_040` - 82.3 @ 224, 82.96 @ 288
+  * `regnety_064` - 83.0 @ 224, 83.65 @ 288
+  * `regnety_080` - 83.17 @ 224, 83.86 @ 288
+  * `regnetv_040` - 82.44 @ 224, 83.18 @ 288   (timm pre-act)
+  * `regnetv_064` - 83.1 @ 224, 83.71 @ 288   (timm pre-act)
+  * `regnetz_040` - 83.67 @ 256, 84.25 @ 320
+  * `regnetz_040h` - 83.77 @ 256, 84.5 @ 320 (w/ extra fc in head)
+  * `resnetv2_50d_gn` - 80.8 @ 224, 81.96 @ 288 (pre-act GroupNorm)
+  * `resnetv2_50d_evos` 80.77 @ 224, 82.04 @ 288 (pre-act EvoNormS)
+  * `regnetz_c16_evos`  - 81.9 @ 256, 82.64 @ 320 (EvoNormS)
+  * `regnetz_d8_evos`  - 83.42 @ 256, 84.04 @ 320 (EvoNormS)
+  * `xception41p` - 82 @ 299   (timm pre-act)
+  * `xception65` -  83.17 @ 299
+  * `xception65p` -  83.14 @ 299   (timm pre-act)
+  * `resnext101_64x4d` - 82.46 @ 224, 83.16 @ 288
+  * `seresnext101_32x8d` - 83.57 @ 224, 84.270 @ 288
+  * `resnetrs200` - 83.85 @ 256, 84.44 @ 320
+* HuggingFace hub support fixed w/ initial groundwork for allowing alternative 'config sources' for pretrained model definitions and weights (generic local file / remote url support soon)
+* SwinTransformer-V2 implementation added. Submitted by [Christoph Reich](https://github.com/ChristophReich1996). Training experiments and model changes by myself are ongoing so expect compat breaks.
+* Swin-S3 (AutoFormerV2) models / weights added from https://github.com/microsoft/Cream/tree/main/AutoFormerV2
+* MobileViT models w/ weights adapted from https://github.com/apple/ml-cvnets
+* PoolFormer models w/ weights adapted from https://github.com/sail-sg/poolformer
+* VOLO models w/ weights adapted from https://github.com/sail-sg/volo
+* Significant work experimenting with non-BatchNorm norm layers such as EvoNorm, FilterResponseNorm, GroupNorm, etc
+* Enhance support for alternate norm + act ('NormAct') layers added to a number of models, esp EfficientNet/MobileNetV3, RegNet, and aligned Xception
+* Grouped conv support added to EfficientNet family
+* Add 'group matching' API to all models to allow grouping model parameters for application of 'layer-wise' LR decay, lr scale added to LR scheduler
+* Gradient checkpointing support added to many models
+* `forward_head(x, pre_logits=False)` fn added to all models to allow separate calls of `forward_features` + `forward_head`
+* All vision transformer and vision MLP models update to return non-pooled / non-token selected features from `foward_features`, for consistency with CNN models, token selection or pooling now applied in `forward_head`
+
+### Feb 2, 2022
+* [Chris Hughes](https://github.com/Chris-hughes10) posted an exhaustive run through of `timm` on his blog yesterday. Well worth a read. [Getting Started with PyTorch Image Models (timm): A Practitioner’s Guide](https://towardsdatascience.com/getting-started-with-pytorch-image-models-timm-a-practitioners-guide-4e77b4bf9055)
+* I'm currently prepping to merge the `norm_norm_norm` branch back to master (ver 0.6.x) in next week or so.
+  * The changes are more extensive than usual and may destabilize and break some model API use (aiming for full backwards compat). So, beware `pip install git+https://github.com/rwightman/pytorch-image-models` installs!
+  * `0.5.x` releases and a `0.5.x` branch will remain stable with a cherry pick or two until dust clears. Recommend sticking to pypi install for a bit if you want stable.
+
+### Jan 14, 2022
+* Version 0.5.4 w/ release to be pushed to pypi. It's been a while since last pypi update and riskier changes will be merged to main branch soon....
+* Add ConvNeXT models /w weights from official impl (https://github.com/facebookresearch/ConvNeXt), a few perf tweaks, compatible with timm features
+* Tried training a few small (~1.8-3M param) / mobile optimized models, a few are good so far, more on the way...
+  * `mnasnet_small` - 65.6 top-1
+  * `mobilenetv2_050` - 65.9
+  * `lcnet_100/075/050` - 72.1 / 68.8 / 63.1
+  * `semnasnet_075` - 73
+  * `fbnetv3_b/d/g` - 79.1 / 79.7 / 82.0
+* TinyNet models added by [rsomani95](https://github.com/rsomani95)
+* LCNet added via MobileNetV3 architecture
+
+## Introduction
+
+Py**T**orch **Im**age **M**odels (`timm`) is a collection of image models, layers, utilities, optimizers, schedulers, data-loaders / augmentations, and reference training / validation scripts that aim to pull together a wide variety of SOTA models with ability to reproduce ImageNet training results.
+
+The work of many others is present here. I've tried to make sure all source material is acknowledged via links to github, arxiv papers, etc in the README, documentation, and code docstrings. Please let me know if I missed anything.
+
+## Models
+
+All model architecture families include variants with pretrained weights. There are specific model variants without any weights, it is NOT a bug. Help training new or better weights is always appreciated.
+
+* Aggregating Nested Transformers - https://arxiv.org/abs/2105.12723
+* BEiT - https://arxiv.org/abs/2106.08254
+* Big Transfer ResNetV2 (BiT) - https://arxiv.org/abs/1912.11370
+* Bottleneck Transformers - https://arxiv.org/abs/2101.11605
+* CaiT (Class-Attention in Image Transformers) - https://arxiv.org/abs/2103.17239
+* CoaT (Co-Scale Conv-Attentional Image Transformers) - https://arxiv.org/abs/2104.06399
+* CoAtNet (Convolution and Attention) - https://arxiv.org/abs/2106.04803
+* ConvNeXt - https://arxiv.org/abs/2201.03545
+* ConvNeXt-V2 - http://arxiv.org/abs/2301.00808
+* ConViT (Soft Convolutional Inductive Biases Vision Transformers)- https://arxiv.org/abs/2103.10697
+* CspNet (Cross-Stage Partial Networks) - https://arxiv.org/abs/1911.11929
+* DeiT - https://arxiv.org/abs/2012.12877
+* DeiT-III - https://arxiv.org/pdf/2204.07118.pdf
+* DenseNet - https://arxiv.org/abs/1608.06993
+* DLA - https://arxiv.org/abs/1707.06484
+* DPN (Dual-Path Network) - https://arxiv.org/abs/1707.01629
+* EdgeNeXt - https://arxiv.org/abs/2206.10589
+* EfficientFormer - https://arxiv.org/abs/2206.01191
+* EfficientNet (MBConvNet Family)
+    * EfficientNet NoisyStudent (B0-B7, L2) - https://arxiv.org/abs/1911.04252
+    * EfficientNet AdvProp (B0-B8) - https://arxiv.org/abs/1911.09665
+    * EfficientNet (B0-B7) - https://arxiv.org/abs/1905.11946
+    * EfficientNet-EdgeTPU (S, M, L) - https://ai.googleblog.com/2019/08/efficientnet-edgetpu-creating.html
+    * EfficientNet V2 - https://arxiv.org/abs/2104.00298
+    * FBNet-C - https://arxiv.org/abs/1812.03443
+    * MixNet - https://arxiv.org/abs/1907.09595
+    * MNASNet B1, A1 (Squeeze-Excite), and Small - https://arxiv.org/abs/1807.11626
+    * MobileNet-V2 - https://arxiv.org/abs/1801.04381
+    * Single-Path NAS - https://arxiv.org/abs/1904.02877
+    * TinyNet - https://arxiv.org/abs/2010.14819
+* EVA - https://arxiv.org/abs/2211.07636
+* EVA-02 - https://arxiv.org/abs/2303.11331
+* FlexiViT - https://arxiv.org/abs/2212.08013
+* FocalNet (Focal Modulation Networks) - https://arxiv.org/abs/2203.11926
+* GCViT (Global Context Vision Transformer) - https://arxiv.org/abs/2206.09959
+* GhostNet - https://arxiv.org/abs/1911.11907
+* gMLP - https://arxiv.org/abs/2105.08050
+* GPU-Efficient Networks - https://arxiv.org/abs/2006.14090
+* Halo Nets - https://arxiv.org/abs/2103.12731
+* HRNet - https://arxiv.org/abs/1908.07919
+* Inception-V3 - https://arxiv.org/abs/1512.00567
+* Inception-ResNet-V2 and Inception-V4 - https://arxiv.org/abs/1602.07261
+* Lambda Networks - https://arxiv.org/abs/2102.08602
+* LeViT (Vision Transformer in ConvNet's Clothing) - https://arxiv.org/abs/2104.01136
+* MaxViT (Multi-Axis Vision Transformer) - https://arxiv.org/abs/2204.01697
+* MLP-Mixer - https://arxiv.org/abs/2105.01601
+* MobileNet-V3 (MBConvNet w/ Efficient Head) - https://arxiv.org/abs/1905.02244
+  * FBNet-V3 - https://arxiv.org/abs/2006.02049
+  * HardCoRe-NAS - https://arxiv.org/abs/2102.11646
+  * LCNet - https://arxiv.org/abs/2109.15099
+* MobileViT - https://arxiv.org/abs/2110.02178
+* MobileViT-V2 - https://arxiv.org/abs/2206.02680
+* MViT-V2 (Improved Multiscale Vision Transformer) - https://arxiv.org/abs/2112.01526
+* NASNet-A - https://arxiv.org/abs/1707.07012
+* NesT - https://arxiv.org/abs/2105.12723
+* NFNet-F - https://arxiv.org/abs/2102.06171
+* NF-RegNet / NF-ResNet - https://arxiv.org/abs/2101.08692
+* PNasNet - https://arxiv.org/abs/1712.00559
+* PoolFormer (MetaFormer) - https://arxiv.org/abs/2111.11418
+* Pooling-based Vision Transformer (PiT) - https://arxiv.org/abs/2103.16302
+* PVT-V2 (Improved Pyramid Vision Transformer) - https://arxiv.org/abs/2106.13797
+* RegNet - https://arxiv.org/abs/2003.13678
+* RegNetZ - https://arxiv.org/abs/2103.06877
+* RepVGG - https://arxiv.org/abs/2101.03697
+* ResMLP - https://arxiv.org/abs/2105.03404
+* ResNet/ResNeXt
+    * ResNet (v1b/v1.5) - https://arxiv.org/abs/1512.03385
+    * ResNeXt - https://arxiv.org/abs/1611.05431
+    * 'Bag of Tricks' / Gluon C, D, E, S variations - https://arxiv.org/abs/1812.01187
+    * Weakly-supervised (WSL) Instagram pretrained / ImageNet tuned ResNeXt101 - https://arxiv.org/abs/1805.00932
+    * Semi-supervised (SSL) / Semi-weakly Supervised (SWSL) ResNet/ResNeXts - https://arxiv.org/abs/1905.00546
+    * ECA-Net (ECAResNet) - https://arxiv.org/abs/1910.03151v4
+    * Squeeze-and-Excitation Networks (SEResNet) - https://arxiv.org/abs/1709.01507
+    * ResNet-RS - https://arxiv.org/abs/2103.07579
+* Res2Net - https://arxiv.org/abs/1904.01169
+* ResNeSt - https://arxiv.org/abs/2004.08955
+* ReXNet - https://arxiv.org/abs/2007.00992
+* SelecSLS - https://arxiv.org/abs/1907.00837
+* Selective Kernel Networks - https://arxiv.org/abs/1903.06586
+* Sequencer2D - https://arxiv.org/abs/2205.01972
+* Swin S3 (AutoFormerV2) - https://arxiv.org/abs/2111.14725
+* Swin Transformer - https://arxiv.org/abs/2103.14030
+* Swin Transformer V2 - https://arxiv.org/abs/2111.09883
+* Transformer-iN-Transformer (TNT) - https://arxiv.org/abs/2103.00112
+* TResNet - https://arxiv.org/abs/2003.13630
+* Twins (Spatial Attention in Vision Transformers) - https://arxiv.org/pdf/2104.13840.pdf
+* Visformer - https://arxiv.org/abs/2104.12533
+* Vision Transformer - https://arxiv.org/abs/2010.11929
+* VOLO (Vision Outlooker) - https://arxiv.org/abs/2106.13112
+* VovNet V2 and V1 - https://arxiv.org/abs/1911.06667
+* Xception - https://arxiv.org/abs/1610.02357
+* Xception (Modified Aligned, Gluon) - https://arxiv.org/abs/1802.02611
+* Xception (Modified Aligned, TF) - https://arxiv.org/abs/1802.02611
+* XCiT (Cross-Covariance Image Transformers) - https://arxiv.org/abs/2106.09681
+
+## Features
+
+Several (less common) features that I often utilize in my projects are included. Many of their additions are the reason why I maintain my own set of models, instead of using others' via PIP:
+
+* All models have a common default configuration interface and API for
+    * accessing/changing the classifier - `get_classifier` and `reset_classifier`
+    * doing a forward pass on just the features - `forward_features` (see [documentation](https://huggingface.co/docs/timm/feature_extraction))
+    * these makes it easy to write consistent network wrappers that work with any of the models
+* All models support multi-scale feature map extraction (feature pyramids) via create_model (see [documentation](https://huggingface.co/docs/timm/feature_extraction))
+    * `create_model(name, features_only=True, out_indices=..., output_stride=...)`
+    * `out_indices` creation arg specifies which feature maps to return, these indices are 0 based and generally correspond to the `C(i + 1)` feature level.
+    * `output_stride` creation arg controls output stride of the network by using dilated convolutions. Most networks are stride 32 by default. Not all networks support this.
+    * feature map channel counts, reduction level (stride) can be queried AFTER model creation via the `.feature_info` member
+* All models have a consistent pretrained weight loader that adapts last linear if necessary, and from 3 to 1 channel input if desired
+* High performance [reference training, validation, and inference scripts](https://huggingface.co/docs/timm/training_script) that work in several process/GPU modes:
+    * NVIDIA DDP w/ a single GPU per process, multiple processes with APEX present (AMP mixed-precision optional)
+    * PyTorch DistributedDataParallel w/ multi-gpu, single process (AMP disabled as it crashes when enabled)
+    * PyTorch w/ single GPU single process (AMP optional)
+* A dynamic global pool implementation that allows selecting from average pooling, max pooling, average + max, or concat([average, max]) at model creation. All global pooling is adaptive average by default and compatible with pretrained weights.
+* A 'Test Time Pool' wrapper that can wrap any of the included models and usually provides improved performance doing inference with input images larger than the training size. Idea adapted from original DPN implementation when I ported (https://github.com/cypw/DPNs)
+* Learning rate schedulers
+  * Ideas adopted from
+     * [AllenNLP schedulers](https://github.com/allenai/allennlp/tree/master/allennlp/training/learning_rate_schedulers)
+     * [FAIRseq lr_scheduler](https://github.com/pytorch/fairseq/tree/master/fairseq/optim/lr_scheduler)
+     * SGDR: Stochastic Gradient Descent with Warm Restarts (https://arxiv.org/abs/1608.03983)
+  * Schedulers include `step`, `cosine` w/ restarts, `tanh` w/ restarts, `plateau`
+* Optimizers:
+    * `rmsprop_tf` adapted from PyTorch RMSProp by myself. Reproduces much improved Tensorflow RMSProp behaviour.
+    * `radam` by [Liyuan Liu](https://github.com/LiyuanLucasLiu/RAdam) (https://arxiv.org/abs/1908.03265)
+    * `novograd` by [Masashi Kimura](https://github.com/convergence-lab/novograd) (https://arxiv.org/abs/1905.11286)
+    * `lookahead` adapted from impl by [Liam](https://github.com/alphadl/lookahead.pytorch) (https://arxiv.org/abs/1907.08610)
+    * `fused<name>` optimizers by name with [NVIDIA Apex](https://github.com/NVIDIA/apex/tree/master/apex/optimizers) installed
+    * `adamp` and `sgdp` by [Naver ClovAI](https://github.com/clovaai) (https://arxiv.org/abs/2006.08217)
+    * `adafactor` adapted from [FAIRSeq impl](https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py) (https://arxiv.org/abs/1804.04235)
+    * `adahessian` by [David Samuel](https://github.com/davda54/ada-hessian) (https://arxiv.org/abs/2006.00719)
+* Random Erasing from [Zhun Zhong](https://github.com/zhunzhong07/Random-Erasing/blob/master/transforms.py)  (https://arxiv.org/abs/1708.04896)
+* Mixup (https://arxiv.org/abs/1710.09412)
+* CutMix (https://arxiv.org/abs/1905.04899)
+* AutoAugment (https://arxiv.org/abs/1805.09501) and RandAugment (https://arxiv.org/abs/1909.13719) ImageNet configurations modeled after impl for EfficientNet training (https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py)
+* AugMix w/ JSD loss (https://arxiv.org/abs/1912.02781), JSD w/ clean + augmented mixing support works with AutoAugment and RandAugment as well
+* SplitBachNorm - allows splitting batch norm layers between clean and augmented (auxiliary batch norm) data
+* DropPath aka "Stochastic Depth" (https://arxiv.org/abs/1603.09382)
+* DropBlock (https://arxiv.org/abs/1810.12890)
+* Blur Pooling (https://arxiv.org/abs/1904.11486)
+* Space-to-Depth by [mrT23](https://github.com/mrT23/TResNet/blob/master/src/models/tresnet/layers/space_to_depth.py) (https://arxiv.org/abs/1801.04590) -- original paper?
+* Adaptive Gradient Clipping (https://arxiv.org/abs/2102.06171, https://github.com/deepmind/deepmind-research/tree/master/nfnets)
+* An extensive selection of channel and/or spatial attention modules:
+    * Bottleneck Transformer - https://arxiv.org/abs/2101.11605
+    * CBAM - https://arxiv.org/abs/1807.06521
+    * Effective Squeeze-Excitation (ESE) - https://arxiv.org/abs/1911.06667
+    * Efficient Channel Attention (ECA) - https://arxiv.org/abs/1910.03151
+    * Gather-Excite (GE) - https://arxiv.org/abs/1810.12348
+    * Global Context (GC) - https://arxiv.org/abs/1904.11492
+    * Halo - https://arxiv.org/abs/2103.12731
+    * Involution - https://arxiv.org/abs/2103.06255
+    * Lambda Layer - https://arxiv.org/abs/2102.08602
+    * Non-Local (NL) -  https://arxiv.org/abs/1711.07971
+    * Squeeze-and-Excitation (SE) - https://arxiv.org/abs/1709.01507
+    * Selective Kernel (SK) - (https://arxiv.org/abs/1903.06586
+    * Split (SPLAT) - https://arxiv.org/abs/2004.08955
+    * Shifted Window (SWIN) - https://arxiv.org/abs/2103.14030
+
+## Results
+
+Model validation results can be found in the [results tables](results/README.md)
+
+## Getting Started (Documentation)
+
+The official documentation can be found at https://huggingface.co/docs/hub/timm. Documentation contributions are welcome.
+
+[Getting Started with PyTorch Image Models (timm): A Practitioner’s Guide](https://towardsdatascience.com/getting-started-with-pytorch-image-models-timm-a-practitioners-guide-4e77b4bf9055) by [Chris Hughes](https://github.com/Chris-hughes10) is an extensive blog post covering many aspects of `timm` in detail.
+
+[timmdocs](http://timm.fast.ai/) is an alternate set of documentation for `timm`. A big thanks to [Aman Arora](https://github.com/amaarora) for his efforts creating timmdocs.
+
+[paperswithcode](https://paperswithcode.com/lib/timm) is a good resource for browsing the models within `timm`.
+
+## Train, Validation, Inference Scripts
+
+The root folder of the repository contains reference train, validation, and inference scripts that work with the included models and other features of this repository. They are adaptable for other datasets and use cases with a little hacking. See [documentation](https://huggingface.co/docs/timm/training_script).
+
+## Awesome PyTorch Resources
+
+One of the greatest assets of PyTorch is the community and their contributions. A few of my favourite resources that pair well with the models and components here are listed below.
+
+### Object Detection, Instance and Semantic Segmentation
+* Detectron2 - https://github.com/facebookresearch/detectron2
+* Segmentation Models (Semantic) - https://github.com/qubvel/segmentation_models.pytorch
+* EfficientDet (Obj Det, Semantic soon) - https://github.com/rwightman/efficientdet-pytorch
+
+### Computer Vision / Image Augmentation
+* Albumentations - https://github.com/albumentations-team/albumentations
+* Kornia - https://github.com/kornia/kornia
+
+### Knowledge Distillation
+* RepDistiller - https://github.com/HobbitLong/RepDistiller
+* torchdistill - https://github.com/yoshitomo-matsubara/torchdistill
+
+### Metric Learning
+* PyTorch Metric Learning - https://github.com/KevinMusgrave/pytorch-metric-learning
+
+### Training / Frameworks
+* fastai - https://github.com/fastai/fastai
+
+## Licenses
+
+### Code
+The code here is licensed Apache 2.0. I've taken care to make sure any third party code included or adapted has compatible (permissive) licenses such as MIT, BSD, etc. I've made an effort to avoid any GPL / LGPL conflicts. That said, it is your responsibility to ensure you comply with licenses here and conditions of any dependent licenses. Where applicable, I've linked the sources/references for various components in docstrings. If you think I've missed anything please create an issue.
+
+### Pretrained Weights
+So far all of the pretrained weights available here are pretrained on ImageNet with a select few that have some additional pretraining (see extra note below). ImageNet was released for non-commercial research purposes only (https://image-net.org/download). It's not clear what the implications of that are for the use of pretrained weights from that dataset. Any models I have trained with ImageNet are done for research purposes and one should assume that the original dataset license applies to the weights. It's best to seek legal advice if you intend to use the pretrained weights in a commercial product.
+
+#### Pretrained on more than ImageNet
+Several weights included or references here were pretrained with proprietary datasets that I do not have access to. These include the Facebook WSL, SSL, SWSL ResNe(Xt) and the Google Noisy Student EfficientNet models. The Facebook models have an explicit non-commercial license (CC-BY-NC 4.0, https://github.com/facebookresearch/semi-supervised-ImageNet1K-models, https://github.com/facebookresearch/WSL-Images). The Google models do not appear to have any restriction beyond the Apache 2.0 license (and ImageNet concerns). In either case, you should contact Facebook or Google with any questions.
+
+## Citing
+
+### BibTeX
+
+```bibtex
+@misc{rw2019timm,
+  author = {Ross Wightman},
+  title = {PyTorch Image Models},
+  year = {2019},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  doi = {10.5281/zenodo.4414861},
+  howpublished = {\url{https://github.com/rwightman/pytorch-image-models}}
+}
+```
+
+### Latest DOI
+
+[![DOI](https://zenodo.org/badge/168799526.svg)](https://zenodo.org/badge/latestdoi/168799526)

avg_checkpoints.py

@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+""" Checkpoint Averaging Script
+
+This script averages all model weights for checkpoints in specified path that match
+the specified filter wildcard. All checkpoints must be from the exact same model.
+
+For any hope of decent results, the checkpoints should be from the same or child
+(via resumes) training session. This can be viewed as similar to maintaining running
+EMA (exponential moving average) of the model weights or performing SWA (stochastic
+weight averaging), but post-training.
+
+Hacked together by / Copyright 2020 Ross Wightman (https://github.com/rwightman)
+"""
+import torch
+import argparse
+import os
+import glob
+import hashlib
+from timm.models import load_state_dict
+try:
+    import safetensors.torch
+    _has_safetensors = True
+except ImportError:
+    _has_safetensors = False
+
+DEFAULT_OUTPUT = "./averaged.pth"
+DEFAULT_SAFE_OUTPUT = "./averaged.safetensors"
+
+parser = argparse.ArgumentParser(description='PyTorch Checkpoint Averager')
+parser.add_argument('--input', default='', type=str, metavar='PATH',
+                    help='path to base input folder containing checkpoints')
+parser.add_argument('--filter', default='*.pth.tar', type=str, metavar='WILDCARD',
+                    help='checkpoint filter (path wildcard)')
+parser.add_argument('--output', default=DEFAULT_OUTPUT, type=str, metavar='PATH',
+                    help=f'Output filename. Defaults to {DEFAULT_SAFE_OUTPUT} when passing --safetensors.')
+parser.add_argument('--no-use-ema', dest='no_use_ema', action='store_true',
+                    help='Force not using ema version of weights (if present)')
+parser.add_argument('--no-sort', dest='no_sort', action='store_true',
+                    help='Do not sort and select by checkpoint metric, also makes "n" argument irrelevant')
+parser.add_argument('-n', type=int, default=10, metavar='N',
+                    help='Number of checkpoints to average')
+parser.add_argument('--safetensors', action='store_true',
+                    help='Save weights using safetensors instead of the default torch way (pickle).')
+
+
+def checkpoint_metric(checkpoint_path):
+    if not checkpoint_path or not os.path.isfile(checkpoint_path):
+        return {}
+    print("=> Extracting metric from checkpoint '{}'".format(checkpoint_path))
+    checkpoint = torch.load(checkpoint_path, map_location='cpu')
+    metric = None
+    if 'metric' in checkpoint:
+        metric = checkpoint['metric']
+    elif 'metrics' in checkpoint and 'metric_name' in checkpoint:
+        metrics = checkpoint['metrics']
+        print(metrics)
+        metric = metrics[checkpoint['metric_name']]
+    return metric
+
+
+def main():
+    args = parser.parse_args()
+    # by default use the EMA weights (if present)
+    args.use_ema = not args.no_use_ema
+    # by default sort by checkpoint metric (if present) and avg top n checkpoints
+    args.sort = not args.no_sort
+
+    if args.safetensors and args.output == DEFAULT_OUTPUT:
+        # Default path changes if using safetensors
+        args.output = DEFAULT_SAFE_OUTPUT
+
+    output, output_ext = os.path.splitext(args.output)
+    if not output_ext:
+        output_ext = ('.safetensors' if args.safetensors else '.pth')
+    output = output + output_ext
+
+    if args.safetensors and not output_ext == ".safetensors":
+        print(
+            "Warning: saving weights as safetensors but output file extension is not "
+            f"set to '.safetensors': {args.output}"
+        )
+
+    if os.path.exists(output):
+        print("Error: Output filename ({}) already exists.".format(output))
+        exit(1)
+
+    pattern = args.input
+    if not args.input.endswith(os.path.sep) and not args.filter.startswith(os.path.sep):
+        pattern += os.path.sep
+    pattern += args.filter
+    checkpoints = glob.glob(pattern, recursive=True)
+
+    if args.sort:
+        checkpoint_metrics = []
+        for c in checkpoints:
+            metric = checkpoint_metric(c)
+            if metric is not None:
+                checkpoint_metrics.append((metric, c))
+        checkpoint_metrics = list(sorted(checkpoint_metrics))
+        checkpoint_metrics = checkpoint_metrics[-args.n:]
+        if checkpoint_metrics:
+            print("Selected checkpoints:")
+            [print(m, c) for m, c in checkpoint_metrics]
+        avg_checkpoints = [c for m, c in checkpoint_metrics]
+    else:
+        avg_checkpoints = checkpoints
+        if avg_checkpoints:
+            print("Selected checkpoints:")
+            [print(c) for c in checkpoints]
+
+    if not avg_checkpoints:
+        print('Error: No checkpoints found to average.')
+        exit(1)
+
+    avg_state_dict = {}
+    avg_counts = {}
+    for c in avg_checkpoints:
+        new_state_dict = load_state_dict(c, args.use_ema)
+        if not new_state_dict:
+            print(f"Error: Checkpoint ({c}) doesn't exist")
+            continue
+        for k, v in new_state_dict.items():
+            if k not in avg_state_dict:
+                avg_state_dict[k] = v.clone().to(dtype=torch.float64)
+                avg_counts[k] = 1
+            else:
+                avg_state_dict[k] += v.to(dtype=torch.float64)
+                avg_counts[k] += 1
+
+    for k, v in avg_state_dict.items():
+        v.div_(avg_counts[k])
+
+    # float32 overflow seems unlikely based on weights seen to date, but who knows
+    float32_info = torch.finfo(torch.float32)
+    final_state_dict = {}
+    for k, v in avg_state_dict.items():
+        v = v.clamp(float32_info.min, float32_info.max)
+        final_state_dict[k] = v.to(dtype=torch.float32)
+
+    if args.safetensors:
+        assert _has_safetensors, "`pip install safetensors` to use .safetensors"
+        safetensors.torch.save_file(final_state_dict, output)
+    else:
+        torch.save(final_state_dict, output)
+
+    with open(output, 'rb') as f:
+        sha_hash = hashlib.sha256(f.read()).hexdigest()
+    print(f"=> Saved state_dict to '{output}, SHA256: {sha_hash}'")
+
+
+if __name__ == '__main__':
+    main()

benchmark.py

@@ -0,0 +1,696 @@
+#!/usr/bin/env python3
+""" Model Benchmark Script
+
+An inference and train step benchmark script for timm models.
+
+Hacked together by Ross Wightman (https://github.com/rwightman)
+"""
+import argparse
+import csv
+import json
+import logging
+import time
+from collections import OrderedDict
+from contextlib import suppress
+from functools import partial
+
+import torch
+import torch.nn as nn
+import torch.nn.parallel
+
+from timm.data import resolve_data_config
+from timm.layers import set_fast_norm
+from timm.models import create_model, is_model, list_models
+from timm.optim import create_optimizer_v2
+from timm.utils import setup_default_logging, set_jit_fuser, decay_batch_step, check_batch_size_retry, ParseKwargs
+
+has_apex = False
+try:
+    from apex import amp
+    has_apex = True
+except ImportError:
+    pass
+
+has_native_amp = False
+try:
+    if getattr(torch.cuda.amp, 'autocast') is not None:
+        has_native_amp = True
+except AttributeError:
+    pass
+
+try:
+    from deepspeed.profiling.flops_profiler import get_model_profile
+    has_deepspeed_profiling = True
+except ImportError as e:
+    has_deepspeed_profiling = False
+
+try:
+    from fvcore.nn import FlopCountAnalysis, flop_count_str, ActivationCountAnalysis
+    has_fvcore_profiling = True
+except ImportError as e:
+    FlopCountAnalysis = None
+    has_fvcore_profiling = False
+
+try:
+    from functorch.compile import memory_efficient_fusion
+    has_functorch = True
+except ImportError as e:
+    has_functorch = False
+
+has_compile = hasattr(torch, 'compile')
+
+if torch.cuda.is_available():
+    torch.backends.cuda.matmul.allow_tf32 = True
+    torch.backends.cudnn.benchmark = True
+_logger = logging.getLogger('validate')
+
+
+parser = argparse.ArgumentParser(description='PyTorch Benchmark')
+
+# benchmark specific args
+parser.add_argument('--model-list', metavar='NAME', default='',
+                    help='txt file based list of model names to benchmark')
+parser.add_argument('--bench', default='both', type=str,
+                    help="Benchmark mode. One of 'inference', 'train', 'both'. Defaults to 'both'")
+parser.add_argument('--detail', action='store_true', default=False,
+                    help='Provide train fwd/bwd/opt breakdown detail if True. Defaults to False')
+parser.add_argument('--no-retry', action='store_true', default=False,
+                    help='Do not decay batch size and retry on error.')
+parser.add_argument('--results-file', default='', type=str,
+                    help='Output csv file for validation results (summary)')
+parser.add_argument('--results-format', default='csv', type=str,
+                    help='Format for results file one of (csv, json) (default: csv).')
+parser.add_argument('--num-warm-iter', default=10, type=int,
+                    help='Number of warmup iterations (default: 10)')
+parser.add_argument('--num-bench-iter', default=40, type=int,
+                    help='Number of benchmark iterations (default: 40)')
+parser.add_argument('--device', default='cuda', type=str,
+                    help="device to run benchmark on")
+
+# common inference / train args
+parser.add_argument('--model', '-m', metavar='NAME', default='resnet50',
+                    help='model architecture (default: resnet50)')
+parser.add_argument('-b', '--batch-size', default=256, type=int,
+                    metavar='N', help='mini-batch size (default: 256)')
+parser.add_argument('--img-size', default=None, type=int,
+                    metavar='N', help='Input image dimension, uses model default if empty')
+parser.add_argument('--input-size', default=None, nargs=3, type=int,
+                    metavar='N N N', help='Input all image dimensions (d h w, e.g. --input-size 3 224 224), uses model default if empty')
+parser.add_argument('--use-train-size', action='store_true', default=False,
+                    help='Run inference at train size, not test-input-size if it exists.')
+parser.add_argument('--num-classes', type=int, default=None,
+                    help='Number classes in dataset')
+parser.add_argument('--gp', default=None, type=str, metavar='POOL',
+                    help='Global pool type, one of (fast, avg, max, avgmax, avgmaxc). Model default if None.')
+parser.add_argument('--channels-last', action='store_true', default=False,
+                    help='Use channels_last memory layout')
+parser.add_argument('--grad-checkpointing', action='store_true', default=False,
+                    help='Enable gradient checkpointing through model blocks/stages')
+parser.add_argument('--amp', action='store_true', default=False,
+                    help='use PyTorch Native AMP for mixed precision training. Overrides --precision arg.')
+parser.add_argument('--amp-dtype', default='float16', type=str,
+                    help='lower precision AMP dtype (default: float16). Overrides --precision arg if args.amp True.')
+parser.add_argument('--precision', default='float32', type=str,
+                    help='Numeric precision. One of (amp, float32, float16, bfloat16, tf32)')
+parser.add_argument('--fuser', default='', type=str,
+                    help="Select jit fuser. One of ('', 'te', 'old', 'nvfuser')")
+parser.add_argument('--fast-norm', default=False, action='store_true',
+                    help='enable experimental fast-norm')
+parser.add_argument('--model-kwargs', nargs='*', default={}, action=ParseKwargs)
+
+# codegen (model compilation) options
+scripting_group = parser.add_mutually_exclusive_group()
+scripting_group.add_argument('--torchscript', dest='torchscript', action='store_true',
+                             help='convert model torchscript for inference')
+scripting_group.add_argument('--torchcompile', nargs='?', type=str, default=None, const='inductor',
+                             help="Enable compilation w/ specified backend (default: inductor).")
+scripting_group.add_argument('--aot-autograd', default=False, action='store_true',
+                             help="Enable AOT Autograd optimization.")
+
+# train optimizer parameters
+parser.add_argument('--opt', default='sgd', type=str, metavar='OPTIMIZER',
+                    help='Optimizer (default: "sgd"')
+parser.add_argument('--opt-eps', default=None, type=float, metavar='EPSILON',
+                    help='Optimizer Epsilon (default: None, use opt default)')
+parser.add_argument('--opt-betas', default=None, type=float, nargs='+', metavar='BETA',
+                    help='Optimizer Betas (default: None, use opt default)')
+parser.add_argument('--momentum', type=float, default=0.9, metavar='M',
+                    help='Optimizer momentum (default: 0.9)')
+parser.add_argument('--weight-decay', type=float, default=0.0001,
+                    help='weight decay (default: 0.0001)')
+parser.add_argument('--clip-grad', type=float, default=None, metavar='NORM',
+                    help='Clip gradient norm (default: None, no clipping)')
+parser.add_argument('--clip-mode', type=str, default='norm',
+                    help='Gradient clipping mode. One of ("norm", "value", "agc")')
+
+
+# model regularization / loss params that impact model or loss fn
+parser.add_argument('--smoothing', type=float, default=0.1,
+                    help='Label smoothing (default: 0.1)')
+parser.add_argument('--drop', type=float, default=0.0, metavar='PCT',
+                    help='Dropout rate (default: 0.)')
+parser.add_argument('--drop-path', type=float, default=None, metavar='PCT',
+                    help='Drop path rate (default: None)')
+parser.add_argument('--drop-block', type=float, default=None, metavar='PCT',
+                    help='Drop block rate (default: None)')
+
+
+def timestamp(sync=False):
+    return time.perf_counter()
+
+
+def cuda_timestamp(sync=False, device=None):
+    if sync:
+        torch.cuda.synchronize(device=device)
+    return time.perf_counter()
+
+
+def count_params(model: nn.Module):
+    return sum([m.numel() for m in model.parameters()])
+
+
+def resolve_precision(precision: str):
+    assert precision in ('amp', 'amp_bfloat16', 'float16', 'bfloat16', 'float32')
+    amp_dtype = None  # amp disabled
+    model_dtype = torch.float32
+    data_dtype = torch.float32
+    if precision == 'amp':
+        amp_dtype = torch.float16
+    elif precision == 'amp_bfloat16':
+        amp_dtype = torch.bfloat16
+    elif precision == 'float16':
+        model_dtype = torch.float16
+        data_dtype = torch.float16
+    elif precision == 'bfloat16':
+        model_dtype = torch.bfloat16
+        data_dtype = torch.bfloat16
+    return amp_dtype, model_dtype, data_dtype
+
+
+def profile_deepspeed(model, input_size=(3, 224, 224), batch_size=1, detailed=False):
+    _, macs, _ = get_model_profile(
+        model=model,
+        input_shape=(batch_size,) + input_size,  # input shape/resolution
+        print_profile=detailed,  # prints the model graph with the measured profile attached to each module
+        detailed=detailed,  # print the detailed profile
+        warm_up=10,  # the number of warm-ups before measuring the time of each module
+        as_string=False,  # print raw numbers (e.g. 1000) or as human-readable strings (e.g. 1k)
+        output_file=None,  # path to the output file. If None, the profiler prints to stdout.
+        ignore_modules=None)  # the list of modules to ignore in the profiling
+    return macs, 0  # no activation count in DS
+
+
+def profile_fvcore(model, input_size=(3, 224, 224), batch_size=1, detailed=False, force_cpu=False):
+    if force_cpu:
+        model = model.to('cpu')
+    device, dtype = next(model.parameters()).device, next(model.parameters()).dtype
+    example_input = torch.ones((batch_size,) + input_size, device=device, dtype=dtype)
+    fca = FlopCountAnalysis(model, example_input)
+    aca = ActivationCountAnalysis(model, example_input)
+    if detailed:
+        fcs = flop_count_str(fca)
+        print(fcs)
+    return fca.total(), aca.total()
+
+
+class BenchmarkRunner:
+    def __init__(
+            self,
+            model_name,
+            detail=False,
+            device='cuda',
+            torchscript=False,
+            torchcompile=None,
+            aot_autograd=False,
+            precision='float32',
+            fuser='',
+            num_warm_iter=10,
+            num_bench_iter=50,
+            use_train_size=False,
+            **kwargs
+    ):
+        self.model_name = model_name
+        self.detail = detail
+        self.device = device
+        self.amp_dtype, self.model_dtype, self.data_dtype = resolve_precision(precision)
+        self.channels_last = kwargs.pop('channels_last', False)
+        if self.amp_dtype is not None:
+            self.amp_autocast = partial(torch.cuda.amp.autocast, dtype=self.amp_dtype)
+        else:
+            self.amp_autocast = suppress
+
+        if fuser:
+            set_jit_fuser(fuser)
+        self.model = create_model(
+            model_name,
+            num_classes=kwargs.pop('num_classes', None),
+            in_chans=3,
+            global_pool=kwargs.pop('gp', 'fast'),
+            scriptable=torchscript,
+            drop_rate=kwargs.pop('drop', 0.),
+            drop_path_rate=kwargs.pop('drop_path', None),
+            drop_block_rate=kwargs.pop('drop_block', None),
+            **kwargs.pop('model_kwargs', {}),
+        )
+        self.model.to(
+            device=self.device,
+            dtype=self.model_dtype,
+            memory_format=torch.channels_last if self.channels_last else None)
+        self.num_classes = self.model.num_classes
+        self.param_count = count_params(self.model)
+        _logger.info('Model %s created, param count: %d' % (model_name, self.param_count))
+
+        data_config = resolve_data_config(kwargs, model=self.model, use_test_size=not use_train_size)
+        self.input_size = data_config['input_size']
+        self.batch_size = kwargs.pop('batch_size', 256)
+
+        self.compiled = False
+        if torchscript:
+            self.model = torch.jit.script(self.model)
+            self.compiled = True
+        elif torchcompile:
+            assert has_compile, 'A version of torch w/ torch.compile() is required, possibly a nightly.'
+            torch._dynamo.reset()
+            self.model = torch.compile(self.model, backend=torchcompile)
+            self.compiled = True
+        elif aot_autograd:
+            assert has_functorch, "functorch is needed for --aot-autograd"
+            self.model = memory_efficient_fusion(self.model)
+            self.compiled = True
+
+        self.example_inputs = None
+        self.num_warm_iter = num_warm_iter
+        self.num_bench_iter = num_bench_iter
+        self.log_freq = num_bench_iter // 5
+        if 'cuda' in self.device:
+            self.time_fn = partial(cuda_timestamp, device=self.device)
+        else:
+            self.time_fn = timestamp
+
+    def _init_input(self):
+        self.example_inputs = torch.randn(
+            (self.batch_size,) + self.input_size, device=self.device, dtype=self.data_dtype)
+        if self.channels_last:
+            self.example_inputs = self.example_inputs.contiguous(memory_format=torch.channels_last)
+
+
+class InferenceBenchmarkRunner(BenchmarkRunner):
+
+    def __init__(
+            self,
+            model_name,
+            device='cuda',
+            torchscript=False,
+            **kwargs
+    ):
+        super().__init__(model_name=model_name, device=device, torchscript=torchscript, **kwargs)
+        self.model.eval()
+
+    def run(self):
+        def _step():
+            t_step_start = self.time_fn()
+            with self.amp_autocast():
+                output = self.model(self.example_inputs)
+            t_step_end = self.time_fn(True)
+            return t_step_end - t_step_start
+
+        _logger.info(
+            f'Running inference benchmark on {self.model_name} for {self.num_bench_iter} steps w/ '
+            f'input size {self.input_size} and batch size {self.batch_size}.')
+
+        with torch.no_grad():
+            self._init_input()
+
+            for _ in range(self.num_warm_iter):
+                _step()
+
+            total_step = 0.
+            num_samples = 0
+            t_run_start = self.time_fn()
+            for i in range(self.num_bench_iter):
+                delta_fwd = _step()
+                total_step += delta_fwd
+                num_samples += self.batch_size
+                num_steps = i + 1
+                if num_steps % self.log_freq == 0:
+                    _logger.info(
+                        f"Infer [{num_steps}/{self.num_bench_iter}]."
+                        f" {num_samples / total_step:0.2f} samples/sec."
+                        f" {1000 * total_step / num_steps:0.3f} ms/step.")
+            t_run_end = self.time_fn(True)
+            t_run_elapsed = t_run_end - t_run_start
+
+        results = dict(
+            samples_per_sec=round(num_samples / t_run_elapsed, 2),
+            step_time=round(1000 * total_step / self.num_bench_iter, 3),
+            batch_size=self.batch_size,
+            img_size=self.input_size[-1],
+            param_count=round(self.param_count / 1e6, 2),
+        )
+
+        retries = 0 if self.compiled else 2  # skip profiling if model is scripted
+        while retries:
+            retries -= 1
+            try:
+                if has_deepspeed_profiling:
+                    macs, _ = profile_deepspeed(self.model, self.input_size)
+                    results['gmacs'] = round(macs / 1e9, 2)
+                elif has_fvcore_profiling:
+                    macs, activations = profile_fvcore(self.model, self.input_size, force_cpu=not retries)
+                    results['gmacs'] = round(macs / 1e9, 2)
+                    results['macts'] = round(activations / 1e6, 2)
+            except RuntimeError as e:
+                pass
+
+        _logger.info(
+            f"Inference benchmark of {self.model_name} done. "
+            f"{results['samples_per_sec']:.2f} samples/sec, {results['step_time']:.2f} ms/step")
+
+        return results
+
+
+class TrainBenchmarkRunner(BenchmarkRunner):
+
+    def __init__(
+            self,
+            model_name,
+            device='cuda',
+            torchscript=False,
+            **kwargs
+    ):
+        super().__init__(model_name=model_name, device=device, torchscript=torchscript, **kwargs)
+        self.model.train()
+
+        self.loss = nn.CrossEntropyLoss().to(self.device)
+        self.target_shape = tuple()
+
+        self.optimizer = create_optimizer_v2(
+            self.model,
+            opt=kwargs.pop('opt', 'sgd'),
+            lr=kwargs.pop('lr', 1e-4))
+
+        if kwargs.pop('grad_checkpointing', False):
+            self.model.set_grad_checkpointing()
+
+    def _gen_target(self, batch_size):
+        return torch.empty(
+            (batch_size,) + self.target_shape, device=self.device, dtype=torch.long).random_(self.num_classes)
+
+    def run(self):
+        def _step(detail=False):
+            self.optimizer.zero_grad()  # can this be ignored?
+            t_start = self.time_fn()
+            t_fwd_end = t_start
+            t_bwd_end = t_start
+            with self.amp_autocast():
+                output = self.model(self.example_inputs)
+                if isinstance(output, tuple):
+                    output = output[0]
+                if detail:
+                    t_fwd_end = self.time_fn(True)
+                target = self._gen_target(output.shape[0])
+                self.loss(output, target).backward()
+                if detail:
+                    t_bwd_end = self.time_fn(True)
+            self.optimizer.step()
+            t_end = self.time_fn(True)
+            if detail:
+                delta_fwd = t_fwd_end - t_start
+                delta_bwd = t_bwd_end - t_fwd_end
+                delta_opt = t_end - t_bwd_end
+                return delta_fwd, delta_bwd, delta_opt
+            else:
+                delta_step = t_end - t_start
+                return delta_step
+
+        _logger.info(
+            f'Running train benchmark on {self.model_name} for {self.num_bench_iter} steps w/ '
+            f'input size {self.input_size} and batch size {self.batch_size}.')
+
+        self._init_input()
+
+        for _ in range(self.num_warm_iter):
+            _step()
+
+        t_run_start = self.time_fn()
+        if self.detail:
+            total_fwd = 0.
+            total_bwd = 0.
+            total_opt = 0.
+            num_samples = 0
+            for i in range(self.num_bench_iter):
+                delta_fwd, delta_bwd, delta_opt = _step(True)
+                num_samples += self.batch_size
+                total_fwd += delta_fwd
+                total_bwd += delta_bwd
+                total_opt += delta_opt
+                num_steps = (i + 1)
+                if num_steps % self.log_freq == 0:
+                    total_step = total_fwd + total_bwd + total_opt
+                    _logger.info(
+                        f"Train [{num_steps}/{self.num_bench_iter}]."
+                        f" {num_samples / total_step:0.2f} samples/sec."
+                        f" {1000 * total_fwd / num_steps:0.3f} ms/step fwd,"
+                        f" {1000 * total_bwd / num_steps:0.3f} ms/step bwd,"
+                        f" {1000 * total_opt / num_steps:0.3f} ms/step opt."
+                    )
+            total_step = total_fwd + total_bwd + total_opt
+            t_run_elapsed = self.time_fn() - t_run_start
+            results = dict(
+                samples_per_sec=round(num_samples / t_run_elapsed, 2),
+                step_time=round(1000 * total_step / self.num_bench_iter, 3),
+                fwd_time=round(1000 * total_fwd / self.num_bench_iter, 3),
+                bwd_time=round(1000 * total_bwd / self.num_bench_iter, 3),
+                opt_time=round(1000 * total_opt / self.num_bench_iter, 3),
+                batch_size=self.batch_size,
+                img_size=self.input_size[-1],
+                param_count=round(self.param_count / 1e6, 2),
+            )
+        else:
+            total_step = 0.
+            num_samples = 0
+            for i in range(self.num_bench_iter):
+                delta_step = _step(False)
+                num_samples += self.batch_size
+                total_step += delta_step
+                num_steps = (i + 1)
+                if num_steps % self.log_freq == 0:
+                    _logger.info(
+                        f"Train [{num_steps}/{self.num_bench_iter}]."
+                        f" {num_samples / total_step:0.2f} samples/sec."
+                        f" {1000 * total_step / num_steps:0.3f} ms/step.")
+            t_run_elapsed = self.time_fn() - t_run_start
+            results = dict(
+                samples_per_sec=round(num_samples / t_run_elapsed, 2),
+                step_time=round(1000 * total_step / self.num_bench_iter, 3),
+                batch_size=self.batch_size,
+                img_size=self.input_size[-1],
+                param_count=round(self.param_count / 1e6, 2),
+            )
+
+        _logger.info(
+            f"Train benchmark of {self.model_name} done. "
+            f"{results['samples_per_sec']:.2f} samples/sec, {results['step_time']:.2f} ms/sample")
+
+        return results
+
+
+class ProfileRunner(BenchmarkRunner):
+
+    def __init__(self, model_name, device='cuda', profiler='', **kwargs):
+        super().__init__(model_name=model_name, device=device, **kwargs)
+        if not profiler:
+            if has_deepspeed_profiling:
+                profiler = 'deepspeed'
+            elif has_fvcore_profiling:
+                profiler = 'fvcore'
+        assert profiler, "One of deepspeed or fvcore needs to be installed for profiling to work."
+        self.profiler = profiler
+        self.model.eval()
+
+    def run(self):
+        _logger.info(
+            f'Running profiler on {self.model_name} w/ '
+            f'input size {self.input_size} and batch size {self.batch_size}.')
+
+        macs = 0
+        activations = 0
+        if self.profiler == 'deepspeed':
+            macs, _ = profile_deepspeed(self.model, self.input_size, batch_size=self.batch_size, detailed=True)
+        elif self.profiler == 'fvcore':
+            macs, activations = profile_fvcore(self.model, self.input_size, batch_size=self.batch_size, detailed=True)
+
+        results = dict(
+            gmacs=round(macs / 1e9, 2),
+            macts=round(activations / 1e6, 2),
+            batch_size=self.batch_size,
+            img_size=self.input_size[-1],
+            param_count=round(self.param_count / 1e6, 2),
+        )
+
+        _logger.info(
+            f"Profile of {self.model_name} done. "
+            f"{results['gmacs']:.2f} GMACs, {results['param_count']:.2f} M params.")
+
+        return results
+
+
+def _try_run(
+        model_name,
+        bench_fn,
+        bench_kwargs,
+        initial_batch_size,
+        no_batch_size_retry=False
+):
+    batch_size = initial_batch_size
+    results = dict()
+    error_str = 'Unknown'
+    while batch_size:
+        try:
+            torch.cuda.empty_cache()
+            bench = bench_fn(model_name=model_name, batch_size=batch_size, **bench_kwargs)
+            results = bench.run()
+            return results
+        except RuntimeError as e:
+            error_str = str(e)
+            _logger.error(f'"{error_str}" while running benchmark.')
+            if not check_batch_size_retry(error_str):
+                _logger.error(f'Unrecoverable error encountered while benchmarking {model_name}, skipping.')
+                break
+            if no_batch_size_retry:
+                break
+        batch_size = decay_batch_step(batch_size)
+        _logger.warning(f'Reducing batch size to {batch_size} for retry.')
+    results['error'] = error_str
+    return results
+
+
+def benchmark(args):
+    if args.amp:
+        _logger.warning("Overriding precision to 'amp' since --amp flag set.")
+        args.precision = 'amp' if args.amp_dtype == 'float16' else '_'.join(['amp', args.amp_dtype])
+    _logger.info(f'Benchmarking in {args.precision} precision. '
+                 f'{"NHWC" if args.channels_last else "NCHW"} layout. '
+                 f'torchscript {"enabled" if args.torchscript else "disabled"}')
+
+    bench_kwargs = vars(args).copy()
+    bench_kwargs.pop('amp')
+    model = bench_kwargs.pop('model')
+    batch_size = bench_kwargs.pop('batch_size')
+
+    bench_fns = (InferenceBenchmarkRunner,)
+    prefixes = ('infer',)
+    if args.bench == 'both':
+        bench_fns = (
+            InferenceBenchmarkRunner,
+            TrainBenchmarkRunner
+        )
+        prefixes = ('infer', 'train')
+    elif args.bench == 'train':
+        bench_fns = TrainBenchmarkRunner,
+        prefixes = 'train',
+    elif args.bench.startswith('profile'):
+        # specific profiler used if included in bench mode string, otherwise default to deepspeed, fallback to fvcore
+        if 'deepspeed' in args.bench:
+            assert has_deepspeed_profiling, "deepspeed must be installed to use deepspeed flop counter"
+            bench_kwargs['profiler'] = 'deepspeed'
+        elif 'fvcore' in args.bench:
+            assert has_fvcore_profiling, "fvcore must be installed to use fvcore flop counter"
+            bench_kwargs['profiler'] = 'fvcore'
+        bench_fns = ProfileRunner,
+        batch_size = 1
+
+    model_results = OrderedDict(model=model)
+    for prefix, bench_fn in zip(prefixes, bench_fns):
+        run_results = _try_run(
+            model,
+            bench_fn,
+            bench_kwargs=bench_kwargs,
+            initial_batch_size=batch_size,
+            no_batch_size_retry=args.no_retry,
+        )
+        if prefix and 'error' not in run_results:
+            run_results = {'_'.join([prefix, k]): v for k, v in run_results.items()}
+        model_results.update(run_results)
+        if 'error' in run_results:
+            break
+    if 'error' not in model_results:
+        param_count = model_results.pop('infer_param_count', model_results.pop('train_param_count', 0))
+        model_results.setdefault('param_count', param_count)
+        model_results.pop('train_param_count', 0)
+    return model_results
+
+
+def main():
+    setup_default_logging()
+    args = parser.parse_args()
+    model_cfgs = []
+    model_names = []
+
+    if args.fast_norm:
+        set_fast_norm()
+
+    if args.model_list:
+        args.model = ''
+        with open(args.model_list) as f:
+            model_names = [line.rstrip() for line in f]
+        model_cfgs = [(n, None) for n in model_names]
+    elif args.model == 'all':
+        # validate all models in a list of names with pretrained checkpoints
+        args.pretrained = True
+        model_names = list_models(pretrained=True, exclude_filters=['*in21k'])
+        model_cfgs = [(n, None) for n in model_names]
+    elif not is_model(args.model):
+        # model name doesn't exist, try as wildcard filter
+        model_names = list_models(args.model)
+        model_cfgs = [(n, None) for n in model_names]
+
+    if len(model_cfgs):
+        _logger.info('Running bulk validation on these pretrained models: {}'.format(', '.join(model_names)))
+        results = []
+        try:
+            for m, _ in model_cfgs:
+                if not m:
+                    continue
+                args.model = m
+                r = benchmark(args)
+                if r:
+                    results.append(r)
+                time.sleep(10)
+        except KeyboardInterrupt as e:
+            pass
+        sort_key = 'infer_samples_per_sec'
+        if 'train' in args.bench:
+            sort_key = 'train_samples_per_sec'
+        elif 'profile' in args.bench:
+            sort_key = 'infer_gmacs'
+        results = filter(lambda x: sort_key in x, results)
+        results = sorted(results, key=lambda x: x[sort_key], reverse=True)
+    else:
+        results = benchmark(args)
+
+    if args.results_file:
+        write_results(args.results_file, results, format=args.results_format)
+
+    # output results in JSON to stdout w/ delimiter for runner script
+    print(f'--result\n{json.dumps(results, indent=4)}')
+
+
+def write_results(results_file, results, format='csv'):
+    with open(results_file, mode='w') as cf:
+        if format == 'json':
+            json.dump(results, cf, indent=4)
+        else:
+            if not isinstance(results, (list, tuple)):
+                results = [results]
+            if not results:
+                return
+            dw = csv.DictWriter(cf, fieldnames=results[0].keys())
+            dw.writeheader()
+            for r in results:
+                dw.writerow(r)
+            cf.flush()
+
+
+if __name__ == '__main__':
+    main()

bulk_runner.py

@@ -0,0 +1,184 @@
+#!/usr/bin/env python3
+""" Bulk Model Script Runner
+
+Run validation or benchmark script in separate process for each model
+
+Benchmark all 'vit*' models:
+python bulk_runner.py  --model-list 'vit*' --results-file vit_bench.csv benchmark.py --amp -b 512
+
+Validate all models:
+python bulk_runner.py  --model-list all --results-file val.csv --pretrained validate.py /imagenet/validation/ --amp -b 512 --retry
+
+Hacked together by Ross Wightman (https://github.com/rwightman)
+"""
+import argparse
+import os
+import sys
+import csv
+import json
+import subprocess
+import time
+from typing import Callable, List, Tuple, Union
+
+
+from timm.models import is_model, list_models
+
+
+parser = argparse.ArgumentParser(description='Per-model process launcher')
+
+# model and results args
+parser.add_argument(
+    '--model-list', metavar='NAME', default='',
+    help='txt file based list of model names to benchmark')
+parser.add_argument(
+    '--results-file', default='', type=str, metavar='FILENAME',
+    help='Output csv file for validation results (summary)')
+parser.add_argument(
+    '--sort-key', default='', type=str, metavar='COL',
+    help='Specify sort key for results csv')
+parser.add_argument(
+    "--pretrained", action='store_true',
+    help="only run models with pretrained weights")
+
+parser.add_argument(
+    "--delay",
+    type=float,
+    default=0,
+    help="Interval, in seconds, to delay between model invocations.",
+)
+parser.add_argument(
+    "--start_method", type=str, default="spawn", choices=["spawn", "fork", "forkserver"],
+    help="Multiprocessing start method to use when creating workers.",
+)
+parser.add_argument(
+    "--no_python",
+    help="Skip prepending the script with 'python' - just execute it directly. Useful "
+         "when the script is not a Python script.",
+)
+parser.add_argument(
+    "-m",
+    "--module",
+    help="Change each process to interpret the launch script as a Python module, executing "
+         "with the same behavior as 'python -m'.",
+)
+
+# positional
+parser.add_argument(
+    "script", type=str,
+    help="Full path to the program/script to be launched for each model config.",
+)
+parser.add_argument("script_args", nargs=argparse.REMAINDER)
+
+
+def cmd_from_args(args) -> Tuple[Union[Callable, str], List[str]]:
+    # If ``args`` not passed, defaults to ``sys.argv[:1]``
+    with_python = not args.no_python
+    cmd: Union[Callable, str]
+    cmd_args = []
+    if with_python:
+        cmd = os.getenv("PYTHON_EXEC", sys.executable)
+        cmd_args.append("-u")
+        if args.module:
+            cmd_args.append("-m")
+        cmd_args.append(args.script)
+    else:
+        if args.module:
+            raise ValueError(
+                "Don't use both the '--no_python' flag"
+                " and the '--module' flag at the same time."
+            )
+        cmd = args.script
+    cmd_args.extend(args.script_args)
+
+    return cmd, cmd_args
+
+
+def main():
+    args = parser.parse_args()
+    cmd, cmd_args = cmd_from_args(args)
+
+    model_cfgs = []
+    model_names = []
+    if args.model_list == 'all':
+        # NOTE should make this config, for validation / benchmark runs the focus is 1k models,
+        # so we filter out 21/22k and some other unusable heads. This will change in the future...
+        exclude_model_filters = ['*in21k', '*in22k', '*dino', '*_22k']
+        model_names = list_models(
+            pretrained=args.pretrained,  # only include models w/ pretrained checkpoints if set
+            exclude_filters=exclude_model_filters
+        )
+        model_cfgs = [(n, None) for n in model_names]
+    elif not is_model(args.model_list):
+        # model name doesn't exist, try as wildcard filter
+        model_names = list_models(args.model_list)
+        model_cfgs = [(n, None) for n in model_names]
+
+    if not model_cfgs and os.path.exists(args.model_list):
+        with open(args.model_list) as f:
+            model_names = [line.rstrip() for line in f]
+            model_cfgs = [(n, None) for n in model_names]
+
+    if len(model_cfgs):
+        results_file = args.results_file or './results.csv'
+        results = []
+        errors = []
+        print('Running script on these models: {}'.format(', '.join(model_names)))
+        if not args.sort_key:
+            if 'benchmark' in args.script:
+                if any(['train' in a for a in args.script_args]):
+                    sort_key = 'train_samples_per_sec'
+                else:
+                    sort_key = 'infer_samples_per_sec'
+            else:
+                sort_key = 'top1'
+        else:
+            sort_key = args.sort_key
+        print(f'Script: {args.script}, Args: {args.script_args}, Sort key: {sort_key}')
+
+        try:
+            for m, _ in model_cfgs:
+                if not m:
+                    continue
+                args_str = (cmd, *[str(e) for e in cmd_args], '--model', m)
+                try:
+                    o = subprocess.check_output(args=args_str).decode('utf-8').split('--result')[-1]
+                    r = json.loads(o)
+                    results.append(r)
+                except Exception as e:
+                    # FIXME batch_size retry loop is currently done in either validation.py or benchmark.py
+                    # for further robustness (but more overhead), we may want to manage that by looping here...
+                    errors.append(dict(model=m, error=str(e)))
+                if args.delay:
+                    time.sleep(args.delay)
+        except KeyboardInterrupt as e:
+            pass
+
+        errors.extend(list(filter(lambda x: 'error' in x, results)))
+        if errors:
+            print(f'{len(errors)} models had errors during run.')
+            for e in errors:
+                print(f"\t {e['model']} ({e.get('error', 'Unknown')})")
+        results = list(filter(lambda x: 'error' not in x, results))
+
+        no_sortkey = list(filter(lambda x: sort_key not in x, results))
+        if no_sortkey:
+            print(f'{len(no_sortkey)} results missing sort key, skipping sort.')
+        else:
+            results = sorted(results, key=lambda x: x[sort_key], reverse=True)
+
+        if len(results):
+            print(f'{len(results)} models run successfully. Saving results to {results_file}.')
+            write_results(results_file, results)
+
+
+def write_results(results_file, results):
+    with open(results_file, mode='w') as cf:
+        dw = csv.DictWriter(cf, fieldnames=results[0].keys())
+        dw.writeheader()
+        for r in results:
+            dw.writerow(r)
+        cf.flush()
+
+
+if __name__ == '__main__':
+    main()

clean_checkpoint.py

@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+""" Checkpoint Cleaning Script
+
+Takes training checkpoints with GPU tensors, optimizer state, extra dict keys, etc.
+and outputs a CPU  tensor checkpoint with only the `state_dict` along with SHA256
+calculation for model zoo compatibility.
+
+Hacked together by / Copyright 2020 Ross Wightman (https://github.com/rwightman)
+"""
+import torch
+import argparse
+import os
+import hashlib
+import shutil
+import tempfile
+from timm.models import load_state_dict
+try:
+    import safetensors.torch
+    _has_safetensors = True
+except ImportError:
+    _has_safetensors = False
+
+parser = argparse.ArgumentParser(description='PyTorch Checkpoint Cleaner')
+parser.add_argument('--checkpoint', default='', type=str, metavar='PATH',
+                    help='path to latest checkpoint (default: none)')
+parser.add_argument('--output', default='', type=str, metavar='PATH',
+                    help='output path')
+parser.add_argument('--no-use-ema', dest='no_use_ema', action='store_true',
+                    help='use ema version of weights if present')
+parser.add_argument('--no-hash', dest='no_hash', action='store_true',
+                    help='no hash in output filename')
+parser.add_argument('--clean-aux-bn', dest='clean_aux_bn', action='store_true',
+                    help='remove auxiliary batch norm layers (from SplitBN training) from checkpoint')
+parser.add_argument('--safetensors', action='store_true',
+                    help='Save weights using safetensors instead of the default torch way (pickle).')
+
+
+def main():
+    args = parser.parse_args()
+
+    if os.path.exists(args.output):
+        print("Error: Output filename ({}) already exists.".format(args.output))
+        exit(1)
+
+    clean_checkpoint(
+        args.checkpoint,
+        args.output,
+        not args.no_use_ema,
+        args.no_hash,
+        args.clean_aux_bn,
+        safe_serialization=args.safetensors,
+    )
+
+
+def clean_checkpoint(
+        checkpoint,
+        output,
+        use_ema=True,
+        no_hash=False,
+        clean_aux_bn=False,
+        safe_serialization: bool=False,
+):
+    # Load an existing checkpoint to CPU, strip everything but the state_dict and re-save
+    if checkpoint and os.path.isfile(checkpoint):
+        print("=> Loading checkpoint '{}'".format(checkpoint))
+        state_dict = load_state_dict(checkpoint, use_ema=use_ema)
+        new_state_dict = {}
+        for k, v in state_dict.items():
+            if clean_aux_bn and 'aux_bn' in k:
+                # If all aux_bn keys are removed, the SplitBN layers will end up as normal and
+                # load with the unmodified model using BatchNorm2d.
+                continue
+            name = k[7:] if k.startswith('module.') else k
+            new_state_dict[name] = v
+        print("=> Loaded state_dict from '{}'".format(checkpoint))
+
+        ext = ''
+        if output:
+            checkpoint_root, checkpoint_base = os.path.split(output)
+            checkpoint_base, ext = os.path.splitext(checkpoint_base)
+        else:
+            checkpoint_root = ''
+            checkpoint_base = os.path.split(checkpoint)[1]
+            checkpoint_base = os.path.splitext(checkpoint_base)[0]
+
+        temp_filename = '__' + checkpoint_base
+        if safe_serialization:
+            assert _has_safetensors, "`pip install safetensors` to use .safetensors"
+            safetensors.torch.save_file(new_state_dict, temp_filename)
+        else:
+            torch.save(new_state_dict, temp_filename)
+
+        with open(temp_filename, 'rb') as f:
+            sha_hash = hashlib.sha256(f.read()).hexdigest()
+
+        if ext:
+            final_ext = ext
+        else:
+            final_ext = ('.safetensors' if safe_serialization else '.pth')
+
+        if no_hash:
+            final_filename = checkpoint_base + final_ext
+        else:
+            final_filename = '-'.join([checkpoint_base, sha_hash[:8]]) + final_ext
+
+        shutil.move(temp_filename, os.path.join(checkpoint_root, final_filename))
+        print("=> Saved state_dict to '{}, SHA256: {}'".format(final_filename, sha_hash))
+        return final_filename
+    else:
+        print("Error: Checkpoint ({}) doesn't exist".format(checkpoint))
+        return ''
+
+
+if __name__ == '__main__':
+    main()

convert/convert_from_mxnet.py

@@ -0,0 +1,107 @@
+import argparse
+import hashlib
+import os
+
+import mxnet as mx
+import gluoncv
+import torch
+from timm import create_model
+
+parser = argparse.ArgumentParser(description='Convert from MXNet')
+parser.add_argument('--model', default='all', type=str, metavar='MODEL',
+                    help='Name of model to train (default: "all"')
+
+
+def convert(mxnet_name, torch_name):
+    # download and load the pre-trained model
+    net = gluoncv.model_zoo.get_model(mxnet_name, pretrained=True)
+
+    # create corresponding torch model
+    torch_net = create_model(torch_name)
+
+    mxp = [(k, v) for k, v in net.collect_params().items() if 'running' not in k]
+    torchp = list(torch_net.named_parameters())
+    torch_params = {}
+
+    # convert parameters
+    # NOTE: we are relying on the fact that the order of parameters
+    # are usually exactly the same between these models, thus no key name mapping
+    # is necessary. Asserts will trip if this is not the case.
+    for (tn, tv), (mn, mv) in zip(torchp, mxp):
+        m_split = mn.split('_')
+        t_split = tn.split('.')
+        print(t_split, m_split)
+        print(tv.shape, mv.shape)
+
+        # ensure ordering of BN params match since their sizes are not specific
+        if m_split[-1] == 'gamma':
+            assert t_split[-1] == 'weight'
+        if m_split[-1] == 'beta':
+            assert t_split[-1] == 'bias'
+
+        # ensure shapes match
+        assert all(t == m for t, m in zip(tv.shape, mv.shape))
+
+        torch_tensor = torch.from_numpy(mv.data().asnumpy())
+        torch_params[tn] = torch_tensor
+
+    # convert buffers (batch norm running stats)
+    mxb = [(k, v) for k, v in net.collect_params().items() if any(x in k for x in ['running_mean', 'running_var'])]
+    torchb = [(k, v) for k, v in torch_net.named_buffers() if 'num_batches' not in k]
+    for (tn, tv), (mn, mv) in zip(torchb, mxb):
+        print(tn, mn)
+        print(tv.shape, mv.shape)
+
+        # ensure ordering of BN params match since their sizes are not specific
+        if 'running_var' in tn:
+            assert 'running_var' in mn
+        if 'running_mean' in tn:
+            assert 'running_mean' in mn
+            
+        torch_tensor = torch.from_numpy(mv.data().asnumpy())
+        torch_params[tn] = torch_tensor
+
+    torch_net.load_state_dict(torch_params)
+    torch_filename = './%s.pth' % torch_name
+    torch.save(torch_net.state_dict(), torch_filename)
+    with open(torch_filename, 'rb') as f:
+        sha_hash = hashlib.sha256(f.read()).hexdigest()
+    final_filename = os.path.splitext(torch_filename)[0] + '-' + sha_hash[:8] + '.pth'
+    os.rename(torch_filename, final_filename)
+    print("=> Saved converted model to '{}, SHA256: {}'".format(final_filename, sha_hash))
+
+
+def map_mx_to_torch_model(mx_name):
+    torch_name = mx_name.lower()
+    if torch_name.startswith('se_'):
+        torch_name = torch_name.replace('se_', 'se')
+    elif torch_name.startswith('senet_'):
+        torch_name = torch_name.replace('senet_', 'senet')
+    elif torch_name.startswith('inceptionv3'):
+        torch_name = torch_name.replace('inceptionv3', 'inception_v3')
+    torch_name = 'gluon_' + torch_name
+    return torch_name
+
+
+ALL = ['resnet18_v1b', 'resnet34_v1b', 'resnet50_v1b', 'resnet101_v1b', 'resnet152_v1b',
+       'resnet50_v1c', 'resnet101_v1c', 'resnet152_v1c', 'resnet50_v1d', 'resnet101_v1d', 'resnet152_v1d',
+       #'resnet50_v1e', 'resnet101_v1e', 'resnet152_v1e',
+       'resnet50_v1s', 'resnet101_v1s', 'resnet152_v1s', 'resnext50_32x4d', 'resnext101_32x4d', 'resnext101_64x4d',
+       'se_resnext50_32x4d', 'se_resnext101_32x4d', 'se_resnext101_64x4d', 'senet_154', 'inceptionv3']
+
+
+def main():
+    args = parser.parse_args()
+
+    if not args.model or args.model == 'all':
+        for mx_model in ALL:
+            torch_model = map_mx_to_torch_model(mx_model)
+            convert(mx_model, torch_model)
+    else:
+        mx_model = args.model
+        torch_model = map_mx_to_torch_model(mx_model)
+        convert(mx_model, torch_model)
+
+
+if __name__ == '__main__':
+    main()

convert/convert_nest_flax.py

@@ -0,0 +1,109 @@
+"""
+Convert weights from https://github.com/google-research/nested-transformer
+NOTE: You'll need https://github.com/google/CommonLoopUtils, not included in requirements.txt
+"""
+
+import sys
+
+import numpy as np
+import torch
+
+from clu import checkpoint
+
+
+arch_depths = {
+    'nest_base': [2, 2, 20],
+    'nest_small': [2, 2, 20],
+    'nest_tiny': [2, 2, 8],
+}
+
+
+def convert_nest(checkpoint_path, arch):
+    """
+    Expects path to checkpoint which is a dir containing 4 files like in each of these folders
+        - https://console.cloud.google.com/storage/browser/gresearch/nest-checkpoints
+    `arch` is needed to 
+    Returns a state dict that can be used with `torch.nn.Module.load_state_dict`
+    Hint: Follow timm.models.nest.Nest.__init__ and 
+    https://github.com/google-research/nested-transformer/blob/main/models/nest_net.py
+    """
+    assert arch in ['nest_base', 'nest_small', 'nest_tiny'], "Your `arch` is not supported"
+
+    flax_dict = checkpoint.load_state_dict(checkpoint_path)['optimizer']['target']
+    state_dict = {}
+
+    # Patch embedding
+    state_dict['patch_embed.proj.weight'] = torch.tensor(
+        flax_dict['PatchEmbedding_0']['Conv_0']['kernel']).permute(3, 2, 0, 1)
+    state_dict['patch_embed.proj.bias'] = torch.tensor(flax_dict['PatchEmbedding_0']['Conv_0']['bias'])
+    
+    # Positional embeddings
+    posemb_keys = [k for k in flax_dict.keys() if k.startswith('PositionEmbedding')]
+    for i, k in enumerate(posemb_keys):
+        state_dict[f'levels.{i}.pos_embed'] = torch.tensor(flax_dict[k]['pos_embedding'])
+    
+    # Transformer encoders
+    depths = arch_depths[arch]
+    for level in range(len(depths)):
+        for layer in range(depths[level]):
+            global_layer_ix = sum(depths[:level]) + layer
+            # Norms
+            for i in range(2):
+                state_dict[f'levels.{level}.transformer_encoder.{layer}.norm{i+1}.weight'] = torch.tensor(
+                    flax_dict[f'EncoderNDBlock_{global_layer_ix}'][f'LayerNorm_{i}']['scale'])
+                state_dict[f'levels.{level}.transformer_encoder.{layer}.norm{i+1}.bias'] = torch.tensor(
+                    flax_dict[f'EncoderNDBlock_{global_layer_ix}'][f'LayerNorm_{i}']['bias'])
+            # Attention qkv
+            w_q = flax_dict[f'EncoderNDBlock_{global_layer_ix}']['MultiHeadAttention_0']['DenseGeneral_0']['kernel']
+            w_kv = flax_dict[f'EncoderNDBlock_{global_layer_ix}']['MultiHeadAttention_0']['DenseGeneral_1']['kernel']
+            # Pay attention to dims here (maybe get pen and paper)
+            w_kv = np.concatenate(np.split(w_kv, 2, -1), 1)
+            w_qkv = np.concatenate([w_q, w_kv], 1)
+            state_dict[f'levels.{level}.transformer_encoder.{layer}.attn.qkv.weight'] = torch.tensor(w_qkv).flatten(1).permute(1,0)
+            b_q = flax_dict[f'EncoderNDBlock_{global_layer_ix}']['MultiHeadAttention_0']['DenseGeneral_0']['bias']
+            b_kv = flax_dict[f'EncoderNDBlock_{global_layer_ix}']['MultiHeadAttention_0']['DenseGeneral_1']['bias']
+            # Pay attention to dims here (maybe get pen and paper)
+            b_kv = np.concatenate(np.split(b_kv, 2, -1), 0)
+            b_qkv = np.concatenate([b_q, b_kv], 0)
+            state_dict[f'levels.{level}.transformer_encoder.{layer}.attn.qkv.bias'] = torch.tensor(b_qkv).reshape(-1)
+            # Attention proj
+            w_proj = flax_dict[f'EncoderNDBlock_{global_layer_ix}']['MultiHeadAttention_0']['proj_kernel']
+            w_proj = torch.tensor(w_proj).permute(2, 1, 0).flatten(1)
+            state_dict[f'levels.{level}.transformer_encoder.{layer}.attn.proj.weight'] = w_proj
+            state_dict[f'levels.{level}.transformer_encoder.{layer}.attn.proj.bias'] = torch.tensor(
+                flax_dict[f'EncoderNDBlock_{global_layer_ix}']['MultiHeadAttention_0']['bias'])
+            # MLP
+            for i in range(2):
+                state_dict[f'levels.{level}.transformer_encoder.{layer}.mlp.fc{i+1}.weight'] = torch.tensor(
+                    flax_dict[f'EncoderNDBlock_{global_layer_ix}']['MlpBlock_0'][f'Dense_{i}']['kernel']).permute(1, 0)
+                state_dict[f'levels.{level}.transformer_encoder.{layer}.mlp.fc{i+1}.bias'] = torch.tensor(
+                    flax_dict[f'EncoderNDBlock_{global_layer_ix}']['MlpBlock_0'][f'Dense_{i}']['bias'])
+
+    # Block aggregations (ConvPool)
+    for level in range(1, len(depths)):
+        # Convs
+        state_dict[f'levels.{level}.pool.conv.weight'] = torch.tensor(
+            flax_dict[f'ConvPool_{level-1}']['Conv_0']['kernel']).permute(3, 2, 0, 1)
+        state_dict[f'levels.{level}.pool.conv.bias'] = torch.tensor(
+            flax_dict[f'ConvPool_{level-1}']['Conv_0']['bias'])
+        # Norms
+        state_dict[f'levels.{level}.pool.norm.weight'] = torch.tensor(
+                    flax_dict[f'ConvPool_{level-1}']['LayerNorm_0']['scale'])
+        state_dict[f'levels.{level}.pool.norm.bias'] = torch.tensor(
+                    flax_dict[f'ConvPool_{level-1}']['LayerNorm_0']['bias'])
+
+    # Final norm
+    state_dict[f'norm.weight'] = torch.tensor(flax_dict['LayerNorm_0']['scale'])
+    state_dict[f'norm.bias'] = torch.tensor(flax_dict['LayerNorm_0']['bias'])
+
+    # Classifier
+    state_dict['head.weight'] = torch.tensor(flax_dict['Dense_0']['kernel']).permute(1, 0)
+    state_dict['head.bias'] = torch.tensor(flax_dict['Dense_0']['bias'])
+
+    return state_dict
+
+
+if __name__ == '__main__':
+    variant = sys.argv[1] # base, small, or tiny
+    state_dict = convert_nest(f'./nest-{variant[0]}_imagenet', f'nest_{variant}')
+    torch.save(state_dict, f'./jx_nest_{variant}.pth')

demo.py

@@ -0,0 +1,120 @@
+
+import os, cv2, time, math
+print("=> Loading libraries...")
+start = time.time()
+
+import requests, torch
+import gradio as gr
+from torchvision import transforms
+from datasets import load_dataset
+from timm.data import create_transform
+from timm.models import create_model, load_checkpoint
+from pytorch_grad_cam import GradCAM
+from pytorch_grad_cam.utils.image import show_cam_on_image
+
+
+print(f"=> Libraries loaded in {time.time()- start:.2f} sec(s).")
+print("=> Loading model...")
+start = time.time()
+
+size = "b"
+img_size = 224
+crop_pct = 0.9
+IMAGENET_DEFAULT_MEAN = (0.485, 0.456, 0.406)
+IMAGENET_DEFAULT_STD = (0.229, 0.224, 0.225)
+
+model = create_model(f"tpmlp_{size}").cuda()
+load_checkpoint(model, f"../tpmlp_{size}.pth.tar", True)
+model.eval()
+
+response = requests.get("https://git.io/JJkYN")
+labels = response.text.split("\n")
+
+augs = create_transform(
+    input_size=(3, 224, 224),
+    is_training=False,
+    use_prefetcher=False,
+    crop_pct=0.9,
+)
+
+
+scale_size = math.floor(img_size / crop_pct)
+resize = transforms.Compose([
+    transforms.Resize(scale_size),
+    transforms.CenterCrop(img_size),
+    transforms.ToTensor()
+])
+normalize = transforms.Normalize(mean=torch.tensor(IMAGENET_DEFAULT_MEAN), std=torch.tensor(IMAGENET_DEFAULT_STD))
+
+def transform(img):
+    img = resize(img.convert("RGB"))
+    tensor = normalize(img)
+    return img, tensor
+
+def predict(inp):
+    img, inp = transform(inp)
+    inp = inp.unsqueeze(0)
+    with GradCAM(model=model, target_layers=[model.layers[3]], use_cuda=True) as cam:
+        grayscale_cam, probs = cam(input_tensor=inp, aug_smooth=False, eigen_smooth=False, return_probs=True)
+        
+        # Here grayscale_cam has only one image in the batch
+        grayscale_cam = grayscale_cam[0, :]
+        probs = probs[0, :]
+
+        cam_image = show_cam_on_image(img.permute(1, 2, 0).detach().cpu().numpy(), grayscale_cam, use_rgb=True, image_weight=0.5, colormap=cv2.COLORMAP_TWILIGHT_SHIFTED)
+        confidences = {labels[i]: float(probs[i]) for i in range(1000)}
+    return confidences, cam_image
+
+print(f"=> Model (tpmlp_{size}) loaded in {time.time()- start:.2f} sec(s).")
+
+if not os.path.isdir("../example-imgs"):
+    os.mkdir("../example-imgs")
+
+print("=> Loading examples.")
+indices = [
+    0,      # Coucal
+    2,      # Volcano
+    7,      # Sombrero
+    9,      # Balance beam
+    10,     # Sulphur-crested cockatoo
+    11,     # Shower cap
+    12,     # Petri dish INCORRECTLY CLASSIFIED as lens
+    14,     # Angora rabbit
+]
+ds = load_dataset("imagenet-1k", split="validation", streaming=True)
+examples = []; idx = 0
+start = time.time()
+for data in ds:
+    if idx == indices:
+        data['image'].save(f"../example-imgs/{idx}.png")
+    idx += 1
+    if idx == max(indices):
+        break
+del ds
+print(f"=> Examples loaded in {time.time()- start:.2f} sec(s).")
+
+# demo = gr.Interface(
+#     fn=predict, 
+#     inputs=gr.inputs.Image(type="pil"),
+#     outputs=[gr.outputs.Label(num_top_classes=4), gr.outputs.Image(type="numpy")],
+#     examples=[f"../example-imgs/{idx}.png" for idx in indices],
+# )
+
+
+with gr.Blocks(theme=gr.themes.Monochrome(font=[gr.themes.GoogleFont("DM Sans"), "sans-serif"])) as demo:
+    gr.HTML("""
+    <h1 align="center">Interactive Demo</h1>
+    <h2 align="center">CS-Mixer: A Cross-Scale Vision MLP Model with Spatial–Channel Mixing</h2>
+    <br><br>
+    """)
+    with gr.Row():
+        input_image = gr.Image(type="pil", min_width=300, label="Input Image")
+        softmax = gr.Label(num_top_classes=4, min_width=200, label="Model Predictions")
+        grad_cam = gr.Image(type="numpy", min_width=300, label="Grad-CAM")
+    with gr.Row():
+        gr.Button("Predict").click(fn=predict, inputs=input_image, outputs=[softmax, grad_cam])
+        gr.ClearButton(input_image)
+    with gr.Row():
+        gr.Examples([f"../example-imgs/{idx}.png" for idx in indices], inputs=input_image, outputs=[softmax, grad_cam], fn=predict, run_on_click=True)
+            
+demo.launch(share=True, allowed_paths=["../example-imgs"])

distributed_train.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+NUM_PROC=$1
+shift
+torchrun --nproc_per_node=$NUM_PROC train.py "$@"
+

docs/archived_changes.md

@@ -0,0 +1,406 @@
+# Archived Changes
+
+### Nov 22, 2021
+* A number of updated weights anew new model defs
+  * `eca_halonext26ts` - 79.5 @ 256
+  * `resnet50_gn` (new) - 80.1 @ 224, 81.3 @ 288
+  * `resnet50` - 80.7 @ 224, 80.9 @ 288 (trained at 176, not replacing current a1 weights as default since these don't scale as well to higher res, [weights](https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-rsb-weights/resnet50_a1h2_176-001a1197.pth))
+  * `resnext50_32x4d` - 81.1 @ 224, 82.0 @ 288
+  * `sebotnet33ts_256` (new) - 81.2 @ 224
+  * `lamhalobotnet50ts_256` - 81.5 @ 256
+  * `halonet50ts` - 81.7 @ 256
+  * `halo2botnet50ts_256` - 82.0 @ 256
+  * `resnet101` - 82.0 @ 224, 82.8 @ 288
+  * `resnetv2_101` (new) - 82.1 @ 224, 83.0 @ 288
+  * `resnet152` - 82.8 @ 224, 83.5 @ 288
+  * `regnetz_d8` (new) - 83.5 @ 256, 84.0 @ 320
+  * `regnetz_e8` (new) - 84.5 @ 256, 85.0 @ 320
+* `vit_base_patch8_224` (85.8 top-1) & `in21k` variant weights added thanks [Martins Bruveris](https://github.com/martinsbruveris)
+* Groundwork in for FX feature extraction thanks to [Alexander Soare](https://github.com/alexander-soare)
+  * models updated for tracing compatibility (almost full support with some distlled transformer exceptions)
+
+### Oct 19, 2021
+* ResNet strikes back (https://arxiv.org/abs/2110.00476) weights added, plus any extra training components used. Model weights and some more details here (https://github.com/rwightman/pytorch-image-models/releases/tag/v0.1-rsb-weights)
+* BCE loss and Repeated Augmentation support for RSB paper
+* 4 series of ResNet based attention model experiments being added (implemented across byobnet.py/byoanet.py). These include all sorts of attention, from channel attn like SE, ECA to 2D QKV self-attention layers such as Halo, Bottlneck, Lambda. Details here (https://github.com/rwightman/pytorch-image-models/releases/tag/v0.1-attn-weights)
+* Working implementations of the following 2D self-attention modules (likely to be differences from paper or eventual official impl):
+  * Halo (https://arxiv.org/abs/2103.12731)
+  * Bottleneck Transformer (https://arxiv.org/abs/2101.11605)
+  * LambdaNetworks (https://arxiv.org/abs/2102.08602)
+* A RegNetZ series of models with some attention experiments (being added to). These do not follow the paper (https://arxiv.org/abs/2103.06877) in any way other than block architecture, details of official models are not available. See more here (https://github.com/rwightman/pytorch-image-models/releases/tag/v0.1-attn-weights)
+* ConvMixer (https://openreview.net/forum?id=TVHS5Y4dNvM), CrossVit (https://arxiv.org/abs/2103.14899), and BeiT (https://arxiv.org/abs/2106.08254) architectures + weights added
+* freeze/unfreeze helpers by [Alexander Soare](https://github.com/alexander-soare)
+
+### Aug 18, 2021
+* Optimizer bonanza!
+  * Add LAMB and LARS optimizers, incl trust ratio clipping options. Tweaked to work properly in PyTorch XLA (tested on TPUs w/ `timm bits` [branch](https://github.com/rwightman/pytorch-image-models/tree/bits_and_tpu/timm/bits))
+  * Add MADGRAD from FB research w/ a few tweaks (decoupled decay option, step handling that works with PyTorch XLA)
+  * Some cleanup on all optimizers and factory. No more `.data`, a bit more consistency, unit tests for all!
+  * SGDP and AdamP still won't work with PyTorch XLA but others should (have yet to test Adabelief, Adafactor, Adahessian myself).
+* EfficientNet-V2 XL TF ported weights added, but they don't validate well in PyTorch (L is better). The pre-processing for the V2 TF training is a bit diff and the fine-tuned 21k -> 1k weights are very sensitive and less robust than the 1k weights.
+* Added PyTorch trained EfficientNet-V2 'Tiny' w/ GlobalContext attn weights. Only .1-.2 top-1 better than the SE so more of a curiosity for those interested.
+
+### July 12, 2021
+* Add XCiT models from [official facebook impl](https://github.com/facebookresearch/xcit). Contributed by [Alexander Soare](https://github.com/alexander-soare)
+
+### July 5-9, 2021
+* Add `efficientnetv2_rw_t` weights, a custom 'tiny' 13.6M param variant that is a bit better than (non NoisyStudent) B3 models. Both faster and better accuracy (at same or lower res)
+  * top-1 82.34 @ 288x288 and 82.54 @ 320x320
+* Add [SAM pretrained](https://arxiv.org/abs/2106.01548) in1k weight for ViT B/16 (`vit_base_patch16_sam_224`) and B/32 (`vit_base_patch32_sam_224`)  models.
+* Add 'Aggregating Nested Transformer' (NesT) w/ weights converted from official [Flax impl](https://github.com/google-research/nested-transformer). Contributed by [Alexander Soare](https://github.com/alexander-soare).
+  * `jx_nest_base` - 83.534, `jx_nest_small` - 83.120, `jx_nest_tiny` - 81.426
+
+### June 23, 2021
+* Reproduce gMLP model training, `gmlp_s16_224` trained to 79.6 top-1, matching [paper](https://arxiv.org/abs/2105.08050). Hparams for this and other recent MLP training [here](https://gist.github.com/rwightman/d6c264a9001f9167e06c209f630b2cc6)
+
+### June 20, 2021
+* Release Vision Transformer 'AugReg' weights from [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://arxiv.org/abs/2106.10270)
+  * .npz weight loading support added, can load any of the 50K+ weights from the [AugReg series](https://console.cloud.google.com/storage/browser/vit_models/augreg)
+  * See [example notebook](https://colab.research.google.com/github/google-research/vision_transformer/blob/master/vit_jax_augreg.ipynb) from [official impl](https://github.com/google-research/vision_transformer/) for navigating the augreg weights
+  * Replaced all default weights w/ best AugReg variant (if possible). All AugReg 21k classifiers work.
+    * Highlights: `vit_large_patch16_384` (87.1 top-1), `vit_large_r50_s32_384` (86.2 top-1), `vit_base_patch16_384` (86.0 top-1)
+  * `vit_deit_*` renamed to just `deit_*`
+  * Remove my old small model, replace with DeiT compatible small w/ AugReg weights
+* Add 1st training of my `gmixer_24_224` MLP /w GLU, 78.1 top-1 w/ 25M params.
+* Add weights from official ResMLP release (https://github.com/facebookresearch/deit)
+* Add `eca_nfnet_l2` weights from my 'lightweight' series. 84.7 top-1 at 384x384.
+* Add distilled BiT 50x1 student and 152x2 Teacher weights from  [Knowledge distillation: A good teacher is patient and consistent](https://arxiv.org/abs/2106.05237)
+* NFNets and ResNetV2-BiT models work w/ Pytorch XLA now
+  * weight standardization uses F.batch_norm instead of std_mean (std_mean wasn't lowered)
+  * eps values adjusted, will be slight differences but should be quite close
+* Improve test coverage and classifier interface of non-conv (vision transformer and mlp) models
+* Cleanup a few classifier / flatten details for models w/ conv classifiers or early global pool
+* Please report any regressions, this PR touched quite a few models.
+
+### June 8, 2021
+* Add first ResMLP weights, trained in PyTorch XLA on TPU-VM w/ my XLA branch. 24 block variant, 79.2 top-1.
+* Add ResNet51-Q model w/ pretrained weights at 82.36 top-1.
+  * NFNet inspired block layout with quad layer stem and no maxpool
+  * Same param count (35.7M) and throughput as ResNetRS-50 but +1.5 top-1 @ 224x224 and +2.5 top-1 at 288x288
+
+### May 25, 2021
+* Add LeViT, Visformer, Convit (PR by Aman Arora), Twins (PR by paper authors) transformer models
+* Cleanup input_size/img_size override handling and testing for all vision transformer models
+* Add `efficientnetv2_rw_m` model and weights (started training before official code). 84.8 top-1, 53M params.
+
+### May 14, 2021
+* Add EfficientNet-V2 official model defs w/ ported weights from official [Tensorflow/Keras](https://github.com/google/automl/tree/master/efficientnetv2) impl.
+  * 1k trained variants: `tf_efficientnetv2_s/m/l`
+  * 21k trained variants: `tf_efficientnetv2_s/m/l_in21k`
+  * 21k pretrained -> 1k fine-tuned: `tf_efficientnetv2_s/m/l_in21ft1k`
+  * v2 models w/ v1 scaling: `tf_efficientnetv2_b0` through `b3`
+  * Rename my prev V2 guess `efficientnet_v2s` -> `efficientnetv2_rw_s`
+  * Some blank `efficientnetv2_*` models in-place for future native PyTorch training
+
+### May 5, 2021
+* Add MLP-Mixer models and port pretrained weights from [Google JAX impl](https://github.com/google-research/vision_transformer/tree/linen)
+* Add CaiT models and pretrained weights from [FB](https://github.com/facebookresearch/deit)
+* Add ResNet-RS models and weights from [TF](https://github.com/tensorflow/tpu/tree/master/models/official/resnet/resnet_rs). Thanks [Aman Arora](https://github.com/amaarora)
+* Add CoaT models and weights. Thanks [Mohammed Rizin](https://github.com/morizin)
+* Add new ImageNet-21k weights & finetuned weights for TResNet, MobileNet-V3, ViT models. Thanks [mrT](https://github.com/mrT23)
+* Add GhostNet models and weights. Thanks [Kai Han](https://github.com/iamhankai)
+* Update ByoaNet attention modles
+   * Improve SA module inits
+   * Hack together experimental stand-alone Swin based attn module and `swinnet`
+   * Consistent '26t' model defs for experiments.
+* Add improved Efficientnet-V2S (prelim model def) weights. 83.8 top-1.
+* WandB logging support
+
+### April 13, 2021
+* Add Swin Transformer models and weights from https://github.com/microsoft/Swin-Transformer
+
+### April 12, 2021
+* Add ECA-NFNet-L1 (slimmed down F1 w/ SiLU, 41M params) trained with this code. 84% top-1 @ 320x320. Trained at 256x256.
+* Add EfficientNet-V2S model (unverified model definition) weights. 83.3 top-1 @ 288x288. Only trained single res 224. Working on progressive training.
+* Add ByoaNet model definition (Bring-your-own-attention) w/ SelfAttention block and corresponding SA/SA-like modules and model defs
+  * Lambda Networks - https://arxiv.org/abs/2102.08602
+  * Bottleneck Transformers - https://arxiv.org/abs/2101.11605
+  * Halo Nets - https://arxiv.org/abs/2103.12731
+* Adabelief optimizer contributed by Juntang Zhuang
+
+### April 1, 2021
+* Add snazzy `benchmark.py` script for bulk `timm` model benchmarking of train and/or inference
+* Add Pooling-based Vision Transformer (PiT) models (from https://github.com/naver-ai/pit)
+  * Merged distilled variant into main for torchscript compatibility
+  * Some `timm` cleanup/style tweaks and weights have hub download support
+* Cleanup Vision Transformer (ViT) models
+  * Merge distilled (DeiT) model into main so that torchscript can work
+  * Support updated weight init (defaults to old still) that closer matches original JAX impl (possibly better training from scratch)
+  * Separate hybrid model defs into different file and add several new model defs to fiddle with, support patch_size != 1 for hybrids
+  * Fix fine-tuning num_class changes (PiT and ViT) and pos_embed resizing (Vit) with distilled variants
+  * nn.Sequential for block stack (does not break downstream compat)
+* TnT (Transformer-in-Transformer) models contributed by author (from https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/cv/TNT)
+* Add RegNetY-160 weights from DeiT teacher model
+* Add new NFNet-L0 w/ SE attn (rename `nfnet_l0b`->`nfnet_l0`) weights 82.75 top-1 @ 288x288
+* Some fixes/improvements for TFDS dataset wrapper
+
+### March 7, 2021
+* First 0.4.x PyPi release w/ NFNets (& related), ByoB (GPU-Efficient, RepVGG, etc).
+* Change feature extraction for pre-activation nets (NFNets, ResNetV2) to return features before activation.
+
+### Feb 18, 2021
+* Add pretrained weights and model variants for NFNet-F* models from [DeepMind Haiku impl](https://github.com/deepmind/deepmind-research/tree/master/nfnets).
+  * Models are prefixed with `dm_`. They require SAME padding conv, skipinit enabled, and activation gains applied in act fn.
+  * These models are big, expect to run out of GPU memory. With the GELU activiation + other options, they are roughly 1/2 the inference speed of my SiLU PyTorch optimized `s` variants.
+  * Original model results are based on pre-processing that is not the same as all other models so you'll see different results in the results csv (once updated).
+  * Matching the original pre-processing as closely as possible I get these results:
+    * `dm_nfnet_f6` - 86.352
+    * `dm_nfnet_f5` - 86.100
+    * `dm_nfnet_f4` - 85.834
+    * `dm_nfnet_f3` - 85.676
+    * `dm_nfnet_f2` - 85.178
+    * `dm_nfnet_f1` - 84.696
+    * `dm_nfnet_f0` - 83.464
+
+### Feb 16, 2021
+* Add Adaptive Gradient Clipping (AGC) as per https://arxiv.org/abs/2102.06171. Integrated w/ PyTorch gradient clipping via mode arg that defaults to prev 'norm' mode. For backward arg compat, clip-grad arg must be specified to enable when using train.py.
+  * AGC w/ default clipping factor `--clip-grad .01 --clip-mode agc`
+  * PyTorch global norm of 1.0 (old behaviour, always norm), `--clip-grad 1.0`
+  * PyTorch value clipping of 10, `--clip-grad 10. --clip-mode value`
+  * AGC performance is definitely sensitive to the clipping factor. More experimentation needed to determine good values for smaller batch sizes and optimizers besides those in paper. So far I've found .001-.005 is necessary for stable RMSProp training w/ NFNet/NF-ResNet.
+
+### Feb 12, 2021
+* Update Normalization-Free nets to include new NFNet-F (https://arxiv.org/abs/2102.06171) model defs
+
+### Feb 10, 2021
+* More model archs, incl a flexible ByobNet backbone ('Bring-your-own-blocks')
+  * GPU-Efficient-Networks (https://github.com/idstcv/GPU-Efficient-Networks), impl in `byobnet.py`
+  * RepVGG (https://github.com/DingXiaoH/RepVGG), impl in `byobnet.py`
+  * classic VGG (from torchvision, impl in `vgg`)
+* Refinements to normalizer layer arg handling and normalizer+act layer handling in some models
+* Default AMP mode changed to native PyTorch AMP instead of APEX. Issues not being fixed with APEX. Native works with `--channels-last` and `--torchscript` model training, APEX does not.
+* Fix a few bugs introduced since last pypi release
+
+### Feb 8, 2021
+* Add several ResNet weights with ECA attention. 26t & 50t trained @ 256, test @ 320. 269d train @ 256, fine-tune @320, test @ 352.
+  * `ecaresnet26t` - 79.88 top-1 @ 320x320, 79.08 @ 256x256
+  * `ecaresnet50t` - 82.35 top-1 @ 320x320, 81.52 @ 256x256
+  * `ecaresnet269d` - 84.93 top-1 @ 352x352, 84.87 @ 320x320
+* Remove separate tiered (`t`) vs tiered_narrow (`tn`) ResNet model defs, all `tn` changed to `t` and `t` models removed (`seresnext26t_32x4d` only model w/ weights that was removed).
+* Support model default_cfgs with separate train vs test resolution `test_input_size` and remove extra `_320` suffix ResNet model defs that were just for test.
+
+### Jan 30, 2021
+* Add initial "Normalization Free" NF-RegNet-B* and NF-ResNet model definitions based on [paper](https://arxiv.org/abs/2101.08692)
+
+### Jan 25, 2021
+* Add ResNetV2 Big Transfer (BiT) models w/ ImageNet-1k and 21k weights from https://github.com/google-research/big_transfer
+* Add official R50+ViT-B/16 hybrid models + weights from https://github.com/google-research/vision_transformer
+* ImageNet-21k ViT weights are added w/ model defs and representation layer (pre logits) support
+  * NOTE: ImageNet-21k classifier heads were zero'd in original weights, they are only useful for transfer learning
+* Add model defs and weights for DeiT Vision Transformer models from https://github.com/facebookresearch/deit
+* Refactor dataset classes into ImageDataset/IterableImageDataset + dataset specific parser classes
+* Add Tensorflow-Datasets (TFDS) wrapper to allow use of TFDS image classification sets with train script
+  * Ex: `train.py /data/tfds --dataset tfds/oxford_iiit_pet --val-split test --model resnet50 -b 256 --amp --num-classes 37 --opt adamw --lr 3e-4 --weight-decay .001 --pretrained -j 2`
+* Add improved .tar dataset parser that reads images from .tar, folder of .tar files, or .tar within .tar
+  * Run validation on full ImageNet-21k directly from tar w/ BiT model: `validate.py /data/fall11_whole.tar --model resnetv2_50x1_bitm_in21k --amp`
+* Models in this update should be stable w/ possible exception of ViT/BiT, possibility of some regressions with train/val scripts and dataset handling
+
+### Jan 3, 2021
+* Add SE-ResNet-152D weights
+  * 256x256 val, 0.94 crop top-1 - 83.75
+  * 320x320 val, 1.0 crop - 84.36
+* Update results files
+
+### Dec 18, 2020
+* Add ResNet-101D, ResNet-152D, and ResNet-200D weights trained @ 256x256
+  * 256x256 val, 0.94 crop (top-1) - 101D (82.33), 152D (83.08), 200D (83.25)
+  * 288x288 val, 1.0 crop - 101D (82.64), 152D (83.48), 200D (83.76)
+  * 320x320 val, 1.0 crop - 101D (83.00), 152D (83.66), 200D (84.01)
+
+### Dec 7, 2020
+* Simplify EMA module (ModelEmaV2), compatible with fully torchscripted models
+* Misc fixes for SiLU ONNX export, default_cfg missing from Feature extraction models, Linear layer w/ AMP + torchscript
+* PyPi release @ 0.3.2 (needed by EfficientDet)
+
+
+### Oct 30, 2020
+* Test with PyTorch 1.7 and fix a small top-n metric view vs reshape issue.
+* Convert newly added 224x224 Vision Transformer weights from official JAX repo. 81.8 top-1 for B/16, 83.1 L/16.
+* Support PyTorch 1.7 optimized, native SiLU (aka Swish) activation. Add mapping to 'silu' name, custom swish will eventually be deprecated.
+* Fix regression for loading pretrained classifier via direct model entrypoint functions. Didn't impact create_model() factory usage.
+* PyPi release @ 0.3.0 version!
+
+### Oct 26, 2020
+* Update Vision Transformer models to be compatible with official code release at https://github.com/google-research/vision_transformer
+* Add Vision Transformer weights (ImageNet-21k pretrain) for 384x384 base and large models converted from official jax impl
+  * ViT-B/16 - 84.2
+  * ViT-B/32 - 81.7
+  * ViT-L/16 - 85.2
+  * ViT-L/32 - 81.5
+
+### Oct 21, 2020
+* Weights added for Vision Transformer (ViT) models. 77.86 top-1 for 'small' and 79.35 for 'base'. Thanks to [Christof](https://www.kaggle.com/christofhenkel) for training the base model w/ lots of GPUs.
+
+### Oct 13, 2020
+* Initial impl of Vision Transformer models. Both patch and hybrid (CNN backbone) variants. Currently trying to train...
+* Adafactor and AdaHessian (FP32 only, no AMP) optimizers
+* EdgeTPU-M (`efficientnet_em`) model trained in PyTorch, 79.3 top-1
+* Pip release, doc updates pending a few more changes...
+
+### Sept 18, 2020
+* New ResNet 'D' weights. 72.7 (top-1) ResNet-18-D, 77.1 ResNet-34-D, 80.5 ResNet-50-D
+* Added a few untrained defs for other ResNet models (66D, 101D, 152D, 200/200D)
+
+### Sept 3, 2020
+* New weights
+  * Wide-ResNet50 - 81.5 top-1 (vs 78.5 torchvision)
+  * SEResNeXt50-32x4d - 81.3 top-1 (vs 79.1 cadene)
+* Support for native Torch AMP and channels_last memory format added to train/validate scripts (`--channels-last`, `--native-amp` vs `--apex-amp`)
+* Models tested with channels_last on latest NGC 20.08 container. AdaptiveAvgPool in attn layers changed to mean((2,3)) to work around bug with NHWC kernel.
+
+### Aug 12, 2020
+* New/updated weights from training experiments
+  * EfficientNet-B3 - 82.1 top-1 (vs 81.6 for official with AA and 81.9 for AdvProp)
+  * RegNetY-3.2GF - 82.0 top-1 (78.9 from official ver)
+  * CSPResNet50 - 79.6 top-1 (76.6 from official ver)
+* Add CutMix integrated w/ Mixup. See [pull request](https://github.com/rwightman/pytorch-image-models/pull/218) for some usage examples
+* Some fixes for using pretrained weights with `in_chans` != 3 on several models.
+
+### Aug 5, 2020
+Universal feature extraction, new models, new weights, new test sets.
+* All models support the `features_only=True` argument for `create_model` call to return a network that extracts feature maps from the deepest layer at each stride.
+* New models
+  * CSPResNet, CSPResNeXt, CSPDarkNet, DarkNet
+  * ReXNet
+  * (Modified Aligned) Xception41/65/71 (a proper port of TF models)
+* New trained weights
+  * SEResNet50 - 80.3 top-1
+  * CSPDarkNet53 - 80.1 top-1
+  * CSPResNeXt50 - 80.0 top-1
+  * DPN68b - 79.2 top-1
+  * EfficientNet-Lite0 (non-TF ver) - 75.5 (submitted by [@hal-314](https://github.com/hal-314))
+* Add 'real' labels for ImageNet and ImageNet-Renditions test set, see [`results/README.md`](results/README.md)
+* Test set ranking/top-n diff script by [@KushajveerSingh](https://github.com/KushajveerSingh)
+* Train script and loader/transform tweaks to punch through more aug arguments
+* README and documentation overhaul. See initial (WIP) documentation at https://rwightman.github.io/pytorch-image-models/
+* adamp and sgdp optimizers added by [@hellbell](https://github.com/hellbell)
+
+### June 11, 2020
+Bunch of changes:
+* DenseNet models updated with memory efficient addition from torchvision (fixed a bug), blur pooling and deep stem additions
+* VoVNet V1 and V2 models added, 39 V2 variant (ese_vovnet_39b) trained to 79.3 top-1
+* Activation factory added along with new activations:
+   * select act at model creation time for more flexibility in using activations compatible with scripting or tracing (ONNX export)
+   * hard_mish (experimental) added with memory-efficient grad, along with ME hard_swish
+   * context mgr for setting exportable/scriptable/no_jit states
+* Norm + Activation combo layers added with initial trial support in DenseNet and VoVNet along with impl of EvoNorm and InplaceAbn wrapper that fit the interface
+* Torchscript works for all but two of the model types as long as using Pytorch 1.5+, tests added for this
+* Some import cleanup and classifier reset changes, all models will have classifier reset to nn.Identity on reset_classifer(0) call
+* Prep for 0.1.28 pip release
+
+### May 12, 2020
+* Add ResNeSt models (code adapted from https://github.com/zhanghang1989/ResNeSt, paper https://arxiv.org/abs/2004.08955))
+
+### May 3, 2020
+* Pruned EfficientNet B1, B2, and B3 (https://arxiv.org/abs/2002.08258) contributed by [Yonathan Aflalo](https://github.com/yoniaflalo)
+
+### May 1, 2020
+* Merged a number of execellent contributions in the ResNet model family over the past month
+  * BlurPool2D and resnetblur models initiated by [Chris Ha](https://github.com/VRandme), I trained resnetblur50 to 79.3.
+  * TResNet models and SpaceToDepth, AntiAliasDownsampleLayer layers by [mrT23](https://github.com/mrT23)
+  * ecaresnet (50d, 101d, light) models and two pruned variants using pruning as per (https://arxiv.org/abs/2002.08258) by [Yonathan Aflalo](https://github.com/yoniaflalo)
+* 200 pretrained models in total now with updated results csv in results folder
+
+### April 5, 2020
+* Add some newly trained MobileNet-V2 models trained with latest h-params, rand augment. They compare quite favourably to EfficientNet-Lite
+  * 3.5M param MobileNet-V2 100 @ 73%
+  * 4.5M param MobileNet-V2 110d @ 75%
+  * 6.1M param MobileNet-V2 140 @ 76.5%
+  * 5.8M param MobileNet-V2 120d @ 77.3%
+
+### March 18, 2020
+* Add EfficientNet-Lite models w/ weights ported from [Tensorflow TPU](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite)
+* Add RandAugment trained ResNeXt-50 32x4d weights with 79.8 top-1. Trained by [Andrew Lavin](https://github.com/andravin) (see Training section for hparams)
+
+### April 5, 2020
+* Add some newly trained MobileNet-V2 models trained with latest h-params, rand augment. They compare quite favourably to EfficientNet-Lite
+  * 3.5M param MobileNet-V2 100 @ 73%
+  * 4.5M param MobileNet-V2 110d @ 75%
+  * 6.1M param MobileNet-V2 140 @ 76.5%
+  * 5.8M param MobileNet-V2 120d @ 77.3%
+
+### March 18, 2020
+* Add EfficientNet-Lite models w/ weights ported from [Tensorflow TPU](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite)
+* Add RandAugment trained ResNeXt-50 32x4d weights with 79.8 top-1. Trained by [Andrew Lavin](https://github.com/andravin) (see Training section for hparams)
+
+### Feb 29, 2020
+* New MobileNet-V3 Large weights trained from stratch with this code to 75.77% top-1
+* IMPORTANT CHANGE - default weight init changed for all MobilenetV3 / EfficientNet / related models
+  * overall results similar to a bit better training from scratch on a few smaller models tried
+  * performance early in training seems consistently improved but less difference by end
+  * set `fix_group_fanout=False` in `_init_weight_goog` fn if you need to reproducte past behaviour
+* Experimental LR noise feature added applies a random perturbation to LR each epoch in specified range of training
+
+### Feb 18, 2020
+* Big refactor of model layers and addition of several attention mechanisms. Several additions motivated by 'Compounding the Performance Improvements...' (https://arxiv.org/abs/2001.06268):
+  * Move layer/module impl into `layers` subfolder/module of `models` and organize in a more granular fashion
+  * ResNet downsample paths now properly support dilation (output stride != 32) for avg_pool ('D' variant) and 3x3 (SENets) networks
+  * Add Selective Kernel Nets on top of ResNet base, pretrained weights
+    * skresnet18 - 73% top-1
+    * skresnet34 - 76.9% top-1 
+    * skresnext50_32x4d (equiv to SKNet50) - 80.2% top-1
+  * ECA and CECA (circular padding) attention layer contributed by [Chris Ha](https://github.com/VRandme)
+  * CBAM attention experiment (not the best results so far, may remove)
+  * Attention factory to allow dynamically selecting one of SE, ECA, CBAM in the `.se` position for all ResNets
+  * Add DropBlock and DropPath (formerly DropConnect for EfficientNet/MobileNetv3) support to all ResNet variants
+* Full dataset results updated that incl NoisyStudent weights and 2 of the 3 SK weights
+
+### Feb 12, 2020
+* Add EfficientNet-L2 and B0-B7 NoisyStudent weights ported from [Tensorflow TPU](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet)
+
+### Feb 6, 2020
+* Add RandAugment trained EfficientNet-ES (EdgeTPU-Small) weights with 78.1 top-1. Trained by [Andrew Lavin](https://github.com/andravin) (see Training section for hparams)
+
+### Feb 1/2, 2020
+* Port new EfficientNet-B8 (RandAugment) weights, these are different than the B8 AdvProp, different input normalization.
+* Update results csv files on all models for ImageNet validation and three other test sets
+* Push PyPi package update
+
+### Jan 31, 2020
+* Update ResNet50 weights with a new 79.038 result from further JSD / AugMix experiments. Full command line for reproduction in training section below.
+
+### Jan 11/12, 2020
+* Master may be a bit unstable wrt to training, these changes have been tested but not all combos
+* Implementations of AugMix added to existing RA and AA. Including numerous supporting pieces like JSD loss (Jensen-Shannon divergence + CE), and AugMixDataset
+* SplitBatchNorm adaptation layer added for implementing Auxiliary BN as per AdvProp paper
+* ResNet-50 AugMix trained model w/ 79% top-1 added
+* `seresnext26tn_32x4d` - 77.99 top-1, 93.75 top-5 added to tiered experiment, higher img/s than 't' and 'd'
+
+### Jan 3, 2020
+* Add RandAugment trained EfficientNet-B0 weight with 77.7 top-1. Trained by [Michael Klachko](https://github.com/michaelklachko) with this code and recent hparams (see Training section)
+* Add `avg_checkpoints.py` script for post training weight averaging and update all scripts with header docstrings and shebangs.
+
+### Dec 30, 2019
+* Merge [Dushyant Mehta's](https://github.com/mehtadushy) PR for SelecSLS (Selective Short and Long Range Skip Connections) networks. Good GPU memory consumption and throughput. Original: https://github.com/mehtadushy/SelecSLS-Pytorch
+
+### Dec 28, 2019
+* Add new model weights and training hparams (see Training Hparams section)
+  * `efficientnet_b3` - 81.5 top-1, 95.7 top-5 at default res/crop, 81.9, 95.8 at 320x320 1.0 crop-pct
+     * trained with RandAugment, ended up with an interesting but less than perfect result (see training section)
+  * `seresnext26d_32x4d`- 77.6 top-1, 93.6 top-5
+     * deep stem (32, 32, 64), avgpool downsample
+     * stem/dowsample from bag-of-tricks paper
+  * `seresnext26t_32x4d`- 78.0 top-1, 93.7 top-5
+     * deep tiered stem (24, 48, 64), avgpool downsample (a modified 'D' variant)
+     * stem sizing mods from Jeremy Howard and fastai devs discussing ResNet architecture experiments
+
+### Dec 23, 2019
+* Add RandAugment trained MixNet-XL weights with 80.48 top-1.
+* `--dist-bn` argument added to train.py, will distribute BN stats between nodes after each train epoch, before eval
+
+### Dec 4, 2019
+* Added weights from the first training from scratch of an EfficientNet (B2) with my new RandAugment implementation. Much better than my previous B2 and very close to the official AdvProp ones (80.4 top-1, 95.08 top-5).
+
+### Nov 29, 2019
+* Brought EfficientNet and MobileNetV3 up to date with my https://github.com/rwightman/gen-efficientnet-pytorch code. Torchscript and ONNX export compat excluded.
+  * AdvProp weights added
+  * Official TF MobileNetv3 weights added
+* EfficientNet and MobileNetV3 hook based 'feature extraction' classes added. Will serve as basis for using models as backbones in obj detection/segmentation tasks. Lots more to be done here...
+* HRNet classification models and weights added from https://github.com/HRNet/HRNet-Image-Classification
+* Consistency in global pooling, `reset_classifer`, and `forward_features` across models
+  * `forward_features` always returns unpooled feature maps now
+* Reasonable chance I broke something... let me know
+
+### Nov 22, 2019
+* Add ImageNet training RandAugment implementation alongside AutoAugment. PyTorch Transform compatible format, using PIL. Currently training two EfficientNet models from scratch with promising results... will update.
+* `drop-connect` cmd line arg finally added to `train.py`, no need to hack model fns. Works for efficientnet/mobilenetv3 based models, ignored otherwise.

docs/changes.md

@@ -0,0 +1,314 @@
+# Recent Changes
+### Jan 5, 2023
+* ConvNeXt-V2 models and weights added to existing `convnext.py`
+  * Paper: [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](http://arxiv.org/abs/2301.00808)
+  * Reference impl: https://github.com/facebookresearch/ConvNeXt-V2 (NOTE: weights currently CC-BY-NC)
+
+### Dec 23, 2022 🎄☃
+* Add FlexiViT models and weights from https://github.com/google-research/big_vision (check out paper at https://arxiv.org/abs/2212.08013)
+  * NOTE currently resizing is static on model creation, on-the-fly dynamic / train patch size sampling is a WIP
+* Many more models updated to multi-weight and downloadable via HF hub now (convnext, efficientnet, mobilenet, vision_transformer*, beit)
+* More model pretrained tag and adjustments, some model names changed (working on deprecation translations, consider main branch DEV branch right now, use 0.6.x for stable use)
+* More ImageNet-12k (subset of 22k) pretrain models popping up:
+  * `efficientnet_b5.in12k_ft_in1k` - 85.9 @ 448x448
+  * `vit_medium_patch16_gap_384.in12k_ft_in1k` - 85.5 @ 384x384
+  * `vit_medium_patch16_gap_256.in12k_ft_in1k` - 84.5 @ 256x256
+  * `convnext_nano.in12k_ft_in1k` - 82.9 @ 288x288
+
+### Dec 8, 2022
+* Add 'EVA l' to `vision_transformer.py`, MAE style ViT-L/14 MIM pretrain w/ EVA-CLIP targets, FT on ImageNet-1k (w/ ImageNet-22k intermediate for some)
+  * original source: https://github.com/baaivision/EVA
+
+| model                                     | top1 | param_count |  gmac | macts | hub                                     |
+|:------------------------------------------|-----:|------------:|------:|------:|:----------------------------------------|
+| eva_large_patch14_336.in22k_ft_in22k_in1k | 89.2 |       304.5 | 191.1 | 270.2 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_large_patch14_336.in22k_ft_in1k       | 88.7 |       304.5 | 191.1 | 270.2 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_large_patch14_196.in22k_ft_in22k_in1k | 88.6 |       304.1 |  61.6 |  63.5 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_large_patch14_196.in22k_ft_in1k       | 87.9 |       304.1 |  61.6 |  63.5 | [link](https://huggingface.co/BAAI/EVA) |
+
+### Dec 6, 2022
+* Add 'EVA g', BEiT style ViT-g/14 model weights w/ both MIM pretrain and CLIP pretrain to `beit.py`. 
+  * original source: https://github.com/baaivision/EVA
+  * paper: https://arxiv.org/abs/2211.07636
+
+| model                                    |   top1 |   param_count |   gmac |   macts | hub                                     |
+|:-----------------------------------------|-------:|--------------:|-------:|--------:|:----------------------------------------|
+| eva_giant_patch14_560.m30m_ft_in22k_in1k |   89.8 |        1014.4 | 1906.8 |  2577.2 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_giant_patch14_336.m30m_ft_in22k_in1k |   89.6 |        1013   |  620.6 |   550.7 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_giant_patch14_336.clip_ft_in1k       |   89.4 |        1013   |  620.6 |   550.7 | [link](https://huggingface.co/BAAI/EVA) |
+| eva_giant_patch14_224.clip_ft_in1k       |   89.1 |        1012.6 |  267.2 |   192.6 | [link](https://huggingface.co/BAAI/EVA) |
+
+### Dec 5, 2022
+
+* Pre-release (`0.8.0dev0`) of multi-weight support (`model_arch.pretrained_tag`). Install with `pip install --pre timm`
+  * vision_transformer, maxvit, convnext are the first three model impl w/ support
+  * model names are changing with this (previous _21k, etc. fn will merge), still sorting out deprecation handling
+  * bugs are likely, but I need feedback so please try it out
+  * if stability is needed, please use 0.6.x pypi releases or clone from [0.6.x branch](https://github.com/rwightman/pytorch-image-models/tree/0.6.x)
+* Support for PyTorch 2.0 compile is added in train/validate/inference/benchmark, use `--torchcompile` argument
+* Inference script allows more control over output, select k for top-class index + prob json, csv or parquet output
+* Add a full set of fine-tuned CLIP image tower weights from both LAION-2B and original OpenAI CLIP models
+
+| model                                            |   top1 |   param_count |   gmac |   macts | hub                                                                                  |
+|:-------------------------------------------------|-------:|--------------:|-------:|--------:|:-------------------------------------------------------------------------------------|
+| vit_huge_patch14_clip_336.laion2b_ft_in12k_in1k  |   88.6 |         632.5 |  391   |   407.5 | [link](https://huggingface.co/timm/vit_huge_patch14_clip_336.laion2b_ft_in12k_in1k)  |
+| vit_large_patch14_clip_336.openai_ft_in12k_in1k  |   88.3 |         304.5 |  191.1 |   270.2 | [link](https://huggingface.co/timm/vit_large_patch14_clip_336.openai_ft_in12k_in1k)  |
+| vit_huge_patch14_clip_224.laion2b_ft_in12k_in1k  |   88.2 |         632   |  167.4 |   139.4 | [link](https://huggingface.co/timm/vit_huge_patch14_clip_224.laion2b_ft_in12k_in1k)  |
+| vit_large_patch14_clip_336.laion2b_ft_in12k_in1k |   88.2 |         304.5 |  191.1 |   270.2 | [link](https://huggingface.co/timm/vit_large_patch14_clip_336.laion2b_ft_in12k_in1k) |
+| vit_large_patch14_clip_224.openai_ft_in12k_in1k  |   88.2 |         304.2 |   81.1 |    88.8 | [link](https://huggingface.co/timm/vit_large_patch14_clip_224.openai_ft_in12k_in1k)  |
+| vit_large_patch14_clip_224.laion2b_ft_in12k_in1k |   87.9 |         304.2 |   81.1 |    88.8 | [link](https://huggingface.co/timm/vit_large_patch14_clip_224.laion2b_ft_in12k_in1k) |
+| vit_large_patch14_clip_224.openai_ft_in1k        |   87.9 |         304.2 |   81.1 |    88.8 | [link](https://huggingface.co/timm/vit_large_patch14_clip_224.openai_ft_in1k)        |
+| vit_large_patch14_clip_336.laion2b_ft_in1k       |   87.9 |         304.5 |  191.1 |   270.2 | [link](https://huggingface.co/timm/vit_large_patch14_clip_336.laion2b_ft_in1k)       |
+| vit_huge_patch14_clip_224.laion2b_ft_in1k        |   87.6 |         632   |  167.4 |   139.4 | [link](https://huggingface.co/timm/vit_huge_patch14_clip_224.laion2b_ft_in1k)        |
+| vit_large_patch14_clip_224.laion2b_ft_in1k       |   87.3 |         304.2 |   81.1 |    88.8 | [link](https://huggingface.co/timm/vit_large_patch14_clip_224.laion2b_ft_in1k)       |
+| vit_base_patch16_clip_384.laion2b_ft_in12k_in1k  |   87.2 |          86.9 |   55.5 |   101.6 | [link](https://huggingface.co/timm/vit_base_patch16_clip_384.laion2b_ft_in12k_in1k)  |
+| vit_base_patch16_clip_384.openai_ft_in12k_in1k   |   87   |          86.9 |   55.5 |   101.6 | [link](https://huggingface.co/timm/vit_base_patch16_clip_384.openai_ft_in12k_in1k)   |
+| vit_base_patch16_clip_384.laion2b_ft_in1k        |   86.6 |          86.9 |   55.5 |   101.6 | [link](https://huggingface.co/timm/vit_base_patch16_clip_384.laion2b_ft_in1k)        |
+| vit_base_patch16_clip_384.openai_ft_in1k         |   86.2 |          86.9 |   55.5 |   101.6 | [link](https://huggingface.co/timm/vit_base_patch16_clip_384.openai_ft_in1k)         |
+| vit_base_patch16_clip_224.laion2b_ft_in12k_in1k  |   86.2 |          86.6 |   17.6 |    23.9 | [link](https://huggingface.co/timm/vit_base_patch16_clip_224.laion2b_ft_in12k_in1k)  |
+| vit_base_patch16_clip_224.openai_ft_in12k_in1k   |   85.9 |          86.6 |   17.6 |    23.9 | [link](https://huggingface.co/timm/vit_base_patch16_clip_224.openai_ft_in12k_in1k)   |
+| vit_base_patch32_clip_448.laion2b_ft_in12k_in1k  |   85.8 |          88.3 |   17.9 |    23.9 | [link](https://huggingface.co/timm/vit_base_patch32_clip_448.laion2b_ft_in12k_in1k)  |
+| vit_base_patch16_clip_224.laion2b_ft_in1k        |   85.5 |          86.6 |   17.6 |    23.9 | [link](https://huggingface.co/timm/vit_base_patch16_clip_224.laion2b_ft_in1k)        |
+| vit_base_patch32_clip_384.laion2b_ft_in12k_in1k  |   85.4 |          88.3 |   13.1 |    16.5 | [link](https://huggingface.co/timm/vit_base_patch32_clip_384.laion2b_ft_in12k_in1k)  |
+| vit_base_patch16_clip_224.openai_ft_in1k         |   85.3 |          86.6 |   17.6 |    23.9 | [link](https://huggingface.co/timm/vit_base_patch16_clip_224.openai_ft_in1k)         |
+| vit_base_patch32_clip_384.openai_ft_in12k_in1k   |   85.2 |          88.3 |   13.1 |    16.5 | [link](https://huggingface.co/timm/vit_base_patch32_clip_384.openai_ft_in12k_in1k)   |
+| vit_base_patch32_clip_224.laion2b_ft_in12k_in1k  |   83.3 |          88.2 |    4.4 |     5   | [link](https://huggingface.co/timm/vit_base_patch32_clip_224.laion2b_ft_in12k_in1k)  |
+| vit_base_patch32_clip_224.laion2b_ft_in1k        |   82.6 |          88.2 |    4.4 |     5   | [link](https://huggingface.co/timm/vit_base_patch32_clip_224.laion2b_ft_in1k)        |
+| vit_base_patch32_clip_224.openai_ft_in1k         |   81.9 |          88.2 |    4.4 |     5   | [link](https://huggingface.co/timm/vit_base_patch32_clip_224.openai_ft_in1k)         |
+
+* Port of MaxViT Tensorflow Weights from official impl at https://github.com/google-research/maxvit
+  * There was larger than expected drops for the upscaled 384/512 in21k fine-tune weights, possible detail missing, but the 21k FT did seem sensitive to small preprocessing
+
+| model                              |   top1 |   param_count |   gmac |   macts | hub                                                                    |
+|:-----------------------------------|-------:|--------------:|-------:|--------:|:-----------------------------------------------------------------------|
+| maxvit_xlarge_tf_512.in21k_ft_in1k |   88.5 |         475.8 |  534.1 |  1413.2 | [link](https://huggingface.co/timm/maxvit_xlarge_tf_512.in21k_ft_in1k) |
+| maxvit_xlarge_tf_384.in21k_ft_in1k |   88.3 |         475.3 |  292.8 |   668.8 | [link](https://huggingface.co/timm/maxvit_xlarge_tf_384.in21k_ft_in1k) |
+| maxvit_base_tf_512.in21k_ft_in1k   |   88.2 |         119.9 |  138   |   704   | [link](https://huggingface.co/timm/maxvit_base_tf_512.in21k_ft_in1k)   |
+| maxvit_large_tf_512.in21k_ft_in1k  |   88   |         212.3 |  244.8 |   942.2 | [link](https://huggingface.co/timm/maxvit_large_tf_512.in21k_ft_in1k)  |
+| maxvit_large_tf_384.in21k_ft_in1k  |   88   |         212   |  132.6 |   445.8 | [link](https://huggingface.co/timm/maxvit_large_tf_384.in21k_ft_in1k)  |
+| maxvit_base_tf_384.in21k_ft_in1k   |   87.9 |         119.6 |   73.8 |   332.9 | [link](https://huggingface.co/timm/maxvit_base_tf_384.in21k_ft_in1k)   |
+| maxvit_base_tf_512.in1k            |   86.6 |         119.9 |  138   |   704   | [link](https://huggingface.co/timm/maxvit_base_tf_512.in1k)            |
+| maxvit_large_tf_512.in1k           |   86.5 |         212.3 |  244.8 |   942.2 | [link](https://huggingface.co/timm/maxvit_large_tf_512.in1k)           |
+| maxvit_base_tf_384.in1k            |   86.3 |         119.6 |   73.8 |   332.9 | [link](https://huggingface.co/timm/maxvit_base_tf_384.in1k)            |
+| maxvit_large_tf_384.in1k           |   86.2 |         212   |  132.6 |   445.8 | [link](https://huggingface.co/timm/maxvit_large_tf_384.in1k)           |
+| maxvit_small_tf_512.in1k           |   86.1 |          69.1 |   67.3 |   383.8 | [link](https://huggingface.co/timm/maxvit_small_tf_512.in1k)           |
+| maxvit_tiny_tf_512.in1k            |   85.7 |          31   |   33.5 |   257.6 | [link](https://huggingface.co/timm/maxvit_tiny_tf_512.in1k)            |
+| maxvit_small_tf_384.in1k           |   85.5 |          69   |   35.9 |   183.6 | [link](https://huggingface.co/timm/maxvit_small_tf_384.in1k)           |
+| maxvit_tiny_tf_384.in1k            |   85.1 |          31   |   17.5 |   123.4 | [link](https://huggingface.co/timm/maxvit_tiny_tf_384.in1k)            |
+| maxvit_large_tf_224.in1k           |   84.9 |         211.8 |   43.7 |   127.4 | [link](https://huggingface.co/timm/maxvit_large_tf_224.in1k)           |
+| maxvit_base_tf_224.in1k            |   84.9 |         119.5 |   24   |    95   | [link](https://huggingface.co/timm/maxvit_base_tf_224.in1k)            |
+| maxvit_small_tf_224.in1k           |   84.4 |          68.9 |   11.7 |    53.2 | [link](https://huggingface.co/timm/maxvit_small_tf_224.in1k)           |
+| maxvit_tiny_tf_224.in1k            |   83.4 |          30.9 |    5.6 |    35.8 | [link](https://huggingface.co/timm/maxvit_tiny_tf_224.in1k)            |
+
+### Oct 15, 2022
+* Train and validation script enhancements
+* Non-GPU (ie CPU) device support
+* SLURM compatibility for train script
+* HF datasets support (via ReaderHfds)
+* TFDS/WDS dataloading improvements (sample padding/wrap for distributed use fixed wrt sample count estimate)
+* in_chans !=3 support for scripts / loader
+* Adan optimizer
+* Can enable per-step LR scheduling via args
+* Dataset 'parsers' renamed to 'readers', more descriptive of purpose
+* AMP args changed, APEX via `--amp-impl apex`, bfloat16 supportedf via `--amp-dtype bfloat16`
+* main branch switched to 0.7.x version, 0.6x forked for stable release of weight only adds
+* master -> main branch rename
+
+### Oct 10, 2022
+* More weights in `maxxvit` series, incl first ConvNeXt block based `coatnext` and `maxxvit` experiments:
+  * `coatnext_nano_rw_224` - 82.0 @ 224 (G) -- (uses ConvNeXt conv block, no BatchNorm)
+  * `maxxvit_rmlp_nano_rw_256` - 83.0 @ 256, 83.7 @ 320  (G) (uses ConvNeXt conv block, no BN)
+  * `maxvit_rmlp_small_rw_224` - 84.5 @ 224, 85.1 @ 320 (G)
+  * `maxxvit_rmlp_small_rw_256` - 84.6 @ 256, 84.9 @ 288 (G) -- could be trained better, hparams need tuning (uses ConvNeXt block, no BN)
+  * `coatnet_rmlp_2_rw_224` - 84.6 @ 224, 85 @ 320  (T)
+  * NOTE: official MaxVit weights (in1k) have been released at https://github.com/google-research/maxvit -- some extra work is needed to port and adapt since my impl was created independently of theirs and has a few small differences + the whole TF same padding fun.
+  
+### Sept 23, 2022
+* LAION-2B CLIP image towers supported as pretrained backbones for fine-tune or features (no classifier)
+  * vit_base_patch32_224_clip_laion2b
+  * vit_large_patch14_224_clip_laion2b
+  * vit_huge_patch14_224_clip_laion2b
+  * vit_giant_patch14_224_clip_laion2b
+
+### Sept 7, 2022
+* Hugging Face [`timm` docs](https://huggingface.co/docs/hub/timm) home now exists, look for more here in the future
+* Add BEiT-v2 weights for base and large 224x224 models from https://github.com/microsoft/unilm/tree/master/beit2
+* Add more weights in `maxxvit` series incl a `pico` (7.5M params, 1.9 GMACs), two `tiny` variants:
+  * `maxvit_rmlp_pico_rw_256` - 80.5 @ 256, 81.3 @ 320  (T)
+  * `maxvit_tiny_rw_224` - 83.5 @ 224 (G)
+  * `maxvit_rmlp_tiny_rw_256` - 84.2 @ 256, 84.8 @ 320 (T)
+
+### Aug 29, 2022
+* MaxVit window size scales with img_size by default. Add new RelPosMlp MaxViT weight that leverages this:
+  * `maxvit_rmlp_nano_rw_256` - 83.0 @ 256, 83.6 @ 320  (T)
+
+### Aug 26, 2022
+* CoAtNet (https://arxiv.org/abs/2106.04803) and MaxVit (https://arxiv.org/abs/2204.01697) `timm` original models
+  * both found in [`maxxvit.py`](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/maxxvit.py) model def, contains numerous experiments outside scope of original papers
+  * an unfinished Tensorflow version from MaxVit authors can be found https://github.com/google-research/maxvit
+* Initial CoAtNet and MaxVit timm pretrained weights (working on more):
+  * `coatnet_nano_rw_224` - 81.7 @ 224  (T)
+  * `coatnet_rmlp_nano_rw_224` - 82.0 @ 224, 82.8 @ 320 (T)
+  * `coatnet_0_rw_224` - 82.4  (T)  -- NOTE timm '0' coatnets have 2 more 3rd stage blocks
+  * `coatnet_bn_0_rw_224` - 82.4  (T)
+  * `maxvit_nano_rw_256` - 82.9 @ 256  (T)
+  * `coatnet_rmlp_1_rw_224` - 83.4 @ 224, 84 @ 320  (T)
+  * `coatnet_1_rw_224` - 83.6 @ 224 (G) 
+  * (T) = TPU trained with `bits_and_tpu` branch training code, (G) = GPU trained
+* GCVit (weights adapted from https://github.com/NVlabs/GCVit, code 100% `timm` re-write for license purposes)
+* MViT-V2 (multi-scale vit, adapted from https://github.com/facebookresearch/mvit)
+* EfficientFormer (adapted from https://github.com/snap-research/EfficientFormer)
+* PyramidVisionTransformer-V2 (adapted from https://github.com/whai362/PVT)
+* 'Fast Norm' support for LayerNorm and GroupNorm that avoids float32 upcast w/ AMP (uses APEX LN if available for further boost)
+
+
+### Aug 15, 2022
+* ConvNeXt atto weights added
+  * `convnext_atto` - 75.7 @ 224, 77.0 @ 288
+  * `convnext_atto_ols` - 75.9  @ 224, 77.2 @ 288
+
+### Aug 5, 2022
+* More custom ConvNeXt smaller model defs with weights 
+  * `convnext_femto` - 77.5 @ 224, 78.7 @ 288
+  * `convnext_femto_ols` - 77.9  @ 224, 78.9 @ 288
+  * `convnext_pico` - 79.5 @ 224, 80.4 @ 288
+  * `convnext_pico_ols` - 79.5 @ 224, 80.5 @ 288
+  * `convnext_nano_ols` - 80.9 @ 224, 81.6 @ 288
+* Updated EdgeNeXt to improve ONNX export, add new base variant and weights from original (https://github.com/mmaaz60/EdgeNeXt)
+
+### July 28, 2022
+* Add freshly minted DeiT-III Medium (width=512, depth=12, num_heads=8) model weights. Thanks [Hugo Touvron](https://github.com/TouvronHugo)!
+
+### July 27, 2022
+* All runtime benchmark and validation result csv files are up-to-date!
+* A few more weights & model defs added:
+  * `darknetaa53` -  79.8 @ 256, 80.5 @ 288
+  * `convnext_nano` - 80.8 @ 224, 81.5 @ 288
+  * `cs3sedarknet_l` - 81.2 @ 256, 81.8 @ 288
+  * `cs3darknet_x` - 81.8 @ 256, 82.2 @ 288
+  * `cs3sedarknet_x` - 82.2 @ 256, 82.7 @ 288
+  * `cs3edgenet_x` - 82.2 @ 256, 82.7 @ 288
+  * `cs3se_edgenet_x` - 82.8 @ 256, 83.5 @ 320
+* `cs3*` weights above all trained on TPU w/ `bits_and_tpu` branch. Thanks to TRC program!
+* Add output_stride=8 and 16 support to ConvNeXt (dilation)
+* deit3 models not being able to resize pos_emb fixed
+* Version 0.6.7 PyPi release (/w above bug fixes and new weighs since 0.6.5)
+
+### July 8, 2022
+More models, more fixes
+* Official research models (w/ weights) added:
+  * EdgeNeXt from (https://github.com/mmaaz60/EdgeNeXt)
+  * MobileViT-V2 from (https://github.com/apple/ml-cvnets)
+  * DeiT III (Revenge of the ViT) from (https://github.com/facebookresearch/deit)
+* My own models:
+  * Small `ResNet` defs added by request with 1 block repeats for both basic and bottleneck (resnet10 and resnet14)
+  * `CspNet` refactored with dataclass config, simplified CrossStage3 (`cs3`) option. These are closer to YOLO-v5+ backbone defs.
+  * More relative position vit fiddling. Two `srelpos` (shared relative position) models trained, and a medium w/ class token.
+  * Add an alternate downsample mode to EdgeNeXt and train a `small` model. Better than original small, but not their new USI trained weights.
+* My own model weight results (all ImageNet-1k training)
+  * `resnet10t` - 66.5 @ 176, 68.3 @ 224
+  * `resnet14t` - 71.3 @ 176, 72.3 @ 224
+  * `resnetaa50` - 80.6 @ 224 , 81.6 @ 288
+  * `darknet53` -  80.0 @ 256, 80.5 @ 288
+  * `cs3darknet_m` - 77.0 @ 256, 77.6 @ 288
+  * `cs3darknet_focus_m` - 76.7 @ 256, 77.3 @ 288
+  * `cs3darknet_l` - 80.4 @ 256, 80.9 @ 288
+  * `cs3darknet_focus_l` - 80.3 @ 256, 80.9 @ 288
+  * `vit_srelpos_small_patch16_224` - 81.1 @ 224, 82.1 @ 320
+  * `vit_srelpos_medium_patch16_224` - 82.3 @ 224, 83.1 @ 320
+  * `vit_relpos_small_patch16_cls_224` - 82.6 @ 224, 83.6 @ 320
+  * `edgnext_small_rw` - 79.6 @ 224, 80.4 @ 320
+* `cs3`, `darknet`, and `vit_*relpos` weights above all trained on TPU thanks to TRC program! Rest trained on overheating GPUs.
+* Hugging Face Hub support fixes verified, demo notebook TBA
+* Pretrained weights / configs can be loaded externally (ie from local disk) w/ support for head adaptation.
+* Add support to change image extensions scanned by `timm` datasets/parsers. See (https://github.com/rwightman/pytorch-image-models/pull/1274#issuecomment-1178303103)
+* Default ConvNeXt LayerNorm impl to use `F.layer_norm(x.permute(0, 2, 3, 1), ...).permute(0, 3, 1, 2)` via `LayerNorm2d` in all cases. 
+  * a bit slower than previous custom impl on some hardware (ie Ampere w/ CL), but overall fewer regressions across wider HW / PyTorch version ranges. 
+  * previous impl exists as `LayerNormExp2d` in `models/layers/norm.py`
+* Numerous bug fixes
+* Currently testing for imminent PyPi 0.6.x release
+* LeViT pretraining of larger models still a WIP, they don't train well / easily without distillation. Time to add distill support (finally)?
+* ImageNet-22k weight training + finetune ongoing, work on multi-weight support (slowly) chugging along (there are a LOT of weights, sigh) ...
+
+### May 13, 2022
+* Official Swin-V2 models and weights added from (https://github.com/microsoft/Swin-Transformer). Cleaned up to support torchscript.
+* Some refactoring for existing `timm` Swin-V2-CR impl, will likely do a bit more to bring parts closer to official and decide whether to merge some aspects.
+* More Vision Transformer relative position / residual post-norm experiments (all trained on TPU thanks to TRC program)
+  * `vit_relpos_small_patch16_224` - 81.5 @ 224, 82.5 @ 320 -- rel pos, layer scale, no class token, avg pool
+  * `vit_relpos_medium_patch16_rpn_224` - 82.3 @ 224, 83.1 @ 320 -- rel pos + res-post-norm, no class token, avg pool
+  * `vit_relpos_medium_patch16_224` - 82.5 @ 224, 83.3 @ 320 -- rel pos, layer scale, no class token, avg pool
+  * `vit_relpos_base_patch16_gapcls_224` - 82.8 @ 224, 83.9 @ 320 -- rel pos, layer scale, class token, avg pool (by mistake)
+* Bring 512 dim, 8-head 'medium' ViT model variant back to life (after using in a pre DeiT 'small' model for first ViT impl back in 2020)
+* Add ViT relative position support for switching btw existing impl and some additions in official Swin-V2 impl for future trials
+* Sequencer2D impl (https://arxiv.org/abs/2205.01972), added via PR from author (https://github.com/okojoalg)
+
+### May 2, 2022
+* Vision Transformer experiments adding Relative Position (Swin-V2 log-coord) (`vision_transformer_relpos.py`) and Residual Post-Norm branches (from Swin-V2) (`vision_transformer*.py`)
+  * `vit_relpos_base_patch32_plus_rpn_256` - 79.5 @ 256, 80.6 @ 320 -- rel pos + extended width + res-post-norm, no class token, avg pool
+  * `vit_relpos_base_patch16_224` - 82.5 @ 224, 83.6 @ 320 -- rel pos, layer scale, no class token, avg pool
+  * `vit_base_patch16_rpn_224` - 82.3 @ 224 -- rel pos + res-post-norm, no class token, avg pool
+* Vision Transformer refactor to remove representation layer that was only used in initial vit and rarely used since with newer pretrain (ie `How to Train Your ViT`)
+* `vit_*` models support removal of class token, use of global average pool, use of fc_norm (ala beit, mae).
+
+### April 22, 2022
+* `timm` models are now officially supported in [fast.ai](https://www.fast.ai/)! Just in time for the new Practical Deep Learning course. `timmdocs` documentation link updated to [timm.fast.ai](http://timm.fast.ai/).
+* Two more model weights added in the TPU trained [series](https://github.com/rwightman/pytorch-image-models/releases/tag/v0.1-tpu-weights). Some In22k pretrain still in progress.
+  * `seresnext101d_32x8d` - 83.69 @ 224, 84.35 @ 288
+  * `seresnextaa101d_32x8d` (anti-aliased w/ AvgPool2d) - 83.85 @ 224, 84.57 @ 288
+
+### March 23, 2022
+* Add `ParallelBlock` and `LayerScale` option to base vit models to support model configs in [Three things everyone should know about ViT](https://arxiv.org/abs/2203.09795)
+* `convnext_tiny_hnf` (head norm first) weights trained with (close to) A2 recipe, 82.2% top-1, could do better with more epochs.
+
+### March 21, 2022
+* Merge `norm_norm_norm`. **IMPORTANT** this update for a coming 0.6.x release will likely de-stabilize the master branch for a while. Branch [`0.5.x`](https://github.com/rwightman/pytorch-image-models/tree/0.5.x) or a previous 0.5.x release can be used if stability is required.
+* Significant weights update (all TPU trained) as described in this [release](https://github.com/rwightman/pytorch-image-models/releases/tag/v0.1-tpu-weights)
+  * `regnety_040` - 82.3 @ 224, 82.96 @ 288
+  * `regnety_064` - 83.0 @ 224, 83.65 @ 288
+  * `regnety_080` - 83.17 @ 224, 83.86 @ 288
+  * `regnetv_040` - 82.44 @ 224, 83.18 @ 288   (timm pre-act)
+  * `regnetv_064` - 83.1 @ 224, 83.71 @ 288   (timm pre-act)
+  * `regnetz_040` - 83.67 @ 256, 84.25 @ 320
+  * `regnetz_040h` - 83.77 @ 256, 84.5 @ 320 (w/ extra fc in head)
+  * `resnetv2_50d_gn` - 80.8 @ 224, 81.96 @ 288 (pre-act GroupNorm)
+  * `resnetv2_50d_evos` 80.77 @ 224, 82.04 @ 288 (pre-act EvoNormS)
+  * `regnetz_c16_evos`  - 81.9 @ 256, 82.64 @ 320 (EvoNormS)
+  * `regnetz_d8_evos`  - 83.42 @ 256, 84.04 @ 320 (EvoNormS)
+  * `xception41p` - 82 @ 299   (timm pre-act)
+  * `xception65` -  83.17 @ 299
+  * `xception65p` -  83.14 @ 299   (timm pre-act)
+  * `resnext101_64x4d` - 82.46 @ 224, 83.16 @ 288
+  * `seresnext101_32x8d` - 83.57 @ 224, 84.270 @ 288
+  * `resnetrs200` - 83.85 @ 256, 84.44 @ 320
+* HuggingFace hub support fixed w/ initial groundwork for allowing alternative 'config sources' for pretrained model definitions and weights (generic local file / remote url support soon)
+* SwinTransformer-V2 implementation added. Submitted by [Christoph Reich](https://github.com/ChristophReich1996). Training experiments and model changes by myself are ongoing so expect compat breaks.
+* Swin-S3 (AutoFormerV2) models / weights added from https://github.com/microsoft/Cream/tree/main/AutoFormerV2
+* MobileViT models w/ weights adapted from https://github.com/apple/ml-cvnets
+* PoolFormer models w/ weights adapted from https://github.com/sail-sg/poolformer
+* VOLO models w/ weights adapted from https://github.com/sail-sg/volo
+* Significant work experimenting with non-BatchNorm norm layers such as EvoNorm, FilterResponseNorm, GroupNorm, etc
+* Enhance support for alternate norm + act ('NormAct') layers added to a number of models, esp EfficientNet/MobileNetV3, RegNet, and aligned Xception
+* Grouped conv support added to EfficientNet family
+* Add 'group matching' API to all models to allow grouping model parameters for application of 'layer-wise' LR decay, lr scale added to LR scheduler
+* Gradient checkpointing support added to many models
+* `forward_head(x, pre_logits=False)` fn added to all models to allow separate calls of `forward_features` + `forward_head`
+* All vision transformer and vision MLP models update to return non-pooled / non-token selected features from `foward_features`, for consistency with CNN models, token selection or pooling now applied in `forward_head`
+
+### Feb 2, 2022
+* [Chris Hughes](https://github.com/Chris-hughes10) posted an exhaustive run through of `timm` on his blog yesterday. Well worth a read. [Getting Started with PyTorch Image Models (timm): A Practitioner’s Guide](https://towardsdatascience.com/getting-started-with-pytorch-image-models-timm-a-practitioners-guide-4e77b4bf9055)
+* I'm currently prepping to merge the `norm_norm_norm` branch back to master (ver 0.6.x) in next week or so.
+  * The changes are more extensive than usual and may destabilize and break some model API use (aiming for full backwards compat). So, beware `pip install git+https://github.com/rwightman/pytorch-image-models` installs!
+  * `0.5.x` releases and a `0.5.x` branch will remain stable with a cherry pick or two until dust clears. Recommend sticking to pypi install for a bit if you want stable.
+
+### Jan 14, 2022
+* Version 0.5.4 w/ release to be pushed to pypi. It's been a while since last pypi update and riskier changes will be merged to main branch soon....
+* Add ConvNeXT models /w weights from official impl (https://github.com/facebookresearch/ConvNeXt), a few perf tweaks, compatible with timm features
+* Tried training a few small (~1.8-3M param) / mobile optimized models, a few are good so far, more on the way...
+  * `mnasnet_small` - 65.6 top-1
+  * `mobilenetv2_050` - 65.9
+  * `lcnet_100/075/050` - 72.1 / 68.8 / 63.1
+  * `semnasnet_075` - 73
+  * `fbnetv3_b/d/g` - 79.1 / 79.7 / 82.0
+* TinyNet models added by [rsomani95](https://github.com/rsomani95)
+* LCNet added via MobileNetV3 architecture
+

docs/feature_extraction.md

@@ -0,0 +1,174 @@
+# Feature Extraction
+
+All of the models in `timm` have consistent mechanisms for obtaining various types of features from the model for tasks besides classification.
+
+## Penultimate Layer Features (Pre-Classifier Features)
+
+The features from the penultimate model layer can be obtained in several ways without requiring model surgery (although feel free to do surgery). One must first decide if they want pooled or un-pooled features.
+
+### Unpooled
+
+There are three ways to obtain unpooled features.
+
+Without modifying the network, one can call `model.forward_features(input)` on any model instead of the usual `model(input)`. This will bypass the head classifier and global pooling for networks.
+
+If one wants to explicitly modify the network to return unpooled features, they can either create the model without a classifier and pooling, or remove it later. Both paths remove the parameters associated with the classifier from the network.
+
+#### forward_features()
+```python hl_lines="3 6"
+import torch
+import timm
+m = timm.create_model('xception41', pretrained=True)
+o = m(torch.randn(2, 3, 299, 299))
+print(f'Original shape: {o.shape}')
+o = m.forward_features(torch.randn(2, 3, 299, 299))
+print(f'Unpooled shape: {o.shape}')
+```
+Output:
+```text
+Original shape: torch.Size([2, 1000])
+Unpooled shape: torch.Size([2, 2048, 10, 10])
+```
+
+#### Create with no classifier and pooling
+```python hl_lines="3"
+import torch
+import timm
+m = timm.create_model('resnet50', pretrained=True, num_classes=0, global_pool='')
+o = m(torch.randn(2, 3, 224, 224))
+print(f'Unpooled shape: {o.shape}')
+```
+Output:
+```text
+Unpooled shape: torch.Size([2, 2048, 7, 7])
+```
+
+#### Remove it later
+```python hl_lines="3 6"
+import torch
+import timm
+m = timm.create_model('densenet121', pretrained=True)
+o = m(torch.randn(2, 3, 224, 224))
+print(f'Original shape: {o.shape}')
+m.reset_classifier(0, '')
+o = m(torch.randn(2, 3, 224, 224))
+print(f'Unpooled shape: {o.shape}')
+```
+Output:
+```text
+Original shape: torch.Size([2, 1000])
+Unpooled shape: torch.Size([2, 1024, 7, 7])
+```
+
+### Pooled
+
+To modify the network to return pooled features, one can use `forward_features()` and pool/flatten the result themselves, or modify the network like above but keep pooling intact. 
+
+#### Create with no classifier
+```python hl_lines="3"
+import torch
+import timm
+m = timm.create_model('resnet50', pretrained=True, num_classes=0)
+o = m(torch.randn(2, 3, 224, 224))
+print(f'Pooled shape: {o.shape}')
+```
+Output:
+```text
+Pooled shape: torch.Size([2, 2048])
+```
+
+#### Remove it later
+```python hl_lines="3 6"
+import torch
+import timm
+m = timm.create_model('ese_vovnet19b_dw', pretrained=True)
+o = m(torch.randn(2, 3, 224, 224))
+print(f'Original shape: {o.shape}')
+m.reset_classifier(0)
+o = m(torch.randn(2, 3, 224, 224))
+print(f'Pooled shape: {o.shape}')
+```
+Output:
+```text
+Original shape: torch.Size([2, 1000])
+Pooled shape: torch.Size([2, 1024])
+```
+
+
+## Multi-scale Feature Maps (Feature Pyramid)
+
+Object detection, segmentation, keypoint, and a variety of dense pixel tasks require access to feature maps from the backbone network at multiple scales. This is often done by modifying the original classification network. Since each network varies quite a bit in structure, it's not uncommon to see only a few backbones supported in any given obj detection or segmentation library.
+
+`timm` allows a consistent interface for creating any of the included models as feature backbones that output feature maps for selected levels. 
+
+A feature backbone can be created by adding the argument `features_only=True` to any `create_model` call. By default 5 strides will be output from most models (not all have that many), with the first starting at 2 (some start at 1 or 4).
+
+### Create a feature map extraction model
+```python hl_lines="3"
+import torch
+import timm
+m = timm.create_model('resnest26d', features_only=True, pretrained=True)
+o = m(torch.randn(2, 3, 224, 224))
+for x in o:
+  print(x.shape)
+```
+Output:
+```text
+torch.Size([2, 64, 112, 112])
+torch.Size([2, 256, 56, 56])
+torch.Size([2, 512, 28, 28])
+torch.Size([2, 1024, 14, 14])
+torch.Size([2, 2048, 7, 7])
+```
+
+### Query the feature information
+
+After a feature backbone has been created, it can be queried to provide channel or resolution reduction information to the downstream heads without requiring static config or hardcoded constants. The `.feature_info` attribute is a class encapsulating the information about the feature extraction points.
+
+```python hl_lines="3 4"
+import torch
+import timm
+m = timm.create_model('regnety_032', features_only=True, pretrained=True)
+print(f'Feature channels: {m.feature_info.channels()}')
+o = m(torch.randn(2, 3, 224, 224))
+for x in o:
+  print(x.shape)
+```
+Output:
+```text
+Feature channels: [32, 72, 216, 576, 1512]
+torch.Size([2, 32, 112, 112])
+torch.Size([2, 72, 56, 56])
+torch.Size([2, 216, 28, 28])
+torch.Size([2, 576, 14, 14])
+torch.Size([2, 1512, 7, 7])
+```
+
+### Select specific feature levels or limit the stride
+
+There are two additional creation arguments impacting the output features. 
+
+* `out_indices` selects which indices to output
+* `output_stride` limits the feature output stride of the network (also works in classification mode BTW)
+
+`out_indices` is supported by all models, but not all models have the same index to feature stride mapping. Look at the code or check feature_info to compare. The out indices generally correspond to the `C(i+1)th` feature level (a `2^(i+1)` reduction). For most models, index 0 is the stride 2 features, and index 4 is stride 32.
+
+`output_stride` is achieved by converting layers to use dilated convolutions. Doing so is not always straightforward, some networks only support `output_stride=32`.
+
+```python hl_lines="3 4 5"
+import torch
+import timm
+m = timm.create_model('ecaresnet101d', features_only=True, output_stride=8, out_indices=(2, 4), pretrained=True)
+print(f'Feature channels: {m.feature_info.channels()}')
+print(f'Feature reduction: {m.feature_info.reduction()}')
+o = m(torch.randn(2, 3, 320, 320))
+for x in o:
+  print(x.shape)
+```
+Output:
+```text
+Feature channels: [512, 2048]
+Feature reduction: [8, 8]
+torch.Size([2, 512, 40, 40])
+torch.Size([2, 2048, 40, 40])
+```

docs/index.md

@@ -0,0 +1,80 @@
+# Getting Started
+
+## Welcome
+
+Welcome to the `timm` documentation, a lean set of docs that covers the basics of `timm`.
+
+For a more comprehensive set of docs (currently under development), please visit [timmdocs](http://timm.fast.ai) by [Aman Arora](https://github.com/amaarora).
+
+## Install
+
+The library can be installed with pip:
+
+```
+pip install timm
+```
+
+I update the PyPi (pip) packages when I'm confident there are no significant model regressions from previous releases. If you want to pip install the bleeding edge from GitHub, use:
+```
+pip install git+https://github.com/rwightman/pytorch-image-models.git
+```
+
+!!! info "Conda Environment"
+    All development and testing has been done in Conda Python 3 environments on Linux x86-64 systems, specifically 3.7, 3.8, 3.9, 3.10
+    
+    Little to no care has been taken to be Python 2.x friendly and will not support it. If you run into any challenges running on Windows, or other OS, I'm definitely open to looking into those issues so long as it's in a reproducible (read Conda) environment.
+    
+    PyTorch versions 1.9, 1.10, 1.11 have been tested with the latest versions of this code.
+    
+    I've tried to keep the dependencies minimal, the setup is as per the PyTorch default install instructions for Conda:
+    ```
+    conda create -n torch-env
+    conda activate torch-env
+    conda install pytorch torchvision cudatoolkit=11.3 -c pytorch
+    conda install pyyaml
+    ```
+
+## Load a Pretrained Model
+
+Pretrained models can be loaded using `timm.create_model`
+
+```python
+import timm
+
+m = timm.create_model('mobilenetv3_large_100', pretrained=True)
+m.eval()
+```
+
+## List Models with Pretrained Weights
+```python
+import timm
+from pprint import pprint
+model_names = timm.list_models(pretrained=True)
+pprint(model_names)
+>>> ['adv_inception_v3',
+ 'cspdarknet53',
+ 'cspresnext50',
+ 'densenet121',
+ 'densenet161',
+ 'densenet169',
+ 'densenet201',
+ 'densenetblur121d',
+ 'dla34',
+ 'dla46_c',
+...
+]
+```
+
+## List Model Architectures by Wildcard
+```python
+import timm
+from pprint import pprint
+model_names = timm.list_models('*resne*t*')
+pprint(model_names)
+>>> ['cspresnet50',
+ 'cspresnet50d',
+ 'cspresnet50w',
+ 'cspresnext50',
+...
+]
+```

docs/javascripts/tables.js

@@ -0,0 +1,6 @@
+app.location$.subscribe(function() {
+  var tables = document.querySelectorAll("article table")
+  tables.forEach(function(table) {
+    new Tablesort(table)
+  })
+})

docs/models.md

@@ -0,0 +1,171 @@
+# Model Summaries
+
+The model architectures included come from a wide variety of sources. Sources, including papers, original impl ("reference code") that I rewrote / adapted, and PyTorch impl that I leveraged directly ("code") are listed below.
+
+Most included models have pretrained weights. The weights are either:
+
+1. from their original sources
+2. ported by myself from their original impl in a different framework (e.g. Tensorflow models)
+3. trained from scratch using the included training script
+
+The validation results for the pretrained weights are [here](results.md)
+
+A more exciting view (with pretty pictures) of the models within `timm` can be found at [paperswithcode](https://paperswithcode.com/lib/timm).
+
+## Big Transfer ResNetV2 (BiT) [[resnetv2.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/resnetv2.py)]
+* Paper: `Big Transfer (BiT): General Visual Representation Learning` - https://arxiv.org/abs/1912.11370
+* Reference code: https://github.com/google-research/big_transfer
+
+## Cross-Stage Partial Networks [[cspnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/cspnet.py)]
+* Paper: `CSPNet: A New Backbone that can Enhance Learning Capability of CNN` - https://arxiv.org/abs/1911.11929
+* Reference impl: https://github.com/WongKinYiu/CrossStagePartialNetworks
+
+## DenseNet [[densenet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/densenet.py)]
+* Paper: `Densely Connected Convolutional Networks` - https://arxiv.org/abs/1608.06993
+* Code: https://github.com/pytorch/vision/tree/master/torchvision/models
+
+## DLA [[dla.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/dla.py)]
+* Paper: https://arxiv.org/abs/1707.06484
+* Code: https://github.com/ucbdrive/dla
+
+## Dual-Path Networks [[dpn.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/dpn.py)]
+* Paper: `Dual Path Networks` - https://arxiv.org/abs/1707.01629
+* My PyTorch code: https://github.com/rwightman/pytorch-dpn-pretrained
+* Reference code: https://github.com/cypw/DPNs
+
+## GPU-Efficient Networks [[byobnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/byobnet.py)]
+* Paper: `Neural Architecture Design for GPU-Efficient Networks` - https://arxiv.org/abs/2006.14090
+* Reference code: https://github.com/idstcv/GPU-Efficient-Networks
+
+## HRNet [[hrnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/hrnet.py)]
+* Paper: `Deep High-Resolution Representation Learning for Visual Recognition` - https://arxiv.org/abs/1908.07919
+* Code: https://github.com/HRNet/HRNet-Image-Classification
+
+## Inception-V3 [[inception_v3.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/inception_v3.py)]
+* Paper: `Rethinking the Inception Architecture for Computer Vision` - https://arxiv.org/abs/1512.00567
+* Code: https://github.com/pytorch/vision/tree/master/torchvision/models
+
+## Inception-V4 [[inception_v4.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/inception_v4.py)]
+* Paper: `Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning` - https://arxiv.org/abs/1602.07261
+* Code: https://github.com/Cadene/pretrained-models.pytorch
+* Reference code: https://github.com/tensorflow/models/tree/master/research/slim/nets
+
+## Inception-ResNet-V2 [[inception_resnet_v2.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/inception_resnet_v2.py)]
+* Paper: `Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning` - https://arxiv.org/abs/1602.07261
+* Code: https://github.com/Cadene/pretrained-models.pytorch
+* Reference code: https://github.com/tensorflow/models/tree/master/research/slim/nets
+
+## NASNet-A [[nasnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/nasnet.py)]
+* Papers: `Learning Transferable Architectures for Scalable Image Recognition` - https://arxiv.org/abs/1707.07012
+* Code: https://github.com/Cadene/pretrained-models.pytorch
+* Reference code: https://github.com/tensorflow/models/tree/master/research/slim/nets/nasnet
+
+## PNasNet-5 [[pnasnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/pnasnet.py)]
+* Papers: `Progressive Neural Architecture Search` - https://arxiv.org/abs/1712.00559
+* Code: https://github.com/Cadene/pretrained-models.pytorch
+* Reference code: https://github.com/tensorflow/models/tree/master/research/slim/nets/nasnet
+
+## EfficientNet [[efficientnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/efficientnet.py)]
+
+* Papers:
+  * EfficientNet NoisyStudent (B0-B7, L2) - https://arxiv.org/abs/1911.04252
+  * EfficientNet AdvProp (B0-B8) - https://arxiv.org/abs/1911.09665
+  * EfficientNet (B0-B7) - https://arxiv.org/abs/1905.11946
+  * EfficientNet-EdgeTPU (S, M, L) - https://ai.googleblog.com/2019/08/efficientnet-edgetpu-creating.html
+  * MixNet - https://arxiv.org/abs/1907.09595
+  * MNASNet B1, A1 (Squeeze-Excite), and Small - https://arxiv.org/abs/1807.11626
+  * MobileNet-V2 - https://arxiv.org/abs/1801.04381
+  * FBNet-C - https://arxiv.org/abs/1812.03443
+  * Single-Path NAS - https://arxiv.org/abs/1904.02877
+* My PyTorch code: https://github.com/rwightman/gen-efficientnet-pytorch
+* Reference code: https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet
+
+## MobileNet-V3 [[mobilenetv3.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/mobilenetv3.py)]
+* Paper: `Searching for MobileNetV3` - https://arxiv.org/abs/1905.02244
+* Reference code: https://github.com/tensorflow/models/tree/master/research/slim/nets/mobilenet
+
+## RegNet [[regnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/regnet.py)]
+* Paper: `Designing Network Design Spaces` - https://arxiv.org/abs/2003.13678
+* Reference code: https://github.com/facebookresearch/pycls/blob/master/pycls/models/regnet.py
+
+## RepVGG [[byobnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/byobnet.py)]
+* Paper: `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697
+* Reference code: https://github.com/DingXiaoH/RepVGG
+
+## ResNet, ResNeXt [[resnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/resnet.py)]
+
+* ResNet (V1B)
+  * Paper: `Deep Residual Learning for Image Recognition` - https://arxiv.org/abs/1512.03385
+  * Code: https://github.com/pytorch/vision/tree/master/torchvision/models
+* ResNeXt
+  * Paper: `Aggregated Residual Transformations for Deep Neural Networks` - https://arxiv.org/abs/1611.05431
+  * Code: https://github.com/pytorch/vision/tree/master/torchvision/models
+* 'Bag of Tricks' / Gluon C, D, E, S ResNet variants
+  * Paper: `Bag of Tricks for Image Classification with CNNs` - https://arxiv.org/abs/1812.01187
+  * Code: https://github.com/dmlc/gluon-cv/blob/master/gluoncv/model_zoo/resnetv1b.py
+* Instagram pretrained / ImageNet tuned ResNeXt101
+  * Paper: `Exploring the Limits of Weakly Supervised Pretraining` - https://arxiv.org/abs/1805.00932
+  * Weights: https://pytorch.org/hub/facebookresearch_WSL-Images_resnext (NOTE: CC BY-NC 4.0 License, NOT commercial friendly)
+* Semi-supervised (SSL) / Semi-weakly Supervised (SWSL) ResNet and ResNeXts
+  * Paper: `Billion-scale semi-supervised learning for image classification` - https://arxiv.org/abs/1905.00546
+  * Weights: https://github.com/facebookresearch/semi-supervised-ImageNet1K-models (NOTE: CC BY-NC 4.0 License, NOT commercial friendly)
+* Squeeze-and-Excitation Networks
+  * Paper: `Squeeze-and-Excitation Networks` - https://arxiv.org/abs/1709.01507
+  * Code: Added to ResNet base, this is current version going forward, old `senet.py` is being deprecated
+* ECAResNet (ECA-Net)
+  * Paper: `ECA-Net: Efficient Channel Attention for Deep CNN` - https://arxiv.org/abs/1910.03151v4
+  * Code: Added to ResNet base, ECA module contributed by @VRandme, reference https://github.com/BangguWu/ECANet
+
+## Res2Net [[res2net.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/res2net.py)]
+* Paper: `Res2Net: A New Multi-scale Backbone Architecture` - https://arxiv.org/abs/1904.01169
+* Code: https://github.com/gasvn/Res2Net
+
+## ResNeSt [[resnest.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/resnest.py)]
+* Paper: `ResNeSt: Split-Attention Networks` - https://arxiv.org/abs/2004.08955
+* Code: https://github.com/zhanghang1989/ResNeSt
+
+## ReXNet [[rexnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/rexnet.py)]
+* Paper: `ReXNet: Diminishing Representational Bottleneck on CNN` - https://arxiv.org/abs/2007.00992
+* Code: https://github.com/clovaai/rexnet
+
+## Selective-Kernel Networks [[sknet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/sknet.py)]
+* Paper: `Selective-Kernel Networks` - https://arxiv.org/abs/1903.06586
+* Code: https://github.com/implus/SKNet, https://github.com/clovaai/assembled-cnn
+
+## SelecSLS [[selecsls.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/selecsls.py)]
+* Paper: `XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera` - https://arxiv.org/abs/1907.00837
+* Code: https://github.com/mehtadushy/SelecSLS-Pytorch
+
+## Squeeze-and-Excitation Networks [[senet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/senet.py)]
+NOTE: I am deprecating this version of the networks, the new ones are part of `resnet.py`
+
+* Paper: `Squeeze-and-Excitation Networks` - https://arxiv.org/abs/1709.01507
+* Code: https://github.com/Cadene/pretrained-models.pytorch 
+
+## TResNet [[tresnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/tresnet.py)]
+* Paper: `TResNet: High Performance GPU-Dedicated Architecture` - https://arxiv.org/abs/2003.13630
+* Code: https://github.com/mrT23/TResNet
+
+## VGG [[vgg.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vgg.py)]
+* Paper: `Very Deep Convolutional Networks For Large-Scale Image Recognition` - https://arxiv.org/pdf/1409.1556.pdf
+* Reference code: https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py
+
+## Vision Transformer [[vision_transformer.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py)]
+* Paper: `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale` - https://arxiv.org/abs/2010.11929
+* Reference code and pretrained weights: https://github.com/google-research/vision_transformer
+
+## VovNet V2 and V1 [[vovnet.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vovnet.py)]
+* Paper: `CenterMask : Real-Time Anchor-Free Instance Segmentation` - https://arxiv.org/abs/1911.06667
+* Reference code: https://github.com/youngwanLEE/vovnet-detectron2
+
+## Xception [[xception.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/xception.py)]
+* Paper: `Xception: Deep Learning with Depthwise Separable Convolutions` - https://arxiv.org/abs/1610.02357
+* Code: https://github.com/Cadene/pretrained-models.pytorch
+
+## Xception (Modified Aligned, Gluon) [[gluon_xception.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/gluon_xception.py)]
+* Paper: `Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation` - https://arxiv.org/abs/1802.02611
+* Reference code: https://github.com/dmlc/gluon-cv/tree/master/gluoncv/model_zoo, https://github.com/jfzhang95/pytorch-deeplab-xception/
+
+## Xception (Modified Aligned, TF) [[aligned_xception.py](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/aligned_xception.py)]
+* Paper: `Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation` - https://arxiv.org/abs/1802.02611
+* Reference code: https://github.com/tensorflow/models/tree/master/research/deeplab

docs/models/.pages

@@ -0,0 +1 @@
+title: Model Pages

docs/models/.templates/code_snippets.md

@@ -0,0 +1,62 @@
+## How do I use this model on an image?
+To load a pretrained model:
+
+```python
+import timm
+model = timm.create_model('{{ model_name }}', pretrained=True)
+model.eval()
+```
+
+To load and preprocess the image:
+```python 
+import urllib
+from PIL import Image
+from timm.data import resolve_data_config
+from timm.data.transforms_factory import create_transform
+
+config = resolve_data_config({}, model=model)
+transform = create_transform(**config)
+
+url, filename = ("https://github.com/pytorch/hub/raw/master/images/dog.jpg", "dog.jpg")
+urllib.request.urlretrieve(url, filename)
+img = Image.open(filename).convert('RGB')
+tensor = transform(img).unsqueeze(0) # transform and add batch dimension
+```
+
+To get the model predictions:
+```python
+import torch
+with torch.no_grad():
+    out = model(tensor)
+probabilities = torch.nn.functional.softmax(out[0], dim=0)
+print(probabilities.shape)
+# prints: torch.Size([1000])
+```
+
+To get the top-5 predictions class names:
+```python
+# Get imagenet class mappings
+url, filename = ("https://raw.githubusercontent.com/pytorch/hub/master/imagenet_classes.txt", "imagenet_classes.txt")
+urllib.request.urlretrieve(url, filename) 
+with open("imagenet_classes.txt", "r") as f:
+    categories = [s.strip() for s in f.readlines()]
+
+# Print top categories per image
+top5_prob, top5_catid = torch.topk(probabilities, 5)
+for i in range(top5_prob.size(0)):
+    print(categories[top5_catid[i]], top5_prob[i].item())
+# prints class names and probabilities like:
+# [('Samoyed', 0.6425196528434753), ('Pomeranian', 0.04062102362513542), ('keeshond', 0.03186424449086189), ('white wolf', 0.01739676296710968), ('Eskimo dog', 0.011717947199940681)]
+```
+
+Replace the model name with the variant you want to use, e.g. `{{ model_name }}`. You can find the IDs in the model summaries at the top of this page.
+
+To extract image features with this model, follow the [timm feature extraction examples](https://rwightman.github.io/pytorch-image-models/feature_extraction/), just change the name of the model you want to use.
+
+## How do I finetune this model?
+You can finetune any of the pre-trained models just by changing the classifier (the last layer).
+```python
+model = timm.create_model('{{ model_name }}', pretrained=True, num_classes=NUM_FINETUNE_CLASSES)
+```
+To finetune on your own dataset, you have to write a training loop or adapt [timm's training
+script](https://github.com/rwightman/pytorch-image-models/blob/master/train.py) to use your dataset.

docs/models/.templates/generate_readmes.py

@@ -0,0 +1,64 @@
+"""
+Run this script to generate the model-index files in `models` from the templates in `.templates/models`.
+"""
+
+import argparse
+from pathlib import Path
+
+from jinja2 import Environment, FileSystemLoader
+
+import modelindex
+
+
+def generate_readmes(templates_path: Path, dest_path: Path):
+    """Add the code snippet template to the readmes"""
+    readme_templates_path = templates_path / "models"
+    code_template_path = templates_path / "code_snippets.md"
+
+    env = Environment(
+        loader=FileSystemLoader([readme_templates_path, readme_templates_path.parent]),
+    )
+
+    for readme in readme_templates_path.iterdir():
+        if readme.suffix == ".md":
+            template = env.get_template(readme.name)
+
+            # get the first model_name for this model family
+            mi = modelindex.load(str(readme))
+            model_name = mi.models[0].name
+
+            full_content = template.render(model_name=model_name)
+
+            # generate full_readme
+            with open(dest_path / readme.name, "w") as f:
+                f.write(full_content)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Model index generation config")
+    parser.add_argument(
+        "-t",
+        "--templates",
+        default=Path(__file__).parent / ".templates",
+        type=str,
+        help="Location of the markdown templates",
+    )
+    parser.add_argument(
+        "-d",
+        "--dest",
+        default=Path(__file__).parent / "models",
+        type=str,
+        help="Destination folder that contains the generated model-index files.",
+    )
+    args = parser.parse_args()
+    templates_path = Path(args.templates)
+    dest_readmes_path = Path(args.dest)
+
+    generate_readmes(
+        templates_path,
+        dest_readmes_path,
+    )
+
+
+if __name__ == "__main__":
+    main()

docs/models/.templates/models/adversarial-inception-v3.md

@@ -0,0 +1,98 @@
+# Adversarial Inception v3
+
+**Inception v3** is a convolutional neural network architecture from the Inception family that makes several improvements including using [Label Smoothing](https://paperswithcode.com/method/label-smoothing), Factorized 7 x 7 convolutions, and the use of an [auxiliary classifer](https://paperswithcode.com/method/auxiliary-classifier) to propagate label information lower down the network (along with the use of batch normalization for layers in the sidehead). The key building block is an [Inception Module](https://paperswithcode.com/method/inception-v3-module).
+
+This particular model was trained for study of adversarial examples (adversarial training).
+
+The weights from this model were ported from [Tensorflow/Models](https://github.com/tensorflow/models).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@article{DBLP:journals/corr/abs-1804-00097,
+  author    = {Alexey Kurakin and
+               Ian J. Goodfellow and
+               Samy Bengio and
+               Yinpeng Dong and
+               Fangzhou Liao and
+               Ming Liang and
+               Tianyu Pang and
+               Jun Zhu and
+               Xiaolin Hu and
+               Cihang Xie and
+               Jianyu Wang and
+               Zhishuai Zhang and
+               Zhou Ren and
+               Alan L. Yuille and
+               Sangxia Huang and
+               Yao Zhao and
+               Yuzhe Zhao and
+               Zhonglin Han and
+               Junjiajia Long and
+               Yerkebulan Berdibekov and
+               Takuya Akiba and
+               Seiya Tokui and
+               Motoki Abe},
+  title     = {Adversarial Attacks and Defences Competition},
+  journal   = {CoRR},
+  volume    = {abs/1804.00097},
+  year      = {2018},
+  url       = {http://arxiv.org/abs/1804.00097},
+  archivePrefix = {arXiv},
+  eprint    = {1804.00097},
+  timestamp = {Thu, 31 Oct 2019 16:31:22 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1804-00097.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: Adversarial Inception v3
+  Paper:
+    Title: Adversarial Attacks and Defences Competition
+    URL: https://paperswithcode.com/paper/adversarial-attacks-and-defences-competition
+Models:
+- Name: adv_inception_v3
+  In Collection: Adversarial Inception v3
+  Metadata:
+    FLOPs: 7352418880
+    Parameters: 23830000
+    File Size: 95549439
+    Architecture:
+    - 1x1 Convolution
+    - Auxiliary Classifier
+    - Average Pooling
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inception-v3 Module
+    - Max Pooling
+    - ReLU
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: adv_inception_v3
+    Crop Pct: '0.875'
+    Image Size: '299'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/inception_v3.py#L456
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/adv_inception_v3-9e27bd63.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 77.58%
+      Top 5 Accuracy: 93.74%
+-->

docs/models/.templates/models/advprop.md

@@ -0,0 +1,457 @@
+# AdvProp (EfficientNet)
+
+**AdvProp** is an adversarial training scheme which treats adversarial examples as additional examples, to prevent overfitting. Key to the method is the usage of a separate auxiliary batch norm for adversarial examples, as they have different underlying distributions to normal examples.
+
+The weights from this model were ported from [Tensorflow/TPU](https://github.com/tensorflow/tpu).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{xie2020adversarial,
+      title={Adversarial Examples Improve Image Recognition}, 
+      author={Cihang Xie and Mingxing Tan and Boqing Gong and Jiang Wang and Alan Yuille and Quoc V. Le},
+      year={2020},
+      eprint={1911.09665},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: AdvProp
+  Paper:
+    Title: Adversarial Examples Improve Image Recognition
+    URL: https://paperswithcode.com/paper/adversarial-examples-improve-image
+Models:
+- Name: tf_efficientnet_b0_ap
+  In Collection: AdvProp
+  Metadata:
+    FLOPs: 488688572
+    Parameters: 5290000
+    File Size: 21385973
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - AdvProp
+    - AutoAugment
+    - Label Smoothing
+    - RMSProp
+    - Stochastic Depth
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: tf_efficientnet_b0_ap
+    LR: 0.256
+    Epochs: 350
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 2048
+    Image Size: '224'
+    Weight Decay: 1.0e-05
+    Interpolation: bicubic
+    RMSProp Decay: 0.9
+    Label Smoothing: 0.1
+    BatchNorm Momentum: 0.99
+  Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/efficientnet.py#L1334
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b0_ap-f262efe1.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 77.1%
+      Top 5 Accuracy: 93.26%
+- Name: tf_efficientnet_b1_ap
+  In Collection: AdvProp
+  Metadata:
+    FLOPs: 883633200
+    Parameters: 7790000
+    File Size: 31515350
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - AdvProp
+    - AutoAugment
+    - Label Smoothing
+    - RMSProp
+    - Stochastic Depth
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: tf_efficientnet_b1_ap
+    LR: 0.256
+    Epochs: 350
+    Crop Pct: '0.882'
+    Momentum: 0.9
+    Batch Size: 2048
+    Image Size: '240'
+    Weight Decay: 1.0e-05
+    Interpolation: bicubic
+    RMSProp Decay: 0.9
+    Label Smoothing: 0.1
+    BatchNorm Momentum: 0.99
+  Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/efficientnet.py#L1344
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b1_ap-44ef0a3d.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.28%
+      Top 5 Accuracy: 94.3%
+- Name: tf_efficientnet_b2_ap
+  In Collection: AdvProp
+  Metadata:
+    FLOPs: 1234321170
+    Parameters: 9110000
+    File Size: 36800745
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - AdvProp
+    - AutoAugment
+    - Label Smoothing
+    - RMSProp
+    - Stochastic Depth
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: tf_efficientnet_b2_ap
+    LR: 0.256
+    Epochs: 350
+    Crop Pct: '0.89'
+    Momentum: 0.9
+    Batch Size: 2048
+    Image Size: '260'
+    Weight Decay: 1.0e-05
+    Interpolation: bicubic
+    RMSProp Decay: 0.9
+    Label Smoothing: 0.1
+    BatchNorm Momentum: 0.99
+  Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/efficientnet.py#L1354
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b2_ap-2f8e7636.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.3%
+      Top 5 Accuracy: 95.03%
+- Name: tf_efficientnet_b3_ap
+  In Collection: AdvProp
+  Metadata:
+    FLOPs: 2275247568
+    Parameters: 12230000
+    File Size: 49384538
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - AdvProp
+    - AutoAugment
+    - Label Smoothing
+    - RMSProp
+    - Stochastic Depth
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: tf_efficientnet_b3_ap
+    LR: 0.256
+    Epochs: 350
+    Crop Pct: '0.904'
+    Momentum: 0.9
+    Batch Size: 2048
+    Image Size: '300'
+    Weight Decay: 1.0e-05
+    Interpolation: bicubic
+    RMSProp Decay: 0.9
+    Label Smoothing: 0.1
+    BatchNorm Momentum: 0.99
+  Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/efficientnet.py#L1364
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b3_ap-aad25bdd.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 81.82%
+      Top 5 Accuracy: 95.62%
+- Name: tf_efficientnet_b4_ap
+  In Collection: AdvProp
+  Metadata:
+    FLOPs: 5749638672
+    Parameters: 19340000
+    File Size: 77993585
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - AdvProp
+    - AutoAugment
+    - Label Smoothing
+    - RMSProp
+    - Stochastic Depth
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: tf_efficientnet_b4_ap
+    LR: 0.256
+    Epochs: 350
+    Crop Pct: '0.922'
+    Momentum: 0.9
+    Batch Size: 2048
+    Image Size: '380'
+    Weight Decay: 1.0e-05
+    Interpolation: bicubic
+    RMSProp Decay: 0.9
+    Label Smoothing: 0.1
+    BatchNorm Momentum: 0.99
+  Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/efficientnet.py#L1374
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b4_ap-dedb23e6.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 83.26%
+      Top 5 Accuracy: 96.39%
+- Name: tf_efficientnet_b5_ap
+  In Collection: AdvProp
+  Metadata:
+    FLOPs: 13176501888
+    Parameters: 30390000
+    File Size: 122403150
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - AdvProp
+    - AutoAugment
+    - Label Smoothing
+    - RMSProp
+    - Stochastic Depth
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: tf_efficientnet_b5_ap
+    LR: 0.256
+    Epochs: 350
+    Crop Pct: '0.934'
+    Momentum: 0.9
+    Batch Size: 2048
+    Image Size: '456'
+    Weight Decay: 1.0e-05
+    Interpolation: bicubic
+    RMSProp Decay: 0.9
+    Label Smoothing: 0.1
+    BatchNorm Momentum: 0.99
+  Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/efficientnet.py#L1384
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b5_ap-9e82fae8.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 84.25%
+      Top 5 Accuracy: 96.97%
+- Name: tf_efficientnet_b6_ap
+  In Collection: AdvProp
+  Metadata:
+    FLOPs: 24180518488
+    Parameters: 43040000
+    File Size: 173237466
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - AdvProp
+    - AutoAugment
+    - Label Smoothing
+    - RMSProp
+    - Stochastic Depth
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: tf_efficientnet_b6_ap
+    LR: 0.256
+    Epochs: 350
+    Crop Pct: '0.942'
+    Momentum: 0.9
+    Batch Size: 2048
+    Image Size: '528'
+    Weight Decay: 1.0e-05
+    Interpolation: bicubic
+    RMSProp Decay: 0.9
+    Label Smoothing: 0.1
+    BatchNorm Momentum: 0.99
+  Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/efficientnet.py#L1394
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b6_ap-4ffb161f.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 84.79%
+      Top 5 Accuracy: 97.14%
+- Name: tf_efficientnet_b7_ap
+  In Collection: AdvProp
+  Metadata:
+    FLOPs: 48205304880
+    Parameters: 66349999
+    File Size: 266850607
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - AdvProp
+    - AutoAugment
+    - Label Smoothing
+    - RMSProp
+    - Stochastic Depth
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: tf_efficientnet_b7_ap
+    LR: 0.256
+    Epochs: 350
+    Crop Pct: '0.949'
+    Momentum: 0.9
+    Batch Size: 2048
+    Image Size: '600'
+    Weight Decay: 1.0e-05
+    Interpolation: bicubic
+    RMSProp Decay: 0.9
+    Label Smoothing: 0.1
+    BatchNorm Momentum: 0.99
+  Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/efficientnet.py#L1405
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b7_ap-ddb28fec.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 85.12%
+      Top 5 Accuracy: 97.25%
+- Name: tf_efficientnet_b8_ap
+  In Collection: AdvProp
+  Metadata:
+    FLOPs: 80962956270
+    Parameters: 87410000
+    File Size: 351412563
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - AdvProp
+    - AutoAugment
+    - Label Smoothing
+    - RMSProp
+    - Stochastic Depth
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: tf_efficientnet_b8_ap
+    LR: 0.128
+    Epochs: 350
+    Crop Pct: '0.954'
+    Momentum: 0.9
+    Batch Size: 2048
+    Image Size: '672'
+    Weight Decay: 1.0e-05
+    Interpolation: bicubic
+    RMSProp Decay: 0.9
+    Label Smoothing: 0.1
+    BatchNorm Momentum: 0.99
+  Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/efficientnet.py#L1416
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/tf_efficientnet_b8_ap-00e169fa.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 85.37%
+      Top 5 Accuracy: 97.3%
+-->

docs/models/.templates/models/big-transfer.md

@@ -0,0 +1,295 @@
+# Big Transfer (BiT)
+
+**Big Transfer (BiT)** is a type of pretraining recipe that pre-trains  on a large supervised source dataset, and fine-tunes the weights on the target task. Models are trained on the JFT-300M dataset. The finetuned models contained in this collection are finetuned on ImageNet.
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{kolesnikov2020big,
+      title={Big Transfer (BiT): General Visual Representation Learning}, 
+      author={Alexander Kolesnikov and Lucas Beyer and Xiaohua Zhai and Joan Puigcerver and Jessica Yung and Sylvain Gelly and Neil Houlsby},
+      year={2020},
+      eprint={1912.11370},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: Big Transfer
+  Paper:
+    Title: 'Big Transfer (BiT): General Visual Representation Learning'
+    URL: https://paperswithcode.com/paper/large-scale-learning-of-general-visual
+Models:
+- Name: resnetv2_101x1_bitm
+  In Collection: Big Transfer
+  Metadata:
+    FLOPs: 5330896
+    Parameters: 44540000
+    File Size: 178256468
+    Architecture:
+    - 1x1 Convolution
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Group Normalization
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    - Weight Standardization
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Mixup
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    - JFT-300M
+    Training Resources: Cloud TPUv3-512
+    ID: resnetv2_101x1_bitm
+    LR: 0.03
+    Epochs: 90
+    Layers: 101
+    Crop Pct: '1.0'
+    Momentum: 0.9
+    Batch Size: 4096
+    Image Size: '480'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/b9843f954b0457af2db4f9dea41a8538f51f5d78/timm/models/resnetv2.py#L444
+  Weights: https://storage.googleapis.com/bit_models/BiT-M-R101x1-ILSVRC2012.npz
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 82.21%
+      Top 5 Accuracy: 96.47%
+- Name: resnetv2_101x3_bitm
+  In Collection: Big Transfer
+  Metadata:
+    FLOPs: 15988688
+    Parameters: 387930000
+    File Size: 1551830100
+    Architecture:
+    - 1x1 Convolution
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Group Normalization
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    - Weight Standardization
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Mixup
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    - JFT-300M
+    Training Resources: Cloud TPUv3-512
+    ID: resnetv2_101x3_bitm
+    LR: 0.03
+    Epochs: 90
+    Layers: 101
+    Crop Pct: '1.0'
+    Momentum: 0.9
+    Batch Size: 4096
+    Image Size: '480'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/b9843f954b0457af2db4f9dea41a8538f51f5d78/timm/models/resnetv2.py#L451
+  Weights: https://storage.googleapis.com/bit_models/BiT-M-R101x3-ILSVRC2012.npz
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 84.38%
+      Top 5 Accuracy: 97.37%
+- Name: resnetv2_152x2_bitm
+  In Collection: Big Transfer
+  Metadata:
+    FLOPs: 10659792
+    Parameters: 236340000
+    File Size: 945476668
+    Architecture:
+    - 1x1 Convolution
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Group Normalization
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    - Weight Standardization
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Mixup
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    - JFT-300M
+    ID: resnetv2_152x2_bitm
+    Crop Pct: '1.0'
+    Image Size: '480'
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/b9843f954b0457af2db4f9dea41a8538f51f5d78/timm/models/resnetv2.py#L458
+  Weights: https://storage.googleapis.com/bit_models/BiT-M-R152x2-ILSVRC2012.npz
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 84.4%
+      Top 5 Accuracy: 97.43%
+- Name: resnetv2_152x4_bitm
+  In Collection: Big Transfer
+  Metadata:
+    FLOPs: 21317584
+    Parameters: 936530000
+    File Size: 3746270104
+    Architecture:
+    - 1x1 Convolution
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Group Normalization
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    - Weight Standardization
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Mixup
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    - JFT-300M
+    Training Resources: Cloud TPUv3-512
+    ID: resnetv2_152x4_bitm
+    Crop Pct: '1.0'
+    Image Size: '480'
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/b9843f954b0457af2db4f9dea41a8538f51f5d78/timm/models/resnetv2.py#L465
+  Weights: https://storage.googleapis.com/bit_models/BiT-M-R152x4-ILSVRC2012.npz
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 84.95%
+      Top 5 Accuracy: 97.45%
+- Name: resnetv2_50x1_bitm
+  In Collection: Big Transfer
+  Metadata:
+    FLOPs: 5330896
+    Parameters: 25550000
+    File Size: 102242668
+    Architecture:
+    - 1x1 Convolution
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Group Normalization
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    - Weight Standardization
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Mixup
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    - JFT-300M
+    Training Resources: Cloud TPUv3-512
+    ID: resnetv2_50x1_bitm
+    LR: 0.03
+    Epochs: 90
+    Layers: 50
+    Crop Pct: '1.0'
+    Momentum: 0.9
+    Batch Size: 4096
+    Image Size: '480'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/b9843f954b0457af2db4f9dea41a8538f51f5d78/timm/models/resnetv2.py#L430
+  Weights: https://storage.googleapis.com/bit_models/BiT-M-R50x1-ILSVRC2012.npz
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.19%
+      Top 5 Accuracy: 95.63%
+- Name: resnetv2_50x3_bitm
+  In Collection: Big Transfer
+  Metadata:
+    FLOPs: 15988688
+    Parameters: 217320000
+    File Size: 869321580
+    Architecture:
+    - 1x1 Convolution
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Group Normalization
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    - Weight Standardization
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Mixup
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    - JFT-300M
+    Training Resources: Cloud TPUv3-512
+    ID: resnetv2_50x3_bitm
+    LR: 0.03
+    Epochs: 90
+    Layers: 50
+    Crop Pct: '1.0'
+    Momentum: 0.9
+    Batch Size: 4096
+    Image Size: '480'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/b9843f954b0457af2db4f9dea41a8538f51f5d78/timm/models/resnetv2.py#L437
+  Weights: https://storage.googleapis.com/bit_models/BiT-M-R50x3-ILSVRC2012.npz
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 83.75%
+      Top 5 Accuracy: 97.12%
+-->

docs/models/.templates/models/csp-darknet.md

@@ -0,0 +1,81 @@
+# CSP-DarkNet
+
+**CSPDarknet53** is a convolutional neural network and backbone for object detection that uses [DarkNet-53](https://paperswithcode.com/method/darknet-53). It employs a CSPNet strategy to partition the feature map of the base layer into two parts and then merges them through a cross-stage hierarchy. The use of a split and merge strategy allows for more gradient flow through the network. 
+
+This CNN is used as the backbone for [YOLOv4](https://paperswithcode.com/method/yolov4).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{bochkovskiy2020yolov4,
+      title={YOLOv4: Optimal Speed and Accuracy of Object Detection}, 
+      author={Alexey Bochkovskiy and Chien-Yao Wang and Hong-Yuan Mark Liao},
+      year={2020},
+      eprint={2004.10934},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: CSP DarkNet
+  Paper:
+    Title: 'YOLOv4: Optimal Speed and Accuracy of Object Detection'
+    URL: https://paperswithcode.com/paper/yolov4-optimal-speed-and-accuracy-of-object
+Models:
+- Name: cspdarknet53
+  In Collection: CSP DarkNet
+  Metadata:
+    FLOPs: 8545018880
+    Parameters: 27640000
+    File Size: 110775135
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Mish
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - CutMix
+    - Label Smoothing
+    - Mosaic
+    - Polynomial Learning Rate Decay
+    - SGD with Momentum
+    - Self-Adversarial Training
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 1x NVIDIA RTX 2070 GPU
+    ID: cspdarknet53
+    LR: 0.1
+    Layers: 53
+    Crop Pct: '0.887'
+    Momentum: 0.9
+    Batch Size: 128
+    Image Size: '256'
+    Warmup Steps: 1000
+    Weight Decay: 0.0005
+    Interpolation: bilinear
+    Training Steps: 8000000
+    FPS (GPU RTX 2070): 66
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/cspnet.py#L441
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/cspdarknet53_ra_256-d05c7c21.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.05%
+      Top 5 Accuracy: 95.09%
+-->

docs/models/.templates/models/csp-resnet.md

@@ -0,0 +1,76 @@
+# CSP-ResNet
+
+**CSPResNet** is a convolutional neural network where we apply the Cross Stage Partial Network (CSPNet) approach to [ResNet](https://paperswithcode.com/method/resnet). The CSPNet partitions the feature map of the base layer into two parts and then merges them through a cross-stage hierarchy. The use of a split and merge strategy allows for more gradient flow through the network.
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{wang2019cspnet,
+      title={CSPNet: A New Backbone that can Enhance Learning Capability of CNN}, 
+      author={Chien-Yao Wang and Hong-Yuan Mark Liao and I-Hau Yeh and Yueh-Hua Wu and Ping-Yang Chen and Jun-Wei Hsieh},
+      year={2019},
+      eprint={1911.11929},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: CSP ResNet
+  Paper:
+    Title: 'CSPNet: A New Backbone that can Enhance Learning Capability of CNN'
+    URL: https://paperswithcode.com/paper/cspnet-a-new-backbone-that-can-enhance
+Models:
+- Name: cspresnet50
+  In Collection: CSP ResNet
+  Metadata:
+    FLOPs: 5924992000
+    Parameters: 21620000
+    File Size: 86679303
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Label Smoothing
+    - Polynomial Learning Rate Decay
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: cspresnet50
+    LR: 0.1
+    Layers: 50
+    Crop Pct: '0.887'
+    Momentum: 0.9
+    Batch Size: 128
+    Image Size: '256'
+    Weight Decay: 0.005
+    Interpolation: bilinear
+    Training Steps: 8000000
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/cspnet.py#L415
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/cspresnet50_ra-d3e8d487.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.57%
+      Top 5 Accuracy: 94.71%
+-->

docs/models/.templates/models/csp-resnext.md

@@ -0,0 +1,77 @@
+# CSP-ResNeXt
+
+**CSPResNeXt** is a convolutional neural network where we apply the Cross Stage Partial Network (CSPNet) approach to [ResNeXt](https://paperswithcode.com/method/resnext). The CSPNet partitions the feature map of the base layer into two parts and then merges them through a cross-stage hierarchy. The use of a split and merge strategy allows for more gradient flow through the network.
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{wang2019cspnet,
+      title={CSPNet: A New Backbone that can Enhance Learning Capability of CNN}, 
+      author={Chien-Yao Wang and Hong-Yuan Mark Liao and I-Hau Yeh and Yueh-Hua Wu and Ping-Yang Chen and Jun-Wei Hsieh},
+      year={2019},
+      eprint={1911.11929},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: CSP ResNeXt
+  Paper:
+    Title: 'CSPNet: A New Backbone that can Enhance Learning Capability of CNN'
+    URL: https://paperswithcode.com/paper/cspnet-a-new-backbone-that-can-enhance
+Models:
+- Name: cspresnext50
+  In Collection: CSP ResNeXt
+  Metadata:
+    FLOPs: 3962945536
+    Parameters: 20570000
+    File Size: 82562887
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Grouped Convolution
+    - Max Pooling
+    - ReLU
+    - ResNeXt Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Label Smoothing
+    - Polynomial Learning Rate Decay
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 1x GPU
+    ID: cspresnext50
+    LR: 0.1
+    Layers: 50
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 128
+    Image Size: '224'
+    Weight Decay: 0.005
+    Interpolation: bilinear
+    Training Steps: 8000000
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/cspnet.py#L430
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/cspresnext50_ra_224-648b4713.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.05%
+      Top 5 Accuracy: 94.94%
+-->

docs/models/.templates/models/densenet.md

@@ -0,0 +1,305 @@
+# DenseNet
+
+**DenseNet** is a type of convolutional neural network that utilises dense connections between layers, through [Dense Blocks](http://www.paperswithcode.com/method/dense-block), where we connect *all layers* (with matching feature-map sizes) directly with each other. To preserve the feed-forward nature, each layer obtains additional inputs from all preceding layers and passes on its own feature-maps to all subsequent layers.
+
+The **DenseNet Blur** variant in this collection by Ross Wightman employs [Blur Pooling](http://www.paperswithcode.com/method/blur-pooling)
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@article{DBLP:journals/corr/HuangLW16a,
+  author    = {Gao Huang and
+               Zhuang Liu and
+               Kilian Q. Weinberger},
+  title     = {Densely Connected Convolutional Networks},
+  journal   = {CoRR},
+  volume    = {abs/1608.06993},
+  year      = {2016},
+  url       = {http://arxiv.org/abs/1608.06993},
+  archivePrefix = {arXiv},
+  eprint    = {1608.06993},
+  timestamp = {Mon, 10 Sep 2018 15:49:32 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/HuangLW16a.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+```
+@misc{rw2019timm,
+  author = {Ross Wightman},
+  title = {PyTorch Image Models},
+  year = {2019},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  doi = {10.5281/zenodo.4414861},
+  howpublished = {\url{https://github.com/rwightman/pytorch-image-models}}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: DenseNet
+  Paper:
+    Title: Densely Connected Convolutional Networks
+    URL: https://paperswithcode.com/paper/densely-connected-convolutional-networks
+Models:
+- Name: densenet121
+  In Collection: DenseNet
+  Metadata:
+    FLOPs: 3641843200
+    Parameters: 7980000
+    File Size: 32376726
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Block
+    - Dense Connections
+    - Dropout
+    - Max Pooling
+    - ReLU
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Kaiming Initialization
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: densenet121
+    LR: 0.1
+    Epochs: 90
+    Layers: 121
+    Dropout: 0.2
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/densenet.py#L295
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/densenet121_ra-50efcf5c.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 75.56%
+      Top 5 Accuracy: 92.65%
+- Name: densenet161
+  In Collection: DenseNet
+  Metadata:
+    FLOPs: 9931959264
+    Parameters: 28680000
+    File Size: 115730790
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Block
+    - Dense Connections
+    - Dropout
+    - Max Pooling
+    - ReLU
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Kaiming Initialization
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: densenet161
+    LR: 0.1
+    Epochs: 90
+    Layers: 161
+    Dropout: 0.2
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/densenet.py#L347
+  Weights: https://download.pytorch.org/models/densenet161-8d451a50.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 77.36%
+      Top 5 Accuracy: 93.63%
+- Name: densenet169
+  In Collection: DenseNet
+  Metadata:
+    FLOPs: 4316945792
+    Parameters: 14150000
+    File Size: 57365526
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Block
+    - Dense Connections
+    - Dropout
+    - Max Pooling
+    - ReLU
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Kaiming Initialization
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: densenet169
+    LR: 0.1
+    Epochs: 90
+    Layers: 169
+    Dropout: 0.2
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/densenet.py#L327
+  Weights: https://download.pytorch.org/models/densenet169-b2777c0a.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 75.9%
+      Top 5 Accuracy: 93.02%
+- Name: densenet201
+  In Collection: DenseNet
+  Metadata:
+    FLOPs: 5514321024
+    Parameters: 20010000
+    File Size: 81131730
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Block
+    - Dense Connections
+    - Dropout
+    - Max Pooling
+    - ReLU
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Kaiming Initialization
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: densenet201
+    LR: 0.1
+    Epochs: 90
+    Layers: 201
+    Dropout: 0.2
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/densenet.py#L337
+  Weights: https://download.pytorch.org/models/densenet201-c1103571.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 77.29%
+      Top 5 Accuracy: 93.48%
+- Name: densenetblur121d
+  In Collection: DenseNet
+  Metadata:
+    FLOPs: 3947812864
+    Parameters: 8000000
+    File Size: 32456500
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Blur Pooling
+    - Convolution
+    - Dense Block
+    - Dense Connections
+    - Dropout
+    - Max Pooling
+    - ReLU
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: densenetblur121d
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/densenet.py#L305
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/densenetblur121d_ra-100dcfbc.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 76.59%
+      Top 5 Accuracy: 93.2%
+- Name: tv_densenet121
+  In Collection: DenseNet
+  Metadata:
+    FLOPs: 3641843200
+    Parameters: 7980000
+    File Size: 32342954
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Block
+    - Dense Connections
+    - Dropout
+    - Max Pooling
+    - ReLU
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: tv_densenet121
+    LR: 0.1
+    Epochs: 90
+    Crop Pct: '0.875'
+    LR Gamma: 0.1
+    Momentum: 0.9
+    Batch Size: 32
+    Image Size: '224'
+    LR Step Size: 30
+    Weight Decay: 0.0001
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/densenet.py#L379
+  Weights: https://download.pytorch.org/models/densenet121-a639ec97.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 74.74%
+      Top 5 Accuracy: 92.15%
+-->

docs/models/.templates/models/dla.md

@@ -0,0 +1,545 @@
+# Deep Layer Aggregation
+
+Extending  “shallow” skip connections, **Dense Layer Aggregation (DLA)** incorporates more depth and sharing. The authors introduce two structures for deep layer aggregation (DLA): iterative deep aggregation (IDA) and hierarchical deep aggregation (HDA). These structures are expressed through an architectural framework, independent of the choice of backbone, for compatibility with current and future networks. 
+
+IDA focuses on fusing resolutions and scales while HDA focuses on merging features from all modules and channels. IDA follows the base hierarchy to refine resolution and aggregate scale stage-bystage. HDA assembles its own hierarchy of tree-structured connections that cross and merge stages to aggregate different levels of representation. 
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{yu2019deep,
+      title={Deep Layer Aggregation}, 
+      author={Fisher Yu and Dequan Wang and Evan Shelhamer and Trevor Darrell},
+      year={2019},
+      eprint={1707.06484},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: DLA
+  Paper:
+    Title: Deep Layer Aggregation
+    URL: https://paperswithcode.com/paper/deep-layer-aggregation
+Models:
+- Name: dla102
+  In Collection: DLA
+  Metadata:
+    FLOPs: 7192952808
+    Parameters: 33270000
+    File Size: 135290579
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 8x GPUs
+    ID: dla102
+    LR: 0.1
+    Epochs: 120
+    Layers: 102
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L410
+  Weights: http://dl.yf.io/dla/models/imagenet/dla102-d94d9790.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.03%
+      Top 5 Accuracy: 93.95%
+- Name: dla102x
+  In Collection: DLA
+  Metadata:
+    FLOPs: 5886821352
+    Parameters: 26310000
+    File Size: 107552695
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 8x GPUs
+    ID: dla102x
+    LR: 0.1
+    Epochs: 120
+    Layers: 102
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L418
+  Weights: http://dl.yf.io/dla/models/imagenet/dla102x-ad62be81.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.51%
+      Top 5 Accuracy: 94.23%
+- Name: dla102x2
+  In Collection: DLA
+  Metadata:
+    FLOPs: 9343847400
+    Parameters: 41280000
+    File Size: 167645295
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 8x GPUs
+    ID: dla102x2
+    LR: 0.1
+    Epochs: 120
+    Layers: 102
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L426
+  Weights: http://dl.yf.io/dla/models/imagenet/dla102x2-262837b6.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.44%
+      Top 5 Accuracy: 94.65%
+- Name: dla169
+  In Collection: DLA
+  Metadata:
+    FLOPs: 11598004200
+    Parameters: 53390000
+    File Size: 216547113
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 8x GPUs
+    ID: dla169
+    LR: 0.1
+    Epochs: 120
+    Layers: 169
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L434
+  Weights: http://dl.yf.io/dla/models/imagenet/dla169-0914e092.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.69%
+      Top 5 Accuracy: 94.33%
+- Name: dla34
+  In Collection: DLA
+  Metadata:
+    FLOPs: 3070105576
+    Parameters: 15740000
+    File Size: 63228658
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: dla34
+    LR: 0.1
+    Epochs: 120
+    Layers: 32
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L362
+  Weights: http://dl.yf.io/dla/models/imagenet/dla34-ba72cf86.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 74.62%
+      Top 5 Accuracy: 92.06%
+- Name: dla46_c
+  In Collection: DLA
+  Metadata:
+    FLOPs: 583277288
+    Parameters: 1300000
+    File Size: 5307963
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: dla46_c
+    LR: 0.1
+    Epochs: 120
+    Layers: 46
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L369
+  Weights: http://dl.yf.io/dla/models/imagenet/dla46_c-2bfd52c3.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 64.87%
+      Top 5 Accuracy: 86.29%
+- Name: dla46x_c
+  In Collection: DLA
+  Metadata:
+    FLOPs: 544052200
+    Parameters: 1070000
+    File Size: 4387641
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: dla46x_c
+    LR: 0.1
+    Epochs: 120
+    Layers: 46
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L378
+  Weights: http://dl.yf.io/dla/models/imagenet/dla46x_c-d761bae7.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 65.98%
+      Top 5 Accuracy: 86.99%
+- Name: dla60
+  In Collection: DLA
+  Metadata:
+    FLOPs: 4256251880
+    Parameters: 22040000
+    File Size: 89560235
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: dla60
+    LR: 0.1
+    Epochs: 120
+    Layers: 60
+    Dropout: 0.2
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L394
+  Weights: http://dl.yf.io/dla/models/imagenet/dla60-24839fc4.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 77.04%
+      Top 5 Accuracy: 93.32%
+- Name: dla60_res2net
+  In Collection: DLA
+  Metadata:
+    FLOPs: 4147578504
+    Parameters: 20850000
+    File Size: 84886593
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: dla60_res2net
+    Layers: 60
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L346
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-res2net/res2net_dla60_4s-d88db7f9.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.46%
+      Top 5 Accuracy: 94.21%
+- Name: dla60_res2next
+  In Collection: DLA
+  Metadata:
+    FLOPs: 3485335272
+    Parameters: 17030000
+    File Size: 69639245
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: dla60_res2next
+    Layers: 60
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L354
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-res2net/res2next_dla60_4s-d327927b.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.44%
+      Top 5 Accuracy: 94.16%
+- Name: dla60x
+  In Collection: DLA
+  Metadata:
+    FLOPs: 3544204264
+    Parameters: 17350000
+    File Size: 70883139
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: dla60x
+    LR: 0.1
+    Epochs: 120
+    Layers: 60
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L402
+  Weights: http://dl.yf.io/dla/models/imagenet/dla60x-d15cacda.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.25%
+      Top 5 Accuracy: 94.02%
+- Name: dla60x_c
+  In Collection: DLA
+  Metadata:
+    FLOPs: 593325032
+    Parameters: 1320000
+    File Size: 5454396
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - DLA Bottleneck Residual Block
+    - DLA Residual Block
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: dla60x_c
+    LR: 0.1
+    Epochs: 120
+    Layers: 60
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dla.py#L386
+  Weights: http://dl.yf.io/dla/models/imagenet/dla60x_c-b870c45c.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 67.91%
+      Top 5 Accuracy: 88.42%
+-->

docs/models/.templates/models/dpn.md

@@ -0,0 +1,256 @@
+# Dual Path Network (DPN)
+
+A **Dual Path Network (DPN)** is a convolutional neural network which presents a new topology of connection paths internally. The intuition is that [ResNets](https://paperswithcode.com/method/resnet) enables feature re-usage while DenseNet enables new feature exploration, and both are important for learning good representations. To enjoy the benefits from both path topologies, Dual Path Networks share common features while maintaining the flexibility to explore new features through dual path architectures. 
+
+The principal building block is an [DPN Block](https://paperswithcode.com/method/dpn-block).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{chen2017dual,
+      title={Dual Path Networks}, 
+      author={Yunpeng Chen and Jianan Li and Huaxin Xiao and Xiaojie Jin and Shuicheng Yan and Jiashi Feng},
+      year={2017},
+      eprint={1707.01629},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: DPN
+  Paper:
+    Title: Dual Path Networks
+    URL: https://paperswithcode.com/paper/dual-path-networks
+Models:
+- Name: dpn107
+  In Collection: DPN
+  Metadata:
+    FLOPs: 23524280296
+    Parameters: 86920000
+    File Size: 348612331
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - DPN Block
+    - Dense Connections
+    - Global Average Pooling
+    - Max Pooling
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 40x K80 GPUs
+    ID: dpn107
+    LR: 0.316
+    Layers: 107
+    Crop Pct: '0.875'
+    Batch Size: 1280
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dpn.py#L310
+  Weights: https://github.com/rwightman/pytorch-dpn-pretrained/releases/download/v0.1/dpn107_extra-1ac7121e2.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.16%
+      Top 5 Accuracy: 94.91%
+- Name: dpn131
+  In Collection: DPN
+  Metadata:
+    FLOPs: 20586274792
+    Parameters: 79250000
+    File Size: 318016207
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - DPN Block
+    - Dense Connections
+    - Global Average Pooling
+    - Max Pooling
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 40x K80 GPUs
+    ID: dpn131
+    LR: 0.316
+    Layers: 131
+    Crop Pct: '0.875'
+    Batch Size: 960
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dpn.py#L302
+  Weights: https://github.com/rwightman/pytorch-dpn-pretrained/releases/download/v0.1/dpn131-71dfe43e0.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.83%
+      Top 5 Accuracy: 94.71%
+- Name: dpn68
+  In Collection: DPN
+  Metadata:
+    FLOPs: 2990567880
+    Parameters: 12610000
+    File Size: 50761994
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - DPN Block
+    - Dense Connections
+    - Global Average Pooling
+    - Max Pooling
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 40x K80 GPUs
+    ID: dpn68
+    LR: 0.316
+    Layers: 68
+    Crop Pct: '0.875'
+    Batch Size: 1280
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dpn.py#L270
+  Weights: https://github.com/rwightman/pytorch-dpn-pretrained/releases/download/v0.1/dpn68-66bebafa7.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 76.31%
+      Top 5 Accuracy: 92.97%
+- Name: dpn68b
+  In Collection: DPN
+  Metadata:
+    FLOPs: 2990567880
+    Parameters: 12610000
+    File Size: 50781025
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - DPN Block
+    - Dense Connections
+    - Global Average Pooling
+    - Max Pooling
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 40x K80 GPUs
+    ID: dpn68b
+    LR: 0.316
+    Layers: 68
+    Crop Pct: '0.875'
+    Batch Size: 1280
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dpn.py#L278
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/dpn68b_ra-a31ca160.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.21%
+      Top 5 Accuracy: 94.42%
+- Name: dpn92
+  In Collection: DPN
+  Metadata:
+    FLOPs: 8357659624
+    Parameters: 37670000
+    File Size: 151248422
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - DPN Block
+    - Dense Connections
+    - Global Average Pooling
+    - Max Pooling
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 40x K80 GPUs
+    ID: dpn92
+    LR: 0.316
+    Layers: 92
+    Crop Pct: '0.875'
+    Batch Size: 1280
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dpn.py#L286
+  Weights: https://github.com/rwightman/pytorch-dpn-pretrained/releases/download/v0.1/dpn92_extra-b040e4a9b.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.99%
+      Top 5 Accuracy: 94.84%
+- Name: dpn98
+  In Collection: DPN
+  Metadata:
+    FLOPs: 15003675112
+    Parameters: 61570000
+    File Size: 247021307
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - DPN Block
+    - Dense Connections
+    - Global Average Pooling
+    - Max Pooling
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 40x K80 GPUs
+    ID: dpn98
+    LR: 0.4
+    Layers: 98
+    Crop Pct: '0.875'
+    Batch Size: 1280
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/dpn.py#L294
+  Weights: https://github.com/rwightman/pytorch-dpn-pretrained/releases/download/v0.1/dpn98-5b90dec4d.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.65%
+      Top 5 Accuracy: 94.61%
+-->

docs/models/.templates/models/ecaresnet.md

@@ -0,0 +1,236 @@
+# ECA-ResNet
+
+An **ECA ResNet** is a variant on a [ResNet](https://paperswithcode.com/method/resnet) that utilises an [Efficient Channel Attention module](https://paperswithcode.com/method/efficient-channel-attention). Efficient Channel Attention is an architectural unit based on [squeeze-and-excitation blocks](https://paperswithcode.com/method/squeeze-and-excitation-block) that reduces model complexity without dimensionality reduction. 
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{wang2020ecanet,
+      title={ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks}, 
+      author={Qilong Wang and Banggu Wu and Pengfei Zhu and Peihua Li and Wangmeng Zuo and Qinghua Hu},
+      year={2020},
+      eprint={1910.03151},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: ECAResNet
+  Paper:
+    Title: 'ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks'
+    URL: https://paperswithcode.com/paper/eca-net-efficient-channel-attention-for-deep
+Models:
+- Name: ecaresnet101d
+  In Collection: ECAResNet
+  Metadata:
+    FLOPs: 10377193728
+    Parameters: 44570000
+    File Size: 178815067
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Efficient Channel Attention
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    - Squeeze-and-Excitation Block
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 4x RTX 2080Ti GPUs
+    ID: ecaresnet101d
+    LR: 0.1
+    Epochs: 100
+    Layers: 101
+    Crop Pct: '0.875'
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/resnet.py#L1087
+  Weights: https://imvl-automl-sh.oss-cn-shanghai.aliyuncs.com/darts/hyperml/hyperml/job_45402/outputs/ECAResNet101D_281c5844.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 82.18%
+      Top 5 Accuracy: 96.06%
+- Name: ecaresnet101d_pruned
+  In Collection: ECAResNet
+  Metadata:
+    FLOPs: 4463972081
+    Parameters: 24880000
+    File Size: 99852736
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Efficient Channel Attention
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    - Squeeze-and-Excitation Block
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: ecaresnet101d_pruned
+    Layers: 101
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/resnet.py#L1097
+  Weights: https://imvl-automl-sh.oss-cn-shanghai.aliyuncs.com/darts/hyperml/hyperml/job_45610/outputs/ECAResNet101D_P_75a3370e.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.82%
+      Top 5 Accuracy: 95.64%
+- Name: ecaresnet50d
+  In Collection: ECAResNet
+  Metadata:
+    FLOPs: 5591090432
+    Parameters: 25580000
+    File Size: 102579290
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Efficient Channel Attention
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    - Squeeze-and-Excitation Block
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 4x RTX 2080Ti GPUs
+    ID: ecaresnet50d
+    LR: 0.1
+    Epochs: 100
+    Layers: 50
+    Crop Pct: '0.875'
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0001
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/resnet.py#L1045
+  Weights: https://imvl-automl-sh.oss-cn-shanghai.aliyuncs.com/darts/hyperml/hyperml/job_45402/outputs/ECAResNet50D_833caf58.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.61%
+      Top 5 Accuracy: 95.31%
+- Name: ecaresnet50d_pruned
+  In Collection: ECAResNet
+  Metadata:
+    FLOPs: 3250730657
+    Parameters: 19940000
+    File Size: 79990436
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Efficient Channel Attention
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    - Squeeze-and-Excitation Block
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: ecaresnet50d_pruned
+    Layers: 50
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/resnet.py#L1055
+  Weights: https://imvl-automl-sh.oss-cn-shanghai.aliyuncs.com/darts/hyperml/hyperml/job_45899/outputs/ECAResNet50D_P_9c67f710.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.71%
+      Top 5 Accuracy: 94.88%
+- Name: ecaresnetlight
+  In Collection: ECAResNet
+  Metadata:
+    FLOPs: 5276118784
+    Parameters: 30160000
+    File Size: 120956612
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Efficient Channel Attention
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    - Squeeze-and-Excitation Block
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    ID: ecaresnetlight
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/resnet.py#L1077
+  Weights: https://imvl-automl-sh.oss-cn-shanghai.aliyuncs.com/darts/hyperml/hyperml/job_45402/outputs/ECAResNetLight_4f34b35b.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.46%
+      Top 5 Accuracy: 95.25%
+-->

docs/models/.templates/models/efficientnet-pruned.md

@@ -0,0 +1,145 @@
+# EfficientNet (Knapsack Pruned)
+
+**EfficientNet** is a convolutional neural network architecture and scaling method that uniformly scales all dimensions of depth/width/resolution using a *compound coefficient*. Unlike conventional practice that arbitrary scales  these factors, the EfficientNet scaling method uniformly scales network width, depth, and resolution with a set of fixed scaling coefficients. For example, if we want to use $2^N$ times more computational resources, then we can simply increase the network depth by $\alpha ^ N$,  width by $\beta ^ N$, and image size by $\gamma ^ N$, where $\alpha, \beta, \gamma$ are constant coefficients determined by a small grid search on the original small model. EfficientNet uses a compound coefficient $\phi$ to uniformly scales network width, depth, and resolution in a  principled way.
+
+The compound scaling method is justified by the intuition that if the input image is bigger, then the network needs more layers to increase the receptive field and more channels to capture more fine-grained patterns on the bigger image.
+
+The base EfficientNet-B0 network is based on the inverted bottleneck residual blocks of [MobileNetV2](https://paperswithcode.com/method/mobilenetv2), in addition to [squeeze-and-excitation blocks](https://paperswithcode.com/method/squeeze-and-excitation-block).
+
+This collection consists of pruned EfficientNet models.
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{tan2020efficientnet,
+      title={EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks}, 
+      author={Mingxing Tan and Quoc V. Le},
+      year={2020},
+      eprint={1905.11946},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG}
+}
+```
+
+```
+@misc{aflalo2020knapsack,
+      title={Knapsack Pruning with Inner Distillation},
+      author={Yonathan Aflalo and Asaf Noy and Ming Lin and Itamar Friedman and Lihi Zelnik},
+      year={2020},
+      eprint={2002.08258},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: EfficientNet Pruned
+  Paper:
+    Title: Knapsack Pruning with Inner Distillation
+    URL: https://paperswithcode.com/paper/knapsack-pruning-with-inner-distillation
+Models:
+- Name: efficientnet_b1_pruned
+  In Collection: EfficientNet Pruned
+  Metadata:
+    FLOPs: 489653114
+    Parameters: 6330000
+    File Size: 25595162
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_b1_pruned
+    Crop Pct: '0.882'
+    Image Size: '240'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1208
+  Weights: https://imvl-automl-sh.oss-cn-shanghai.aliyuncs.com/darts/hyperml/hyperml/job_45403/outputs/effnetb1_pruned_9ebb3fe6.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.25%
+      Top 5 Accuracy: 93.84%
+- Name: efficientnet_b2_pruned
+  In Collection: EfficientNet Pruned
+  Metadata:
+    FLOPs: 878133915
+    Parameters: 8310000
+    File Size: 33555005
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_b2_pruned
+    Crop Pct: '0.89'
+    Image Size: '260'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1219
+  Weights: https://imvl-automl-sh.oss-cn-shanghai.aliyuncs.com/darts/hyperml/hyperml/job_45403/outputs/effnetb2_pruned_203f55bc.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.91%
+      Top 5 Accuracy: 94.86%
+- Name: efficientnet_b3_pruned
+  In Collection: EfficientNet Pruned
+  Metadata:
+    FLOPs: 1239590641
+    Parameters: 9860000
+    File Size: 39770812
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_b3_pruned
+    Crop Pct: '0.904'
+    Image Size: '300'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1230
+  Weights: https://imvl-automl-sh.oss-cn-shanghai.aliyuncs.com/darts/hyperml/hyperml/job_45403/outputs/effnetb3_pruned_5abcc29f.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.86%
+      Top 5 Accuracy: 95.24%
+-->

docs/models/.templates/models/efficientnet.md

@@ -0,0 +1,325 @@
+# EfficientNet
+
+**EfficientNet** is a convolutional neural network architecture and scaling method that uniformly scales all dimensions of depth/width/resolution using a *compound coefficient*. Unlike conventional practice that arbitrary scales  these factors, the EfficientNet scaling method uniformly scales network width, depth, and resolution with a set of fixed scaling coefficients. For example, if we want to use $2^N$ times more computational resources, then we can simply increase the network depth by $\alpha ^ N$,  width by $\beta ^ N$, and image size by $\gamma ^ N$, where $\alpha, \beta, \gamma$ are constant coefficients determined by a small grid search on the original small model. EfficientNet uses a compound coefficient $\phi$ to uniformly scales network width, depth, and resolution in a  principled way.
+
+The compound scaling method is justified by the intuition that if the input image is bigger, then the network needs more layers to increase the receptive field and more channels to capture more fine-grained patterns on the bigger image.
+
+The base EfficientNet-B0 network is based on the inverted bottleneck residual blocks of [MobileNetV2](https://paperswithcode.com/method/mobilenetv2), in addition to [squeeze-and-excitation blocks](https://paperswithcode.com/method/squeeze-and-excitation-block).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{tan2020efficientnet,
+      title={EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks}, 
+      author={Mingxing Tan and Quoc V. Le},
+      year={2020},
+      eprint={1905.11946},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: EfficientNet
+  Paper:
+    Title: 'EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks'
+    URL: https://paperswithcode.com/paper/efficientnet-rethinking-model-scaling-for
+Models:
+- Name: efficientnet_b0
+  In Collection: EfficientNet
+  Metadata:
+    FLOPs: 511241564
+    Parameters: 5290000
+    File Size: 21376743
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_b0
+    Layers: 18
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1002
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/efficientnet_b0_ra-3dd342df.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 77.71%
+      Top 5 Accuracy: 93.52%
+- Name: efficientnet_b1
+  In Collection: EfficientNet
+  Metadata:
+    FLOPs: 909691920
+    Parameters: 7790000
+    File Size: 31502706
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_b1
+    Crop Pct: '0.875'
+    Image Size: '240'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1011
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/efficientnet_b1-533bc792.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.71%
+      Top 5 Accuracy: 94.15%
+- Name: efficientnet_b2
+  In Collection: EfficientNet
+  Metadata:
+    FLOPs: 1265324514
+    Parameters: 9110000
+    File Size: 36788104
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_b2
+    Crop Pct: '0.875'
+    Image Size: '260'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1020
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/efficientnet_b2_ra-bcdf34b7.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.38%
+      Top 5 Accuracy: 95.08%
+- Name: efficientnet_b2a
+  In Collection: EfficientNet
+  Metadata:
+    FLOPs: 1452041554
+    Parameters: 9110000
+    File Size: 49369973
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_b2a
+    Crop Pct: '1.0'
+    Image Size: '288'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1029
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/efficientnet_b3_ra2-cf984f9c.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.61%
+      Top 5 Accuracy: 95.32%
+- Name: efficientnet_b3
+  In Collection: EfficientNet
+  Metadata:
+    FLOPs: 2327905920
+    Parameters: 12230000
+    File Size: 49369973
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_b3
+    Crop Pct: '0.904'
+    Image Size: '300'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1038
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/efficientnet_b3_ra2-cf984f9c.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 82.08%
+      Top 5 Accuracy: 96.03%
+- Name: efficientnet_b3a
+  In Collection: EfficientNet
+  Metadata:
+    FLOPs: 2600628304
+    Parameters: 12230000
+    File Size: 49369973
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_b3a
+    Crop Pct: '1.0'
+    Image Size: '320'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1047
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/efficientnet_b3_ra2-cf984f9c.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 82.25%
+      Top 5 Accuracy: 96.11%
+- Name: efficientnet_em
+  In Collection: EfficientNet
+  Metadata:
+    FLOPs: 3935516480
+    Parameters: 6900000
+    File Size: 27927309
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_em
+    Crop Pct: '0.882'
+    Image Size: '240'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1118
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/efficientnet_em_ra2-66250f76.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.26%
+      Top 5 Accuracy: 94.79%
+- Name: efficientnet_es
+  In Collection: EfficientNet
+  Metadata:
+    FLOPs: 2317181824
+    Parameters: 5440000
+    File Size: 22003339
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_es
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1110
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/efficientnet_es_ra-f111e99c.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.09%
+      Top 5 Accuracy: 93.93%
+- Name: efficientnet_lite0
+  In Collection: EfficientNet
+  Metadata:
+    FLOPs: 510605024
+    Parameters: 4650000
+    File Size: 18820005
+    Architecture:
+    - 1x1 Convolution
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inverted Residual Block
+    - Squeeze-and-Excitation Block
+    - Swish
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: efficientnet_lite0
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/a7f95818e44b281137503bcf4b3e3e94d8ffa52f/timm/models/efficientnet.py#L1163
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/efficientnet_lite0_ra-37913777.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 75.5%
+      Top 5 Accuracy: 92.51%
+-->

docs/models/.templates/models/ensemble-adversarial.md

@@ -0,0 +1,98 @@
+# # Ensemble Adversarial Inception ResNet v2
+
+**Inception-ResNet-v2** is a convolutional neural architecture that builds on the Inception family of architectures but incorporates [residual connections](https://paperswithcode.com/method/residual-connection) (replacing the filter concatenation stage of the Inception architecture).
+
+This particular model was trained for study of adversarial examples (adversarial training).
+
+The weights from this model were ported from [Tensorflow/Models](https://github.com/tensorflow/models).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@article{DBLP:journals/corr/abs-1804-00097,
+  author    = {Alexey Kurakin and
+               Ian J. Goodfellow and
+               Samy Bengio and
+               Yinpeng Dong and
+               Fangzhou Liao and
+               Ming Liang and
+               Tianyu Pang and
+               Jun Zhu and
+               Xiaolin Hu and
+               Cihang Xie and
+               Jianyu Wang and
+               Zhishuai Zhang and
+               Zhou Ren and
+               Alan L. Yuille and
+               Sangxia Huang and
+               Yao Zhao and
+               Yuzhe Zhao and
+               Zhonglin Han and
+               Junjiajia Long and
+               Yerkebulan Berdibekov and
+               Takuya Akiba and
+               Seiya Tokui and
+               Motoki Abe},
+  title     = {Adversarial Attacks and Defences Competition},
+  journal   = {CoRR},
+  volume    = {abs/1804.00097},
+  year      = {2018},
+  url       = {http://arxiv.org/abs/1804.00097},
+  archivePrefix = {arXiv},
+  eprint    = {1804.00097},
+  timestamp = {Thu, 31 Oct 2019 16:31:22 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1804-00097.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: Ensemble Adversarial
+  Paper:
+    Title: Adversarial Attacks and Defences Competition
+    URL: https://paperswithcode.com/paper/adversarial-attacks-and-defences-competition
+Models:
+- Name: ens_adv_inception_resnet_v2
+  In Collection: Ensemble Adversarial
+  Metadata:
+    FLOPs: 16959133120
+    Parameters: 55850000
+    File Size: 223774238
+    Architecture:
+    - 1x1 Convolution
+    - Auxiliary Classifier
+    - Average Pooling
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inception-v3 Module
+    - Max Pooling
+    - ReLU
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: ens_adv_inception_resnet_v2
+    Crop Pct: '0.897'
+    Image Size: '299'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/inception_resnet_v2.py#L351
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/ens_adv_inception_resnet_v2-2592a550.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 1.0%
+      Top 5 Accuracy: 17.32%
+-->

docs/models/.templates/models/ese-vovnet.md

@@ -0,0 +1,92 @@
+# ESE-VoVNet
+
+**VoVNet** is a convolutional neural network that seeks to make [DenseNet](https://paperswithcode.com/method/densenet) more efficient by concatenating all features only once in the last feature map, which makes input size constant and enables enlarging new output channel. 
+
+Read about [one-shot aggregation here](https://paperswithcode.com/method/one-shot-aggregation).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{lee2019energy,
+      title={An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection}, 
+      author={Youngwan Lee and Joong-won Hwang and Sangrok Lee and Yuseok Bae and Jongyoul Park},
+      year={2019},
+      eprint={1904.09730},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: ESE VovNet
+  Paper:
+    Title: 'CenterMask : Real-Time Anchor-Free Instance Segmentation'
+    URL: https://paperswithcode.com/paper/centermask-real-time-anchor-free-instance-1
+Models:
+- Name: ese_vovnet19b_dw
+  In Collection: ESE VovNet
+  Metadata:
+    FLOPs: 1711959904
+    Parameters: 6540000
+    File Size: 26243175
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - Max Pooling
+    - One-Shot Aggregation
+    - ReLU
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: ese_vovnet19b_dw
+    Layers: 19
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/vovnet.py#L361
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/ese_vovnet19b_dw-a8741004.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 76.82%
+      Top 5 Accuracy: 93.28%
+- Name: ese_vovnet39b
+  In Collection: ESE VovNet
+  Metadata:
+    FLOPs: 9089259008
+    Parameters: 24570000
+    File Size: 98397138
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - Max Pooling
+    - One-Shot Aggregation
+    - ReLU
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: ese_vovnet39b
+    Layers: 39
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/vovnet.py#L371
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/ese_vovnet39b-f912fe73.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.31%
+      Top 5 Accuracy: 94.72%
+-->

docs/models/.templates/models/fbnet.md

@@ -0,0 +1,76 @@
+# FBNet
+
+**FBNet** is a type of convolutional neural architectures discovered through [DNAS](https://paperswithcode.com/method/dnas) neural architecture search. It utilises a basic type of image model block inspired by [MobileNetv2](https://paperswithcode.com/method/mobilenetv2) that utilises depthwise convolutions and an inverted residual structure (see components).
+
+The principal building block is the [FBNet Block](https://paperswithcode.com/method/fbnet-block).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{wu2019fbnet,
+      title={FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search}, 
+      author={Bichen Wu and Xiaoliang Dai and Peizhao Zhang and Yanghan Wang and Fei Sun and Yiming Wu and Yuandong Tian and Peter Vajda and Yangqing Jia and Kurt Keutzer},
+      year={2019},
+      eprint={1812.03443},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: FBNet
+  Paper:
+    Title: 'FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural
+      Architecture Search'
+    URL: https://paperswithcode.com/paper/fbnet-hardware-aware-efficient-convnet-design
+Models:
+- Name: fbnetc_100
+  In Collection: FBNet
+  Metadata:
+    FLOPs: 508940064
+    Parameters: 5570000
+    File Size: 22525094
+    Architecture:
+    - 1x1 Convolution
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - FBNet Block
+    - Global Average Pooling
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - SGD with Momentum
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 8x GPUs
+    ID: fbnetc_100
+    LR: 0.1
+    Epochs: 360
+    Layers: 22
+    Dropout: 0.2
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.0005
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/efficientnet.py#L985
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/fbnetc_100-c345b898.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 75.12%
+      Top 5 Accuracy: 92.37%
+-->

docs/models/.templates/models/gloun-inception-v3.md

@@ -0,0 +1,78 @@
+# (Gluon) Inception v3
+
+**Inception v3** is a convolutional neural network architecture from the Inception family that makes several improvements including using [Label Smoothing](https://paperswithcode.com/method/label-smoothing), Factorized 7 x 7 convolutions, and the use of an [auxiliary classifer](https://paperswithcode.com/method/auxiliary-classifier) to propagate label information lower down the network (along with the use of batch normalization for layers in the sidehead). The key building block is an [Inception Module](https://paperswithcode.com/method/inception-v3-module).
+
+The weights from this model were ported from [Gluon](https://cv.gluon.ai/model_zoo/classification.html).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@article{DBLP:journals/corr/SzegedyVISW15,
+  author    = {Christian Szegedy and
+               Vincent Vanhoucke and
+               Sergey Ioffe and
+               Jonathon Shlens and
+               Zbigniew Wojna},
+  title     = {Rethinking the Inception Architecture for Computer Vision},
+  journal   = {CoRR},
+  volume    = {abs/1512.00567},
+  year      = {2015},
+  url       = {http://arxiv.org/abs/1512.00567},
+  archivePrefix = {arXiv},
+  eprint    = {1512.00567},
+  timestamp = {Mon, 13 Aug 2018 16:49:07 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/SzegedyVISW15.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: Gloun Inception v3
+  Paper:
+    Title: Rethinking the Inception Architecture for Computer Vision
+    URL: https://paperswithcode.com/paper/rethinking-the-inception-architecture-for
+Models:
+- Name: gluon_inception_v3
+  In Collection: Gloun Inception v3
+  Metadata:
+    FLOPs: 7352418880
+    Parameters: 23830000
+    File Size: 95567055
+    Architecture:
+    - 1x1 Convolution
+    - Auxiliary Classifier
+    - Average Pooling
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inception-v3 Module
+    - Max Pooling
+    - ReLU
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_inception_v3
+    Crop Pct: '0.875'
+    Image Size: '299'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/inception_v3.py#L464
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/gluon_inception_v3-9f746940.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.8%
+      Top 5 Accuracy: 94.38%
+-->

docs/models/.templates/models/gloun-resnet.md

@@ -0,0 +1,504 @@
+# (Gluon) ResNet
+
+**Residual Networks**, or **ResNets**, learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. Instead of hoping each few stacked layers directly fit a desired underlying mapping, residual nets let these layers fit a residual mapping. They stack [residual blocks](https://paperswithcode.com/method/residual-block) ontop of each other to form network: e.g. a ResNet-50 has fifty layers using these blocks. 
+
+The weights from this model were ported from [Gluon](https://cv.gluon.ai/model_zoo/classification.html).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@article{DBLP:journals/corr/HeZRS15,
+  author    = {Kaiming He and
+               Xiangyu Zhang and
+               Shaoqing Ren and
+               Jian Sun},
+  title     = {Deep Residual Learning for Image Recognition},
+  journal   = {CoRR},
+  volume    = {abs/1512.03385},
+  year      = {2015},
+  url       = {http://arxiv.org/abs/1512.03385},
+  archivePrefix = {arXiv},
+  eprint    = {1512.03385},
+  timestamp = {Wed, 17 Apr 2019 17:23:45 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/HeZRS15.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: Gloun ResNet
+  Paper:
+    Title: Deep Residual Learning for Image Recognition
+    URL: https://paperswithcode.com/paper/deep-residual-learning-for-image-recognition
+Models:
+- Name: gluon_resnet101_v1b
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 10068547584
+    Parameters: 44550000
+    File Size: 178723172
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet101_v1b
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L89
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet101_v1b-3b017079.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.3%
+      Top 5 Accuracy: 94.53%
+- Name: gluon_resnet101_v1c
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 10376567296
+    Parameters: 44570000
+    File Size: 178802575
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet101_v1c
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L113
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet101_v1c-1f26822a.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.53%
+      Top 5 Accuracy: 94.59%
+- Name: gluon_resnet101_v1d
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 10377018880
+    Parameters: 44570000
+    File Size: 178802755
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet101_v1d
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L138
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet101_v1d-0f9c8644.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.4%
+      Top 5 Accuracy: 95.02%
+- Name: gluon_resnet101_v1s
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 11805511680
+    Parameters: 44670000
+    File Size: 179221777
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet101_v1s
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L166
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet101_v1s-60fe0cc1.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.29%
+      Top 5 Accuracy: 95.16%
+- Name: gluon_resnet152_v1b
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 14857660416
+    Parameters: 60190000
+    File Size: 241534001
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet152_v1b
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L97
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet152_v1b-c1edb0dd.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.69%
+      Top 5 Accuracy: 94.73%
+- Name: gluon_resnet152_v1c
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 15165680128
+    Parameters: 60210000
+    File Size: 241613404
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet152_v1c
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L121
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet152_v1c-a3bb0b98.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.91%
+      Top 5 Accuracy: 94.85%
+- Name: gluon_resnet152_v1d
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 15166131712
+    Parameters: 60210000
+    File Size: 241613584
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet152_v1d
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L147
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet152_v1d-bd354e12.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.48%
+      Top 5 Accuracy: 95.2%
+- Name: gluon_resnet152_v1s
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 16594624512
+    Parameters: 60320000
+    File Size: 242032606
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet152_v1s
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L175
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet152_v1s-dcc41b81.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 81.02%
+      Top 5 Accuracy: 95.42%
+- Name: gluon_resnet18_v1b
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 2337073152
+    Parameters: 11690000
+    File Size: 46816736
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet18_v1b
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L65
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet18_v1b-0757602b.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 70.84%
+      Top 5 Accuracy: 89.76%
+- Name: gluon_resnet34_v1b
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 4718469120
+    Parameters: 21800000
+    File Size: 87295112
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet34_v1b
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L73
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet34_v1b-c6d82d59.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 74.59%
+      Top 5 Accuracy: 92.0%
+- Name: gluon_resnet50_v1b
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 5282531328
+    Parameters: 25560000
+    File Size: 102493763
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet50_v1b
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L81
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet50_v1b-0ebe02e2.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 77.58%
+      Top 5 Accuracy: 93.72%
+- Name: gluon_resnet50_v1c
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 5590551040
+    Parameters: 25580000
+    File Size: 102573166
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet50_v1c
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L105
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet50_v1c-48092f55.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.01%
+      Top 5 Accuracy: 93.99%
+- Name: gluon_resnet50_v1d
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 5591002624
+    Parameters: 25580000
+    File Size: 102573346
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet50_v1d
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L129
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet50_v1d-818a1b1b.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.06%
+      Top 5 Accuracy: 94.46%
+- Name: gluon_resnet50_v1s
+  In Collection: Gloun ResNet
+  Metadata:
+    FLOPs: 7019495424
+    Parameters: 25680000
+    File Size: 102992368
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Bottleneck Residual Block
+    - Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnet50_v1s
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L156
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnet50_v1s-1762acc0.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.7%
+      Top 5 Accuracy: 94.25%
+-->

docs/models/.templates/models/gloun-resnext.md

@@ -0,0 +1,142 @@
+# (Gluon) ResNeXt
+
+A **ResNeXt** repeats a [building block](https://paperswithcode.com/method/resnext-block) that aggregates a set of transformations with the same topology. Compared to a [ResNet](https://paperswithcode.com/method/resnet), it exposes a new dimension,  *cardinality* (the size of the set of transformations) $C$, as an essential factor in addition to the dimensions of depth and width. 
+
+The weights from this model were ported from [Gluon](https://cv.gluon.ai/model_zoo/classification.html).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@article{DBLP:journals/corr/XieGDTH16,
+  author    = {Saining Xie and
+               Ross B. Girshick and
+               Piotr Doll{\'{a}}r and
+               Zhuowen Tu and
+               Kaiming He},
+  title     = {Aggregated Residual Transformations for Deep Neural Networks},
+  journal   = {CoRR},
+  volume    = {abs/1611.05431},
+  year      = {2016},
+  url       = {http://arxiv.org/abs/1611.05431},
+  archivePrefix = {arXiv},
+  eprint    = {1611.05431},
+  timestamp = {Mon, 13 Aug 2018 16:45:58 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/XieGDTH16.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: Gloun ResNeXt
+  Paper:
+    Title: Aggregated Residual Transformations for Deep Neural Networks
+    URL: https://paperswithcode.com/paper/aggregated-residual-transformations-for-deep
+Models:
+- Name: gluon_resnext101_32x4d
+  In Collection: Gloun ResNeXt
+  Metadata:
+    FLOPs: 10298145792
+    Parameters: 44180000
+    File Size: 177367414
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Grouped Convolution
+    - Max Pooling
+    - ReLU
+    - ResNeXt Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnext101_32x4d
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L193
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnext101_32x4d-b253c8c4.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.33%
+      Top 5 Accuracy: 94.91%
+- Name: gluon_resnext101_64x4d
+  In Collection: Gloun ResNeXt
+  Metadata:
+    FLOPs: 19954172928
+    Parameters: 83460000
+    File Size: 334737852
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Grouped Convolution
+    - Max Pooling
+    - ReLU
+    - ResNeXt Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnext101_64x4d
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L201
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnext101_64x4d-f9a8e184.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.63%
+      Top 5 Accuracy: 95.0%
+- Name: gluon_resnext50_32x4d
+  In Collection: Gloun ResNeXt
+  Metadata:
+    FLOPs: 5472648192
+    Parameters: 25030000
+    File Size: 100441719
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Grouped Convolution
+    - Max Pooling
+    - ReLU
+    - ResNeXt Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_resnext50_32x4d
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L185
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_resnext50_32x4d-e6a097c1.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.35%
+      Top 5 Accuracy: 94.42%
+-->

docs/models/.templates/models/gloun-senet.md

@@ -0,0 +1,63 @@
+# (Gluon) SENet
+
+A **SENet** is a convolutional neural network architecture that employs [squeeze-and-excitation blocks](https://paperswithcode.com/method/squeeze-and-excitation-block) to enable the network to perform dynamic channel-wise feature recalibration.
+
+The weights from this model were ported from [Gluon](https://cv.gluon.ai/model_zoo/classification.html).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{hu2019squeezeandexcitation,
+      title={Squeeze-and-Excitation Networks}, 
+      author={Jie Hu and Li Shen and Samuel Albanie and Gang Sun and Enhua Wu},
+      year={2019},
+      eprint={1709.01507},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: Gloun SENet
+  Paper:
+    Title: Squeeze-and-Excitation Networks
+    URL: https://paperswithcode.com/paper/squeeze-and-excitation-networks
+Models:
+- Name: gluon_senet154
+  In Collection: Gloun SENet
+  Metadata:
+    FLOPs: 26681705136
+    Parameters: 115090000
+    File Size: 461546622
+    Architecture:
+    - Convolution
+    - Dense Connections
+    - Global Average Pooling
+    - Max Pooling
+    - Softmax
+    - Squeeze-and-Excitation Block
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_senet154
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L239
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_senet154-70a1a3c0.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 81.23%
+      Top 5 Accuracy: 95.35%
+-->

docs/models/.templates/models/gloun-seresnext.md

@@ -0,0 +1,136 @@
+# (Gluon) SE-ResNeXt
+
+**SE ResNeXt** is a variant of a [ResNext](https://www.paperswithcode.com/method/resnext) that employs [squeeze-and-excitation blocks](https://paperswithcode.com/method/squeeze-and-excitation-block) to enable the network to perform dynamic channel-wise feature recalibration.
+
+The weights from this model were ported from [Gluon](https://cv.gluon.ai/model_zoo/classification.html).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{hu2019squeezeandexcitation,
+      title={Squeeze-and-Excitation Networks}, 
+      author={Jie Hu and Li Shen and Samuel Albanie and Gang Sun and Enhua Wu},
+      year={2019},
+      eprint={1709.01507},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: Gloun SEResNeXt
+  Paper:
+    Title: Squeeze-and-Excitation Networks
+    URL: https://paperswithcode.com/paper/squeeze-and-excitation-networks
+Models:
+- Name: gluon_seresnext101_32x4d
+  In Collection: Gloun SEResNeXt
+  Metadata:
+    FLOPs: 10302923504
+    Parameters: 48960000
+    File Size: 196505510
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Grouped Convolution
+    - Max Pooling
+    - ReLU
+    - ResNeXt Block
+    - Residual Connection
+    - Softmax
+    - Squeeze-and-Excitation Block
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_seresnext101_32x4d
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L219
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_seresnext101_32x4d-cf52900d.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.87%
+      Top 5 Accuracy: 95.29%
+- Name: gluon_seresnext101_64x4d
+  In Collection: Gloun SEResNeXt
+  Metadata:
+    FLOPs: 19958950640
+    Parameters: 88230000
+    File Size: 353875948
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Grouped Convolution
+    - Max Pooling
+    - ReLU
+    - ResNeXt Block
+    - Residual Connection
+    - Softmax
+    - Squeeze-and-Excitation Block
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_seresnext101_64x4d
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L229
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_seresnext101_64x4d-f9926f93.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 80.88%
+      Top 5 Accuracy: 95.31%
+- Name: gluon_seresnext50_32x4d
+  In Collection: Gloun SEResNeXt
+  Metadata:
+    FLOPs: 5475179184
+    Parameters: 27560000
+    File Size: 110578827
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Grouped Convolution
+    - Max Pooling
+    - ReLU
+    - ResNeXt Block
+    - Residual Connection
+    - Softmax
+    - Squeeze-and-Excitation Block
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_seresnext50_32x4d
+    Crop Pct: '0.875'
+    Image Size: '224'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_resnet.py#L209
+  Weights: https://github.com/rwightman/pytorch-pretrained-gluonresnet/releases/download/v0.1/gluon_seresnext50_32x4d-90cf2d6e.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.92%
+      Top 5 Accuracy: 94.82%
+-->

docs/models/.templates/models/gloun-xception.md

@@ -0,0 +1,66 @@
+# (Gluon) Xception
+
+**Xception** is a convolutional neural network architecture that relies solely on [depthwise separable convolution](https://paperswithcode.com/method/depthwise-separable-convolution) layers.
+
+The weights from this model were ported from [Gluon](https://cv.gluon.ai/model_zoo/classification.html).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{chollet2017xception,
+      title={Xception: Deep Learning with Depthwise Separable Convolutions}, 
+      author={François Chollet},
+      year={2017},
+      eprint={1610.02357},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: Gloun Xception
+  Paper:
+    Title: 'Xception: Deep Learning with Depthwise Separable Convolutions'
+    URL: https://paperswithcode.com/paper/xception-deep-learning-with-depthwise
+Models:
+- Name: gluon_xception65
+  In Collection: Gloun Xception
+  Metadata:
+    FLOPs: 17594889728
+    Parameters: 39920000
+    File Size: 160551306
+    Architecture:
+    - 1x1 Convolution
+    - Convolution
+    - Dense Connections
+    - Depthwise Separable Convolution
+    - Global Average Pooling
+    - Max Pooling
+    - ReLU
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Data:
+    - ImageNet
+    ID: gluon_xception65
+    Crop Pct: '0.903'
+    Image Size: '299'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/gluon_xception.py#L241
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/gluon_xception-7015a15c.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.7%
+      Top 5 Accuracy: 94.87%
+-->

docs/models/.templates/models/hrnet.md

@@ -0,0 +1,358 @@
+# HRNet
+
+**HRNet**, or **High-Resolution Net**, is a general purpose convolutional neural network for tasks like semantic segmentation, object detection and image classification. It is able to maintain high resolution representations through the whole process. We start from a high-resolution convolution stream, gradually add high-to-low resolution convolution streams one by one, and connect the multi-resolution streams in parallel. The resulting network consists of several ($4$ in the paper) stages and the $n$th stage contains $n$ streams corresponding to $n$ resolutions. The authors conduct repeated multi-resolution fusions by exchanging the information across the parallel streams over and over.
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{sun2019highresolution,
+      title={High-Resolution Representations for Labeling Pixels and Regions}, 
+      author={Ke Sun and Yang Zhao and Borui Jiang and Tianheng Cheng and Bin Xiao and Dong Liu and Yadong Mu and Xinggang Wang and Wenyu Liu and Jingdong Wang},
+      year={2019},
+      eprint={1904.04514},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: HRNet
+  Paper:
+    Title: Deep High-Resolution Representation Learning for Visual Recognition
+    URL: https://paperswithcode.com/paper/190807919
+Models:
+- Name: hrnet_w18
+  In Collection: HRNet
+  Metadata:
+    FLOPs: 5547205500
+    Parameters: 21300000
+    File Size: 85718883
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - ReLU
+    - Residual Connection
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 4x NVIDIA V100 GPUs
+    ID: hrnet_w18
+    Epochs: 100
+    Layers: 18
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/hrnet.py#L800
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-hrnet/hrnetv2_w18-8cb57bb9.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 76.76%
+      Top 5 Accuracy: 93.44%
+- Name: hrnet_w18_small
+  In Collection: HRNet
+  Metadata:
+    FLOPs: 2071651488
+    Parameters: 13190000
+    File Size: 52934302
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - ReLU
+    - Residual Connection
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 4x NVIDIA V100 GPUs
+    ID: hrnet_w18_small
+    Epochs: 100
+    Layers: 18
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/hrnet.py#L790
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-hrnet/hrnet_w18_small_v1-f460c6bc.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 72.34%
+      Top 5 Accuracy: 90.68%
+- Name: hrnet_w18_small_v2
+  In Collection: HRNet
+  Metadata:
+    FLOPs: 3360023160
+    Parameters: 15600000
+    File Size: 62682879
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - ReLU
+    - Residual Connection
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 4x NVIDIA V100 GPUs
+    ID: hrnet_w18_small_v2
+    Epochs: 100
+    Layers: 18
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/hrnet.py#L795
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-hrnet/hrnet_w18_small_v2-4c50a8cb.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 75.11%
+      Top 5 Accuracy: 92.41%
+- Name: hrnet_w30
+  In Collection: HRNet
+  Metadata:
+    FLOPs: 10474119492
+    Parameters: 37710000
+    File Size: 151452218
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - ReLU
+    - Residual Connection
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 4x NVIDIA V100 GPUs
+    ID: hrnet_w30
+    Epochs: 100
+    Layers: 30
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/hrnet.py#L805
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-hrnet/hrnetv2_w30-8d7f8dab.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.21%
+      Top 5 Accuracy: 94.22%
+- Name: hrnet_w32
+  In Collection: HRNet
+  Metadata:
+    FLOPs: 11524528320
+    Parameters: 41230000
+    File Size: 165547812
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - ReLU
+    - Residual Connection
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 4x NVIDIA V100 GPUs
+    Training Time: 60 hours
+    ID: hrnet_w32
+    Epochs: 100
+    Layers: 32
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/hrnet.py#L810
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-hrnet/hrnetv2_w32-90d8c5fb.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.45%
+      Top 5 Accuracy: 94.19%
+- Name: hrnet_w40
+  In Collection: HRNet
+  Metadata:
+    FLOPs: 16381182192
+    Parameters: 57560000
+    File Size: 230899236
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - ReLU
+    - Residual Connection
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 4x NVIDIA V100 GPUs
+    ID: hrnet_w40
+    Epochs: 100
+    Layers: 40
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/hrnet.py#L815
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-hrnet/hrnetv2_w40-7cd397a4.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.93%
+      Top 5 Accuracy: 94.48%
+- Name: hrnet_w44
+  In Collection: HRNet
+  Metadata:
+    FLOPs: 19202520264
+    Parameters: 67060000
+    File Size: 268957432
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - ReLU
+    - Residual Connection
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 4x NVIDIA V100 GPUs
+    ID: hrnet_w44
+    Epochs: 100
+    Layers: 44
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/hrnet.py#L820
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-hrnet/hrnetv2_w44-c9ac8c18.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 78.89%
+      Top 5 Accuracy: 94.37%
+- Name: hrnet_w48
+  In Collection: HRNet
+  Metadata:
+    FLOPs: 22285865760
+    Parameters: 77470000
+    File Size: 310603710
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - ReLU
+    - Residual Connection
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 4x NVIDIA V100 GPUs
+    Training Time: 80 hours
+    ID: hrnet_w48
+    Epochs: 100
+    Layers: 48
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/hrnet.py#L825
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-hrnet/hrnetv2_w48-abd2e6ab.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.32%
+      Top 5 Accuracy: 94.51%
+- Name: hrnet_w64
+  In Collection: HRNet
+  Metadata:
+    FLOPs: 37239321984
+    Parameters: 128060000
+    File Size: 513071818
+    Architecture:
+    - Batch Normalization
+    - Convolution
+    - ReLU
+    - Residual Connection
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 4x NVIDIA V100 GPUs
+    ID: hrnet_w64
+    Epochs: 100
+    Layers: 64
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 256
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/hrnet.py#L830
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-hrnet/hrnetv2_w64-b47cc881.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 79.46%
+      Top 5 Accuracy: 94.65%
+-->

docs/models/.templates/models/ig-resnext.md

@@ -0,0 +1,209 @@
+# Instagram ResNeXt WSL
+
+A **ResNeXt** repeats a [building block](https://paperswithcode.com/method/resnext-block) that aggregates a set of transformations with the same topology. Compared to a [ResNet](https://paperswithcode.com/method/resnet), it exposes a new dimension,  *cardinality* (the size of the set of transformations) $C$, as an essential factor in addition to the dimensions of depth and width. 
+
+This model was trained on billions of Instagram images using thousands of distinct hashtags as labels exhibit excellent transfer learning performance. 
+
+Please note the CC-BY-NC 4.0 license on theses weights, non-commercial use only.
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{mahajan2018exploring,
+      title={Exploring the Limits of Weakly Supervised Pretraining}, 
+      author={Dhruv Mahajan and Ross Girshick and Vignesh Ramanathan and Kaiming He and Manohar Paluri and Yixuan Li and Ashwin Bharambe and Laurens van der Maaten},
+      year={2018},
+      eprint={1805.00932},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: IG ResNeXt
+  Paper:
+    Title: Exploring the Limits of Weakly Supervised Pretraining
+    URL: https://paperswithcode.com/paper/exploring-the-limits-of-weakly-supervised
+Models:
+- Name: ig_resnext101_32x16d
+  In Collection: IG ResNeXt
+  Metadata:
+    FLOPs: 46623691776
+    Parameters: 194030000
+    File Size: 777518664
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Grouped Convolution
+    - Max Pooling
+    - ReLU
+    - ResNeXt Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - IG-3.5B-17k
+    - ImageNet
+    Training Resources: 336x GPUs
+    ID: ig_resnext101_32x16d
+    Epochs: 100
+    Layers: 101
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 8064
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/resnet.py#L874
+  Weights: https://download.pytorch.org/models/ig_resnext101_32x16-c6f796b0.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 84.16%
+      Top 5 Accuracy: 97.19%
+- Name: ig_resnext101_32x32d
+  In Collection: IG ResNeXt
+  Metadata:
+    FLOPs: 112225170432
+    Parameters: 468530000
+    File Size: 1876573776
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Grouped Convolution
+    - Max Pooling
+    - ReLU
+    - ResNeXt Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - IG-3.5B-17k
+    - ImageNet
+    Training Resources: 336x GPUs
+    ID: ig_resnext101_32x32d
+    Epochs: 100
+    Layers: 101
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 8064
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+    Minibatch Size: 8064
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/resnet.py#L885
+  Weights: https://download.pytorch.org/models/ig_resnext101_32x32-e4b90b00.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 85.09%
+      Top 5 Accuracy: 97.44%
+- Name: ig_resnext101_32x48d
+  In Collection: IG ResNeXt
+  Metadata:
+    FLOPs: 197446554624
+    Parameters: 828410000
+    File Size: 3317136976
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Grouped Convolution
+    - Max Pooling
+    - ReLU
+    - ResNeXt Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - IG-3.5B-17k
+    - ImageNet
+    Training Resources: 336x GPUs
+    ID: ig_resnext101_32x48d
+    Epochs: 100
+    Layers: 101
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 8064
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/resnet.py#L896
+  Weights: https://download.pytorch.org/models/ig_resnext101_32x48-3e41cc8a.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 85.42%
+      Top 5 Accuracy: 97.58%
+- Name: ig_resnext101_32x8d
+  In Collection: IG ResNeXt
+  Metadata:
+    FLOPs: 21180417024
+    Parameters: 88790000
+    File Size: 356056638
+    Architecture:
+    - 1x1 Convolution
+    - Batch Normalization
+    - Convolution
+    - Global Average Pooling
+    - Grouped Convolution
+    - Max Pooling
+    - ReLU
+    - ResNeXt Block
+    - Residual Connection
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Nesterov Accelerated Gradient
+    - Weight Decay
+    Training Data:
+    - IG-3.5B-17k
+    - ImageNet
+    Training Resources: 336x GPUs
+    ID: ig_resnext101_32x8d
+    Epochs: 100
+    Layers: 101
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Batch Size: 8064
+    Image Size: '224'
+    Weight Decay: 0.001
+    Interpolation: bilinear
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/resnet.py#L863
+  Weights: https://download.pytorch.org/models/ig_resnext101_32x8-c38310e5.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 82.7%
+      Top 5 Accuracy: 96.64%
+-->

docs/models/.templates/models/inception-resnet-v2.md

@@ -0,0 +1,72 @@
+# Inception ResNet v2
+
+**Inception-ResNet-v2** is a convolutional neural architecture that builds on the Inception family of architectures but incorporates [residual connections](https://paperswithcode.com/method/residual-connection) (replacing the filter concatenation stage of the Inception architecture).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@misc{szegedy2016inceptionv4,
+      title={Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning}, 
+      author={Christian Szegedy and Sergey Ioffe and Vincent Vanhoucke and Alex Alemi},
+      year={2016},
+      eprint={1602.07261},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: Inception ResNet v2
+  Paper:
+    Title: Inception-v4, Inception-ResNet and the Impact of Residual Connections on
+      Learning
+    URL: https://paperswithcode.com/paper/inception-v4-inception-resnet-and-the-impact
+Models:
+- Name: inception_resnet_v2
+  In Collection: Inception ResNet v2
+  Metadata:
+    FLOPs: 16959133120
+    Parameters: 55850000
+    File Size: 223774238
+    Architecture:
+    - Average Pooling
+    - Dropout
+    - Inception-ResNet-v2 Reduction-B
+    - Inception-ResNet-v2-A
+    - Inception-ResNet-v2-B
+    - Inception-ResNet-v2-C
+    - Reduction-A
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Label Smoothing
+    - RMSProp
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 20x NVIDIA Kepler GPUs
+    ID: inception_resnet_v2
+    LR: 0.045
+    Dropout: 0.2
+    Crop Pct: '0.897'
+    Momentum: 0.9
+    Image Size: '299'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/inception_resnet_v2.py#L343
+  Weights: https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/inception_resnet_v2-940b1cd6.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 0.95%
+      Top 5 Accuracy: 17.29%
+-->

docs/models/.templates/models/inception-v3.md

@@ -0,0 +1,85 @@
+# Inception v3
+
+**Inception v3** is a convolutional neural network architecture from the Inception family that makes several improvements including using [Label Smoothing](https://paperswithcode.com/method/label-smoothing), Factorized 7 x 7 convolutions, and the use of an [auxiliary classifer](https://paperswithcode.com/method/auxiliary-classifier) to propagate label information lower down the network (along with the use of batch normalization for layers in the sidehead). The key building block is an [Inception Module](https://paperswithcode.com/method/inception-v3-module).
+
+{% include 'code_snippets.md' %}
+
+## How do I train this model?
+
+You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
+
+## Citation
+
+```BibTeX
+@article{DBLP:journals/corr/SzegedyVISW15,
+  author    = {Christian Szegedy and
+               Vincent Vanhoucke and
+               Sergey Ioffe and
+               Jonathon Shlens and
+               Zbigniew Wojna},
+  title     = {Rethinking the Inception Architecture for Computer Vision},
+  journal   = {CoRR},
+  volume    = {abs/1512.00567},
+  year      = {2015},
+  url       = {http://arxiv.org/abs/1512.00567},
+  archivePrefix = {arXiv},
+  eprint    = {1512.00567},
+  timestamp = {Mon, 13 Aug 2018 16:49:07 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/SzegedyVISW15.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<!--
+Type: model-index
+Collections:
+- Name: Inception v3
+  Paper:
+    Title: Rethinking the Inception Architecture for Computer Vision
+    URL: https://paperswithcode.com/paper/rethinking-the-inception-architecture-for
+Models:
+- Name: inception_v3
+  In Collection: Inception v3
+  Metadata:
+    FLOPs: 7352418880
+    Parameters: 23830000
+    File Size: 108857766
+    Architecture:
+    - 1x1 Convolution
+    - Auxiliary Classifier
+    - Average Pooling
+    - Average Pooling
+    - Batch Normalization
+    - Convolution
+    - Dense Connections
+    - Dropout
+    - Inception-v3 Module
+    - Max Pooling
+    - ReLU
+    - Softmax
+    Tasks:
+    - Image Classification
+    Training Techniques:
+    - Gradient Clipping
+    - Label Smoothing
+    - RMSProp
+    - Weight Decay
+    Training Data:
+    - ImageNet
+    Training Resources: 50x NVIDIA Kepler GPUs
+    ID: inception_v3
+    LR: 0.045
+    Dropout: 0.2
+    Crop Pct: '0.875'
+    Momentum: 0.9
+    Image Size: '299'
+    Interpolation: bicubic
+  Code: https://github.com/rwightman/pytorch-image-models/blob/d8e69206be253892b2956341fea09fdebfaae4e3/timm/models/inception_v3.py#L442
+  Weights: https://download.pytorch.org/models/inception_v3_google-1a9a5a14.pth
+  Results:
+  - Task: Image Classification
+    Dataset: ImageNet
+    Metrics:
+      Top 1 Accuracy: 77.46%
+      Top 5 Accuracy: 93.48%
+-->