README.md

---
language: en
license: apache-2.0
library_name: pytorch
tags:
- deep-reinforcement-learning
- reinforcement-learning
- DI-engine
- TicTacToe-play-with-bot
benchmark_name: OpenAI/Gym/Atari
task_name: TicTacToe-play-with-bot
pipeline_tag: reinforcement-learning
model-index:
- name: AlphaZero
  results:
  - task:
      type: reinforcement-learning
      name: reinforcement-learning
    dataset:
      name: TicTacToe-play-with-bot
      type: TicTacToe-play-with-bot
    metrics:
    - type: mean_reward
      value: 0.5 +/- 0.5
      name: mean_reward
---

# Play **TicTacToe-play-with-bot** with **AlphaZero** Policy

## Model Description
<!-- Provide a longer summary of what this model is. -->

This implementation applies **AlphaZero** to the OpenAI/Gym/Atari **TicTacToe-play-with-bot** environment using [LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine).

**LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).

## Model Usage
### Install the Dependencies
<details close>
<summary>(Click for Details)</summary>

```shell
# install huggingface_ding
git clone https://github.com/opendilab/huggingface_ding.git
pip3 install -e ./huggingface_ding/
# install environment dependencies if needed

pip3 install DI-engine[common_env,video]
pip3 install LightZero

```
</details>

### Git Clone from Huggingface and Run the Model

<details close>
<summary>(Click for Details)</summary>

```shell
# running with trained model
python3 -u run.py
```
**run.py**
```python
from lzero.agent import AlphaZeroAgent
from ding.config import Config
from easydict import EasyDict
import torch

# Pull model from files which are git cloned from huggingface
policy_state_dict = torch.load("pytorch_model.bin", map_location=torch.device("cpu"))
cfg = EasyDict(Config.file_to_dict("policy_config.py").cfg_dict)
# Instantiate the agent
agent = AlphaZeroAgent(
    env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-AlphaZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict
)
# Continue training
agent.train(step=5000)
# Render the new agent performance
agent.deploy(enable_save_replay=True)

```
</details>

### Run Model by Using Huggingface_ding

<details close>
<summary>(Click for Details)</summary>

```shell
# running with trained model
python3 -u run.py
```
**run.py**
```python
from lzero.agent import AlphaZeroAgent
from huggingface_ding import pull_model_from_hub

# Pull model from Hugggingface hub
policy_state_dict, cfg = pull_model_from_hub(repo_id="OpenDILabCommunity/TicTacToe-play-with-bot-AlphaZero")
# Instantiate the agent
agent = AlphaZeroAgent(
    env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-AlphaZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict
)
# Continue training
agent.train(step=5000)
# Render the new agent performance
agent.deploy(enable_save_replay=True)

```
</details>

## Model Training

### Train the Model and Push to Huggingface_hub

<details close>
<summary>(Click for Details)</summary>

```shell
#Training Your Own Agent
python3 -u train.py
```
**train.py**
```python
from lzero.agent import AlphaZeroAgent
from huggingface_ding import push_model_to_hub

# Instantiate the agent
agent = AlphaZeroAgent(env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-AlphaZero")
# Train the agent
return_ = agent.train(step=int(500000))
# Push model to huggingface hub
push_model_to_hub(
    agent=agent.best,
    env_name="OpenAI/Gym/Atari",
    task_name="TicTacToe-play-with-bot",
    algo_name="AlphaZero",
    github_repo_url="https://github.com/opendilab/LightZero",
    github_doc_model_url=None,
    github_doc_env_url=None,
    installation_guide='''
pip3 install DI-engine[common_env,video]
pip3 install LightZero
''',
    usage_file_by_git_clone="./alphazero/tictactoe_play_with_bot_alphazero_deploy.py",
    usage_file_by_huggingface_ding="./alphazero/tictactoe_play_with_bot_alphazero_download.py",
    train_file="./alphazero/tictactoe_play_with_bot_alphazero.py",
    repo_id="OpenDILabCommunity/TicTacToe-play-with-bot-AlphaZero",
    platform_info="[LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine)",
    model_description="**LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).",
    create_repo=True
)

```
</details>

**Configuration**
<details close>
<summary>(Click for Details)</summary>


```python
exp_config = {
    'main_config': {
        'exp_name': 'TicTacToe-play-with-bot-AlphaZero',
        'seed': 0,
        'env': {
            'env_id': 'TicTacToe-play-with-bot',
            'board_size': 3,
            'battle_mode': 'play_with_bot_mode',
            'bot_action_type': 'v0',
            'channel_last': False,
            'collector_env_num': 8,
            'evaluator_env_num': 5,
            'n_evaluator_episode': 5,
            'manager': {
                'shared_memory': False
            },
            'agent_vs_human': False,
            'prob_random_agent': 0,
            'prob_expert_agent': 0,
            'scale': True,
            'alphazero_mcts_ctree': False,
            'save_replay_gif': False,
            'replay_path_gif': './replay_gif'
        },
        'policy': {
            'on_policy': False,
            'cuda': True,
            'multi_gpu': False,
            'bp_update_sync': True,
            'traj_len_inf': False,
            'model': {
                'observation_shape': [3, 3, 3],
                'action_space_size': 9,
                'num_res_blocks': 1,
                'num_channels': 16,
                'fc_value_layers': [8],
                'fc_policy_layers': [8]
            },
            'torch_compile': False,
            'tensor_float_32': False,
            'sampled_algo': False,
            'gumbel_algo': False,
            'update_per_collect': 50,
            'model_update_ratio': 0.1,
            'batch_size': 256,
            'optim_type': 'Adam',
            'learning_rate': 0.003,
            'weight_decay': 0.0001,
            'momentum': 0.9,
            'grad_clip_value': 0.5,
            'value_weight': 1.0,
            'collector_env_num': 8,
            'evaluator_env_num': 5,
            'lr_piecewise_constant_decay': False,
            'threshold_training_steps_for_final_lr': 500000,
            'manual_temperature_decay': False,
            'threshold_training_steps_for_final_temperature': 100000,
            'fixed_temperature_value': 0.25,
            'mcts': {
                'num_simulations': 25
            },
            'other': {
                'replay_buffer': {
                    'replay_buffer_size': 1000000,
                    'save_episode': False
                }
            },
            'cfg_type': 'AlphaZeroPolicyDict',
            'mcts_ctree': False,
            'simulation_env_name': 'tictactoe',
            'simulation_env_config_type': 'play_with_bot',
            'board_size': 3,
            'entropy_weight': 0.0,
            'n_episode': 8,
            'eval_freq': 2000
        },
        'wandb_logger': {
            'gradient_logger': False,
            'video_logger': False,
            'plot_logger': False,
            'action_logger': False,
            'return_logger': False
        }
    },
    'create_config': {
        'env': {
            'type': 'tictactoe',
            'import_names': ['zoo.board_games.tictactoe.envs.tictactoe_env']
        },
        'env_manager': {
            'type': 'subprocess'
        },
        'policy': {
            'type': 'alphazero',
            'import_names': ['lzero.policy.alphazero']
        },
        'collector': {
            'type': 'episode_alphazero',
            'import_names': ['lzero.worker.alphazero_collector']
        },
        'evaluator': {
            'type': 'alphazero',
            'import_names': ['lzero.worker.alphazero_evaluator']
        }
    }
}

```
</details>

**Training Procedure** 
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
- **Weights & Biases (wandb):** [monitor link](<TODO>)

## Model Information
<!-- Provide the basic links for the model. -->
- **Github Repository:** [repo link](https://github.com/opendilab/LightZero)
- **Doc**: [Algorithm link](<TODO>)
- **Configuration:** [config link](https://huggingface.co/OpenDILabCommunity/TicTacToe-play-with-bot-AlphaZero/blob/main/policy_config.py)
- **Demo:** [video](https://huggingface.co/OpenDILabCommunity/TicTacToe-play-with-bot-AlphaZero/blob/main/replay.mp4)
<!-- Provide the size information for the model. -->
- **Parameters total size:** 51.13 KB
- **Last Update Date:** 2023-12-20

## Environments
<!-- Address questions around what environment the model is intended to be trained and deployed at, including the necessary information needed to be provided for future users. -->
- **Benchmark:** OpenAI/Gym/Atari
- **Task:** TicTacToe-play-with-bot
- **Gym version:** 0.25.1
- **DI-engine version:** v0.5.0
- **PyTorch version:** 2.0.1+cu117
- **Doc**: [Environments link](<TODO>)