--- language: en license: apache-2.0 library_name: pytorch tags: - deep-reinforcement-learning - reinforcement-learning - DI-engine - TicTacToe-play-with-bot benchmark_name: OpenAI/Gym/Atari task_name: TicTacToe-play-with-bot pipeline_tag: reinforcement-learning model-index: - name: MuZero results: - task: type: reinforcement-learning name: reinforcement-learning dataset: name: TicTacToe-play-with-bot type: TicTacToe-play-with-bot metrics: - type: mean_reward value: 0.5 +/- 0.81 name: mean_reward --- # Play **TicTacToe-play-with-bot** with **MuZero** Policy ## Model Description This implementation applies **MuZero** to the OpenAI/Gym/Atari **TicTacToe-play-with-bot** environment using [LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine). **LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348). ## Model Usage ### Install the Dependencies
(Click for Details) ```shell # install huggingface_ding git clone https://github.com/opendilab/huggingface_ding.git pip3 install -e ./huggingface_ding/ # install environment dependencies if needed pip3 install DI-engine[common_env,video] pip3 install LightZero ```
### Git Clone from Huggingface and Run the Model
(Click for Details) ```shell # running with trained model python3 -u run.py ``` **run.py** ```python from lzero.agent import MuZeroAgent from ding.config import Config from easydict import EasyDict import torch # Pull model from files which are git cloned from huggingface policy_state_dict = torch.load("pytorch_model.bin", map_location=torch.device("cpu")) cfg = EasyDict(Config.file_to_dict("policy_config.py").cfg_dict) # Instantiate the agent agent = MuZeroAgent( env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-MuZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict ) # Continue training agent.train(step=5000) # Render the new agent performance agent.deploy(enable_save_replay=True) ```
### Run Model by Using Huggingface_ding
(Click for Details) ```shell # running with trained model python3 -u run.py ``` **run.py** ```python from lzero.agent import MuZeroAgent from huggingface_ding import pull_model_from_hub # Pull model from Hugggingface hub policy_state_dict, cfg = pull_model_from_hub(repo_id="OpenDILabCommunity/TicTacToe-play-with-bot-MuZero") # Instantiate the agent agent = MuZeroAgent( env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-MuZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict ) # Continue training agent.train(step=5000) # Render the new agent performance agent.deploy(enable_save_replay=True) ```
## Model Training ### Train the Model and Push to Huggingface_hub
(Click for Details) ```shell #Training Your Own Agent python3 -u train.py ``` **train.py** ```python from lzero.agent import MuZeroAgent from huggingface_ding import push_model_to_hub # Instantiate the agent agent = MuZeroAgent(env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-MuZero") # Train the agent return_ = agent.train(step=int(500000)) # Push model to huggingface hub push_model_to_hub( agent=agent.best, env_name="OpenAI/Gym/Atari", task_name="TicTacToe-play-with-bot", algo_name="MuZero", github_repo_url="https://github.com/opendilab/LightZero", github_doc_model_url=None, github_doc_env_url=None, installation_guide=''' pip3 install DI-engine[common_env,video] pip3 install LightZero ''', usage_file_by_git_clone="./muzero/tictactoe_play_with_bot_muzero_deploy.py", usage_file_by_huggingface_ding="./muzero/tictactoe_play_with_bot_muzero_download.py", train_file="./muzero/tictactoe_play_with_bot_muzero.py", repo_id="OpenDILabCommunity/TicTacToe-play-with-bot-MuZero", platform_info="[LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine)", model_description="**LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).", create_repo=True ) ```
**Configuration**
(Click for Details) ```python exp_config = { 'main_config': { 'exp_name': 'TicTacToe-play-with-bot-MuZero', 'seed': 0, 'env': { 'env_id': 'TicTacToe-play-with-bot', 'battle_mode': 'play_with_bot_mode', 'collector_env_num': 8, 'evaluator_env_num': 5, 'n_evaluator_episode': 5, 'manager': { 'shared_memory': False } }, 'policy': { 'on_policy': False, 'cuda': True, 'multi_gpu': False, 'bp_update_sync': True, 'traj_len_inf': False, 'model': { 'observation_shape': [3, 3, 3], 'action_space_size': 9, 'image_channel': 3, 'num_res_blocks': 1, 'num_channels': 16, 'fc_reward_layers': [8], 'fc_value_layers': [8], 'fc_policy_layers': [8], 'support_scale': 10, 'reward_support_size': 21, 'value_support_size': 21, 'norm_type': 'BN' }, 'use_rnd_model': False, 'sampled_algo': False, 'gumbel_algo': False, 'mcts_ctree': True, 'collector_env_num': 8, 'evaluator_env_num': 5, 'env_type': 'board_games', 'action_type': 'varied_action_space', 'battle_mode': 'play_with_bot_mode', 'monitor_extra_statistics': True, 'game_segment_length': 5, 'transform2string': False, 'gray_scale': False, 'use_augmentation': False, 'augmentation': ['shift', 'intensity'], 'ignore_done': False, 'update_per_collect': 50, 'model_update_ratio': 0.1, 'batch_size': 256, 'optim_type': 'Adam', 'learning_rate': 0.003, 'target_update_freq': 100, 'target_update_freq_for_intrinsic_reward': 1000, 'weight_decay': 0.0001, 'momentum': 0.9, 'grad_clip_value': 0.5, 'n_episode': 8, 'num_simulations': 25, 'discount_factor': 1, 'td_steps': 9, 'num_unroll_steps': 3, 'reward_loss_weight': 1, 'value_loss_weight': 0.25, 'policy_loss_weight': 1, 'policy_entropy_loss_weight': 0, 'ssl_loss_weight': 0, 'lr_piecewise_constant_decay': False, 'threshold_training_steps_for_final_lr': 50000, 'manual_temperature_decay': False, 'threshold_training_steps_for_final_temperature': 100000, 'fixed_temperature_value': 0.25, 'use_ture_chance_label_in_chance_encoder': False, 'use_priority': True, 'priority_prob_alpha': 0.6, 'priority_prob_beta': 0.4, 'root_dirichlet_alpha': 0.3, 'root_noise_weight': 0.25, 'random_collect_episode_num': 0, 'eps': { 'eps_greedy_exploration_in_collect': False, 'type': 'linear', 'start': 1.0, 'end': 0.05, 'decay': 100000 }, 'cfg_type': 'MuZeroPolicyDict', 'reanalyze_ratio': 0.0, 'eval_freq': 2000, 'replay_buffer_size': 10000 }, 'wandb_logger': { 'gradient_logger': False, 'video_logger': False, 'plot_logger': False, 'action_logger': False, 'return_logger': False } }, 'create_config': { 'env': { 'type': 'tictactoe', 'import_names': ['zoo.board_games.tictactoe.envs.tictactoe_env'] }, 'env_manager': { 'type': 'subprocess' }, 'policy': { 'type': 'muzero', 'import_names': ['lzero.policy.muzero'] } } } ```
**Training Procedure** - **Weights & Biases (wandb):** [monitor link]() ## Model Information - **Github Repository:** [repo link](https://github.com/opendilab/LightZero) - **Doc**: [Algorithm link]() - **Configuration:** [config link](https://huggingface.co/OpenDILabCommunity/TicTacToe-play-with-bot-MuZero/blob/main/policy_config.py) - **Demo:** [video](https://huggingface.co/OpenDILabCommunity/TicTacToe-play-with-bot-MuZero/blob/main/replay.mp4) - **Parameters total size:** 91.5 KB - **Last Update Date:** 2023-12-20 ## Environments - **Benchmark:** OpenAI/Gym/Atari - **Task:** TicTacToe-play-with-bot - **Gym version:** 0.25.1 - **DI-engine version:** v0.5.0 - **PyTorch version:** 2.0.1+cu117 - **Doc**: [Environments link]()