| [2024-10-20 15:27:19,191][matcha.utils.utils][INFO] - Enforcing tags! <cfg.extras.enforce_tags=True> | |
| [2024-10-20 15:27:19,197][matcha.utils.utils][INFO] - Printing config tree with Rich! <cfg.extras.print_config=True> | |
| [2024-10-20 15:27:19,263][__main__][INFO] - Instantiating datamodule <matcha.data.text_mel_datamodule.TextMelDataModule> | |
| [2024-10-20 15:27:19,726][__main__][INFO] - Instantiating model <matcha.models.matcha_tts.MatchaTTS> | |
| [2024-10-20 15:27:20,313][__main__][INFO] - Instantiating callbacks... | |
| [2024-10-20 15:27:20,314][matcha.utils.instantiators][INFO] - Instantiating callback <lightning.pytorch.callbacks.ModelCheckpoint> | |
| [2024-10-20 15:27:20,317][matcha.utils.instantiators][INFO] - Instantiating callback <lightning.pytorch.callbacks.RichModelSummary> | |
| [2024-10-20 15:27:20,318][matcha.utils.instantiators][INFO] - Instantiating callback <lightning.pytorch.callbacks.RichProgressBar> | |
| [2024-10-20 15:27:20,318][__main__][INFO] - Instantiating loggers... | |
| [2024-10-20 15:27:20,319][matcha.utils.instantiators][INFO] - Instantiating logger <lightning.pytorch.loggers.tensorboard.TensorBoardLogger> | |
| [2024-10-20 15:27:20,323][__main__][INFO] - Instantiating trainer <lightning.pytorch.trainer.Trainer> | |
| [2024-10-20 15:27:20,367][__main__][INFO] - Logging hyperparameters! | |
| [2024-10-20 15:27:20,422][__main__][INFO] - Starting training! | |
| [2024-10-22 01:31:06,012][matcha.utils.utils][ERROR] - | |
| Traceback (most recent call last): | |
| File "/workspace/Matcha-TTS/matcha/utils/utils.py", line 77, in wrap | |
| metric_dict, object_dict = task_func(cfg=cfg) | |
| File "/workspace/Matcha-TTS/matcha/train.py", line 79, in train | |
| trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path")) | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 538, in fit | |
| call._call_and_handle_interrupt( | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 47, in _call_and_handle_interrupt | |
| return trainer_fn(*args, **kwargs) | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 574, in _fit_impl | |
| self._run(model, ckpt_path=ckpt_path) | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 981, in _run | |
| results = self._run_stage() | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1025, in _run_stage | |
| self.fit_loop.run() | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 206, in run | |
| self.on_advance_end() | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 378, in on_advance_end | |
| call._call_callback_hooks(trainer, "on_train_epoch_end", monitoring_callbacks=True) | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 218, in _call_callback_hooks | |
| fn(trainer, trainer.lightning_module, *args, **kwargs) | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 326, in on_train_epoch_end | |
| self._save_last_checkpoint(trainer, monitor_candidates) | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 696, in _save_last_checkpoint | |
| self._save_checkpoint(trainer, filepath) | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 390, in _save_checkpoint | |
| trainer.save_checkpoint(filepath, self.save_weights_only) | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1365, in save_checkpoint | |
| self.strategy.save_checkpoint(checkpoint, filepath, storage_options=storage_options) | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 490, in save_checkpoint | |
| self.checkpoint_io.save_checkpoint(checkpoint, filepath, storage_options=storage_options) | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/fabric/plugins/io/torch_io.py", line 58, in save_checkpoint | |
| _atomic_save(checkpoint, path) | |
| File "/opt/conda/lib/python3.10/site-packages/lightning/fabric/utilities/cloud_io.py", line 90, in _atomic_save | |
| f.write(bytesbuffer.getvalue()) | |
| File "/opt/conda/lib/python3.10/site-packages/fsspec/implementations/local.py", line 373, in write | |
| return self.f.write(*args, **kwargs) | |
| OSError: [Errno 28] No space left on device | |
| [2024-10-22 01:31:06,020][matcha.utils.utils][INFO] - Output dir: /workspace/Matcha-TTS/logs/train/darkman/runs/2024-10-20_15-27-18 | |