[2024-10-20 15:27:19,191][matcha.utils.utils][INFO] - Enforcing tags! <cfg.extras.enforce_tags=True>
[2024-10-20 15:27:19,197][matcha.utils.utils][INFO] - Printing config tree with Rich! <cfg.extras.print_config=True>
[2024-10-20 15:27:19,263][__main__][INFO] - Instantiating datamodule <matcha.data.text_mel_datamodule.TextMelDataModule>
[2024-10-20 15:27:19,726][__main__][INFO] - Instantiating model <matcha.models.matcha_tts.MatchaTTS>
[2024-10-20 15:27:20,313][__main__][INFO] - Instantiating callbacks...
[2024-10-20 15:27:20,314][matcha.utils.instantiators][INFO] - Instantiating callback <lightning.pytorch.callbacks.ModelCheckpoint>
[2024-10-20 15:27:20,317][matcha.utils.instantiators][INFO] - Instantiating callback <lightning.pytorch.callbacks.RichModelSummary>
[2024-10-20 15:27:20,318][matcha.utils.instantiators][INFO] - Instantiating callback <lightning.pytorch.callbacks.RichProgressBar>
[2024-10-20 15:27:20,318][__main__][INFO] - Instantiating loggers...
[2024-10-20 15:27:20,319][matcha.utils.instantiators][INFO] - Instantiating logger <lightning.pytorch.loggers.tensorboard.TensorBoardLogger>
[2024-10-20 15:27:20,323][__main__][INFO] - Instantiating trainer <lightning.pytorch.trainer.Trainer>
[2024-10-20 15:27:20,367][__main__][INFO] - Logging hyperparameters!
[2024-10-20 15:27:20,422][__main__][INFO] - Starting training!
[2024-10-22 01:31:06,012][matcha.utils.utils][ERROR] - 
Traceback (most recent call last):
  File "/workspace/Matcha-TTS/matcha/utils/utils.py", line 77, in wrap
    metric_dict, object_dict = task_func(cfg=cfg)
  File "/workspace/Matcha-TTS/matcha/train.py", line 79, in train
    trainer.fit(model=model, datamodule=datamodule, ckpt_path=cfg.get("ckpt_path"))
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 538, in fit
    call._call_and_handle_interrupt(
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 47, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 574, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 981, in _run
    results = self._run_stage()
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1025, in _run_stage
    self.fit_loop.run()
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 206, in run
    self.on_advance_end()
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 378, in on_advance_end
    call._call_callback_hooks(trainer, "on_train_epoch_end", monitoring_callbacks=True)
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 218, in _call_callback_hooks
    fn(trainer, trainer.lightning_module, *args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 326, in on_train_epoch_end
    self._save_last_checkpoint(trainer, monitor_candidates)
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 696, in _save_last_checkpoint
    self._save_checkpoint(trainer, filepath)
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/callbacks/model_checkpoint.py", line 390, in _save_checkpoint
    trainer.save_checkpoint(filepath, self.save_weights_only)
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1365, in save_checkpoint
    self.strategy.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
  File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 490, in save_checkpoint
    self.checkpoint_io.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
  File "/opt/conda/lib/python3.10/site-packages/lightning/fabric/plugins/io/torch_io.py", line 58, in save_checkpoint
    _atomic_save(checkpoint, path)
  File "/opt/conda/lib/python3.10/site-packages/lightning/fabric/utilities/cloud_io.py", line 90, in _atomic_save
    f.write(bytesbuffer.getvalue())
  File "/opt/conda/lib/python3.10/site-packages/fsspec/implementations/local.py", line 373, in write
    return self.f.write(*args, **kwargs)
OSError: [Errno 28] No space left on device
[2024-10-22 01:31:06,020][matcha.utils.utils][INFO] - Output dir: /workspace/Matcha-TTS/logs/train/darkman/runs/2024-10-20_15-27-18