Since I am testing on 1000 data points from Task A where the response= xyz, Here is my Task B data point with a response= abc. If I train model1 for Task A and fine-tune model2 on model1, my model2 is not able to respond with 'xyz which is response of model1 for taskA.

If we merge adapters then this work for continuous learning ?
I expect the new train model to provide output for previously trained data as well.

model_path = "model/path"

Reload model in FP16 and merge it with LoRA weights

base_model = AutoModelForCausalLM.from_pretrained(
model_name,
low_cpu_mem_usage=True,
return_dict=True,
torch_dtype=torch.float16,
device_map=device_map,
)
model = PeftModel.from_pretrained(base_model, new_model)
model = model.merge_and_unload()

Reload tokenizer to save it

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

Save the merged model

model.save_pretrained(model_path)
tokenizer.save_pretrained(model_path)