Description
If there is almost no difference between the tensor of the base model and the tensor of a model, task vector values are close to zero. If all the task vectors are close to zero, then the merge method of the merge method class just returns the base tensor. The current warning message "No task vectors. Returning the base model tensor." is not very helpful:
class TaskArithmetic(MergeMethod):
def merge(
self,
weight: ModelWeight,
base_model_tensor: torch.Tensor,
models_tensors: Dict[Model, torch.Tensor],
merge_method_settings: Union[TaskArithmeticSettings, TiesMergingSettings],
base_model: Model,
) -> torch.Tensor:
base_tensor_dtype = base_model_tensor.dtype
task_vectors: Dict[Model, torch.Tensor] = self._get_task_vectors(
base_model_tensor, models_tensors
)
if not task_vectors:
logger.warning("No task vectors. Returning the base model tensor.")
return base_model_tensor
...
It should provide a better explanation.
Description
If there is almost no difference between the tensor of the base model and the tensor of a model, task vector values are close to zero. If all the task vectors are close to zero, then the merge method of the merge method class just returns the base tensor. The current warning message "No task vectors. Returning the base model tensor." is not very helpful:
It should provide a better explanation.