[SageMaker] Pipeline execution overrides user-defined SageMaker configuration when running locally

[thvasilo]

SageMaker local execution allows users to configure the Docker containers using a local file under $HOME/.sagemaker/config.yaml. See https://aws.amazon.com/blogs/machine-learning/configure-and-use-defaults-for-amazon-sagemaker-resources-with-the-sagemaker-python-sdk/ for details

An example file can be:

local:
    local_code: true # Using everything locally
    region_name: "us-east-1" # Name of the region
    container_config: # Additional docker container config
        shm_size: "58G"
        environment:
          - AWS_REGION: "us-east-1"

when creating a local session this configuration is saved as a dict in a config parameter:

from sagemaker.workflow.pipeline_context import LocalPipelineSession

local_session = LocalPipelineSession()
config_dict = local_session.config

However in execute_pipeline.py we override this config to set the shm size for the container:

graphstorm/sagemaker/pipeline/execute_sm_pipeline.py

Lines 164 to 172 in a145677

	local_session = LocalPipelineSession()
	pipeline_generator = GraphStormPipelineGenerator(
	pipeline_deploy_args, input_session=local_session
	)
	# Set shared memory to half of the host's size, as SM does
	instance_mem_mb = int(psutil.virtual_memory().total // (1024 * 1024))
	local_session.config = {
	"local": {"container_config": {"shm_size": f"{instance_mem_mb//2}M"}}
	}

What we should be doing instead is only update the shm_size if it's not already configured.