Repository navigation
LightningCLI: Variable interpolation and command line arguments聽#13607
Description
馃悰 Bug
Hi! I'm not sure this is actually a bug or if the one doing something wrong, but here's the problem.
I'm using OmegaConf鈥檚 variable interpolation to link one parameter in the configuration file to a second one, which I want to provide with the command line. The problem is that, when using the command line to provide one of the two parameters, the second one, linked to the first one with variable interpolation, doesn't change.
To Reproduce
config.yaml
foo: default_value
bar: ${foo}main.py
from pytorch_lightning.utilities.cli import LightningCLI
class MyLightningCLI(LightningCLI):
def add_arguments_to_parser(self, parser):
parser.add_argument("--foo", type=str)
parser.add_argument("--bar", type=str)
if __name__ == "__main__":
cli = MyLightningCLI(parser_kwargs={"parser_mode": "omegaconf"})- Run
python main.py fit \
--config=config.yaml \
--foo=some_value \
--print_config- Result
The result of this is that the foo parameter is equal to the value assigned with the command line, some_value, but the bar parameter isn't, and it is instead keeping its default value set in the config file (default_value).
seed_everything: null
trainer:
logger: true
checkpoint_callback: null
...
foo: some_value
bar: default_value
...Expected behavior
I expected the bar parameter to be equal to the foo聽parameter, that is to have the value some_value:
seed_everything: null
trainer:
logger: true
checkpoint_callback: null
...
foo: some_value
bar: some_value
...Environment
- PyTorch Lightning Version (e.g., 1.5.0): 1.6.4
- PyTorch Version (e.g., 1.10): 1.10.2
- Python version (e.g., 3.9): 3.9
- OS (e.g., Linux): Mac
- GPU models and configuration: None (CPU)
- How you installed PyTorch (
conda,pip, source):pip
Activity
lodo1995 commentedon Jul 11, 2022
I agree that this would be a very nice feature to have. Unfortunately, what you are seeing is a limitation of
jsonargparse, the library that Lightning uses to handle both command line arguments and configuration files.Their documentation on variable interpolation explicitly states that (emphasis mine)
mikcnt commentedon Jul 11, 2022
@lodo1995 Thanks a lot for pointing that out! I honestly didn't notice that, that's unfortunate 馃槶 Do you happen to know if there is some workaround for that?
lodo1995 commentedon Jul 11, 2022
@mikcnt I guess there must be a smart way to abuse argument linking to achieve what you want, but I'm not sure how.
Alternatively, you can "brute force" it by overriding
before_instantiate_classes, manually looking for the pattern${*}in every string argument and doing the appropriate lookup and replacement based on other arguments. Watch out for infinite recursion if you try this!But no, I don't know of any simple workaround.
mauvilsa commentedon Jul 12, 2022
@mikcnt with your example using
fooandbarit is not known what was the real use case behind it. Could you please your actual case? There are many cases in which link_arguments should be used, but people go for variable interpolation because it is a better known concept.mikcnt commentedon Jul 12, 2022
@lodo1995 thanks a lot for the tips! I'll try something out, even though I still want to keep things simple.
@mauvilsa Sure, my example isn't really explanatory. My use case is similar to the the first example of #13558. Basically, I want to provide one single parameter through the command line (let's say the name of the experiment), and I want to use that to fill some other parameter as well (e.g., wandb run, directory for the checkpoints, etc).
mauvilsa commentedon Jul 14, 2022
To me it seems that what you want is not interpolation. There is a single parameter, and its value will always be used in several specific places. It is not responsibility of who uses the CLI to do this in a config file. Is your intention that the loggers are always
WandbLoggerandCSVLogger, or you want the amount and which loggers to be configurable?link_argumentsdoes not support linking to a target which is an element of a list, like theloggerwhen there are multiple. But if the loggers are fixed I think it might be already possible to do what you want.mikcnt commentedon Jul 14, 2022
I see what you're saying. The reason why I was thinking about interpolation is that I didn't want to necessarily use the same parameter for every single location, but most definitely a slight modification of this, and that was something easily achievable with interpolation. Nevertheless, I think that would suffice.
Yes, I'll use only WandbLogger for every experiment.
mauvilsa commentedon Jul 15, 2022
There were some cases that
link_argumentsdid not support which are required for your logger example. I implemented them but not yet in a release. To try out the code below you will need to install jsonargparse from source.If the CLI will have one or more fixed loggers, then better to make the choice of loggers non-configurable. Only configurable would be the parameters of each fixed logger. For a single logger, this could be done as:
The
save_dir_from_experiment_namefunction is just to illustrate that linked values can be modified, which is what I understood you wanted. The config would have a structure like:For multiple loggers it could be as:
And the config structure:
mikcnt commentedon Jul 26, 2022
Sorry for the superlate response. That's exactly what I was searching for! Thanks for helping out and taking the time to write a comprehensive guide :).
carmocca commentedon Jul 27, 2022
@mikcnt If you don't have any further questions, you should close the issue. Thanks! :)
mauvilsa commentedon Aug 28, 2025
Even though the solution to this was argument linking, I thought of mentioning here that there is an experimental improvement to omegaconf's variable interpolation in jsonargparse. It supports interpolation across configs and command line arguments. If any of you are interested, have a look at mauvilsa/jsonargparse#765