Change default parser mode to json_or_yaml and say which format failed in parse errors

CHANGELOG.rst

@@ -41,8 +41,8 @@ Added
   ``add_method_arguments``, and links applied on instantiate can target the
   parameters of these groups (`#986
   <https://github.com/mauvilsa/jsonargparse/pull/986>`__).
-- New ``__include__`` key accepted at any level of a config, which merges into it
-  the configs that it gives, so that a config can be composed from others.
+- New ``__include__`` key accepted at any level of a config, which merges into
+  it the configs that it gives, so that a config can be composed from others.
   Enabled with the new ``config_include_enabled`` of ``set_parsing_settings``
   (`#990 <https://github.com/mauvilsa/jsonargparse/pull/990>`__).
 - Support for ``TypeAliasType`` aliases whose value is a string, e.g.
@@ -56,13 +56,11 @@ Added
   ``JSONARGPARSE_ALLOW_ABBREV`` environment variable, to accept abbreviated
   options in parsers not given ``allow_abbrev`` (`#993
   <https://github.com/mauvilsa/jsonargparse/pull/993>`__).
-- New ``parser_mode`` setting in ``set_parsing_settings`` and
-  ``JSONARGPARSE_PARSER_MODE`` environment variable, to change the mode of
-  parsers not given ``parser_mode`` (`#997
-  <https://github.com/mauvilsa/jsonargparse/pull/997>`__).
 - Support for ``datetime``, ``date`` and ``time`` (ISO 8601), ``re.Pattern``
   (str only) and ``Fraction`` types (`#1002
   <https://github.com/mauvilsa/jsonargparse/pull/1002>`__).
+- New ``json_or_yaml`` parser mode, which parses as JSON and if that fails as
+  YAML (`#1004 <https://github.com/mauvilsa/jsonargparse/pull/1004>`__).
 
 Fixed
 ^^^^^
@@ -114,16 +112,16 @@ Fixed
 - ``--print_config`` failing when a required subcommand is not given, unlike
   other required arguments (`#984
   <https://github.com/mauvilsa/jsonargparse/pull/984>`__).
-- ``instantiate``, ``auto_cli`` and ``from_config`` failing when a signature
-  has positional-only parameters (`#986
+- ``instantiate``, ``auto_cli`` and ``from_config`` failing when a signature has
+  positional-only parameters (`#986
   <https://github.com/mauvilsa/jsonargparse/pull/986>`__).
 - AST resolver failing for a method whose ``self`` is positional-only (`#986
   <https://github.com/mauvilsa/jsonargparse/pull/986>`__).
 - A typed positional with ``nargs="*"`` set in a config file being reset to
   empty when no values for it are given in the command line (`#986
   <https://github.com/mauvilsa/jsonargparse/pull/986>`__).
-- A positional-only parameter that has a default being required, even though
-  the call can omit it (`#986
+- A positional-only parameter that has a default being required, even though the
+  call can omit it (`#986
   <https://github.com/mauvilsa/jsonargparse/pull/986>`__).
 - Parse errors logged multiple times when raised from a nested parse, e.g. an
   invalid value in a config given as argument (`#989
@@ -145,14 +143,17 @@ Fixed
 - Recursive type aliases, e.g. ``type Json = int | list[Json]``, failing with
   ``RecursionError`` (`#991
   <https://github.com/mauvilsa/jsonargparse/pull/991>`__).
-- Sentinel defaults, e.g. ``UNSET`` for ``x: int | UnsetType = UNSET``, shown
-  in the help as ``<UnsetType object at 0x...>``, and failing to serialize when
-  the sentinel is not defined in the module of its class (`#991
+- Sentinel defaults, e.g. ``UNSET`` for ``x: int | UnsetType = UNSET``, shown in
+  the help as ``<UnsetType object at 0x...>``, and failing to serialize when the
+  sentinel is not defined in the module of its class (`#991
   <https://github.com/mauvilsa/jsonargparse/pull/991>`__).
 - ``TypedDict`` with ``extra_items`` (PEP 728) rejecting the keys it doesn't
   declare, failing with ``Unexpected keys`` instead of validating them against
   the ``extra_items`` type (`#1000
   <https://github.com/mauvilsa/jsonargparse/pull/1000>`__).
+- ``jsonnet`` parser mode hiding Jsonnet evaluation errors when the config also
+  parses as YAML, e.g. ``{"a": error "boom"}`` giving ``a`` the string ``error
+  "boom"`` (`#1004 <https://github.com/mauvilsa/jsonargparse/pull/1004>`__).
 
 Changed
 ^^^^^^^
@@ -178,8 +179,8 @@ Changed
   the only way to strip it (`#969
   <https://github.com/mauvilsa/jsonargparse/pull/969>`__).
 - ``pyyaml`` is no longer a required dependency, install the ``yaml`` extra to
-  have it. Without it, using ``yaml`` raises an informative ``ImportError`` (`#970
-  <https://github.com/mauvilsa/jsonargparse/pull/970>`__).
+  have it. Without it, using ``yaml`` raises an informative ``ImportError``
+  (`#970 <https://github.com/mauvilsa/jsonargparse/pull/970>`__).
 - The ``json`` dump format is now indented, so that the print config argument
   gives a more readable output. Use the new ``json_compact`` format for the
   previous single line output (`#970
@@ -193,7 +194,8 @@ Changed
   <https://github.com/mauvilsa/jsonargparse/pull/973>`__).
 - The ``comments`` flag of the print config argument is now always accepted and
   listed in the help, and fails with an informative error when ``ruamel.yaml``
-  is not installed (`#975 <https://github.com/mauvilsa/jsonargparse/pull/975>`__).
+  is not installed (`#975
+  <https://github.com/mauvilsa/jsonargparse/pull/975>`__).
 - Relative paths in config files are now resolved without changing the process
   working directory (`#979
   <https://github.com/mauvilsa/jsonargparse/pull/979>`__).
@@ -226,14 +228,16 @@ Changed
   <https://github.com/mauvilsa/jsonargparse/pull/990>`__).
 - ``from_config`` now fails with ``ArgumentError`` for an invalid config,
   including a missing required parameter, instead of ``TypeError`` or
-  ``ImportDenied`` (`#990 <https://github.com/mauvilsa/jsonargparse/pull/990>`__).
+  ``ImportDenied`` (`#990
+  <https://github.com/mauvilsa/jsonargparse/pull/990>`__).
 - ``allow_abbrev`` now defaults to ``False``, so abbreviated options, e.g.
   ``--max`` for ``--max_epochs``, fail instead of being expanded (`#993
   <https://github.com/mauvilsa/jsonargparse/pull/993>`__).
-- ``parser_mode`` now defaults to ``json`` instead of ``yaml``, so YAML configs
-  and values with YAML syntax, e.g. ``--list=[a, b]`` or ``--flag=yes``, fail
-  unless ``parser_mode="yaml"`` is given. The ``jsonnet`` parser mode no longer
-  accepts YAML (`#997 <https://github.com/mauvilsa/jsonargparse/pull/997>`__).
+- ``parser_mode`` now defaults to the new ``json_or_yaml`` mode instead of
+  ``yaml``, so values are parsed as JSON and only if that fails as YAML, which
+  without the ``yaml`` extra makes YAML configs fail. Load errors now say which
+  format failed to parse (`#1004
+  <https://github.com/mauvilsa/jsonargparse/pull/1004>`__).
 
 Removed
 ^^^^^^^

DOCUMENTATION.rst

@@ -1354,13 +1354,11 @@ Configuration files
 
 jsonargparse can parse configuration files (config files). The dot notation
 hierarchy of the arguments (see :ref:`nested-namespaces`) defines the structure
-expected in these files. The default format is JSON. To change it, use the
-``parser_mode`` parameter of the parser, e.g.
-``ArgumentParser(parser_mode="yaml")``, or set it globally, which also applies
-to parsers created by other code, with ``set_parsing_settings(parser_mode=...)``
-or the environment variable ``JSONARGPARSE_PARSER_MODE``. The examples in this
-documentation use ``yaml``, which requires the ``yaml`` extra, see
-:ref:`installation`.
+expected in these files. The default ``parser_mode`` is ``json_or_yaml``, which
+parses as JSON and, if that fails, as YAML. YAML requires the ``yaml`` extra,
+see :ref:`installation`, so without it only JSON is accepted. To change the
+mode, use the ``parser_mode`` parameter of the parser, e.g.
+``ArgumentParser(parser_mode="json")``.
 
 The :py:attr:`.ArgumentParser.default_config_files` property holds patterns of
 config files to search for, e.g.
@@ -1521,7 +1519,8 @@ From Python, a config object is serialized with the :meth:`dump
 <.ArgumentParser.dump>` and :meth:`save <.ArgumentParser.save>` methods. The
 supported formats are ``yaml``, ``toml``, ``json``/``json_indented``,
 ``json_compact`` and ``parser_mode``, the default, which uses the format of the
-parser, or ``json`` when the parser mode has no dumper. The ``yaml`` format
+parser, ``yaml`` for ``json_or_yaml`` when the ``yaml`` extra is installed, or
+``json`` when the parser mode has no dumper. The ``yaml`` format
 dumps with a subclass of `yaml.SafeDumper
 <https://pyyaml.org/wiki/PyYAMLDocumentation#dumper>`__ that writes multi-line
 strings as literal blocks, i.e. ``|``, instead of escaping the line breaks. More
@@ -3173,9 +3172,9 @@ The parser then uses these variables, unless the command line overrides them:
     'from env 2'
 
 Note the ``default_env=True`` given to the parser. By default :meth:`parse_args
-<.ArgumentParser.parse_args>` does not parse environment variables. If
-``default_env`` is left unset, they can also be enabled by setting
-``JSONARGPARSE_DEFAULT_ENV=true`` in the shell.
+<.ArgumentParser.parse_args>` does not parse environment variables. Setting
+``JSONARGPARSE_DEFAULT_ENV`` to ``true`` or ``false`` in the shell overrides
+``default_env`` for all parsers, even ones given it explicitly.
 
 The :meth:`parse_env <.ArgumentParser.parse_env>` method parses only environment
 variables, useful when there is no command line call.
@@ -3782,7 +3781,8 @@ globally, which also applies to parsers created by other code:
     set_parsing_settings(allow_abbrev=False)
 
 Without changing Python code, the same is achieved by setting the environment
-variable ``JSONARGPARSE_ALLOW_ABBREV=true``.
+variable ``JSONARGPARSE_ALLOW_ABBREV=true``. Neither applies to parsers given
+``allow_abbrev`` explicitly.
 
 User defined types
 ------------------

README.rst

@@ -190,11 +190,12 @@ specifying one or more of the following extras (optional dependencies):
 ``signatures``, ``docstrings``, ``typeshed``, ``yaml``, ``jsonschema``,
 ``jsonnet``, ``urls``, ``fsspec``, ``toml``, ``ruamel``, ``omegaconf``,
 ``shtab``, and ``argcomplete``. The ``yaml`` extra installs `PyYAML
-<https://pypi.org/project/PyYAML/>`__, required for ``parser_mode="yaml"``. The
-``signatures`` extra includes ``docstrings`` and ``typeshed``, which can also be
-installed individually. Additionally, the ``all`` extras can be used to enable
-all optional features (excluding tab completion ones). To install
-``jsonargparse`` with extras, use the following syntax:
+<https://pypi.org/project/PyYAML/>`__, without which config files are parsed
+only as JSON. The ``signatures`` extra includes ``docstrings`` and
+``typeshed``, which can also be installed individually. Additionally, the
+``all`` extras can be used to enable all optional features (excluding tab
+completion ones). To install ``jsonargparse`` with extras, use the following
+syntax:
 
 .. code-block:: bash
 

jsonargparse/_actions.py

@@ -130,12 +130,19 @@ def apply_config(parser, cfg, dest, value) -> None:
                 cfg_path: Path | None = Path(value, mode=_get_config_read_mode())
             except TypeError as ex_path:
                 try:
-                    if isinstance(load_value(value), str):
+                    try:
+                        loaded = load_value(value)
+                    except get_loader_exceptions() as ex_load:
+                        if value.strip()[:1] in {"[", "{"}:  # clearly not intended as a path
+                            raise ex_load
+                        errors = f"- {ex_path}\n- {indent_text(str(ex_load), first_line=False)}"
+                        raise TypeError(f"Not a valid path nor a config string:\n{errors}") from ex_load
+                    if isinstance(loaded, str):
                         raise ex_path
                     cfg_path = None
                     cfg_file = parser.parse_string(value, **kwargs)
                 except (TypeError, ValueError) + get_loader_exceptions() as ex_str:
-                    raise TypeError(f'Parser key "{dest}": {ex_str}') from ex_str
+                    raise TypeError(f'Parser key "{dest}": {indent_text(str(ex_str), first_line=False)}') from ex_str
             else:
                 cfg_file = parser.parse_path(value, **kwargs)
             cfg_merged = merge_config(parser, cfg_file, cfg)

jsonargparse/_common.py

@@ -281,7 +281,6 @@ class ImportDenied(ImportError, ValueError):
 
 parsing_settings: dict = {
     "allow_abbrev": False,
-    "parser_mode": "json",
     "validate_defaults": False,
     "validate_subclass_spec_in_any": False,
     "instantiate_subclass_spec_in_any": False,
@@ -361,7 +360,6 @@ def get_env_var_bool(name: str) -> bool:
 def set_parsing_settings(
     *,
     allow_abbrev: bool | None = None,
-    parser_mode: str | None = None,
     validate_defaults: bool | None = None,
     validate_subclass_spec_in_any: bool | None = None,
     instantiate_subclass_spec_in_any: bool | None = None,
@@ -388,9 +386,6 @@ def set_parsing_settings(
             ``--max`` for ``--max_epochs``. Default is ``False``, unlike
             argparse. Parsers given ``allow_abbrev`` take precedence, and the
             ``JSONARGPARSE_ALLOW_ABBREV`` environment variable over this.
-        parser_mode: Mode for parsing values and config files of parsers not
-            given ``parser_mode``. Default is ``json``. The
-            ``JSONARGPARSE_PARSER_MODE`` environment variable takes precedence.
         validate_defaults: Whether default values must be valid according to the
             argument type. Defaults are always validated to normalize them, but
             with the default ``False`` an invalid default is kept as is, like
@@ -484,12 +479,6 @@ class when a value for a type that accepts any value, i.e. ``Any``,
             parsing_settings[name] = value
         elif value is not None:
             raise ValueError(f"{name} must be a boolean, but got {value}.")
-    # parser_mode
-    if parser_mode is not None:
-        from ._loaders_dumpers import check_parser_mode
-
-        check_parser_mode(parser_mode)
-        parsing_settings["parser_mode"] = parser_mode
     # unset_sentinel
     if isinstance(unset_sentinel, bool):
         parsing_settings["unset_sentinel"] = Unset if unset_sentinel else None
@@ -518,7 +507,6 @@ class when a value for a type that accepts any value, i.e. ``Any``,
 
 parsing_settings_env_vars = {
     "allow_abbrev": "JSONARGPARSE_ALLOW_ABBREV",
-    "parser_mode": "JSONARGPARSE_PARSER_MODE",
     "add_print_completion_argument": "JSONARGPARSE_ADD_PRINT_COMPLETION_ARGUMENT",
 }
 
@@ -528,23 +516,10 @@ def get_parsing_setting(name: str):
         raise ValueError(f"Unknown parsing setting {name}.")
     var_name = parsing_settings_env_vars.get(name, "")
     if var_name in os.environ:
-        if name == "parser_mode":
-            return get_env_var_parser_mode(var_name)
         return get_env_var_bool(var_name)
     return parsing_settings[name]
 
 
-def get_env_var_parser_mode(name: str) -> str:
-    from ._loaders_dumpers import check_parser_mode
-
-    value = os.environ[name]
-    try:
-        check_parser_mode(value)
-    except ValueError as ex:
-        raise ValueError(f"Invalid value for environment variable {name}: {ex}") from ex
-    return value
-
-
 def validate_default(container: ActionsContainer, action: argparse.Action, logger: logging.Logger) -> None:
     if action.default is get_parsing_setting("unset_sentinel") or not hasattr(action, "_check_type"):
         return

jsonargparse/_core.py

@@ -45,11 +45,11 @@
 from ._jsonschema import ActionJsonSchema
 from ._link_arguments import ActionLink, ArgumentLinking
 from ._loaders_dumpers import (
-    check_parser_mode,
     check_valid_dump_format,
     dump_using_format,
     get_loader_exceptions,
     load_value,
+    loaders,
 )
 from ._namespace import (
     Namespace,
@@ -70,6 +70,8 @@
     _get_config_read_mode,
     fsspec_support,
     import_fsspec,
+    import_jsonnet,
+    import_pyyaml,
     omegaconf_apply,
     pydantic_model_accepts_extra,
     pydantic_support,
@@ -109,6 +111,7 @@
     get_argument_group_class,
     get_private_kwargs,
     identity,
+    indent_text,
     load_config_path_context,
     merge_config,
     resolve_config_includes,
@@ -297,7 +300,7 @@ def __init__(
         logger: logging.Logger | bool | str | dict = False,
         version: str | None = None,
         print_config: str | None = "--print_%s",
-        parser_mode: str | None = None,
+        parser_mode: str = "json_or_yaml",
         dump_header: list[str] | None = None,
         default_config_files: list[str | os.PathLike] | None = None,
         default_env: bool = False,
@@ -316,8 +319,8 @@ def __init__(
             logger: Logger to use or configuration for logger.
             version: Program version which will be printed by the ``--version`` argument.
             print_config: Name for print config argument, ``%s`` is replaced by config dest, set ``None`` to disable.
-            parser_mode: Mode for parsing values: ``json``, ``yaml``, ``jsonnet`` or added via :func:`.set_loader`,
-                ``None`` to use the global setting.
+            parser_mode: Mode for parsing values: ``json_or_yaml``, ``json``, ``yaml``, ``jsonnet`` or added via
+                :func:`.set_loader`.
             dump_header: Header to include as comment when dumping a config object.
             default_config_files: Default config file locations, e.g. ``['~/.config/myapp/*.yaml']``.
             default_env: Set the default value on whether to parse environment variables.
@@ -800,7 +803,7 @@ def _load_config_parser_mode(
         try:
             cfg_dict = load_value(content, path=path, ext_vars=ext_vars)
         except get_loader_exceptions() as ex:
-            raise TypeError(f"Problems parsing config: {ex}") from ex
+            raise TypeError(f"Problems parsing config:\n{indent_text(str(ex))}") from ex
         if not isinstance(cfg_dict, dict):
             raise TypeError(f"Unexpected config: {content}")
         return self._apply_actions(resolve_config_includes(cfg_dict), prev_cfg=prev_cfg)
@@ -1712,24 +1715,25 @@ def env_prefix(self, env_prefix: bool | str):
 
     @property
     def parser_mode(self) -> str:
-        """Mode for parsing config files, ``json``, ``yaml``, ``jsonnet`` or ones added via :func:`.set_loader`.
-
-        When set to ``None``, the ``parser_mode`` setting of :func:`.set_parsing_settings` is used.
+        """Mode for parsing config files, ``json_or_yaml``, ``json``, ``yaml``, ``jsonnet`` or via :func:`.set_loader`.
 
         :getter: Returns the current parser mode.
-        :setter: Sets the parser mode or ``None``.
+        :setter: Sets the parser mode.
 
         Raises:
             ValueError: If an invalid value is given.
         """
-        if self._parser_mode is None:
-            return get_parsing_setting("parser_mode")
         return self._parser_mode
 
     @parser_mode.setter
-    def parser_mode(self, parser_mode: str | None):
-        if parser_mode is not None:
-            check_parser_mode(parser_mode)
+    def parser_mode(self, parser_mode: str):
+        accepted = set(loaders).union({"omegaconf", "omegaconf+"})
+        if parser_mode not in accepted:
+            raise ValueError(f"The only accepted values for parser_mode are {accepted}.")
+        if parser_mode == "jsonnet":
+            import_jsonnet("parser_mode=jsonnet")
+        elif parser_mode == "yaml":
+            import_pyyaml("parser_mode=yaml")
         self._parser_mode = parser_mode
         if self._subcommands_action:
             for subparser in self._subcommands_action._name_parser_map.values():