Allows stack.yaml files to use !include directives

ChangeLog.md

@@ -34,6 +34,16 @@ Other enhancements:
 * In YAML configuration files, the `recent-snapshots` key is introduced (under
   the `urls` key), to specify the URL used by Stack's `ls snapshots remote`
   command.
+* In YAML configuration files (`stack.yaml` and `config.yaml`), the `!include`
+  YAML directive is now supported, allowing common configuration to be shared
+  across multiple files. For example, projects that maintain multiple
+  `stack.yaml` files for testing against different snapshots can use `!include`
+  to avoid duplicating shared settings.
+* Stack's `config set` command now raises an error (message S-6088) if the
+  target configuration file contains `!include` directives and the key being
+  set is not already present in the file, as appending a new key to such files
+  cannot be done safely. Existing keys can be modified even in files that use
+  `!include`.
 
 Bug fixes:
 

doc/commands/config_command.md

@@ -87,6 +87,14 @@ to be set. See `stack config set` for the available keys.
     The `config set` commands support an existing key only in the form
     `key: value` on a single line.
 
+!!! warning
+
+    The `config set` commands cannot add a new key to a configuration file that
+    uses [`!include`](../configure/yaml/include.md) directives. Stack will report
+    an error if it detects `!include` directives in the target configuration file
+    and the key being set is not already present. Existing keys can be modified
+    even in files that use `!include`.
+
 ## The `stack config set install-ghc` command
 
 ~~~text

doc/configure/yaml/include.md

@@ -0,0 +1,131 @@
+<div class="hidden-warning"><a href="https://docs.haskellstack.org/"><img src="https://cdn.jsdelivr.net/gh/commercialhaskell/stack/doc/img/hidden-warning.svg"></a></div>
+
+# The `!include` directive
+
+Stack's configuration files are in the [YAML](https://yaml.org/) format. Stack
+supports a non-standard `!include` YAML directive that allows the content of
+one YAML file to be included in another. The directive can be used in both
+[project-level and global](index.md#project-level-and-global-configuration-files)
+configuration files.
+
+The included file path is relative to the directory containing the file with the
+`!include` directive.
+
+!!! warning
+
+    The [`stack config set`](../../commands/config_command.md#the-stack-config-set-commands)
+    commands cannot modify a configuration file that uses `!include` directives.
+
+## Including a value
+
+A value for a key can be provided by an included file. For example, given a file
+`snapshot.yaml` with the content:
+
+~~~yaml
+lts-23.24
+~~~
+
+the following project-level configuration file would use `lts-23.24` as the
+snapshot:
+
+~~~yaml
+snapshot: !include snapshot.yaml
+packages:
+- .
+~~~
+
+The included file replaces the `!include` directive with its content, so this is
+equivalent to:
+
+~~~yaml
+snapshot: lts-23.24
+packages:
+- .
+~~~
+
+## Merging mappings
+
+YAML's merge key (`<<`) can be combined with `!include` to merge the content of
+an included file into the current mapping. For example, given a file
+`shared-config.yaml` with the content:
+
+~~~yaml
+ghc-options:
+  "$everything": -Wall
+flags:
+  my-package:
+    dev: true
+~~~
+
+the following project-level configuration file would merge those options:
+
+~~~yaml
+snapshot: lts-23.24
+<<: !include shared-config.yaml
+packages:
+- .
+~~~
+
+This is equivalent to:
+
+~~~yaml
+snapshot: lts-23.24
+ghc-options:
+  "$everything": -Wall
+flags:
+  my-package:
+    dev: true
+packages:
+- .
+~~~
+
+The `!include` directive can also be placed on the line after the merge key:
+
+~~~yaml
+snapshot: lts-23.24
+<<:
+  !include shared-config.yaml
+packages:
+- .
+~~~
+
+## Including list items
+
+The `!include` directive can also be used to include the contents of a file as a
+list item. For example, given a file `extra-deps.yaml` with the content:
+
+~~~yaml
+- acme-missiles-0.3
+- text-short-0.1.6
+~~~
+
+the following would use those as extra dependencies:
+
+~~~yaml
+snapshot: lts-23.24
+extra-deps: !include extra-deps.yaml
+~~~
+
+## Nested includes
+
+Included files can themselves contain `!include` directives, allowing for nested
+composition of configuration. Stack detects and raises an error for cyclic
+includes.
+
+## Use with global configuration
+
+The `!include` directive can also be used in the global configuration file
+(`config.yaml`). For example, given a file `ghc-options.yaml` with the content:
+
+~~~yaml
+ghc-options:
+  "$everything": -j4
+~~~
+
+the global configuration file could include it:
+
+~~~yaml
+<<: !include ghc-options.yaml
+install-ghc: true
+system-ghc: false
+~~~

doc/configure/yaml/index.md

@@ -6,7 +6,8 @@ title: Configuration files
 # Configuration files
 
 Stack is configured by the content of files in the [YAML](https://yaml.org/)
-format.
+format. Stack also supports a non-standard [`!include` directive](include.md)
+for composing configuration from multiple YAML files.
 
 ## Project-specific and non-project specific options
 

doc/maintainers/stack_errors.md

@@ -90,6 +90,7 @@ to take stock of the errors that Stack itself can raise, by reference to the
 
         ~~~haskell
         [S-3136] = NoProjectConfigAvailable
+        [S-6088] | ConfigFileContainsIncludes (Path Abs File)
         ~~~
 
     -   `Stack.Constants.ConstantsException`

mkdocs.yml

@@ -100,6 +100,7 @@ nav:
     - configure/yaml/index.md
     - Project-specific configuration: configure/yaml/project.md
     - Non-project specific configuration: configure/yaml/non-project.md
+    - The !include directive: configure/yaml/include.md
   - Global flags and options: configure/global_flags.md
   - Customisation scripts: configure/customisation_scripts.md
 - Topics:

src/Stack/Config.hs

@@ -55,6 +55,7 @@ import           Data.Monoid.Map ( MonoidMap (..) )
 import qualified Data.Set as Set
 import qualified Data.Text as T
 import qualified Data.Yaml as Yaml
+import qualified Data.Yaml.Include as YamlInclude
 import qualified Distribution.PackageDescription as PD
 import           Distribution.System
                    ( Arch (..), OS (..), Platform (..), buildPlatform )
@@ -1216,7 +1217,7 @@ loadYaml ::
   -> Path Abs File
   -> RIO env (Either Yaml.ParseException a)
 loadYaml parser path =
-  liftIO (Yaml.decodeFileEither (toFilePath path)) >>= \case
+  liftIO (YamlInclude.decodeFileEither (toFilePath path)) >>= \case
     Left err -> pure (Left err)
     Right val ->
       case Yaml.parseEither parser val of

src/Stack/ConfigCmd.hs

@@ -19,6 +19,7 @@ module Stack.ConfigCmd
   , cfgCmdBuildFiles
   , cfgCmdBuildFilesName
   , cfgCmdName
+  , yamlContainsInclude
   ) where
 
 import qualified Data.Aeson.Key as Key
@@ -65,12 +66,18 @@ import           System.Environment ( getEnvironment )
 -- "Stack.ConfigCmd" module.
 data ConfigCmdException
   = NoProjectConfigAvailable
+  | ConfigFileContainsIncludes !(Path Abs File)
   deriving Show
 
 instance Exception ConfigCmdException where
   displayException NoProjectConfigAvailable =
     "Error: [S-3136]\n"
     ++ "'config' command used when no project configuration available."
+  displayException (ConfigFileContainsIncludes configFile) =
+    "Error: [S-6088]\n"
+    ++ "The 'config set' command cannot add a new key to a configuration file \
+       \that uses !include directives: "
+    ++ toFilePath configFile
 
 -- | Function underlying Stack's @config set@ command.
 cfgCmdSet ::
@@ -101,6 +108,8 @@ cfgCmdSet cmd = do
       primaryCmdKey = NE.last $ NE.head cmdKeys
   newYamlLines <- case hits of
     [] -> do
+      when (yamlContainsInclude rawConfig) $
+        throwIO (ConfigFileContainsIncludes configFilePath)
       prettyInfoL
         [ pretty configFilePath
         , flow "has been extended."
@@ -280,6 +289,26 @@ cfgCmdSetKeys (ConfigCmdSetRecommendStackUpgrade _ _) =
 cfgCmdSetKeys (ConfigCmdSetDownloadPrefix _ _) =
   [["package-index", "download-prefix"]]
 
+-- | Check if YAML content contains a @!include@ directive in value position.
+-- This covers both inline values (e.g. @key: !include path@) and values on
+-- the next line after indentation. Stack config keys do not contain spaces or
+-- colons, so the first @:@ is always the value separator.
+yamlContainsInclude :: Text -> Bool
+yamlContainsInclude =
+ let
+   lineContainsInclude yamlLine =
+     let stripped = T.stripStart yamlLine
+     in  includeAsValue stripped || includeOnOwnLine stripped
+
+   includeAsValue strippedLine =
+     let (_key, rest) = T.breakOn ":" strippedLine
+     in  "!include" `T.isPrefixOf` T.stripStart (T.drop 1 rest)
+
+   includeOnOwnLine strippedLine =
+     "!include" `T.isPrefixOf` strippedLine
+ in
+   any lineContainsInclude . T.lines
+
 -- | The name of Stack's @config@ command.
 cfgCmdName :: String
 cfgCmdName = "config"

tests/integration/tests/6879-stack-yaml-includes/Main.hs

@@ -0,0 +1,57 @@
+import StackTest
+
+import Control.Monad (unless)
+import Data.List (isInfixOf)
+import System.Directory (getCurrentDirectory)
+import System.Environment (setEnv)
+import System.FilePath ( (</>) )
+
+main :: IO ()
+main = do
+  let
+    checkFor expected actual =
+      unless (expected == actual) $
+        error ("expected " <> show expected <> "but got: " <> show actual)
+
+  -- Check that includes in stack.yaml files are included
+  stackCheckStdout
+    ["--stack-yaml","stack-including-flags.yaml","run"]
+    (checkFor "TEST_FLAG was set\n")
+
+  stackCheckStdout
+    ["--stack-yaml","stack-including-flags-with-newline.yaml","run"]
+    (checkFor "TEST_FLAG was set\n")
+
+  stackCheckStdout
+    ["--stack-yaml","stack-not-including-flags.yaml","run"]
+    (checkFor "TEST_FLAG was not set\n")
+
+  -- Check that includes in config.yaml files are included
+  currentDir <- getCurrentDirectory
+  setEnv "STACK_CONFIG" (currentDir </> "config-including-flags.yaml")
+  stackCheckStdout
+    ["--stack-yaml","stack-not-including-flags.yaml","run"]
+    (checkFor "TEST_FLAG was set\n")
+
+  -- Check that 'config set' succeeds when the key already exists in a
+  -- stack.yaml file that uses !include directives
+  stackCheckStderr
+    ["--stack-yaml","stack-including-flags.yaml","config","set","snapshot","lts-24.37"]
+    (expectMessage "already contained the intended configuration")
+
+  -- Check that 'config set' succeeds when the key already exists in a
+  -- stack.yaml file that uses !include directives (with newline variant)
+  stackCheckStderr
+    ["--stack-yaml","stack-including-flags-with-newline.yaml","config","set","snapshot","lts-24.37"]
+    (expectMessage "already contained the intended configuration")
+
+  -- Check that 'config set' raises an error when the key does not exist in a
+  -- stack.yaml file that uses !include directives
+  stackErrStderr
+    ["--stack-yaml","stack-including-file-with-install-ghc.yaml","config","set","install-ghc","true"]
+    (expectMessage "!include")
+
+expectMessage :: String -> String -> IO ()
+expectMessage msg stderr' = do
+  unless (msg `isInfixOf` stderr')
+         (error $ "Expected stderr to contain " ++ show msg ++ " but got:\n" ++ stderr')

tests/integration/tests/6879-stack-yaml-includes/files/app/Main.hs

@@ -0,0 +1,14 @@
+{-# LANGUAGE CPP #-}
+
+module Main
+    ( main
+    ) where
+
+main :: IO ()
+main =
+#if TEST_FLAG
+  putStrLn "TEST_FLAG was set"
+#else
+  putStrLn "TEST_FLAG was not set"
+#endif
+

tests/integration/tests/6879-stack-yaml-includes/files/config-flags.yaml

@@ -0,0 +1,2 @@
+ghc-options:
+  "$everything": -DTEST_FLAG

tests/integration/tests/6879-stack-yaml-includes/files/config-including-flags.yaml

@@ -0,0 +1 @@
+<<: !include config-flags.yaml

tests/integration/tests/6879-stack-yaml-includes/files/files.cabal

@@ -0,0 +1,17 @@
+name:                files
+version:             0.1.0.0
+build-type:          Simple
+cabal-version:       >=1.10
+
+flag test-flag
+  description: Generate a compiler error for test purposes
+  default:     False
+  manual:      True
+
+executable test-exe
+  hs-source-dirs:      app
+  main-is:             Main.hs
+  build-depends:       base >= 4.7 && < 5
+  default-language:    Haskell2010
+  if flag(test-flag)
+    cpp-options: -DTEST_FLAG

tests/integration/tests/6879-stack-yaml-includes/files/install-ghc.yaml

@@ -0,0 +1 @@
+install-ghc: true

tests/integration/tests/6879-stack-yaml-includes/files/stack-flags.yaml

@@ -0,0 +1,3 @@
+flags:
+  files:
+    test-flag: true

tests/integration/tests/6879-stack-yaml-includes/files/stack-including-file-with-install-ghc.yaml

@@ -0,0 +1,2 @@
+snapshot: lts-24.37
+<<: !include install-ghc.yaml

tests/integration/tests/6879-stack-yaml-includes/files/stack-including-flags-with-newline.yaml

@@ -0,0 +1,3 @@
+snapshot: lts-24.37
+<<:
+  !include stack-flags.yaml

tests/integration/tests/6879-stack-yaml-includes/files/stack-including-flags.yaml

@@ -0,0 +1,2 @@
+snapshot: lts-24.37
+<<: !include stack-flags.yaml

tests/integration/tests/6879-stack-yaml-includes/files/stack-not-including-flags.yaml

@@ -0,0 +1 @@
+snapshot: lts-24.37