BAT dependencies, package load time(s) and modularization

[oschulz]

This issue is intended to keep track of modularizing (splitting up) BAT to reduce package load time(s) and increase flexibility.

The current high load time of BAT is mainly due to it's dependencies. Dependency optiions and load-time cost, preliminary analysis:

Unavoidable expensive core deps:

• Distributions (includes StatsBase)
• StaticArrays (indirect through many packages)

Unavoidable non-negligible deps:

• ArraysOfArrays
• ValueShapes
• RecipesBase
• Possibly some ArrayInterface packages (ArrayInterfaceCore is now very lightweight)

Hard-to-avoid non-negligible direct/indirect deps

• BangBang
• Transducers (non-negligible cost on top of BangBang)
• TerminalLoggers
• PrettyTables (may be avoidable)
• DataStructures

Cheap deps:

• StructArrays: Very cheap on top of StaticArrays
• AbstractDifferentiation: Almost free on top of ChainRulesCore

Autodiff choices:

• ForwardDiff: Quite cheap on top of StaticArrays and Distributions
• FiniteDifferences: Basically free on top of StaticArrays and Distributions
• FiniteDiff: Instead of FiniteDifferences, would pull in ArrayInterface
• Zygote: Expensive
• (Future option) Diffractor: Quite cheap on top of StaticArrays

Math deps:

• LinearMaps: About 100 ms, no deps

Statistics deps:

• LogarithmicNumbers: Very cheap on top of unavoidable deps
• MeasureBase: Not that expensive on top of unavoidable deps (uses LogarithmicNumbers)

Optimizer deps:

• NLSolversBase: Cheap itself, but depends on ArrayInterface, ChainRulesCore, SpecialFunctions and ForwardDiff
• Optim: Cheap itself, but pulls in ForwardDiff and ArrayInterface via FiniteDiff,
  BAT would need only Nelder-Mead and L-BFGS for core functionality though
• LBFGSB: Cheap, but indroduces binary deps
• Manopt: Would be chop on top of StaticArrays if it didn't pull in ColorSchemes

Sampler deps:

• AbstractMCMC: Cheap on top of StaticArrays, Distributions, Transducers, TerminalLoggers
• AdvancedHMC: Quite cheap on top of AbstractMCMC
• PSIS: Cheap on top of unavoidable deps

Integator deps:

• CUBA: Cheap, but indroduces binary deps
• MonteCarloIntegration: Almost free on top of unavoidable deps
• HCubature: Almost free on top of unavoidable deps
• QuadGK: Cheap, but only univariate

Deps to avoid:

• Folds: Significant load time cost on top of Transducers
• SciMLBase: Really expensive
• GalacticOptim: Not cheap even on top of SciMLBase, also load-time interplay with SciMLBase
• Optimization: Over 1000 ms load time on top of it's heavy deps (Optim, GPUArrays, RecursiveArrayTools, SciMLBase)

Expensive deps to get rid of:

• DoubleFloats
• Polynomials
• ... ?

Non-negligible deps to get rid of:

• KernelDensity
• PrettyTables (maybe, see above)
• AdaptiveRejectionSampling (cost due to ForwardDiff though)
• ... ?

[ChrisRackauckas]

GalacticOptim: Really expensive even on top of SciMLBase

Even with v3 splitting out the solvers?

[ChrisRackauckas]

SciMLBase: Really expensive

Interesting, how expensive, and due to what? There's not much in there 😅

[oschulz]

SciMLBase: Really expensive Interesting, how expensive, and due to what? There's not much in there

I was very surprised as well, I had always assumed SciMLBase to be very lightweight, I just never timed it before.

GalacticOptim: Really expensive even on top of SciMLBase

Hm, maybe I had GalacticOptim v2 due to some dep in my tests - though even with v3 it has a not insignificant load time:

Session 1:

pkg> st SciMLBase GalacticOptim
Status `/user/.julia/environments/temp/Project.toml`
  [a75be94c] GalacticOptim v3.1.1
  [0bca4576] SciMLBase v1.31.3

julia> using InverseFunctions # load a super-lightweight package to get some initial Pkg costs out of the way

julia> @time using SciMLBase
  2.356626 seconds (5.58 M allocations: 423.052 MiB, 4.17% gc time, 49.19% compilation time)

Session 2:

julia> using InverseFunctions

julia> @time using SciMLBase, GalacticOptim
  2.990720 seconds (6.56 M allocations: 475.628 MiB, 5.91% gc time, 60.39% compilation time)

Session 3:

julia> using InverseFunctions

julia> @time_imports using SciMLBase, GalacticOptim
     10.9 ms    ┌ MacroTools
     18.5 ms  ┌ ZygoteRules
      0.2 ms    ┌ IteratorInterfaceExtensions
      0.7 ms  ┌ TableTraits
      3.5 ms  ┌ Compat
      0.5 ms  ┌ Requires
    141.0 ms  ┌ FillArrays
      0.2 ms  ┌ DataValueInterfaces
    532.6 ms  ┌ StaticArrays
      3.2 ms    ┌ DocStringExtensions
      0.2 ms    ┌ IfElse
     19.4 ms    ┌ RecipesBase
     37.2 ms      ┌ Static
    718.0 ms    ┌ ArrayInterface
      0.7 ms    ┌ Adapt
     58.6 ms    ┌ ChainRulesCore
    864.2 ms  ┌ RecursiveArrayTools
      1.4 ms    ┌ DataAPI
     14.6 ms  ┌ Tables
      0.2 ms  ┌ CommonSolve
      0.8 ms  ┌ ConstructionBase
      0.2 ms  ┌ TreeViews
   1829.6 ms  SciMLBase
      2.7 ms  ┌ DiffResults
      0.3 ms  ┌ Reexport
      6.3 ms    ┌ AbstractTrees
      2.9 ms    ┌ ProgressLogging
      5.6 ms    ┌ LeftChildRightSiblingTrees
     16.7 ms  ┌ TerminalLoggers
      4.8 ms  ┌ ProgressMeter
      2.1 ms  ┌ LoggingExtras
      0.8 ms  ┌ ConsoleProgressMonitor
    321.1 ms  GalacticOptim

Also, in comparison (though Optim and GalacticOptim are not actually replacements for each other of course) - loading Optim:

julia> using InverseFunctions

julia> @time using Distributions, StaticArrays, ArrayInterface # Pretty much unavoidable for BAT
  1.918963 seconds (4.79 M allocations: 333.435 MiB, 3.12% gc time, 37.83% compilation time)

julia> @time using Optim
  0.181226 seconds (475.28 k allocations: 30.906 MiB, 14.12% gc time, 13.72% compilation time)

Loading GalacticOptim:

julia> using InverseFunctions

julia> @time using Distributions, StaticArrays, ArrayInterface # Unavoidable deps for BAT
  1.921450 seconds (4.79 M allocations: 333.419 MiB, 3.04% gc time, 37.15% compilation time)

julia> @time using GalacticOptim
  0.936281 seconds (2.32 M allocations: 181.437 MiB, 3.88% gc time, 59.31% compilation time)

julia> @time using SciMLBase
  0.595445 seconds (861.23 k allocations: 45.663 MiB, 6.73% gc time, 99.98% compilation time)

Why does loading SciMLBase after GalacticOptim take any time at all? This is really weird:

julia> using InverseFunctions

julia> @time using Distributions, StaticArrays, ArrayInterface # Unavoidable deps for BAT
  1.913109 seconds (4.79 M allocations: 333.435 MiB, 3.06% gc time, 37.17% compilation time)

julia> @time using SciMLBase, GalacticOptim
  1.513281 seconds (3.17 M allocations: 227.054 MiB, 4.97% gc time, 75.43% compilation time)

Why is loading SciMLBase and GalacticOptim more expensive than just loading GalacticOptim, which depends on SciMLBase? Some strange Requires effect?

[oschulz]

SciMLBase seems to have some strange load time effects depending on order of package loading in general. I don't get why ...

When timing it all in one go it's not so extreme, but around 300 ms still seems to be very high for the actual code of a "...Base" package:

julia> using InverseFunctions

julia> @time_imports using StaticArrays, ArrayInterface, RecursiveArrayTools, SciMLBase
    527.0 ms  StaticArrays
      3.3 ms  ┌ Compat
      0.4 ms  ┌ Requires
      0.1 ms  ┌ IfElse
     36.9 ms  ┌ Static
    721.1 ms  ArrayInterface
     10.4 ms    ┌ MacroTools
     11.0 ms  ┌ ZygoteRules
    149.6 ms  ┌ FillArrays
      3.3 ms  ┌ DocStringExtensions
     19.0 ms  ┌ RecipesBase
      0.6 ms  ┌ Adapt
     59.7 ms  ┌ ChainRulesCore
    309.3 ms  RecursiveArrayTools
      0.2 ms    ┌ IteratorInterfaceExtensions
      0.6 ms  ┌ TableTraits
      0.2 ms  ┌ DataValueInterfaces
      1.3 ms    ┌ DataAPI
     14.5 ms  ┌ Tables
      0.2 ms  ┌ CommonSolve
      0.8 ms  ┌ ConstructionBase
      0.2 ms  ┌ TreeViews
    307.5 ms  SciMLBase

StaticArrays and especially ArrayInterface make up more of the total load time, of course, together with RecursiveArrayTools which is also not exactly lightweight.

[ChrisRackauckas]

I wonder how requires is measured. My guess is that it's triggering requires in ArrayInterface and that's measured as part of the SciMLBase time.

[oschulz]

I would think so ... probably have to ask Tim or so. :-)

[oschulz]

that it's triggering requires in ArrayInterface

Oh wow, ArrayInterface has a lot of requires!

[ChrisRackauckas]

Hence the idea to make it like GalacticOptim in terms of subpackages.

[oschulz]

You mean JuliaArrays/ArrayInterface.jl#211? Yes, that would be great - and if there a more lightweight parts, maybe some of the requires can be turned into thoses packages depending on it? I feel we have quite a few dependencies in the ecosystem right now that should be the other way round, or common interface packages are missing, and lot's of requires to compensate for it.

Update: We have a very lightweight ArrayInterfaceCore now.

[ChrisRackauckas]

and if there a more lightweight parts, maybe some of the requires can be turned into thoses packages depending on it?

Indeed, that's the dream.