[BUG] OnlineOptim.MULTIPROCESS fails with spawn/forkserver start methods

[mickaelbegon]

Description

OnlineOptim.MULTIPROCESS fails when Python uses a multiprocessing start method that requires the process target to be pickled (spawn or forkserver).

This is now directly visible on Linux with Python 3.14, where forkserver replaced fork as the default start method.

Reproduction

Run an OCP containing CasADi symbolic expressions with:

solver = Solver.IPOPT(online_optim=OnlineOptim.MULTIPROCESS)
sol = ocp.solve(solver)

Using Python 3.14 on Linux raises an exception during:

self.plot_process.start()

The traceback ends with:

Exception: Cannot pickle SX objects without a casadi context
when serializing dict item 'dt'
when serializing NonLinearProgram
when serializing OptimalControlProgram
when serializing ProcessPlotter

Root-cause hypothesis

OnlineCallbackMultiprocess creates a process whose target is a ProcessPlotter containing the complete OCP:

self.plotter = self.ProcessPlotter(self.ocp)
self.plot_process = mp.Process(target=self.plotter, ...)

With fork, the child inherits the parent memory and the OCP is not serialized. With spawn or forkserver, the process target must be pickled, including the OCP and its CasADi SX/MX expressions.

A local comparison found:

• CasADi 3.6.7: regular pickling of an SX fails with TypeError: cannot pickle 'SwigPyObject' object.
• CasADi 3.7.2: regular pickling fails with the explicit CasADi-context error, while global_pickle_context() plus global_unpickle_context() can serialize and restore a simple SX graph.

CasADi added this contextual mechanism in casadi/casadi@e1bb6d7, associated with casadi/casadi#3624. Python documents the POSIX default change to forkserver in Python 3.14.

This suggests a latent dependency on fork: Python 3.14 makes serialization mandatory and CasADi determines the resulting error. This hypothesis should still be confirmed against Bioptim's complete OCP graph and supported environments.

The current OnlineOptim.DEFAULT uses MULTIPROCESS_SERVER, which avoids this path. The problem concerns explicit use of OnlineOptim.MULTIPROCESS.

Possible directions

• Serialize only the plotting payload before starting the child, then deserialize it inside a top-level process entry point.
• Investigate CasADi's global pickle/unpickle contexts for the complete required graph.
• Reuse the OcpSerializable representation used by the server backend.
• If safe support is not practical, detect unsupported start methods early and provide an actionable error recommending OnlineOptim.MULTIPROCESS_SERVER.

Explicitly forcing fork could confirm the diagnosis or provide a temporary Linux workaround, but should probably not be the long-term solution because Python moved away from it for multithreading safety.

Acceptance criteria

• OnlineOptim.MULTIPROCESS either works with forkserver/spawn, or fails early with a clear documented fallback.
• A functional test actually starts the online callback using a non-fork context.
• Python 3.14 is covered explicitly.
• Supported combinations of operating system, Python start method and OnlineOptim backend are documented.

@EveCharbie, could you please confirm or refine the reproduction details from your new environment, especially the Python and CasADi versions?

@pariterre, could you please complete or nuance the CasADi analysis above? In particular, does the contextual pickling mechanism change the implementation options you had in mind?

Could you both also indicate how you would prioritize this relative to improving feature parity for plot_ipopt_outputs and plot_check_conditioning on the server backend?

[pariterre]

@mickaelbegon If there is a way to serialize/deserialize the MX/SX graph (which I was unaware of), this should indeed be used to prepare the plots when multiprocess is used.

[EveCharbie]

Old:
biorbd = 1.11.2
casadi = 1.6.5
python = 3.12.4

New:
biorbd = 1.12.3
casadi = 3.7.2
python = 3.14.6

see PR #1081

[EveCharbie]

plot_check_conditioning is not really used by anyone, so low priority (but I would not delete this piece of code as I refered someone to it lately)
plot_ipopt_outputs I recommend everyone struggling to understand why their ocp is not converging to use it, so I would prioretize.

[mickaelbegon]

@EveCharbie @pariterre — thanks for the environment details and the priority feedback. I have pushed a working prototype branch for architectural review:

https://github.com/mickaelbegon/BiorbdOptim/tree/codex/fix-1082-multiprocess-pickling

Root cause confirmed

Using CasADi's global pickle/unpickle contexts makes MX/SX serializable, but serializing the full OptimalControlProgram still fails immediately afterwards because biorbd_casadi.biorbd.Model contains a non-picklable SWIG object. Serializing the entire OCP is therefore not sufficient for Python 3.14 forkserver/spawn startup.

Architecture implemented

The prototype mirrors the proven client/server split:

• the solver process keeps the complete OCP and initializes PlotOcp(..., only_initialize_variables=True);
• it computes xdata/ydata with the real plotting functions;
• only a JSON OcpSerializable, numerical phase times, and serialized numerical xdata/ydata are sent to the plotting process;
• the child reconstructs a plotting-only OCP and updates the figures without receiving the full OCP, MX/SX graph, or biorbd model;
• intermediary IPOPT iteration saving remains in the solver process;
• shutdown now performs a conditional kill, join, and queue close.

Validation

Tested locally with Python 3.14.4, CasADi 3.7.2 and a real forkserver context:

• a new automated test starts a fresh forkserver interpreter, reconstructs the serialized plotter, and updates its figures;
• a real pendulum IPOPT solve with OnlineOptim.MULTIPROCESS completed one iteration, executed two callbacks, kept the plotting process alive, and closed it cleanly;
• callback cost on this small example was about 2.2 ms per call;
• 29 targeted and neighboring plotting/backend-selection tests pass;
• the full local shard-1 run reached 270 passed and 8 skipped. Its 27 failures were exclusively ACADOS tests because acados_template is not installed locally.

Current limits

• plot_ipopt_outputs and plot_check_conditioning are explicitly unsupported in this serialized multiprocess path for now because both still require symbolic functions and the complete OCP. This avoids silent partial behavior.
• parse_data now runs in the solver process. It is inexpensive on the small example, but it may matter for large OCPs and should be benchmarked.
• explicit OnlineOptim.MULTIPROCESS remains disabled by the existing macOS guard; this prototype targets the Python 3.14 Linux/forkserver failure.

Proposed next steps

1. Please validate or nuance this parent-computes/child-displays split before I open a PR.
2. If accepted, keep the first PR focused on standard plots and forkserver compatibility.
3. Prioritize restoring plot_ipopt_outputs in a separate change using a dedicated numerical payload; keep plot_check_conditioning as a lower-priority follow-up, consistent with Eve's feedback.
4. Investigate serializing only the CasADi functions required by parse_data, instead of the full OCP, if benchmarks justify moving that computation back out of the solver process.

Does this architecture look appropriate, or would you prefer the first implementation to target serialization of the individual plotting functions?

[pariterre]

@mickaelbegon lambda function, as far as I am concerned, will never be pickable. Since, easy problem do not need to check conditioning, while complex problems need, but probably require lambda functions to be properly declared, I suspect it won't be possible anyway. Can you confirm, there is a way to serialize the lambda function, or at least the target can be saved so it is properly serialized/deserialized. If so, a serialize method should be implemented for the OCP. Git history can be used to see a previous implementation of which and/or ReducedOCP can also be used as a template

[mickaelbegon]

I let you decide since you better know Bioptimp and Python than I.
It might (will) affect the speed.

Python lambdas cannot be serialized with the standard pickle mechanism. Tools such as cloudpickle can sometimes serialize them, but this would not solve the general Bioptim case because many plotting lambdas capture an nlp, CasADi functions, or biorbd/SWIG objects.

The current prototype therefore does not attempt to serialize the lambdas. They remain in the solver process, where parse_data evaluates them, and only the resulting numerical xdata/ydata are sent to the plotting process. This also handles Bioptim’s built-in plots, which themselves use lambdas internally.

Targets and compiled casadi.Function objects are different: numerical targets can be serialized directly, and a casadi.Function can be serialized/deserialized using CasADi’s API. This could support a future explicit serialization protocol, but a target alone would not be sufficient to reconstruct arbitrary user logic, mappings, weights, or closures.

I checked the history as you suggested. The former reduced-OCP implementation removed models, symbolic variables, and callbacks before pickling, while a later implementation rebuilt the OCP from its constructor arguments. It is a useful template for a future OCP serialization method, but it does not generically solve lambdas contained in those arguments. Also, commit 2905d3d5 deliberately removed plot functions from CustomPlotSerializable and moved their evaluation to the client side, which is close to the architecture used by this prototype.

For plot_check_conditioning, we could avoid serializing lambdas entirely by evaluating the Jacobian/Hessian functions in the solver process and sending only the resulting matrices. Similarly, plot_ipopt_outputs could use a dedicated numerical payload.

My suggestion would therefore be:

1. keep the first PR focused on the parent-computes/child-displays solution for standard plots and forkserver compatibility;
2. restore plot_ipopt_outputs separately using numerical data or serialized CasADi functions;
3. consider a more general OCP serialization protocol, inspired by ReducedOCP/model serialize() methods, as a distinct larger project.

Does that distinction between serializing the Python callable and serializing its compiled CasADi representation address your concern?