Next-generation nonlinear

[blegat]

We discussed with @ccoffrin, @harshangrjn and @kaarthiksundar about what the next-generation NLP could look like.
Here is a gist of the idea:
We create two new function types. First NonlinearExpressionFunction which will be used by JuMP to give the objective function and the constraints (as NonlinearExpressionFunction-in-EqualTo/GreaterThan/LessThan) to the MOI backend.

struct NonlinearExpressionFunction <: AbstractScalarFunction
    expression
end

Then SecondOrderBlackBoxFunction will be what NLP solvers such as Ipopt supports:

struct SecondOrderBlackBoxFunction <: AbstractScalarFunction
    evaluation_oracle::Function
    gradient_oracle::Function # May need to replace it by `append_to_jacobian_sparsity!` and `fill_constraint_jacobian!`
    hessian_oracle::Function
end

But NLP solvers such as Alpine can directly support NonlinearExpressionFunction as it exploits the expression structure.

We can then define the bridges AffinetoSecondOrderBlackBox and QuadratictoSecondOrderBlackBoxBridge which transforms respectively ScalarAffineFunction and ScalarQuadraticFunction into SecondOrderBlackBoxFunctionBridge (as is currently done in the Ipopt MOI wrapper for instance).

The transformation from NonlinearExpressionFunction to SecondOrderBlackBoxFunction can be achieved by a AutomaticDifferentiationBridge{ADBackend} which computes the callbacks using ADBackend.
We can move the current AD implementation form JuMP to MOI.Utilities into an DefaultADBackend and full_bridge_optimizer adds AutomaticDifferentiationBridge{DefaultADBackend}.
If the user wants to change the AD backend to OtherADBackend, there will be a JuMP function that internally removes AutomaticDifferentiationBridge{DefaultADBackend} and adds AutomaticDifferentiationBridge{OtherADBackend}.

@kaarthiksundar would be interested to work on this and we thought a good first step is to define SecondOrderBlackBoxFunction and create the bridges AffinetoSecondOrderBlackBox and QuadratictoSecondOrderBlackBoxBridge.

Let us know what you think !

[ccoffrin]

Thanks @blegat, at least at a high level this is sounding reasonable to me. Is this required to break the NLP block into separate MOI constraints or is there an intermediate step, where we first address that point?

[blegat]

The first step does not require it but resolving the whole issue requires it.

[mlubin]

At a high level this seems fine. Don't forget that people want to query derivatives from a JuMP model without acting as a solver (http://www.juliaopt.org/JuMP.jl/v0.19.2/nlp/#Querying-derivatives-from-a-JuMP-model-1). It's not clear how that fits in with bridges.

The big details I'm worried about are the specification of the expression graph format. Using Julia expression graphs would be pretty inefficient because every term in the expression is a GC-tracked object.

[rschwarz]

From the perspective of SCIP (consuming expression graphs directly) this sounds good to me. I understand we would not need any bridge then, and could easily port the NLPBlock based code to be used in direct_model mode.

[mewilhel]

Is anyone currently working on this?

If so, I'll be happy to pitch in and collaborate on this. Otherwise, I'll start taking a pass at this. Just let me know.

[odow]

No one is working on this. But this is a massive project to undertake (on the order of months).

We're working on getting funding to support a sustained development effort, but things are still uncertain due to COVID etc.

It won't happen before JuMP 1.0. If you have development time for NLP, it's probably better fixing some of the outstanding issues in Ipopt etc.

[ccoffrin]

Migration of Pavito, Pajarito, Alpine to MOI are good near-term NLP activities.

[mewilhel]

@odow Entirely understood. I'll keep pushing out fixes as I come across them in the meantime.

That said my interest in this is somewhat selfish.

There’s a family of nonsmooth NLP solvers which require generalized derivative information (Lexicographic or Clarke type) that I’m getting interested in. So being able to specify a backend other than the standard AD scheme is of interest. I can do this currently by grabbing the NLPEvaluator from JuMP and manipulating it to construct a new evaluator. That said, this is a bit clunky (and prone to breaking when someone updates the AD in JuMP).

Also, I’ve also had a few people chat with me about wanting to extract computational tapes from JuMP models (for testing things akin forward-reverse interval contractors and so on…).

So, while this is definitely a long-term task, I’m interested in starting to get the MOI foundation for its setup.

@ccoffrin I’d be happy to do that if I ever have some genuinely free time 😊. That said, there are some research activities I’m looking at that may benefit from a more extendable nlp interface (which is motivating my interest in this).

[odow]

prone to breaking when someone updates the AD in JuMP

This isn't going to change in the near-term.

being able to specify a backend other than the standard AD scheme is of interest

The first change on the AD in JuMP will be to enable this feature :)

[mewilhel]

Awesome! Thanks for the clarification.