automatically differentiate CL kernels

[robbert-harms]

Hello,

I am looking for a software package that can do automatic differentiation of OpenCL kernels. Can this package do that?

For example, suppose I have a string containing an OpenCL kernel. Can I use this software package to automatically compute the gradient of the function in that kernel? Or would I need to change the kernel before the gradient can be computed?

Best regards,

Robbert

[msupernaw]

Hi Robert- This code should work for what you are trying to do. I haven't looked at it in a while, so let me review what I wrote and I'll get back to you with an example of how to use it. In the meantime, you may find it informative to look at the test in in "tests/matrix". If I recall correctly, I wrote this to collect the adjoint derivatives on the gpu and put them in a struct called "gradient_structure". The gradient is computed on the cpu in reverse mode. I do recall shelving the project until opencl 2.0 was released so that I could update it to take advantage of the shared memory component rather than copy data back and forth from cpu to gpu. Thanks. Matthew

…

On Tue, Jul 17, 2018 at 10:51 AM, Robbert Harms ***@***.***> wrote: Hello, I am looking for a software package that can do automatic differentiation of OpenCL kernels. Can this package do that? For example, suppose I have a string containing an OpenCL kernel. Can I use this software package to automatically compute the gradient of the function in that kernel? Or would I need to change the kernel before the gradient can be computed? Best regards, Robbert — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEtJEES7tsE6oc8w0pTgukjGdEBtKxEXks5uHfoHgaJpZM4VS_8n> .

-- Matthew Supernaw Scientific Programmer *National Oceanic and Atmospheric Administration* National Marine Fisheries Service Sustainable Fisheries Division St. Petersburg, FL, 33701 Phone 727-551-5606 Fax 727-824-5300

[msupernaw]

Also, you may find this <https://github.com/msupernaw/ATL2> project useful. It's more refined. It doesn't use the gpu, but can utilize threads depending on the formulation of your problem. On Tue, Jul 17, 2018 at 11:09 AM, Matthew Supernaw - NOAA Federal < matthew.supernaw@noaa.gov> wrote:

…

Hi Robert- This code should work for what you are trying to do. I haven't looked at it in a while, so let me review what I wrote and I'll get back to you with an example of how to use it. In the meantime, you may find it informative to look at the test in in "tests/matrix". If I recall correctly, I wrote this to collect the adjoint derivatives on the gpu and put them in a struct called "gradient_structure". The gradient is computed on the cpu in reverse mode. I do recall shelving the project until opencl 2.0 was released so that I could update it to take advantage of the shared memory component rather than copy data back and forth from cpu to gpu. Thanks. Matthew On Tue, Jul 17, 2018 at 10:51 AM, Robbert Harms ***@***.***> wrote: > Hello, > > I am looking for a software package that can do automatic differentiation > of OpenCL kernels. Can this package do that? > > For example, suppose I have a string containing an OpenCL kernel. Can I > use this software package to automatically compute the gradient of the > function in that kernel? Or would I need to change the kernel before the > gradient can be computed? > > Best regards, > > Robbert > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#2>, or mute the thread > <https://github.com/notifications/unsubscribe-auth/AEtJEES7tsE6oc8w0pTgukjGdEBtKxEXks5uHfoHgaJpZM4VS_8n> > . > -- Matthew Supernaw Scientific Programmer *National Oceanic and Atmospheric Administration* National Marine Fisheries Service Sustainable Fisheries Division St. Petersburg, FL, 33701 Phone 727-551-5606 Fax 727-824-5300

-- Matthew Supernaw Scientific Programmer *National Oceanic and Atmospheric Administration* National Marine Fisheries Service Sustainable Fisheries Division St. Petersburg, FL, 33701 Phone 727-551-5606 Fax 727-824-5300

[robbert-harms]

Hi Matthew,

Thank you for your quick answer, your package sounds very promising.

At the moment I am working on scientific image processing software using Python and OpenCL. The Python code generates OpenCL kernels (storing them as a string) which are then compiled and executed. For some applications I need the gradient and/or Hessian of some of these kernels. At the moment I implemented this using numerical differentiation, although AD would be a cleaner method.

As I understand from your first answer, given a kernel, AD4CL computes a sort of gradient computation tree on the GPU, which is then traversed on the CPU to compute the final gradient?

Another question, can it also compute the the second derivatives, the Hessian matrix?

I am a bit new to AD tools, apologies for perhaps using the wrong terminology.

Best wishes,

Robbert

[msupernaw]

Hi Robert, AD4CL cannot compute the Hessian, however ATL can. ATL can also compute up to third-order mixed derivatives rather quickly considering. The unfortunate reason I stopped working on AD4CL was the cost of transferring data back and forth from the host to the gpu. It really nullified the benefits of using the gpu in the first place. Are your kernels complicated? Matthew

…

On Tue, Jul 17, 2018 at 11:27 AM, Robbert Harms ***@***.***> wrote: Hi Matthew, Thank you for your quick answer, your package sounds very promising. At the moment I am working on scientific image processing software using Python and OpenCL. The Python code generates OpenCL kernels (storing them as a string) which are then compiled and executed. For some applications I need the gradient and/or Hessian of some of these kernels. At the moment I implemented this using numerical differentiation, although AD would be a cleaner method. As I understand from your first answer, given a kernel, AD4CL computes a sort of gradient computation tree on the GPU, which is then traversed on the CPU to compute the final gradient? Another question, can it also compute the the second derivatives, the Hessian matrix? I am a bit new to AD tools, apologies for perhaps using the wrong terminology. Best wishes, Robbert — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEtJEL2povzj_riRxgEjlP0o7erMDDdxks5uHgJegaJpZM4VS_8n> .

-- Matthew Supernaw Scientific Programmer *National Oceanic and Atmospheric Administration* National Marine Fisheries Service Sustainable Fisheries Division St. Petersburg, FL, 33701 Phone 727-551-5606 Fax 727-824-5300

[robbert-harms]

Hi Matthew,

I see, so basically ATL is a successor to AD4CL. Does it do source code transformation, or how can it use only the CPU?

Here is an example of an automatically generated model function from my software. The last function _composite_model_function is the function I wish to differentiate with respect to some of the input parameters. The mot_float_type type is typedefined to float or double depending on the desired precision. This is one of the easier modeling functions, there are more complex models involving some 1000 lines of CL code.

A few remarks, in the model functions I only use private and constant memory for the variables. Also, since these kernels are largely dynamically generated, I can do some source code transformations.

Perhaps we can setup a case-study to test your software for my purposes? If it works, I would be interested in either making a Python port, or creating a Python interface to your code to use it with my library.

mot_float_type4 SphericalToCartesian(mot_float_type theta, mot_float_type phi){

    mot_float_type cos_theta;
    mot_float_type sin_theta = sincos(theta, &cos_theta);
    mot_float_type cos_phi;
    mot_float_type sin_phi = sincos(phi, &cos_phi);

    return (mot_float_type4)(cos_phi * sin_theta, sin_phi * sin_theta, cos_theta, 0.0);
}

double S0(mot_float_type s0){ 
    return s0;
}

double Weight(mot_float_type w){
    return w;
}

double Ball(mot_float_type b, mot_float_type d){
    return exp(-d * b);
}

double Stick(mot_float_type4 g, mot_float_type b, mot_float_type d, mot_float_type theta, mot_float_type phi){
    return exp(-b * d * pown(dot(g, SphericalToCartesian(theta, phi)), 2));
}

double _composite_model_function(mot_float_type b, mot_float_type4 g, 
                                 mot_float_type S0_s0, mot_float_type w_ball_w, 
                                 mot_float_type Ball_d, mot_float_type w_stick0_w, 
                                 mot_float_type Stick0_d, mot_float_type Stick0_theta, 
                                 mot_float_type Stick0_phi){
    return ((
            S0(S0_s0) * ((
                Weight(w_ball_w) * Ball(b, Ball_d)
            ) + (
                Weight(w_stick0_w) * Stick(g, b, Stick0_d, Stick0_theta, Stick0_phi)
            ))
        ));
}

Best,

Robbert

[robbert-harms]

On a different note, I looked up your profile in Linked-In and I saw you listed yourself as Scientific Software Engineer. Very interesting, I was planning on a similar career path. Was it difficult to find new jobs for your consultancy practice? Please do not feel obliged to answer.

Considering your research, what might be interesting to you is https://github.com/cbclab/MOT, my OpenCL implementation of a few non-linear optimization routines. It accepts model functions similar to the ones above, combines them with a optimization routine (in OpenCL) and optimizes multiple independent problems on the GPU.