Skip to content

Post 2026 update #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ffrancesco94 wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Post 2026 update #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7f59b00
Fixed schedule: separated linear from non-linear regression, removed …
ffrancesco94 Jun 23, 2026
7c47354
Removed overlapping linear algebra stuff with julia-intro
ffrancesco94 Jun 23, 2026
34a5f5b
Used OneHotArrays for both Flux and Lux for consistency
ffrancesco94 Jun 23, 2026
36076bc
Implemented feedback for data science part
ffrancesco94 Jun 24, 2026
b7ada1d
Fixed imports for mean, std in data science
ffrancesco94 Jun 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
210 changes: 35 additions & 175 deletions content/data-science.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,166 +20,6 @@ Data science and machine learning
- 50 min exercises


Working with data
-----------------

In the Data Formats and Dataframes lesson, we explored a Julian approach
to manipulation and visualisation of data.


Here we will learn and clustering, classification, machine learning and deep learning with some toy examples.


Download a dataset
^^^^^^^^^^^^^^^^^^

We start by downloading a dataset containing measurements
of characteristic features of different penguin species.


.. figure:: img/lter_penguins.png
:align: center

Artwork by @allison_horst

.. exercise::

To obtain the data we simply add the PalmerPenguins package.

.. code-block:: julia

using Pkg
Pkg.add("PalmerPenguins")
using PalmerPenguins


As it was done in the Data Formats and Dataframes lesson, we can

.. code-block:: julia

dropmissing!(df)

The main features we are interested in for each penguin observation are
`bill_length_mm`, `bill_depth_mm`, `flipper_length_mm` and `body_mass_g`.
What the first three features mean is illustrated in the picture below.

.. figure:: img/culmen_depth.png
:align: center

Artwork by @allison_horst


Saving the Current Setup
------------------------

There are several ways to save the current setup in Julia.
This section will cover three parts: saving the environment to
have reproducible code and saving data using CSV files or ``JLD``.

1. Saving the Environment
^^^^^^^^^^^^^^^^^^^^^^^^^

.. exercise::
To check the current status of your Julia environment, you can use the status command in the package manager.

.. code-block:: julia

using Pkg
Pkg.status()

.. code-block:: text

Status `~/.julia/environments/v1.9/Project.toml`
[336ed68f] CSV v0.10.11
[aaaa29a8] Clustering v0.15.4
[a93c6f00] DataFrames v1.6.1
[682c06a0] JSON v0.21.4
[8b842266] PalmerPenguins v0.1.4

This will display the list of packages in the current environment along with their versions.

To save the state of your environment, Julia uses two files: ``Project.toml`` and ``Manifest.toml``.
The ``Project.toml`` file specifies the packages that you explicitly added to your environment,
while the ``Manifest.toml`` file records the exact versions of these packages and all their dependencies1.

When you add packages using ``Pkg.add()``, Julia automatically updates these files.
Therefore, your environment’s state (i.e., the set of loaded packages) is automatically saved.
``Project.toml`` and ``Manifest.toml`` are located in the directory of your current Julia environment; in our case, ``~/.julia/environments/v1.9/``.

If you want to replicate this environment on another machine or in another folder, you can do the following:

1. Copy both ``Project.toml`` and ``Manifest.toml`` to the new location.
2. In Julia, navigate to that folder and activate the environment using ``Pkg.activate(".")``.
3. Use ``Pkg.instantiate()`` to download all the necessary packages.

More information in section `Environments` at https://enccs.github.io/julia-intro/development/

2. Saving Data as a CSV File
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

As shown in the Data Formats and DataFrames lesson, a DataFrame can easily dumped into a CSV file using
the ``CSV.jl`` package, which also allows for reading tabular data.

.. exercise::

You can use the CSV.jl package to save a DataFrame as a CSV file, which can be re-read later.

.. code-block:: julia

# using Pkg
# Pkg.add("CSV")
using CSV
CSV.write("penguins.csv", df)

And you can load it back with:

.. code-block:: julia

df = CSV.read("penguins.csv", DataFrame)

3. Saving Data Using JLD/JLD2
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Another option is to use `JLD.jl <https://github.com/JuliaIO/JLD.jl>`_
The ``JLD.jl`` package provides a way to save and load Julia variables while preserving native types.
It is based on HDF5, a cross-platform, multi-language data storage format most frequently used for scientific data.
However, it is written in pure Julia and does not require any of the original C HDF5 implementation.

The ``JLD`` package can be imported in the usual way:

.. code-block:: julia

using Pkg
Pkg.add("JLD")

A DataFrame can be saved to file in the following way:

.. code-block:: julia

using JLD
save("penguins.jld", "df", df)

Here we're saving ``df`` as "df" within ``penguins.jld``. You can load this DataFrame back in with:

.. code-block:: julia

df = load("penguins.jld", "df")

This will return the DataFrame ``df`` from the file and assign it back to ``df``.
In the past years, the ``JLD2.jl`` package came forward as an alternative to ``JLD``. It
is also based on HDF5 and can read h5 files saved by other HDF5 implementations. It exposes an interface
similar to ``JLD`` with ``save()`` and ``load()`` functions, but the more user-friendly function ``jldsave()``
is also available:

.. code-block:: julia

using JLD2
jldsave("penguins.jld2"; df) # This is equivalent to the save command above
df = load("penguins.jld2", "df")

Moreover, a ``jldopen()`` function provides a file-like interface. More information can be found
`here <https://github.com/JuliaIO/JLD2.jl>`__.

Machine learning
----------------

Expand Down Expand Up @@ -595,7 +435,7 @@ Whereas for Lux:

.. code-block:: julia

using MLJ: partition, ConfusionMatrix
using MLJ: partition, ConfusionMatrix, mean, std
using DataFrames
using PalmerPenguins
using OneHotArrays
Expand Down Expand Up @@ -643,7 +483,7 @@ Whereas for Lux:

# onecold (opposite to onehot) gives back the original representation
function accuracy(x, y)
return sum(Flux.onecold(model(x)) .== Flux.onecold(y)) / size(y, 2)
return sum(OneHotArrays.onecold(model(x)) .== OneHotArrays.onecold(y)) / size(y, 2)
end

``model`` will be our neural network, so we go ahead and define it:
Expand Down Expand Up @@ -672,7 +512,6 @@ Whereas for Lux:
ytrain[:,1:5]
# accuracy before training
accuracy(xtrain, ytrain)
accuracy(xtest, ytest)

Finally we are ready to train the model. Let's run 100 epochs:

Expand All @@ -687,9 +526,6 @@ Whereas for Lux:
accuracy(xtrain, ytrain)
accuracy(xtest, ytest)

The performance of the model is probably somewhat underwhelming, but you will
fix that in an exercise below!

We finally create a confusion matrix to quantify the performance of the model:

.. code-block:: julia
Expand All @@ -698,6 +534,15 @@ Whereas for Lux:
true_species = OneHotArrays.onecold(ytest, ["Adelie", "Gentoo", "Chinstrap"])
ConfusionMatrix()(predicted_species, true_species)

The model can be serialised for later inference using JLD2:

.. code-block:: julia

@save "model_trained.jld2" model_state=Flux.state(model)
@load "model_trained.jld2" model_state
model = model(); # So the definition has to be available
Flux.loadmodel!(model, model_state);

.. group-tab:: Lux

.. code-block:: julia
Expand Down Expand Up @@ -737,7 +582,6 @@ Whereas for Lux:
.. code-block:: julia

accuracy(model, ps, st, xtrain, ytrain)
accuracy(model, ps, st, xtest, ytest)

Now we can start the training loop!

Expand Down Expand Up @@ -765,19 +609,35 @@ Whereas for Lux:
true_species = OneHotArrays.onecold(ytest, ["Adelie", "Gentoo", "Chinstrap"])
ConfusionMatrix()(predicted_species, true_species)

The model parameters can be saved using JLD2:

.. code-block:: julia

@save "trained_model.jld2" ps st
@load "trained_model.jld2" ps st

y, st = model(x, ps, st)

As you can see, the performance of the model is less than optimal. This is due
to a combination of two effects:
- The features have very different ranges (from about 15 to 6000)
- We are using a sigmoid activation function

The sigmoid function is capped between 0 and 1 and tends to "squish" the very
large and very small values. When the features have different ranges, sigmoid
leads to a loss of information, which prevents the network from learning
effectively. We will try to fix this, as well as apply other optimisations, in
the exercises below.

Exercises
---------

.. exercise:: Improve the deep learning model

Improve the performance of the neural network we trained above!
The network is not improving much because of the large numerical
range of the input features (from around 15 to around 6000) combined
with the fact that we use a ``sigmoid`` activation function. A standard
method in machine learning is to normalize features by "batch
normalization". Replace the network definition with the following and
see if the performance improves:
A standard method in machine learning is to normalize features by "batch
normalization". Replace the network definition with the following and see if
the performance improves:

.. tabs::

Expand Down Expand Up @@ -866,8 +726,8 @@ Exercises
accuracy(xtest, ytest)
# 0.9850746268656716

predicted_species = Flux.onecold(model(xtest), ["Adelie", "Gentoo", "Chinstrap"])
true_species = Flux.onecold(ytest, ["Adelie", "Gentoo", "Chinstrap"])
predicted_species = OneHotArrays.onecold(model(xtest), ["Adelie", "Gentoo", "Chinstrap"])
true_species = OneHotArrays.onecold(ytest, ["Adelie", "Gentoo", "Chinstrap"])
ConfusionMatrix()(predicted_species, true_species)

.. group-tab:: Lux
Expand Down
19 changes: 19 additions & 0 deletions content/dataformats-dataframes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,25 @@ Here's how you can create a new dataframe:
df = DataFrame(Parquet2.Dataset("penguins.parquet"))


Finally, we would also like to introduce a Julia-native format called `JLD2
<https://github.com/JuliaIO/JLD2.jl>`_. This format is based on HDF5, which is
a standard format for scientific data, but does not bring external dependencies
(like the HDF5 headers). It can be used to serialise native Julia *data*,
including custom structs. While not as interoperable as Arrow, JSON or CSV, it
*can*, in principle, be opened by any HDF5 compatible client.

Our penguins dataset can be saved and loaded as follows:

.. code-block:: julia

using JLD2
@save "penguins.jld2" df
@load "penguins.jld2" df

It is possible to save and load several variables with a dictionary-like
syntax, as well as using a file-like syntax with a do-block. For more
information, the `documentation <https://juliaio.github.io/JLD2.jl/dev/>`_ is a
good place to start.

Inspect dataset
^^^^^^^^^^^^^^^
Expand Down
12 changes: 6 additions & 6 deletions content/guide.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,15 +86,15 @@ The next three days were scheduled as follows:
+-------------+--------------------------------------------+
| Time | Section |
+=============+============================================+
| 9:00-9:45 | Machine learning |
| 9:00-9:45 | Clustering and classification |
+-------------+--------------------------------------------+
| 9:45-10:00 | Break |
+-------------+--------------------------------------------+
| 10:00-10:45 | Clustering and Classification |
| 10:00-10:45 | Deep learning |
+-------------+--------------------------------------------+
| 10:45-11:00 | Break |
+-------------+--------------------------------------------+
| 11:00-12:00 | Deep learning |
| 11:00-12:00 | Non-linear regression |
+-------------+--------------------------------------------+


Expand All @@ -103,15 +103,15 @@ The next three days were scheduled as follows:
+-------------+--------------------------------------------+
| Time | Section |
+=============+============================================+
| 9:00-9:45 | Linear regression |
| 9:00-9:45 | Non-linear regression (cont.) |
+-------------+--------------------------------------------+
| 9:45-10:00 | Break |
+-------------+--------------------------------------------+
| 10:00-10:45 | Non-linear regression |
| 10:00-10:45 | Scientific machine learning |
+-------------+--------------------------------------------+
| 10:45-11:00 | Break |
+-------------+--------------------------------------------+
| 11:00-11:45 | Scientific machine learning |
| 11:00-11:45 | Scientific machine learning (cont.) |
+-------------+--------------------------------------------+
| 11:45-12:00 | Conclusions and outlook |
+-------------+--------------------------------------------+
Expand Down
3 changes: 2 additions & 1 deletion content/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@ please visit the lesson `Julia for high-performance scientific computing <https:
dataformats-dataframes
linear-algebra
data-science
regression
linear-regression
non-linear-regression
sciml

.. toctree::
Expand Down
Loading
Loading