Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
73599cf
Rename test directory
GjjvdBurg Feb 11, 2021
0478480
Reorder makefile
GjjvdBurg Feb 11, 2021
ac03a41
Add github action
GjjvdBurg Feb 11, 2021
1747e19
install numpy prerequisite when testing
GjjvdBurg Feb 11, 2021
a79d10f
make takes care of install
GjjvdBurg Feb 11, 2021
9f28856
install numpy before compile
GjjvdBurg Feb 11, 2021
28ace55
install system dependencies
GjjvdBurg Feb 12, 2021
87c0c17
Recursively clone submodules
GjjvdBurg Aug 1, 2021
3253bc7
Ensure we have cython when building c extension
GjjvdBurg Aug 1, 2021
f08d7b6
Fix numpy deprecation warning
GjjvdBurg Aug 1, 2021
de2265d
Test on macOS as well
GjjvdBurg Aug 1, 2021
d3536b9
Skip brew update step
GjjvdBurg Aug 1, 2021
2037b1e
Set openblas env vars for osx
GjjvdBurg Aug 4, 2021
1a4a32a
set env var properly
GjjvdBurg Aug 4, 2021
21cb151
Attempt to build on windows
GjjvdBurg Aug 4, 2021
58f4181
Reduce duplicate builds
GjjvdBurg Aug 4, 2021
5c86cfd
Only build on windows for debugging
GjjvdBurg Aug 4, 2021
9619449
Try to convey openblas install to numpy
GjjvdBurg Aug 4, 2021
3b5212f
Use Powershell for echo
GjjvdBurg Aug 4, 2021
b47ed6c
Fix multiline run command
GjjvdBurg Aug 4, 2021
660ca32
append to config
GjjvdBurg Aug 4, 2021
4a84454
debugging
GjjvdBurg Aug 4, 2021
f96c5f2
Try to use Tree
GjjvdBurg Aug 4, 2021
65e96a4
Incorporate openblas version
GjjvdBurg Aug 4, 2021
a985092
make list
GjjvdBurg Aug 4, 2021
873718c
attempt to enable distutils debug
GjjvdBurg Aug 4, 2021
f794f9b
Update tree command
GjjvdBurg Aug 4, 2021
2b35a40
attempt to hack the blas libs using setup.py
GjjvdBurg Aug 4, 2021
f225890
set distutils verbosity
GjjvdBurg Aug 4, 2021
0e4c6f6
Try un-escaping slashes
GjjvdBurg Aug 4, 2021
9d29cbd
fix setup.py
GjjvdBurg Aug 4, 2021
e8d4a68
rename libopenblas.dll.a to libopenblas.lib
GjjvdBurg Aug 4, 2021
c07a2b3
add bitness dir
GjjvdBurg Aug 4, 2021
fb3aa1a
disable distutils debugging
GjjvdBurg Aug 4, 2021
c1b192f
run tree after rename
GjjvdBurg Aug 4, 2021
8b1cf8d
set extra compile args
GjjvdBurg Aug 4, 2021
65bc52d
rename other lib too
GjjvdBurg Aug 4, 2021
02205ef
try random line from the web
GjjvdBurg Aug 4, 2021
7643069
fix path
GjjvdBurg Aug 4, 2021
6a37776
disable flag (not using vscode now)
GjjvdBurg Aug 4, 2021
2b60756
Let's start trying with cibuildwheel
GjjvdBurg Aug 4, 2021
101677a
disable the compiler change for now
GjjvdBurg Aug 4, 2021
dbbfbcc
set openblas version
GjjvdBurg Aug 4, 2021
ec0a736
Install numpy and cython before build
GjjvdBurg Aug 4, 2021
7e0e360
move windows build commands to script
GjjvdBurg Aug 4, 2021
7cb8701
add ps1 extension
GjjvdBurg Aug 4, 2021
7ef79e2
set openblas path based on bitness
GjjvdBurg Aug 5, 2021
b29db2a
Fix test command
GjjvdBurg Aug 5, 2021
13a6a4f
set compile flag again
GjjvdBurg Aug 5, 2021
1b2457b
Attempt to use scikit-learn style build scripts
GjjvdBurg Aug 5, 2021
fcecb2c
Remove extra text
GjjvdBurg Aug 5, 2021
fa7b6e0
bump minimal python version
GjjvdBurg Aug 5, 2021
2e28934
Add distributor init file
GjjvdBurg Aug 5, 2021
0836454
Move distributor init up
GjjvdBurg Aug 5, 2021
5164957
attempt to debug wheel contents
GjjvdBurg Aug 5, 2021
f11c9b3
tree not available
GjjvdBurg Aug 5, 2021
3fdb121
debugging
GjjvdBurg Aug 5, 2021
a04fc6e
debug import failures
GjjvdBurg Aug 5, 2021
17fc2a6
inspect distributor init
GjjvdBurg Aug 5, 2021
9d71567
attempt to inspect dll file
GjjvdBurg Aug 5, 2021
72afea3
attempt to inspect pyd file
GjjvdBurg Aug 5, 2021
ff21568
remove inspect (no dumpbin)
GjjvdBurg Aug 5, 2021
b23af8d
try to vendorize openblas lib too
GjjvdBurg Aug 5, 2021
89a26df
fix typos
GjjvdBurg Aug 5, 2021
bec828f
package native dlls
GjjvdBurg Aug 5, 2021
a0cd514
debug why not found
GjjvdBurg Aug 5, 2021
76b9a0c
Print some more info
GjjvdBurg Sep 9, 2021
2b13863
try different dlls
GjjvdBurg Sep 9, 2021
1de2fdc
remove test lib paths (not dlls)
GjjvdBurg Sep 9, 2021
bfc6c03
Copy all dlls that come with openblas
GjjvdBurg Sep 9, 2021
7d63cef
debug where failure occurs
GjjvdBurg Sep 9, 2021
3750fca
revert to single dll
GjjvdBurg Sep 9, 2021
6dd8eb8
try again with alternative files
GjjvdBurg Sep 9, 2021
bb8a1de
Increase build verbosity
GjjvdBurg Sep 12, 2021
527531c
Reinstate dlls
GjjvdBurg Sep 12, 2021
6a9d6dd
try building with bin instead of lib
GjjvdBurg Sep 12, 2021
f4d2a87
undo previous
GjjvdBurg Sep 12, 2021
3735fe4
more debugging
GjjvdBurg Sep 12, 2021
c06e3bc
try taking the dll from numpy
GjjvdBurg Sep 12, 2021
3d864aa
fix typo
GjjvdBurg Sep 12, 2021
966dedb
Give tmate a try
GjjvdBurg Oct 2, 2021
7e04f5c
Mark which dll is loaded
GjjvdBurg Oct 18, 2021
4ed9856
disable tmate
GjjvdBurg Oct 18, 2021
5cfc113
fix cron line
GjjvdBurg Apr 14, 2022
f46ff7e
try adding lib to the manifest
GjjvdBurg Apr 17, 2022
984b549
try to grab the wrapper.lib when vendorizing
GjjvdBurg Apr 17, 2022
59ea325
also load the wrapper lib as dll
GjjvdBurg Apr 17, 2022
2908915
remove old stuff
GjjvdBurg Apr 17, 2022
55a8eb3
increase sleep for better log formatting
GjjvdBurg Apr 17, 2022
b05ef6e
use the basename when loading
GjjvdBurg Apr 17, 2022
c10a6e5
ship wrapper.lib but don't load it
GjjvdBurg Apr 17, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
name: build

on:
push:
branches:
- master
pull_request:
branches:
- master
schedule:
- cron: 42 5 */14 * *

jobs:
test:
name: Unit tests for Python package
runs-on: ${{ matrix.os }}
strategy:
matrix:
#os: [ 'ubuntu-latest', 'macos-latest', 'windows-latest' ]
os: [ 'windows-latest' ]
# minimal and latest
#py: [ '3.6', '3.9' ]
py: [ '3.6' ]
openblas_version: [ '0.2.14.1' ]
steps:
- name: Install Python ${{ matrix.py }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.py }}

- name: Install system dependencies (Linux)
if: runner.os == 'Linux'
run: sudo apt-get install libatlas-base-dev liblapack-dev liblapacke-dev

- name: Install system dependencies (OSX)
if: runner.os == 'macOS'
run: brew install openblas

- name: Install system dependencies (Windows)
if: runner.os == 'Windows'
run: |
nuget install OpenBLAS -Version ${{ matrix.openblas_version }} -OutputDirectory /cibw/openblas -Verbosity detailed
echo "[openblas]" | Out-File -FilePath ~/.numpy-site.cfg -Encoding utf8
echo "libraries = openblas" | Out-File -FilePath ~/.numpy-site.cfg -Encoding utf8 -Append
echo 'library_dirs = D:\cibw\openblas\OpenBLAS.${{ matrix.openblas_version }}\lib\native\lib\x64' | Out-File -FilePath ~/.numpy-site.cfg -Encoding utf8 -Append
echo 'include_dir = D:\cibw\openblas\OpenBLAS.${{ matrix.openblas_version }}\lib\native\include' | Out-File -FilePath ~/.numpy-site.cfg -Encoding utf8 -Append
TREE "D:\\cibw\\openblas" /F
Rename-Item "D:\\cibw\\openblas\\OpenBLAS.${{ matrix.openblas_version }}\\lib\\native\\lib\\x64\\libopenblas.dll.a" "libopenblas.lib"
Rename-Item "D:\\cibw\\openblas\\OpenBLAS.${{ matrix.openblas_version }}\\lib\\native\\lib\\win32\\libopenblas.dll.a" "libopenblas.lib"
TREE "D:\\cibw\\openblas" /F
#echo "[build]`ncompiler=mingw32" | Out-File -Encoding ASCII
#~/pydistutils.cfg

- name: Checkout
uses: actions/checkout@v2
with:
submodules: recursive

- name: Run unit tests (Linux)
if: runner.os == 'Linux'
run: make test_direct

- name: Run unit tests (OSX)
if: runner.os == 'macOS'
env:
LDFLAGS: "-L/usr/local/opt/openblas/lib"
CPPFLAGS: "-I/usr/local/opt/openblas/include"
run: make test_direct

- name: Run unit tests (Windows)
if: runner.os == 'Windows'
run: make test_direct


52 changes: 52 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: Deploy to PyPI

on:
pull_request:
branches:
- master

jobs:
build_wheels:
name: Build wheels on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ 'windows-latest' ]
python: [37]
openblas_version: [ '0.2.14.1' ]
include:
- os: windows-latest
bitness: 64
platform_id: win_amd64
- os: windows-latest
bitness: 32
platform_id: win32

steps:
- name: Checkout
uses: actions/checkout@v2
with:
submodules: recursive

- name: Install Python ${{ matrix.py }}
uses: actions/setup-python@v2
with:
python-version: '3.7'

# - name: Setup tmate session
# uses: mxschmitt/action-tmate@v3

- name: Build and test wheels
env:
CIBW_BUILD: cp${{ matrix.python }}-${{ matrix.platform_id }}
CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: bash build_tools/github/repair_windows_wheels.sh {wheel} {dest_dir} ${{ matrix.bitness }}
CIBW_BEFORE_TEST_WINDOWS: bash build_tools/github/build_minimal_windows_image.sh ${{ matrix.python }} ${{ matrix.bitness }}
CIBW_TEST_COMMAND: "python -VV && python -m unittest discover -f -s {project}/tests/"
CIBW_TEST_EXTRAS: "dev"
CIBW_SKIP: "pp* cp27-* cp33-* cp34-* cp35-*"
CIBW_BEFORE_ALL_WINDOWS: pwsh build_tools/github/setup_windows.ps1
CIBW_BEFORE_BUILD: "pip install numpy Cython"
CIBW_BUILD_VERBOSITY: 1
#CIBW_ARCHS_MACOS: x86_64 arm64 universal2
#CIBW_ARCHS_LINUX: auto aarch64
run: bash build_tools/github/build_wheels.sh
72 changes: 52 additions & 20 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -17,33 +17,73 @@ help:
awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m\
%s\n", $$1, $$2}'

in: inplace
inplace:
python setup.py build_ext -i
################
# Installation #
################

.PHONY: inplace install

inplace: ## Build C extensions
pip install numpy Cython && python setup.py build_ext -i

install: ## Install for the current user using the default python command
python setup.py build_ext --inplace
python setup.py install --user
python setup.py build_ext --inplace && \
python setup.py install --user

test: venv ## Run nosetests using the default nosetests command
source $(VENV_DIR)/bin/activate && green -a -vv -f
################
# Distribution #
################

develop: ## Install a development version of the package needed for testing
python setup.py develop --user
.PHONY: release dist

dist: ## Make Python source distribution
python setup.py sdist

release:
release: ## Prepare a release
python make_release.py

###########
# Testing #
###########

.PHONY: test test_direct

test: venv ## Run nosetests using the default nosetests command
source $(VENV_DIR)/bin/activate && green -a -vv -f ./tests

test_direct: inplace ## Run unit tests without a virtual environment
pip install wheel numpy && pip install . && \
python -m unittest discover ./tests

#################
# Documentation #
#################

docs: doc
doc: venv ## Build documentation with Sphinx
source $(VENV_DIR)/bin/activate && m2r README.md && mv README.rst $(DOC_DIR)
source $(VENV_DIR)/bin/activate && m2r CHANGELOG.md && mv CHANGELOG.rst $(DOC_DIR)
source $(VENV_DIR)/bin/activate && $(MAKE) -C $(DOC_DIR) html

clean: ## Clean build dist and egg directories left after install
#######################
# Virtual environment #
#######################

.PHONY: venv

venv: $(VENV_DIR)/bin/activate

$(VENV_DIR)/bin/activate: setup.py
test -d $(VENV_DIR) || python -m venv $(VENV_DIR)
source $(VENV_DIR)/bin/activate && pip install -U numpy && \
pip install -e .[dev]
touch $(VENV_DIR)/bin/activate

############
# Clean up #
############

clean: ## Clean up after build or dist and remove compiled code
rm -rf ./dist
rm -rf ./build
rm -rf ./$(PACKAGE).egg-info
Expand All @@ -55,13 +95,5 @@ clean: ## Clean build dist and egg directories left after install
find . -type f -iname '*.pyc' -delete
find . -type d -name '__pycache__' -empty -delete

cleaner: clean
cleaner: clean ## Remove Cython output too
rm -f ./src/wrapper.c

venv: $(VENV_DIR)/bin/activate

$(VENV_DIR)/bin/activate: setup.py
test -d $(VENV_DIR) || python -m venv $(VENV_DIR)
source $(VENV_DIR)/bin/activate && pip install -U numpy && \
pip install -e .[dev]
touch $(VENV_DIR)/bin/activate
15 changes: 15 additions & 0 deletions build_tools/github/Windows
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Get the Python version of the base image from a build argument
ARG PYTHON_VERSION
FROM winamd64/python:$PYTHON_VERSION-windowsservercore

ARG WHEEL_NAME
ARG CONFTEST_NAME
ARG CIBW_TEST_REQUIRES

# Copy and install the Windows wheel
COPY $WHEEL_NAME $WHEEL_NAME
COPY $CONFTEST_NAME $CONFTEST_NAME
RUN pip install $env:WHEEL_NAME

# Install the testing dependencies
RUN pip install $env:CIBW_TEST_REQUIRES.split(" ")
31 changes: 31 additions & 0 deletions build_tools/github/build_minimal_windows_image.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/bin/bash

set -e
set -x

PYTHON_VERSION=$1
BITNESS=$2

if [[ "$PYTHON_VERSION" == "36" || "$BITNESS" == "32" ]]; then
# Python 3.6 and 32-bit architectures are not supported by the
# official Docker images: Tests will just be run on the host (instead
# of the minimal Docker container)
exit 0
fi

TEMP_FOLDER="$HOME/AppData/Local/Temp"
WHEEL_PATH=$(ls -d $TEMP_FOLDER/*/repaired_wheel/*)
WHEEL_NAME=$(basename $WHEEL_PATH)

cp $WHEEL_PATH $WHEEL_NAME

# Dot the Python version for identifying the base Docker image
PYTHON_VERSION=$(echo ${PYTHON_VERSION:0:1}.${PYTHON_VERSION:1:2})

# Build a minimal Windows Docker image for testing the wheels
docker build --build-arg PYTHON_VERSION=$PYTHON_VERSION \
--build-arg WHEEL_NAME=$WHEEL_NAME \
--build-arg CONFTEST_NAME=$CONFTEST_NAME \
--build-arg CIBW_TEST_REQUIRES="$CIBW_TEST_REQUIRES" \
-f build_tools/github/windows \
-t gensvm/minimal-windows .
11 changes: 11 additions & 0 deletions build_tools/github/build_wheels.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/bin/bash

set -e
set -x

# The version of the built dependencies are specified
# in the pyproject.toml file, while the tests are run
# against the most recent version of the dependencies

python -m pip install cibuildwheel
python -m cibuildwheel --output-dir wheelhouse
1 change: 1 addition & 0 deletions build_tools/github/inspect_pyd.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
dumpbin /dependents gensvm-0.2.8/gensvm/cython_wrapper/wrapper.cp37-win32.pyd
32 changes: 32 additions & 0 deletions build_tools/github/repair_windows_wheels.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/bin/bash

# TODO: This file is adapted from scikit-learn

set -e
set -x

WHEEL=$1
DEST_DIR=$2
BITNESS=$3

# By default, the Windows wheels are not repaired.
# In this case, we need to vendor VCRUNTIME140.dll
wheel unpack "$WHEEL"
WHEEL_DIRNAME=$(ls -d gensvm-*)

ls -R -a "${WHEEL_DIRNAME}"

python build_tools/github/vendor.py "$WHEEL_DIRNAME" "$BITNESS"

sleep 1

ls -R -a "${WHEEL_DIRNAME}"

sleep 1

cat "${WHEEL_DIRNAME}/gensvm/_distributor_init.py"

sleep 1

wheel pack "$WHEEL_DIRNAME" -d "$DEST_DIR"
rm -rf "$WHEEL_DIRNAME"
16 changes: 16 additions & 0 deletions build_tools/github/setup_windows.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@


nuget install OpenBLAS -Version 0.2.14.1 -OutputDirectory /cibw/openblas -Verbosity detailed

echo "[openblas]" | Out-File -FilePath ~/.numpy-site.cfg -Encoding utf8

echo "libraries = openblas" | Out-File -FilePath ~/.numpy-site.cfg -Encoding utf8 -Append
echo 'library_dirs = D:\cibw\openblas\OpenBLAS.0.2.14.1\lib\native\lib\x64' | Out-File -FilePath ~/.numpy-site.cfg -Encoding utf8 -Append
echo 'include_dir = D:\cibw\openblas\OpenBLAS.0.2.14.1\lib\native\include' | Out-File -FilePath ~/.numpy-site.cfg -Encoding utf8 -Append

TREE "D:\\cibw\\openblas" /F

Rename-Item "D:\\cibw\\openblas\\OpenBLAS.0.2.14.1\\lib\\native\\lib\\x64\\libopenblas.dll.a" "libopenblas.lib"
Rename-Item "D:\\cibw\\openblas\\OpenBLAS.0.2.14.1\\lib\\native\\lib\\win32\\libopenblas.dll.a" "libopenblas.lib"

TREE "D:\\cibw\\openblas" /F
Loading