Building error

[nandub]

After installing yotta, I'm trying to build microbit-samples to make sure everything is setup right, but after running yt build I get the following error:

info: generate for target: bbc-microbit-classic-gcc 0.2.3 at /Users/nandub/development/code/microbit/microbit-samples/yotta_targets/bbc-microbit-classic-gcc
error: Can't pickle <function Pack.runScript at 0x10d7c3820>: it's not the same object as yotta.lib.pack.Pack.runScript

Python version: 3.8.3 on a virtualenv

[0xc0170]

how can I reproduce this (step by step) ? This looks like multiprocessing error in yotta.

[nandub]

Steps to reproduce:

I'm using pyenv with pyenv-virtualenv to create a clean python environment. Also I'm using a Mac / OS HIgh Sierra

pyenv install 3.8.3
pyenv global 3.8.3
pyenv virtualenv microbit
pyenv activate microbit

Install yotta

pip install yotta

Compile microbit-samples

git clone https://github.com/lancaster-university/microbit-samples.git
cd microbit-samples
yt clean
yt target bbc-microbit-classic-gcc
yt build
info: generate for target: bbc-microbit-classic-gcc 0.2.3 at /Users/nandub/development/code/microbit/microbit-samples/yotta_targets/bbc-microbit-classic-gcc
error: Can't pickle <function Pack.runScript at 0x10cc4d700>: it's not the same object as yotta.lib.pack.Pack.runScript

[carlosperate]

I'm also seeing the same issue on macOS with Python 3.8.2 and 3.8.3. Following the steps indicated by @nandub:

info: generate for target: bbc-microbit-classic-gcc 0.2.3 at ...
error: Can't pickle <function Pack.runScript at 0x10f07c160>: it's not the same object as yotta.lib.pack.Pack.runScript

$ pip list
Package                       Version    
----------------------------- -----------
appdirs                       1.4.4      
argcomplete                   1.11.1     
certifi                       2020.6.20  
cffi                          1.14.0     
chardet                       3.0.4      
cmsis-pack-manager            0.2.10     
colorama                      0.3.9      
cryptography                  2.9.2      
Deprecated                    1.2.10     
future                        0.18.2     
hgapi                         1.7.4      
hidapi                        0.9.0.post3
idna                          2.10       
intelhex                      2.2.1      
intervaltree                  3.0.2      
Jinja2                        2.11.2     
jsonpointer                   1.14       
jsonschema                    2.6.0      
MarkupSafe                    1.1.1      
mbed-test-wrapper             1.0.0      
milksnake                     0.1.5      
pathlib                       1.0.1      
pip                           19.2.3     
prettytable                   0.7.2      
project-generator             0.8.17     
project-generator-definitions 0.2.40     
psutil                        5.7.0      
pycparser                     2.20       
pyelftools                    0.23       
PyGithub                      1.51       
PyJWT                         1.7.1      
pylink-square                 0.6.1      
pyocd                         0.26.1     
pyOpenSSL                     19.1.0     
pyusb                         1.0.2      
PyYAML                        5.3.1      
requests                      2.24.0     
semantic-version              2.8.5      
setuptools                    41.2.0     
six                           1.15.0     
sortedcontainers              2.2.2      
urllib3                       1.25.9     
valinor                       1.1.4      
wrapt                         1.12.1     
xmltodict                     0.12.0     
yotta                         0.20.5  

Seems to be working on Ubuntu 16.04 and Windows 10 with Python 3.8.

On macOS it works fine with Python 3.6.8.

[nandub]

@carlosperate awesome, thanks for testing and confirming the error. I'm able to install and use python 3.6.8.

There must be something wrong with newer version of python, not very familiar with python's internal to try to figure this error out. If someone else knows how to debug the error, please go ahead and try to figure out.

[autopulated]

The default pickle protocol changed between python 3.6 and 3.8, so maybe something to do with that?

If you temporarily remove the @dropRootPrivs decorator from runScript , does the build fail later on a different function, or is this the only one causing a problem?

You could try changing the runScript method to make it use a new module-level function which uses the @dropRootPrivs decorator, and leave the class method undecorated.

From Python 3.9 there is support to specify the user and group to subprocess.Popen directly, so could just change the subprocess call to use that in the future, instead of the whacky dropRootPrivs decorator.

[nandub]

@autopulated thanks for the feedback. If I removed @dropRootPrivs I get a similar error but on a different function.

error: Can't pickle <function DerivedTarget.build at 0x10260b160>: it's not the same object as yotta.lib.target.DerivedTarget.build

If I do the same to build and removed @dropRootPrivs the yt build then succeeds.

[0xc0170]

@nandub Thanks for testing, would you be able to send a pull request fixing it? I assume the solution that would work not just with python 3.9.

[nandub]

@0xc0170 python is not my forte, I'd rather let someone else with more expertise on this area to come up with a pull request with the fixes.

[amurchick]

Same error MacOs 11.1

$ yt build
info: generate for target: bbc-microbit-classic-gcc 0.2.3 at /Users/user/Documents/microbit-samples/yotta_targets/bbc-microbit-classic-gcc
error: Can't pickle <function Pack.runScript at 0x10dbaeb80>: it's not the same object as yotta.lib.pack.Pack.runScript

Install as described in http://docs.yottabuild.org/#installing

brew tap ARMmbed/homebrew-formulae
brew install python cmake ninja arm-none-eabi-gcc
pip install yotta

$ python --version
Python 3.9.1

---

EDIT: After I am commented all @fsutils.dropRootPrivs in /usr/local/lib/python3.9/site-packages/yotta/lib/[pack|target].py - running yt build was done with success.