Environment markers in dependencies should be merged, not overriden by each other

[Jamim]

Issue description

Pipenv.lock file might contain different markers after pipenv lock.

This issue is quite annoying when you deal with a Pipfile that contains a lot of requirements especially when most of them are outdated.

Expected result

Markers remain unchanged after pipenv lock if nothing has been changed in Pipfile and on the PyPI.

Actual result

Markers change unpredictably.

Steps to replicate

You can use this script to reproduce the bug.
Please see the asciicast for the details.

Environment

I experience this issue

• on latest Gentoo with latest pipenv from PyPI and from master of this repo
• on Ubuntu 16.04 with latest pipenv from PyPI

$ pipenv --support # Gentoo

Pipenv version: '2018.7.1.dev0'

Pipenv location: '/home/mim/src/pipenv/pipenv'

Python location: '/usr/bin/python3.7'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 3.7: /usr/bin/python3.7m

• 3.7: /usr/bin/python3.7

• 3.7.0: /usr/bin/python

• 2.7.15: /usr/bin/python2

• 3.7.0: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.17.10-gentoo',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Jul 25 23:37:33 +03 2018',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'linux'}

System environment variables:

• LC_ALL
• LS_COLORS
• STY
• ANDROID_HOME
• XDG_MENU_PREFIX
• LANG
• LESS
• DISPLAY
• OPENGL_PROFILE
• CONFIG_PROTECT_MASK
• EDITOR
• GPG_TTY
• COLORTERM
• ASCIINEMA_REC
• JAVA_HOME
• GCC_SPECS
• SSH_AUTH_SOCK
• GLADE_CATALOG_PATH
• VBOX_APP_HOME
• HUSHLOGIN
• USER
• GLADE_MODULE_PATH
• PAGER
• DESKTOP_SESSION
• PWD
• HOME
• MANPAGER
• WINDOW
• SSH_AGENT_PID
• GSETTINGS_BACKEND
• XDG_DATA_DIRS
• JDK_HOME
• GLADE_PIXMAP_PATH
• GTK_MODULES
• MAIL
• WINDOWPATH
• CONFIG_PROTECT
• SHELL
• TERM
• VTE_VERSION
• JAVAC
• XDG_CURRENT_DESKTOP
• OPENCL_PROFILE
• ANDROID_SWT
• MOZ_GMP_PATH
• SHLVL
• LANGUAGE
• MANPATH
• WINDOWID
• LOGNAME
• DBUS_SESSION_BUS_ADDRESS
• XSESSION
• XAUTHORITY
• ANT_HOME
• XDG_CONFIG_DIRS
• PATH
• INFOPATH
• TERMCAP
• SESSION_MANAGER
• LESSOPEN
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/mim/.local/bin:/home/mim/.local/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/8.1.0:/usr/lib/llvm/6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/home/mim/.npm-global/bin:/home/mim/.npm-global/bin:
• SHELL: /bin/bash
• EDITOR: /bin/nano
• LANG: ru_RU.utf8
• PWD: /home/mim

---

$ pipenv --support # Ubuntu 16.04

Pipenv version: '2018.7.1'

Pipenv location: '/home/mim/.local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python3.6'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.5: /usr/bin/python3.5m

• 3.5: /usr/bin/python3.5

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 2.7.12: /usr/bin/python

• 2.7.12: /usr/bin/python2

• 3.5.2: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.13.0-45-generic',
 'platform_system': 'Linux',
 'platform_version': '#50~16.04.1-Ubuntu SMP Wed May 30 11:18:27 UTC 2018',
 'python_full_version': '3.6.6',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• XDG_VTNR
• LC_PAPER
• XDG_SESSION_ID
• LC_ADDRESS
• XDG_GREETER_DATA_DIR
• LC_MONETARY
• CLUTTER_IM_MODULE
• SESSION
• GPG_AGENT_INFO
• XDG_MENU_PREFIX
• TERM
• SHELL
• QT_LINUX_ACCESSIBILITY_ALWAYS_ON
• WINDOWID
• LC_NUMERIC
• UPSTART_SESSION
• GNOME_KEYRING_CONTROL
• GTK_MODULES
• USER
• LS_COLORS
• QT_ACCESSIBILITY
• LC_TELEPHONE
• XDG_SESSION_PATH
• UNITY_HAS_3D_SUPPORT
• XDG_SEAT_PATH
• SSH_AUTH_SOCK
• SESSION_MANAGER
• DEFAULTS_PATH
• XDG_CONFIG_DIRS
• UNITY_DEFAULT_PROFILE
• PATH
• DESKTOP_SESSION
• QT_QPA_PLATFORMTHEME
• QT_IM_MODULE
• LC_IDENTIFICATION
• __fish_sysconfdir
• XDG_SESSION_TYPE
• PWD
• JOB
• XMODIFIERS
• __fish_datadir
• LANG
• GNOME_KEYRING_PID
• MANDATORY_PATH
• GDM_LANG
• LC_MEASUREMENT
• IM_CONFIG_PHASE
• COMPIZ_CONFIG_PROFILE
• PAPERSIZE
• GDMSESSION
• SESSIONTYPE
• GTK2_MODULES
• XDG_SEAT
• SHLVL
• HOME
• LANGUAGE
• GNOME_DESKTOP_SESSION_ID
• UPSTART_INSTANCE
• __fish_help_dir
• XDG_SESSION_DESKTOP
• UPSTART_EVENTS
• LOGNAME
• COMPIZ_BIN_PATH
• XDG_DATA_DIRS
• QT4_IM_MODULE
• DBUS_SESSION_BUS_ADDRESS
• LESSOPEN
• UPSTART_JOB
• INSTANCE
• XDG_RUNTIME_DIR
• DISPLAY
• XDG_CURRENT_DESKTOP
• GTK_IM_MODULE
• LESSCLOSE
• LC_TIME
• __fish_bin_dir
• XAUTHORITY
• LC_NAME
• COLORTERM
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/mim/bin:/home/mim/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/mim/.npm-global/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /home/mim

---

[nikolay-saskovets]

I've also faced with the same issue:

I've tried the script from @Jamim and here are my results:

diff --git a/Pipfile.lock b/Pipfile.lock
index d77c7e4..c306df2 100644
--- a/Pipfile.lock
+++ b/Pipfile.lock
@@ -80,7 +80,7 @@
                 "sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca",
                 "sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50"
             ],
-            "markers": "python_version < '3.0'",
+            "markers": "python_version < '3.3'",
             "version": "==1.0.2"
         },
         "mock": {

Something was changed on attempt 5

my env on Ubuntu 18.04:

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/home/develop/.local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python3'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 2.7.15: /usr/bin/python

• 2.7.15: /usr/bin/python2

• 3.6.5: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-29-generic',
 'platform_system': 'Linux',
 'platform_version': '#31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• CLUTTER_IM_MODULE
• NVM_DIR
• LS_COLORS
• LC_MEASUREMENT
• LESSCLOSE
• LC_PAPER
• LC_MONETARY
• LANG
• GDM_LANG
• HISTCONTROL
• DISPLAY
• HISTTIMEFORMAT
• QT_STYLE_OVERRIDE
• WORKON_HOME
• EDITOR
• COLORTERM
• XDG_VTNR
• SSH_AUTH_SOCK
• MANDATORY_PATH
• LC_NAME
• XDG_SESSION_ID
• XDG_GREETER_DATA_DIR
• USER
• DESKTOP_SESSION
• QT4_IM_MODULE
• DEFAULTS_PATH
• PWD
• HOME
• SSH_AGENT_PID
• XDG_SESSION_TYPE
• XDG_DATA_DIRS
• XDG_SESSION_DESKTOP
• LC_ADDRESS
• LC_NUMERIC
• VISUAL
• TERM
• SHELL
• XDG_SEAT_PATH
• XMODIFIERS
• GPG_AGENT_INFO
• COLORFGBG
• XDG_SEAT
• SHLVL
• LANGUAGE
• WINDOWID
• LC_TELEPHONE
• GDMSESSION
• LOGNAME
• DBUS_SESSION_BUS_ADDRESS
• XDG_RUNTIME_DIR
• HOSTFILE
• XAUTHORITY
• XDG_SESSION_PATH
• XDG_CONFIG_DIRS
• PATH
• LC_IDENTIFICATION
• PS1
• HISTSIZE
• HISTFILESIZE
• LESSOPEN
• GTK_IM_MODULE
• LC_TIME
• OLDPWD
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/develop/bin/my:/home/develop/bin:/home/develop/bin/my:/home/develop/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/root/bin/:/usr/local/rvm/bin/
• SHELL: /bin/bash
• EDITOR: vim
• LANG: en_US.UTF-8
• PWD: /home/develop/tmp

---

[uranusjr]

@nikolay-saskovets What is the package that produces the different output? This information would save us much time to investigate. Without looking into this too deeply, it is likely due to how a certain package is distributed in an undeterministic way, but it would be much easier to explain this with a concrete example.

[nikolay-saskovets]

@uranusjr this output is about installing those two packages pytest-cov pytest-vcr. And markers are changes for one of their dependency — the package funcsigs:

develop:$ git diff -U5
diff --git a/Pipfile.lock b/Pipfile.lock
index d77c7e4..c306df2 100644
--- a/Pipfile.lock
+++ b/Pipfile.lock
@@ -78,11 +78,11 @@
         "funcsigs": {
             "hashes": [
                 "sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca",
                 "sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50"
             ],
-            "markers": "python_version < '3.0'",
+            "markers": "python_version < '3.3'",
             "version": "==1.0.2"
         },

But it seems like such issues may caught with some other packages from PyPi. We are faced with such problem in a real project with other packages, too.

I think @Jamim may provide more info as «topic starter» :-)

[uranusjr]

Thanks! I think I have an idea of what’s going on, but I’ll need some time to investigate it further to make sure my guess is correct. It would be super helpful if more details could be provideded in the meantime.

What would be helpful are:

• Output of pipenv lock --verbose
• Full Pipfile.lock results
• Whether pipenv lock --lock --verbose changes the output (and if it does, what exactly changed)
• pipenv graph and pipenv run pip list results on both environments

[Jamim]

Thank you, @nikolay-saskovets!

Hi @uranusjr,

The simplest way to reproduce the issue is to run the script from Steps to replicate.
It installs pytest-cov and pytest-vcr and locks the dependencies in a loop until the Pipfile.lock has changed. I specially made this example using as less requirements as possible.

In the example the issue affects funcsigs==1.0.2 only, but in the real project it affects:

• backports.functools-lru-cache==1.5
• configparser==3.5.0
• contextlib2==0.5.5
• funcsigs==1.0.2

So every time when I run pipenv lock it changes some markers :-/

There are a lot of pipenv lock --verbose at the asciicast. I hope it should be really helpful.

[Jamim]

@uranusjr You can found Pipfile and Pipfile.lock in this repository.
It's the repository that was created during the asciicast recording.

[Jamim]

pipenv graph:

pytest-cov==2.5.1
  - coverage [required: >=3.7.1, installed: 4.5.1]
  - pytest [required: >=2.6.0, installed: 3.6.3]
    - atomicwrites [required: >=1.0, installed: 1.1.5]
    - attrs [required: >=17.4.0, installed: 18.1.0]
    - funcsigs [required: Any, installed: 1.0.2]
    - more-itertools [required: >=4.0.0, installed: 4.2.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - pluggy [required: >=0.5,<0.7, installed: 0.6.0]
    - py [required: >=1.5.0, installed: 1.5.4]
    - setuptools [required: Any, installed: 40.0.0]
    - six [required: >=1.10.0, installed: 1.11.0]
pytest-vcr==0.3.0
  - pytest [required: >=3.0.0, installed: 3.6.3]
    - atomicwrites [required: >=1.0, installed: 1.1.5]
    - attrs [required: >=17.4.0, installed: 18.1.0]
    - funcsigs [required: Any, installed: 1.0.2]
    - more-itertools [required: >=4.0.0, installed: 4.2.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - pluggy [required: >=0.5,<0.7, installed: 0.6.0]
    - py [required: >=1.5.0, installed: 1.5.4]
    - setuptools [required: Any, installed: 40.0.0]
    - six [required: >=1.10.0, installed: 1.11.0]
  - vcrpy [required: Any, installed: 1.13.0]
    - contextlib2 [required: Any, installed: 0.5.5]
    - mock [required: Any, installed: 2.0.0]
      - funcsigs [required: >=1, installed: 1.0.2]
      - pbr [required: >=0.11, installed: 4.2.0]
      - six [required: >=1.9, installed: 1.11.0]
    - PyYAML [required: Any, installed: 3.13]
    - six [required: >=1.5, installed: 1.11.0]
    - wrapt [required: Any, installed: 1.10.11]

pipenv run pip list:

Package        Version
-------------- -------
atomicwrites   1.1.5  
attrs          18.1.0 
contextlib2    0.5.5  
coverage       4.5.1  
funcsigs       1.0.2  
mock           2.0.0  
more-itertools 4.2.0  
pbr            4.2.0  
pip            18.0   
pluggy         0.6.0  
py             1.5.4  
pytest         3.6.3  
pytest-cov     2.5.1  
pytest-vcr     0.3.0  
PyYAML         3.13   
setuptools     40.0.0 
six            1.11.0 
vcrpy          1.13.0 
wheel          0.31.1 
wrapt          1.10.11

[uranusjr]

Quick update: I think the problem is a race condition, for funcsig it’s between mock and pytest. They are picking up different conditions from each source. They should be instead consolidated into one instead (ideally python_version < '3.3', but I don’t think we have that level of merker parsing ability, and would need to settle with python_version <'3.0' or python_version < '3.3').

[techalchemy]

is this only affecting python_version? This is not straightforward and also it's not really a bug, they are functionally identical which is why it doesn't matter. I get that it is an annoyance, but it is not a resolution bug, it is a bug with having to ignore lockfile changes sometimes