poetry icon indicating copy to clipboard operation
poetry copied to clipboard
verified publisher icon python-poetry

Fix dependency versions does not prevent duplicate dependency resolution process.

Open krlng opened this issue 3 years ago • 0 comments

  • [x] I am on the latest Poetry version.
  • [x] I have searched the issues of this repo: Related to issue 5121, but while they want a more intelligent resolution, I wonder why version pinning of dependencies does not work.
  • [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

Infos:

  • OS version and name**:
  • Poetry version**:
  • Link of a Gist with the contents of your pyproject.toml file**: https://gist.github.com/krlng/ae115db28da790860d378577c0659a66

Description: The pyproject I linked is a minimal version of our real pyorject.yaml (see below). It contains two major dependencies, that both rely on different versions of numpy depending on the python-version, platform, and chipset you are on:

pandas:

install_requires =
    numpy>=1.18.5; platform_machine!='aarch64' and platform_machine!='arm64' and python_version<'3.10'
    numpy>=1.19.2; platform_machine=='aarch64' and python_version<'3.10'
    numpy>=1.20.0; platform_machine=='arm64' and python_version<'3.10'
    numpy>=1.21.0; python_version>='3.10'

opencv-python:

  "numpy==1.13.3; python_version=='3.6' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.14.5; python_version=='3.7' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.17.3; python_version=='3.8' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.19.3; python_version<='3.9' and sys_platform == 'linux' and platform_machine == 'aarch64'",
  "numpy==1.21.0; python_version<='3.9' and sys_platform == 'darwin' and platform_machine == 'arm64'",
  "numpy==1.19.3; python_version=='3.9' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.21.2; python_version>='3.10'"

When you resolve the project (poetry lock -vvv), poetry realizes that there are 16 different possible combinations and resolves all of them.

Excerpt of the logs:


   0: Duplicate dependencies for numpy
   0: Different requirements found for numpy (>=1.21.2) with markers python_version >= "3.6" and platform_system == "Darwin" and platform_machine == "arm64", numpy (>=1.19.3) with markers python_version >= "3.6" and platform_system == "Linux" and platform_machine == "aarch64" or python_version >= "3.9", numpy (>=1.14.5) with markers python_version >= "3.7" and numpy (>=1.17.3) with markers python_version >= "3.8".
   1: Version solving took 0.902 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('opencv-python-headless', '4.6.0.66'): {'numpy': <Dependency numpy (>=1.21.2)>}}).
   1: fact: resolve-dependencies is 0.1.0
   1: derived: resolve-dependencies
   1: fact: resolve-dependencies depends on opencv-python-headless (^4.6)
   1: fact: resolve-dependencies depends on pandas (^1.3)
   1: fact: resolve-dependencies depends on numpy (=1.21.2)
   1: selecting resolve-dependencies (0.1.0)
   1: derived: numpy (==1.21.2)
   1: derived: pandas (>=1.3,<2.0)
   1: derived: opencv-python-headless (>=4.6,<5.0)
   1: selecting numpy (1.21.2)
   1: fact: opencv-python-headless (4.6.0.66) depends on numpy (>=1.21.2)
   1: selecting opencv-python-headless (4.6.0.66)
PyPI: Getting info for pandas (1.4.3) from PyPI
   0: Duplicate dependencies for numpy
   0: Different requirements found for numpy (>=1.18.5) with markers platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10", numpy (>=1.19.2) with markers platform_machine == "aarch64" and python_version < "3.10" and numpy (>=1.20.0) with markers platform_machine == "arm64" and python_version < "3.10".
   1: Version solving took 0.138 seconds.
   1: Tried 1 solutions.

This will continue for 15 more strategies, so in the end, there will be these combinations:


({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) >	}
})

It can quickly be seen, that no library has a more upper bound than numpy < 2.0, so using the newest version of numpy should just be fine. As the resolution system does not realize that on its own, I pinned my numpy to 1.21.2. However, this simply gets ignored, and the process is exactly the same, no matter if I specified this.

Why is this the case? I would suspect it is related to the different platform_machine and sys_platform markers. This is why I also tried to pin numpy for every possible combination of those, but this also did not help.

Is there any way to prevent poetry from resolving all those different strategies?

While this is more a small blemish in the posted example, it is a major issue in our real use case, as that one includes multiple packages living in a private package repository, which will interrupt the connection after a few minutes because of hitting the rate limit, which makes it practically impossible to use poetry at all.

krlng avatar Jul 29 '22 09:07 krlng