未验证 提交 157f7e35 编写于 作者: R Roman Donchenko 提交者: GitHub

Implement Python dependency pinning via pip-compile-multi (#6048)

This improves the reproducibility of the server build process. Now new
versions of dependencies can no longer break the server unless we
explicitly upgrade to them.

To minimize changes, I did not update any of the version constraints we
currently have; however, in the future, we should be able to relax a lot
of them.

Resolves #5310.
上级 a3534979
......@@ -46,16 +46,12 @@ ARG PIP_VERSION
ARG PIP_DISABLE_PIP_VERSION_CHECK=1
RUN --mount=type=cache,target=/root/.cache/pip/http \
python3 -m pip install -U pip==${PIP_VERSION}
COPY cvat/requirements/ /tmp/requirements/
COPY utils/dataset_manifest/ /tmp/dataset_manifest/
COPY cvat/requirements/ /tmp/cvat/requirements/
COPY utils/dataset_manifest/requirements.txt /tmp/utils/dataset_manifest/requirements.txt
# The server implementation depends on the dataset_manifest utility
# so we need to build its dependencies too
# https://github.com/opencv/cvat/issues/5096
RUN --mount=type=cache,target=/root/.cache/pip/http \
DATUMARO_HEADLESS=1 python3 -m pip wheel \
-r /tmp/requirements/${DJANGO_CONFIGURATION}.txt \
-r /tmp/dataset_manifest/requirements.txt \
DATUMARO_HEADLESS=1 python3 -m pip wheel --no-deps \
-r /tmp/cvat/requirements/${DJANGO_CONFIGURATION}.txt \
-w /tmp/wheelhouse
FROM ${BASE_IMAGE}
......
......@@ -23,6 +23,7 @@ RUN apt-get update && \
rm -rf /var/lib/apt/lists/*;
COPY cvat/requirements/ /tmp/cvat/requirements/
COPY utils/dataset_manifest/requirements.txt /tmp/utils/dataset_manifest/requirements.txt
RUN DATUMARO_HEADLESS=1 python3 -m pip install --no-cache-dir -r /tmp/cvat/requirements/${DJANGO_CONFIGURATION}.txt && \
python3 -m pip install --no-cache-dir coveralls
......
To regenerate the `*.txt` files in this directory, run:
DATUMARO_HEADLESS=1 pip-compile-multi -d cvat/requirements \
--backtracking --allow-unsafe --autoresolve --skip-constraints
Make sure to use the same Python version as is used in the main Dockerfile.
-r development.in
-r production.in
-r testing.in
# SHA1:06f1cef9f12e8fbb0059de4b040a5f3a4fee3469
#
# This file is autogenerated by pip-compile-multi
# To update, run:
#
# pip-compile-multi
#
-r development.txt
-r production.txt
-r staging.txt
-r testing.txt
--no-binary av
# The following packages are considered to be unsafe in a requirements file:
-r ../../utils/dataset_manifest/requirements.in
attrs==21.4.0
click==8.1.3
Django==3.2.18
django-appconf==1.0.4
django-auth-ldap==2.2.0
django-compressor==2.4
# dj-rest-auth[with_social] includes django-allauth but with version range: >=0.40.0,<0.53.0
# This does not suit us in the case when one of the previous allauth version was installed.
# Despite direct indication allauth in requirements we should keep 'with_social' for dj-rest-auth
# to avoid possible further versions conflicts (we use registration functionality)
# https://dj-rest-auth.readthedocs.io/en/latest/installation.html#registration-optional
django-allauth>=0.52.0
dj-rest-auth[with_social]==2.2.7
django-rq==2.3.2
EasyProcess==0.3
Pillow==9.3.0
numpy==1.22.0
python-ldap==3.4.3
pytz==2020.1
pyunpack==0.2.1
rcssmin==1.0.6
redis==4.5.4
rjsmin==1.1.0
requests==2.26.0
rq==1.5.2
rq-scheduler==0.10.0
sqlparse==0.4.2
django-sendfile2==0.7.0
dj-pagination==2.5.0
python-logstash-async==2.5.0
rules==2.2
GitPython==3.1.30
coreapi==2.3.3
django-filter==2.4.0
Markdown==3.2.2
djangorestframework==3.12.4
Pygments==2.7.4
drf-spectacular==0.22.1
Shapely==1.7.1
pdf2image==1.14.0
opencv-python-headless==4.5.5.62
h5py==3.6.0
django-cors-headers==3.5.0
furl==2.1.0
av==9.2.0 --no-binary=av
tensorflow==2.11.1 # Optional requirement of Datumaro. Use tensorflow-macos==2.8.0 for Mac M1
# The package is used by pyunpack as a command line tool to support multiple
# archives. Don't use as a python module because it has GPL license.
patool==1.12
diskcache==5.4.0
boto3==1.17.61
azure-storage-blob==12.13.0
google-cloud-storage==1.42.0
git+https://github.com/cvat-ai/datumaro.git@0817144ade1ddc514e182ca1835e322cb9af00a0
urllib3>=1.26.5 # not directly required, pinned by Snyk to avoid a vulnerability
natsort==8.0.0
mistune>=2.0.1 # not directly required, pinned by Snyk to avoid a vulnerability
dnspython==2.2.0
setuptools==65.5.1
django-health-check==3.17.0
psutil==5.9.4
clickhouse-connect==0.5.10
django-crum==0.7.9
wheel>=0.38.0 # not directly required, pinned by Snyk to avoid a vulnerability
psycopg2-binary==2.9.5
\ No newline at end of file
# SHA1:50365e5ee7471df51668fecdb1438bd0a1483d14
#
# This file is autogenerated by pip-compile-multi
# To update, run:
#
# pip-compile-multi
#
-r ../../utils/dataset_manifest/requirements.txt
--no-binary av
absl-py==1.4.0
# via
# tensorboard
# tensorflow
asgiref==3.6.0
# via django
astunparse==1.6.3
# via tensorflow
async-timeout==4.0.2
# via redis
attrs==21.4.0
# via
# -r cvat/requirements/base.in
# datumaro
# jsonschema
azure-core==1.26.4
# via
# azure-storage-blob
# msrest
azure-storage-blob==12.13.0
# via -r cvat/requirements/base.in
boto3==1.17.61
# via -r cvat/requirements/base.in
botocore==1.20.112
# via
# boto3
# s3transfer
cachetools==5.3.0
# via google-auth
certifi==2022.12.7
# via
# clickhouse-connect
# msrest
# requests
cffi==1.15.1
# via cryptography
charset-normalizer==2.0.12
# via requests
click==8.1.3
Django==3.2.18
# via
# -r cvat/requirements/base.in
# rq
clickhouse-connect==0.5.10
# via -r cvat/requirements/base.in
contourpy==1.0.7
# via matplotlib
coreapi==2.3.3
# via -r cvat/requirements/base.in
coreschema==0.0.4
# via coreapi
croniter==1.3.14
# via rq-scheduler
cryptography==40.0.2
# via
# azure-storage-blob
# pyjwt
cycler==0.11.0
# via matplotlib
datumaro @ git+https://github.com/cvat-ai/datumaro.git@0817144ade1ddc514e182ca1835e322cb9af00a0
# via -r cvat/requirements/base.in
defusedxml==0.7.1
# via
# datumaro
# python3-openid
deprecated==1.2.13
# via limits
diskcache==5.4.0
# via -r cvat/requirements/base.in
dj-pagination==2.5.0
# via -r cvat/requirements/base.in
dj-rest-auth[with_social]==2.2.7
# via -r cvat/requirements/base.in
django==3.2.18
# via
# -r cvat/requirements/base.in
# dj-rest-auth
# django-allauth
# django-appconf
# django-auth-ldap
# django-cors-headers
# django-crum
# django-filter
# django-health-check
# django-rq
# django-sendfile2
# djangorestframework
# drf-spectacular
django-allauth==0.52.0
# via
# -r cvat/requirements/base.in
# dj-rest-auth
django-appconf==1.0.4
# via
# -r cvat/requirements/base.in
# django-compressor
django-auth-ldap==2.2.0
# via -r cvat/requirements/base.in
django-compressor==2.4
# dj-rest-auth[with_social] includes django-allauth but with version range: >=0.40.0,<0.53.0
# This does not suit us in the case when one of the previous allauth version was installed.
# Despite direct indication allauth in requirements we should keep 'with_social' for dj-rest-auth
# to avoid possible further versions conflicts (we use registration functionality)
# https://dj-rest-auth.readthedocs.io/en/latest/installation.html#registration-optional
django-allauth>=0.52.0
dj-rest-auth[with_social]==2.2.7
# via -r cvat/requirements/base.in
django-cors-headers==3.5.0
# via -r cvat/requirements/base.in
django-crum==0.7.9
# via -r cvat/requirements/base.in
django-filter==2.4.0
# via -r cvat/requirements/base.in
django-health-check==3.17.0
# via -r cvat/requirements/base.in
django-rq==2.3.2
EasyProcess==0.3
Pillow==9.3.0
numpy==1.22.0
# via -r cvat/requirements/base.in
django-sendfile2==0.7.0
# via -r cvat/requirements/base.in
djangorestframework==3.12.4
# via
# -r cvat/requirements/base.in
# dj-rest-auth
# drf-spectacular
dnspython==2.2.0
# via -r cvat/requirements/base.in
drf-spectacular==0.22.1
# via -r cvat/requirements/base.in
easyprocess==0.3
# via
# -r cvat/requirements/base.in
# pyunpack
entrypoint2==1.1
# via pyunpack
flatbuffers==23.3.3
# via tensorflow
fonttools==4.39.3
# via matplotlib
furl==2.1.0
# via -r cvat/requirements/base.in
gast==0.4.0
# via tensorflow
gitdb==4.0.10
# via gitpython
gitpython==3.1.30
# via -r cvat/requirements/base.in
google-api-core==2.11.0
# via
# google-cloud-core
# google-cloud-storage
google-auth==2.17.3
# via
# google-api-core
# google-auth-oauthlib
# google-cloud-core
# google-cloud-storage
# tensorboard
google-auth-oauthlib==0.4.6
# via tensorboard
google-cloud-core==2.3.2
# via google-cloud-storage
google-cloud-storage==1.42.0
# via -r cvat/requirements/base.in
google-crc32c==1.5.0
# via google-resumable-media
google-pasta==0.2.0
# via tensorflow
google-resumable-media==2.4.1
# via google-cloud-storage
googleapis-common-protos==1.59.0
# via google-api-core
grpcio==1.54.0
# via
# tensorboard
# tensorflow
h5py==3.6.0
# via
# -r cvat/requirements/base.in
# datumaro
# tensorflow
idna==3.4
# via requests
importlib-resources==5.12.0
# via limits
inflection==0.5.1
# via drf-spectacular
isodate==0.6.1
# via msrest
itypes==1.2.0
# via coreapi
jinja2==3.1.2
# via coreschema
jmespath==0.10.0
# via
# boto3
# botocore
jsonschema==4.17.3
# via drf-spectacular
keras==2.11.0
# via tensorflow
kiwisolver==1.4.4
# via matplotlib
libclang==16.0.0
# via tensorflow
limits==3.4.0
# via python-logstash-async
lxml==4.9.2
# via datumaro
lz4==4.3.2
# via clickhouse-connect
markdown==3.2.2
# via
# -r cvat/requirements/base.in
# tensorboard
markupsafe==2.1.2
# via
# jinja2
# werkzeug
matplotlib==3.7.1
# via
# datumaro
# pycocotools
mistune==2.0.5
# via -r cvat/requirements/base.in
msrest==0.7.1
# via azure-storage-blob
networkx==3.1
# via datumaro
nibabel==5.1.0
# via datumaro
oauthlib==3.2.2
# via requests-oauthlib
opt-einsum==3.3.0
# via tensorflow
orderedmultidict==1.0.1
# via furl
orjson==3.8.10
# via datumaro
packaging==23.1
# via
# limits
# matplotlib
# nibabel
# tensorboardx
# tensorflow
pandas==2.0.0
# via datumaro
patool==1.12
# via -r cvat/requirements/base.in
pdf2image==1.14.0
# via -r cvat/requirements/base.in
protobuf==3.19.6
# via
# google-api-core
# googleapis-common-protos
# tensorboard
# tensorboardx
# tensorflow
psutil==5.9.4
# via -r cvat/requirements/base.in
psycopg2-binary==2.9.5
# via -r cvat/requirements/base.in
pyasn1==0.4.8
# via
# pyasn1-modules
# python-ldap
# rsa
pyasn1-modules==0.2.8
# via
# google-auth
# python-ldap
pycocotools==2.0.6
# via datumaro
pycparser==2.21
# via cffi
pygments==2.7.4
# via -r cvat/requirements/base.in
pyjwt[crypto]==2.6.0
# via django-allauth
pylogbeat==2.0.0
# via python-logstash-async
pyparsing==3.0.9
# via matplotlib
pyrsistent==0.19.3
# via jsonschema
python-dateutil==2.8.2
# via
# botocore
# croniter
# matplotlib
# pandas
python-ldap==3.4.3
# via
# -r cvat/requirements/base.in
# django-auth-ldap
python-logstash-async==2.5.0
# via -r cvat/requirements/base.in
python3-openid==3.2.0
# via django-allauth
pytz==2020.1
# via
# -r cvat/requirements/base.in
# clickhouse-connect
# django
# pandas
pyunpack==0.2.1
# via -r cvat/requirements/base.in
pyyaml==6.0
# via
# datumaro
# drf-spectacular
rcssmin==1.0.6
# via
# -r cvat/requirements/base.in
# django-compressor
redis==4.5.4
rjsmin==1.1.0
# via
# -r cvat/requirements/base.in
# django-rq
# rq
requests==2.26.0
# via
# -r cvat/requirements/base.in
# azure-core
# coreapi
# datumaro
# django-allauth
# google-api-core
# google-cloud-storage
# msrest
# python-logstash-async
# requests-oauthlib
# tensorboard
requests-oauthlib==1.3.1
# via
# django-allauth
# google-auth-oauthlib
# msrest
rjsmin==1.1.0
# via
# -r cvat/requirements/base.in
# django-compressor
rq==1.5.2
# via
# -r cvat/requirements/base.in
# django-rq
# rq-scheduler
rq-scheduler==0.10.0
sqlparse==0.4.2
django-sendfile2==0.7.0
dj-pagination==2.5.0
python-logstash-async==2.5.0
# via -r cvat/requirements/base.in
rsa==4.9
# via google-auth
ruamel-yaml==0.17.21
# via datumaro
ruamel-yaml-clib==0.2.7
# via ruamel-yaml
rules==2.2
GitPython==3.1.30
coreapi==2.3.3
django-filter==2.4.0
Markdown==3.2.2
djangorestframework==3.12.4
Pygments==2.7.4
drf-spectacular==0.22.1
Shapely==1.7.1
pdf2image==1.14.0
opencv-python-headless==4.5.5.62
h5py==3.6.0
django-cors-headers==3.5.0
furl==2.1.0
av==9.2.0 --no-binary=av
tensorflow==2.11.1 # Optional requirement of Datumaro. Use tensorflow-macos==2.8.0 for Mac M1
# The package is used by pyunpack as a command line tool to support multiple
# archives. Don't use as a python module because it has GPL license.
patool==1.12
diskcache==5.4.0
boto3==1.17.61
azure-storage-blob==12.13.0
google-cloud-storage==1.42.0
git+https://github.com/cvat-ai/datumaro.git@0817144ade1ddc514e182ca1835e322cb9af00a0
urllib3>=1.26.5 # not directly required, pinned by Snyk to avoid a vulnerability
natsort==8.0.0
mistune>=2.0.1 # not directly required, pinned by Snyk to avoid a vulnerability
dnspython==2.2.0
# via -r cvat/requirements/base.in
s3transfer==0.4.2
# via boto3
scipy==1.10.1
# via datumaro
shapely==1.7.1
# via -r cvat/requirements/base.in
six==1.16.0
# via
# astunparse
# azure-core
# django-compressor
# furl
# google-auth
# google-pasta
# isodate
# orderedmultidict
# python-dateutil
# tensorflow
smmap==5.0.0
# via gitdb
sqlparse==0.4.2
# via
# -r cvat/requirements/base.in
# django
tensorboard==2.11.2
# via tensorflow
tensorboard-data-server==0.6.1
# via tensorboard
tensorboard-plugin-wit==1.8.1
# via tensorboard
tensorboardx==2.6
# via datumaro
tensorflow==2.11.1
# via -r cvat/requirements/base.in
tensorflow-estimator==2.11.0
# via tensorflow
tensorflow-io-gcs-filesystem==0.32.0
# via tensorflow
termcolor==2.2.0
# via tensorflow
typing-extensions==4.5.0
# via
# azure-core
# datumaro
# limits
# tensorflow
tzdata==2023.3
# via pandas
uritemplate==4.1.1
# via
# coreapi
# drf-spectacular
urllib3==1.26.15
# via
# -r cvat/requirements/base.in
# botocore
# clickhouse-connect
# requests
werkzeug==2.2.3
# via tensorboard
wheel==0.40.0
# via
# -r cvat/requirements/base.in
# astunparse
# tensorboard
wrapt==1.15.0
# via
# deprecated
# tensorflow
zstandard==0.21.0
# via clickhouse-connect
# The following packages are considered to be unsafe in a requirements file:
setuptools==65.5.1
django-health-check==3.17.0
psutil==5.9.4
clickhouse-connect==0.5.10
django-crum==0.7.9
wheel>=0.38.0 # not directly required, pinned by Snyk to avoid a vulnerability
psycopg2-binary==2.9.5
\ No newline at end of file
# via
# -r cvat/requirements/base.in
# limits
# tensorboard
# tensorflow
-r base.in
pylint==2.14.5
pylint-django==2.5.3
pylint-plugin-utils==0.7
rope==0.17.0
django-extensions==3.0.8
snakeviz==2.1.0
django-silk==5.0.3
\ No newline at end of file
# SHA1:4ea6010ac7e1df247f335663badaef30aabd8a11
#
# This file is autogenerated by pip-compile-multi
# To update, run:
#
# pip-compile-multi
#
-r base.txt
--no-binary av
astroid==2.11.7
# via pylint
autopep8==2.0.2
# via django-silk
dill==0.3.6
# via pylint
django-extensions==3.0.8
# via -r cvat/requirements/development.in
django-silk==5.0.3
# via -r cvat/requirements/development.in
gprof2dot==2022.7.29
# via django-silk
isort==5.12.0
# via pylint
lazy-object-proxy==1.9.0
# via astroid
mccabe==0.7.0
# via pylint
platformdirs==3.2.0
# via pylint
pycodestyle==2.10.0
# via autopep8
pylint==2.14.5
# via
# -r cvat/requirements/development.in
# pylint-django
# pylint-plugin-utils
pylint-django==2.5.3
# via -r cvat/requirements/development.in
pylint-plugin-utils==0.7
# via
# -r cvat/requirements/development.in
# pylint-django
rope==0.17.0
django-extensions==3.0.8
# via -r cvat/requirements/development.in
snakeviz==2.1.0
django-silk==5.0.3
\ No newline at end of file
# via -r cvat/requirements/development.in
tomli==2.0.1
# via
# autopep8
# pylint
tomlkit==0.11.7
# via pylint
tornado==6.3
# via snakeviz
# The following packages are considered to be unsafe in a requirements file:
-r base.in
mod-wsgi==4.9.4
# SHA1:2ed11382b8b8c472f9271bd4bd0701d52103ebd2
#
# This file is autogenerated by pip-compile-multi
# To update, run:
#
# pip-compile-multi
#
-r base.txt
--no-binary av
mod-wsgi==4.9.4
# via -r cvat/requirements/production.in
# The following packages are considered to be unsafe in a requirements file:
-r development.in
coveralls
fakeredis==2.10.3
# SHA1:8dec2a1cde7e68bd79ee31cdec270084f7e3266b
#
# This file is autogenerated by pip-compile-multi
# To update, run:
#
# pip-compile-multi
#
-r development.txt
coveralls
--no-binary av
coverage==6.5.0
# via coveralls
coveralls==3.3.1
# via -r cvat/requirements/testing.in
docopt==0.6.2
# via coveralls
fakeredis==2.10.3
# via -r cvat/requirements/testing.in
sortedcontainers==2.4.0
# via fakeredis
# The following packages are considered to be unsafe in a requirements file:
......@@ -81,12 +81,19 @@ description: 'Installing a development environment for different operating syste
pip install -U pip wheel setuptools
pip install \
-r cvat/requirements/development.txt \
-r cvat/requirements/production.txt \
-r utils/dataset_manifest/requirements.txt
-r cvat/requirements/production.txt
python manage.py migrate
python manage.py collectstatic
```
Note that the `.txt` files in the `cvat/requirements` directory
have pinned dependencies intended for the main target OS/Python version
(the one used in the main Dockerfile).
If you're unable to install those dependency versions,
you can substitute the corresponding `.in` files instead.
That way, you're more likely to be able to install the dependencies,
but their versions might not correspond to those used in production.
> Note for Mac users
>
> If you have any problems with installing dependencies from
......
......@@ -98,7 +98,7 @@ Create an environment and install the necessary python modules:
python3 -m venv .env
. .env/bin/activate
pip install -U pip
pip install -r utils/dataset_manifest/requirements.txt
pip install -r utils/dataset_manifest/requirements.in
```
> Please note that if used with video this way, the results may be different from what
......
av==9.2.0 --no-binary=av # Pinned for the whole CVAT
opencv-python-headless>=4.4.0.42
Pillow==9.3.0
tqdm>=4.58.0
natsort>=8.0.0
av==9.2.0 --no-binary=av # Pinned for the whole CVAT
opencv-python-headless>=4.4.0.42
Pillow==9.3.0
tqdm>=4.58.0
natsort>=8.0.0
# SHA1:172275fac2f1fa6324a38130619bb7994d5aa413
#
# This file is autogenerated by pip-compile-multi
# To update, run:
#
# pip-compile-multi
#
--no-binary av
av==9.2.0
# via -r utils/dataset_manifest/requirements.in
natsort==8.0.0
# via -r utils/dataset_manifest/requirements.in
numpy==1.22.0
# via opencv-python-headless
opencv-python-headless==4.5.5.62
# via -r utils/dataset_manifest/requirements.in
pillow==9.3.0
# via -r utils/dataset_manifest/requirements.in
tqdm==4.65.0
# via -r utils/dataset_manifest/requirements.in
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册