Merge branch 'feature/dockerfile' into 'master'

tools: add Dockerfile

Closes IDFGH-1304 and IDF-590

See merge request espressif/esp-idf!5432

docs/en/api-guides/index.rst

@@ -34,3 +34,4 @@ API Guides
    BluFi <blufi>
    External SPI-connected RAM <external-ram>
    Linker Script Generation <linker-script-generation>
+   Tools <tools/index>

docs/en/api-guides/tools/idf-docker-image.rst

@@ -0,0 +1,96 @@
+****************
+IDF Docker Image
+****************
+
+.. highlight:: bash
+
+IDF Docker image (``espressif/idf``) is intended for building applications and libraries with specific versions of ESP-IDF, when doing automated builds.
+
+The image contains:
+
+- Common utilities such as git, wget, curl, zip.
+- Python 3.6 or newer.
+- A copy of a specific version of ESP-IDF (see below for information about versions). ``IDF_PATH`` environment variable is set, and points to ESP-IDF location in the container.
+- All the build tools required for the specific version of ESP-IDF: CMake, make, ninja, cross-compiler toolchains, etc.
+- All Python packages required by ESP-IDF are installed in a virtual environment.
+
+The image entrypoint sets up ``PATH`` environment variable to point to the correct version of tools, and activates the Python virtual environment. As a result, the environment is ready to use the ESP-IDF build system.
+
+The image can also be used as a base for custom images, if additional utilities are required.
+
+Tags
+====
+
+Multiple tags of this image are maintained:
+
+- ``latest``: tracks ``master`` branch of ESP-IDF
+- ``vX.Y``: corresponds to ESP-IDF release ``vX.Y``
+- ``release-vX.Y``: tracks ``release/vX.Y`` branch of ESP-IDF
+
+.. note::
+
+    Versions of ESP-IDF released before this feature was introduced do not have corresponding Docker image versions. You can check the up-to-date list of available tags at https://hub.docker.com/r/espressif/idf/tags.
+
+Usage
+=====
+
+Setting up Docker
+~~~~~~~~~~~~~~~~~
+
+Before using the ``espressif/idf`` Docker image locally, make sure you have Docker installed. Follow the instructions at https://docs.docker.com/install/, if it is not installed yet.
+
+If using the image in CI environment, consult the documentation of your CI service on how to specify the image used for the build process.
+
+Building a project with CMake
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the project directory, run::
+
+    docker run --rm -v $PWD:/project -w /project espressif/idf idf.py build
+
+
+The above command explained:
+
+- ``docker run``: runs a Docker image. It is a shorter form of the command ``docker container run``.
+- ``--rm``: removes the container when the build is finished
+- ``-v $PWD:/project``: mounts the current directory on the host (``$PWD``) as ``/project`` directory in the container
+- ``espressif/idf``: uses Docker image ``espressif/idf`` with tag ``latest`` (implicitly added by Docker when no tag is specified)
+- ``idf.py build``: runs this command inside the container
+
+To build with a specific docker image tag, specify it as ``espressif/idf:TAG``::
+
+    docker run --rm -v $PWD:/project -w /project espressif/idf:v4.0 idf.py build
+
+.. note::
+
+    At the time of writing, v4.0 release of ESP-IDF does not exist, yet, so the above command will not work. You can check the up-to-date list of available tags at https://hub.docker.com/r/espressif/idf/tags.
+
+
+Building a project with GNU Make
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Same as for CMake, except that the build command is different::
+
+    docker run --rm -v $PWD:/project -w /project espressif/idf make defconfig all -j4
+
+
+.. note::
+
+    If the ``sdkconfig`` file does not exist, the default behavior of GNU Make build system is to open the menuconfig UI. This may be not desired in automated build environments. To ensure that the ``sdkconfig`` file exists, ``defconfig`` target is added before ``all``.
+
+Using the image interactively
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It is also possible to do builds interactively, to debug build issues or test the automated build scripts. Start the container with `-i -t` flags::
+
+    docker run --rm -v $PWD:/project -w /project -it espressif/idf
+
+
+Then inside the container, use ``idf.py`` as usual::
+
+    idf.py menuconfig
+    idf.py build
+
+.. note::
+
+    Commands which communicate with the development board, such as ``idf.py flash`` and ``idf.py monitor`` will not work in the container unless the serial port is passed through into the container. However currently this is not possible with Docker for Windows (https://github.com/docker/for-win/issues/1018) and Docker for Mac (https://github.com/docker/for-mac/issues/900).

docs/en/api-guides/tools/index.rst

@@ -0,0 +1,8 @@
+Tools
+*****
+
+.. toctree::
+   :maxdepth: 1
+
+   IDF Monitor <idf-monitor>
+   IDF Docker image <idf-docker-image>

docs/zh_CN/api-guides/index.rst

@@ -34,3 +34,4 @@ API 指南
    BluFi <blufi>
    External SPI-connected RAM <external-ram>
    链接脚本生成机制 <linker-script-generation>
+   Tools <tools/index>

docs/zh_CN/api-guides/tools/idf-docker-image.rst

@@ -0,0 +1 @@
+.. include:: ../../../en/api-guides/tools/idf-docker-image.rst

docs/zh_CN/api-guides/tools/index.rst

@@ -0,0 +1,8 @@
+工具
+*****
+
+.. toctree::
+   :maxdepth: 1
+
+   IDF 监视器 <idf-monitor>
+   IDF Docker image <idf-docker-image>

tools/ci/config/build.yml

@@ -259,3 +259,26 @@ test_build_system_cmake:
     - mkdir test_build_system
     - cd test_build_system
     - ${IDF_PATH}/tools/ci/test_build_system_cmake.sh
+
+build_docker:
+  stage: build
+  image: espressif/docker-builder:1
+  tags:
+    - build_docker_amd64_brno
+  only:
+    refs:
+      - master
+      - /^release\/v/
+      - /^v\d+\.\d+(\.\d+)?($|-)/
+      - schedules
+  variables:
+    DOCKER_TMP_IMAGE_NAME: "idf_tmp_image"
+  before_script: []
+  script:
+    - export DOCKER_BUILD_ARGS="--build-arg IDF_CLONE_URL=${CI_REPOSITORY_URL} --build-arg IDF_CLONE_BRANCH_OR_TAG=${CI_COMMIT_REF_NAME} --build-arg IDF_CHECKOUT_REF=${CI_COMMIT_TAG:-$CI_COMMIT_SHA}"
+    # Build
+    - docker build --tag ${DOCKER_TMP_IMAGE_NAME} ${DOCKER_BUILD_ARGS} tools/docker/
+    # We can't mount $PWD/examples/get-started/blink into the container, see https://gitlab.com/gitlab-org/gitlab-ce/issues/41227.
+    # The workaround mentioned there works, but leaves around directories which need to be cleaned up manually.
+    # Therefore, build a copy of the example located inside the container.
+    - docker run --rm --workdir /opt/esp/idf/examples/get-started/blink ${DOCKER_TMP_IMAGE_NAME} idf.py build

tools/ci/executable-list.txt

@@ -49,6 +49,7 @@ tools/ci/test_build_system_cmake.sh
 tools/ci/test_configure_ci_environment.sh
 tools/cmake/convert_to_cmake.py
 tools/cmake/run_cmake_lint.sh
+tools/docker/entrypoint.sh
 tools/elf_to_ld.sh
 tools/esp_app_trace/logtrace_proc.py
 tools/esp_app_trace/sysviewtrace_proc.py

tools/docker/Dockerfile

@@ -0,0 +1,62 @@
+FROM ubuntu:18.04
+
+ARG DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && apt-get install -y \
+    apt-utils \
+    bison \
+    ca-certificates \
+    ccache \
+    check \
+    cmake \
+    curl \
+    flex \
+    git \
+    gperf \
+    lcov \
+    libncurses-dev \
+    libusb-1.0-0-dev \
+    make \
+    ninja-build \
+    python3 \
+    python3-pip \
+    unzip \
+    wget \
+    xz-utils \
+    zip \
+   && apt-get autoremove -y \
+   && rm -rf /var/lib/apt/lists/* \
+   && update-alternatives --install /usr/bin/python python /usr/bin/python3 10
+
+RUN python -m pip install --upgrade pip virtualenv
+
+# To build the image for a branch or a tag of IDF, pass --build-arg IDF_CLONE_BRANCH_OR_TAG=name.
+# To build the image with a specific commit ID of IDF, pass --build-arg IDF_CHECKOUT_REF=commit-id.
+# It is possibe to combine both, e.g.:
+#   IDF_CLONE_BRANCH_OR_TAG=release/vX.Y
+#   IDF_CHECKOUT_REF=<some commit on release/vX.Y branch>.
+
+ARG IDF_CLONE_URL=https://github.com/espressif/esp-idf.git
+ARG IDF_CLONE_BRANCH_OR_TAG=master
+ARG IDF_CHECKOUT_REF=
+
+ENV IDF_PATH=/opt/esp/idf
+ENV IDF_TOOLS_PATH=/opt/esp
+
+RUN echo IDF_CHECKOUT_REF=$IDF_CHECKOUT_REF IDF_CLONE_BRANCH_OR_TAG=$IDF_CLONE_BRANCH_OR_TAG && \
+    git clone --recursive \
+      ${IDF_CLONE_BRANCH_OR_TAG:+-b $IDF_CLONE_BRANCH_OR_TAG} \
+      $IDF_CLONE_URL $IDF_PATH && \
+    if [ -n "$IDF_CHECKOUT_REF" ]; then \
+      cd $IDF_PATH && \
+      git checkout $IDF_CHECKOUT_REF && \
+      git submodule update --init --recursive; \
+    fi
+
+RUN $IDF_PATH/install.sh && \
+  rm -rf $IDF_TOOLS_PATH/dist
+
+COPY entrypoint.sh /opt/esp/entrypoint.sh
+
+ENTRYPOINT [ "/opt/esp/entrypoint.sh" ]
+CMD [ "/bin/bash" ]

tools/docker/entrypoint.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+set -e
+
+. $IDF_PATH/export.sh
+
+exec "$@"