diff --git a/components/freertos/readme_smp.txt b/components/freertos/readme_smp.txt deleted file mode 100644 index fdd9b146b..000000000 --- a/components/freertos/readme_smp.txt +++ /dev/null @@ -1,27 +0,0 @@ -This version of FreeRTOS has been modified by Espressif to be SMP-aware. The -API is similar to the original FreeRTOS API, with the following changes: - -- The xTaskCreate() function now creates tasks that will run on the first -core only, for backwards compatibility. To schedule tasks on another core, -use xTaskCreatePinnedToCore(), which will accept a core ID as the last -argument. If this is the constant tskNO_AFFINITY, the task will be dynamically -scheduled on whichever core has time. - -- vTaskSuspendAll/vTaskResumeAll in non-SMP FreeRTOS will suspend the scheduler -so no other tasks than the current one will run. In this SMP version, it will -only suspend the scheduler ON THE CURRENT CORE. That is, tasks scheduled to -run on the other core(s) or without a specific CPU affinity, will still be -able to run. - -- Enabling and disabling interrupts will only affect the current core. -Disabling the interrupts will not disallow other tasks to run as -it would on a single-core system: the other core still will keep on -executing all it's own. Use a mux, queue or semaphore to protect your -structures instead. - -- This FreeRTOS version has the task local storage backported from the 8.2.x -versions. It, however, has an addition: you can also set a callback when you -set the pointer. This callback will be called by the idle task, with the -pointer as an argument, when the thread is destroyed. This depends on the idle -task getting CPU time; when a thread is hogging the CPU without yielding, -the idle thread won't be called and the delete callback won't be called either. diff --git a/docs/_static/freertos-ready-task-list-smp-pxIndex.png b/docs/_static/freertos-ready-task-list-smp-pxIndex.png new file mode 100644 index 000000000..08346858e Binary files /dev/null and b/docs/_static/freertos-ready-task-list-smp-pxIndex.png differ diff --git a/docs/_static/freertos-ready-task-list-smp.png b/docs/_static/freertos-ready-task-list-smp.png new file mode 100644 index 000000000..820007afc Binary files /dev/null and b/docs/_static/freertos-ready-task-list-smp.png differ diff --git a/docs/_static/freertos-ready-task-list.png b/docs/_static/freertos-ready-task-list.png new file mode 100644 index 000000000..ea1489545 Binary files /dev/null and b/docs/_static/freertos-ready-task-list.png differ diff --git a/docs/api-guides/freertos-smp.rst b/docs/api-guides/freertos-smp.rst new file mode 100644 index 000000000..7ccba1b10 --- /dev/null +++ b/docs/api-guides/freertos-smp.rst @@ -0,0 +1,369 @@ +ESP-IDF FreeRTOS SMP Changes +============================ + +Overview +-------- + +The vanilla FreeRTOS is designed to run on a single core. However the ESP32 is +dual core containing a Protocol CPU (known as **CPU 0** or **PRO_CPU**) and an +Application CPU (known as **CPU 1** or **APP_CPU**). The two cores are +identical in practice and share the same memory. This allows the two cores to +run tasks interchangeably between them. + +The ESP-IDF FreeRTOS is a modified version of vanilla FreeRTOS which supports +symmetric multiprocessing (SMP). ESP-IDF FreeRTOS is based on the Xtensa port +of FreeRTOS v8.2.0, however features such as static task creation and Thread +Local Storage Pointers have been backported from later versions of FreeRTOS. +This guide outlines the major differences between vanilla FreeRTOS and +ESP-IDF FreeRTOS. The API reference for vanilla FreeRTOS can be found +via http://www.freertos.org/a00106.html + +:ref:`tasks-and-task-creation`: Use ``xTaskCreatePinnedToCore()`` or +``xTaskCreateStaticPinnedToCore()`` to create tasks in ESP-IDF FreeRTOS. The +last parameter of the two functions is ``xCoreID``. This parameter specifies +which core the task is pinned to. Acceptable values are ``0`` for **PRO_CPU**, +``1`` for **APP_CPU**, or ``tskNO_AFFINITY`` which allows the task to run on +both. + +:ref:`round-robin-scheduling`: The ESP-IDF FreeRTOS scheduler will skip tasks when +implementing Round-Robin scheduling between multiple tasks in the Ready state +that are of the same priority. To avoid this behavior, ensure that those tasks either +enter a blocked state, or are distributed across a wider range of priorities. + +:ref:`scheduler-suspension`: Suspending the scheduler in ESP-IDF FreeRTOS will only +affect the scheduler on the the calling core. In other words, calling +``vTaskSuspendAll()`` on **PRO_CPU** will not prevent **APP_CPU** from scheduling, and +vice versa. Use critical sections or semaphores instead for simultaneous +access protection. + +:ref:`tick-interrupt-synchronicity`: Tick interrupts of **PRO_CPU** and **APP_CPU** +are not synchronized. Do not expect to use ``vTaskDelay`` or +``vTaskDelayUntil`` as an accurate method of synchronizing task execution +between the two cores. Use a counting semaphore instead as their context +switches are not tied to tick interrupts due to preemption. + +:ref:`critical-sections`: In ESP-IDF FreeRTOS, critical sections are implemented using +mutexes. Entering critical sections involve taking a mutex, then disabling the +scheduler and interrupts of the calling core. However the other core is left +unaffected. If the other core attemps to take same mutex, it will spin until +the calling core has released the mutex by exiting the critical section. + +:ref:`deletion-callbacks`: ESP-IDF FreeRTOS has +backported the Thread Local Storage Pointers feature. However they have the +extra feature of deletion callbacks. Deletion callbacks are used to +automatically free memory used by Thread Local Storage Pointers during the task +deletion. Call ``vTaskSetThreadLocalStoragePointerAndDelCallback()`` +to set Thread Local Storage Pointers and deletion callbacks. + +:ref:`esp-idf-freertos-configuration`: Several aspects of ESP-IDF FreeRTOS can be +configured using ``make meunconfig`` such as running ESP-IDF in Unicore Mode, +or configuring the number of Thread Local Storage Pointers each task will have. + + +.. _tasks-and-task-creation: + +Tasks and Task Creation +----------------------- + +Tasks in ESP-IDF FreeRTOS are designed to run on a particular core, therefore +two new task creation functions have been added to ESP-IDF FreeRTOS by +appending ``PinnedToCore`` to the names of the task creation functions in +vanilla FreeRTOS. The vanilla FreeRTOS functions of ``xTaskCreate()`` +and ``xTaskCreateStatic()`` have led to the addition of +``xTaskCreatePinnedToCore()`` and ``xTaskCreateStaticPinnedToCore()`` in +ESP-IDF FreeRTOS. + +For more details see :component_file:`freertos/task.c` + +The ESP-IDF FreeRTOS task creation functions are nearly identical to their +vanilla counterparts with the exception of the extra parameter known as +``xCoreID``. This parameter specifies the core on which the task should run on +and can be one of the following values. + + - ``0`` pins the task to **PRO_CPU** + - ``1`` pins the task to **APP_CPU** + - ``tskNO_AFFINITY`` allows the task to be run on both CPUs + +For example ``xTaskCreatePinnedToCore(tsk_callback, “APP_CPU Task”, 1000, NULL, 10, NULL, 1)`` +creates a task of priority 10 that is pinned to **APP_CPU** with a stack size +of 1000 bytes. It should be noted that the ``uxStackDepth`` parameter in +vanilla FreeRTOS specifies a task’s stack depth in terms of the number of +words, whereas ESP-IDF FreeRTOS specifies the stack depth in terms of bytes. + +Note that the vanilla FreeRTOS functions ``xTaskCreate`` and +``xTaskCreateStatic`` have been macro defined in ESP-IDF FreeRTOS to call +``xTaskCreatePinnedToCore()`` and ``xTaskCreateStaticPinnedToCore()`` +respectively with ``tskNO_AFFINITY`` as the ``xCoreID`` value. + +Each Task Control Block (TCB) in ESP-IDF stores the ``xCoreID`` as a member. +Hence when each core calls the scheduler to select a task to run, the +``xCoreID`` member will allow the scheduler to determine if a given task is +permitted to run on the core that called it. + +Scheduling +---------- + +The vanilla FreeRTOS implements scheduling in the ``vTaskSwitchContext()`` +function. This function is responsible for selecting the highest priority task +to run from a list of tasks in the Ready state known as the Ready Tasks List +(described in the next section). In ESP-IDF FreeRTOS, each core will call +``vTaskSwitchContext()`` independently to select a task to run from the +Ready Tasks List which is shared between both cores. There are several +differences in scheduling behavior between vanilla and ESP-IDF FreeRTOS such as +differences in Round Robin scheduling, scheduler suspension, and tick interrupt +synchronicity. + +.. _round-robin-scheduling: + +Round Robin Scheduling +^^^^^^^^^^^^^^^^^^^^^^ + +Given multiple tasks in the Ready state and of the same priority, vanilla +FreeRTOS implements Round Robin scheduling between each task. This will result +in running those tasks in turn each time the scheduler is called +(e.g. every tick interrupt). On the other hand, the ESP-IDF FreeRTOS scheduler +may skip tasks when Round Robin scheduling multiple Ready state tasks of the +same priority. + +The issue of skipping tasks during Round Robin scheduling arises from the way +the Ready Tasks List is implemented in FreeRTOS. In vanilla FreeRTOS, +``pxReadyTasksList`` is used to store a list of tasks that are in the Ready +state. The list is implemented as an array of length ``configMAX_PRIORITIES`` +where each element of the array is a linked list. Each linked list is of type +``List_t`` and contains TCBs of tasks of the same priority that are in the +Ready state. The following diagram illustrates the ``pxReadyTasksList`` +structure. + +.. figure:: ../_static/freertos-ready-task-list.png + :align: center + :alt: Vanilla FreeRTOS Ready Task List Structure + + Illustration of FreeRTOS Ready Task List Data Structure + + +Each linked list also contains a ``pxIndex`` which points to the last TCB +returned when the list was queried. This index allows the ``vTaskSwitchContext()`` +to start traversing the list at the TCB immediately after ``pxIndex`` hence +implementing Round Robin Scheduling between tasks of the same priority. + +In ESP-IDF FreeRTOS, the Ready Tasks List is shared between cores hence +``pxReadyTasksList`` will contain tasks pinned to different cores. When a core +calls the scheduler, it is able to look at the ``xCoreID`` member of each TCB +in the list to determine if a task is allowed to run on calling the core. The +ESP-IDF FreeRTOS ``pxReadyTasksList`` is illustrated below. + +.. figure:: ../_static/freertos-ready-task-list-smp.png + :align: center + :alt: ESP-IDF FreeRTOS Ready Task List Structure + + Illustration of FreeRTOS Ready Task List Data Structure in ESP-IDF + +Therefore when **PRO_CPU** calls the scheduler, it will only consider the tasks +in blue or purple. Whereas when **APP_CPU** calls the scheduler, it will only +consider the tasks in orange or purple. + +Although each TCB has an ``xCoreID`` in ESP-IDF FreeRTOS, the linked list of +each priority only has a single ``pxIndex``. Therefore when the scheduler is +called from a particular core and traverses the linked list, it will skip all +TCBs pinned to the other core and point the pxIndex at the selected task. If +the other core then calls the scheduler, it will traverse the linked list +starting at the TCB immediately after ``pxIndex``. Therefore, TCBs skipped on +the previous scheduler call from the other core would not be considered on the +current scheduler call. This issue is demonstrated in the following +illustration. + +.. figure:: ../_static/freertos-ready-task-list-smp-pxIndex.png + :align: center + :alt: ESP-IDF pxIndex Behavior + + Illustration of pxIndex behavior in ESP-IDF FreeRTOS + +Referring to the illustration above, assume that priority 9 is the highest +priority, and none of the tasks in priority 9 will block hence will always be +either in the running or Ready state. + +1) **PRO_CPU** calls the scheduler and selects Task A to run, hence moves +``pxIndex`` to point to Task A + +2) **APP_CPU** calls the scheduler and starts traversing from the task after +``pxIndex`` which is Task B. However Task B is not selected to run as it is not +pinned to **APP_CPU** hence it is skipped and Task C is selected instead. +``pxIndex`` now points to Task C + +3) **PRO_CPU** calls the scheduler and starts traversing from Task D. It skips +Task D and selects Task E to run and points ``pxIndex`` to Task E. Notice that +Task B isn’t traversed because it was skipped the last time **APP_CPU** called +the scheduler to traverse the list. + +4) The same situation with Task D will occur if **APP_CPU** calls the +scheduler again as ``pxIndex`` now points to Task E + +One solution to the issue of task skipping is to ensure that every task will +enter a blocked state so that they are removed from the Ready Task List. +Another solution is to distribute tasks across multiple priorities such that +a given priority will not be assigned multiple tasks that are pinned to +different cores. + +.. _scheduler-suspension: + +Scheduler Suspension +^^^^^^^^^^^^^^^^^^^^ + +In vanilla FreeRTOS, suspending the scheduler via ``vTaskSuspendAll()`` will +prevent calls of ``vTaskSwitchContext()`` from context switching until the +scheduler has been resumed with ``vTaskResumeAll()``. However servicing ISRs +are still permitted. Therefore any changes in task states as a result from the +current running task or ISRSs will not be executed until the scheduler is +resumed. Scheduler suspension in vanilla FreeRTOS is a common protection method +against simultaneous access of data shared between tasks, whilst still allowing +ISRs to be serviced. + +In ESP-IDF FreeRTOS, ``vTaskSuspendAll()`` will only prevent calls of +``vTaskSwitchContext()`` from switching contexts on the core that called for the +suspension. Hence if **PRO_CPU** calls ``vTaskSuspendAll()``, **APP_CPU** will +still be able to switch contexts. If data is shared between tasks that are +pinned to different cores, scheduler suspension is **NOT** a valid method of +protection against simultaneous access. Consider using critical sections +(disables interrupts) or semaphores (does not disable interrupts) instead when +protecting shared resources in ESP-IDF FreeRTOS. + +.. _tick-interrupt-synchronicity: + +Tick Interrupt Synchronicity +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In ESP-IDF FreeRTOS, tasks on different cores that unblock on the same tick +count might not run at exactly the same time due to the scheduler calls from +each core being independent, and the tick interrupts to each core being +unsynchronized. + +In vanilla FreeRTOS the tick interrupt triggers a call to +``xTaskIncrementTick()`` which is responsible for incrementing the tick +counter, checking if tasks which have called ``vTaskDelay()`` have fulfilled +their delay period, and moving those tasks from the Delayed Task List to the +Ready Task List. The tick interrupt will then call the scheduler if a context +switch is necessary. + +In ESP-IDF FreeRTOS, delayed tasks are unblocked with reference to the tick +interrupt on PRO_CPU as PRO_CPU is responsible for incrementing the shared tick +count. However tick interrupts to each core might not be synchronized (same +frequency but out of phase) hence when PRO_CPU receives a tick interrupt, +APP_CPU might not have received it yet. Therefore if multiple tasks of the same +priority are unblocked on the same tick count, the task pinned to PRO_CPU will +run immediately whereas the task pinned to APP_CPU must wait until APP_CPU +receives its out of sync tick interrupt. Upon receiving the tick interrupt, +APP_CPU will then call for a context switch and finally switches contexts to +the newly unblocked task. + +Therefore, task delays should **NOT** be used as a method of synchronization +between tasks in ESP-IDF FreeRTOS. Instead, consider using a counting semaphore +to unblock multiple tasks at the same time. + +.. _critical-sections: + +Critical Sections & Disabling Interrupts +---------------------------------------- + +Vanilla FreeRTOS implements critical sections in ``vTaskEnterCritical`` which +disables the scheduler and calls ``portDISABLE_INTERRUPTS``. This prevents +context switches and servicing of ISRs during a critical section. Therefore, +critical sections are used as a valid protection method against simultaneous +access in vanilla FreeRTOS. + +On the other hand, the ESP32 has no hardware method for cores to disable each +other’s interrupts. Calling ``portDISABLE_INTERRUPTS()`` will have no effect on +the interrupts of the other core. Therefore, disabling interrupts is **NOT** +a valid protection method against simultaneous access to shared data as it +leaves the other core free to access the data even if the current core has +disabled its own interrupts. + +For this reason, ESP-IDF FreeRTOS implements critical sections using mutexes, +and calls to enter or exit a critical must provide a mutex that is associated +with a shared resource requiring access protection. When entering a critical +section in ESP-IDF FreeRTOS, the calling core will disable its scheduler and +interrupts similar to the vanilla FreeRTOS implementation. However, the calling +core will also take the mutex whilst the other core is left unaffected during +the critical section. If the other core attempts to take the same mutex, it +will spin until the mutex is released. Therefore, the ESP-IDF FreeRTOS +implementation of critical sections allows a core to have protected access to a +shared resource without disabling the other core. The other core will only be +affected if it tries to concurrently access the same resource. + +The ESP-IDF FreeRTOS critical section functions have been modified as follows… + + - ``taskENTER_CRITICAL(mux)``, ``taskENTER_CRITICAL_ISR(mux)``, + ``portENTER_CRITICAL(mux)``, ``portENTER_CRITICAL_ISR(mux)`` are all macro + defined to call ``vTaskEnterCritical()`` + + - ``taskEXIT_CRITICAL(mux)``, ``taskEXIT_CRITICAL_ISR(mux)``, + ``portEXIT_CRITICAL(mux)``, ``portEXIT_CRITICAL_ISR(mux)`` are all macro + defined to call ``vTaskExitCritical()`` + +For more details see :component_file:`freertos/include/freertos/portmacro.h` +and :component_file:`freertos/task.c` + +It should be noted that when modifying vanilla FreeRTOS code to be ESP-IDF +FreeRTOS compatible, it is trivial to modify the type of critical section +called as they are all defined to call the same function. As long as the same +mutex is provided upon entering and exiting, the type of call should not +matter. + +.. _deletion-callbacks: + +Thread Local Storage Pointers & Deletion Callbacks +-------------------------------------------------- + +Thread Local Storage Pointers are pointers stored directly in the TCB which +allows each task to have a pointer to a data structure containing that is +specific to that task. However vanilla FreeRTOS provides no functionality to +free the memory pointed to by the Thread Local Storage Pointers. Therefore if +the memory pointed to by the Thread Local Storage Pointers is not explicitly +freed by the user before a task is deleted, memory leak will occur. + +ESP-IDF FreeRTOS provides the added feature of deletion callbacks. These +deletion callbacks are used to automatically free the memory pointed to by the +Thread Local Storage Pointers when a task is deleted. Each Thread Local Storage +Pointer can have its own call back, and these call backs are called when the +Idle tasks cleans up a deleted tasks. + +Vanilla FreeRTOS sets a Thread Local Storage Pointers using +``vTaskSetThreadLocalStoragePointer()`` whereas ESP-IDF FreeRTOS sets a Thread +Local Storage Pointers and Deletion Callbacks using +``vTaskSetThreadLocalStoragePointerAndDelCallback()`` which accepts a pointer +to the deletion call back as an extra parameter of type +```TlsDeleteCallbackFunction_t``. Calling the vanilla FreeRTOS API +``vTaskSetThreadLocalStoragePointer()`` is still valid however it is internally +defined to call ``vTaskSetThreadLocalStoragePointerAndDelCallback()`` with a +``NULL`` pointer as the deletion call back. This results in the selected Thread +Local Storage Pointer to have no deletion call back. + +For more details see :component_file:`freertos/include/freertos/task.h` + +.. _esp-idf-freertos-configuration: + +Configuring ESP-IDF FreeRTOS +---------------------------- + +The ESP-IDF FreeRTOS can be configured using ``make menuconfig`` under +``Component_Config/FreeRTOS``. The following section highlights some of the +ESP-IDF FreeRTOS configuration options. For a full list of ESP-IDF +FreeRTOS configurations, see :doc:`FreeRTOS <../api-reference/kconfig>` + +:ref:`CONFIG_FREERTOS_UNICORE` will run ESP-IDF FreeRTOS only +on **PRO_CPU**. Note that this is **not equivalent to running vanilla +FreeRTOS**. Behaviors of multiple components in ESP-IDF will be modified such +as :component_file:`esp32/cpu_start.c`. For more details regarding the +effects of running ESP-IDF FreeRTOS on a single core, search for +occurences of ``CONFIG_FREERTOS_UNICORE`` in the ESP-IDF components. + +:ref:`CONFIG_FREERTOS_THREAD_LOCAL_STORAGE_POINTERS` will define the +number of Thread Local Storage Pointers each task will have in ESP-IDF +FreeRTOS. + +:ref:`CONFIG_SUPPORT_STATIC_ALLOCATION` will enable the backported +functionality of ``xTaskCreateStaticPinnedToCore()`` in ESP-IDF FreeRTOS + +:ref:`CONFIG_FREERTOS_ASSERT_ON_UNTESTED_FUNCTION` will trigger a halt in +particular functions in ESP-IDF FreeRTOS which have not been fully tested +in an SMP context. + \ No newline at end of file diff --git a/docs/api-guides/index.rst b/docs/api-guides/index.rst index 0f155bafb..72bd9f04b 100644 --- a/docs/api-guides/index.rst +++ b/docs/api-guides/index.rst @@ -9,6 +9,7 @@ API Guides Deep Sleep Wake Stubs ESP32 Core Dump Flash Encryption <../security/flash-encryption> + FreeRTOS SMP Changes High Level Interrupts JTAG Debugging Partition Tables