OVMS3-idf/docs/api-guides/freertos-smp.rst

ESP-IDF FreeRTOS SMP Changes
============================

Overview
--------

The vanilla FreeRTOS is designed to run on a single core. However the ESP32 is 
dual core containing a Protocol CPU (known as **CPU 0** or **PRO_CPU**) and an 
Application CPU (known as **CPU 1** or **APP_CPU**). The two cores are 
identical in practice and share the same memory. This allows the two cores to 
run tasks interchangeably between them.

The ESP-IDF FreeRTOS is a modified version of vanilla FreeRTOS which supports 
symmetric multiprocessing (SMP). ESP-IDF FreeRTOS is based on the Xtensa port 
of FreeRTOS v8.2.0, however features such as static task creation and Thread 
Local Storage Pointers have been backported from later versions of FreeRTOS. 
This guide outlines the major differences between vanilla FreeRTOS and 
ESP-IDF FreeRTOS. The API reference for vanilla FreeRTOS can be found 
via http://www.freertos.org/a00106.html

:ref:`tasks-and-task-creation`: Use ``xTaskCreatePinnedToCore()`` or 
``xTaskCreateStaticPinnedToCore()`` to create tasks in ESP-IDF FreeRTOS. The 
last parameter of the two functions is ``xCoreID``. This parameter specifies 
which core the task is pinned to. Acceptable values are ``0`` for **PRO_CPU**, 
``1`` for **APP_CPU**, or ``tskNO_AFFINITY`` which allows the task to run on
both.
        
:ref:`round-robin-scheduling`: The ESP-IDF FreeRTOS scheduler will skip tasks when 
implementing Round-Robin scheduling between multiple tasks in the Ready state 
that are of the same priority. To avoid this behavior, ensure that those tasks either 
enter a blocked state, or are distributed across a wider range of priorities.
        
:ref:`scheduler-suspension`: Suspending the scheduler in ESP-IDF FreeRTOS will only 
affect the scheduler on the the calling core. In other words, calling 
``vTaskSuspendAll()`` on **PRO_CPU** will not prevent **APP_CPU** from scheduling, and
vice versa. Use critical sections or semaphores instead for simultaneous
access protection.
    
:ref:`tick-interrupt-synchronicity`: Tick interrupts of **PRO_CPU** and **APP_CPU** 
are not synchronized. Do not expect to use ``vTaskDelay`` or 
``vTaskDelayUntil`` as an accurate method of synchronizing task execution 
between the two cores. Use a counting semaphore instead as their context 
switches are not tied to tick interrupts due to preemption.
        
:ref:`critical-sections`: In ESP-IDF FreeRTOS, critical sections are implemented using
mutexes. Entering critical sections involve taking a mutex, then disabling the 
scheduler and interrupts of the calling core. However the other core is left 
unaffected. If the other core attemps to take same mutex, it will spin until
the calling core has released the mutex by exiting the critical section.
        
:ref:`deletion-callbacks`: ESP-IDF FreeRTOS has 
backported the Thread Local Storage Pointers feature. However they have the 
extra feature of deletion callbacks. Deletion callbacks are used to
automatically free memory used by Thread Local Storage Pointers during the task
deletion. Call ``vTaskSetThreadLocalStoragePointerAndDelCallback()``
to set Thread Local Storage Pointers and deletion callbacks.

:ref:`esp-idf-freertos-configuration`: Several aspects of ESP-IDF FreeRTOS can be 
configured using ``make meunconfig`` such as running ESP-IDF in Unicore Mode,
or configuring the number of Thread Local Storage Pointers each task will have.


.. _tasks-and-task-creation:

Tasks and Task Creation
-----------------------

Tasks in ESP-IDF FreeRTOS are designed to run on a particular core, therefore 
two new task creation functions have been added to ESP-IDF FreeRTOS by 
appending ``PinnedToCore`` to the names of the task creation functions in 
vanilla FreeRTOS. The vanilla FreeRTOS functions of ``xTaskCreate()``
and ``xTaskCreateStatic()`` have led to the addition of 
``xTaskCreatePinnedToCore()`` and ``xTaskCreateStaticPinnedToCore()`` in 
ESP-IDF FreeRTOS.

For more details see :component_file:`freertos/task.c`

The ESP-IDF FreeRTOS task creation functions are nearly identical to their 
vanilla counterparts with the exception of the extra parameter known as 
``xCoreID``. This parameter specifies the core on which the task should run on 
and can be one of the following values.

    -	``0`` pins the task to **PRO_CPU**
    -	``1`` pins the task to **APP_CPU**
    -	``tskNO_AFFINITY`` allows the task to be run on both CPUs

For example ``xTaskCreatePinnedToCore(tsk_callback, “APP_CPU Task”, 1000, NULL, 10, NULL, 1)`` 
creates a task of priority 10 that is pinned to **APP_CPU** with a stack size 
of 1000 bytes. It should be noted that the ``uxStackDepth`` parameter in 
vanilla FreeRTOS specifies a task’s stack depth in terms of the number of 
words, whereas ESP-IDF FreeRTOS specifies the stack depth in terms of bytes.

Note that the vanilla FreeRTOS functions ``xTaskCreate`` and 
``xTaskCreateStatic`` have been macro defined in ESP-IDF FreeRTOS to call 
``xTaskCreatePinnedToCore()`` and ``xTaskCreateStaticPinnedToCore()``
respectively with ``tskNO_AFFINITY`` as the ``xCoreID`` value. 

Each Task Control Block (TCB) in ESP-IDF stores the ``xCoreID`` as a member. 
Hence when each core calls the scheduler to select a task to run, the 
``xCoreID`` member will allow the scheduler to determine if a given task is  
permitted to run on the core that called it.

Scheduling
----------

The vanilla FreeRTOS implements scheduling in the ``vTaskSwitchContext()`` 
function. This function is responsible for selecting the highest priority task
to run from a list of tasks in the Ready state known as the Ready Tasks List 
(described in the next section). In ESP-IDF FreeRTOS, each core will call 
``vTaskSwitchContext()`` independently to select a task to run from the 
Ready Tasks List which is shared between both cores. There are several 
differences in scheduling behavior between vanilla and ESP-IDF FreeRTOS such as 
differences in Round Robin scheduling, scheduler suspension, and tick interrupt 
synchronicity. 

.. _round-robin-scheduling:

Round Robin Scheduling
^^^^^^^^^^^^^^^^^^^^^^

Given multiple tasks in the Ready state and of the same priority, vanilla 
FreeRTOS implements Round Robin scheduling between each task. This will result
in running those tasks in turn each time the scheduler is called 
(e.g. every tick interrupt). On the other hand, the ESP-IDF FreeRTOS scheduler 
may skip tasks when Round Robin scheduling multiple Ready state tasks of the 
same priority.

The issue of skipping tasks during Round Robin scheduling arises from the way 
the Ready Tasks List is implemented in FreeRTOS. In vanilla FreeRTOS, 
``pxReadyTasksList`` is used to store a list of tasks that are in the Ready 
state. The list is implemented as an array of length ``configMAX_PRIORITIES`` 
where each element of the array is a linked list. Each linked list is of type 
``List_t`` and contains TCBs of tasks of the same priority that are in the 
Ready state. The following diagram illustrates the ``pxReadyTasksList`` 
structure.

.. figure:: ../_static/freertos-ready-task-list.png
    :align: center
    :alt: Vanilla FreeRTOS Ready Task List Structure
    
    Illustration of FreeRTOS Ready Task List Data Structure 


Each linked list also contains a ``pxIndex`` which points to the last TCB 
returned when the list was queried. This index allows the ``vTaskSwitchContext()`` 
to start traversing the list at the TCB immediately after ``pxIndex`` hence 
implementing Round Robin Scheduling between tasks of the same priority.

In ESP-IDF FreeRTOS, the Ready Tasks List is shared between cores hence 
``pxReadyTasksList`` will contain tasks pinned to different cores. When a core 
calls the scheduler, it is able to look at the ``xCoreID`` member of each TCB 
in the list to determine if a task is allowed to run on calling the core. The 
ESP-IDF FreeRTOS ``pxReadyTasksList`` is illustrated below.

.. figure:: ../_static/freertos-ready-task-list-smp.png
    :align: center
    :alt: ESP-IDF FreeRTOS Ready Task List Structure
    
    Illustration of FreeRTOS Ready Task List Data Structure in ESP-IDF
    
Therefore when **PRO_CPU** calls the scheduler, it will only consider the tasks 
in blue or purple. Whereas when **APP_CPU** calls the scheduler, it will only 
consider the tasks in orange or purple.

Although each TCB has an ``xCoreID`` in ESP-IDF FreeRTOS, the linked list of 
each priority only has a single ``pxIndex``. Therefore when the scheduler is 
called from a particular core and traverses the linked list, it will skip all 
TCBs pinned to the other core and point the pxIndex at the selected task. If 
the other core then calls the scheduler, it will traverse the linked list 
starting at the TCB immediately after ``pxIndex``. Therefore, TCBs skipped on
the previous scheduler call from the other core would not be considered on the 
current scheduler call. This issue is demonstrated in the following 
illustration.

.. figure:: ../_static/freertos-ready-task-list-smp-pxIndex.png
    :align: center
    :alt: ESP-IDF pxIndex Behavior
    
    Illustration of pxIndex behavior in ESP-IDF FreeRTOS

Referring to the illustration above, assume that priority 9 is the highest 
priority, and none of the tasks in priority 9 will block hence will always be 
either in the running or Ready state.

1)	**PRO_CPU** calls the scheduler and selects Task A to run, hence moves 
``pxIndex`` to point to Task A

2)	**APP_CPU** calls the scheduler and starts traversing from the task after 
``pxIndex`` which is Task B. However Task B is not selected to run as it is not 
pinned to **APP_CPU** hence it is skipped and Task C is selected instead. 
``pxIndex`` now points to Task C

3)	**PRO_CPU** calls the scheduler and starts traversing from Task D. It skips 
Task D and selects Task E to run and points ``pxIndex`` to Task E. Notice that 
Task B isn’t traversed because it was skipped the last time **APP_CPU** called 
the scheduler to traverse the list.

4)	The same situation with Task D will occur if **APP_CPU** calls the 
scheduler again as ``pxIndex`` now points to Task E

One solution to the issue of task skipping is to ensure that every task will
enter a blocked state so that they are removed from the Ready Task List.
Another solution is to distribute tasks across multiple priorities such that 
a given priority will not be assigned multiple tasks that are pinned to 
different cores.

.. _scheduler-suspension:

Scheduler Suspension
^^^^^^^^^^^^^^^^^^^^

In vanilla FreeRTOS, suspending the scheduler via ``vTaskSuspendAll()`` will 
prevent calls of ``vTaskSwitchContext()`` from context switching until the 
scheduler has been resumed with ``vTaskResumeAll()``. However servicing ISRs 
are still permitted. Therefore any changes in task states as a result from the
current running task or ISRSs will not be executed until the scheduler is 
resumed. Scheduler suspension in vanilla FreeRTOS is a common protection method 
against simultaneous access of data shared between tasks, whilst still allowing 
ISRs to be serviced.

In ESP-IDF FreeRTOS, ``vTaskSuspendAll()`` will only prevent calls of 
``vTaskSwitchContext()`` from switching contexts on the core that called for the
suspension. Hence if **PRO_CPU** calls ``vTaskSuspendAll()``, **APP_CPU** will 
still be able to switch contexts. If data is shared between tasks that are 
pinned to different cores, scheduler suspension is **NOT** a valid method of 
protection against simultaneous access. Consider using critical sections 
(disables interrupts) or semaphores (does not disable interrupts) instead when 
protecting shared resources in ESP-IDF FreeRTOS.

If the task running on the CPU with scheduler suspended calls a function (like
``xSemaphoreGive``, ``xQueueSend``) that wakes another task, this task will not run
until after ``vTaskResumeAll()`` is called. This is true even if the woken task is
on the other CPU (where the scheduler is still running).

In general, it's better to use other RTOS primitives like mutex semaphores to protect
against data shared between tasks, rather than ``vTaskSuspendAll()``.

.. _tick-interrupt-synchronicity:

Tick Interrupt Synchronicity 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In ESP-IDF FreeRTOS, tasks on different cores that unblock on the same tick 
count might not run at exactly the same time due to the scheduler calls from 
each core being independent, and the tick interrupts to each core being 
unsynchronized.

In vanilla FreeRTOS the tick interrupt triggers a call to 
``xTaskIncrementTick()`` which is responsible for incrementing the tick 
counter, checking if tasks which have called ``vTaskDelay()`` have fulfilled 
their delay period, and moving those tasks from the Delayed Task List to the 
Ready Task List. The tick interrupt will then call the scheduler if a context 
switch is necessary.

In ESP-IDF FreeRTOS, delayed tasks are unblocked with reference to the tick 
interrupt on PRO_CPU as PRO_CPU is responsible for incrementing the shared tick 
count. However tick interrupts to each core might not be synchronized (same 
frequency but out of phase) hence when PRO_CPU receives a tick interrupt, 
APP_CPU might not have received it yet. Therefore if multiple tasks of the same 
priority are unblocked on the same tick count, the task pinned to PRO_CPU will 
run immediately whereas the task pinned to APP_CPU must wait until APP_CPU 
receives its out of sync tick interrupt. Upon receiving the tick interrupt, 
APP_CPU will then call for a context switch and finally switches contexts to
the newly unblocked task.

Therefore, task delays should **NOT** be used as a method of synchronization 
between tasks in ESP-IDF FreeRTOS. Instead, consider using a counting semaphore 
to unblock multiple tasks at the same time.

.. _critical-sections:

Critical Sections & Disabling Interrupts
----------------------------------------

Vanilla FreeRTOS implements critical sections in ``vTaskEnterCritical`` which 
disables the scheduler and calls ``portDISABLE_INTERRUPTS``. This prevents 
context switches and servicing of ISRs during a critical section. Therefore, 
critical sections are used as a valid protection method against simultaneous 
access in vanilla FreeRTOS.

On the other hand, the ESP32 has no hardware method for cores to disable each 
other’s interrupts. Calling ``portDISABLE_INTERRUPTS()`` will have no effect on 
the interrupts of the other core. Therefore, disabling interrupts is **NOT** 
a valid protection method against simultaneous access to shared data as it 
leaves the other core free to access the data even if the current core has 
disabled its own interrupts. 

For this reason, ESP-IDF FreeRTOS implements critical sections using mutexes, 
and calls to enter or exit a critical must provide a mutex that is associated 
with a shared resource requiring access protection. When entering a critical 
section in ESP-IDF FreeRTOS, the calling core will disable its scheduler and 
interrupts similar to the vanilla FreeRTOS implementation. However, the calling 
core will also take the mutex whilst the other core is left unaffected during 
the critical section. If the other core attempts to take the same mutex, it 
will spin until the mutex is released. Therefore, the ESP-IDF FreeRTOS 
implementation of critical sections allows a core to have protected access to a
shared resource without disabling the other core. The other core will only be 
affected if it tries to concurrently access the same resource.

The ESP-IDF FreeRTOS critical section functions have been modified as follows…

 - ``taskENTER_CRITICAL(mux)``, ``taskENTER_CRITICAL_ISR(mux)``, 
   ``portENTER_CRITICAL(mux)``, ``portENTER_CRITICAL_ISR(mux)`` are all macro 
   defined to call ``vTaskEnterCritical()`` 

 - ``taskEXIT_CRITICAL(mux)``, ``taskEXIT_CRITICAL_ISR(mux)``, 
   ``portEXIT_CRITICAL(mux)``, ``portEXIT_CRITICAL_ISR(mux)`` are all macro 
   defined to call ``vTaskExitCritical()``

For more details see :component_file:`freertos/include/freertos/portmacro.h` 
and :component_file:`freertos/task.c`

It should be noted that when modifying vanilla FreeRTOS code to be ESP-IDF 
FreeRTOS compatible, it is trivial to modify the type of critical section 
called as they are all defined to call the same function. As long as the same 
mutex is provided upon entering and exiting, the type of call should not 
matter.

.. _deletion-callbacks:

Thread Local Storage Pointers & Deletion Callbacks
--------------------------------------------------

Thread Local Storage Pointers are pointers stored directly in the TCB which 
allows each task to have a pointer to a data structure containing that is 
specific to that task. However vanilla FreeRTOS provides no functionality to 
free the memory pointed to by the Thread Local Storage Pointers. Therefore if 
the memory pointed to by the Thread Local Storage Pointers is not explicitly 
freed by the user before a task is deleted, memory leak will occur.

ESP-IDF FreeRTOS provides the added feature of deletion callbacks. These 
deletion callbacks are used to automatically free the memory pointed to by the 
Thread Local Storage Pointers when a task is deleted. Each Thread Local Storage 
Pointer can have its own call back, and these call backs are called when the 
Idle tasks cleans up a deleted tasks.

Vanilla FreeRTOS sets a Thread Local Storage Pointers using 
``vTaskSetThreadLocalStoragePointer()`` whereas ESP-IDF FreeRTOS sets a Thread 
Local Storage Pointers and Deletion Callbacks using 
``vTaskSetThreadLocalStoragePointerAndDelCallback()`` which accepts a pointer 
to the deletion call back as an extra parameter of type 
```TlsDeleteCallbackFunction_t``. Calling the vanilla FreeRTOS API 
``vTaskSetThreadLocalStoragePointer()`` is still valid however it is internally
defined to call ``vTaskSetThreadLocalStoragePointerAndDelCallback()`` with a
``NULL`` pointer as the deletion call back. This results in the selected Thread 
Local Storage Pointer to have no deletion call back.

For more details see :component_file:`freertos/include/freertos/task.h`

.. _esp-idf-freertos-configuration:

Configuring ESP-IDF FreeRTOS
----------------------------

The ESP-IDF FreeRTOS can be configured using ``make menuconfig`` under 
``Component_Config/FreeRTOS``. The following section highlights some of the
ESP-IDF FreeRTOS configuration options. For a full list of ESP-IDF
FreeRTOS configurations, see :doc:`FreeRTOS <../api-reference/kconfig>`

:ref:`CONFIG_FREERTOS_UNICORE` will run ESP-IDF FreeRTOS only
on **PRO_CPU**. Note that this is **not equivalent to running vanilla 
FreeRTOS**. Behaviors of multiple components in ESP-IDF will be modified such 
as :component_file:`esp32/cpu_start.c`. For more details regarding the 
effects of running ESP-IDF FreeRTOS on a single core, search for 
occurences of ``CONFIG_FREERTOS_UNICORE`` in the ESP-IDF components.
    
:ref:`CONFIG_FREERTOS_THREAD_LOCAL_STORAGE_POINTERS` will define the 
number of Thread Local Storage Pointers each task will have in ESP-IDF 
FreeRTOS.

:ref:`CONFIG_SUPPORT_STATIC_ALLOCATION` will enable the backported
functionality of ``xTaskCreateStaticPinnedToCore()`` in ESP-IDF FreeRTOS
    
:ref:`CONFIG_FREERTOS_ASSERT_ON_UNTESTED_FUNCTION` will trigger a halt in
particular functions in ESP-IDF FreeRTOS which have not been fully tested
in an SMP context.