Crash_Override/OVMS3-idf
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Doc/review api ref storage

Browse source
This commit is contained in:
Kirill Chalov 2019-06-17 14:23:52 +08:00 committed by Krzysztof Budzynski
parent e20b37aff9
commit 4faf2de035
13 changed files with 538 additions and 535 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

132
components/nvs_flash/README.rst
View file

@ -4,54 +4,60 @@ Non-volatile storage library
Introduction
------------
Non-volatile storage (NVS) library is designed to store key-value pairs in flash. This sections introduces some concepts used by NVS.
Non-volatile storage (NVS) library is designed to store key-value pairs in flash. This section introduces some concepts used by NVS.
Underlying storage
^^^^^^^^^^^^^^^^^^
Currently NVS uses a portion of main flash memory through ``spi_flash_{read|write|erase}`` APIs. The library uses the all the partitions with ``data`` type and ``nvs`` subtype. The application can choose to use the partition with label ``nvs`` through ``nvs_open`` API or any of the other partition by specifying its name through ``nvs_open_from_part`` API.
Currently, NVS uses a portion of main flash memory through ``spi_flash_{read|write|erase}`` APIs. The library uses all the partitions with ``data`` type and ``nvs`` subtype. The application can choose to use the partition with the label ``nvs`` through the ``nvs_open`` API function or any other partition by specifying its name using the ``nvs_open_from_part`` API function.
Future versions of this library may add other storage backends to keep data in another flash chip (SPI or I2C), RTC, FRAM, etc.
Future versions of this library may have other storage backends to keep data in another flash chip (SPI or I2C), RTC, FRAM, etc.
.. note:: if an NVS partition is truncated (for example, when the partition table layout is changed), its contents should be erased. ESP-IDF build system provides a ``make erase_flash`` target to erase all contents of the flash chip.
.. note:: NVS works best for storing many small values, rather than a few large values of type 'string' and 'blob'. If storing large blobs or strings is required, consider using the facilities provided by the FAT filesystem on top of the wear levelling library.
.. note:: NVS works best for storing many small values, rather than a few large values of the type 'string' and 'blob'. If you need to store large blobs or strings, consider using the facilities provided by the FAT filesystem on top of the wear levelling library.
Keys and values
^^^^^^^^^^^^^^^
NVS operates on key-value pairs. Keys are ASCII strings, maximum key length is currently 15 characters. Values can have one of the following types:
NVS operates on key-value pairs. Keys are ASCII strings; the maximum key length is currently 15 characters. Values can have one of the following types:
- integer types: ``uint8_t``, ``int8_t``, ``uint16_t``, ``int16_t``, ``uint32_t``, ``int32_t``, ``uint64_t``, ``int64_t``
- zero-terminated string
- variable length binary data (blob)
.. note::
String values are currently limited to 4000 bytes. This includes the null terminator. Blob values are limited to 508000 bytes or (97.6% of the partition size - 4000) bytes whichever is lower.
Additional types, such as ``float`` and ``double`` may be added later.
String values are currently limited to 4000 bytes. This includes the null terminator. Blob values are limited to 508000 bytes or 97.6% of the partition size - 4000 bytes, whichever is lower.
Keys are required to be unique. Writing a value for a key which already exists behaves as follows:
Additional types, such as ``float`` and ``double`` might be added later.
- if the new value is of the same type as old one, value is updated
- if the new value has different data type, an error is returned
Keys are required to be unique. Assigning a new value to an existing key works as follows:
- if the new value is of the same type as the old one, value is updated
- if the new value has a different data type, an error is returned
Data type check is also performed when reading a value. An error is returned if the data type of the read operation does not match the data type of the value.
Data type check is also performed when reading a value. An error is returned if data type of read operation doesn’t match the data type of the value.
Namespaces
^^^^^^^^^^
To mitigate potential conflicts in key names between different components, NVS assigns each key-value pair to one of namespaces. Namespace names follow the same rules as key names, i.e. 15 character maximum length. Namespace name is specified in the ``nvs_open`` or ``nvs_open_from_part`` call. This call returns an opaque handle, which is used in subsequent calls to ``nvs_read_*``, ``nvs_write_*``, and ``nvs_commit`` functions. This way, handle is associated with a namespace, and key names will not collide with same names in other namespaces.
Please note that the namespaces with same name in different NVS partitions are considered as separate namespaces.
To mitigate potential conflicts in key names between different components, NVS assigns each key-value pair to one of namespaces. Namespace names follow the same rules as key names, i.e., the maximum length is 15 characters. Namespace name is specified in the ``nvs_open`` or ``nvs_open_from_part`` call. This call returns an opaque handle, which is used in subsequent calls to the ``nvs_read_*``, ``nvs_write_*``, and ``nvs_commit`` functions. This way, a handle is associated with a namespace, and key names will not collide with same names in other namespaces.
Please note that the namespaces with the same name in different NVS partitions are considered as separate namespaces.
Security, tampering, and robustness
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NVS is not directly compatible with the ESP32 flash encryption system. However, data can still be stored in encrypted form if NVS encryption is used together with ESP32 flash encryption. Please refer to :ref:`nvs_encryption` for more details.
If NVS encryption is not used, it is possible for anyone with physical access to the flash chip to alter, erase, or add key-value pairs. With NVS encryption enabled, it is not possible to alter or add a key-value pair and get recognized as a valid pair without knowing corresponding NVS encryption keys. However, there is no tamper-resistance against erase operation.
If NVS encryption is not used, it is possible for anyone with physical access to the flash chip to alter, erase, or add key-value pairs. With NVS encryption enabled, it is not possible to alter or add a key-value pair and get recognized as a valid pair without knowing corresponding NVS encryption keys. However, there is no tamper-resistance against the erase operation.
The library does try to recover from conditions when flash memory is in an inconsistent state. In particular, one should be able to power off the device at any point and time and then power it back on. This should not result in loss of data, except for the new key-value pair if it was being written at the moment of powering off. The library should also be able to initialize properly with any random data present in flash memory.
The library does try to recover from conditions when flash memory is in an inconsistent state. In particular, one should be able to power off the device at any point and time and then power it back on. This should not result in loss of data, except for the new key-value pair if it was being written at the moment of power off. The library should also be able to initialize properly with any random data present in flash memory.
Internals
---------
@ -59,7 +65,7 @@ Internals
Log of key-value pairs
^^^^^^^^^^^^^^^^^^^^^^
NVS stores key-value pairs sequentially, with new key-value pairs being added at the end. When a value of any given key has to be updated, new key-value pair is added at the end of the log and old key-value pair is marked as erased.
NVS stores key-value pairs sequentially, with new key-value pairs being added at the end. When a value of any given key has to be updated, a new key-value pair is added at the end of the log and the old key-value pair is marked as erased.
Pages and entries
^^^^^^^^^^^^^^^^^
@ -67,22 +73,22 @@ Pages and entries
NVS library uses two main entities in its operation: pages and entries. Page is a logical structure which stores a portion of the overall log. Logical page corresponds to one physical sector of flash memory. Pages which are in use have a *sequence number* associated with them. Sequence numbers impose an ordering on pages. Higher sequence numbers correspond to pages which were created later. Each page can be in one of the following states:
Empty/uninitialized
Flash storage for the page is empty (all bytes are ``0xff``). Page isn't used to store any data at this point and doesn’t have a sequence number.
Flash storage for the page is empty (all bytes are ``0xff``). Page is not used to store any data at this point and does not have a sequence number.
Active
Flash storage is initialized, page header has been written to flash, page has a valid sequence number. Page has some empty entries and data can be written there. At most one page can be in this state at any given moment.
Flash storage is initialized, page header has been written to flash, page has a valid sequence number. Page has some empty entries and data can be written there. No more than one page can be in this state at any given moment.
Full
Flash storage is in a consistent state and is filled with key-value pairs.
Writing new key-value pairs into this page is not possible. It is still possible to mark some key-value pairs as erased.
Erasing
Non-erased key-value pairs are being moved into another page so that the current page can be erased. This is a transient state, i.e. page should never stay in this state when any API call returns. In case of a sudden power off, move-and-erase process will be completed upon next power on.
Non-erased key-value pairs are being moved into another page so that the current page can be erased. This is a transient state, i.e., page should never stay in this state at the time when any API call returns. In case of a sudden power off, the move-and-erase process will be completed upon the next power-on.
Corrupted
Page header contains invalid data, and further parsing of page data was canceled. Any items previously written into this page will not be accessible. Corresponding flash sector will not be erased immediately, and will be kept along with sectors in *uninitialized* state for later use. This may be useful for debugging.
Page header contains invalid data, and further parsing of page data was canceled. Any items previously written into this page will not be accessible. The corresponding flash sector will not be erased immediately and will be kept along with sectors in *uninitialized* state for later use. This may be useful for debugging.
Mapping from flash sectors to logical pages doesn't have any particular order. Library will inspect sequence numbers of pages found in each flash sector and organize pages in a list based on these numbers.
Mapping from flash sectors to logical pages does not have any particular order. The library will inspect sequence numbers of pages found in each flash sector and organize pages in a list based on these numbers.
::
@ -101,11 +107,11 @@ Mapping from flash sectors to logical pages doesn't have any particular order. L
Structure of a page
^^^^^^^^^^^^^^^^^^^
For now we assume that flash sector size is 4096 bytes and that ESP32 flash encryption hardware operates on 32-byte blocks. It is possible to introduce some settings configurable at compile-time (e.g. via menuconfig) to accommodate flash chips with different sector sizes (although it is not clear if other components in the system, e.g. SPI flash driver and SPI flash cache can support these other sizes).
For now, we assume that flash sector size is 4096 bytes and that ESP32 flash encryption hardware operates on 32-byte blocks. It is possible to introduce some settings configurable at compile-time (e.g., via menuconfig) to accommodate flash chips with different sector sizes (although it is not clear if other components in the system, e.g., SPI flash driver and SPI flash cache can support these other sizes).
Page consists of three parts: header, entry state bitmap, and entries themselves. To be compatible with ESP32 flash encryption, entry size is 32 bytes. For integer types, entry holds one key-value pair. For strings and blobs, an entry holds part of key-value pair (more on that in the entry structure description).
The following diagram illustrates page structure. Numbers in parentheses indicate size of each part in bytes. ::
The following diagram illustrates the page structure. Numbers in parentheses indicate the size of each part in bytes. ::
+-----------+--------------+-------------+-------------------------+
| State (4) | Seq. no. (4) | version (1) | Unused (19) | CRC32 (4) | Header (32)
@ -122,23 +128,23 @@ The following diagram illustrates page structure. Numbers in parentheses indicat
| Entry 125 (32) |
+------------------------------------------------------------------+
Page header and entry state bitmap are always written to flash unencrypted. Entries are encrypted if flash encryption feature of the ESP32 is used.
Page header and entry state bitmap are always written to flash unencrypted. Entries are encrypted if flash encryption feature of ESP32 is used.
Page state values are defined in such a way that changing state is possible by writing 0 into some of the bits. Therefore it not necessary to erase the page to change page state, unless that is a change to *erased* state.
Page state values are defined in such a way that changing state is possible by writing 0 into some of the bits. Therefore it is not necessary to erase the page to change its state unless that is a change to the *erased* state.
The version field in the header reflects NVS format version used. For backward compatibility reasons, it is decremented for every version upgrade starting at 0xff (i.e. 0xff for version-1, 0xfe for version-2 and so on).
The version field in the header reflects the NVS format version used. For backward compatibility reasons, it is decremented for every version upgrade starting at 0xff (i.e., 0xff for version-1, 0xfe for version-2 and so on).
CRC32 value in header is calculated over the part which doesn't include state value (bytes 4 to 28). Unused part is currently filled with ``0xff`` bytes.
CRC32 value in the header is calculated over the part which does not include a state value (bytes 4 to 28). The unused part is currently filled with ``0xff`` bytes.
The following sections describe structure of entry state bitmap and entry itself.
The following sections describe the structure of entry state bitmap and entry itself.
Entry and entry state bitmap
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Each entry can be in one of the following three states. Each state is represented with two bits in the entry state bitmap. Final four bits in the bitmap (256 - 2 * 126) are unused.
Each entry can be in one of the following three states represented with two bits in the entry state bitmap. The final four bits in the bitmap (256 - 2 * 126) are not used.
Empty (2'b11)
Nothing is written into the specific entry yet. It is in an uninitialized state (all bytes ``0xff``).
Nothing is written into the specific entry yet. It is in an uninitialized state (all bytes are ``0xff``).
Written (2'b10)
A key-value pair (or part of key-value pair which spans multiple entries) has been written into the entry.
@ -152,7 +158,7 @@ Erased (2'b00)
Structure of entry
^^^^^^^^^^^^^^^^^^
For values of primitive types (currently integers from 1 to 8 bytes long), entry holds one key-value pair. For string and blob types, entry holds part of the whole key-value pair. For strings, in case when a key-value pair spans multiple entries, all entries are stored in the same page. Blobs are allowed to span over multiple pages by dividing them into smaller chunks. For the purpose tracking these chunks, an additional fixed length metadata entry is stored called "blob index" entry. Earlier format of blobs are still supported (can be read and modified). However, once the blobs are modified, they are stored using the new format.
For values of primitive types (currently integers from 1 to 8 bytes long), entry holds one key-value pair. For string and blob types, entry holds part of the whole key-value pair. For strings, in case when a key-value pair spans multiple entries, all entries are stored in the same page. Blobs are allowed to span over multiple pages by dividing them into smaller chunks. For tracking these chunks, an additional fixed length metadata entry is stored called "blob index". Earlier formats of blobs are still supported (can be read and modified). However, once the blobs are modified, they are stored using the new format.
::
@ -160,10 +166,10 @@ For values of primitive types (currently integers from 1 to 8 bytes long), entry
| NS (1) | Type (1) | Span (1) | ChunkIndex (1) | CRC32 (4) | Key (16) | Data (8) |
+--------+----------+----------+----------------+-----------+---------------+----------+
Primitive +--------------------------------+
+--------> | Data (8) |
Primitive +--------------------------------+
+--------> | Data (8) |
| Types +--------------------------------+
+-> Fixed length --
+-> Fixed length --
| | +---------+--------------+---------------+-------+
| +--------> | Size(4) | ChunkCount(1)| ChunkStart(1) | Rsv(2)|
Data format ---+ Blob Index +---------+--------------+---------------+-------+
@ -176,25 +182,25 @@ For values of primitive types (currently integers from 1 to 8 bytes long), entry
Individual fields in entry structure have the following meanings:
NS
Namespace index for this entry. See section on namespaces implementation for explanation of this value.
Namespace index for this entry. For more information on this value, see the section on namespaces implementation.
Type
One byte indicating data type of value. See ``ItemType`` enumeration in ``nvs_types.h`` for possible values.
One byte indicating the value data type. See the ``ItemType`` enumeration in ``nvs_types.h`` for possible values.
Span
Number of entries used by this key-value pair. For integer types, this is equal to 1. For strings and blobs this depends on value length.
Number of entries used by this key-value pair. For integer types, this is equal to 1. For strings and blobs, this depends on value length.
ChunkIndex
Used to store index of the blob-data chunk for blob types. For other types, this should be ``0xff``.
Used to store the index of a blob-data chunk for blob types. For other types, this should be ``0xff``.
CRC32
Checksum calculated over all the bytes in this entry, except for the CRC32 field itself.
Key
Zero-terminated ASCII string containing key name. Maximum string length is 15 bytes, excluding zero terminator.
Zero-terminated ASCII string containing a key name. Maximum string length is 15 bytes, excluding a zero terminator.
Data
For integer types, this field contains the value itself. If the value itself is shorter than 8 bytes it is padded to the right, with unused bytes filled with ``0xff``.
For integer types, this field contains the value itself. If the value itself is shorter than 8 bytes, it is padded to the right, with unused bytes filled with ``0xff``.
For "blob index" entry, these 8 bytes hold the following information about data-chunks:
@ -205,23 +211,23 @@ Data
(Only for blob index.) Total number of blob-data chunks into which the blob was divided during storage.
- ChunkStart
(Only for blob index.) ChunkIndex of the first blob-data chunk of this blob. Subsequent chunks have chunkIndex incrementally allocated (step of 1)
(Only for blob index.) ChunkIndex of the first blob-data chunk of this blob. Subsequent chunks have chunkIndex incrementally allocated (step of 1).
For string and blob data chunks, these 8 bytes hold additional data about the value, described next:
For string and blob data chunks, these 8 bytes hold additional data about the value, which are described below:
- Size
(Only for strings and blobs.) Size, in bytes, of actual data. For strings, this includes zero terminator.
(Only for strings and blobs.) Size, in bytes, of actual data. For strings, this includes zero terminators.
- CRC32
(Only for strings and blobs.) Checksum calculated over all bytes of data.
Variable length values (strings and blobs) are written into subsequent entries, 32 bytes per entry. `Span` field of the first entry indicates how many entries are used.
Variable length values (strings and blobs) are written into subsequent entries, 32 bytes per entry. The `Span` field of the first entry indicates how many entries are used.
Namespaces
^^^^^^^^^^
As mentioned above, each key-value pair belongs to one of the namespaces. Namespaces identifiers (strings) are stored as keys of key-value pairs in namespace with index 0. Values corresponding to these keys are indexes of these namespaces.
As mentioned above, each key-value pair belongs to one of the namespaces. Namespace identifiers (strings) are stored as keys of key-value pairs in namespace with index 0. Values corresponding to these keys are indexes of these namespaces.
::
@ -239,23 +245,23 @@ As mentioned above, each key-value pair belongs to one of the namespaces. Namesp
Item hash list
^^^^^^^^^^^^^^
To reduce the number of reads performed from flash memory, each member of Page class maintains a list of pairs: (item index; item hash). This list makes searches much quicker. Instead of iterating over all entries, reading them from flash one at a time, ``Page::findItem`` first performs search for item hash in the hash list. This gives the item index within the page, if such an item exists. Due to a hash collision it is possible that a different item will be found. This is handled by falling back to iteration over items in flash.
To reduce the number of reads from flash memory, each member of the Page class maintains a list of pairs: item index; item hash. This list makes searches much quicker. Instead of iterating over all entries, reading them from flash one at a time, ``Page::findItem`` first performs a search for the item hash in the hash list. This gives the item index within the page if such an item exists. Due to a hash collision, it is possible that a different item will be found. This is handled by falling back to iteration over items in flash.
Each node in hash list contains a 24-bit hash and 8-bit item index. Hash is calculated based on item namespace, key name and ChunkIndex. CRC32 is used for calculation, result is truncated to 24 bits. To reduce overhead of storing 32-bit entries in a linked list, list is implemented as a doubly-linked list of arrays. Each array holds 29 entries, for the total size of 128 bytes, together with linked list pointers and 32-bit count field. Minimal amount of extra RAM usage per page is therefore 128 bytes, maximum is 640 bytes.
Each node in the hash list contains a 24-bit hash and 8-bit item index. Hash is calculated based on item namespace, key name, and ChunkIndex. CRC32 is used for calculation; the result is truncated to 24 bits. To reduce the overhead for storing 32-bit entries in a linked list, the list is implemented as a double-linked list of arrays. Each array holds 29 entries, for the total size of 128 bytes, together with linked list pointers and a 32-bit count field. The minimum amount of extra RAM usage per page is therefore 128 bytes; maximum is 640 bytes.
.. _nvs_encryption:
NVS Encryption
--------------
Data stored in NVS partitions can be encrypted using AES-XTS in the manner similar to one mentioned in disc encryption standard IEEE P1619. For the purpose of encryption, each entry is considered as one `sector` and relative address of the entry (w.r.t. partition-start) is fed to the encryption algorithm as `sector-number`. The keys required for nvs encryption are stored in yet another partition, which is protected using :doc:`Flash Encryption <../../security/flash-encryption>`. Therefore, enabling :doc:`Flash Encryption <../../security/flash-encryption>` is a prerequisite for NVS encryption.
Data stored in NVS partitions can be encrypted using AES-XTS in the manner similar to the one mentioned in disk encryption standard IEEE P1619. For the purpose of encryption, each entry is treated as one `sector` and relative address of the entry (w.r.t. partition-start) is fed to the encryption algorithm as `sector-number`. The keys required for NVS encryption are stored in yet another partition, which is protected using :doc:`Flash Encryption <../../security/flash-encryption>`. Therefore, enabling :doc:`Flash Encryption <../../security/flash-encryption>` is a prerequisite for NVS encryption.
.. _nvs_key_partition:
NVS key partition
^^^^^^^^^^^^^^^^^
An application requiring NVS encryption support needs to be compiled with a key-partition of type `data` and subtype `key`. This partition should be marked as `encrypted`. Refer to :doc:`Partition Tables <../../api-guides/partition-tables>` for more details. The size of the partition should be 4096 bytes (minimum partition size). The structure of this partition is depicted below.
An application requiring NVS encryption support needs to be compiled with a key-partition of the type `data` and subtype `key`. This partition should be marked as `encrypted`. Refer to :doc:`Partition Tables <../../api-guides/partition-tables>` for more details. The size of the partition should be 4096 bytes (minimum partition size). The structure of this partition is depicted below.
::
@ -267,25 +273,33 @@ An application requiring NVS encryption support needs to be compiled with a key-
| CRC32(4) |
+---------------------------------------------+
This partition can be generated using `nvs partition generator` utility and flashed onto the device. Since the partition is marked `encrypted` and :doc:`Flash Encryption <../../security/flash-encryption>` is enabled, bootloader will encrypt this partition using flash encryption key on first boot. Alternatively, the keys can be generated after startup using ``nvs_flash_generate_keys`` API provided by ``nvs_flash.h``, which will then write those keys onto the key-partition in encrypted form.
This partition can be generated using `nvs partition generator` utility and flashed onto the device. Since the partition is marked `encrypted` and :doc:`Flash Encryption <../../security/flash-encryption>` is enabled, bootloader will encrypt this partition using flash encryption key on the first boot. Alternatively, the keys can be generated after startup using the ``nvs_flash_generate_keys`` API function provided by ``nvs_flash.h``, which will then write those keys onto the key-partition in encrypted form.
It is possible for an application to use different keys for different NVS partitions and thereby have multiple key-partitions. However, it is a responsibility of the application to provide correct key-partition/keys for the purpose of encryption/decryption.
Encrypted Read/Write
^^^^^^^^^^^^^^^^^^^^
The same NVS APIs ``nvs_read_*`` or ``nvs_write_*`` can be used for reading and writing of encrypted nvs partition as well. However, the APIs for initialising NVS partitions are different. ``nvs_flash_secure_init`` and ``nvs_flash_secure_init_partition`` are used for initialising instead of ``nvs_flash_init`` and ``nvs_flash_init_partition`` respectively. ``nvs_sec_cfg_t`` structure required for these APIs can be populated using ``nvs_flash_generate_keys`` or ``nvs_flash_read_security_cfg``.
The same NVS API functions ``nvs_read_*`` or ``nvs_write_*`` can be used for reading of, and writing to an encrypted nvs partition as well. However, the API functions for initialising NVS partitions are different: ``nvs_flash_secure_init`` and ``nvs_flash_secure_init_partition`` instead of ``nvs_flash_init`` and ``nvs_flash_init_partition`` respectively. The ``nvs_sec_cfg_t`` structure required for these API functions can be populated using ``nvs_flash_generate_keys`` or ``nvs_flash_read_security_cfg``.
Applications are expected to follow the following steps in order to perform NVS read/write operations with encryption enabled.
Applications are expected to follow the steps below in order to perform NVS read/write operations with encryption enabled.
1. Find key partition and NVS data partition using ``esp_partition_find*`` APIs.
2. Populate ``nvs_sec_cfg_t`` struct using ``nvs_flash_read_security_cfg`` or ``nvs_flash_generate_keys`` APIs.
3. Initialise NVS flash partition using ``nvs_flash_secure_init`` or ``nvs_flash_secure_init_partition`` APIs.
4. Open a namespace using ``nvs_open`` or ``nvs_open_from_part`` APIs
5. Perform NVS read/write operations using ``nvs_read_*`` or ``nvs_write_*``
6. Deinitialise NVS partition using ``nvs_flash_deinit``.
1. Find key partition and NVS data partition using ``esp_partition_find*`` API functions.
2. Populate the ``nvs_sec_cfg_t`` struct using the ``nvs_flash_read_security_cfg`` or ``nvs_flash_generate_keys`` API functions.
3. Initialise NVS flash partition using the ``nvs_flash_secure_init`` or ``nvs_flash_secure_init_partition`` API functions.
4. Open a namespace using the ``nvs_open`` or ``nvs_open_from_part`` API functions.
5. Perform NVS read/write operations using ``nvs_read_*`` or ``nvs_write_*``.
6. Deinitialise an NVS partition using ``nvs_flash_deinit``.
NVS iterators
^^^^^^^^^^^^^
Iterators allow to list key-value pairs stored in NVS based on specified partition name, namespace and data type. ``nvs_entry_find`` returns an opaque handle, which is used in subsequent calls to ``nvs_entry_next`` and ``nvs_entry_info`` function. ``nvs_entry_next`` function returns iterator to the next key-value pair. If none or no other key-value pair was found for given criteria, ``nvs_entry_find`` and ``nvs_entry_next`` functions return NULL. In that case, iterator does not have to be released. Otherwise, ``nvs_release_iterator`` function has to be used, when iterator is no longer needed. Information about each key-value pair can be obtained from ``nvs_entry_info`` function.
Iterators allow to list key-value pairs stored in NVS, based on specified partition name, namespace, and data type.
There are the following functions available:
- ``nvs_entry_find`` returns an opaque handle, which is used in subsequent calls to the ``nvs_entry_next`` and ``nvs_entry_info`` functions.
- ``nvs_entry_next`` returns iterator to the next key-value pair.
- ``nvs_entry_info`` returns information about each key-value pair
If none or no other key-value pair was found for given criteria, ``nvs_entry_find`` and ``nvs_entry_next`` return NULL. In that case, the iterator does not have to be released. If the iterator is no longer needed, you can release it by using the function ``nvs_release_iterator``.

141
components/nvs_flash/nvs_partition_generator/README.rst
View file

@ -4,64 +4,81 @@ NVS Partition Generator Utility
Introduction
------------
:component_file:`nvs_flash/nvs_partition_generator/nvs_partition_gen.py` utility is designed to help create a binary file, compatible with NVS architecture defined in :doc:`Non-Volatile Storage </api-reference/storage/nvs_flash>`, based on user provided key-value pairs in a CSV file.
Utility is ideally suited for generating a binary blob, containing data specific to ODM/OEM, which can be flashed externally at the time of device manufacturing. This helps manufacturers set unique value for various parameters for each device, e.g. serial number, while using same application firmware for all devices.
The utility :component_file:`nvs_flash/nvs_partition_generator/nvs_partition_gen.py` creates a binary file based on key-value pairs provided in a CSV file. The binary file is compatible with NVS architecture defined in :doc:`Non-Volatile Storage </api-reference/storage/nvs_flash>`.
This utility is ideally suited for generating a binary blob, containing data specific to ODM/OEM, which can be flashed externally at the time of device manufacturing. This allows manufacturers to generate many instances of the same application firmware with customized parameters for each device, such as a serial number.
Prerequisites
-------------
To use this utility in encryption mode, the following packages need to be installed:
To use this utility in encryption mode, install the following packages:
- cryptography package
These dependencies is already captured by including these packages in `requirement.txt` in top level IDF directory.
All the required packages are included in `requirements.txt` in the root of the esp-idf directory.
CSV file format
---------------
Each row of the .csv file should have 4 parameters, separated by comma. Below is the description of each of these parameters:
Each line of a .csv file should contain 4 parameters, separated by a comma. The table below provides the description for each of these parameters.
Key
Key of the data. Data can later be accessed from an application via this key.
+-----+-----------+----------------------------------------------------------------------+-----------------------------------------------------+
| No. | Parameter | Description | Notes |
+=====+===========+======================================================================+=====================================================+
| 1 | Key | Key of the data. The data can be accessed later from | |
| | | an application using this key. | |
+-----+-----------+----------------------------------------------------------------------+-----------------------------------------------------+
| 2 | Type | Supported values are ``file``, ``data`` and ``namespace``. | |
+-----+-----------+----------------------------------------------------------------------+-----------------------------------------------------+
| 3 | Encoding | Supported values are: ``u8``, ``i8``, ``u16``, ``u32``, | As of now, for the ``file`` type, |
| | | ``i32``, ``string``, ``hex2bin``, ``base64`` and ``binary``. | only ``hex2bin``, ``base64``, ``string``, |
| | | This specifies how actual data values are encoded in the | and ``binary`` encoding is supported. |
| | | resulting binary file. The difference between the ``string`` | |
| | | and ``binary`` encoding is that ``string`` data is terminated | |
| | | with a NULL character, whereas ``binary`` data is not. | |
+-----+-----------+----------------------------------------------------------------------+-----------------------------------------------------+
| 4 | Value | Data value. | Encoding and Value cells for the ``namespace`` |
| | | | field type should be empty. Encoding and Value |
| | | | of ``namespace`` is fixed and is not configurable. |
| | | | Any values in these cells are ignored. |
+-----+-----------+----------------------------------------------------------------------+-----------------------------------------------------+
Type
Supported values are ``file``, ``data`` and ``namespace``.
.. note:: The first line of the CSV file should always be the column header and it is not configurable.
Encoding
Supported values are: ``u8``, ``i8``, ``u16``, ``u32``, ``i32``, ``string``, ``hex2bin``, ``base64`` and ``binary``. This specifies how actual data values are encoded in the resultant binary file. Difference between ``string`` and ``binary`` encoding is that ``string`` data is terminated with a NULL character, whereas ``binary`` data is not.
.. note:: For ``file`` type, only ``hex2bin``, ``base64``, ``string`` and ``binary`` is supported as of now.
Value
Data value.
.. note:: Encoding and Value cells for ``namespace`` field type should be empty. Encoding and Value of ``namespace`` is fixed and isn't configurable. Any value in these cells are ignored.
.. note:: First row of the CSV file should always be column header and isn't configurable.
Below is an example dump of such CSV file::
Below is an example dump of such a CSV file::
key,type,encoding,value <-- column header
namespace_name,namespace,, <-- First entry should be of type "namespace"
key1,data,u8,1
key2,file,string,/path/to/file
.. note:: Make sure there are no spaces before and after ',' or at the end of each line in CSV file.
.. note::
Make sure there are **no spaces**:
- before and after ','
- at the end of each line in a CSV file
NVS Entry and Namespace association
-----------------------------------
When a new namespace entry is encountered in the CSV file, each follow-up entries will be part of that namespace, until next namespace entry is found, in which case all the follow-up entries will be part of the new namespace.
When a namespace entry is encountered in a CSV file, each following entry will be treated as part of that namespace until the next namespace entry is found. At this point, all the following entries will be treated as part of the new namespace.
.. note:: First entry in a CSV file should always be a ``namespace`` entry.
.. note:: First entry in a CSV file should always be ``namespace`` entry.
Multipage Blob Support
----------------------
By default, binary blobs are allowed to span over multiple pages and written in the format mentioned in section :ref:`structure_of_entry`.
If older format is intended to be used, the utility provides an option to disable this feature.
By default, binary blobs are allowed to span over multiple pages and are written in the format mentioned in Section :ref:`structure_of_entry`.
If you intend to use an older format, the utility provides an option to disable this feature.
Encryption Support
-------------------
This utility allows you to create an enrypted binary file also. Encryption used is AES-XTS encryption. Refer to :ref:`nvs_encryption` for more details.
The NVS Partition Generator utility also allows you to create an encrypted binary file. The utility uses the AES-XTS encryption. Please refer to :ref:`nvs_encryption` for more details.
Running the utility
-------------------
@ -74,29 +91,32 @@ Running the utility
[--keyfile KEYFILE] [--outdir OUTDIR]
+------------------------+----------------------------------------------------------------------------------------------+
| Arguments | Description |
+========================+==============================================================================================+
| --input INPUT | Path to CSV file to parse. |
+------------------------+----------------------------------------------------------------------------------------------+
| --output OUTPUT | Path to output generated binary file. |
+------------------------+----------------------------------------------------------------------------------------------+
| --size SIZE | Size of NVS Partition in bytes (must be multiple of 4096) |
+------------------------+----------------------------------------------------------------------------------------------+
| --version {v1,v2} | Set version. Default: v2 |
+------------------------+----------------------------------------------------------------------------------------------+
| --keygen {true,false} | Generate keys for encryption. |
+------------------------+----------------------------------------------------------------------------------------------+
| --encrypt {true,false} | Set encryption mode. Default: false |
+------------------------+----------------------------------------------------------------------------------------------+
| --keyfile KEYFILE | File having key for encryption (Applicable only if encryption mode is true) |
+------------------------+----------------------------------------------------------------------------------------------+
| --outdir OUTDIR | The output directory to store the files created (Default: current directory) |
+------------------------+----------------------------------------------------------------------------------------------+
+------------------------+---------------------------------------------------+-------------------+
| Arguments | Description | Default Value |
+========================+===================================================+===================+
| --input INPUT | Path to a CSV file to parse. | |
+------------------------+---------------------------------------------------+-------------------+
| --output OUTPUT | Path to the generated binary file. | |
+------------------------+---------------------------------------------------+-------------------+
| --size SIZE | Size of NVS Partition in bytes | |
| | (must be multiple of 4096). | |
+------------------------+---------------------------------------------------+-------------------+
| --version {v1,v2} | Set version. | v2 |
+------------------------+---------------------------------------------------+-------------------+
| --keygen {true,false} | Generate keys for encryption. | |
+------------------------+---------------------------------------------------+-------------------+
| --encrypt {true,false} | Set encryption mode. Default: false. | false |
+------------------------+---------------------------------------------------+-------------------+
| --keyfile KEYFILE | File containing the key for encryption | |
| | (Applicable only if encryption mode is true). | |
+------------------------+---------------------------------------------------+-------------------+
| --outdir OUTDIR | The output directory to store the created files. | current directory |
+------------------------+---------------------------------------------------+-------------------+
You can run this utility in two modes:
- Default mode - Binary generated in this mode is an unencrypted binary file.
- Encryption mode - Binary generated in this mode is an encrypted binary file.
- **Default mode**: You get an unencrypted binary file.
- **Encryption mode**: You get an encrypted binary file.
**In default mode:**
@ -109,12 +129,11 @@ You can run this utility in two modes:
[--keygen {true,false}] [--encrypt {true,false}]
[--keyfile KEYFILE] [--outdir OUTDIR]
You can run the utility using below command::
You can run the utility using the command below::
python nvs_partition_gen.py --input sample.csv --output sample.bin --size 0x3000
**In encryption mode:**
-----------------------
@ -126,21 +145,21 @@ You can run the utility using below command::
[--version {v1,v2}] [--outdir OUTDIR]
You can run the utility using below commands:
You can run the utility using one of the commands below:
- By enabling generation of encryption keys::
- By enabling generation of encryption keys::
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keygen true
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keygen true
- By taking encryption keys as an input file. A sample encryption keys binary file is provided with the utility::
- By taking encryption keys as an input file. A sample binary file containing encryption keys is provided with the utility::
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keyfile testdata/sample_encryption_keys.bin
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keyfile testdata/sample_encryption_keys.bin
- By enabling generation of encryption keys and storing the keys in custom filename::
- By enabling generation of encryption keys and storing the keys in a binary file with a custom filename::
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keygen true --keyfile encryption_keys_generated.bin
python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keygen true --keyfile encryption_keys_generated.bin
.. note:: If `--keygen` is given with `--keyfile` argument, generated keys will be stored in `--keyfile` file. If `--keygen` argument is absent, `--keyfile` is taken as input file having key for encryption.
.. note:: If `--keygen` is given with the `--keyfile` argument, generated keys will be stored in the `--keyfile` file. If `--keygen` argument is absent, `--keyfile` is taken as input file containing encryption keys.
*To generate* **only** *encryption keys with this utility*::
@ -181,7 +200,7 @@ A sample CSV file is provided with the utility::
Caveats
-------
- Utility doesn't check for duplicate keys and will write data pertaining to both keys. User needs to make sure keys are distinct.
- Once a new page is created, no data will be written in the space left in previous page. Fields in the CSV file need to be ordered in such a way so as to optimize memory.
- Utility does not check for duplicate keys and will write data pertaining to both keys. You need to make sure that the keys are distinct.
- Once a new page is created, no data will be written in the space left on the previous page. Fields in the CSV file need to be ordered in such a way as to optimize memory.
- 64-bit datatype is not yet supported.

151
components/spi_flash/README.rst
View file

@ -1,80 +1,55 @@
SPI Flash APIs
==============
SPI Flash API
=============
Overview
--------
The spi_flash component contains APIs related to reading, writing, erasing,
memory mapping data in the external SPI flash. It also has higher-level
APIs which work with partitions defined in the :doc:`partition table </api-guides/partition-tables>`.
The spi_flash component contains API functions related to reading, writing, erasing, memory mapping for data in the external SPI flash. The spi_flash component also has higher-level API functions which work with partitions defined in the :doc:`partition table </api-guides/partition-tables>`.
Note that all the functionality is limited to the "main" SPI flash chip,
the same SPI flash chip from which program runs. For ``spi_flash_*`` functions,
this is a software limitation. The underlying ROM functions which work with SPI flash
do not have provisions for working with flash chips attached to SPI peripherals
other than SPI0.
Note that all the functionality is limited to the "main" SPI flash chip, the same SPI flash chip from which programs are runs. For ``spi_flash_*`` functions, this is a software limitation. The underlying ROM functions which work with SPI flash do not have provisions for working with flash chips attached to SPI peripherals other than SPI0.
SPI flash access APIs
---------------------
SPI flash access API
--------------------
This is the set of APIs for working with data in flash:
This is the set of API functions for working with data in flash:
- :cpp:func:`spi_flash_read` used to read data from flash to RAM
- :cpp:func:`spi_flash_write` used to write data from RAM to flash
- :cpp:func:`spi_flash_erase_sector` used to erase individual sectors of flash
- :cpp:func:`spi_flash_erase_range` used to erase range of addresses in flash
- :cpp:func:`spi_flash_read` reads data from flash to RAM
- :cpp:func:`spi_flash_write` writes data from RAM to flash
- :cpp:func:`spi_flash_erase_sector` erases individual sectors of flash
- :cpp:func:`spi_flash_erase_range` erases ranges of addresses in flash
- :cpp:func:`spi_flash_get_chip_size` returns flash chip size, in bytes, as configured in menuconfig
Generally, try to avoid using the raw SPI flash functions in favour of
:ref:`partition-specific functions <flash-partition-apis>`.
Generally, try to avoid using the raw SPI flash functions in favor of :ref:`partition-specific functions <flash-partition-apis>`.
SPI Flash Size
--------------
The SPI flash size is configured by writing a field in the software bootloader
image header, flashed at offset 0x1000.
The SPI flash size is configured by writing a field in the software bootloader image header, flashed at offset 0x1000.
By default, the SPI flash size is detected by esptool.py when this bootloader is
written to flash, and the header is updated with the correct
size. Alternatively, it is possible to generate a fixed flash size by setting
:envvar:`CONFIG_ESPTOOLPY_FLASHSIZE` in ``make menuconfig``.
By default, the SPI flash size is detected by esptool.py when this bootloader is written to flash, and the header is updated with the correct size. Alternatively, it is possible to generate a fixed flash size by setting :envvar:`CONFIG_ESPTOOLPY_FLASHSIZE` in ``make menuconfig``.
If it is necessary to override the configured flash size at runtime, is is
possible to set the ``chip_size`` member of ``g_rom_flashchip`` structure. This
size is used by ``spi_flash_*`` functions (in both software & ROM) for bounds
checking.
If it is necessary to override the configured flash size at runtime, it is possible to set the ``chip_size`` member of the ``g_rom_flashchip`` structure. This size is used by ``spi_flash_*`` functions (in both software & ROM) to check the bounds.
Concurrency Constraints
-----------------------
Because the SPI flash is also used for firmware execution (via the instruction &
data caches), these caches must be disabled while reading/writing/erasing. This
means that both CPUs must be running code from IRAM and only reading data from
DRAM while flash write operations occur.
Because the SPI flash is also used for firmware execution via the instruction & data caches, these caches must be disabled while reading/writing/erasing. This means that both CPUs must be running code from IRAM and must only be reading data from DRAM while flash write operations occur.
If you use the APIs documented here, then this happens automatically and
transparently. However note that it will have some performance impact on other
tasks in the system.
If you use the API functions documented here, then these constraints are applied automatically and transparently. However, note that it will have some performance impact on other tasks in the system.
Refer to the :ref:`application memory layout <memory-layout>` documentation for
an explanation of the differences between IRAM, DRAM and flash cache.
For differences between IRAM, DRAM, and flash cache, please refer to the :ref:`application memory layout <memory-layout>` documentation.
To avoid reading flash cache accidentally, when one CPU commences a flash write
or erase operation the other CPU is put into a blocked state and all
non-IRAM-safe interrupts are disabled on both CPUs, until the flash operation
completes.
To avoid reading flash cache accidentally, when one CPU initiates a flash write or erase operation, the other CPU is put into a blocked state, and all non-IRAM-safe interrupts are disabled on both CPUs until the flash operation completes.
If one CPU initiates a flash write or erase operation, the other CPU is put into a blocked state to avoid reading flash cache accidentally. All interrupts not safe for IRAM are disabled on both CPUs until the flash operation completes.
.. _iram-safe-interrupt-handlers:
IRAM-Safe Interrupt Handlers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you have an interrupt handler that you want to execute even when a flash
operation is in progress (for example, for low latency operations), set the
``ESP_INTR_FLAG_IRAM`` flag when the :doc:`interrupt handler is registered
</api-reference/system/intr_alloc>`.
If you have an interrupt handler that you want to execute while a flash operation is in progress (for example, for low latency operations), set the ``ESP_INTR_FLAG_IRAM`` flag when the :doc:`interrupt handler is registered </api-reference/system/intr_alloc>`.
You must ensure all data and functions accessed by these interrupt handlers are
located in IRAM or DRAM. This includes any functions that the handler calls.
You must ensure that all data and functions accessed by these interrupt handlers, including the ones that handlers call, are located in IRAM or DRAM.
Use the ``IRAM_ATTR`` attribute for functions::
@ -93,80 +68,56 @@ Use the ``DRAM_ATTR`` and ``DRAM_STR`` attributes for constant data::
const static char *MSG = DRAM_STR("I am a string stored in RAM");
}
Note that knowing which data should be marked with ``DRAM_ATTR`` can be hard,
the compiler will sometimes recognise that a variable or expression is constant
(even if it is not marked ``const``) and optimise it into flash, unless it is
marked with ``DRAM_ATTR``.
Note that knowing which data should be marked with ``DRAM_ATTR`` can be hard, the compiler will sometimes recognize that a variable or expression is constant (even if it is not marked ``const``) and optimize it into flash, unless it is marked with ``DRAM_ATTR``.
If a function or symbol is not correctly put into IRAM/DRAM and the interrupt
handler reads from the flash cache during a flash operation, it will cause a
crash due to Illegal Instruction exception (for code which should be in IRAM) or
garbage data to be read (for constant data which should be in DRAM).
If a function or symbol is not correctly put into IRAM/DRAM, and the interrupt handler reads from the flash cache during a flash operation, it will cause a crash due to Illegal Instruction exception (for code which should be in IRAM) or garbage data to be read (for constant data which should be in DRAM).
.. _flash-partition-apis:
Partition table APIs
--------------------
Partition table API
-------------------
ESP-IDF projects use a partition table to maintain information about various regions of
SPI flash memory (bootloader, various application binaries, data, filesystems).
More information about partition tables can be found :doc:`here </api-guides/partition-tables>`.
ESP-IDF projects use a partition table to maintain information about various regions of SPI flash memory (bootloader, various application binaries, data, filesystems). More information on partition tables can be found :doc:`here </api-guides/partition-tables>`.
This component provides APIs to enumerate partitions found in the partition table
and perform operations on them. These functions are declared in ``esp_partition.h``:
This component provides API functions to enumerate partitions found in the partition table and perform operations on them. These functions are declared in ``esp_partition.h``:
- :cpp:func:`esp_partition_find` used to search partition table for entries with
specific type, returns an opaque iterator
- :cpp:func:`esp_partition_get` returns a structure describing the partition, for the given iterator
- :cpp:func:`esp_partition_next` advances iterator to the next partition found
- :cpp:func:`esp_partition_iterator_release` releases iterator returned by ``esp_partition_find``
- :cpp:func:`esp_partition_find_first` is a convenience function which returns structure
describing the first partition found by ``esp_partition_find``
- :cpp:func:`esp_partition_read`, :cpp:func:`esp_partition_write`, :cpp:func:`esp_partition_erase_range`
are equivalent to :cpp:func:`spi_flash_read`, :cpp:func:`spi_flash_write`,
:cpp:func:`spi_flash_erase_range`, but operate within partition boundaries
- :cpp:func:`esp_partition_find` checks a partition table for entries with specific type, returns an opaque iterator.
- :cpp:func:`esp_partition_get` returns a structure describing the partition for a given iterator.
- :cpp:func:`esp_partition_next` shifts the iterator to the next found partition.
- :cpp:func:`esp_partition_iterator_release` releases iterator returned by ``esp_partition_find``.
- :cpp:func:`esp_partition_find_first` - a convenience function which returns the structure describing the first partition found by ``esp_partition_find``.
- :cpp:func:`esp_partition_read`, :cpp:func:`esp_partition_write`, :cpp:func:`esp_partition_erase_range` are equivalent to :cpp:func:`spi_flash_read`, :cpp:func:`spi_flash_write`, :cpp:func:`spi_flash_erase_range`, but operate within partition boundaries.
.. note::
Most application code should use these ``esp_partition_*`` APIs instead of lower level
``spi_flash_*`` APIs. Partition APIs do bounds checking and calculate correct
offsets in flash based on data stored in partition table.
Application code should mostly use these ``esp_partition_*`` API functions instead of lower level ``spi_flash_*`` API functions. Partition table API functions do bounds checking and calculate correct offsets in flash, based on data stored in a partition table.
SPI Flash Encryption
--------------------
It is possible to encrypt SPI flash contents, and have it transparenlty decrypted by hardware.
It is possible to encrypt the contents of SPI flash and have it transparently decrypted by hardware.
Refer to the :doc:`Flash Encryption documentation </security/flash-encryption>` for more details.
Memory mapping APIs
-------------------
Memory mapping API
------------------
ESP32 features memory hardware which allows regions of flash memory to be mapped
into instruction and data address spaces. This mapping works only for read operations,
it is not possible to modify contents of flash memory by writing to mapped memory
region. Mapping happens in 64KB pages. Memory mapping hardware can map up to
4 megabytes of flash into data address space, and up to 16 megabytes of flash into
instruction address space. See the technical reference manual for more details
about memory mapping hardware.
ESP32 features memory hardware which allows regions of flash memory to be mapped into instruction and data address spaces. This mapping works only for read operations. It is not possible to modify contents of flash memory by writing to a mapped memory region.
Note that some number of 64KB pages is used to map the application
itself into memory, so the actual number of available 64KB pages may be less.
Mapping happens in 64KB pages. Memory mapping hardware can map up to four megabytes of flash into data address space and up to 16 megabytes of flash into instruction address space. See the technical reference manual for more details about memory mapping hardware.
Reading data from flash using a memory mapped region is the only way to decrypt
contents of flash when :doc:`flash encryption </security/flash-encryption>` is enabled.
Decryption is performed at hardware level.
Note that some 64KB pages are used to map the application itself into memory, so the actual number of available 64KB pages may be less.
Memory mapping APIs are declared in ``esp_spi_flash.h`` and ``esp_partition.h``:
Reading data from flash using a memory mapped region is the only way to decrypt contents of flash when :doc:`flash encryption </security/flash-encryption>` is enabled. Decryption is performed at the hardware level.
- :cpp:func:`spi_flash_mmap` maps a region of physical flash addresses into instruction space or data space of the CPU
- :cpp:func:`spi_flash_munmap` unmaps previously mapped region
- :cpp:func:`esp_partition_mmap` maps part of a partition into the instruction space or data space of the CPU
Memory mapping API are declared in ``esp_spi_flash.h`` and ``esp_partition.h``:
- :cpp:func:`spi_flash_mmap` maps a region of physical flash addresses into instruction space or data space of the CPU.
- :cpp:func:`spi_flash_munmap` unmaps previously mapped region.
- :cpp:func:`esp_partition_mmap` maps part of a partition into the instruction space or data space of the CPU.
Differences between :cpp:func:`spi_flash_mmap` and :cpp:func:`esp_partition_mmap` are as follows:
- :cpp:func:`spi_flash_mmap` must be given a 64KB aligned physical address
- :cpp:func:`esp_partition_mmap` may be given any arbitrary offset within the partition,
it will adjust returned pointer to mapped memory as necessary
- :cpp:func:`spi_flash_mmap` must be given a 64KB aligned physical address.
- :cpp:func:`esp_partition_mmap` may be given any arbitrary offset within the partition, it will adjust the returned pointer to mapped memory as necessary
Note that because memory mapping happens in 64KB blocks, it may be possible to
read data outside of the partition provided to ``esp_partition_mmap``.
Note that since memory mapping happens in 64KB blocks, it may be possible to read data outside of the partition provided to ``esp_partition_mmap``.

81
components/vfs/README.rst
View file

@ -4,18 +4,17 @@ Virtual filesystem component
Overview
--------
Virtual filesystem (VFS) component provides a unified interface for drivers which can perform operations on file-like objects. This can be a real filesystems (FAT, SPIFFS, etc.), or device drivers which exposes file-like interface.
Virtual filesystem (VFS) component provides a unified interface for drivers which can perform operations on file-like objects. These can be real filesystems (FAT, SPIFFS, etc.) or device drivers which provide a file-like interface.
This component allows C library functions, such as fopen and fprintf, to work with FS drivers. At high level, each FS driver is associated with some path prefix. When one of C library functions needs to open a file, VFS component searches for the FS driver associated with the file's path, and forwards the call to that driver. VFS also forwards read, write, and other calls for the given file to the same FS driver.
This component allows C library functions, such as fopen and fprintf, to work with FS drivers. At a high level, each FS driver is associated with some path prefix. When one of C library functions needs to open a file, the VFS component searches for the FS driver associated with the file path and forwards the call to that driver. VFS also forwards read, write, and other calls for the given file to the same FS driver.
For example, one can register a FAT filesystem driver with the ``/fat`` prefix and call ``fopen("/fat/file.txt", "w")``. The VFS component will then call the function ``open`` of the FAT driver and pass the argument ``/file.txt`` to it together with appropriate mode flags. All subsequent calls to C library functions for the returned ``FILE*`` stream will also be forwarded to the FAT driver.
For example, one can register a FAT filesystem driver with ``/fat`` prefix, and call ``fopen("/fat/file.txt", "w")``. VFS component will then call ``open`` function of FAT driver and pass ``/file.txt`` argument to it (and appropriate mode flags). All subsequent calls to C library functions for the returned ``FILE*`` stream will also be forwarded to the FAT driver.
FS registration
---------------
To register an FS driver, application needs to define in instance of :cpp:type:`esp_vfs_t` structure and populate it with function pointers to FS APIs:
To register an FS driver, an application needs to define an instance of the :cpp:type:`esp_vfs_t` structure and populate it with function pointers to FS APIs:
.. highlight:: c
@ -32,9 +31,9 @@ To register an FS driver, application needs to define in instance of :cpp:type:`
ESP_ERROR_CHECK(esp_vfs_register("/data", &myfs, NULL));
Depending on the way FS driver declares its APIs, either ``read``, ``write``, etc., or ``read_p``, ``write_p``, etc. should be used.
Depending on the way how the FS driver declares its API functions, either ``read``, ``write``, etc., or ``read_p``, ``write_p``, etc., should be used.
Case 1: API functions are declared without an extra context pointer (FS driver is a singleton)::
Case 1: API functions are declared without an extra context pointer (the FS driver is a singleton)::
ssize_t myfs_write(int fd, const void * data, size_t size);
@ -46,7 +45,7 @@ Case 1: API functions are declared without an extra context pointer (FS driver i
// When registering FS, context pointer (third argument) is NULL:
ESP_ERROR_CHECK(esp_vfs_register("/data", &myfs, NULL));
Case 2: API functions are declared with an extra context pointer (FS driver supports multiple instances)::
Case 2: API functions are declared with an extra context pointer (the FS driver supports multiple instances)::
ssize_t myfs_write(myfs_t* fs, int fd, const void * data, size_t size);
@ -68,8 +67,8 @@ Synchronous input/output multiplexing
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you want to use synchronous input/output multiplexing by :cpp:func:`select`
then you need to register the VFS with :cpp:func:`start_select` and
:cpp:func:`end_select` functions similarly to the following example:
then you need to register VFS with the functions :cpp:func:`start_select` and
:cpp:func:`end_select` similar to the following example:
.. highlight:: c
@ -85,22 +84,27 @@ detection of read/write/error conditions on file descriptors belonging to the
given VFS. :cpp:func:`end_select` is called to stop/deinitialize/free the
environment which was setup by :cpp:func:`start_select`. Please refer to the
reference implementation for the UART peripheral in
:component_file:`vfs/vfs_uart.c` and most particularly to functions
:cpp:func:`esp_vfs_dev_uart_register`, :cpp:func:`uart_start_select` and
:component_file:`vfs/vfs_uart.c` and most particularly to the functions
:cpp:func:`esp_vfs_dev_uart_register`, :cpp:func:`uart_start_select`, and
:cpp:func:`uart_end_select`.
Examples demonstrating the use of :cpp:func:`select` with VFS file descriptors
are the :example:`peripherals/uart_select` and the :example:`system/select`
examples.
Please check the following examples that demonstrate the use of :cpp:func:`select` with VFS file descriptors:
- :example:`peripherals/uart_select`
- :example:`system/select`
<<<<<<< HEAD
If :cpp:func:`select` is used for socket file descriptors only then one can
enable the :envvar:`CONFIG_LWIP_USE_ONLY_LWIP_SELECT` option which can reduce the code
=======
If you use :cpp:func:`select` for socket file descriptors, you can enable the :envvar:`CONFIG_LWIP_USE_ONLY_LWIP_SELECT` option to reduce the code
>>>>>>> afc2fdf27... Review all the files in the esp-idf's api_ref/storage directory
size and improve performance.
Paths
-----
Each registered FS has a path prefix associated with it. This prefix may be considered a "mount point" of this partition.
Each registered FS has a path prefix associated with it. This prefix can be considered as a "mount point" of this partition.
In case when mount points are nested, the mount point with the longest matching path prefix is used when opening the file. For instance, suppose that the following filesystems are registered in VFS:
@ -111,45 +115,49 @@ Then:
- FS 1 will be used when opening a file called ``/data/log.txt``
- FS 2 will be used when opening a file called ``/data/static/index.html``
- Even if ``/index.html"`` doesn't exist in FS 2, FS 1 will *not* be searched for ``/static/index.html``.
- Even if ``/index.html"`` does not exist in FS 2, FS 1 will *not* be searched for ``/static/index.html``.
As a general rule, mount point names must start with the path separator (``/``) and must contain at least one character after path separator. However an empty mount point name is also supported, and may be used in cases when application needs to provide "fallback" filesystem, or override VFS functionality altogether. Such filesystem will be used if no prefix matches the path given.
As a general rule, mount point names must start with the path separator (``/``) and must contain at least one character after path separator. However, an empty mount point name is also supported and might be used in cases when an application needs to provide a "fallback" filesystem or to override VFS functionality altogether. Such filesystem will be used if no prefix matches the path given.
VFS does not handle dots (``.``) in path names in any special way. VFS does not treat ``..`` as a reference to the parent directory. I.e. in the above example, using a path ``/data/static/../log.txt`` will not result in a call to FS 1 to open ``/log.txt``. Specific FS drivers (such as FATFS) may handle dots in file names differently.
VFS does not handle dots (``.``) in path names in any special way. VFS does not treat ``..`` as a reference to the parent directory. In the above example, using a path ``/data/static/../log.txt`` will not result in a call to FS 1 to open ``/log.txt``. Specific FS drivers (such as FATFS) might handle dots in file names differently.
When opening files, FS driver will only be given relative path to files. For example:
When opening files, the FS driver receives only relative paths to files. For example:
- ``myfs`` driver is registered with ``/data`` as path prefix
- and application calls ``fopen("/data/config.json", ...)``
- then VFS component will call ``myfs_open("/config.json", ...)``.
- ``myfs`` driver will open ``/config.json`` file
1. The ``myfs`` driver is registered with ``/data`` as a path prefix.
2. The application calls ``fopen("/data/config.json", ...)``.
3. The VFS component calls ``myfs_open("/config.json", ...)``.
4. The ``myfs`` driver opens the ``/config.json`` file.
VFS does not impose any limit on total file path length, but it does limit the FS path prefix to ``ESP_VFS_PATH_MAX`` characters. Individual FS drivers may have their own filename length limitations.
VFS doesn't impose a limit on total file path length, but it does limit FS path prefix to ``ESP_VFS_PATH_MAX`` characters. Individual FS drivers may have their own filename length limitations.
File descriptors
----------------
File descriptors are small positive integers from ``0`` to ``FD_SETSIZE - 1`` where ``FD_SETSIZE`` is defined in newlib's ``sys/types.h``. The largest file descriptors (configured by ``CONFIG_LWIP_MAX_SOCKETS``) are reserved for sockets. The VFS component contains a lookup-table called ``s_fd_table`` for mapping global file descriptors to VFS driver indexes registered in the ``s_vfs`` array.
File descriptors are small positive integers from ``0`` to ``FD_SETSIZE - 1``, where ``FD_SETSIZE`` is defined in newlib's ``sys/types.h``. The largest file descriptors (configured by ``CONFIG_LWIP_MAX_SOCKETS``) are reserved for sockets. The VFS component contains a lookup-table called ``s_fd_table`` for mapping global file descriptors to VFS driver indexes registered in the ``s_vfs`` array.
Standard IO streams (stdin, stdout, stderr)
-------------------------------------------
If "UART for console output" menuconfig option is not set to "None", then ``stdin``, ``stdout``, and ``stderr`` are configured to read from, and write to, a UART. It is possible to use UART0 or UART1 for standard IO. By default, UART0 is used, with 115200 baud rate, TX pin is GPIO1 and RX pin is GPIO3. These parameters can be changed in menuconfig.
If the menuconfig option ``UART for console output`` is not set to ``None``, then ``stdin``, ``stdout``, and ``stderr`` are configured to read from, and write to, a UART. It is possible to use UART0 or UART1 for standard IO. By default, UART0 is used with 115200 baud rate; TX pin is GPIO1; RX pin is GPIO3. These parameters can be changed in menuconfig.
Writing to ``stdout`` or ``stderr`` will send characters to the UART transmit FIFO. Reading from ``stdin`` will retrieve characters from the UART receive FIFO.
By default, VFS uses simple functions for reading from and writing to UART. Writes busy-wait until all data is put into UART FIFO, and reads are non-blocking, returning only the data present in the FIFO. Because of this non-blocking read behavior, higher level C library calls, such as ``fscanf("%d\n", &var);`` may not have desired results.
By default, VFS uses simple functions for reading from and writing to UART. Writes busy-wait until all data is put into UART FIFO, and reads are non-blocking, returning only the data present in the FIFO. Due to this non-blocking read behavior, higher level C library calls, such as ``fscanf("%d\n", &var);``, might not have desired results.
Applications which use UART driver may instruct VFS to use the driver's interrupt driven, blocking read and write functions instead. This can be done using a call to ``esp_vfs_dev_uart_use_driver`` function. It is also possible to revert to the basic non-blocking functions using a call to ``esp_vfs_dev_uart_use_nonblocking``.
Applications which use the UART driver can instruct VFS to use the driver's interrupt driven, blocking read and write functions instead. This can be done using a call to the ``esp_vfs_dev_uart_use_driver`` function. It is also possible to revert to the basic non-blocking functions using a call to ``esp_vfs_dev_uart_use_nonblocking``.
VFS also provides optional newline conversion feature for input and output. Internally, most applications send and receive lines terminated by LF (''\n'') character. Different terminal programs may require different line termination, such as CR or CRLF. Applications can configure this separately for input and output either via menuconfig, or by calls to ``esp_vfs_dev_uart_set_rx_line_endings`` and ``esp_vfs_dev_uart_set_tx_line_endings`` functions.
VFS also provides an optional newline conversion feature for input and output. Internally, most applications send and receive lines terminated by the LF (''\n'') character. Different terminal programs may require different line termination, such as CR or CRLF. Applications can configure this separately for input and output either via menuconfig, or by calls to the functions ``esp_vfs_dev_uart_set_rx_line_endings`` and ``esp_vfs_dev_uart_set_tx_line_endings``.
Standard streams and FreeRTOS tasks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``FILE`` objects for ``stdin``, ``stdout``, and ``stderr`` are shared between all FreeRTOS tasks, but the pointers to these objects are are stored in per-task ``struct _reent``. The following code:
``FILE`` objects for ``stdin``, ``stdout``, and ``stderr`` are shared between all FreeRTOS tasks, but the pointers to these objects are stored in per-task ``struct _reent``.
The following code is transferred to ``fprintf(__getreent()->_stderr, "42\n");`` by the preprocessor:
.. highlight:: c
@ -157,14 +165,11 @@ Standard streams and FreeRTOS tasks
fprintf(stderr, "42\n");
actually is translated to to this (by the preprocessor)::
fprintf(__getreent()->_stderr, "42\n");
where the ``__getreent()`` function returns a per-task pointer to ``struct _reent`` (:component_file:`newlib/include/sys/reent.h#L370-L417`). This structure is allocated on the TCB of each task. When a task is initialized, ``_stdin``, ``_stdout`` and ``_stderr`` members of ``struct _reent`` are set to the values of ``_stdin``, ``_stdout`` and ``_stderr`` of ``_GLOBAL_REENT`` (i.e. the structure which is used before FreeRTOS is started).
The ``__getreent()`` function returns a per-task pointer to ``struct _reent`` (:component_file:`newlib/include/sys/reent.h#L370-L417`). This structure is allocated on the TCB of each task. When a task is initialized, ``_stdin``, ``_stdout``, and ``_stderr`` members of ``struct _reent`` are set to the values of ``_stdin``, ``_stdout``, and ``_stderr`` of ``_GLOBAL_REENT`` (i.e., the structure which is used before FreeRTOS is started).
Such a design has the following consequences:
- It is possible to set ``stdin``, ``stdout``, and ``stderr`` for any given task without affecting other tasks, e.g. by doing ``stdin = fopen("/dev/uart/1", "r")``.
- Closing default ``stdin``, ``stdout``, or ``stderr`` using ``fclose`` will close the ``FILE`` stream object — this will affect all other tasks.
- It is possible to set ``stdin``, ``stdout``, and ``stderr`` for any given task without affecting other tasks, e.g., by doing ``stdin = fopen("/dev/uart/1", "r")``.
- Closing default ``stdin``, ``stdout``, or ``stderr`` using ``fclose`` will close the ``FILE`` stream object, which will affect all other tasks.
- To change the default ``stdin``, ``stdout``, ``stderr`` streams for new tasks, modify ``_GLOBAL_REENT->_stdin`` (``_stdout``, ``_stderr``) before creating the task.

70
components/wear_levelling/README.rst
View file

@ -1,55 +1,47 @@
Wear Levelling APIs
===================
Wear Levelling API
==================
Overview
--------
Most of the flash devices and specially SPI flash devices that are used in ESP32
have sector based organization and have limited amount of erase/modification cycles
per memory sector. To avoid situation when one sector reach the limit of erases when
other sectors was used not often, we have made a component that avoid this situation.
The wear levelling component share the amount of erases between all sectors in the
memory without user interaction.
The wear_levelling component contains APIs related to reading, writing, erasing,
memory mapping data in the external SPI flash through the partition component. It
also has higher-level APIs which work with FAT filesystem defined in
the :doc:`FAT filesystem </api-reference/storage/fatfs>`.
Most of flash memory and especially SPI flash that is used in ESP32 has a sector-based organization and also has a limited number of erase/modification cycles per memory sector. The wear levelling component helps to distribute wear and tear among sectors more evenly without requiring any attention from the user.
The wear levelling component, together with FAT FS component, works with FAT FS sector size 4096
bytes which is standard size of the flash devices. In this mode the component has best performance,
but needs additional memoty in the RAM. To save internal memory the component has two additional modes
to work with sector size 512 bytes: Performance and Safety modes. In Performance mode by erase sector
operation data will be stored to the RAM, sector will be erased and then data will be stored
back to the flash. If by this operation power off situation will occur, the complete 4096 bytes
will be lost. To prevent this the Safety mode was implemented. In safety mode the data will be first
stored to the flash and after sector will be erased, will be stored back. If power off situation will
occur, after power on, the data will be recovered.
By default defined the sector size 512 bytes and Performance mode. To change these values please use
the configuration menu.
The wear levelling component provides API functions related to reading, writing, erasing, and memory mapping of data in external SPI flash through the partition component. The component also has higher-level API functions which work with the FAT filesystem defined in :doc:`FAT filesystem </api-reference/storage/fatfs>`.
The wear levelling component, together with the FAT FS component, uses FAT FS sectors of 4096 bytes, which is a standard size for flash memory. With this size, the component shows the best performance but needs additional memory in RAM.
To save internal memory, the component has two additional modes which both use sectors of 512 bytes:
- **Performance mode.** Erase sector operation data is stored in RAM, the sector is erased, and then data is copied back to flash memory. However, if a device is powered off for any reason, all 4096 bytes of data is lost.
- **Safety mode.** The data is first saved to flash memory, and after the sector is erased, the data is saved back. If a device is powered off, the data can be recovered as soon as the device boots up.
The default settings are as follows:
- Sector size is 512 bytes
- Performance mode
You can change the settings through the configuration menu.
The wear levelling component does not cache data in RAM. Write and erase functions
modify flash directly, and flash contents is consistent when the function returns.
The wear levelling component does not cache data in RAM. The write and erase functions modify flash directly, and flash contents are consistent when the function returns.
Wear Levelling access APIs
--------------------------
Wear Levelling access API functions
-----------------------------------
This is the set of APIs for working with data in flash:
This is the set of API functions for working with data in flash:
- ``wl_mount`` mount wear levelling module for defined partition
- ``wl_unmount`` used to unmount levelling module
- ``wl_erase_range`` used to erase range of addresses in flash
- ``wl_write`` used to write data to the partition
- ``wl_read`` used to read data from the partition
- ``wl_size`` return size of avalible memory in bytes
- ``wl_sector_size`` returns size of one sector
- ``wl_mount`` - initializes the wear levelling module and mounts the specified partition
- ``wl_unmount`` - unmounts the partition and deinitializes the wear levelling module
- ``wl_erase_range`` - erases a range of addresses in flash
- ``wl_write`` - writes data to a partition
- ``wl_read`` - reads data from a partition
- ``wl_size`` - returns the size of available memory in bytes
- ``wl_sector_size`` - returns the size of one sector
As a rule, try to avoid using raw wear levelling functions and use filesystem-specific functions instead.
Generally, try to avoid using the raw wear levelling functions in favor of
filesystem-specific functions.
Memory Size
-----------
The memory size calculated in the wear Levelling module based on parameters of
partition. The module use few sectors of flash for internal data.
The memory size is calculated in the wear levelling module based on partition parameters. The module uses some sectors of flash for internal data.

44
docs/en/api-reference/storage/fatfs.rst
View file

@ -1,36 +1,43 @@
FAT Filesystem Support
======================
ESP-IDF uses `FatFs <http://elm-chan.org/fsw/ff/00index_e.html>`_ library to work with FAT filesystems. FatFs library resides in ``fatfs`` component. Although it can be used directly, many of its features can be accessed via VFS using C standard library and POSIX APIs.
ESP-IDF uses the `FatFs <http://elm-chan.org/fsw/ff/00index_e.html>`_ library to work with FAT filesystems. FatFs resides in the ``fatfs`` component. Although the library can be used directly, many of its features can be accessed via VFS, using the C standard library and POSIX API functions.
Additionally, FatFs has been modified to support the runtime pluggable disk I/O layer. This allows mapping of FatFs drives to physical disks at runtime.
Additionally, FatFs has been modified to support run-time pluggable disk IO layer. This allows mapping of FatFs drives to physical disks at run-time.
Using FatFs with VFS
--------------------
:component_file:`fatfs/src/esp_vfs_fat.h` header file defines functions to connect FatFs with VFS. :cpp:func:`esp_vfs_fat_register` function allocates a ``FATFS`` structure, and registers a given path prefix in VFS. Subsequent operations on files starting with this prefix are forwarded to FatFs APIs. :cpp:func:`esp_vfs_fat_unregister_path` function deletes the registration with VFS, and frees the ``FATFS`` structure.
The header file :component_file:`fatfs/src/esp_vfs_fat.h` defines the functions for connecting FatFs and VFS.
Most applications will use the following flow when working with ``esp_vfs_fat_`` functions:
The function :cpp:func:`esp_vfs_fat_register` allocates a ``FATFS`` structure and registers a given path prefix in VFS. Subsequent operations on files starting with this prefix are forwarded to FatFs APIs.
The function :cpp:func:`esp_vfs_fat_unregister_path` deletes the registration with VFS, and frees the ``FATFS`` structure.
1. Call :cpp:func:`esp_vfs_fat_register`, specifying path prefix where the filesystem has to be mounted (e.g. ``"/sdcard"``, ``"/spiflash"``), FatFs drive number, and a variable which will receive a pointer to ``FATFS`` structure.
Most applications use the following workflow when working with ``esp_vfs_fat_`` functions:
2. Call :cpp:func:`ff_diskio_register` function to register disk IO driver for the drive number used in step 1.
1. Call :cpp:func:`esp_vfs_fat_register` to specify:
- Path prefix where to mount the filesystem (e.g. ``"/sdcard"``, ``"/spiflash"``)
- FatFs drive number
- A variable which will receive the pointer to the ``FATFS`` structure
3. Call FatFs ``f_mount`` function (and optionally ``f_fdisk``, ``f_mkfs``) to mount the filesystem using the same drive number which was passed to :cpp:func:`esp_vfs_fat_register`. See `FatFs documentation for more details <http://www.elm-chan.org/fsw/ff/doc/mount.html>`.
2. Call :cpp:func:`ff_diskio_register` to register the disk I/O driver for the drive number used in Step 1.
4. Call POSIX and C standard library functions to open, read, write, erase, copy files, etc. Use paths starting with the prefix passed to :cpp:func:`esp_vfs_register` (such as ``"/sdcard/hello.txt"``).
3. Call the FatFs function ``f_mount``, and optionally ``f_fdisk``, ``f_mkfs``, to mount the filesystem using the same drive number which was passed to :cpp:func:`esp_vfs_fat_register`. For more information, see `FatFs documentation <http://www.elm-chan.org/fsw/ff/doc/mount.html>`.
5. Optionally, call FatFs library functions directly. Use paths without a VFS prefix in this case (``"/hello.txt"``).
4. Call the C standard library and POSIX API functions to perform such actions on files as open, read, write, erase, copy, etc. Use paths starting with the path prefix passed to :cpp:func:`esp_vfs_register` (for example, ``"/sdcard/hello.txt"``).
5. Optionally, call the FatFs library functions directly. In this case, use paths without a VFS prefix (for example, ``"/hello.txt"``).
6. Close all open files.
7. Call FatFs ``f_mount`` function for the same drive number, with NULL ``FATFS*`` argument, to unmount the filesystem.
7. Call the FatFs function ``f_mount`` for the same drive number, with NULL ``FATFS*`` argument, to unmount the filesystem.
8. Call FatFs :cpp:func:`ff_diskio_register` with NULL ``ff_diskio_impl_t*`` argument and the same drive number.
8. Call the FatFs function :cpp:func:`ff_diskio_register` with NULL ``ff_diskio_impl_t*`` argument and the same drive number to unregister the disk I/O driver.
9. Call :cpp:func:`esp_vfs_fat_unregister_path` with the path where the file system is mounted to remove FatFs from VFS, and free the ``FATFS`` structure allocated on step 1.
9. Call :cpp:func:`esp_vfs_fat_unregister_path` with the path where the file system is mounted to remove FatFs from VFS, and free the ``FATFS`` structure allocated in Step 1.
Convenience functions, ``esp_vfs_fat_sdmmc_mount`` and ``esp_vfs_fat_sdmmc_unmount``, which wrap these steps and also handle SD card initialization, are described in the next section.
The convenience functions ``esp_vfs_fat_sdmmc_mount`` and ``esp_vfs_fat_sdmmc_unmount`` wrap the steps described above and also handle SD card initialization. These two functions are described in the next section.
.. doxygenfunction:: esp_vfs_fat_register
.. doxygenfunction:: esp_vfs_fat_unregister_path
@ -39,7 +46,9 @@ Convenience functions, ``esp_vfs_fat_sdmmc_mount`` and ``esp_vfs_fat_sdmmc_unmou
Using FatFs with VFS and SD cards
---------------------------------
:component_file:`fatfs/src/esp_vfs_fat.h` header file also provides a convenience function to perform steps 1–3 and 7–9, and also handle SD card initialization: :cpp:func:`esp_vfs_fat_sdmmc_mount`. This function does only limited error handling. Developers are encouraged to look at its source code and incorporate more advanced versions into production applications. :cpp:func:`esp_vfs_fat_sdmmc_unmount` function unmounts the filesystem and releases resources acquired by :cpp:func:`esp_vfs_fat_sdmmc_mount`.
The header file :component_file:`fatfs/src/esp_vfs_fat.h` defines convenience functions :cpp:func:`esp_vfs_fat_sdmmc_mount` and :cpp:func:`esp_vfs_fat_sdmmc_unmount`. These function perform Steps 1–3 and 7–9 respectively and handle SD card initialization, but provide only limited error handling. Developers are encouraged to check its source code and incorporate more advanced features into production applications.
The convenience function :cpp:func:`esp_vfs_fat_sdmmc_unmount` unmounts the filesystem and releases the resources acquired by :cpp:func:`esp_vfs_fat_sdmmc_mount`.
.. doxygenfunction:: esp_vfs_fat_sdmmc_mount
.. doxygenstruct:: esp_vfs_fat_mount_config_t
@ -50,17 +59,18 @@ Using FatFs with VFS and SD cards
Using FatFs with VFS in read-only mode
--------------------------------------
Convenience functions, :cpp:func:`esp_vfs_fat_rawflash_mount` and :cpp:func:`esp_vfs_fat_rawflash_unmount`, are provided by :component_file:`fatfs/src/esp_vfs_fat.h` header file in order to perform steps 1-3 and 7-9 for read-only FAT partitions. These are particularly helpful for data partitions written only once during factory provisioning and need not be changed by production application throughout the lifetime.
The header file :component_file:`fatfs/src/esp_vfs_fat.h` also defines the convenience functions :cpp:func:`esp_vfs_fat_rawflash_mount` and :cpp:func:`esp_vfs_fat_rawflash_unmount`. These functions perform Steps 1-3 and 7-9 respectively for read-only FAT partitions. These are particularly helpful for data partitions written only once during factory provisioning which will not be changed by production application throughout the lifetime of the hardware.
.. doxygenfunction:: esp_vfs_fat_rawflash_mount
.. doxygenfunction:: esp_vfs_fat_rawflash_unmount
FatFS disk IO layer
-------------------
FatFs has been extended with an API to register disk IO driver at runtime.
FatFs has been extended with API functions that register the disk I/O driver at runtime.
Implementation of disk IO functions for SD/MMC cards is provided. It can be registered for the given FatFs drive number using :cpp:func:`ff_diskio_register_sdmmc` function.
They provide implementation of disk I/O functions for SD/MMC cards and can be registered for the given FatFs drive number using the function :cpp:func:`ff_diskio_register_sdmmc`.
.. doxygenfunction:: ff_diskio_register
.. doxygenstruct:: ff_diskio_impl_t

2
docs/en/api-reference/storage/index.rst
View file

@ -15,4 +15,4 @@ Storage API
Mass Manufacturing Utility <mass_mfg.rst>
Example code for this API section is provided in :example:`storage` directory of ESP-IDF examples.
Code examples for this API section are provided in the :example:`storage` directory of ESP-IDF examples.

18
docs/en/api-reference/storage/nvs_flash.rst
View file

@ -3,29 +3,29 @@
NVS Partition Generator Utility
-------------------------------
This utility helps in generating NVS-esque partition binary file which can be flashed separately on a dedicated partition via a flashing utility. Key-value pairs to be flashed onto the partition can be provided via a CSV file. Refer to :doc:`NVS Partition Generator Utility <nvs_partition_gen>` for more details.
This utility helps generate NVS partition binary files which can be flashed separately on a dedicated partition via a flashing utility. Key-value pairs to be flashed onto the partition can be provided via a CSV file. For more details, please refer to :doc:`NVS Partition Generator Utility <nvs_partition_gen>`.
Application Example
-------------------
Two examples are provided in :example:`storage` directory of ESP-IDF examples:
You can find two code examples in the :example:`storage` directory of ESP-IDF examples:
:example:`storage/nvs_rw_value`
Demonstrates how to read and write a single integer value using NVS.
Demonstrates how to read a single integer value from, and write it to NVS.
The value holds the number of ESP32 module restarts. Since it is written to NVS, the value is preserved between restarts.
The value checked in this example holds the number of the ESP32 module restarts. The value's function as a counter is only possible due to its storing in NVS.
Example also shows how to check if read / write operation was successful, or certain value is not initialized in NVS. Diagnostic is provided in plain text to help track program flow and capture any issues on the way.
The example also shows how to check if a read / write operation was successful, or if a certain value has not been initialized in NVS. The diagnostic procedure is provided in plain text to help you track the program flow and capture any issues on the way.
:example:`storage/nvs_rw_blob`
Demonstrates how to read and write a single integer value and a blob (binary large object) using NVS to preserve them between ESP32 module restarts.
Demonstrates how to read a single integer value and a blob (binary large object), and write them to NVS to preserve this value between ESP32 module restarts.
* value - tracks number of ESP32 module soft and hard restarts.
* blob - contains a table with module run times. The table is read from NVS to dynamically allocated RAM. New run time is added to the table on each manually triggered soft restart and written back to NVS. Triggering is done by pulling down GPIO0.
* value - tracks the number of the ESP32 module soft and hard restarts.
* blob - contains a table with module run times. The table is read from NVS to dynamically allocated RAM. A new run time is added to the table on each manually triggered soft restart, and then the added run time is written to NVS. Triggering is done by pulling down GPIO0.
Example also shows how to implement diagnostics if read / write operation was successful.
The example also shows how to implement the diagnostic procedure to check if the read / write operation was successful.
API Reference

98
docs/en/api-reference/storage/sdmmc.rst
View file

@ -4,77 +4,93 @@ SD/SDIO/MMC Driver
Overview
--------
SD/SDIO/MMC driver currently supports SD memory, SDIO cards, and eMMC chips. This protocol level driver builds on top of SDMMC and SD SPI host drivers.
The SD/SDIO/MMC driver currently supports SD memory, SDIO cards, and eMMC chips. This is a protocol level driver built on top of SDMMC and SD SPI host drivers.
SDMMC and SD SPI host drivers (``driver/sdmmc_host.h``) provide APIs to send commands to the slave device(s), send and receive data, and handle error conditions on the bus.
- See :doc:`SDMMC Host API <../peripherals/sdmmc_host>` for functions used to initialize and configure SDMMC host.
- See :doc:`SD SPI Host API <../peripherals/sdspi_host>` for functions used to initialize and configure SD SPI host.
SDMMC and SD SPI host drivers (:component:`driver/include/driver/sdmmc_host.h`) provide API functions for:
SDMMC protocol layer (``sdmmc_cmd.h``), described in this document, handles specifics of SD protocol such as card initialization and data transfer commands.
- Sending commands to slave devices
- Sending and receiving data
- Handling error conditions within the bus
For functions used to initialize and configure:
- SDMMC host, see :doc:`SDMMC Host API <../peripherals/sdmmc_host>`
- SD SPI host, see :doc:`SD SPI Host API <../peripherals/sdspi_host>`
The SDMMC protocol layer described in this document handles the specifics of the SD protocol, such as the card initialization and data transfer commands.
The protocol layer works with the host via the :cpp:class:`sdmmc_host_t` structure. This structure contains pointers to various functions of the host.
Protocol layer works with the host via :cpp:class:`sdmmc_host_t` structure. This structure contains pointers to various functions of the host.
Application Example
-------------------
An example which combines SDMMC driver with FATFS library is provided in ``examples/storage/sd_card`` directory. This example initializes the card, writes and reads data from it using POSIX and C library APIs. See README.md file in the example directory for more information.
An example which combines the SDMMC driver with the FATFS library is provided in the :example:`storage/sd_card` directory of ESP-IDF examples. This example initializes the card, then writes and reads data from it using POSIX and C library APIs. See README.md file in the example directory for more information.
Protocol layer APIs
-------------------
Protocol layer is given :cpp:class:`sdmmc_host_t` structure which describes the SD/MMC host driver, lists its capabilites, and provides pointers to functions of the driver. Protocol layer stores card-specific information in :cpp:class:`sdmmc_card_t` structure. When sending commands to the SD/MMC host driver, protocol layer uses :cpp:class:`sdmmc_command_t` structure to describe the command, argument, expected return value, and data to transfer, if any.
Protocol layer API
------------------
Usage with SD memory cards
^^^^^^^^^^^^^^^^^^^^^^^^^^
1. Call the host driver functions to initialize the host (e.g. :cpp:func:`sdmmc_host_init`, :cpp:func:`sdmmc_host_init_slot`).
2. Call :cpp:func:`sdmmc_card_init` to initialize the card, passing it host driver information (``host``) and a pointer to :cpp:class:`sdmmc_card_t` structure which will be filled in (``card``).
3. To read and write sectors of the card, use :cpp:func:`sdmmc_read_sectors` and :cpp:func:`sdmmc_write_sectors`, passing the pointer to card information structure (``card``).
4. When card is not used anymore, call the host driver function to disable the host peripheral and free resources allocated by the driver (e.g. :cpp:func:`sdmmc_host_deinit`).
Usage with eMMC chips
^^^^^^^^^^^^^^^^^^^^^
From the perspective of the protocol layer, eMMC memory chips behave the same way as SD memory cards. Because of similarity of the protocol, even though eMMC are chips don't have the "card" form factor, same terminology is used as for SD cards (`sdmmc_card_t`, `sdmmc_card_init`). Note that eMMC chips can not be used over SPI, therefore are incompatible with SD SPI host driver.
To initialize eMMC memory and do read/write operations, follow the steps listed above for SD cards.
The protocol layer is given the :cpp:class:`sdmmc_host_t` structure. This structure describes the SD/MMC host driver, lists its capabilities, and provides pointers to functions of the driver. The protocol layer stores card-specific information in the :cpp:class:`sdmmc_card_t` structure. When sending commands to the SD/MMC host driver, the protocol layer uses the :cpp:class:`sdmmc_command_t` structure to describe the command, arguments, expected return values, and data to transfer if there is any.
Usage with SDIO cards
^^^^^^^^^^^^^^^^^^^^^
Using API with SD memory cards
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Initialization an probing process is the same as with SD memory cards. Only data transfer commands differ in SDIO mode.
1. To initialize the host, call the host driver functions, e.g., :cpp:func:`sdmmc_host_init`, :cpp:func:`sdmmc_host_init_slot`.
2. To initialize the card, call :cpp:func:`sdmmc_card_init` and pass to it the parameters ``host`` - the host driver information, and ``card`` - a pointer to the structure :cpp:class:`sdmmc_card_t` which will be filled with information about the card when the function completes.
3. To read and write sectors of the card, use :cpp:func:`sdmmc_read_sectors` and :cpp:func:`sdmmc_write_sectors` respectively and pass to it the parameter ``card`` - a pointer to the card information structure.
4. If the card is not used anymore, call the host driver function - e.g., :cpp:func:`sdmmc_host_deinit` - to disable the host peripheral and free the resources allocated by the driver.
During probing and card initialization (done by :cpp:func:`sdmmc_card_init`), the driver only configures the following registers of the IO card:
1. The IO portion of the card is reset by setting RES bit in "I/O Abort" (0x06) register.
2. If 4-line mode is enalbed in host and slot configuration, driver attempts to set "Bus width" field in "Bus Interface Control" (0x07) register. If that succeeds (which means that slave supports 4-line mode), host is also switched to 4-line mode.
3. If high-speed mode is enabled in host configuration, SHS bit is set in "High Speed" (0x13) register.
Using API with eMMC chips
^^^^^^^^^^^^^^^^^^^^^^^^^
In particular, the driver does not set any of the bits in I/O Enable, Int Enable registers, IO block sizes, etc. Applications can set these by calling :cpp:func:`sdmmc_io_write_byte`.
From the protocol layer's perspective, eMMC memory chips behave exactly like SD memory cards. Even though eMMCs are chips and do not have a card form factor, the terminology for SD cards can still be applied to eMMC due to the similarity of the protocol (`sdmmc_card_t`, `sdmmc_card_init`). Note that eMMC chips cannot be used over SPI, which makes them incompatible with the SD SPI host driver.
For card configuration and data transfer, use one of the following functions:
To initialize eMMC memory and perform read/write operations, follow the steps listed for SD cards in the previous section.
- :cpp:func:`sdmmc_io_read_byte`, :cpp:func:`sdmmc_io_write_byte` — read and write single byte using IO_RW_DIRECT (CMD52).
- :cpp:func:`sdmmc_io_read_bytes`, :cpp:func:`sdmmc_io_write_bytes` — read and write multiple bytes using IO_RW_EXTENDED (CMD53), in byte mode.
- :cpp:func:`sdmmc_io_read_blocks`, :cpp:func:`sdmmc_io_write_blocks` — read and write blocks of data using IO_RW_EXTENDED (CMD53), in block mode.
SDIO interrupts can be enabled by the application using :cpp:func:`sdmmc_io_enable_int` function. When using SDIO in 1-line mode, D1 line also needs to be connected to use SDIO interrupts.
Using API with SDIO cards
^^^^^^^^^^^^^^^^^^^^^^^^^
Initialization and the probing process is the same as with SD memory cards. The only difference is in data transfer commands in SDIO mode.
During the card initialization and probing, performed with :cpp:func:`sdmmc_card_init`, the driver only configures the following registers of the IO card:
1. The IO portion of the card is reset by setting RES bit in the I/O Abort (0x06) register.
2. If 4-line mode is enabled in host and slot configuration, the driver attempts to set the Bus width field in the Bus Interface Control (0x07) register. If setting the filed is successful, which means that the slave supports 4-line mode, the host is also switched to 4-line mode.
3. If high-speed mode is enabled in the host configuration, the SHS bit is set in the High Speed (0x13) register.
In particular, the driver does not set any bits in (1) I/O Enable and Int Enable registers, (2) I/O block sizes, etc. Applications can set them by calling :cpp:func:`sdmmc_io_write_byte`.
For card configuration and data transfer, choose the pair of functions relevant to your case from the table below.
========================================================================= ================================= =================================
Action Read Function Write Function
========================================================================= ================================= =================================
Read and write a single byte using IO_RW_DIRECT (CMD52) :cpp:func:`sdmmc_io_read_byte` :cpp:func:`sdmmc_io_write_byte`
Read and write multiple bytes using IO_RW_EXTENDED (CMD53) in byte mode :cpp:func:`sdmmc_io_read_bytes` :cpp:func:`sdmmc_io_write_bytes`
Read and write blocks of data using IO_RW_EXTENDED (CMD53) in block mode :cpp:func:`sdmmc_io_read_blocks` :cpp:func:`sdmmc_io_write_blocks`
========================================================================= ================================= =================================
SDIO interrupts can be enabled by the application using the function :cpp:func:`sdmmc_io_enable_int`. When using SDIO in 1-line mode, the D1 line also needs to be connected to use SDIO interrupts.
If you want the application to wait until the SDIO interrupt occurs, use :cpp:func:`sdmmc_io_wait_int`.
The application can wait for SDIO interrupt to occur using :cpp:func:`sdmmc_io_wait_int`.
Combo (memory + IO) cards
^^^^^^^^^^^^^^^^^^^^^^^^^
The driver does not support SD combo cards. Combo cards will be treated as IO cards.
The driver does not support SD combo cards. Combo cards are treated as IO cards.
Thread safety
^^^^^^^^^^^^^
Most applications need to use the protocol layer only in one task; therefore the protocol layer doesn't implement any kind of locking on the :cpp:class:`sdmmc_card_t` structure, or when accessing SDMMC or SD SPI host drivers. Such locking is usually implemented in the higher layer (e.g. in the filesystem driver).
Most applications need to use the protocol layer only in one task. For this reason, the protocol layer does not implement any kind of locking on the :cpp:class:`sdmmc_card_t` structure, or when accessing SDMMC or SD SPI host drivers. Such locking is usually implemented on a higher layer, e.g., in the filesystem driver.
API Reference

31
docs/en/api-reference/storage/spi_flash.rst
View file

@ -5,7 +5,7 @@ See also
- :doc:`Partition Table documentation <../../api-guides/partition-tables>`
- :doc:`Over The Air Update (OTA) API <../system/ota>` provides high-level API for updating app firmware stored in flash.
- :doc:`Non-Volatile Storage (NVS) API <nvs_flash>` provides a structured API for storing small items of data in SPI flash.
- :doc:`Non-Volatile Storage (NVS) API <nvs_flash>` provides a structured API for storing small pieces of data in SPI flash.
.. _spi-flash-implementation-details:
@ -13,35 +13,20 @@ See also
Implementation details
----------------------
In order to perform some flash operations, we need to make sure both CPUs
are not running any code from flash for the duration of the flash operation.
In a single-core setup this is easy: we disable interrupts/scheduler and do
the flash operation. In the dual-core setup this is slightly more complicated.
We need to make sure that the other CPU doesn't run any code from flash.
In order to perform some flash operations, it is necessary to make sure that both CPUs are not running any code from flash for the duration of the flash operation:
- In a single-core setup, the SDK does it by disabling interrupts/scheduler before performing the flash operation.
- In a dual-core setup, this is slightly more complicated as the SDK needs to make sure that the other CPU is not running any code from flash.
When SPI flash API is called on CPU A (can be PRO or APP), we start
spi_flash_op_block_func function on CPU B using esp_ipc_call API. This API
wakes up high priority task on CPU B and tells it to execute given function,
in this case spi_flash_op_block_func. This function disables cache on CPU B and
signals that cache is disabled by setting s_flash_op_can_start flag.
Then the task on CPU A disables cache as well, and proceeds to execute flash
operation.
When SPI flash API is called on CPU A (can be PRO or APP), start the spi_flash_op_block_func function on CPU B using the esp_ipc_call API. This API wakes up a high priority task on CPU B and tells it to execute a given function, in this case, spi_flash_op_block_func. This function disables cache on CPU B and signals that the cache is disabled by setting the s_flash_op_can_start flag. Then the task on CPU A disables cache as well and proceeds to execute flash operation.
While flash operation is running, interrupts can still run on CPUs A and B.
We assume that all interrupt code is placed into RAM. Once interrupt allocation
API is added, we should add a flag to request interrupt to be disabled for
the duration of flash operations.
While a flash operation is running, interrupts can still run on CPUs A and B. It is assumed that all interrupt code is placed into RAM. Once the interrupt allocation API is added, a flag should be added to request the interrupt to be disabled for the duration of a flash operations.
Once flash operation is complete, function on CPU A sets another flag,
s_flash_op_complete, to let the task on CPU B know that it can re-enable
cache and release the CPU. Then the function on CPU A re-enables the cache on
CPU A as well and returns control to the calling code.
Once the flash operation is complete, the function on CPU A sets another flag, s_flash_op_complete, to let the task on CPU B know that it can re-enable cache and release the CPU. Then the function on CPU A re-enables the cache on CPU A as well and returns control to the calling code.
Additionally, all API functions are protected with a mutex (s_flash_op_mutex).
In a single core environment (:ref:`CONFIG_FREERTOS_UNICORE` enabled), we simply
disable both caches, no inter-CPU communication takes place.
In a single core environment (:ref:`CONFIG_FREERTOS_UNICORE` enabled), you need to disable both caches, so that no inter-CPU communication can take place.
API Reference - SPI Flash
-------------------------

103
docs/en/api-reference/storage/spiffs.rst
View file

@ -4,15 +4,16 @@ SPIFFS Filesystem
Overview
--------
SPIFFS is a file system intended for SPI NOR flash devices on embedded targets.
It supports wear leveling, file system consistency checks and more.
SPIFFS is a file system intended for SPI NOR flash devices on embedded targets. It supports wear levelling, file system consistency checks, and more.
Notes
-----
- Currently, SPIFFS does not support directories. It produces a flat structure. If SPIFFS is mounted under ``/spiffs``, then creating a file with path ``/spiffs/tmp/myfile.txt`` will create a file called ``/tmp/myfile.txt`` in SPIFFS, instead of ``myfile.txt`` under directory ``/spiffs/tmp``.
- It is not a realtime stack. One write operation might last much longer than another.
- Currently, it does not detect or handle bad blocks.
- Currently, SPIFFS does not support directories, it produces a flat structure. If SPIFFS is mounted under ``/spiffs``, then creating a file with the path ``/spiffs/tmp/myfile.txt`` will create a file called ``/tmp/myfile.txt`` in SPIFFS, instead of ``myfile.txt`` in the directory ``/spiffs/tmp``.
- It is not a real-time stack. One write operation might take much longer than another.
- For now, it does not detect or handle bad blocks.
Tools
-----
@ -21,29 +22,25 @@ spiffsgen.py
^^^^^^^^^^^^
:component_file:`spiffsgen.py<spiffs/spiffsgen.py>` is a write-only Python SPIFFS implementation used to create filesystem
images from the contents of a host folder. To use ``spiffsgen.py``, simply invoke it from your favorite terminal::
images from the contents of a host folder. To use ``spiffsgen.py``, open Terminal and run::
python spiffsgen.py <image_size> <base_dir> <output_file>
- image_size: size of the partition on which the created SPIFFS image will be flashed to
- base_dir: directory to create the SPIFFS image of
- output_file: SPIFFS image output file
The required arguments are as follows:
Besides the three required arguments: *image_size*, *base_dir* and *output_file*, there are other arguments
that control image generation. Documentation on these arguments exist in the tool's help::
- **image_size**: size of the partition onto which the created SPIFFS image will be flashed.
- **base_dir**: directory for which the SPIFFS image needs to be created.
- **output_file**: SPIFFS image output file.
There are also other arguments that control image generation. Documentation on these arguments can be found in the tool's help::
python spiffsgen.py --help
These optional arguments correspond to possible SPIFFS build configuration.
User should make sure that the image is generated with the same arguments/configuration as
SPIFFS was built with, else the user ends up with an invalid image. As a guide, the help output indicates the SPIFFS
build configuration the argument corresponds to. In cases when these arguments
are not specified, the default values shown in the help output are used.
These optional arguments correspond to a possible SPIFFS build configuration. To generate the right image, please make sure that you use the same arguments/configuration as were used to build SPIFFS. As a guide, the help output indicates the SPIFFS build configuration to which the argument corresponds. In cases when these arguments are not specified, the default values shown in the help output will be used.
Once the image has been created, it can be flashed using ``esptool.py`` or ``parttool.py``.
When the image is created, it can be flashed using ``esptool.py`` or ``parttool.py``.
Aside from invoking ``spiffsgen.py`` standalone, it is also possible to use it directly from the build system by calling
``spiffs_create_partition_image``.
Aside from invoking the ``spiffsgen.py`` standalone by manually running it from the command line or a script, it is also possible to invoke ``spiffsgen.py`` directly from the build system by calling ``spiffs_create_partition_image``.
Make::
@ -53,44 +50,42 @@ CMake::
spiffs_create_partition_image(<partition> <base_dir> [FLASH_IN_PROJECT])
This is more convenient as the build configuration is automatically passed to the tool,
ensuring that the image generated is valid for that build. An example of this is while the *image_size* is required
for the standalone invocation, only the *partition* name is required when using ``spiffs_create_partition_image`` --
the image size is automatically obtained from the project's partition table.
It is important to note that due to the differences in structure between the two build systems,
when using Make, ``spiffs_create_partition_image`` must be called from the project Makefile;
for CMake, it should be called from one of the component CMakeLists.txt. For both build systems, the image will be created in the build directory
with filename *partition*.bin.
This is more convenient as the build configuration is automatically passed to the tool, ensuring that the generated image is valid for that build. An example of this is while the *image_size* is required for the standalone invocation, only the *partition* name is required when using ``spiffs_create_partition_image`` -- the image size is automatically obtained from the project's partition table.
Optionally, user can opt to have the image automatically flashed together with the app binaries, partition tables, etc. on
Due to the differences in structure between Make and Cmake, it is important to note that:
- for Make ``spiffs_create_partition_image`` must be called from the project Makefile
- for CMake ``spiffs_create_partition_image`` must be called from one of the component CMakeLists.txt files
For both build systems, the image will be created in the build directory with the filename *partition*.bin.
Optionally, you can opt to have the image automatically flashed together with the app binaries, partition tables, etc., with
``idf.py flash`` or ``make flash`` by specifying ``FLASH_IN_PROJECT``. For example::
spiffs_create_partition_image(my_spiffs_partition my_folder FLASH_IN_PROJECT)
If FLASH_IN_PROJECT is not specified, the image is still generated,
but user has to flash it manually using ``esptool.py``, ``parttool.py`` or a custom build system target.
If FLASH_IN_PROJECT is not specified, the image will still be generated, but you will have to flash it manually using ``esptool.py``, ``parttool.py``, or a custom build system target.
For an example, see :example:`examples/storage/spiffsgen>`.
+For an example, see :example:`examples/storage/spiffsgen>`.
mkspiffs
^^^^^^^^
Another tool for creating SPIFS partition images is `mkspiffs <https://github.com/igrr/mkspiffs>`_.
Like ``spiffsgen.py``, it can be used to create image from a given folder and then flash that image with ``esptool.py``
Another tool for creating SPIFFS partition images is `mkspiffs <https://github.com/igrr/mkspiffs>`_.
Similar to ``spiffsgen.py``, it can be used to create an image from a given folder and then flash that image using ``esptool.py``
To do that you need to obtain some parameters:
For that, you need to obtain the following parameters:
- Block Size: 4096 (standard for SPI Flash)
- Page Size: 256 (standard for SPI Flash)
- Image Size: Size of the partition in bytes (can be obtained from partition table)
- Partition Offset: Starting address of the partition (can be obtained from partition table)
- **Block Size**: 4096 (standard for SPI Flash)
- **Page Size**: 256 (standard for SPI Flash)
- **Image Size**: Size of the partition in bytes (can be obtained from a partition table)
- **Partition Offset**: Starting address of the partition (can be obtained from a partition table)
To pack a folder into 1 Megabyte image::
To pack a folder into a 1-Megabyte image, run::
mkspiffs -c [src_folder] -b 4096 -p 256 -s 0x100000 spiffs.bin
To flash the image to ESP32 at offset 0x110000::
To flash the image onto ESP32 at offset 0x110000, run::
python esptool.py --chip esp32 --port [port] --baud [baud] write_flash -z 0x110000 spiffs.bin
@ -98,21 +93,15 @@ To flash the image to ESP32 at offset 0x110000::
Notes on which SPIFFS tool to use
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The two tools presented above offer very similar functionality. There are, however, reasons to prefer one
over the other depending on the use case.
The two tools presented above offer very similar functionality. However, there are reasons to prefer one over the other, depending on the use case.
If the intent is to simply generate a SPIFFS image during build, ``spiffsgen.py`` makes it very convenient
by providing functions/commands from the build system itself. This makes it easy to generate SPIFFS images
that match the build configuration and can be flashed together with the application.
Another case for choosing ``spiffsgen.py`` is when the host has no C/C++ compiler available, since ``mkspiffs``
requires compilation.
Use ``spiffsgen.py`` in the following cases:
1. If you want to simply generate a SPIFFS image during the build. ``spiffsgen.py`` makes it very convenient by providing functions/commands from the build system itself.
2. If the host has no C/C++ compiler available, because ``spiffsgen.py`` does not require compilation.
On the other hand, ``mkspiffs`` offers unpacking SPIFFS images in addition to image generation. This is not
possible with ``spiffsgen.py``, at least not yet. There might also be environments where a Python interpreter
is not available, but a host compiler is or a pre-compiled ``mkspiffs`` binary
can do the job. However, there is no build system integration for ``mkspiffs`` and the user has to
do the corresponding work: compiling ``mkspiffs`` during build (if a pre-compiled binary is not used), creating build rules/targets
for the output files, passing proper parameters to the tool, etc.
Use ``mkspiffs`` in the following cases:
1. If you need to unpack SPIFFS images in addition to image generation. For now, it is not possible with ``spiffsgen.py``.
2. If you have an environment where a Python interpreter is not available, but a host compiler is available. Otherwise, a pre-compiled ``mkspiffs`` binary can do the job. However, there is no build system integration for ``mkspiffs`` and the user has to do the corresponding work: compiling ``mkspiffs`` during build (if a pre-compiled binary is not used), creating build rules/targets for the output files, passing proper parameters to the tool, etc.
See also
@ -120,14 +109,14 @@ See also
- :doc:`Partition Table documentation <../../api-guides/partition-tables>`
Application Example
-------------------
An example for using SPIFFS is provided in :example:`storage/spiffs` directory. This example initializes and mounts SPIFFS partition, and writes and reads data from it using POSIX and C library APIs. See README.md file in the example directory for more information.
An example of using SPIFFS is provided in the :example:`storage/spiffs` directory. This example initializes and mounts a SPIFFS partition, then writes and reads data from it using POSIX and C library APIs. See the README.md file in the example directory for more information.
High level API Reference
High-level API Reference
------------------------
* :component_file:`spiffs/include/esp_spiffs.h`
.. include:: /_build/inc/esp_spiffs.inc

2
docs/en/api-reference/storage/wear-levelling.rst
View file

@ -9,7 +9,7 @@ See also
Application Example
-------------------
An example which combines wear levelling driver with FATFS library is provided in ``examples/storage/wear_levelling`` directory. This example initializes the wear levelling driver, mounts FATFS partition, and writes and reads data from it using POSIX and C library APIs. See README.md file in the example directory for more information.
An example which combines the wear levelling driver with the FATFS library is provided in the :example:`storage/wear_levelling` directory. This example initializes the wear levelling driver, mounts FATFS partition, as well as writes and reads data from it using POSIX and C library APIs. See the :example:`storage/wear_levelling/README.md` file for more information.
High level API Reference
------------------------

200
tools/mass_mfg/docs/README.rst
View file

@ -1,50 +1,56 @@
Manufacturing Utility
=====================
Introduction
----------------
------------
This utility is designed to create instances of factory NVS partition images on a per-device basis for mass manufacturing purposes. The NVS partition images are created from CSV files containing user-provided configurations and values.
Please note that this utility only creates manufacturing binary images which then need to be flashed onto your devices using:
- esptool.py
- Flash Download tool (available on Windows only)
- Direct flash programming
This utility is designed to create per device instances factory nvs partition images for mass manufacturing purposes.
These images are created from user provided configuration and values csv files.
This utility only creates the manufacturing binary images and you can choose to use esptool.py or Windows based flash programming utility or direct flash programming to program these images at the time of manufacturing.
Prerequisites
------------------
-------------
**This utility is dependent on the esp-idf nvs partition utility.**
**This utility is dependent on esp-idf's NVS partition utility.**
* Operating System requirements:
- Linux / MacOS / Windows (standard distributions)
* Operating System requirements:
- Linux / MacOS / Windows (standard distributions)
* The following packages are needed for using this utility:
- Python version: 2.7 (minimum) is required.
- Link to install python: <https://www.python.org/downloads/>
* The following packages are needed to use this utility:
- Python version: 2.7 (minimum) is required. You can find it here: <https://www.python.org/downloads/>
.. note:: Make sure the python path is set in the PATH environment variable before using this utility.
.. note::
Before using this utility, please make sure that:
- The path to Python is added to the PATH environment variable.
- You have installed the packages from `requirement.txt`, the file in the root of the esp-idf directory.
Make sure to include packages from `requirement.txt` in top level IDF directory.
Workflow
-----------
--------
.. blockdiag::
blockdiag {
A [label = "CSV Configuration file"];
B [label = "Master CSV Values file"];
B [label = "Master Value CSV file"];
C [label = "Binary files", stacked];
A -- B -> C
}
CSV Configuration File:
------------------------
CSV Configuration File
----------------------
This file contains the configuration of the device to be manufactured.
This file contains the configuration of the device to be flashed.
The data in configuration file **must** have the following format (`REPEAT` tag is optional)::
The data in the configuration file has the following format (the `REPEAT` tag is optional)::
name1,namespace, <-- First entry should be of type "namespace"
key1,type1,encoding1
@ -53,12 +59,12 @@ The data in configuration file **must** have the following format (`REPEAT` tag
key3,type3,encoding3
key4,type4,encoding4
.. note:: First entry in this file should always be ``namespace`` entry.
.. note:: The first line in this file should always be the ``namespace`` entry.
Each row should have these 3 parameters: ``key,type,encoding`` separated by comma.
If ``REPEAT`` tag is present, the value corresponding to this key in the Master CSV Values File will be the same for all devices.
Each line should have three parameters: ``key,type,encoding``, separated by a comma.
If the ``REPEAT`` tag is present, the value corresponding to this key in the master value CSV file will be the same for all devices.
*Please refer to README of nvs_partition utility for detailed description of each parameter.*
*Please refer to README of the NVS Partition Generator utility for detailed description of each parameter.*
Below is a sample example of such a configuration file::
@ -69,115 +75,131 @@ Below is a sample example of such a configuration file::
device_no,data,i32
.. note:: Make sure there are no spaces before and after ',' in the configuration file.
.. note::
Master CSV Values File:
------------------------
Make sure there are **no spaces**:
- before and after ','
- at the end of each line in a CSV file
This file contains details of the device to be manufactured. Each row in this file corresponds to a device instance.
Master Value CSV File
---------------------
This file contains details of the devices to be flashed. Each line in this file corresponds to a device instance.
The data in values file **must** have the following format::
The data in the master value CSV file has the following format::
key1,key2,key3,.....
value1,value2,value3,....
.. note:: First line in this file should always be the ``key`` names. All the keys from the configuration file should be present here in the **same order**. This file can have additional columns(keys) and they will act like metadata and would not be part of final binary files.
.. note:: The first line in the file should always contain the ``key`` names. All the keys from the configuration file should be present here in the **same order**. This file can have additional columns (keys). The additional keys will be treated as metadata and would not be part of the final binary files.
Each row should have the ``value`` of the corresponding keys, separated by comma. If key has ``REPEAT`` tag, then its corresponding value **must** be entered in the second line only. Keep the entry empty for this value in the next lines. Below is the description of this parameter:
Each line should contain the ``value`` of the corresponding keys, separated by a comma. If the key has the ``REPEAT`` tag, its corresponding value **must** be entered in the second line only. Keep the entry empty for this value in the following lines.
The description of this parameter is as follows:
``value``
Data value.
Data value
Below is a sample example of such a values file::
Data value is the value of data corresponding to the key.
Below is a sample example of a master value CSV file::
id,firmware_key,serial_no,device_no
1,1a2b3c4d5e6faabb,A1,101
2,1a2b3c4d5e6fccdd,,102
3,1a2b3c4d5e6feeff,,103
.. note:: *If 'REPEAT' tag is present, a new Master CSV Values File is created in the same folder as the input Master CSV File with the values inserted at each line for the key with 'REPEAT' tag.*
.. note:: *If the 'REPEAT' tag is present, a new master value CSV file will be created in the same folder as the input Master CSV File with the values inserted at each line for the key with the 'REPEAT' tag*.
.. note:: *Intermediate CSV files are created by this utility which are input to the nvs partition utility to generate the binary files.*
This utility creates intermediate CSV files which are used as input for the NVS partition utility to generate the binary files.
The format of this intermediate csv file will be::
The format of this intermediate CSV file is as follows::
key,type,encoding,value
key,namespace, ,
key1,type1,encoding1,value1
key2,type2,encoding2,value2
.. note:: An intermediate csv file will be created for each device instance.
An instance of an intermediate CSV file will be created for each device on an individual basis.
Running the utility
----------------------
-------------------
The mfg\_gen.py utility is using the generated CSV Configuration file and Master CSV Values file and is generating per device instance factory images.
The mfg\_gen.py utility uses the generated CSV Configuration file and the master value CSV file to generate factory images for each device.
*Sample CSV Configuration file and Master CSV Values file is provided with this utility.*
*A sample CSV Configuration file and a master value CSV file are both provided with this utility.*
**Usage**::
$ ./mfg_gen.py [-h] [--conf CONFIG_FILE] [--values VALUES_FILE]
./mfg_gen.py [-h] [--conf CONFIG_FILE] [--values VALUES_FILE]
[--prefix PREFIX] [--fileid FILEID] [--outdir OUTDIR]
[--size PART_SIZE] [--version {v1,v2}]
[--keygen {true,false}] [--encrypt {true,false}]
[--keyfile KEYFILE]
The description of the arguments is given in the table below.
+------------------------+----------------------------------------------------------------------------------------------+
| Arguments | Description |
+========================+==============================================================================================+
| --conf CONFIG_FILE | the input configuration csv file |
+------------------------+----------------------------------------------------------------------------------------------+
| --values VALUES_FILE | the input values csv file |
+------------------------+----------------------------------------------------------------------------------------------+
| --prefix PREFIX | the unique name as each filename prefix |
+------------------------+----------------------------------------------------------------------------------------------+
| --fileid FILEID | the unique file identifier(any key in values file) |
| | as each filename suffix (Default: numeric value(1,2,3...)) |
+------------------------+----------------------------------------------------------------------------------------------+
| --outdir OUTDIR | the output directory to store the files created (Default: current directory) |
+------------------------+----------------------------------------------------------------------------------------------+
| --size PART_SIZE | Size of NVS Partition in bytes (must be multiple of 4096) |
+------------------------+----------------------------------------------------------------------------------------------+
| --version {v1,v2} | Set version. Default: v2 |
+------------------------+----------------------------------------------------------------------------------------------+
| --keygen {true,false} | Generate keys for encryption. |
| | Default: false |
+------------------------+----------------------------------------------------------------------------------------------+
| --encrypt {true,false} | Set encryption mode. Default: false |
+------------------------+----------------------------------------------------------------------------------------------+
| --keyfile KEYFILE | File having key for encryption (Applicable only if encryption mode is true) |
+------------------------+----------------------------------------------------------------------------------------------+
+------------------------+------------------------------------------------------------+-------------------+
| Arguments | Description | Default Value |
+========================+============================================================+===================+
| --conf CONFIG_FILE | Path to existing CSV configuration file | |
+------------------------+------------------------------------------------------------+-------------------+
| --values VALUES_FILE | Path to existing master value CSV file | |
+------------------------+------------------------------------------------------------+-------------------+
| --prefix PREFIX | Unique filename prefix | |
+------------------------+------------------------------------------------------------+-------------------+
| --fileid FILEID | Unique file identifier (any key in the file with values) | numeric value |
| | as a filename suffix | (1,2,3...) |
+------------------------+------------------------------------------------------------+-------------------+
| --outdir OUTDIR | Output directory to store created files | current directory |
+------------------------+------------------------------------------------------------+-------------------+
| --size PART_SIZE | Size of NVS Partition in bytes (must be multiple of 4096) | |
+------------------------+------------------------------------------------------------+-------------------+
| --version {v1,v2} | Set version | v2 |
+------------------------+------------------------------------------------------------+-------------------+
| --keygen {true,false} | Generate keys for encryption | false |
+------------------------+------------------------------------------------------------+-------------------+
| --encrypt {true,false} | Set encryption mode | false |
+------------------------+------------------------------------------------------------+-------------------+
| --keyfile KEYFILE | File storing key for encryption (Applicable only if | |
| | Encryption mode is true). | |
+------------------------+------------------------------------------------------------+-------------------+
*You can use the below commands to run this utility with the sample files provided*::
$ ./mfg_gen.py --conf samples/sample_config.csv --values samples/sample_values_singlepage_blob.csv --prefix Fan --size 0x3000
*To run this utility with the provided sample files, use the commands below*::
$ ./mfg_gen.py --conf samples/sample_config.csv --values samples/sample_values_multipage_blob.csv --prefix Fan --size 0x4000
.. note:: When you use this utility to generate per device instance factory images --conf, --values, --prefix and --size arguments are mandatory.
$ ./mfg_gen.py --conf samples/sample_config.csv --values samples/sample_values_singlepage_blob.csv --prefix Fan --size 0x3000 --outdir tmp
./mfg_gen.py --conf samples/sample_config.csv --values samples/sample_values_singlepage_blob.csv --prefix Fan --size 0x3000
.. note:: The --outdir directory is created if not present.
./mfg_gen.py --conf samples/sample_config.csv --values samples/sample_values_multipage_blob.csv --prefix Fan --size 0x4000
.. note:: The file path given in the ``file`` type in the values file is expected to be relative to the current directory from which you are running the utility.
$ ./mfg_gen.py --conf samples/sample_config.csv --values samples/sample_values_singlepage_blob.csv --prefix Fan --size 0x3000 --encrypt true --keygen true
When you use this utility to generate factory images on a per device basis, keep in mind that the arguments --conf, --values, --prefix, and --size are mandatory.
.. note:: ``keys/`` directory is generated with the encryption keys filename of the form ``prefix-fileid-keys.bin``.
*You can also run the below command to use the utility to* **only** *generate encryption keys binary file ( following example 'keys/' directory is created in current path), which can further be used to encrypt per device instance factory images*::
$ ./mfg_gen.py --keygen true
$ ./mfg_gen.py --keygen true --keyfile encr_keys.bin
./mfg_gen.py --conf samples/sample_config.csv --values samples/sample_values_singlepage_blob.csv --prefix Fan --size 0x3000 --outdir tmp
.. note:: When running utility to generate only ``keys``, if --keyfile is given it will generate encryption keys with filename given in --keyfile argument.
.. note:: If the --outdir directory does not exist, it will be created.
.. note:: When you use this utility to generate only encryption keys --keygen argument is mandatory.
The master value CSV file should have the path in the ``file`` type relative to the directory from which you are running the utility.
.. note:: The default numeric value: 1,2,3... of ``fileid`` argument, corresponds to each row having device instance values in master csv values file.
./mfg_gen.py --conf samples/sample_config.csv --values samples/sample_values_singlepage_blob.csv --prefix Fan --size 0x3000 --encrypt true --keygen true
.. note:: ``bin/`` **and** ``csv/`` **sub-directories are created in the** ``outdir`` **directory specified while running this utility. The binary files generated will be stored in** ``bin/`` **and the intermediate csv files generated will be stored in** ``csv/``.
.. note:: The generated ``keys/`` directory is named as the file with encryption keys of the form ``prefix-fileid-keys.bin``.
*If you* **only** *want to generate a binary file with encryption keys, you can run the command below.*::
./mfg_gen.py --keygen true
.. note:: When you use this utility to generate encryption keys only, the --keygen argument is mandatory.
In the following example, the 'keys/' directory will be created at the current path. This binary file can further be used to encrypt factory images created on the per device basis*.::
./mfg_gen.py --keygen true --keyfile encr_keys.bin
.. note:: When running the utility to generate encryption keys only, if --keyfile is given, it will generate encryption keys with the filename given in the --keyfile argument.
The default numeric value: 1,2,3... of the ``fileid`` argument corresponds to each line bearing device instance values in the master value CSV file.
While running the manufacturing utility, the following folders will be created in the specified ``outdir`` directory:
- ``bin/`` for storing the generated binary files
- ``csv/`` for storing the generated intermediate CSV files