diff --git a/components/nvs_flash/nvs_partition_generator/README.rst b/components/nvs_flash/nvs_partition_generator/README.rst new file mode 100644 index 000000000..5b62ebc39 --- /dev/null +++ b/components/nvs_flash/nvs_partition_generator/README.rst @@ -0,0 +1,60 @@ +NVS Partition Generator Utility +=============================== + +Introduction +------------ + +:component_file:`nvs_flash/nvs_partition_generator/nvs_partition_gen.py` utility is designed to help create a binary file, compatible with NVS architecture defined in :doc:`Non-Volatile Storage `, based on user provided key-value pairs in a CSV file. +Utility is ideally suited for generating a binary blob, containing data specific to ODM/OEM, which can be flashed externally at the time of device manufacturing. This helps manufacturers set unique value for various parameters for each device, e.g. serial number, while using same application firmaware for all devices. + +CSV file format +--------------- + +Each row of the .csv file should have 4 parameters, separated by comma. Below is the description of each of these parameters: + +Key + Key of the data. Data can later be accessed from an application via this key. + +Type + Supported values are ``file``, ``data`` and ``namespace``. + +Encoding + Supported values are: ``u8``, ``i8``, ``u16``, ``u32``, ``i32``, ``string``, ``hex2bin`` and ``binary``. This specifies how actual data values are encoded in the resultant binary file. Difference between ``string`` and ``binary`` encoding is that ``string`` data is terminated with a NULL character, whereas ``binary`` data is not. + + .. note:: For ``file`` type, only ``hex2bin``, ``string`` and ``binary`` is supported as of now. + +Value + Data value. + +.. note:: Encoding and Value cells for ``namespace`` field type should be empty. Encoding and Value of ``namespace`` is fixed and isn't configurable. Any value in these cells are ignored. + +.. note:: First row of the CSV file should always be column header and isn't configurable. + +Below is an example dump of such CSV file:: + + key,type,encoding,value <-- column header + namespace_name,namespace,, <-- First entry should be of type "namespace" + key1,data,u8,1 + key2,file,string,/path/to/file + +.. note:: Make sure there are no spaces before and after ',' in CSV file. + +NVS Entry and Namespace association +----------------------------------- + +When a new namespace entry is encountered in the CSV file, each follow-up entries will be part of that namespace, until next namespace entry is found, in which case all the follow-up entries will be part of the new namespace. + +.. note:: First entry in a CSV file should always be ``namespace`` entry. + +Running the utility +------------------- + +A sample CSV file provided with the utility. You can run the utility using below command:: + + python nvs_partition_generator.py sample.csv sample.bin + +Caveats +------- +- Utility doesn't check for duplicate keys and will write data pertaining to both keys. User needs to make sure keys are distinct. +- Once a new page is created, no data will be written in the space left in previous page. Fields in the CSV file need to be ordered in such a way so as to optimize memory. +- 64-bit datatype is not yet supported. diff --git a/components/nvs_flash/nvs_partition_generator/nvs_partition_gen.py b/components/nvs_flash/nvs_partition_generator/nvs_partition_gen.py new file mode 100644 index 000000000..ecbe9b5db --- /dev/null +++ b/components/nvs_flash/nvs_partition_generator/nvs_partition_gen.py @@ -0,0 +1,361 @@ +#!/usr/bin/env python +# +# esp-idf NVS partition generation tool. Tool helps in generating NVS-compatible +# partition binary, with key-value pair entries provided via a CSV file. +# +# Copyright 2018 Espressif Systems (Shanghai) PTE LTD +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +import sys +import argparse +import binascii +import getopt +import struct +import os +import array +import csv +import zlib +from os import path + +""" Class for standard NVS page structure """ +class Page(object): + PAGE_PARAMS = { + "max_size": 4096, + "max_blob_size": 1984, + "max_entries": 126 + } + + # Item type codes + U8 = 0x01 + I8 = 0x11 + U16 = 0x02 + I16 = 0x12 + U32 = 0x04 + I32 = 0x14 + SZ = 0x21 + BLOB = 0x41 + + # Few Page constants + HEADER_SIZE = 32 + BITMAPARRAY_OFFSET = 32 + BITMAPARRAY_SIZE_IN_BYTES = 32 + FIRST_ENTRY_OFFSET = 64 + SINGLE_ENTRY_SIZE = 32 + + def __init__(self, page_num): + self.entry_num = 0 + self.bitmap_array = array.array('B') + self.page_buf = bytearray(b'\xff')*Page.PAGE_PARAMS["max_size"] + self.bitmap_array = self.create_bitmap_array() + self.set_header(page_num) + + def set_header(self, page_num): + # set page state to active + page_header= bytearray(b'\xff')*32 + page_state_active_seq = 0xFFFFFFFE + page_header[0:4] = struct.pack(' Page.PAGE_PARAMS["max_blob_size"]: + raise InputError("%s: Size exceeds max allowed length." % key) + + # Calculate no. of entries data will require + rounded_size = (datalen + 31) & ~31 + data_entry_count = rounded_size / 32 + total_entry_count = data_entry_count + 1 # +1 for the entry header + + # Check if page is already full and new page is needed to be created right away + if (self.entry_num + total_entry_count) >= Page.PAGE_PARAMS["max_entries"]: + raise PageFullError() + + # Entry header + entry_struct = bytearray('\xff')*32 + entry_struct[0] = ns_index # namespace index + entry_struct[2] = data_entry_count + 1 # Span + + # set key + key_array = bytearray('\x00')*16 + entry_struct[8:24] = key_array + entry_struct[8:8 + len(key)] = key + + # set Type + if encoding == "string": + entry_struct[1] = Page.SZ + elif encoding == "hex2bin" or encoding == "binary": + entry_struct[1] = Page.BLOB + + # compute CRC of data + entry_struct[24:26] = struct.pack('= Page.PAGE_PARAMS["max_entries"]: + raise PageFullError() + + entry_struct = bytearray('\xff')*32 + entry_struct[0] = ns_index # namespace index + entry_struct[2] = 0x01 # Span + + # write key + key_array = bytearray('\x00')*16 + entry_struct[8:24] = key_array + entry_struct[8:8 + len(key)] = key + + if encoding == "u8": + entry_struct[1] = Page.U8 + entry_struct[24] = struct.pack(' #include +#include +#include #define TEST_ESP_ERR(rc, res) CHECK((rc) == (res)) #define TEST_ESP_OK(rc) CHECK((rc) == ESP_OK) @@ -1459,6 +1461,53 @@ TEST_CASE("Recovery from power-off when the entry being erased is not on active /* Add new tests above */ /* This test has to be the final one */ +TEST_CASE("check partition generation utility", "[nvs_part_gen]") +{ + int childpid = fork(); + if (childpid == 0) { + exit(execlp("python", "python", + "../nvs_partition_generator/nvs_partition_gen.py", + "../nvs_partition_generator/sample.csv", + "../nvs_partition_generator/partition.bin", NULL)); + } else { + CHECK(childpid > 0); + int status; + waitpid(childpid, &status, 0); + CHECK(WEXITSTATUS(status) != -1); + } +} + +TEST_CASE("read data from partition generated via partition generation utility", "[nvs_part_gen]") +{ + SpiFlashEmulator emu("../nvs_partition_generator/partition.bin"); + nvs_handle handle; + TEST_ESP_OK( nvs_flash_init_custom("test", 0, 2) ); + TEST_ESP_OK( nvs_open_from_partition("test", "dummyNamespace", NVS_READONLY, &handle)); + uint8_t u8v; + TEST_ESP_OK( nvs_get_u8(handle, "dummyU8Key", &u8v)); + CHECK(u8v == 127); + int8_t i8v; + TEST_ESP_OK( nvs_get_i8(handle, "dummyI8Key", &i8v)); + CHECK(i8v == -128); + uint16_t u16v; + TEST_ESP_OK( nvs_get_u16(handle, "dummyU16Key", &u16v)); + CHECK(u16v == 32768); + uint32_t u32v; + TEST_ESP_OK( nvs_get_u32(handle, "dummyU32Key", &u32v)); + CHECK(u32v == 4294967295); + int32_t i32v; + TEST_ESP_OK( nvs_get_i32(handle, "dummyI32Key", &i32v)); + CHECK(i32v == -2147483648); + char buf[64] = {0}; + size_t buflen = 64; + TEST_ESP_OK( nvs_get_str(handle, "dummyStringKey", buf, &buflen)); + CHECK(strncmp(buf, "0A:0B:0C:0D:0E:0F", buflen) == 0); + buflen = 64; + uint8_t hexdata[] = {0x01, 0x02, 0x03, 0xab, 0xcd, 0xef}; + TEST_ESP_OK( nvs_get_blob(handle, "dummyHex2BinKey", buf, &buflen)); + CHECK(memcmp(buf, hexdata, buflen) == 0); +} + TEST_CASE("dump all performance data", "[nvs]") { std::cout << "====================" << std::endl << "Dumping benchmarks" << std::endl; diff --git a/docs/en/api-reference/storage/index.rst b/docs/en/api-reference/storage/index.rst index a8cefae6f..2a8074478 100644 --- a/docs/en/api-reference/storage/index.rst +++ b/docs/en/api-reference/storage/index.rst @@ -7,6 +7,7 @@ Storage API SPI Flash and Partition APIs SD/SDIO/MMC Driver Non-Volatile Storage + NVS Partition Generation Utility Virtual Filesystem FAT Filesystem Wear Levelling diff --git a/docs/en/api-reference/storage/nvs_flash.rst b/docs/en/api-reference/storage/nvs_flash.rst index 184daff12..4daa17ea2 100644 --- a/docs/en/api-reference/storage/nvs_flash.rst +++ b/docs/en/api-reference/storage/nvs_flash.rst @@ -1,5 +1,10 @@ .. include:: ../../../../components/nvs_flash/README.rst +NVS Partition Generator Utility +------------------------------- + +This utility helps in generating NVS-esque partition binary file which can be flashed separately on a dedicated partition via a flashing utility. Key-value pairs to be flashed onto the partition can be provided via a CSV file. Refer to :doc:`NVS Partition Generator Utility ` for more details. + Application Example ------------------- diff --git a/docs/en/api-reference/storage/nvs_partition_gen.rst b/docs/en/api-reference/storage/nvs_partition_gen.rst new file mode 100644 index 000000000..c94a708c8 --- /dev/null +++ b/docs/en/api-reference/storage/nvs_partition_gen.rst @@ -0,0 +1 @@ +.. include:: /../../components/nvs_flash/nvs_partition_generator/README.rst diff --git a/docs/zh_CN/api-reference/storage/nvs_partition_gen.rst b/docs/zh_CN/api-reference/storage/nvs_partition_gen.rst new file mode 100644 index 000000000..c94a708c8 --- /dev/null +++ b/docs/zh_CN/api-reference/storage/nvs_partition_gen.rst @@ -0,0 +1 @@ +.. include:: /../../components/nvs_flash/nvs_partition_generator/README.rst