Crash_Override/OVMS3-idf
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

nvs: check value size before writing, document limitations

Browse source 
Writing values longer than half of the page size (with header taken into
account) causes fragmentation issues. Previously it was suggested on the
forum that using long values may cause issues, but this wasn’t checked
in the library itself, and wasn’t documented. This change adds necessary
checks and introduces the new error code.

Documentation is also fixed to reflect the fact that the maximum length
of the key is 15 characters, not 16.
This commit is contained in:
Ivan Grokhotkov 2017-05-31 12:59:24 +08:00
parent d718cbd873
commit a25a4a0a7c
5 changed files with 70 additions and 19 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
components/nvs_flash/README.rst
View file

@ -15,6 +15,8 @@ Future versions of this library may add other storage backends to keep data in a
.. note:: if an NVS partition is truncated (for example, when the partition table layout is changed), its contents should be erased. ESP-IDF build system provides a ``make erase_flash`` target to erase all contents of the flash chip.
.. note:: NVS works best for storing many small values, rather than a few large values of type 'string' and 'blob'. If storing large blobs or strings is required, consider using the facilities provided by the FAT filesystem on top of the wear levelling library.
Keys and values
^^^^^^^^^^^^^^^
@ -24,6 +26,9 @@ NVS operates on key-value pairs. Keys are ASCII strings, maximum key length is c
- zero-terminated string
- variable length binary data (blob)
.. note::
String and blob values are currently limited to 1984 bytes. For strings, this includes the null terminator.
Additional types, such as ``float`` and ``double`` may be added later.
Keys are required to be unique. Writing a value for a key which already exists behaves as follows:

22
components/nvs_flash/include/nvs.h
View file

@ -42,6 +42,7 @@ typedef uint32_t nvs_handle;
#define ESP_ERR_NVS_INVALID_STATE (ESP_ERR_NVS_BASE + 0x0b) /*!< NVS is in an inconsistent state due to a previous error. Call nvs_flash_init and nvs_open again, then retry. */
#define ESP_ERR_NVS_INVALID_LENGTH (ESP_ERR_NVS_BASE + 0x0c) /*!< String or blob length is not sufficient to store data */
#define ESP_ERR_NVS_NO_FREE_PAGES (ESP_ERR_NVS_BASE + 0x0d) /*!< NVS partition doesn't contain any empty pages. This may happen if NVS partition was truncated. Erase the whole partition and call nvs_flash_init again. */
#define ESP_ERR_NVS_VALUE_TOO_LONG (ESP_ERR_NVS_BASE + 0x0c) /*!< String or blob length is longer than supported by the implementation */
/**
* @brief Mode of opening the non-volatile storage
@ -61,7 +62,7 @@ typedef enum {
*
* @param[in] name Namespace name. Maximal length is determined by the
* underlying implementation, but is guaranteed to be
* at least 16 characters. Shouldn't be empty.
* at least 15 characters. Shouldn't be empty.
* @param[in] open_mode NVS_READWRITE or NVS_READONLY. If NVS_READONLY, will
* open a handle for reading only. All write requests will
* be rejected for this handle.
@ -89,8 +90,10 @@ esp_err_t nvs_open(const char* name, nvs_open_mode open_mode, nvs_handle *out_ha
* Handles that were opened read only cannot be used.
* @param[in] key Key name. Maximal length is determined by the underlying
* implementation, but is guaranteed to be at least
* 16 characters. Shouldn't be empty.
* 15 characters. Shouldn't be empty.
* @param[in] value The value to set.
* For strings, the maximum length (including null character) is
* 1984 bytes.
*
* @return
* - ESP_OK if value was set successfully
@ -103,6 +106,7 @@ esp_err_t nvs_open(const char* name, nvs_open_mode open_mode, nvs_handle *out_ha
* write operation has failed. The value was written however, and
* update will be finished after re-initialization of nvs, provided that
* flash operation doesn't fail again.
* - ESP_ERR_NVS_VALUE_TOO_LONG if the string value is too long
*/
esp_err_t nvs_set_i8 (nvs_handle handle, const char* key, int8_t value);
esp_err_t nvs_set_u8 (nvs_handle handle, const char* key, uint8_t value);
@ -123,11 +127,10 @@ esp_err_t nvs_set_str (nvs_handle handle, const char* key, const char* value);
*
* @param[in] handle Handle obtained from nvs_open function.
* Handles that were opened read only cannot be used.
* @param[in] key Key name. Maximal length is determined by the underlying
* implementation, but is guaranteed to be at least
* 16 characters. Shouldn't be empty.
* @param[in] key Key name. Maximal length is 15 characters. Shouldn't be empty.
* @param[in] value The value to set.
* @param[in] length length of binary value to set, in bytes.
* @param[in] length length of binary value to set, in bytes; Maximum length is
* 1984 bytes.
*
* @return
* - ESP_OK if value was set successfully
@ -140,6 +143,7 @@ esp_err_t nvs_set_str (nvs_handle handle, const char* key, const char* value);
* write operation has failed. The value was written however, and
* update will be finished after re-initialization of nvs, provided that
* flash operation doesn't fail again.
* - ESP_ERR_NVS_VALUE_TOO_LONG if the value is too long
*/
esp_err_t nvs_set_blob(nvs_handle handle, const char* key, const void* value, size_t length);
@ -169,7 +173,7 @@ esp_err_t nvs_set_blob(nvs_handle handle, const char* key, const void* value, si
* @param[in] handle Handle obtained from nvs_open function.
* @param[in] key Key name. Maximal length is determined by the underlying
* implementation, but is guaranteed to be at least
* 16 characters. Shouldn't be empty.
* 15 characters. Shouldn't be empty.
* @param out_value Pointer to the output value.
* May be NULL for nvs_get_str and nvs_get_blob, in this
* case required length will be returned in length argument.
@ -230,7 +234,7 @@ esp_err_t nvs_get_u64 (nvs_handle handle, const char* key, uint64_t* out_value);
* @param[in] handle Handle obtained from nvs_open function.
* @param[in] key Key name. Maximal length is determined by the underlying
* implementation, but is guaranteed to be at least
* 16 characters. Shouldn't be empty.
* 15 characters. Shouldn't be empty.
* @param out_value Pointer to the output value.
* May be NULL for nvs_get_str and nvs_get_blob, in this
* case required length will be returned in length argument.
@ -262,7 +266,7 @@ esp_err_t nvs_get_blob(nvs_handle handle, const char* key, void* out_value, size
*
* @param[in] key Key name. Maximal length is determined by the underlying
* implementation, but is guaranteed to be at least
* 16 characters. Shouldn't be empty.
* 15 characters. Shouldn't be empty.
*
* @return
* - ESP_OK if erase operation was successful

4
components/nvs_flash/src/nvs_page.cpp
View file

@ -175,6 +175,10 @@ esp_err_t Page::writeItem(uint8_t nsIndex, ItemType datatype, const char* key, c
if (keySize > Item::MAX_KEY_LENGTH) {
return ESP_ERR_NVS_KEY_TOO_LONG;
}
if (dataSize > Page::BLOB_MAX_SIZE) {
return ESP_ERR_NVS_VALUE_TOO_LONG;
}
size_t totalSize = ENTRY_SIZE;
size_t entriesCount = 1;

2
components/nvs_flash/src/nvs_page.hpp
View file

@ -45,6 +45,8 @@ public:
static const size_t ENTRY_SIZE = 32;
static const size_t ENTRY_COUNT = 126;
static const uint32_t INVALID_ENTRY = 0xffffffff;
static const size_t BLOB_MAX_SIZE = ENTRY_SIZE * (ENTRY_COUNT / 2 - 1);
static const uint8_t NS_INDEX = 0;
static const uint8_t NS_ANY = 255;

56
components/nvs_flash/test_nvs_host/test_nvs.cpp
View file

@ -231,6 +231,34 @@ TEST_CASE("different key names are distinguished even if the pointer is the same
}
}
TEST_CASE("Page validates key size", "[nvs]")
{
SpiFlashEmulator emu(4);
Page page;
TEST_ESP_OK(page.load(0));
// 16-character key fails
TEST_ESP_ERR(page.writeItem(1, "0123456789123456", 1), ESP_ERR_NVS_KEY_TOO_LONG);
// 15-character key is okay
TEST_ESP_OK(page.writeItem(1, "012345678912345", 1));
}
TEST_CASE("Page validates blob size", "[nvs]")
{
SpiFlashEmulator emu(4);
Page page;
TEST_ESP_OK(page.load(0));
char buf[2048] = { 0 };
// There are two potential errors here:
// - not enough space in the page (because one value has been written already)
// - value is too long
// Check that the second one is actually returned.
TEST_ESP_ERR(page.writeItem(1, ItemType::BLOB, "2", buf, Page::ENTRY_COUNT * Page::ENTRY_SIZE), ESP_ERR_NVS_VALUE_TOO_LONG);
// Should fail as well
TEST_ESP_ERR(page.writeItem(1, ItemType::BLOB, "2", buf, Page::BLOB_MAX_SIZE + 1), ESP_ERR_NVS_VALUE_TOO_LONG);
TEST_ESP_OK(page.writeItem(1, ItemType::BLOB, "2", buf, Page::BLOB_MAX_SIZE));
}
TEST_CASE("can init PageManager in empty flash", "[nvs]")
{
SpiFlashEmulator emu(4);
@ -327,17 +355,19 @@ TEST_CASE("storage can find items on second page if first is not fully written a
Storage storage;
CHECK(storage.init(0, 3) == ESP_OK);
int bar = 0;
uint8_t bigdata[100 * 32] = {0};
uint8_t bigdata[Page::BLOB_MAX_SIZE] = {0};
// write one big chunk of data
ESP_ERROR_CHECK(storage.writeItem(0, ItemType::BLOB, "first", bigdata, sizeof(bigdata)));
ESP_ERROR_CHECK(storage.writeItem(0, ItemType::BLOB, "1", bigdata, sizeof(bigdata)));
// write another big chunk of data
ESP_ERROR_CHECK(storage.writeItem(0, ItemType::BLOB, "2", bigdata, sizeof(bigdata)));
// write second one; it will not fit into the first page
ESP_ERROR_CHECK(storage.writeItem(0, ItemType::BLOB, "second", bigdata, sizeof(bigdata)));
// write third one; it will not fit into the first page
ESP_ERROR_CHECK(storage.writeItem(0, ItemType::BLOB, "3", bigdata, sizeof(bigdata)));
size_t size;
ESP_ERROR_CHECK(storage.getItemDataSize(0, ItemType::BLOB, "first", size));
ESP_ERROR_CHECK(storage.getItemDataSize(0, ItemType::BLOB, "1", size));
CHECK(size == sizeof(bigdata));
ESP_ERROR_CHECK(storage.getItemDataSize(0, ItemType::BLOB, "second", size));
ESP_ERROR_CHECK(storage.getItemDataSize(0, ItemType::BLOB, "3", size));
CHECK(size == sizeof(bigdata));
}
@ -1128,15 +1158,21 @@ TEST_CASE("read/write failure (TW8406)", "[nvs]")
TEST_CASE("nvs_flash_init checks for an empty page", "[nvs]")
{
const size_t blob_size = 2048; // big enough so that only one can fit into a page
const size_t blob_size = Page::BLOB_MAX_SIZE;
uint8_t blob[blob_size] = {0};
SpiFlashEmulator emu(5);
TEST_ESP_OK( nvs_flash_init_custom(0, 5) );
nvs_handle handle;
TEST_ESP_OK( nvs_open("test", NVS_READWRITE, &handle) );
TEST_ESP_OK( nvs_set_blob(handle, "1", blob, blob_size) );
TEST_ESP_OK( nvs_set_blob(handle, "2", blob, blob_size) );
TEST_ESP_OK( nvs_set_blob(handle, "3", blob, blob_size) );
// Fill first page
TEST_ESP_OK( nvs_set_blob(handle, "1a", blob, blob_size) );
TEST_ESP_OK( nvs_set_blob(handle, "1b", blob, blob_size) );
// Fill second page
TEST_ESP_OK( nvs_set_blob(handle, "2a", blob, blob_size) );
TEST_ESP_OK( nvs_set_blob(handle, "2b", blob, blob_size) );
// Fill third page
TEST_ESP_OK( nvs_set_blob(handle, "3a", blob, blob_size) );
TEST_ESP_OK( nvs_set_blob(handle, "3b", blob, blob_size) );
TEST_ESP_OK( nvs_commit(handle) );
nvs_close(handle);
// first two pages are now full, third one is writable, last two are empty