Commit graph

29 commits

Author SHA1 Message Date
Sagar Bijwe
5a27a63541 nvs-flash: Support for blobs larger than half of SPI Flash sector size
This change removes the earlier limitation of 1984 bytes for storing data-blobs.
Blobs larger than the sector size are split and stored on multiple sectors.
For this purpose, two new datatypes (multi-page index and multi-page data) are
added for entries stored in the sectors. The underlying read, write, erase and find
operations are modified to support these large blobs. The change is transparent
to users of the library and no special APIs need to be used to store these large
blobs.
2018-07-20 00:13:15 +05:30
Sagar Bijwe
7ae1df1c5e nvs: Fix recovery from power-off while page is being freed
Currently when page is being freed, items are individually moved from
FREEING page to ACTIVE page and erased. If power-off happens during the
process, the remaining entries are moved to ACTIVE page during recovery.
The problem with this approach is there may not be enough space on
ACTIVE page for all items if an item was partially written before
power-off and erased during recovery. This change moves all the items
from FREEING to ACTIVE page and then erased the FREEING page, If
power-off happens during the process, then ACTIVE page is erased and the
process is restarted.
2018-05-02 12:18:02 +05:30
Ivan Grokhotkov
17ab60d642 Merge branch 'bugfix/nvs_failed_crc_assert' into 'master'
nvs: don’t expect items with bad CRC to be in cache

See merge request idf/esp-idf!2239
2018-04-18 11:01:21 +08:00
Sagar Bijwe
9a3c4b7134 nvs: Fix recovery after power-off during erase operation
Current code for recovery after power-off do not clean-up partially
erased items for FULL pages. If the erasure was part of modification
operation, this gets luckily cleaned-up because of duplicate detection
logic. For erase-only operation, the problem still exists. This patch
adds the recovery for FULL pages also.

Closes TW<20284>
2018-04-17 12:24:21 +05:30
Ivan Grokhotkov
2c3644a09f nvs: don’t expect items with bad CRC to be in cache
When erasing a variable length item with an incorrect CRC32, the span
value of the item can not be trusted, so the item will be erased with
span = 1. Subsequent entries represent the data of the variable
length item, and these will be treated as separate items. For each
entry CRC32 is checked, the check most likely fails (because the
entry contains arbitrary data, and not a proper NVS item), and the
entry is erased. Erase function assumed that every item should be
present in cache, but it is not the case for the entries which are
just parts of item’s payload. This change allows for the item to be
not found in the hashlist, if the CRC32 check fails.
2018-04-16 09:36:17 +00:00
konstantin
c93626db3f nvs:Add functions for calculating used/free entries
Users needs functions to count the number of free and used entries.

1. `nvs_get_stats()` This function return structure of statistic about the uspace NVS.
(Struct: used_entries, free_entries, total_entries and namespace_count)
2. `nvs_get_used_entry_count()` The second function return amount of entries in the namespace (by handler)
3. Added unit tests.

Closes TW<12282>
2018-04-13 10:34:13 +05:00
Ivan Grokhotkov
c0269bc1cd nvs: check CRC of items on full pages
Previously NVS did check CRC values of key-value pairs on the active
page, but the check for full pages was missing. This adds the necessary
check and a test for it.
2017-09-04 14:37:33 +08:00
Ivan Grokhotkov
a25a4a0a7c nvs: check value size before writing, document limitations
Writing values longer than half of the page size (with header taken into
account) causes fragmentation issues. Previously it was suggested on the
forum that using long values may cause issues, but this wasn’t checked
in the library itself, and wasn’t documented. This change adds necessary
checks and introduces the new error code.

Documentation is also fixed to reflect the fact that the maximum length
of the key is 15 characters, not 16.
2017-05-31 12:59:24 +08:00
Ivan Grokhotkov
bf01525fc1 nvs: remove search cache at page level
Since read cache was introduced at page level, search cache became
useless in terms of reducing the number of flash read operations.
In addition to that, search cache used an assumption that if pointers to
keys are identical, the keys are also identical, which was proven wrong
by applications which generate key names dynamically.

This change removes CachedFindInfo, and all its uses. This is done at
expense of a small extra number of CPU operations (looking up a value in
the read cache is slightly more expensive) but no extra flash read
operations.

Ref TW12505
Ref https://github.com/espressif/arduino-esp32/issues/365
2017-05-12 12:18:08 +08:00
Ivan Grokhotkov
447ffb23d5 nvs: print page state as text in nvs_dump 2017-03-15 17:28:07 +08:00
Ivan Grokhotkov
0b265dc2a7 nvs, spi_flash: handle case when source data is in DROM 2016-11-18 20:11:17 +08:00
Ivan Grokhotkov
e314f42b0c nvs: fix Page::findItem and Storage::findItem regression
When read caching was added, Page::findItem started modifying itemIndex reference argument even if item wasn't found.
Incidentally, Storage::findItem reused itemIndex when starting search at next page.
So,
- if the first page had a cached index (findItem was called for that page), and it pointed to a non-zero index,
- first page has a few empty items at the end (but is marked full),
- next search looked up the item on the second page,
- index of the item on the second page was less than the cached index on the first page,
then the search would fail because cached starting index was reused.

This change fixes both sides of the problem:
- Page::findItem shouldn't modify itemIndex argument if item is not found
- Storage::findItem should not reuse itemIndex between pages

Two tests have been added.
2016-11-03 20:18:30 +08:00
Ivan Grokhotkov
abea6c50f1 nvs_flash: delete all duplicate entries in a page while loading
Due to previous flash write bug it was possible to create multiple duplicate entries in a single page.
Recovery logic detected that case and bailed out with an assert.
This change adds graceful recovery from this condition.
Tests included.
2016-10-31 21:10:47 +08:00
Ivan Grokhotkov
d9cdc7de58 nvs_flash: don't allow more operation to be done on page in PageState::INVALID
Currently a restart is required to recover a page from invalid state.
The long-term solution is to detect such a condition and recover automatically (without a restart). This will be implemented in a separate change set.
2016-10-31 19:48:28 +08:00
Ivan Grokhotkov
e34fc7a46c Merge branch 'bugfix/nvs_leaks' into 'master'
nvs: fix memory leaks in HashList and nvs_close

Fixes TW8162.
Associated test case is run under Instruments on macOS, until I set up valgrind to test this automatically on Linux.

See merge request !150
2016-10-31 17:30:00 +08:00
Ivan Grokhotkov
9f0f05d520 spi_flash: change pointer type to void* 2016-10-27 17:58:42 +08:00
Ivan Grokhotkov
2c5340d47e spi_flash: change argument types
spi_flash_read and spi_flash_write currently have a limitation that source and destination must be word-aligned.
This can be fixed by adding code paths for various unaligned scenarios, but function signatures also need to be adjusted.
As a first step (since we are pre-1.0 and can still change function signatures) alignment checks are added, and pointer types are relaxed to uint8_t.
Later we will add handling of unaligned operations.
This change also introduces spi_flash_erase_range and spi_flash_get_chip_size functions.

We probably need something like spi_flash_chip_size_detect which will detect actual chip size.
This is to allow single application binary to be used on a variety of boards and modules.
2016-10-27 17:58:42 +08:00
Ivan Grokhotkov
74a658c765 nvs: fix memory leaks in HashList and nvs_close
Fixes TW8162.
Associated test case is run under Instruments on macOS, until I set up valgrind to test this automatically on Linux.
2016-10-26 12:25:53 +08:00
Ivan Grokhotkov
1cd572c7b9 Add test for compiling in release mode, fix warnings and errors which appeared 2016-10-17 13:40:10 +08:00
Ivan Grokhotkov
2a68f60874 components/nvs: add erase function
This change exposes functions to erase single key and to erase all keys from namespace.
TW6769, TW6839
2016-09-23 08:08:03 +08:00
Ivan Grokhotkov
e87d80d478 components/nvs: fix formatting 2016-09-22 21:05:47 +08:00
Ivan Grokhotkov
076141aab9 components/nvs: batch writes when possible
Introduces new internal function, Page::alterEntryRangeState, which gathers changes to multiple elements of entry state table into a single write, provided that these changes fall into a single word. This allows changing state of up to 16 entries in a single write.
Also adds new function, writeEntryData, which writes the whole payload of SZ and BLOB type entries in one go, instead of splitting it into multiple 32-byte writes.
This reduces number of writes required for SZ and BLOB entries.
2016-09-22 21:05:17 +08:00
Ivan Grokhotkov
12a0786e2a components/nvs: maintain item hash list at page level 2016-09-21 18:02:52 +08:00
Ivan Grokhotkov
f06ebeba86 components/nvs: avoid reading just-erased page 2016-09-21 18:02:52 +08:00
Ivan Grokhotkov
9ef827ae20 components/nvs: strlcpy is not available on Linux, replace with strncpy and terminate strings explicitly 2016-08-23 15:14:13 +08:00
Ivan Grokhotkov
7998b6ca2e components/nvs: handle more cases where sudden power off may happen
This commit fixes several issues with state handling in nvs::Page. It also adds extra consistency checks in nvs::PageManger initialization.
These changes were verified with a new long-running test ("test recovery from sudden poweroff"). This test works by repeatedly performing same pseudorandom sequence of calls to nvs_ APIs. Each time it repeats the sequence, it introduces a failure into one of flash operations (write or erase). So if one iteration of this test needs, say, 25000 flash operations, then this test will run 25000 iterations, each time introducing the failure point at different location.
2016-08-23 12:08:00 +08:00
Ivan Grokhotkov
f04c894123 components/nvs: add debugging facilities and runtime checks
Extra runtime sanity checks run when compiled for the host (i.e. with no ESP_PLATFORM define)
2016-08-23 12:08:00 +08:00
Ivan Grokhotkov
3ce433cfd1 components/nvs: make some methods const 2016-08-23 12:08:00 +08:00
Ivan Grokhotkov
bd6ea4393c Initial public version 2016-08-17 23:08:22 +08:00