This change removes the earlier limitation of 1984 bytes for storing data-blobs.
Blobs larger than the sector size are split and stored on multiple sectors.
For this purpose, two new datatypes (multi-page index and multi-page data) are
added for entries stored in the sectors. The underlying read, write, erase and find
operations are modified to support these large blobs. The change is transparent
to users of the library and no special APIs need to be used to store these large
blobs.
Currently when page is being freed, items are individually moved from
FREEING page to ACTIVE page and erased. If power-off happens during the
process, the remaining entries are moved to ACTIVE page during recovery.
The problem with this approach is there may not be enough space on
ACTIVE page for all items if an item was partially written before
power-off and erased during recovery. This change moves all the items
from FREEING to ACTIVE page and then erased the FREEING page, If
power-off happens during the process, then ACTIVE page is erased and the
process is restarted.
* Philosophical: "explicit is better than implicit".
* Practical: Allows useful errors if invalid directories given in components as the defaults aren't
always used. Also trims the -I path from a number of components that have no actual include
directory.
* Simplifies knowing which components will be header-only and which won't
Ideally suited for generating a binary externally, containing key-value pairs specific
to device manufacturers. Utility allows creation of a binary, compatible
with NVS structure, which can be separately flashed onto a new
partition. This helps device manufacturers set different values for
different devices, e.g. serial numbers, but using a single firmaware
image.
Current code for recovery after power-off do not clean-up partially
erased items for FULL pages. If the erasure was part of modification
operation, this gets luckily cleaned-up because of duplicate detection
logic. For erase-only operation, the problem still exists. This patch
adds the recovery for FULL pages also.
Closes TW<20284>
When erasing a variable length item with an incorrect CRC32, the span
value of the item can not be trusted, so the item will be erased with
span = 1. Subsequent entries represent the data of the variable
length item, and these will be treated as separate items. For each
entry CRC32 is checked, the check most likely fails (because the
entry contains arbitrary data, and not a proper NVS item), and the
entry is erased. Erase function assumed that every item should be
present in cache, but it is not the case for the entries which are
just parts of item’s payload. This change allows for the item to be
not found in the hashlist, if the CRC32 check fails.
Current page selection algorithm selects a page for compaction based on just erased counts
and gives up when it does not find any page with erased count greater than 0. This is
problematic since the current allocation procedure skips the active page if there is not
enough room for the item in that page leaving free chunks on the pages. This change modifies
the algorithm to consider both erased as well as free counts on the candidate pages.
Closes TW<20297>
Users needs functions to count the number of free and used entries.
1. `nvs_get_stats()` This function return structure of statistic about the uspace NVS.
(Struct: used_entries, free_entries, total_entries and namespace_count)
2. `nvs_get_used_entry_count()` The second function return amount of entries in the namespace (by handler)
3. Added unit tests.
Closes TW<12282>
* Change snprintf for strlcat does not complain w/gcc7.2.0 and it is safer, thanks @projectgus
* Use proper quotes for character literals
Merges https://github.com/espressif/esp-idf/pull/1163
Previously, nvs_flash_init_custom would not be called if Storage for a
particular partition was already created. This caused issues if the
first call to nvs_flash_init failed due to Storage init error. This issue
exhibited itself as random failures of NVS CI test.
With this change, storage object is deleted (and not added to storage
list) if initialization fails.
Previously NVS did check CRC values of key-value pairs on the active
page, but the check for full pages was missing. This adds the necessary
check and a test for it.
This commit adds support for multiple NVS partitions. This provides application a flexibility to have multiple NVS
partitions such as separate partition with read-only manufacturing data and read-write partition with configuration.
Application can also use this to separate out application's configuration storage from system configuration.
This feature does not change any of the basic property of NVS subsystem. The same-named namespaces across partitions are
considered to be different namespaces. The original NVS API available for the applications remains unchanged. The only
difference is that instead of first NVS partition in the partition table, it now operates on the partition with label
"nvs" (which is default in the IDF provided partition table files). Additional APIs are provided to open a handle and
erase NVS with partition name as a parameter.
A test case is added in the host tests and it is made sure that all the host tests pass. nvs_rw_value app is also tested
with multiple partitions.
Signed-off-by: Amey Inamdar <amey.inamdar@gmail.com>
NVS is used to store PHY calibration data, WiFi configuration, and BT
configuration. Previously BT examples did not call nvs_flash_init,
relying on the fact that it is called during PHY init. However PHY init
did not handle possible NVS initialization errors.
This change moves PHY init procedure into the application, and adds
diagnostic messages to BT config management routines if NVS is not
initialized.
Writing values longer than half of the page size (with header taken into
account) causes fragmentation issues. Previously it was suggested on the
forum that using long values may cause issues, but this wasn’t checked
in the library itself, and wasn’t documented. This change adds necessary
checks and introduces the new error code.
Documentation is also fixed to reflect the fact that the maximum length
of the key is 15 characters, not 16.
Since read cache was introduced at page level, search cache became
useless in terms of reducing the number of flash read operations.
In addition to that, search cache used an assumption that if pointers to
keys are identical, the keys are also identical, which was proven wrong
by applications which generate key names dynamically.
This change removes CachedFindInfo, and all its uses. This is done at
expense of a small extra number of CPU operations (looking up a value in
the read cache is slightly more expensive) but no extra flash read
operations.
Ref TW12505
Ref https://github.com/espressif/arduino-esp32/issues/365
nvs_flash_init may return an error code in some cases, and applications
should check this error code (or at least assert on it being ESP_OK, to
make potential issues more immediately obvious).
This change modifies all the examples which use NVS to check the error
code. Most examples get a simple ESP_ERROR_CHECK assert, while NVS
examples, OTA example, and NVS unit tests get a more verbose check which
may be used in real applications.
This change adds a check for the free page count to nvs_flash_init.
Under normal operation, NVS keeps at least one free page available,
except for transient states such as freeing up new page. Due to external
factors (such as NVS partition size reduction) this free page could be
lost, making NVS operation impossible. Previously this would cause an
error when performing any nvs_set operation or opening a new namespace.
With this change, an error is returned from nvs_flash_init to indicate
that NVS partition is in such a state.
One common pattern of using assert function looks as follows:
int ret = do_foo();
assert(ret == 0); // which reads as: “do_foo should never fail here, by design”
The problem with such code is that if ‘assert’ is removed by the preprocessor in release build,
variable ret is no longer used, and the compiler issues a warning about this.
Changing assert definition in the way done here make the variable used, from language syntax perspective.
Semantically, the variable is still unused at run time (as sizeof can be evaluated at compile time), so the compiler
can optimize things away if possible.
rename nvs host test folder, modify .gitlab-ci.yml
remove unit-test-app build
re-format unit test files
remove extra newlines in project.mk
some refactoring for unit test part in project.mk
add build files of unit-test-app in gitignore
add README.md for unit test app
correct headings in README.md
remove files and make minor tweaks in unit test app
update .gitlab-ci.yml to use unit test app
delete unused lines in component_wrapper.mk
delete periph_i2s.h and lcd test
add text floating point in components/esp32/test/Kconfig
correct idf test build paths in .gitlab-ci.yml
Docs: new documentation and warnings cleanup
This change set
- adds a high-level description of application startup flow. Some parts are missing, but hopefully we can use this description as a base to expand on.
- adds a few notes about memory regions and their use in ESP-IDF.
- add SPI flash and partition APIs page
- fixes all Doxygen warnings in header files
- enables build failures on new Doxygen warnings
See merge request !201
New makefile component_wrapper.mk allows some variables to be set
before component.mk is evaluated. This properly fixes problems with
sdkconfig being hard to access in all phases of the build.
Including component_common.mk is no longer necessary and will print a
deprecation warning for components which use it.
When read caching was added, Page::findItem started modifying itemIndex reference argument even if item wasn't found.
Incidentally, Storage::findItem reused itemIndex when starting search at next page.
So,
- if the first page had a cached index (findItem was called for that page), and it pointed to a non-zero index,
- first page has a few empty items at the end (but is marked full),
- next search looked up the item on the second page,
- index of the item on the second page was less than the cached index on the first page,
then the search would fail because cached starting index was reused.
This change fixes both sides of the problem:
- Page::findItem shouldn't modify itemIndex argument if item is not found
- Storage::findItem should not reuse itemIndex between pages
Two tests have been added.
Fix spi_flash_write regression and nvs error recovery
In the previous set of changes related to spi_flash, new alignment checks were added. One of these checks, word alignment of `src` buffer in `nvs_flas_write`, was unnecessary. ROM `SPIWrite` function can handle unaligned source buffers.
This change caused an error to be returned to nvs for some legitimate write operations. Due to a bug in nvs, further write operations were possible, while the internal state of `Page` instance was broken. In WiFi functional tests, this inflicted havoc on the nvs partition, creating multiple duplicate items in the affected page. Because multiple duplicate items per page were never supposed to be handled during page load, duplicates were not removed. Thankfully this caused an assert at a later point in the loading process, otherwise this bug would be very difficult to detect.
This change set fixes the original spi_flash regression, handling of `INVALID` state of `nvs::Page`, and handling of duplicate items.
See merge request !161
Due to previous flash write bug it was possible to create multiple duplicate entries in a single page.
Recovery logic detected that case and bailed out with an assert.
This change adds graceful recovery from this condition.
Tests included.
Currently a restart is required to recover a page from invalid state.
The long-term solution is to detect such a condition and recover automatically (without a restart). This will be implemented in a separate change set.
Previously the test bench would check failure recovery by introducing error after each write operation.
This makes checks a bit more extensive (and much longer) by failing after every word written. Surprisingly, this change didn't expose any bugs.
nvs: fix memory leaks in HashList and nvs_close
Fixes TW8162.
Associated test case is run under Instruments on macOS, until I set up valgrind to test this automatically on Linux.
See merge request !150
spi_flash_read and spi_flash_write currently have a limitation that source and destination must be word-aligned.
This can be fixed by adding code paths for various unaligned scenarios, but function signatures also need to be adjusted.
As a first step (since we are pre-1.0 and can still change function signatures) alignment checks are added, and pointer types are relaxed to uint8_t.
Later we will add handling of unaligned operations.
This change also introduces spi_flash_erase_range and spi_flash_get_chip_size functions.
We probably need something like spi_flash_chip_size_detect which will detect actual chip size.
This is to allow single application binary to be used on a variety of boards and modules.
Introduces new internal function, Page::alterEntryRangeState, which gathers changes to multiple elements of entry state table into a single write, provided that these changes fall into a single word. This allows changing state of up to 16 entries in a single write.
Also adds new function, writeEntryData, which writes the whole payload of SZ and BLOB type entries in one go, instead of splitting it into multiple 32-byte writes.
This reduces number of writes required for SZ and BLOB entries.
Fixes problems with Eclipse trying to build in directories it shouldn't.
This is a breaking change for existing repositories, they need to rename
any component Makefiles to component.mk and rename their references to
$(IDF_PATH)/make/component.mk to $(IDF_PATH)/make/component_common.mk
This commit fixes several issues with state handling in nvs::Page. It also adds extra consistency checks in nvs::PageManger initialization.
These changes were verified with a new long-running test ("test recovery from sudden poweroff"). This test works by repeatedly performing same pseudorandom sequence of calls to nvs_ APIs. Each time it repeats the sequence, it introduces a failure into one of flash operations (write or erase). So if one iteration of this test needs, say, 25000 flash operations, then this test will run 25000 iterations, each time introducing the failure point at different location.
