We have fixed a bug in test script. Before that, we didn't do strict check when generate the mesh tree, which makes the test pass.
Temp disable these cases, to let CI pass before bug fixed.
1. The test step for error code REASON_ASSOC_EXPIRE is not reliable. SoftAP
could also send error code WIFI_REASON_NOT_AUTHED
2. STA will be disconnected now if softAP restart DHCP server
* Philosophical: "explicit is better than implicit".
* Practical: Allows useful errors if invalid directories given in components as the defaults aren't
always used. Also trims the -I path from a number of components that have no actual include
directory.
* Simplifies knowing which components will be header-only and which won't
use performance matrix to tracking bin size is not convenient. bin size
do change for a lot reasons.
we'll implement a new method to track it. before that, set a large
enough threshold, so we can still keep tracking it.
we'll auto assign cases to test jobs so we don't need those config
files. cases with different wifi modes will now be generated when
creating test case database, we need to remove them from YAML files.
CI: add reset between running each unit test cases
Unit test is designed to detect bug **within** each test case. Therefore we'll reset between each case to provide a clean context. We will later add stress cases to run unit test cases together to detect potential bugs.
See merge request !636
Add python script that parses list of unit test cases for CI from component test folder
Modify .gitlab-ci.yml to run this script as part of build unit tests stage
1. (^)TCPIP_DHCP_0101, (^)TCPIP_DHCP_0301: IDF set IP 0.0.0.0 may return error, don't check the result of setting IP to 0.0.0.0;
2. rename (^)TCPIP_DHCP_0102 to (^)TCPIP_DHCP_0212 as it's a DHCP server test case;
3. (^)TCPIP_TCP_0204,(^)TCPIP_TCP_0210,(^)TCPIP_UDP_0201,(^)TCPIP_UDP_0202: recv thread can't be deleted, change case to not create recv thread when create socket ;
4. (^)TCPIP_TCP_0206,(^)TCPIP_TCP_0212: query TCP server status format changed.
5. WIFI_SCAN_0301: check command error and test environment not correct
6. WIFI_SCAN_0302, WIFI_SCAN_0303, WIFI_SCAN_0304: test environment not correct
1. unsupported cases by "phy" command
2. DHCP server behavior different from ESP8266
3. TCP failed cases causes by unstable TCP behavior
4. UDP failed cases by poor UDP performance
1. test will be categorized by test level
2. add test level as attribute to test cases
3. will select TestCaseScript by the test cases added to CI (currently no test case uses test case script)
4. adding test level to test jobs
5. update .gitlab-ci.yml, each job need to set its test case file path
6. update .gitlab-ci.yml, test case path for test report is changed to idf_test
