b72d22041c
1. test will be categorized by test level 2. add test level as attribute to test cases 3. will select TestCaseScript by the test cases added to CI (currently no test case uses test case script) 4. adding test level to test jobs 5. update .gitlab-ci.yml, each job need to set its test case file path 6. update .gitlab-ci.yml, test case path for test report is changed to idf_test
61 lines
No EOL
2.8 KiB
Markdown
61 lines
No EOL
2.8 KiB
Markdown
|
|
# Note: The test cases in this folder are for Espressif internal use.
|
|
|
|
# Goto internal project wiki Testing page for detail about this folder.
|
|
|
|
## File Structure
|
|
|
|
```
|
|
test --- CIConfigs --- sanity_test1.yml (Runner config files)
|
|
| |-- stress_test1.yml
|
|
|-- TestCaseAll.yml (TestCaseFiles)
|
|
|-- TestEnvAll.yml (TestCaseFiles)
|
|
|-- InitialConditionAll.yml (TestCaseFiles)
|
|
|-- TestCaseScript --- ... (Test case scripts)
|
|
```
|
|
|
|
1. CIConfigs folder
|
|
* config for CI config files are put in this folder
|
|
* CI config files configs the cases and some other options for the CI job with same name
|
|
1. Test case files
|
|
* TestCaseAll.yml (test cases)
|
|
* InitialConditionAll.yml (initial conditions)
|
|
* TestEnvAll.yml (test environments)
|
|
* [how to modify test cases](https://gitlab.espressif.cn:6688/yinling/auto_test_script/blob/master/public/Design/TestCaseFiles.DesignNote.md)
|
|
1. Test case scripts
|
|
* some cases are implemented by specified script. those scripts are put in this folder.
|
|
|
|
|
|
## Modify test cases
|
|
|
|
1. check if the "SDK" of the test case only contain the current SDK
|
|
* if Yes, then just modify the test case behavior
|
|
* if No:
|
|
1. then remove current SDK name from "SDK" of the test case
|
|
2. Add a new test case, and set "SDK" only support current SDK name
|
|
2. use [auto_test_script](https://gitlab.espressif.cn:6688/yinling/auto_test_script) to load the modified case and verify the modification
|
|
3. create a merge request and assign to HYL (or add comment @yinling for merging test).
|
|
After review it will be merged to SDK and will be The cases will be synced to database in auto_test_script.
|
|
|
|
|
|
## Run test case locally
|
|
|
|
1. clone auto_test_script (ssh://git@gitlab.espressif.cn:27227/yinling/auto_test_script.git) from gitlab
|
|
2. create test environment:
|
|
1. search test case (search test case ID in components/test/TestCaseAll.yml, get the "test environment" value
|
|
2. goto [test environment list](https://gitlab.espressif.cn:6688/yinling/auto_test_script/blob/master/public/Documents/TestEnvList.md), find the link and goto the environment detail page
|
|
3. create test environment according to figure and description
|
|
4. [config test environment](https://gitlab.espressif.cn:6688/yinling/auto_test_script/blob/master/public/Design/TestEnvConfig.DesignNote.md). All parameters in table "Parameters require config before use" MUST be configured.
|
|
3. run test cases with [CIRunner.py](https://gitlab.espressif.cn:6688/yinling/auto_test_script/blob/master/public/Design/RunnerConfigs.DesignNote.md)
|
|
|
|
|
|
|
|
## exclude known issues for CI
|
|
the test cases listed in file "KnownIssues" will be excluded by CI when calculating results
|
|
|
|
Editing KnownIssues file is very simple, one single line for the ID for each case.
|
|
```
|
|
TCPIP_TCP_0101
|
|
TCPIP_TCP_0201
|
|
...
|
|
``` |