Re: [Fuego] [PATCH 01/12] add network testcase to spec

Sorry for reply late.

> Is there a reason all these individual stress tests are broken out into their own specs?
> This seems like a lot of new specs.  Do you actually run the networking tests separately, or are they usually run together?
In LTP's test suites, all these individual stress tests are broken out into their own suites.
All the suites can be tested in one spec, but you must modify the spec file when you want to test some of them.
When one spec has one suite, you can choose jobs for testing some of them.

Thanks,
 -- QiuTingting

-----Original Message-----
From: Bird, Tim [mailto:[email protected]] 
Sent: Thursday, July 9, 2020 5:01 AM
To: Qiu, Tingting/仇 婷婷 <[email protected]>; [email protected]
Subject: RE: [Fuego] [PATCH 01/12] add network testcase to spec



> -----Original Message-----
> From: Fuego <[email protected]> On Behalf Of 
> qiutt
> 
> include
>  net.features net.ipv6 net.ipv6_lib net.multicast  net.nfs net.rpc 
> net.rpc_tests net.sctp  net_stress.appl net_stress.broken_ip 
> net_stress.interface  net_stress.ipsec_dccp net_stress.ipsec_icmp 
> net_stress.ipsec_sctp  net_stress.ipsec_tcp net_stress.ipsec_udp 
> net_stress.multicast  net_stress.route net.tcp_cmds net.tirpc_tests 
> network_commands smack
> 
> Signed-off-by: qiutt <[email protected]>
> ---
>  tests/Functional.LTP/spec.json | 111 
> +++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 111 insertions(+)
> 
> diff --git a/tests/Functional.LTP/spec.json 
> b/tests/Functional.LTP/spec.json index d23edc1..8da77b0 100644
> --- a/tests/Functional.LTP/spec.json
> +++ b/tests/Functional.LTP/spec.json
> @@ -73,6 +73,117 @@
>              "tests": "cve",
>              "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
>              "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": 
> "skiplist.txt"}
> +        },
> +        "net.ipv6": {
> +            "tests": "net.ipv6",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net.rpc_tests": {
> +            "tests": "net.rpc_tests",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net.features": {
> +            "tests": "net.features",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net.ipv6_lib": {
> +            "tests": "net.ipv6_lib",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "BOARD_NAME": "m3ulcb"
> +        },
> +        "net.multicast": {
> +            "tests": "net.multicast",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net.nfs": {
> +            "tests": "net.nfs",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net.rpc": {
> +            "tests": "net.rpc",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net.sctp": {
> +            "tests": "net.sctp",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net_stress.appl": {
> +            "tests": "net_stress.appl",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net_stress.broken_ip": {
> +            "tests": "net_stress.broken_ip",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net_stress.interface": {
> +            "tests": "net_stress.interface",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net_stress.ipsec_dccp": {
> +            "tests": "net_stress.ipsec_dccp",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net_stress.ipsec_icmp": {
> +            "tests": "net_stress.ipsec_icmp",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net_stress.ipsec_sctp": {
> +            "tests": "net_stress.ipsec_sctp",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net_stress.ipsec_tcp": {
> +            "tests": "net_stress.ipsec_tcp",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net_stress.ipsec_udp": {
> +            "tests": "net_stress.ipsec_udp",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net_stress.multicast": {
> +            "tests": "net_stress.multicast",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net_stress.route": {
> +            "tests": "net_stress.route",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },

Is there a reason all these individual stress tests are broken out into their own specs?

> +        "net.tcp_cmds": {
> +            "tests": "net.tcp_cmds ",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "net.tirpc_tests": {
> +            "tests": "net.tirpc_tests",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "network_commands": {
> +            "tests": "network_commands",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
> +        "smack": {
> +            "tests": "smack",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": 
> + "skiplist.txt"}
>          }
>      }
>  }
> --
> 2.11.0

This seems like a lot of new specs.  Do you actually run the networking tests separately, or are they usually run together?

Thanks,
 -- Tim