Networking issues (ankh/lwip/drivers/?)

"Stark, Josef" <[email protected]> Fri, 13 Mar 2015 16:07:55 +0000

Newsgroups	gmane.comp.micro-kernel.l4.devel
Message-ID	<[email protected]>

Hi all,

while experimenting with network apps in L4Re, I experienced some issues:

1. (not important) DHCP isn't working:
The lwip example (ankh/examples/lwip) doesn't seem to work, neither with the integrated DHCP-Server of QEMU, nor with a dnsmasq running on a tap interface plugged into a vde_switch. Workaround: Replacing "dhcp_start(&my_netif);" with "netif_set_default(&my_netif); netif_set_up(&my_netif);" and, beforehand, setting the adress manually with "IP4_ADDR()" (#include <ipv4/lwip/ip4_addr.h>).
I also couldn't get DNS to work.

2. (not so important) some NIC-drivers aren't working:
QEMU-x86 can emulate the following interfaces: ne2k_pci, i82551, i82557b, i82559er, rtl8139, e1000, pcnet and virtio.
According to its Makefile, ankh supports the following interfaces: NE2K, LOOP, RTL8139, RTL8169, PCNET, E100, E1000, E1000E and UX.
However, of the intersection (ne2k_pci, rtl8139, pcnet und e1000) only ne2k_pci seems to work somewhat reliably.
The others simply don't work with or without some errors being shown by ankh.
With ne2k_pci I can only get a throughput of about 1MB/s, I'm guessing because the card is pretty old.

3. Sporadic delays/ errors:
Although this didn't happen too often, sometimes, when a client tried to send some small data to a server in another QEMU-instance, I experienced delays in the range of about 1 second before the receiver got the data.
On other occassions, a connection could not be established at all. However, no error messages were shown.

4. Multithreaded server issues:
I couldn't manage to get a multithreaded server to work. Neither a server,
a) which receives data on two different ports, each handled in a separate thread, nor
b) which receives data on a single port, handles however multiple clients simultaneously via separate threads,
did work.

Since a) and b) seem somewhat related I will elaborate only on b):

As long as just one client is connected to the server, sending and receiving on both ends works fine.
As soon as a second client (in another QEMU-instance, all connected via a vde_switch) tries to send data after connecting, both connections from server to the first and second client break down and no data gets through anymore. No errors shown. Finally the write() calls timeout.
Just as I wrote this, it did one time actually work, both clients connected, able to send and receive data. But I couldn't reproduce this success.
After removing the L4Re-specific parts of the code I compiled and ran it on Linux, where it does work flawlessly, even with 6 simultaneous client processes, not just 2.

I'm not sure why these issues occur, either there are bugs somewhere in L4Re's network stack or the NIC drivers, but it's also perfectly possible that I made some mistakes or that vde_switch doesn't work correctly (Although I did test whichever was possible (e.g. 4.a ) also only using simple QEMU TCP sockets). In order to be able to reproduce the results, I attached my configuration (see below and attachments).

Hopefully, someone can tell me what's going on here. I would really be glad.

Thank you!
Josef

----------------------------------------
Configuration:

First of all, I run "sudo vde_switch -daemon -mod 660 -group users". This is only neccessary once; the vde_switch is then used by all QEMU-instances.
To start the scenario, I use a simple script (see attachment run.sh), which starts all the QEMU-instances and sets different MAC-adresses for each instance (in cooperation with Makeconf.boot). server.c and client.c are heavily based on the lwip example.
While server_linux.c and client_linux.c are the stripped-down files I used for testing the network code on linux, all the other attached files are needed for running the L4Re-scenario.

_______________________________________________
l4-hackers mailing list
[email protected]
http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers

Makeconf.boot (application/octet-stream, 197 B) - not displayed

run.sh (application/x-shellscript, 385 B)

#!/bin/sh
#QEMU_OPTIONS="-net socket,listen=:8010" \
MAC="52:54:00:00:00:01" \
make qemu E=server &


sleep 1

#QEMU_OPTIONS="-net socket,connect=127.0.0.1:8010" \
QEMU_OPTIONS="-serial stdio" \
MAC="52:54:00:00:00:02" \
make qemu E=client &


sleep 1

#QEMU_OPTIONS="-net socket,connect=127.0.0.1:8010" \
QEMU_OPTIONS="-serial stdio" \
MAC="52:54:00:00:00:03" \
make qemu E=client &

client.cfg (application/octet-stream, 1.2 KB) - not displayed

server.cfg (application/octet-stream, 1.2 KB) - not displayed

modules.list (application/octet-stream, 1.6 KB) - not displayed

client.c (text/x-csrc, 3.5 KB)

#include <assert.h>
#include <getopt.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <pthread-l4.h>

#include <l4/util/util.h>
#include <l4/ankh/client-c.h>
#include <l4/ankh/lwip-ankh.h>
#include <l4/ankh/session.h>
/*
 * Need to include this file before others.
 * Sets our byteorder.
 */
#include "arch/cc.h"

#include <unistd.h>
#include <netdb.h>

#include "netif/etharp.h"
#include <lwip-util.h>
#include "lwip/dhcp.h"

#define GETOPT_LIST_END { 0, 0, 0, 0 }

enum options
{
	BUF_SIZE,
	SHM_NAME,
};

/* Network interface variables */
ip_addr_t ipaddr, netmask, gw;
struct netif netif;
/* Set network address variables */
#if 0
IP4_ADDR(&gw, 192,168,0,1);
IP4_ADDR(&ipaddr, 192,168,0,2);
IP4_ADDR(&netmask, 255,255,255,0);
#endif

struct netif my_netif;
extern err_t ankhif_init(struct netif*);

static ankh_config_info cfg = { 1024, L4_INVALID_CAP, L4_INVALID_CAP, "" };



static void server(void)
{
	int err, fd;
	struct sockaddr_in in, clnt;
	in.sin_family = AF_INET;
	in.sin_port = htons(3000);
	ip_addr_t target;
	IP4_ADDR(&target,192,168,0,2);
	in.sin_addr.s_addr = target.addr;

	int sock = socket(PF_INET, SOCK_STREAM, 0);
	printf("socket created: %d\n", sock);

	//err = bind(sock, (struct sockaddr*)&in, sizeof(in));
	err = connect(sock, (struct sockaddr*)&in, sizeof(in));
	printf("connected to addr: %d\n", err);

	//err = listen(sock, 10);
	//printf("listen(): %d\n", err);
	char buf[1024];
	strcpy(buf,"helloa\n");
	buf[7]=0;

	sleep(1); //otherwise wont work
	for (;;) {

		//socklen_t clnt_size = sizeof(clnt);
		//fd = accept(sock, (struct sockaddr*)&clnt, &clnt_size);

		//printf("Got connection from  ");
		//lwip_util_print_ip((ip_addr_t*)&clnt.sin_addr); printf("\n");

		//err = read(fd, buf, sizeof(buf));
		err = write(sock, buf, 8);
		printf("Written to fd %d (err %d)\n", sock, err);
		err = read(sock, buf, sizeof(buf));
		printf("Read from fd %d (err %d)\n", sock, err);
		printf("%s", buf);

		buf[5]++;
		sleep(2);
	}
}


int main(int argc, char **argv)
{
	IP4_ADDR(&gw, 192,168,0,1);
	//IP4_ADDR(&ipaddr, 192,168,0,3);
	IP4_ADDR(&ipaddr, 0,0,0,0);
	IP4_ADDR(&netmask, 255,255,255,0);

	if (l4ankh_init())
	  return 1;

	l4_cap_idx_t c = pthread_getl4cap(pthread_self());
	cfg.send_thread = c;

	static struct option long_opts[] = {
		{"bufsize", 1, 0, BUF_SIZE },
		{"shm", 1, 0, SHM_NAME },
		GETOPT_LIST_END
	};

	while (1) {
		int optind = 0;
		int opt = getopt_long(argc, argv, "b:s:", long_opts, &optind);
		printf("getopt: %d\n", opt);

		if (opt == -1)
			break;

		switch(opt) {
			case BUF_SIZE:
				printf("buf size: %d\n", atoi(optarg));
				cfg.bufsize = atoi(optarg);
				break;
			case SHM_NAME:
				printf("shm name: %s\n", optarg);
				snprintf(cfg.shm_name, CFG_SHM_NAME_SIZE, "%s", optarg);
				break;
			default:
				break;
		}
	}

	// Start the TCP/IP thread & init stuff
	tcpip_init(NULL, NULL);
	struct netif *n = netif_add(&my_netif,
	                            &ipaddr, &netmask, &gw,
	                            &cfg, // configuration state
	                            ankhif_init, ethernet_input);

	printf("netif_add: %p (%p)\n", n, &my_netif);
	printf("lsb hwaddr %i\n",my_netif.hwaddr[5]);
	assert(n == &my_netif);
	IP4_ADDR(&ipaddr, 192,168,0,(my_netif.hwaddr[5]%240)+3);
	netif_set_ipaddr(&my_netif,&ipaddr);

	netif_set_default(&my_netif);
	netif_set_up(&my_netif);

	while (!netif_is_up(&my_netif))
		l4_sleep(1000);
	printf("Network interface is up.\n");


	printf("IP: "); lwip_util_print_ip(&my_netif.ip_addr); printf("\n");
	printf("GW: "); lwip_util_print_ip(&my_netif.gw); printf("\n");


	server();

	return 0;
}

server.c (text/x-csrc, 3.9 KB)

#include <assert.h>
#include <getopt.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <pthread-l4.h>

#include <l4/util/util.h>
#include <l4/ankh/client-c.h>
#include <l4/ankh/lwip-ankh.h>
#include <l4/ankh/session.h>
/*
 * Need to include this file before others.
 * Sets our byteorder.
 */
#include "arch/cc.h"

#include <unistd.h>
#include <netdb.h>

#include "netif/etharp.h"
#include <lwip-util.h>
#include "lwip/dhcp.h"



#include <ipv4/lwip/ip4_addr.h>


#define GETOPT_LIST_END { 0, 0, 0, 0 }

enum options
{
	BUF_SIZE,
	SHM_NAME,
};

/* Network interface variables */
ip_addr_t ipaddr, netmask, gw;
struct netif netif;
/* Set network address variables */
#if 0
IP4_ADDR(&gw, 192,168,0,1);
IP4_ADDR(&ipaddr, 192,168,0,2);
IP4_ADDR(&netmask, 255,255,255,0);
#endif

struct netif my_netif;
extern err_t ankhif_init(struct netif*);

static ankh_config_info cfg = { 1024, L4_INVALID_CAP, L4_INVALID_CAP, "" };




static void *connection_handler(void *socket_desc)
{
    int fd = *(int*)socket_desc;
    char buf[1024];

	int err=1;
	while(err>0)
	{
		err = read(fd, buf, sizeof(buf));
		printf("Read from fd %d (err %d)\n", fd, err);
		printf("%s", buf);
		if(err<=0) break;
		err = write(fd, buf, strlen(buf)+1);
		printf("Written to fd %d (err %d)\n", fd, err);
	}
	printf("connection terminated\n");

    //Free the socket pointer
    free(socket_desc);

    return 0;
}




static void server(void)
{
	int* new_sock;
	int err, fd;
	struct sockaddr_in in, clnt;
	in.sin_family = AF_INET;
	in.sin_port = htons(3000);
	in.sin_addr.s_addr = my_netif.ip_addr.addr;

	int sock = socket(PF_INET, SOCK_STREAM, 0);
	printf("socket created: %d\n", sock);

	err = bind(sock, (struct sockaddr*)&in, sizeof(in));
	printf("bound to addr: %d\n", err);

	err = listen(sock, 10);
	printf("listen(): %d\n", err);

	for (;;) {
		char buf[1024];
		socklen_t clnt_size = sizeof(clnt);
		fd = accept(sock, (struct sockaddr*)&clnt, &clnt_size);

		printf("Got connection from  ");
		lwip_util_print_ip((ip_addr_t*)&clnt.sin_addr); printf("\n");

		pthread_t sniffer_thread;
		new_sock=malloc(sizeof(int));
		*new_sock=fd;


		pthread_create(&sniffer_thread,NULL,connection_handler,(void*)new_sock);

		printf("Handler assigned\n");
	//	err=1;
	//	while(err>0)
	//	{
	//		err = read(fd, buf, sizeof(buf));
	//		printf("Read from fd %d (err %d)\n", fd, err);
	//		printf("%s", buf);
	//	}
	//	printf("connection terminated\n");
	}
}


int main(int argc, char **argv)
{
	IP4_ADDR(&gw, 192,168,0,1);
	IP4_ADDR(&ipaddr, 192,168,0,2);
	IP4_ADDR(&netmask, 255,255,255,0);

	if (l4ankh_init())
	  return 1;

	l4_cap_idx_t c = pthread_getl4cap(pthread_self());
	cfg.send_thread = c;

	static struct option long_opts[] = {
		{"bufsize", 1, 0, BUF_SIZE },
		{"shm", 1, 0, SHM_NAME },
		GETOPT_LIST_END
	};

	while (1) {
		int optind = 0;
		int opt = getopt_long(argc, argv, "b:s:", long_opts, &optind);
		printf("getopt: %d\n", opt);

		if (opt == -1)
			break;

		switch(opt) {
			case BUF_SIZE:
				printf("buf size: %d\n", atoi(optarg));
				cfg.bufsize = atoi(optarg);
				break;
			case SHM_NAME:
				printf("shm name: %s\n", optarg);
				snprintf(cfg.shm_name, CFG_SHM_NAME_SIZE, "%s", optarg);
				break;
			default:
				break;
		}
	}

	// Start the TCP/IP thread & init stuff
	tcpip_init(NULL, NULL);
	struct netif *n = netif_add(&my_netif,
	                            &ipaddr, &netmask, &gw,
	                            &cfg, // configuration state
	                            ankhif_init, ethernet_input);

	printf("netif_add: %p (%p)\n", n, &my_netif);
	assert(n == &my_netif);

	netif_set_default(&my_netif);
	netif_set_up(&my_netif);

	while (!netif_is_up(&my_netif))
		l4_sleep(1000);
	printf("Network interface is up.\n");

	printf("IP: "); lwip_util_print_ip(&my_netif.ip_addr); printf("\n");
	printf("GW: "); lwip_util_print_ip(&my_netif.gw); printf("\n");

	pthread_t thread;
	//pthread_create(&thread,NULL,server,NULL);
	server();

	return 0;
}

client_linux.c (text/x-csrc, 1.2 KB)

#include <assert.h>
#include <getopt.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
//#include <pthread-l4.h>

#include <unistd.h>
#include <netdb.h>



static void server(void)
{
	int err, fd;
	struct sockaddr_in in, clnt;
	in.sin_family = AF_INET;
	in.sin_port = htons(3000);
	
//TODO
	in.sin_addr.s_addr = inet_addr("127.0.0.1");

	int sock = socket(PF_INET, SOCK_STREAM, 0);
	printf("socket created: %d\n", sock);

	//err = bind(sock, (struct sockaddr*)&in, sizeof(in));
	err = connect(sock, (struct sockaddr*)&in, sizeof(in));
	printf("connected to addr: %d\n", err);

	//err = listen(sock, 10);
	//printf("listen(): %d\n", err);
	char buf[1024];
	strcpy(buf,"helloa\n");
	buf[7]=0;

	sleep(1); //otherwise wont work
	for (;;) {

		//socklen_t clnt_size = sizeof(clnt);
		//fd = accept(sock, (struct sockaddr*)&clnt, &clnt_size);

		//printf("Got connection from  ");
		//lwip_util_print_ip((ip_addr_t*)&clnt.sin_addr); printf("\n");

		//err = read(fd, buf, sizeof(buf));
		err = write(sock, buf, 8);
		printf("Written to fd %d (err %d)\n", sock, err);
		err = read(sock, buf, sizeof(buf));
		printf("Read from fd %d (err %d)\n", sock, err);
		printf("%s", buf);

		buf[5]++;
		sleep(2);
	}
}


int main(int argc, char **argv)
{
	server();

	return 0;
}

server_linux.c (text/x-csrc, 1.6 KB)

 
#include <assert.h>
#include <getopt.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
//#include <pthread-l4.h>

#include <unistd.h>
#include <netdb.h>





static void *connection_handler(void *socket_desc)
{
    int fd = *(int*)socket_desc;
    char buf[1024];

	int err=1;
	while(err>0)
	{
		err = read(fd, buf, sizeof(buf));
		printf("Read from fd %d (err %d)\n", fd, err);
		printf("%s", buf);
		if(err<=0) break;
		err = write(fd, buf, strlen(buf)+1);
		printf("Written to fd %d (err %d)\n", fd, err);
	}
	printf("connection terminated\n");

    //Free the socket pointer
    free(socket_desc);

    return 0;
}




static void server(void)
{
	int* new_sock;
	int err, fd;
	struct sockaddr_in in, clnt;
	in.sin_family = AF_INET;
	in.sin_port = htons(3000);
	in.sin_addr.s_addr = 0;

	int sock = socket(PF_INET, SOCK_STREAM, 0);
	printf("socket created: %d\n", sock);

	err = bind(sock, (struct sockaddr*)&in, sizeof(in));
	printf("bound to addr: %d\n", err);

	err = listen(sock, 10);
	printf("listen(): %d\n", err);

	for (;;) {
		char buf[1024];
		socklen_t clnt_size = sizeof(clnt);
		fd = accept(sock, (struct sockaddr*)&clnt, &clnt_size);

		printf("Got connection from  ");

		pthread_t sniffer_thread;
		new_sock=malloc(sizeof(int));
		*new_sock=fd;


		pthread_create(&sniffer_thread,NULL,connection_handler,(void*)new_sock);

		printf("Handler assigned\n");
	//	err=1;
	//	while(err>0)
	//	{
	//		err = read(fd, buf, sizeof(buf));
	//		printf("Read from fd %d (err %d)\n", fd, err);
	//		printf("%s", buf);
	//	}
	//	printf("connection terminated\n");
	}
}


int main(int argc, char **argv)
{
	server();

	return 0;
}