receive hangs; was TCP send() performance and eventual hangs

zigg
Posts: 28
Joined: Wed Jul 01, 2009 2:42 pm

receive hangs; was TCP send() performance and eventual hangs

Post by zigg » Thu Jul 02, 2009 12:54 am

The topic of this thread has changed somewhat. In my code, I've taken TCP out of the equation. The TFTP implementation I'm using now is UDP-based, and we're addressing hangs seen there.

Original OP follows.

***

What I'm trying to do here is send the contents of the EEPROM/Flash (for my project, savehost) to a PC.

Here's a complete program that I think is the smallest reproduction of what I'm running into. Plug it into the arm9 example template as main.c, add -ldswifi9 to the libraries, and build. Use netcat to connect to port 1 on your DS (i.e. nc 1.2.3.4 1), which will kick off a neverending stream of bytes to your PC.

If you want to mirror the logic that I use in savehost right now, uncomment the vblanks and use the alternate to_send calculation.

What I'm running into is that maybe 80-90% of the time it seems to work okay, and quickly, but sometimes things choke up. I'll start getting alternating send() returns of 1024, then 396 (which add up to 1420, which seems to be the maximum bytes that can be sent in a single packet.) And when it chokes it goes really slow... and sometimes it hangs entirely.

I'm hoping that I'm merely something wrong with my use of send() and friends here. But I also think that I might be running into a problem with the TCP stack.

Code: Select all

#include <nds.h>
#include <dswifi9.h>

#include <netinet/in.h>
#include <stdio.h>

#define BUFSIZE 65536
#define PORT 1

#define min(a,b) (((a) < (b)) ? (a) : (b))

int main(void) {
	int serv_sock, sock, conn_addrlen, i, sent, to_send;
	struct sockaddr_in serv_addr, conn_addr;
	uint32 addr;
	char *buf, *buf_p;

	consoleDemoInit();

	iprintf("filling buffer\n");

	buf = malloc(BUFSIZE);
	for(i = 0; i < BUFSIZE; i ++) {
		buf[i] = (char)(i & 0xff);
	}

	iprintf("associating\n");

	Wifi_InitDefault(WFC_CONNECT);

	addr = Wifi_GetIP();
	iprintf("address is %d.%d.%d.%d\n", addr & 0xff, (addr >> 8) & 0xff,
			(addr >> 16) & 0xff, (addr >> 24) & 0xff);

	serv_sock = socket(AF_INET, SOCK_STREAM, 0);
	if(serv_sock == -1) return -1;

	serv_addr.sin_family = AF_INET;
	serv_addr.sin_port = htons(PORT);
	serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
	if(bind(serv_sock,
		(struct sockaddr *)&serv_addr,
		sizeof(serv_addr))) return -1;

	if(listen(serv_sock, 0)) return -1;

	iprintf("listening\n");

	conn_addrlen = sizeof(conn_addr);
	sock = accept(serv_sock, (struct sockaddr *)&conn_addr,
			&conn_addrlen);
	if(sock == -1) return -1;

	closesocket(serv_sock);

	iprintf("sending\n");

	while(1) {
		buf_p = buf;
		while((buf_p - buf) < BUFSIZE) {
			to_send = BUFSIZE - (buf_p - buf);
			/* to_send = min(BUFSIZE - (buf_p - buf), 1024); */
				
			iprintf("%d bytes at %d: ", to_send, (buf_p - buf));
			sent = send(sock, buf_p, to_send, 0);
			if(sent < 1) {
				iprintf("connection dropped\n");
				return -1;
			}
			iprintf("sent %d\n", sent);
			/* swiWaitForVBlank(); */
			/* swiWaitForVBlank(); */
			buf_p += sent;
		}
	}

}
Last edited by zigg on Tue Jul 07, 2009 4:52 pm, edited 1 time in total.

zigg
Posts: 28
Joined: Wed Jul 01, 2009 2:42 pm

Re: TCP send() performance and eventual hangs

Post by zigg » Thu Jul 02, 2009 2:58 am

A little more info: I ran wireshark on my PC while the slowdowns were happening. Normally there are a handful of duplicate ACKs and the occasional retransmission, but when the slowdowns kick in there seems to be a snowball of duplicate ACKs from the PC and repeated retransmissions from the DS—50 is not an uncommon number of duplicate ACKs to see.

Not really sure where to take it from here, though that does explain why I am generally able to send() 1420 bytes at a time—most all the packets coming from the DS have a payload of 1420 bytes; when one is finally determined to be acknowledged then there's clearly room for the next one.

elhobbs
Posts: 358
Joined: Thu Jul 02, 2009 1:19 pm

Re: TCP send() performance and eventual hangs

Post by elhobbs » Thu Jul 02, 2009 1:42 pm

I suspect when send returns less than 1420 that the internal send buffers are full and that you are exceeding the transmission rate on the ds. it is only 2Mbit/s - though I think people typically see 1Mbit/s at best. you could try waiting a frame or two if the send is smaller than 1420.

I think I remember reading somewhere where the author sgstair indicated that the retransmission code in the current version of dswifi is not particulary robust and that it can lead to excessive retransmissions if pushed too hard.

zigg
Posts: 28
Joined: Wed Jul 01, 2009 2:42 pm

Re: TCP send() performance and eventual hangs

Post by zigg » Thu Jul 02, 2009 1:54 pm

elhobbs wrote:I suspect when send returns less than 1420 that the internal send buffers are full and that you are exceeding the transmission rate on the ds. it is only 2Mbit/s - though I think people typically see 1Mbit/s at best. you could try waiting a frame or two if the send is smaller than 1420.
Actually, what I will get if I take out my vblanks and don't try to limit the send size (since I'm happily doing this in blocking mode) is a stream of 1420-sized packets. They just run a good deal more slowly than when the transmit initially starts and I'm sending ~1000 bytes at a time.
elhobbs wrote:I think I remember reading somewhere where the author sgstair indicated that the retransmission code in the current version of dswifi is not particulary robust and that it can lead to excessive retransmissions if pushed too hard.
Makes sense. I wish I knew how to "not push it too hard", though, without killing my performance outright. Ultimately, for this all to work out, I need to get all that data over the air, somehow. :)

I was thinking I might try a TFTP-like approach if there's no joy forthcoming here.

elhobbs
Posts: 358
Joined: Thu Jul 02, 2009 1:19 pm

Re: TCP send() performance and eventual hangs

Post by elhobbs » Fri Jul 03, 2009 3:29 am

if you are able to utilize the full 2 mbit bandwidth than this is only ~4 packets a frame. so, probably 2-3 packets per frame is going to be the best you can hope for - most likely less if there is a lot of other network traffic.

zigg
Posts: 28
Joined: Wed Jul 01, 2009 2:42 pm

Re: TCP send() performance and eventual hangs

Post by zigg » Sat Jul 04, 2009 12:05 am

So, just because it's a long weekend and I figured it might be fun anyway, I implemented a simple TFTP server.

The first time around, I did it entirely with blocking sockets and select() to implement timeouts. This one managed to shuffle off 3-5 1-megabyte payloads before hanging on the select().

Then I removed select() from the equation, switched to non-blocking, and used a loop calling swiWaitForVBlank() if I got EAGAIN, figuring a simplistic timeout just ofthe time() call. This time I got 15 1 megabyte payloads, before it started hanging again, I think on the recvfrom() call.

I don't necessarily think it's the TCP stack's fault, though I suspect it might be aggravating some sort of condition.

I've even tried using dswifi9d to try to dig and a printf for dbgprint, but it's no help, really. Messages look the same as any other.

Anyone got any ideas?

elhobbs
Posts: 358
Joined: Thu Jul 02, 2009 1:19 pm

Re: TCP send() performance and eventual hangs

Post by elhobbs » Sat Jul 04, 2009 2:32 am

do you have an exception handler in place? this will install the default one:

Code: Select all

defaultExceptionHandler();
it uses iprintf to display. make sure that you are using the latest version of libnds and dswifi. when libnds switched ipc to use fifo it introduced a lot of instability to the dswifi module. it has been adjusted a few times but I am not sure that it has been completely corrected.

zigg
Posts: 28
Joined: Wed Jul 01, 2009 2:42 pm

Re: TCP send() performance and eventual hangs

Post by zigg » Sat Jul 04, 2009 3:06 pm

elhobbs wrote:do you have an exception handler in place? this will install the default one:

Code: Select all

defaultExceptionHandler();
it uses iprintf to display.
Just did this. Didn't show anything new when it did hang. Though, exceptions? Maybe I'm going to show my ignorance here, but isn't that a C++ feature? Doing pure C here. :)
elhobbs wrote:make sure that you are using the latest version of libnds and dswifi. when libnds switched ipc to use fifo it introduced a lot of instability to the dswifi module. it has been adjusted a few times but I am not sure that it has been completely corrected.
I am using the latest release versions, yeah.

One thing I did notice when I did rebuild with dswifi9d and stuck dbgprint in there is that I get this just before the hang:

R:001AE90B008B 00224391F6B6 0800R:001AE90B008B 00224391F6B6 0800

Typically, only one of these R: are displayed; however every so often I will see at least two. And they always seem identical (though I haven't looked too far back as they scroll off rather quickly) ;)

I wonder if this is a low-level retransmit issue of some sort? wireshark isn't picking up any retransmits apart from the ones that get sent as part of what I coded in as part of my TFTP implementation.

Maybe it might be worth building my own copy of the library (eek!) ;) and sticking some trace prints in to get a better handle on what's going on.

(By the way, just because I don't think I made it clear, TFTP bypasses TCP entirely. It uses UDP exclusively. All protocol-level retransmits are entirely in the hands of my code, basically. My first attempt at this problem used TCP; I've taken it out of the equation now.)

elhobbs
Posts: 358
Joined: Thu Jul 02, 2009 1:19 pm

Re: TCP send() performance and eventual hangs

Post by elhobbs » Sat Jul 04, 2009 5:27 pm

it is to support hardware exceptions - accessing invalid memory addresses, stack overflow, etc.

can you post your source code and binary somewhere? I would be interested in taking a look.

melw
Posts: 13
Joined: Thu Mar 20, 2008 11:43 am

Re: TCP send() performance and eventual hangs

Post by melw » Sat Jul 04, 2009 7:18 pm

zigg wrote: One thing I did notice when I did rebuild with dswifi9d and stuck dbgprint in there is that I get this just before the hang:

R:001AE90B008B 00224391F6B6 0800R:001AE90B008B 00224391F6B6 0800
There are probably coming from Wifi_Update() in wifi_arm9.c, after reading a packet from the RX buffer. The RX buffer is checked in a loop, and obviously there's at least two packets to be processed when your application hangs. Pretty much only thing that happens after this is processing the received packet(s).
zigg wrote:Maybe it might be worth building my own copy of the library (eek!) ;) and sticking some trace prints in to get a better handle on what's going on.
Might be worth a shot. Or like Elhobbs suggested, code sharing has its advantages.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests