Downloading large files

[Derrik]

I've got some code which can create and send an HTTP request but I'm having trouble saving the response.

Here's a snippet of my code:

Code: Select all

	char doneHeader = 0;
	int recvd_len;
	char incoming_buffer[BUFFER_SIZE];
	
	int dataIn = 0;
	
	while((recvd_len = recv(my_socket, incoming_buffer, BUFFER_SIZE, 0)) != 0) {
		if(recvd_len > 0) {
			int contentLength = recvd_len;
			char *contentBuffer = incoming_buffer;
			if(!doneHeader) {
				contentLength -= getDataStart(incoming_buffer) - incoming_buffer;
				contentBuffer = getDataStart(incoming_buffer);
				doneHeader = 1;
			}
			FILE *o = fopen(localFilename, "ab");
			fwrite(contentBuffer, contentLength, 1, o);
			fclose(o);
			dataIn += BUFFER_SIZE;
			printf("%d / 1,424,880\n", dataIn);
		}
	}
	
	iprintf("Done!\n");

What I am trying to do is download the file in chunks of BUFFER_SIZE (which is currently 1024) by appending the response to the end of the file each time new data is received.

The problem is, my DS outputs this:

1024 / 1,424,880
2048 / 1,424,880
...
105472 / 1,424,880

And then freezes.

My guess is that the DS is running out of RAM because dswifi isn't freeing the data that I've already written to the file and don't need anymore.

So how can I get dswifi to download the file in chunks, freeing up memory once it has been delt with?

[Derrik]

OK; I ran the program again and got to "267264 / 1,424,880" before freezing.

What's going on?

[WinterMute]

Opening and closing the file for each iteration of the loop probably isn't a good idea although it should work in theory.

There's no guarantee that recv will wait for BUFFER_SIZE bytes before returning, you should loop until the buffer is full if you want to save in BUFFER_SIZE chunks. You could also just use recvd_len as the number of bytes to write - libfat will buffer before writing to SD anyway.

[Derrik]

Thanks, I've modified my code to only open and close the file once:

Code: Select all

    FILE *o = fopen(localFilename, "ab");
    while((recvd_len = recv(my_socket, incoming_buffer, BUFFER_SIZE, 0)) != 0) {
      if(recvd_len > 0) {
         int contentLength = recvd_len;
         char *contentBuffer = incoming_buffer;
         if(!doneHeader) {
            contentLength -= getDataStart(incoming_buffer) - incoming_buffer;
            contentBuffer = getDataStart(incoming_buffer);
            doneHeader = 1;
         }
         fwrite(contentBuffer, contentLength, 1, o);
         dataIn += BUFFER_SIZE;
         printf("%d\n", dataIn);
      }
   }
   
   iprintf("Done!\n");
   fclose(o);

It usually is contentLength usually is recvd_len, apart from the first receive which needs to take away the size of the HTTP header.

I'm not really sure what else to try, it is still freezing at arbitrary points. The protocol is TCP so I shouldn't need to do my own timeout -> retry code should I?

[Derrik]

My code is a lot more robust, I use the HTTP command "Range: bytes=start-end\r\n" in my header so I can download the file from the server in parts.

The code all works, it's just that it randomly freezes.

In the myConnect function I set my socket to be nonblocking:

Code: Select all

fcntl(my_socket, F_SETFL, O_NONBLOCK);

So the code shouldn't just be waiting for recvd function, since it is nonblocking. However once the application freezes I cannot get anything else to happen, it is frozen.

The only thing I can think of is that dswifi doesn't use this method to be nonblocking:

Code: Select all

fcntl(my_socket, F_SETFL, O_NONBLOCK);

And that my code hasn't frozen, it is just stuck at recv function because my sockets are blocking.

Please confirm that this code should make my sockets non blocking.

I tried this:

Code: Select all

u_long one = 1;
ioctlsocket(my_socket, FIONBIO, &one);

But I get compile errors (can't find ioctlsocket or FIONBIO).

[Derrik]

Found it, it was:

ioctl()

Not ioctlsocket()

Like on Windows.

[Derrik]

So now my sockets are non-blocking, I can see if it has been 5 seconds without receiving data, if so, retry.

Everytime I retry, it fails to download any more data.

It will go to like:

Downloading part 73...
Downloading part 74...
Downloading part 75...
Timout!
Downloading part 75...
Timout!
Downloading part 75...
...

Note, the amount of data changes each time, sometimes it makes it to part 63, other times, 75, other times 84, etc...

Is there some kind of way of flushing dswifi buffers, so that when I get to this point, I can totally retry from the beginning.

[Derrik]

I've even tried this:

Code: Select all

			shutdown(my_socket, 0);
			closesocket(my_socket);
			
			Wifi_DisconnectAP();
			
			if(!Wifi_InitDefault(WFC_CONNECT)) {
				printf(" Failed to connect to WiFi!\n");
			}
			
			printf(" Reconnected to WiFi\n");
			myConnect(host);
			
			printf(" Reconnected to host\n");

But it's not enough to get the DS to start receiving data again.

[Derrik]

Although using the code from hbmenu to reboot the NDS file, and the DS can receive data again.

I've tried to take the code from hbmenu that resets cache so that the DS can receive data again, but it is hard to isolate the WiFi reset code.

[Derrik]

OK; when I try to connect after recreating the socket (after shutting it down, closing it, calling socket) I get errno 119:

Code: Select all

errno = 0;
if(myConnect() == SOCKET_ERROR) {
	printf(" connect() failed (%d)!\n", errno);
}

(myConnect just returns what connect() returns).

I've looked up the error code:

Code: Select all

#define ENAVAIL     119 /* No XENIX semaphores available */

What does this mean?