dslink bugfixes...

nocash
Posts: 8
Joined: Mon Sep 26, 2016 10:21 pm

dslink bugfixes...

Post by nocash » Mon Nov 21, 2016 4:30 pm

Just checked the recent (inofficial?) dslink binary, and its recent source code changes...

The binary boots some of my files (eg. magicflr), but seems to always hang with white screens upon other files.

The arm9/arm7 stack pointers are still wrong. Oh, but ignore the values from the dslink email bug report (I've mistyped some 380Fxxxh values as 3803xxxh). Correct values should be:
ARM7 sp_svc = 380FFC0h
ARM7 sp_irq = 380FF80h
ARM7 sp_sys = 380FD80h
ARM9 sp_svc = 3003FC0h
ARM9 sp_irq = 3003F80h
ARM9 sp_sys = 3002F7Ch

Relocating ARM7 binary from 2000000h to actual ARM7 dest looks wrong if source and dest do overlap. Or wait, one of your recent changes seems to have specifically fixed an issue for overlapping memcopy... looks as if it's just fixing that problem?

For ARM9, just clearing the PROTECT_ENABLE bit isn't enough, that does still leave some other bits at wrong state (eg. ITCM remains enabled, which isn't the correct boot state). Better use this code:
ldr r0,=00012078h
mov p15,0,c1,c0,0,r0
Of course, that disables ITCM, so you can't execute your relocated code in ITCM. But there's no reason to relocate the final bootcode to ITCM anyways, you could as well execute in Main RAM. Original NDS firmware would relocate the final bootcode to 23FEE00h..23FEFFFh, but as you aren't zerofilling the memory occupied by dslink, you could as well execute the bootcode without any relocation.
Setting PROTECT_ENABLE before jumping to ARM9 entrypoint is also wrong. The protection unit should be kept switched off.
And, value 02FFF017h for PU region 7 is wrong (should be 027FF017h).
Some possibly important other settings might be POWCNT1=820Fh and EXMEMCNT=E880h (not checked if you already have them as so). And less important, you could also set DIVDENOM=1 and such things.

I've also got my own dswifi (and dslink) ASM port finished. The good news is that it does successfully start the uploaded file after EVERY upload. On the other hand, my uploader does occassionally hang after DHCP_Start or after the final association Grat message. I am not sure if it's a problem in my ASM port, or if it does also occur in Stephen's original dswifi library. Did you encounter that kind of problem, too?

My overall impression with previous dslink version has been that it crashed about each second time AFTER upload, but rarely crashed BEFORE upload when connecting to the access point (which unfortunately happens in my ASM version, results there are around 90% successful connections, and 10% unsuccessful ones, requiring to reboot the uploader in that case).

nocash
Posts: 8
Joined: Mon Sep 26, 2016 10:21 pm

Re: dslink bugfixes...

Post by nocash » Tue Nov 22, 2016 12:44 am

Some more things in ARM9 main...

In the progressRead, I would clip the chunksize to min 1024 bytes (or to the bootblock size if it's smaller than 1024). Currently, when receiving a small 100 byte bootblock, your code would transfer it in tiny 1-byte chunks which is kinda inefficient (although the actual wifi hardware should transfer it all in one packet).
And the worst case, if the bootblock is SMALLER than 100, then your "chunksize=size/100" would probably just cause the function to hang, trying to receive 0-byte blocks.

And the while loop in the main function, I would split it into two loops, one for the UDP part, and after finishing the UDP stuff, a second loop for the TCP part. Then you could also use the PC's actual IP address from "sa_udp_remote" for the TCP part. That would look better than INADDR_ANY to me.

Though it might be good to keep handling UDP in the second loop too (in case the UDP response got lost). But when repeating UDP responses, one should also close the UDP response (instead of just adding more and more sockets on each response).

For my connect problems, I've found some general bug in my ASM code (after Stephen pointed me on the DHCP retry-handling in dswifi), so connecting should be more stable now... not quite sure yet if it's already as stable as in the HLL version though.

nocash
Posts: 8
Joined: Mon Sep 26, 2016 10:21 pm

Re: dslink bugfixes...

Post by nocash » Tue Nov 22, 2016 3:54 pm

Forget about the "use sa_udp_remote instead of INADDR_ANY" idea, that was nonsense. And hence, doing everything in a single loop looks about right. Except, you shouldn't create (multiple) respSock's INSIDE of the loop. Using a single socket for UDP recvfrom AND sendto should work fine, so you don't need to create the respSock at all, ie. you can just use sock_udp instead of respSock.

WinterMute
Site Admin
Posts: 1845
Joined: Tue Aug 09, 2005 3:21 am
Location: UK
Contact:

Re: dslink bugfixes...

Post by WinterMute » Tue Nov 22, 2016 6:02 pm

nocash wrote:Just checked the recent (inofficial?) dslink binary, and its recent source code changes...
Still got a few things to tweak & check before I roll another binary release. Don't really want to link the current test binaries publically. I've emailed you a link to the latest test binaries.
The binary boots some of my files (eg. magicflr), but seems to always hang with white screens upon other files.
Can you send me some files that don't boot?
The arm9/arm7 stack pointers are still wrong. Oh, but ignore the values from the dslink email bug report (I've mistyped some 380Fxxxh values as 3803xxxh). Correct values should be:
4K seems rather excessive for the irq stack tbh. ARM best practices say only the dispatcher should operate in irq mode with the various handlers operating in system mode. It doesn't really make any difference to libnds code or wmb binaries though so I'm not averse to tweaking these. You should really consider more setup in your code rather than depending on the environment you're loaded into being set up in a particular way.
Relocating ARM7 binary from 2000000h to actual ARM7 dest looks wrong if source and dest do overlap. Or wait, one of your recent changes seems to have specifically fixed an issue for overlapping memcopy... looks as if it's just fixing that problem?
Yes, the overlapping copy was fixed recently.
For ARM9, just clearing the PROTECT_ENABLE bit isn't enough, that does still leave some other bits at wrong state (eg. ITCM remains enabled, which isn't the correct boot state). Better use this code:
ldr r0,=00012078h
mov p15,0,c1,c0,0,r0
Of course, that disables ITCM, so you can't execute your relocated code in ITCM. But there's no reason to relocate the final bootcode to ITCM anyways, you could as well execute in Main RAM. Original NDS firmware would relocate the final bootcode to 23FEE00h..23FEFFFh, but as you aren't zerofilling the memory occupied by dslink, you could as well execute the bootcode without any relocation.

Setting PROTECT_ENABLE before jumping to ARM9 entrypoint is also wrong. The protection unit should be kept switched off.
The memory occupied by dslink is being zero filled. The 48K not being overwritten at the end of eram is the reboot code which is placed there either by hbmenu or the code which loads dslink from wifi flash.

Why are we bothering to set the MPU regions at all if the protection unit is switched off? It seems to me that code which enables the protection unit without setting the regions it needs is basically just wrong. Even with this "default" setup more regions are needed for itcm and bios before enabling MPU. There are also issues attempting to start binaries where both arm9 and arm7 are running in ewram with cache disabled.

And, value 02FFF017h for PU region 7 is wrong (should be 027FF017h).
My concern here is that using the 0x2f address will work in both DSi and DS modes where setting it to 0x27 prevents use of the whole 16 meg ewram. Even Nintendo have fixed their code to use 0x2f range. Again, why does this actually matter if the protection unit is switched off? I was considering setting region 1 for 16meg range as well tbh.
Some possibly important other settings might be POWCNT1=820Fh and EXMEMCNT=E880h (not checked if you already have them as so). And less important, you could also set DIVDENOM=1 and such things.
I've also got my own dswifi (and dslink) ASM port finished. The good news is that it does successfully start the uploaded file after EVERY upload. On the other hand, my uploader does occassionally hang after DHCP_Start or after the final association Grat message. I am not sure if it's a problem in my ASM port, or if it does also occur in Stephen's original dswifi library. Did you encounter that kind of problem, too?
The only problem I'm really seeing with this now is that it doesn't seem to respond to network traffic after a while if I leave it to sit. It's fine if I just reboot dslink though - pressing start does that when it's installed in wifi flash chip. If I automate uploading dslink then it'll quite happily cycle all day without issue.

Also, to be fair here, dslink as far as I can tell is successfully starting the uploaded file after every upload too. Whether or not the uploaded code then crashes has more to do with the uploaded code than the launcher - code using libnds or Nintendo's SDK mostly resets the hardware to a known state by itself on startup.
My overall impression with previous dslink version has been that it crashed about each second time AFTER upload, but rarely crashed BEFORE upload when connecting to the access point
Is the latest one working for you now? I'm not seeing any failures at all with the latest changes. All of Nintendo's wmb demos work fine when transferred in DS mode although a lot of them break in DSi mode.

In the progressRead, I would clip the chunksize to min 1024 bytes (or to the bootblock size if it's smaller than 1024). Currently, when receiving a small 100 byte bootblock, your code would transfer it in tiny 1-byte chunks which is kinda inefficient (although the actual wifi hardware should transfer it all in one packet).
And the worst case, if the bootblock is SMALLER than 100, then your "chunksize=size/100" would probably just cause the function to hang, trying to receive 0-byte blocks.
I'm not sure the inefficiency really matters that much tbh - it's mainly there as a visual indication of the transfer which is rather inefficient already. Logically & in reality it's slower but the transfer is subjectively faster for end users. The actual wifi transfer isn't limited by this so it's really just pulling data from a buffer. It shouldn't fail with < 100 byte blocks so I'll definitely fix that.
And the while loop in the main function, I would split it into two loops, one for the UDP part, and after finishing the UDP stuff, a second loop for the TCP part. Then you could also use the PC's actual IP address from "sa_udp_remote" for the TCP part. That would look better than INADDR_ANY to me.
This is done because not all networks will allow broadcast UDP so the user must specify the DS address from the PC. Obviously we can't know the address of the PC in advance without the broadcast packets and splitting into two loops wouldn't allow the user to make a direct connection.
Help keep devkitPro toolchains free, Donate today

Personal Blog

nocash
Posts: 8
Joined: Mon Sep 26, 2016 10:21 pm

Re: dslink bugfixes...

Post by nocash » Tue Nov 22, 2016 7:26 pm

The stacks and mpu settings are important because dslink is booting NDS software, so they should be booted with same state as when booted via regular NDS firmware. It's just about compatibility with the official firmware, so it really doesn't matter if 4Kbyte IRQ stack is suitable for any purposes - it's just the default boot setting.

Splitting into two loops would work, I've actually did that, but I've dropped that idea meanwhile (since the single loop is better for supporting re-transmits of lost UDP replies).

For bind'ing with INADDR_ANY, I've just misunderstood what bind is doing. It's allowing to specify the destination address, not the source address. Ie. one could bind either of these three values:
* INADDR_BROADCAST - to receive broadcast packets
* IP Address of the NDS console - to receive packets specifcally sent to the console
* INADDR_ANY - to receive either of the above
For setting the remote IP of the PC... it seems that one should use "connect" for that, but dswifi doesn't seem to have that implemented for UDP. Anyways, that would be important only if you would receive more than UDP packet (eg. if you were doing the whole upload via UDP instead TCP), and wanted to ensure that all UDP packets are from the same source.

Oops, yes, the recent dslink version is zerofilling memory. Anyways, better reloc your code to 23FEE00h (official bootstub area) instead of to (disabled) ITCM. And zerofill ITCM before disabling it.

Yup, having both ARM7 and ARM9 entrypoints in MainRAM would require the game to support that (eg. run a SWI wait by loop on ARM7 until the ARM9 bootcode has enabled caches). If a game doesn't support that then it won't work on "real" hardware... and I would prefer such bugged games not to work with dslink either - just to force people to create proper code that works with real firmware.

As long as the PU is off most of it's settings really don't matter too much, unless somebody writes some 1Kbyte-compo game that relies on certain PU regions to have correct initial settings. Or unless somebody writes a tool for testing the initial settings - which can't reinitialize the initial settings before testing them (that's actually what I am doing, and it's kinda impossible to run that kind of code with most or all dslink versions).

Yes, things like 02FFF017h vs 027FF017h can differ on DSi. The DSi is a beast of its own:
If the game is a DSi title (and runs on a DSi console), then it should have several things initialized differently (and especially you would need to upload the ARM9i and ARM7i areas, additionally to the ARM9 and ARM7 areas).
If the game is a NDS title, then it should have everything initialized as when running on a real NDS console, the problem there is that Nintendo screwed up that part, too. The DSi firmware is booting NDS titles with wrong stack pointers (and wrong DTCM address). And the DSi firmware is more restrictive about the ARM7 destination address (the ARM7 entrypoint MUST be in the ARM7 area, so the ARM7 area MUST exist, and as far as I remember the ARM7 area isn't allowed to be located at 2000000h-2xxxxxxh, although the original NDS firmware did support that).

So, if you want to be compatible with official firmware, you could go two ways for booting NDS titles on DSi: Maintain compatibility with original NDS firmware. Or reproduce the NDS-incompatibility issues of the original DSi firmware ; (

For best NDS compatibility on DSi you would need to disable the DSi hardware on ARM7 side (which unfortunately isn't possible with current exploits, particulary some of the Shared RAM will be forcefully overlapped by NewShared RAM). However, you can do some things: Switch to 4MByte Main RAM mode (if you aren't already doing so), and switch the touchscreen/sound controller to NDS-compatiblity mode (I've revengineered some code/flowchart for doing that, if you are interested - it's still a little bit messy, but should work).

nocash
Posts: 8
Joined: Mon Sep 26, 2016 10:21 pm

Re: dslink bugfixes...

Post by nocash » Wed Nov 23, 2016 10:17 pm

I've got the DHCP'ing working reliable in my dswife ASM port. It's been caused by a stuck W_IF flag (the counter half-overflow flags cannot be acknowledged while the counter MSB is set, so one must FIRST read the counters (to reset them), and THEN acknowlege the irq flag). So almost everything works fine now. Except one thing...
The only problem I'm really seeing with this now is that it doesn't seem to respond to network traffic after a while if I leave it to sit.
Yeah, I seem to have some similar problem in my ASM code. The upload doesn't start when doing this:
- start the receiver code on NDS side
- wait about 2 minutes
- start the transmitter code on PC side
The "wait 2 minutes" is causing problems, without that wait everything works fine.

As far as I remember I did have had some similar issue with the old dslink version (and as far as I understood, that problem showed up for you, too). Hmmm, I've just checked... dslink doesn't seem to die after 2 minutes... maybe it dies at some point on... but even if so, I don't know if it's the same problem that I am having in my ASM code.

Anyways, if it's the same problem, do you have some idea what is getting wrong there? I've spent the whole day on (trying to) track it down, but I am still quite clueless. The access point is still saying that it's connected to the NDS even after two minutes. The NDS's power-saving features (like W_BEACONCOUNT2) don't seem to be causing the problem either. And the Heap isn't full. But something is apparently dying : /

nocash
Posts: 8
Joined: Mon Sep 26, 2016 10:21 pm

Re: dslink bugfixes...

Post by nocash » Thu Nov 24, 2016 4:24 pm

Oh, cool you have changed the stack points : ) when you said that you don't want to "tweak" stacks, it sounded to me as if you were saying "leave me alone with your stack pointers" : ( but I might have just misread the sentence, and you meant to fix the issue instead of leaving it in "tweaky" state. Well, anyways, looks better now. Only the sp_sys=3002F7Ch seems to be still slightly off?

And PU is now switched off, too. But why does the comment still say "@ Switch MPU back on", seriously??? And "00012078" is fine, and it does disable ITCM, but your code is still running in ITCM at that point. Theoretically that should crash your program... unless, it might work if the CPU has already prefetched the next opcode... which, it probably does so... but it's kinda frightening to see code that does rely on such stuff.

WinterMute
Site Admin
Posts: 1845
Joined: Tue Aug 09, 2005 3:21 am
Location: UK
Contact:

Re: dslink bugfixes...

Post by WinterMute » Fri Nov 25, 2016 3:11 am

Hehe, I just like to moan about stuff while I'm doing it tbh. I usually commit working code while I'm tidying & rearranging so comments may not reflect what's actually happening while work is in progress.

There's a 5 stage pipeline on the 946e-s so it's perfectly legitimate code - that last branch is already in the pipeline ready to be executed by the time the itcm gets switched off.

I looped this loading itself overnight & it was still uploading quite happily 13 hours later so I think we're reasonably reliable now. I'll probably roll a proper release of this version for now. It's in the same location as my last PM if you want to try it in the meantime.

Code compiled with libnds now is automatically hybrid & can access the extra ram & even nand/sd card when booted via dsiwarehax. We didn't bother setting flags or dealing with extra binaries and I never really considered dealing with that with dslink. I guess I can give it a go and see if it's even possible to boot Nintendo DSi binaries by loading the extra parts in DSi mode - download play on the DSi operates in DS mode though so I don't really expect to find anything with extra parts that will boot this way. Unless of course you know different?
Help keep devkitPro toolchains free, Donate today

Personal Blog

nocash
Posts: 8
Joined: Mon Sep 26, 2016 10:21 pm

Re: dslink bugfixes...

Post by nocash » Fri Nov 25, 2016 5:38 pm

At the moment the extra ARM7i and ARM9i parts are probably only rhetorically needed (for the day when somebody develops a hombrew game that needs those parts, considering the activity in the DSi scene that probably happen in next 20 years, or well, maybe somebody will actually want to use that areas earlier).

For the hybrid mode, ideally that should rely on cart header[12h].bit1. If the bit is cleared then switch everything to NDS mode as possible (eg. disable 16MByte RAM, so that 23FFxxxh will mirror to 27FFxxxh and 2FFFxxxh, as how NDS titles would expect it, and switch TSC to NDS mode to get working touchscreen/sound/microphone). Of course, that will disable DSi support in the current wanna-be-DSi-compatible homebrews (assuming that most of them still aren't identifyling themselves as DSi-compatible via header[12h].bit1). I hope you are setting that flag in libnds builds that do include DSi hardware support! (?)

Good to know that your code survives 13 hours idle time! Then any such problems on my side should be related to bugs in my ASM code, not to the original dswifi library. I've found two ASM bugs today, that seems to have fixed my die-after-2-minutes problem. But I am still having a similar problem, where it's randomly dying once and then, even when starting the upload instantly after connecting to the access point.

No, even if it's working reliable, relying on the CPU pipeline prefetch isn't exactly perfectly legitimate code.

WinterMute
Site Admin
Posts: 1845
Joined: Tue Aug 09, 2005 3:21 am
Location: UK
Contact:

Re: dslink bugfixes...

Post by WinterMute » Sat Nov 26, 2016 10:56 pm

I think we might be talking at cross purposes here to be honest.

From an emulation perspective it may be better for you to be able to identify an application as DSi compatible through flags and the existence of extra binaries so you can switch no$gba into DSi mode. Personally I'm much more interested in having DS code run regardless of whether it's booted in DSi mode or DS mode so actually I'd rather not have something automatically switch.

DSi homebrew basically fell by the wayside when everyone moved to 3DS so, currently, there's no support in the tools for setting the DSi mode flags in the header or adding extra binaries to the nds file ... it's never really mattered for anything. I'll do it at some point but there are other demands on my time right now unfortunately & it's not something straightforward. There are a lot of decisions to be made about if and how the tools should support extra binaries. Certainly I don't think the homebrew tools should just slavishly follow how Nintendo do things.

The 13 hours I referred to was a test of uploading dslink repeatedly, not idle time. I haven't been able to replicate it not responding to network traffic either recently.
Help keep devkitPro toolchains free, Donate today

Personal Blog

Post Reply

Who is online

Users browsing this forum: No registered users and 18 guests