Compilation issues in migrating to libnds 2.0

[ssstolk]

Hello devkitPro,

I was hoping to get some help with some issues that I have encountered in migrating from a late 1.x libnds to libnds 2.0. Context: My intention is to release a full RTS for the NDS/DSi along with its source code, which hopefully will be able to boast the use of libnds 2.0.

Having used the devkitPro Updater 3.0.3 to update to the latest devkitPro, the compilation of my source code was, well, very unsuccessful.
Errors along the line of "calico could not be found" and "This header file is only for NDS ARM9 or ARM7" were present, immediately stopping the compilation process. After having removed all that was installed (and updated) in the devkitPro folder, and running the devkitPro Updater again, the fresh install at least prevented those errors and the migration could begin. Two further problems exist that complicate the compilation process. My Makefile is mostly still the example template for NDS development.

1. Compilation of an assembly file for some reason ignores the "-I" arguments, meaning that the include for asminc.h fails. The .c files in that same directory do get passed the appropriate -I arguments, though, so this seems specific to .s files.
   
   Running the command below, which is successful, at least allowed me to continue with the compilation process.

   Code: Select all

   c:\devkitPro\devkitARM\bin\arm-none-eabi-gcc -MMD -MP -MF build/fastCopyOAM.d -x assembler-with-cpp -D__NDS__ -I"c:\devkitpro\libnds\include" -g -march=armv5te -mtune=arm946e-s -mthumb -c source/fastCopyOAM.s -o fastCopyOAM.o

2. At the rom creation stage, I get the error message below, indicating that a default.elf is missing. Indeed no such file is present in that directory. Am I right in assuming it's attempting to incorporate the default ARM7 binary? Where should that be located?
   

If it helps, I am happy to upload the entire current migrated source code to a private github repo and provide you (fics? wintermute?) with access for testing.

All best wishes,
Sander

[fincs]

In order to ease transition to libnds v2, we changed some of the common Makefile infrastructure to automatically supply some of the new required flags. However, it's possible you may have customized your Makefile a bit "too much", and the logic we added does not work. Could you post your Makefile here so that we can take a look? And also, please make sure all your packages are up to date (pacman -Syu).

[ssstolk]

Of course. Updated all packages just now; errors remain. The Makefile is below. Thank you for your time and effort.
EDIT: After I compile the .s files manually, adding the right -I arguments, and run the Makefile again... I now get a new error. Baffling.
"*** No rule to make target 'source/fastCopyOAM.s', needed by 'fastCopyOAM.o'. Stop."

Code: Select all

#---------------------------------------------------------------------------------
.SUFFIXES:
#---------------------------------------------------------------------------------
ifndef GAME_TITLE
GAME_TITLE		:=	RTS4DS
GAME_SUBTITLE1	:=	by 
GAME_SUBTITLE2	:=	Sander Stolk
GAME_ICON		:=	$(CURDIR)/../logo.bmp
endif

ifeq ($(strip $(DEVKITARM)),)
$(error "Please set DEVKITARM in your environment. export DEVKITARM=<path to>devkitARM")
endif

include $(DEVKITARM)/ds_rules

#---------------------------------------------------------------------------------
# TARGET is the name of the output
# BUILD is the directory where object files & intermediate files will be placed
# SOURCES is a list of directories containing source code
# INCLUDES is a list of directories containing extra header files
#---------------------------------------------------------------------------------
TARGET		:=	$(shell basename $(CURDIR))
BUILD		:=	build
SOURCES		:=	gfx source data source/astar source/gif
INCLUDES	:=	include build
NITRODATA	:=	fs

#---------------------------------------------------------------------------------
# options for code generation
#---------------------------------------------------------------------------------
ARCH	:=	-march=armv5te -mtune=arm946e-s -mthumb

CFLAGS	:=	-g -Wall -O2 -ffunction-sections -fdata-sections\
			$(ARCH)
# note: add -fstack-protector-all to CFLAGS to enable stack overflow detection

CFLAGS	+=	$(INCLUDE) -DARM9 $(TARGET_CFLAGS)
CXXFLAGS	:= $(CFLAGS) -fno-rtti -fno-exceptions

ASFLAGS	:=	-g $(ARCH)
LDFLAGS	=	-specs=ds_arm9.specs -g $(ARCH) -Wl,-Map,$(notdir $*.map)

#---------------------------------------------------------------------------------
# any extra libraries we wish to link with the project
#---------------------------------------------------------------------------------
LIBS	:= -lfat -lfilesystem -lmm9 -lnds9
 
#---------------------------------------------------------------------------------
# list of directories containing libraries, this must be the top level containing
# include and lib
#---------------------------------------------------------------------------------
LIBDIRS	:=	$(LIBNDS)
 
#---------------------------------------------------------------------------------
# no real need to edit anything past this point unless you need to add additional
# rules for different file extensions
#---------------------------------------------------------------------------------
ifneq ($(BUILD),$(notdir $(CURDIR)))
#---------------------------------------------------------------------------------
 
export OUTPUT	:=	$(CURDIR)/$(TARGET)
 
export VPATH	:=	$(foreach dir,$(SOURCES),$(CURDIR)/$(dir))
export DEPSDIR	:=	$(CURDIR)/$(BUILD)

ifneq ($(strip $(NITRODATA)),)
	ifdef NITRODATA_SUBFOLDER
		export NITRO_FILES	:=	$(CURDIR)/$(NITRODATA)/$(NITRODATA_SUBFOLDER)
	else
		export NITRO_FILES	:=	$(CURDIR)/$(NITRODATA)/rts4ds
	endif
endif

CFILES		:=	$(foreach dir,$(SOURCES),$(notdir $(wildcard $(dir)/*.c)))
CPPFILES	:=	$(foreach dir,$(SOURCES),$(notdir $(wildcard $(dir)/*.cpp)))
SFILES		:=	$(foreach dir,$(SOURCES),$(notdir $(wildcard $(dir)/*.s)))
BINFILES	:=	$(foreach dir,$(SOURCES),$(notdir $(wildcard $(dir)/*.bin)))
 
#---------------------------------------------------------------------------------
# use CXX for linking C++ projects, CC for standard C
#---------------------------------------------------------------------------------
ifeq ($(strip $(CPPFILES)),)
#---------------------------------------------------------------------------------
	export LD	:=	$(CC)
#---------------------------------------------------------------------------------
else
#---------------------------------------------------------------------------------
	export LD	:=	$(CXX)
#---------------------------------------------------------------------------------
endif
#---------------------------------------------------------------------------------

export OFILES	:=	$(BINFILES:.bin=.o) \
					$(CPPFILES:.cpp=.o) $(CFILES:.c=.o) $(SFILES:.s=.o)
 
export INCLUDE	:=	$(foreach dir,$(INCLUDES),-I$(CURDIR)/$(dir)) \
					$(foreach dir,$(LIBDIRS),-I$(dir)/include) \
					-I$(CURDIR)/$(BUILD)
 
export LIBPATHS	:=	$(foreach dir,$(LIBDIRS),-L$(dir)/lib)
 
.PHONY: $(BUILD) uw_demo1 uw_demo2 uw_demoX debug clean
 
#---------------------------------------------------------------------------------
$(BUILD):
	@[ -d $@ ] || mkdir -p $@
	@$(MAKE) --no-print-directory -C $(BUILD) -f $(CURDIR)/Makefile
 
#---------------------------------------------------------------------------------
uw_demo1:
	make uw_demoX DEMO_NUMBER=1

uw_demo2:
	make uw_demoX DEMO_NUMBER=2

uw_demoX:
	make TARGET_CFLAGS='-DDEFAULT_LOCATION_NDS=\"fat:/uw_demo$(DEMO_NUMBER).nds\" -DFS_ROOT_FAT=\"fat:/uw_demo$(DEMO_NUMBER)/\"' \
			GAME_TITLE="Ulterior Warzone (demo $(DEMO_NUMBER))" \
			GAME_SUBTITLE1="by Sander Stolk and" \
			GAME_SUBTITLE2="Violation Entertainment" \
			GAME_ICON=$(CURDIR)/logo_uw.bmp \
			NITRODATA_SUBFOLDER=uw_demo$(DEMO_NUMBER)
	mv $(TARGET).elf uw_demo$(DEMO_NUMBER).elf
	mv $(TARGET).nds uw_demo$(DEMO_NUMBER).nds
	mv $(TARGET).nds.ezflash5 uw_demo$(DEMO_NUMBER).nds.ezflash5
	@rm -fr $(TARGET).elf $(TARGET).nds $(TARGET).nds.ezflash5 $(TARGET).ds.gba

debug:
	make TARGET_CFLAGS=-DDEBUG_BUILD

clean:
	@echo clean ...
	@rm -fr $(BUILD) *.elf *.nds *.ezflash5 *.ds.gba 
 
 
#---------------------------------------------------------------------------------
else
 
DEPENDS	:=	$(OFILES:.o=.d)

#---------------------------------------------------------------------------------
# overriding default %.nds rule in order to also generate a homebrew binary compatible with ezflash5
#---------------------------------------------------------------------------------
%.nds	: %.elf
	@ndstool -c $@ -9 $< -b $(GAME_ICON) "$(GAME_TITLE);$(GAME_SUBTITLE1);$(GAME_SUBTITLE2)" $(_ADDFILES)
	@echo built ... $(notdir $@)
	@ndstool -c [email protected] -9 $< -b $(GAME_ICON) "$(GAME_TITLE);$(GAME_SUBTITLE1);$(GAME_SUBTITLE2)" -g "PASS" "00" "RTS4DS" $(_ADDFILES)
	@echo built ... $(notdir $@).ezflash5
	@dlditool $(CURDIR)/../dldi/EZ5V2.dldi [email protected]

#---------------------------------------------------------------------------------
# main targets
#---------------------------------------------------------------------------------
$(OUTPUT).nds	: 	$(OUTPUT).elf
$(OUTPUT).elf	:	$(OFILES)

#---------------------------------------------------------------------------------
%.o	:	%.bin
#---------------------------------------------------------------------------------
	@echo $(notdir $<)
	$(bin2o)
 
 
-include $(DEPENDS)
 
#---------------------------------------------------------------------------------------
endif
#---------------------------------------------------------------------------------------

[WinterMute]

You really shouldn't need to compile .s files manually, it's not clear from your Makefile what might be causing that. There's a possibility that something has gone awry with the update process, it might help if you were to run the update from the msys2 shell rather than using the updater which unfortunately has a habit of hiding problems since we moved to using pacman. Open an msys2 bash shell (Start->devkitPro->msys2) then run the following command there

Code: Select all

pacman -Syu

Report back with any errors you get there (or if it just succeeds)

Access to your private github may be helpful - https://github.com/fincs & https://github.com/WinterMute

Ideally you also shouldn't be adding build steps to your Makefile for specific flashcards or use absolute paths in your builds like this. This can and will cause problems for other cards and DSi compatibility. It inevitably leads to "special" builds for individual cards when really the better option for everyone is having a menu/launcher which properly supports homebrew binaries. We generally recommend our own nds-hb-menu for this and avoid pointing people to launchers intended for piracy.

[ssstolk]

Report back with any errors you get there (or if it just succeeds)

No errors from pacman -Syu via msys. Just updated two packages, it seems. Ran it again afterwards to confirm no update was needed anymore.

Code: Select all

:: Synchronizing package databases...
 msys is up to date
 dkp-libs is up to date
 dkp-windows is up to date
:: Starting core system upgrade...
 there is nothing to do
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (2) diffutils-3.11-1  libpcre2_8-10.45-1

Total Download Size:   0.51 MiB
Total Installed Size:  2.06 MiB
Net Upgrade Size:      0.11 MiB

:: Proceed with installation? [Y/n] Y
:: Retrieving packages...
 diffutils-3.11-1-x86_64         384.2 KiB   881 KiB/s 00:00 [###############################] 100%
 libpcre2_8-10.45-1-x86_64       141.9 KiB   276 KiB/s 00:01 [###############################] 100%
 Total (2/2)                     526.1 KiB   821 KiB/s 00:01 [###############################] 100%
(2/2) checking keys in keyring                               [###############################] 100%
(2/2) checking package integrity                             [###############################] 100%
(2/2) loading package files                                  [###############################] 100%
(2/2) checking for file conflicts                            [###############################] 100%
(2/2) checking available disk space                          [###############################] 100%
:: Processing package changes...
(1/2) upgrading diffutils                                    [###############################] 100%
(2/2) upgrading libpcre2_8                                   [###############################] 100%
:: Running post-transaction hooks...
(1/1) Updating the info directory file...

sssto@ST-SURFACE-PRO7 MSYS ~
$ pacman -Syu
:: Synchronizing package databases...
 msys is up to date
 dkp-libs is up to date
 dkp-windows is up to date
:: Starting core system upgrade...
 there is nothing to do
:: Starting full system upgrade...
 there is nothing to do

Problem with compiling .s persists, however, after this updating of devkitPro toolchain.

Access to your private github may be helpful

I've invited you as collaborators.

Ideally you also shouldn't be adding build steps to your Makefile for specific flashcards or use absolute paths in your builds like this.

Understood and I agree. Will remove it before I release the code. I have it in place only to remove one step in my testing cycle (either DLDI patching first or, on the DS, first starting up the homebrew launcher).

[ssstolk]

Some progress, but could still use your help.

Compilation now succeeds.

• Updated Makefile in Github with custom .s rules (so that -I is set properly; ugly but works) and removed custom .nds rule (so that default ARM7 .elf is located successfully).
• Added game assets. Run "make uw_demo1" to compile the game and add the game assets through NitroFS.

I now stumble upon an error, however, in no$gba: NitroFS fails to initialize.
It works properly in my code that works with libnds1.x; not with libnds2.0 though.
Can you tell me what I'm doing wrong? The code sets argv to a default location (uw_demo1.nds), since no argv is apparently passed by no$gba?

source/fileio.c code based on libnds1.x

Code: Select all

    // initialize NitroFS, in combination with FAT
    if (!( __system_argv->argvMagic == ARGV_MAGIC && __system_argv->argc >= 1 )) {
        // attempt default .nds location by setting it in argv
        __system_argv->argvMagic = ARGV_MAGIC;
        __system_argv->argc = 1;
        __system_argv->argv[0] = malloc(strlen(DEFAULT_LOCATION_NDS)+1);
        strcpy(__system_argv->argv[0], DEFAULT_LOCATION_NDS);
    }
    if (!nitroFSInit(NULL))
        error("Initializing file access failed.",
              "Please start the .nds file  by  "
              "using a launcher such as        "
              "github.com/devkitPro/nds-hb-menu\n");

source/fileio.c code based on libnds2.0

Code: Select all

    // initialize NitroFS, in combination with FAT
    if (!( g_envNdsArgvHeader->magic == ENV_NDS_ARGV_MAGIC && g_envNdsArgvHeader->argc >= 1 )) {
        // attempt default .nds location by setting it in argv
        g_envNdsArgvHeader->magic = ENV_NDS_ARGV_MAGIC;
        g_envNdsArgvHeader->argc = 1;
        g_envNdsArgvHeader->argv[0] = malloc(strlen(DEFAULT_LOCATION_NDS)+1);
        strcpy(g_envNdsArgvHeader->argv[0], DEFAULT_LOCATION_NDS);
    }
    if (!nitroFSInit(NULL))
        error("Initializing file access failed.",
              "Please start the .nds file  by  "
              "using a launcher such as        "
              "github.com/devkitPro/nds-hb-menu\n");

[ssstolk]

Well, it appears that libnds2.0 is more strict when it comes to passing the location of the .nds file for initNitroFS.
Removing the attempt of setting a default .nds location, in case no argv was passed, and removing the Guitar Hero grip REG_EXMEMCNT lines, makes it all run in no$gba.

Still, although the game runs, it does crash pretty soon once truly ingame (level 1).
Could this be due to how I track gameticks (gameticks.c), which I used to do using timers, and now tried to migrate to libnds2.0 in a somewhat awkward manner by calling tickInit and tickGetCount --- looks similar in my own setup with the timers but is actually meant for threads? I track gameticks mainly to see how much time there is left in a frame in order to run A*-pathfinding for combat units. I am not not sure whether using threads would be a viable alternative instead and, if so, how to best approach that.

Of course, it could also be another issue. In libnds1.x, there is quite a serious issue with music + soundeffects. Sporadic crashes (e.g., after half an hour or an hour or...). Disabling music prevents is. Disabling playing samples prevents it. And, I noticed only yesterday, not using killSound() prevents it from happening, which I used every time I wanted another sample to take precedence over one that was already playing. For libnds1.x I now never use killSound, circumventing its use by having certain channels reserved for sounds with a higher priority so that (the likelihood of me needing to) stopping already playing samples is almost nil.

[ssstolk]

Issue does indeed seem to be in (my use of) gameticks, based on further tests. Suggestions for dealing with gameticks in a better way for the purpose at hand would be most welcome.

P.S. I should note that libnds2.0 does appear to make the game run quite a bit slower in no$gba than it did with my libnds1.x compatible .nds binary, and DeSmuME now only results in white screens with libnds2.0. Is that to be expected?