Directory creation within makefiles can now be handled in a consistent manner that is both thread safe and dependency driven. Directories will be created on demand and the mkdir command invoked within a shell only when needed. Bug 680246 has added utility makefile logic to support the functionality.
=== makefiles ===
MKDIR ?= mkdir -p
TOUCH ?= touch
# Dependency generator function
mkdir_deps =$(foreach dir,$(getargv),$(dir)/.mkdir.done)
# Target rule to create the generated dependency.
%/.mkdir.done: # target rule
@$(MKDIR) -p $(dir $@)
@$(TOUCH) $@
=== Problems ===
=== why the extra dep? (.mkdir.done) ===
Placing dependencies on directories is a very old makefile problem. They contribute to generated targets being artificially stale even after gmake has been run. Make will query the inode table when making timestamp comparisons. Unfortunately whenever directories/files/links are created/deleted/modified/renamed within a directory the inode table alteration will be detected by make as a stale dependency/target to regenerate. Generated targets that have the directory as a pre-requisite are re-created whenever the directory timestamp is newer than the target. This can occur immediately after a successful gmake run.
==== mkdir_deps: user function ====
A simple workaround for these problems is to use real makefile dependencies to control if/when a directory should be created. The user function ‘mkdir_deps’ can be called, passing in a list of directory paths. The function will return a list of arguments suffixed by the filename ‘.mkdir.done’.
include $(topsrcdir)/config/rules.mk
deps = $(call mkdir_deps,foo,bar,tans)
$(info deps=$(deps))
deps=foo/.mkdir.done bar/.mkdir.done tans/.mkdir/done
==== mkdir_deps: target rule ====
The second component if the library is a wildcard target rule that will match (~directory) dependencies suffixed by ‘.mkdir.done’. The rule will use the given target name to create a directory and dependency file contained within that directory.
%/.mkdir.done:
$(MKDIR) -p $(dir $@)
$(TOUCH) $@
==== dependencies ====
Using dependencies for directories have a few benefits. They convey filesystem state information to gmake so the command can decide when a directory should be created rather than spawing a shell to run a command that will determine if the directory should be created. The dep can be mentioned as often as needed (pre-requisites) to trigger directory creation. Overhead for each reference will be a file stat or far more likely checking cached filesystem information.
With dependencies in use, existence of a .mkdir.done file will short-circuit all directory creation attempts because the target is up-to-date. To resolve race conditions and critical section problems when a directory does not exist, “mkdir -p” will step in. When multiple threads attempt to create the same directory one thread will win and create it. All other threads will attempt creation, fail and ignore status because the directory already exists.
=== USAGE ===
==== Multiple makefile targets ====
By far the simplest syntax to use for directory creation is when multiple targets in a makefile require a common directory. Assign a list of directories to GENERATED_DIRS then include rules.mk. Dependencies will be generated to automatically create and remove (clean target) directory deps as needed.
dirs = foo bar tans
GENERATED_DIRS = $(dirs)
include $(topsrcdir)/config/rules.mk
libs::
find $(dirs) -ls
Pros:
- simple syntax
- automated cration and cleanup
Cons:
- Shotgun approach, deps created may be irrelevant to a target.
==== Individual makefile targets ====
GENERATED_DIRS may be a fast and easy answer but it is not always correct. Sometimes a more refined answer may be needed . Logic requiring directory creation by common logic residing within an included makefile or may be target specific can directly utilize the mkdir_deps function to create pre-requisites. An example of unwanted dependencies would be “gmake check” being able to create a directory only used by the install target.
Creating individual targets
- Declare a local make var for the directory dependency or use ${target}-preqs if a target is using several.
- $(call mkdir_deps,dir[,dirN]) to generate dependencies.
- Assign deps to targets as needed.
- Declare ‘GARBAGE_DIRS =’ if ‘gmake clean’ targets are allowed to remove the directories.
dirs = utils
GARBAGE_DIRS += $(dirs) # gmake clean
include $(topsrcdir)/config/rules.mk
# call mkdir_deps after rules.mk has defined the function.
# allocate dep once for re-use
dir-utils = $(call mkdir_deps,$(dirs))
# inline dep allocation for pre-requisites
install-preqs =\
$(call mkdir_deps,install_here) \
$(dir-utils) \
$(NULL)
install: $(install-preqs)
do_something_constructive.sh
export: $(dir-utils)
Pros
- surgical approach. Directory creation can be target specific.
Cons
- manual clean target removal required.
- care must be taken when deps are used as a pre-requisite to produce correct behavior.
==== caveats ====
- mkdir_deps should be called after the function has been defined by “include rules.mk”. Calling prior to the include will return $(NULL) rather than a dependency — handled by make as a NOP instead of a directory creation request.
- Be aware of how dependencies are being used as a pre-requisite for targets.
- Common targets [export, libs, tools] are internally flagged as phony by rules.mk and will function correctly.
- Targets explicitly marked as .PHONY: will function correctly.
- Targets that contain other sources as a pre-requisite will function properly.
- A target that has no pre-requisites and are not .PHONY will not function properly w/o help.
dir = $(call mkdir_deps,mydir)
target: $(dir)
do_something_productive.sh > $@
After the directory is created on the initial call subsequent checks will determine {erroneously} that the .mkdir.done dep exists and is older than the target so no work is required. At a minimum the target should be dependent on *.sh and any other related sources or timestamps so target processing will be conditional on change and not simply on directory existence.
==== client.mk example ====
Directory creation is easy. Conditional directory creation is also supported with use of the $(if ) makefile directive.
configure:: configure-files
ifdef MOZ_BUILD_PROJECTS
@if test ! -d $(MOZ_OBJDIR); then $(MKDIR) $(MOZ_OBJDIR); else true; fi
endif
@if test ! -d $(OBJDIR); then $(MKDIR) $(OBJDIR); else true; fi
@echo cd $(OBJDIR);
@echo $(CONFIGURE) $(CONFIGURE_ARGS)
@cd $(OBJDIR) && $(BUILD_PROJECT_ARG) $(CONFIGURE_ENV_ARGS) $(CONFIGURE) $(CONFIGURE_ARGS) \
|| ( echo "*** Fix above errors and then restart with\
\"$(MAKE) -f client.mk build\"" && exit 1 )
@touch $(OBJDIR)/Makefile
Using mkdir_deps(), the creation calls can be written in terms of make dependencies and listed as pre-requisites for a target. In place of in-lined shell commands reducing the number of shells that will need to be spawned.
configure-preqs = \
configure-files \
$(call mkdir_deps,$(OBJDIR)) \
$(if $(MOZ_BUILD_PROJECTS),$(call mkdir_deps,$(MOZ_OBJDIR))) \
$(NULL)
configure:: $(configure-preqs)
@echo cd $(OBJDIR);
@echo $(CONFIGURE) $(CONFIGURE_ARGS)
@cd $(OBJDIR) && $(BUILD_PROJECT_ARG) $(CONFIGURE_ENV_ARGS) $(CONFIGURE) $(CONFIGURE_ARGS) \
|| ( echo "*** Fix above errors and then restart with\
\"$(MAKE) -f client.mk build\"" && exit 1 )
@touch $(OBJDIR)/Makefile
=== MAKEFILE: dirdep.mk ===
* Directory dependency makefile example.
* Howto generate stale makefile targets.
# -*- makefile -*-
all: work/dep-on-dir work/dep-on-file
work/dep-on-dir: work
date > $@
work/dep-on-file:
sleep 3; touch $@
dep-3: work/dep-on-dir
touch > $@
work:
@mkdir -v -p $@
clean:
$(RM) -r work
show:
@echo
/bin/ls -dl work work/*
===== Frst attempt: clean state =====
- No targets exist, create directory 'work' and files 'dep-on-dir' and 'dep-on-file'.
- Sleep to ensure dep-on-file will have a newer timestamp than the containing directory 'work'.
% gmake -f dirdep.mk clean all
rm -f -r work
mkdir: created directory `work'
date > work/dep-on-dir
sleep 3; touch work/dep-on-file
- Directory 'work' and both files exist so dependencies are expected to all be up-to-date.
% gmake -f dirdep.mk show
/bin/ls -dl work work/*
drwxrwxr-x 2 joey joey 4096 2012-04-12 12:12 work
-rw-rw-r-- 1 joey joey 29 2012-04-12 12:12 work/dep-on-dir
-rw-rw-r-- 1 joey joey 0 2012-04-12 12:12 work/dep-on-file
===== Second attempt: timestamps are incorrect, tasks needlessly performed =====
- Re-run gmake a 2nd time to verify the assumption that all targets are up-to-date. "Nothing to be done for 'all"" should be reported.
% gmake -f dirdep.mk all
date > work/dep-on-dir
Alas no, since dep-on-file was modified 3 seconds after the containing directory 'work' was created dependencies are stale yet again and make will need to re-create dep-on-dir when called.
===== Third attempt: success =====
- Run make a 3rd time. Work is now newer than 'dep-on-file' so we are finally up-to-date.
% gmake -f dirdep.mk all
gmake: Nothing to be done for `all'.
===== The plot thickens =====
- At least for this example the dependencies are now up-to-date after running gmake adnauseum (not a good practice...). The problem worsens as dependency chains expand and intra-target chains form circular dependencies that invalidate one-another or are forced to always be stale triggering rebuilds. As more dependencies are placed on work/dep-on-dir (target: dep-3) the chain will contribute to even more stale dependencies that require additional time and effort to try and resolve.
dep-3: work/dep-on-dir
touch $@
% gmake -f dirdep.mk clean all
[snip]
date > work/dep-on-dir
touch dep-3
## dep-on-dir, dep-3 and any targets dependent on them will be considered stale.
% gmake -f dirdep.mk all
date > work/dep-on-dir
touch dep-3
- To uncover dependency problems like these, always run make twice in succession after modifying your makefiles.
% gmake clean all; time gmake all
gmake: Nothing to be done for `all'
If "Nothing to be done' is reported dependencies are functioning properly. If make reports any other progress status or spends a lot of time doing nothing the makefile should be reviewed. This is a clear sign that dependencies are not correct which in turn will forge gmake to spending extra effort needlessly regenerating targets, dependencies and doing random work.
