make: custom up-to-date

[Boris Kolpackov]

Good day,

GNU make has only one way of determining up-to-date-ness: modification
timestamp. This is an efficient and in most cases sufficient mechanism.
But every now and then a situation arises where you want a custom
up-to-date check.

If such a need is local and you have full control over the makefile
fragment then everything can be arranged using so-called "sentinel"
file technique.

The technique is based on the way make decides when to update targets
and whether the targets were actually updated. Consider a small example:

  foo: foo.o
	$(CC) -o $@ $^

  foo.o: foo.c
	$(CC) -o $@ -c $<

Let's assume both `foo.o' and `foo.c' exist and `foo.c' is older than
`foo.o'. After make compares timestamps of `foo.c' and `foo.o' it
realizes that `foo.o' should be updated and runs the corresponding
command. After the command is successfully completed, make compares
modification times of `foo.o' before and after the command execution.
If they are the same, make concludes that `foo.o' was up-to-date and
there is no need to rebuild anything that depends on it (`foo' in our
case). The "sentinel" file technique is based on this before-after
comparison.

Suppose `foo.c' gets updated frequently but the content of the file often
does not change. It would be nice to use custom up-to-date-ness check on
this file which will trigger updates only if content of the file has
changed. One way to achieve this is to use `foo.c.md5' "sentinel" file:

  foo: foo.o
	$(CC) -o $@ $^

  foo.o: foo.c.md5
	$(CC) -o $@ -c $(<:.md5=)

  foo.c.md5: foo.c
	@md5sum $< | cmp -s $@ -; if test $$? -ne 0; then md5sum $< > $@; fi


In this example `foo.c.md5' serves two purposes: it keeps the previous
md5 hash sum and triggers an update of `foo.o' when the previous and
current hash sums do not match.

While this example may look neat, it has one problem: it is not
user-makefile-transparent. To understand what I mean let's transform
our original example above to be a little closer to real life:

  # Things we cannot change.
  #
  foo: foo.o
  foo.o: foo.c foo.h


  # Things we can change.
  #
  %: %.o
	$(CC) -o $@ $^

  %.o: %.c
	$(CC) -o $@ -c $<


To implement md5-based up-to-date check we need to change this line

  foo.o: foo.c foo.h

to be

  foo.o: foo.c.md5 foo.h.md5

This is not very practical since it will most likely lead to changes in
a lot of user makefiles (even if it is acceptable, sure it would be nice
not to have to). It is also bad because we embed a particular method into
user makefiles which makes the whole construction very inflexible.
The ideal solution would require changes only to the part marked "Things
we can change" which is usually located in a common file included by
every user makefile.

A long story short you can't do it in current GNU make. But we are just
a few logical features away from being able to. Right now pattern rules
allow you to add prerequisites to the original rule. What's missing is
the ability to remove and query prerequisites from the original rule. If
we had this ability we would be able to automatically mangle `foo.c foo.h'
to become `foo.c.md5 foo.h.md5'. In fact I went ahead and implemented those
two features in `bk' patch-set ( http://kolpackov.net/projects/make/bk/ ).
Here's how it works:

  # Things we cannot change.
  #
  foo: foo.o
  foo.o: foo.c foo.h


  # Things we can change.
  #
  %: %.o
	$(CC) -o $@ $^


  %.o: %.c.md5 $$(addsuffix\ .md5,$$^) $$-
	$(CC) -o $@ -c $(<:.md5=)

  .PRECIOUS: %.md5

  %.md5: %
	@md5sum $< | cmp -s $@ -; if test $$? -ne 0; then md5sum $< > $@; fi


The second pattern rule is where all the action happens. The first
prerequisite pattern (`%.c.md5') ensures that we have a `.c' file for
every `.o' file. The second prerequisite (`$$(addsuffix\ .md5,$$^)') is
where changing `foo.c foo.h' to `foo.c.md5 foo.h.md5' is done. `$$^'
expands to the list of prerequisites from the original rule (just like
in command script). Note, that we need to escape it so that it will
survive the first round of expansions (happens when makefile is read).
And the last prerequisite (`$$-') is a special variable that expands to
nothing but also removes all the prerequisites that came from the original
rule (`foo.c foo.h' in our case).

The complete example shown above is available in the `bk5' patch-set.

If you have made it this far, thank you for your time. Permission is
granted to copy, distribute and/or modify this document under the terms
of the GNU Free Documentation License, Version 1.2; with no Invariant
Sections, no Front-Cover Texts, and no Back-Cover Texts.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 652 bytes
Desc: Digital signature
Url : http://www.kolpackov.net/pipermail/notes/attachments/20040901/939d6ee2/attachment.bin