build process

usually when starting new project it uses either IDE build-in 'project' that tells compiler how to make it, either uses fast written makefiles to make it flexible or autoconf and automake for generating makefiles for us.

although all mentioned before methods sound familiar there are few problems with each of them:

• IDE-integrated build does not allow easy porting.
• hand written, project-specific makefiles are error-prone and usually not very nice-looking after a while.
• auto-tools are good if you want to prepare release code for user but using them during development, when many things changes in structure over time is not very nice job.
• usually no support for automatic and manual test of code is present.
• (almost?) all proposed solutions build in directory with sources. this is bad because:
  • mixing sources and output files decrease readability of both.
  • when you use different profiles (ex: debug, test and release) you need to clean object files and try to build once again whole project. this is inacceptable in bigger projects that may require tens of minutes or even hours to compile!

because of these, and few more, i've deiced to create generic, easy to port and adopt, build process which i can use in any sort of project.

NOTE: as i'm coding mostly in C++ and C, presented build process works with this type of code.

there are few basic concepts that are implemented. they will be shortly described bellow.

as mentioned in introduction part there is a strong need to separate source files from object files. in proposed solution it is done by creating special 'gen/' directory in project root directory. subdirectories of it are profiles that are build (ex: test, release, debug, etc…). then this directory is a root for project output files. each library and application will have it's own directory for output.

in most cases there is a set of predefined profiles that user may need. in this case following are available:

• debug – build debug information on everything that is possible
• release – build with optimizations and dissable asserts.
• profile – build in profiling mode. this will require gprof installed to read generated stats on execution time of particular calls.
• test – build automatic tests for chosen library/application.
• mtest – build manual tests for chosen library/application.
• doc – generates doxygen documentation.

to make file simpler there is always requirement to test each and every part of software. automatic tests allows to quickly see if all of them works fine. in proposed solution tut framework is used. this creates single binary for each library/application. when you run it it shows all tests that pass and long description of those which failed.

though automatic tests are extremely usefull in every day life, there are some situations when writting automatic test is not possible, or at least very hard and not worth all this time (ex: consider wrapper for system call that show something on the screen). not to leave such a piece of code untouched you should write manual test for this situation. this is binary that calls single, tested functionality and you can see and judge results your self.

to automatically solve dependencies between separate sources is obvious. to go one step ahead dependencies between components (libs/apps) are solved semi-automatically. all you need to do is specify what libraries are needed to build each one and building tree from this is done for you.

some libraries does not provide specific libraries that are need but instead supply some executable that generates flags for given compiler and linker. to cope with those ones there is a backend allowing to place any binary that will produce a list of libraries to link with, link parameters and compile time switches.

there is sometimes need to do something like static polymorphism on source-code level. this happens if some code does not have sense in some cases, or if we need to build different code depending on other circumstances (ex: depending on profile). this mechanism is called split backend. in general, in C and C++ projects it works by specifying single header set and two or more sources that need to be build depending on given mode.

example library tree for having split backend could look like this:

lib1/MyClass.hpp
lib1/FakeHandler/MyClass.cpp
lib1/RealHandler/MyClass.cpp

each time only one MyClass.cpp will be build and linked in.

when specifying library's API you need to show which headers are public and can be used by others and which are library privates. in build process i propose by default all are private. to make them visible to others one must put special marker into header file:

/* public header */

files marked this way will be visible by external libraries and applications. rest is visible only inside library.

example usage can be found in test_app/lib2/Lib2/ObjectToForceLink.cpp file, inside build process' release.

there is a set of makefiles to be used for building library, application, project, etc… theses are ready to use and fully automatic. there is no need for user to change anything inside them. all he have to do is to put symbolic links in proper places.

this allows easy changing build process version. all you need to do is replace old directory with makefiles set with new ones! since all symbolic links point in the same place, but files itself have changed, changes are visible straight away.

in most cases there is no need to go into any Makefile. thought from time to time it happens that some code could be added there to prevent or force doing something. since assumption is not to change any build process files, there is a back door provided. you can put files of special names in project, library or application root directory and they will be automatically included in generic makefiles allowing inserting own code.

the most important of those is config.mk placed in project's root. you can put default setting for build there, so that all you'll need to write to build default mode is:

make

toolchain to be used for building can be easily selected by TC parameter to make (i can be also put in build_config/config.mk). currently available is 'local' that used default toolchain to build everyting. new ones can be provided by creating new toolchain description files under build_process/makefiles/common/toolchains/<tc-name> directory.

currently following toolchains are supported:

1. gcc – uses local, default tools (gcc, g++, ar, etc…)
2. avr – uses tools provided with avr-gcc, and related
3. intel – uses Intel's icc toolchain (icc, xiar, etc…)
4. clang – uses clang C/C++ compiler's suit

new parameter MEM_DEBUG=1 can be passed to make causing linker to use library electric fence to be used as memory manager. it helps to debug many memory-related problems.

integrated possibility to use test data for tests. if test requires ex. reading external file(s), it can be provided with one, that is accessible from test's output directory. dependencies of test data are traced as well as the code, i.e. when you change test data file it will be automatically copied to the destination place.

ccache is now supported out of the box. to run build with ccache simply pass WITH_CCACHE:=1 option when calling 'make', or add it to build_config/config.mk (note: since v1.1.0 it is enabled by default).

distcc support is now available. to use is add WITH_DISTCC:=1 option when calling 'make', or add it to build_config/config.mk.

3 new variables have been added to build_config/config.mk, that allows to specify extra compilation flags for a given project:

1. USER_OPT_FLAGS – flags to be used when doing optimized build
2. USER_DBG_FLAGS – flags to be used when doing debug build
3. USER_PRF_FLAGS – flags to be used when doing profile build

some components can be now build only in some of the profiles now. this is especially useful when building (m)test profiles, since it allows to create test-only components that prepare test data and provide proper stubs, commonly used when testing. these components never appear in release mode and similar.

if you create small project, consisting of just a few files, in most cases it would be an overkill to use full-featured build, that requires some configuration to be done first. instead one may use 'small projects build process', which provides most main features of full-build-process, but consists of only one make file and does not require explicit configuration. it is just enough to put it in sources directory.

example of usage can be seen in test application (small_prj_build_process/test_app/), available in build process' release.

force-link allows forcing linker to include given symbols (i.e. object files) in final binary, even though they would be normally discarded as not used. this is mainly useful when using auto-registration mechanisms for design patterns like abstract factory.

usage is extremely simple – in object file that has registration code, and must be included in final binary, even though linker can remove it, one must include one (C-code) header and use macro to define unique name of this element. see the example:

#include "BuildProcess/ForceLink.hpp"
// ...
FORCE_LINK_THIS_OBJECT(MyNamespace1_MyNamespace2_MyObjectToForceLink)
// ...

build process now allows easy importing of components based on auto-tools (./configure && make && make install). now it is enough to extract such a component to directory with typical structure and link make to AutoToolsMakefile. build process will do the rest!

you can set dependencies for such components as well, which makes it possible for quick and easy inclusion of external libs which have some dependencies between them as well.

since v1.7.0 build process allows 'features', that are more flexible (i.e. scalable) extension of 'modes' concept (see: 'split backend'). features can be build or not, i.e. being turned on/off during compile time, by selecting (or not) given name.

as a simple example, to give an idea how it works, consider two implementations of greetings for your program: normal and slang. lat us assume following directory structure:

mycomponent/MyComponent/hello.hpp            // void hell(void);
mycomponent/MyComponent/Impl/normalHello.cpp // void hello(void) { cout<<"hello world!"<<endl; }
mycomponent/MyComponent/Impl/slangHello.cpp  // void hello(void) { cout<<"yo there!"<<endl; }

to enable features add extra directory 'features' in 'modes' with proper names. for example:

mycomponent/features/modes/default # MyComponent/*.[ch]pp
mycomponent/features/modes/features/normal # MyComponent/Impl/normalHello.cpp
mycomponent/features/modes/features/slang  # MyComponent/Impl/slangHello.cpp

to enable given feature just add it to build parameters, 'FEATURES' variable:

make FEATURES="normal" # builds with 'normal' feature turned on
make FEATURES="slang"  # builds with 'slag' feature turned on

you can specify multiple features separate with spaces. features does not need to be implemented in every component ('modes' does!).

note that 'features' are often very useful along with automatic registration mechanisms like abstract factory, but can be used as a more selective 'split backend' as well (see previous example).

the example of some features can be seen in action in downloaded build process file. there is test_app that is kind of playground/example to see and test.

bellow are some examples of calling build process to build specific source with given parameters.

to build all type:

make PROFILE=release TC=local

or to build single app/lib:

make PROFILE=release TC=local somelib

to build all type:

make PROFILE=debug TC=local

or to build single app/lib:

make PROFILE=debug TC=local somelib

to build all type:

make PROFILE=profile TC=local

or to build single app/lib:

make PROFILE=profile TC=local somelib

to build single app/lib in test mode enter:

make PROFILE=test TC=local somelib

note that all dependencies of 'somelib' will be build in debug profile!

to build single app/lib in test mode enter:

make PROFILE=mtest TC=local somelib

note that all dependencies of 'somelib' will be build in debug profile!

default mode is called simply 'default'. if you need to use other mode, you'll need to specify it explicitly (or via build_config/config.mk). example o building release on mode named 'fakedriver'.

make PROFILE=release TC=local MODE=fakedriver

when creating new project following rules must be respected:

1. all libraries and applications need to be placed in separate directories in project's root.
2. all makefiles should be links to corresponding build process makefiles. in particular main project makefile should be link to ProjectMakefile inside build process.
3. build_config/components.lst file must be created in project root, with spaces separated list of all libraries and applications that are present in projects (use directories names).
4. for each library/application must be build following way:
   1. makefile must be link to proper build process makefile.
   2. sources should be placed in single directory.
   3. 'features' directory should be present with following content:
      1. 'deps' dir with files that have names like profile names. each file specifies dependencies on other libs/apps depending on given profile.
      2. 'modes' dir with files that names correspond to each mode. each file has line separated file expressions to selecting source/headers to compile.
      3. 'testdata' dir – here should be placed all data that will be needed for test and mtest applications (they are copied to gen/ directory).

here you can download build process. current version is 2.0.2.

• fixed display of problem, when compiling with icc and gcc 4.5 libs.

• added check for GCC-4.5 when using ICC - it does not compile, but can fallback to gcc-4.4 when present.
• fixed issue with no warning when build process changed.

• added clang toolchain support.
• new, more general format of toolchains selection mechanism.
• dependencies now include system headers.
• since icc 12.0.0 is fixed and works fine with ccache, warning is displayed only if older version is detected.
• gcc toolchain is now 'gcc' (name 'local' is deprecated form now on - proper warning is shown, when used).
• small-projects-makefile now supports mtest target as well.

• fixed issue with not re-running pre_configure.dep target when re-building component after a change.
• added host-specific optimization flags for gcc and icc compilers.

• build mode is now present in build directories in gen, to distinguish those builds.
• added 'features' to allows build of special parts of code on demand (i.e. turn them on/off) by specifying which files should build).
• automatic link with last 'features' set build with a short name (compatible with previous releases).
• libraries that have only headers changed are not relinked now (no need to do so).
• fixed problem when 'clean' target built components' dependencies first and then removed them.
• fixed problem with passing flags to make, like '-k'.
• build process now checks if all required features do exist in at least one component.
• sources now rebuild when compilation flags changes (from deps).
• binaries now relinks when link falgs changes (from deps).

• reorganized component dependencies backend to allow better parallelism (ex. building object files while linking dependent libraries).
• component dependencies are now computed much faster for bigger projects.
• auto-tools based components now obey V=1 make flag (verbose mode).
• output form slightly improved in few places.
• minor code fixes.

• auto-tools components are now non-verbose (this prevents screen from flooding with compilation details).

• added user-provided events before runnign configure and after running installation of auto-tools based component.
• -Werror is not propagated to external (auto-tools-based) components, since they almost always cause problems.
• CDEPS computation has been optimized - it's now ~4 times faster.
• added distcc support (via WITH_DISTCC make option).

• added user (project) specific link flags.
• implemented component-mods feature.
• added no-doxygen component mod, for disabling warnings from doxygen on certian (ex.: external) components.
• automated support for auto-tools based externa components is now available.=== v1.3.1 ====
• fixed problem with small-prjs-build when no directories where present (just files).

• fixed problem with small-prjs-build when no directories where present (just files).

• fixed display of available toolchains, when wrong is given.
• intel's icc toolchain is now supported.
• build_process now warns when using icc with ccache, which (due to icc issues) does not work properly in all cases.
• build_config/config.mk is now checked for changes as well.

• building 'doc' profile now stops when something's not/wrongly commented.
• removed redundant libraries form linking.

• added force-link functionality.
• fixed small-projects-build-process and made it official part of build_process release.
• added test application for small-prj-build-process.
• added support for components build only in some profiles (ex. tests).
• list of build-process files checked during dependency is now more readable.
• implemented 'local flags' - compilation flags available for single component only and not passed to other components, dependent on this one.
• 'test' and 'mtest' profiles now build their own 'debug' libraries; it is required since compilation flags for debug and (m)test may differ.
• verbose output when computing components' dependencies.
• flags passed to compilation are now reduced to absolute minimum.
• fixed problem with not rebuilding when new dependency file appeared (ex. 'common' or '$profile').
• files with the same name in C and C++ are now allowed (ex.: myfile.c and myfile.cpp).

• fixed 'missing dependency header' problem.

• flags for compilation and linking of profile mode are less restrictive.
• moved -rdynamic to linking, instead of compiling flags.
• switched from electric-fence to duma library for memory debugging, since the first one caused problems with pthreads.

• added automatic check for project's rebuilding when build_process changed.
• fixed problem with building MEM_DEBUG=1 version to directory with “normal” version.
• template for tut-tests has been simplified a little.
• ccache support by setting WITH_CCACHE=1 option.
• fixed bug when linking tests for dynamic library.
• moved project-specific flags to the config.mk (i.e. in order to optimize builds for given hardware, etc…).
• gen directory now has links to header files instead of copyies.
• removing public headers and unmarking header to non-public again are now supported.
• added parallel build to bzr_build.conf for multicore computers.
• fixed bug with library relinking when component had common prefix with other library name.
• test application for AVRs added.
• out-of-the-box support for gcc toolchain for AVR uCs.
• inter-component dependencies and flags checker has been rewritten from scratch.
• no features/deps/* file are needed now if component depends on nothing.
• features/deps/common_dynamic_compile and features/deps/common_dynamic_link files are now in use.
• added description of inter-component dependencies tracking in build_process/templates/features/deps/HOWTO.txt file.

• fixed bug with common dependencies/libs-links for different TC/MEM_CHECK settings.

• copying ublic headers has been improved.
• -I order has been preserved - other components won't be affected anyway.
• all dependency files are now optional - if you don't use them, you don't need them to exist.
• added 'common' dependency file, that is included for all profiles.
• testdata directory is now with own dependencies - anything that changes (add/modified) is also updated in gen/.
• change '/* public header */' to links in separete directory has been rejected due to problems with keeping two directories structures in sync in order to make it work.
• separate gen/* directories for different build options for profiles (np: toolchain).
• build_process tests now if proper variables are set (either via config or cmd-line parameter).
• added -rdynamic when building in debug (this allows backtrace reading in runtime).
• added reviesd BSD as a second license.
• fixed bug with missing symbols for applications
• dynamic link/compile + links in features directory.
• tests group's template has been simplified.
• added .out surfix to application naems, to avoid problem when app has the same name as its source direcotry.
• annoing message when dependecies have chenged is now removed - deps are updated automatically.
• toolchain selection is now possible via TC.
• added MEM_DEBUG falg - setting it to 1 causes to link with memory debug lib electric fence.
• moved profiles into separate files.
• fixed typo in test group template.
• added missing header to main.t.cpp.
• fixed bug with loosing compile-flags for dependencies.

• make exits with error when there is white char in its full path.