Mozbuild Conversion Estimates

mozbuild conversion estimates
=============================
 
Back in June/July time estimates were made for completion of the mozbuild conversions.
Stats were based on elapsed time for wip conversion bugs.  Bug dependencies, variables to convert, etc.  Some time has elapsed but here is a digest of information that was gathered.
 
 
Detail from past/ongoing conversions:
 
Stats cover
  o one person doing conversion work.
  o conversions are mostly verbatim/simple passthrough so overhead from code reviews can be ignored.
  o elements from different variable conversions may be in play at any given time.
  o end-to-end times for past conversion bugs has been a 2 month avg (min=1, max=5).
  o 6 days of try jobs submissions for each variable:
      build[3]: platforms[subset], bugfix_or_repo, platforms[all].
       test[3]: platforms[subset], bugfix_or_report, platforms[all].
 
Makefile content detail (collection date: August 8th):
  o To determine how many variables will need to be converted stats are gathered from Makefile.in and *.mk
  o Exclude as needed, /third_party/, /nsprpub/, Android.mk, etc, etc.
  o A list of variables to convert ‘valid’ are gathered from beneath config/.
    – Variables referenced by rules.mk, makefiles/*.mk (anything except config/Makefile.in) are considered.
    – Variable names must be in all uppercase.
    – These variable names are considered valid and require conversion.
  o Gather Makefile.in and *.mk from the sandbox and parse out a list of variable assignments.
    – Only variables in all uppercase are considered.
    – If the variable is mentioned beneath config/ it will be considered a conversion variable.
    – All other variables (mixed or lower case, not referenced) are considered local variables.
    – From data gathering 77 variables were identified as variables to be converted.
    – Makefiles which contain a valid variable are included in makefile stats.
 
Stats as of/between 07/17 and 08/05:
   Makefiles found: 1112, *.mk files: 75
   Vars to convert: 77 (07/17: 80, 08/05: 77)
   Avg conversion time: 2 months
   Pending conversion bugs: 49 (deps of 847009)
 
Serially mozbuild conversions will require a long, long time to complete at this pace.
The conversion process can benefit heavily from threading, one person per variable.
 
Though threading will introduce a few new issues/bottlenecks to contend with:
  – conflicts due to mozbuild changes or competing makefile edits.
  – having to re-test patches – testing invalidated due to recent patch landings.
 
Possible workarounds:
  – land all mozbuild logic changes early, before makefile patches are generated.
  – submit several smaller conversion patches (a few per directory) to reduce the chance for contention.
  – avoid testing altogether and most of these problems will disappear :).
 
Posted in Uncategorized | Tagged | Leave a comment

mozbuild conversion status

Hmmm one month has elapsed since the last conversion blog post so it is proably time for a status update.

If you are interested in expediting migration of a project over to mozbuild or helping out with the conversion effort to deprecate Makefile.in read this earlier blog post for help on getting started:

For conversion status here are all the gory details:

Converted variable:

    CMMSRCS
    CPPSRCS
    EXTRA_JS_MODULES
    EXTRA_PP_COMPONENTS
    HOST_CPPSRCS
    HOST_CSRCS
    HOST_LIBRARY_NAME
    SDK_HEADERS
    SSRCS
    XPCSHELL_TESTS

In progress: supported but some Makefile.in variables linger.

    ASFILES
    CSRCS
    EXTRA_COMPONENTS
    HOST_CPPSRCS
    LIBRARY_NAME
    SDK_LIBRARY
    SIMPLE_PROGRAMS

** Item of note:
Content for converted variables should now be maintained within the appropriate moz.build file. In the very near future, attempting to build while any of these variables are declared within Makefile.in will be handled as an error condition.

[1]
This wiki reports current status for conversion of mozilla-central makefile variables. jcranmer has been maintaining comm-central info on this page so you can also find conversion info for thunderbird/seamonkey/…

[2] – Feel free to take and work on any bugs linked to by the nomakefiles tracking bug. Any conversion bugs that are not assigned or have patches attached have yet to be worked on.

Posted in mozbuild | Leave a comment

How you can help with moz.build conversions.

If you would like to help out with moz.build conversions simply begin maintaining build information within the moz.build file along with porting existing Makefile.in content.

Logic to support individual variables will need to exist within mozbuild before porting can take place.

To determine if a variable has already been converted there are a few options:

  • python/mozbuild/mozbuild/frontend/sandbox_symbols.py
    A list of supported tokens can be found mentioned in the VARIABLES= assignment.
  • Bugzilla/wiki – variables with ‘logic’ in the in_mozbuild column are supported.
  • Simple answer: just try it
    • Add a variable in moz.build: FOO_BAR += ['value']
    • Run ./mach configure
    • If configure fails, scroll back and search for ‘ERROR PROCESSING MOZBUILD FILE’. If the error reported was “The underlying problem is an attempt to read a reserved UPPERCASE variable that does not exist”. Chances are good support for the variable has not yet been added in moz.build. Feel free to open bugs for missing variables as needed so they can be included and converted.

Code reviews for Makefile.in/moz.build changes can be forwarded to:

    Joey Armstrong[:jarmstrong:]
    Mike Shal[:mshal:]
    Greg Szorc[:gps:]
    or build::config.

Code reviews for python/mozbuild changes to support new variables can be forwarded to:

    Greg Szorc[:gps:]
    or build::config peers


0:02.72 mozbuild.frontend.reader.BuildReaderError: 
0:02.72 ERROR PROCESSING MOZBUILD FILE
0:02.72
0:02.72 The error occurred while processing the following file:
0:02.72
0:02.72     /media/_Q/mozilla/bugs/870407/mfbt/moz.build
0:02.72
0:02.72 The error was triggered on line 11 of this file:
0:02.72
0:02.72     INVALID
0:02.72
0:02.72 The underlying problem is an attempt to read a reserved UPPERCASE variable that does not exist.
0:02.72
0:02.72 The variable read causing the error is:
0:02.72
0:02.72     INVALID
0:02.72
0:02.72 Please change the file to not use this variable.
0:02.72
0:02.72 For reference, the set of valid variables is:
0:02.72
0:02.72 ASFILES, CMMSRCS, CONFIGURE_SUBST_FILES, CPP_SOURCES, CSRCS, DEFINES, DIRS, EXPORTS, EXTERNAL_MAKE_DIRS, EXTRA_COMPONENTS, HOST_CSRCS, HOST_LIBRARY_NAME, MODULE, PARALLEL_DIRS, PARALLEL_EXTERNAL_MAKE_DIRS, PROGRAM, SIMPLE_PROGRAMS, TEST_DIRS, TEST_TOOL_DIRS, TIERS, TOOL_DIRS, XPCSHELL_TESTS_MANIFESTS, XPIDL_FLAGS, XPIDL_MODULE, XPIDL_SOURCES
Posted in mozbuild | Tagged | Leave a comment

mozbuild variable conversions part deux

mozbuild conversions are ongoing, the following is a status update around progress.

The following variables are now supported by mozbuild. Any content changes or addition to them should be made within directory specific moz.build files rather than editing Makefile.in:

  • ASFILES
  • CSRCS
  • CPP_SOURCES
  • EXPORT*
  • SIMPLE_PROGRAMS
  • XPCSHELL_TESTS

Some Makefile.in content for variables will persist while conversions are ongoing. As individual variables are converted and cleanup work is complete, expect changes to be made that will reject future use of converted variables within Makefile.in.

Pending variable conversions

  • CMMSRCS
  • DEFINES
  • HOST_LIBRARY_NAME
  • PREF_JS_EXPORTS
  • SUBMAKEFILES

Pass-through variable conversions:

To expedite migrating content from makefile to mozbuild, per-variable passthrough conversions can be added in python/mozbuild/ without having to fully implement a subroutine to handle each variable and nuances.

The following approach has helped avoid taking an all-or-nothing approach for transitioning variables.

  • Add support for an individual variable within python/mozbuild.
  • Convert makefiles within several directories.
  • loop on: review, try, submit
  • Submit several smaller patches containing bundles of files.
  • post conversion makefile cleanup.
  • reject variable use within Makefile.in
  • deprecate Makefile.in if all logic and variables have been converted.

These operations can be done asynchronously and test/random failures will not imply backing out the entire set of changed files for isolated problems. In contrast some of the early conversion patches contained 800+ files and cycled through several iterations and wide area failures before finally landing.

If you would like to work on some of the passthrough conversions an example from the CSRCS= conversion can be found here:

http://hg.mozilla.org/mozilla-central/rev/e3faa44c33e4

Add support for new varibles + unit test within python/mozbuild.
Convert makefile content to mozbuild.

Ongoing status tracking for variable conversions can be found here:

A list of variables currently suppored by moz.build can be found in sandbox_symbols.py:

http://hg.mozilla.org/mozilla-central/file/ddb7b23166ef/python/mozbuild/mozbuild/frontend/sandbox_symbols.py

Posted in mozbuild | 1 Comment

Subject: Transition to mozbuild: ASFILES – Assembly language

Subject: Transition to mozbuild: ASFILES – Assembly language source files
 
Part #1 of the mozbuild conversion for bug 869135 has landed:
 
This patch has added ability to define lists of assembly language files within mozbuild configuration.  Files are declared within a list context specifying one file per line:
 
ASFILES=  variables will continue to exist in Makefile.in until all files have been converted:
Variable names of ‘DISABLED_ASFILES’ indicate the file list has been converted.
Variable names of ‘ASFILES’ have yet to be converted.
Independent of the *ASFILES variable name, always insert the name of new files within the mozbuild config file.  DISABLED_ASFILE variables will be removed from Makefile.in after all ASFILES references have been converted.  Afterward the mozbuild program will be updated to disallow use of ASFILES within Makefile.in
 
Current state for variable conversions and relevant bugs can be found here:
 
Aside | Posted on by | Leave a comment

Transitioning to moz.build

If you have been wondering about status for the transition to mozbuild as a replacement for makefile infrastructure, a source is becoming available. People involved with the conversion have been meeting bi-weekly on the topic – details and content can be found in etherpad meeting notes:

https://etherpad.mozilla.org/build-config-mtg-2013-05-09

Currently an effort is underway to begin bulk porting makefile variables over to moz.build to help finalize this portion of the transition. As makfile variables are converted advanced notice will be given before the changes go live. Also examples will likely be provided to document syntax needed for adding new content.

Stats are being gathered for amount of effort required to convert individual variables. Bugs are being opened to track conversion of individual variables and after being digested a bit more a summary page will be maintained to allow viewing state of conversion progress at a glance.

Posted in mozbuild | Leave a comment

Creating directories by dependency now a little easier

https://bugzilla.mozilla.org/show_bug.cgi?id=750303

Bug 750303 has landed and will help make dependent directory creation for {+/-nop} builds a bit easier.

Current handling involves generating dependencies with the mkdir_deps library function then listing them as explicit target pre-requisites within a makefile:

    lib-preqs = $(call mkdir_deps,install_dir)
    my-lib-targets: $(lib-preqs)

Bug 750303 has added a new set of GENERATED_DIRS_* variables to handle dep/directory creation. Variables are assigned a simple list of directories that will be automagically created when a target is processed.

    GENERATED_DIRS_export := foo
    GENERATED_DIRS_libs := bar tans
    GENERATED_DIRS_tools := fans

WRT the ‘_tools’ variable, a fans directory will be created when ‘make tools’ is invoked and the fans/ directory does not already exist.

Using a variable assignment the -preqs example above can now be written as:

    GENERATED_DIRS_libs := install_here
    include $(topsrcdir)/config/rules.mk

    libs:: tests/input.sh
    $(CP) $< install_here

One item of note: GENERATED_DIRS_* will also be a stepping stone for converting makefiles into moz.build synatx. When directory creation logic has been isolated within a fixed set of variables they can easily be identified and extracted in bulk for conversion.

Posted in makefiles | Leave a comment