moveconfig.py

#!/usr/bin/env python2
#
# Author: Masahiro Yamada <yamada.masahiro@socionext.com>
#
# SPDX-License-Identifier:	GPL-2.0+
#

"""
Move config options from headers to defconfig files.

Since Kconfig was introduced to U-Boot, we have worked on moving
config options from headers to Kconfig (defconfig).

This tool intends to help this tremendous work.


Usage
-----

First, you must edit the Kconfig to add the menu entries for the configs
you are moving.

And then run this tool giving CONFIG names you want to move.
For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE,
simply type as follows:

  $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE

The tool walks through all the defconfig files and move the given CONFIGs.

The log is also displayed on the terminal.

The log is printed for each defconfig as follows:

<defconfig_name>
    <action1>
    <action2>
    <action3>
    ...

<defconfig_name> is the name of the defconfig.

<action*> shows what the tool did for that defconfig.
It looks like one of the following:

 - Move 'CONFIG_... '
   This config option was moved to the defconfig

 - CONFIG_... is not defined in Kconfig.  Do nothing.
   The entry for this CONFIG was not found in Kconfig.  The option is not
   defined in the config header, either.  So, this case can be just skipped.

 - CONFIG_... is not defined in Kconfig (suspicious).  Do nothing.
   This option is defined in the config header, but its entry was not found
   in Kconfig.
   There are two common cases:
     - You forgot to create an entry for the CONFIG before running
       this tool, or made a typo in a CONFIG passed to this tool.
     - The entry was hidden due to unmet 'depends on'.
   The tool does not know if the result is reasonable, so please check it
   manually.

 - 'CONFIG_...' is the same as the define in Kconfig.  Do nothing.
   The define in the config header matched the one in Kconfig.
   We do not need to touch it.

 - Compiler is missing.  Do nothing.
   The compiler specified for this architecture was not found
   in your PATH environment.
   (If -e option is passed, the tool exits immediately.)

 - Failed to process.
   An error occurred during processing this defconfig.  Skipped.
   (If -e option is passed, the tool exits immediately on error.)

Finally, you will be asked, Clean up headers? [y/n]:

If you say 'y' here, the unnecessary config defines are removed
from the config headers (include/configs/*.h).
It just uses the regex method, so you should not rely on it.
Just in case, please do 'git diff' to see what happened.


How does it work?
-----------------

This tool runs configuration and builds include/autoconf.mk for every
defconfig.  The config options defined in Kconfig appear in the .config
file (unless they are hidden because of unmet dependency.)
On the other hand, the config options defined by board headers are seen
in include/autoconf.mk.  The tool looks for the specified options in both
of them to decide the appropriate action for the options.  If the given
config option is found in the .config, but its value does not match the
one from the board header, the config option in the .config is replaced
with the define in the board header.  Then, the .config is synced by
"make savedefconfig" and the defconfig is updated with it.

For faster processing, this tool handles multi-threading.  It creates
separate build directories where the out-of-tree build is run.  The
temporary build directories are automatically created and deleted as
needed.  The number of threads are chosen based on the number of the CPU
cores of your system although you can change it via -j (--jobs) option.


Toolchains
----------

Appropriate toolchain are necessary to generate include/autoconf.mk
for all the architectures supported by U-Boot.  Most of them are available
at the kernel.org site, some are not provided by kernel.org.

The default per-arch CROSS_COMPILE used by this tool is specified by
the list below, CROSS_COMPILE.  You may wish to update the list to
use your own.  Instead of modifying the list directly, you can give
them via environments.


Tips and trips
--------------

To sync only X86 defconfigs:

   ./tools/moveconfig.py -s -d <(grep -l X86 configs/*)

or:

   grep -l X86 configs/* | ./tools/moveconfig.py -s -d -

To process CONFIG_CMD_FPGAD only for a subset of configs based on path match:

   ls configs/{hrcon*,iocon*,strider*} | \
       ./tools/moveconfig.py -Cy CONFIG_CMD_FPGAD -d -


Finding implied CONFIGs
-----------------------

Some CONFIG options can be implied by others and this can help to reduce
the size of the defconfig files. For example, CONFIG_X86 implies
CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and
all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to
each of the x86 defconfig files.

This tool can help find such configs. To use it, first build a database:

    ./tools/moveconfig.py -b

Then try to query it:

    ./tools/moveconfig.py -i CONFIG_CMD_IRQ
    CONFIG_CMD_IRQ found in 311/2384 defconfigs
    44 : CONFIG_SYS_FSL_ERRATUM_IFC_A002769
    41 : CONFIG_SYS_FSL_ERRATUM_A007075
    31 : CONFIG_SYS_FSL_DDR_VER_44
    28 : CONFIG_ARCH_P1010
    28 : CONFIG_SYS_FSL_ERRATUM_P1010_A003549
    28 : CONFIG_SYS_FSL_ERRATUM_SEC_A003571
    28 : CONFIG_SYS_FSL_ERRATUM_IFC_A003399
    25 : CONFIG_SYS_FSL_ERRATUM_A008044
    22 : CONFIG_ARCH_P1020
    21 : CONFIG_SYS_FSL_DDR_VER_46
    20 : CONFIG_MAX_PIRQ_LINKS
    20 : CONFIG_HPET_ADDRESS
    20 : CONFIG_X86
    20 : CONFIG_PCIE_ECAM_SIZE
    20 : CONFIG_IRQ_SLOT_COUNT
    20 : CONFIG_I8259_PIC
    20 : CONFIG_CPU_ADDR_BITS
    20 : CONFIG_RAMBASE
    20 : CONFIG_SYS_FSL_ERRATUM_A005871
    20 : CONFIG_PCIE_ECAM_BASE
    20 : CONFIG_X86_TSC_TIMER
    20 : CONFIG_I8254_TIMER
    20 : CONFIG_CMD_GETTIME
    19 : CONFIG_SYS_FSL_ERRATUM_A005812
    18 : CONFIG_X86_RUN_32BIT
    17 : CONFIG_CMD_CHIP_CONFIG
    ...

This shows a list of config options which might imply CONFIG_CMD_EEPROM along
with how many defconfigs they cover. From this you can see that CONFIG_X86
implies CONFIG_CMD_EEPROM. Therefore, instead of adding CONFIG_CMD_EEPROM to
the defconfig of every x86 board, you could add a single imply line to the
Kconfig file:

    config X86
        bool "x86 architecture"
        ...
        imply CMD_EEPROM

That will cover 20 defconfigs. Many of the options listed are not suitable as
they are not related. E.g. it would be odd for CONFIG_CMD_GETTIME to imply
CMD_EEPROM.

Using this search you can reduce the size of moveconfig patches.


Available options
-----------------

 -c, --color
   Surround each portion of the log with escape sequences to display it
   in color on the terminal.

 -C, --commit
   Create a git commit with the changes when the operation is complete. A
   standard commit message is used which may need to be edited.

 -d, --defconfigs
  Specify a file containing a list of defconfigs to move.  The defconfig
  files can be given with shell-style wildcards. Use '-' to read from stdin.

 -n, --dry-run
   Perform a trial run that does not make any changes.  It is useful to
   see what is going to happen before one actually runs it.

 -e, --exit-on-error
   Exit immediately if Make exits with a non-zero status while processing
   a defconfig file.

 -s, --force-sync
   Do "make savedefconfig" forcibly for all the defconfig files.
   If not specified, "make savedefconfig" only occurs for cases
   where at least one CONFIG was moved.

 -S, --spl
   Look for moved config options in spl/include/autoconf.mk instead of
   include/autoconf.mk.  This is useful for moving options for SPL build
   because SPL related options (mostly prefixed with CONFIG_SPL_) are
   sometimes blocked by CONFIG_SPL_BUILD ifdef conditionals.

 -H, --headers-only
   Only cleanup the headers; skip the defconfig processing

 -j, --jobs
   Specify the number of threads to run simultaneously.  If not specified,
   the number of threads is the same as the number of CPU cores.

 -r, --git-ref
   Specify the git ref to clone for building the autoconf.mk. If unspecified
   use the CWD. This is useful for when changes to the Kconfig affect the
   default values and you want to capture the state of the defconfig from
   before that change was in effect. If in doubt, specify a ref pre-Kconfig
   changes (use HEAD if Kconfig changes are not committed). Worst case it will
   take a bit longer to run, but will always do the right thing.

 -v, --verbose
   Show any build errors as boards are built

 -y, --yes
   Instead of prompting, automatically go ahead with all operations. This
   includes cleaning up headers, CONFIG_SYS_EXTRA_OPTIONS, the config whitelist
   and the README.

To see the complete list of supported options, run

  $ tools/moveconfig.py -h

"""

import collections
import copy
import difflib
import filecmp
import fnmatch
import glob
import multiprocessing
import optparse
import os
import Queue
import re
import shutil
import subprocess
import sys
import tempfile
import threading
import time

SHOW_GNU_MAKE = 'scripts/show-gnu-make'
SLEEP_TIME=0.03

# Here is the list of cross-tools I use.
# Most of them are available at kernel.org
# (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the following:
# arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases
# nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz
# nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545
# sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu
CROSS_COMPILE = {
    'arc': 'arc-linux-',
    'aarch64': 'aarch64-linux-',
    'arm': 'arm-unknown-linux-gnueabi-',
    'm68k': 'm68k-linux-',
    'microblaze': 'microblaze-linux-',
    'mips': 'mips-linux-',
    'nds32': 'nds32le-linux-',
    'nios2': 'nios2-linux-gnu-',
    'powerpc': 'powerpc-linux-',
    'sh': 'sh-linux-gnu-',
    'x86': 'i386-linux-',
    'xtensa': 'xtensa-linux-'
}

STATE_IDLE = 0
STATE_DEFCONFIG = 1
STATE_AUTOCONF = 2
STATE_SAVEDEFCONFIG = 3

ACTION_MOVE = 0
ACTION_NO_ENTRY = 1
ACTION_NO_ENTRY_WARN = 2
ACTION_NO_CHANGE = 3

COLOR_BLACK        = '0;30'
COLOR_RED          = '0;31'
COLOR_GREEN        = '0;32'
COLOR_BROWN        = '0;33'
COLOR_BLUE         = '0;34'
COLOR_PURPLE       = '0;35'
COLOR_CYAN         = '0;36'
COLOR_LIGHT_GRAY   = '0;37'
COLOR_DARK_GRAY    = '1;30'
COLOR_LIGHT_RED    = '1;31'
COLOR_LIGHT_GREEN  = '1;32'
COLOR_YELLOW       = '1;33'
COLOR_LIGHT_BLUE   = '1;34'
COLOR_LIGHT_PURPLE = '1;35'
COLOR_LIGHT_CYAN   = '1;36'
COLOR_WHITE        = '1;37'

AUTO_CONF_PATH = 'include/config/auto.conf'
CONFIG_DATABASE = 'moveconfig.db'


### helper functions ###
def get_devnull():
    """Get the file object of '/dev/null' device."""
    try:
        devnull = subprocess.DEVNULL # py3k
    except AttributeError:
        devnull = open(os.devnull, 'wb')
    return devnull

def check_top_directory():
    """Exit if we are not at the top of source directory."""
    for f in ('README', 'Licenses'):
        if not os.path.exists(f):
            sys.exit('Please run at the top of source directory.')

def check_clean_directory():
    """Exit if the source tree is not clean."""
    for f in ('.config', 'include/config'):
        if os.path.exists(f):
            sys.exit("source tree is not clean, please run 'make mrproper'")

def get_make_cmd():
    """Get the command name of GNU Make.

    U-Boot needs GNU Make for building, but the command name is not
    necessarily "make". (for example, "gmake" on FreeBSD).
    Returns the most appropriate command name on your system.
    """
    process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
    ret = process.communicate()
    if process.returncode:
        sys.exit('GNU Make not found')
    return ret[0].rstrip()

def get_matched_defconfig(line):
    """Get the defconfig files that match a pattern

    Args:
        line: Path or filename to match, e.g. 'configs/snow_defconfig' or
            'k2*_defconfig'. If no directory is provided, 'configs/' is
            prepended

    Returns:
        a list of matching defconfig files
    """
    dirname = os.path.dirname(line)
    if dirname:
        pattern = line
    else:
        pattern = os.path.join('configs', line)
    return glob.glob(pattern) + glob.glob(pattern + '_defconfig')

def get_matched_defconfigs(defconfigs_file):
    """Get all the defconfig files that match the patterns in a file.

    Args:
        defconfigs_file: File containing a list of defconfigs to process, or
            '-' to read the list from stdin

    Returns:
        A list of paths to defconfig files, with no duplicates
    """
    defconfigs = []
    if defconfigs_file == '-':
        fd = sys.stdin
        defconfigs_file = 'stdin'
    else:
        fd = open(defconfigs_file)
    for i, line in enumerate(fd):
        line = line.strip()
        if not line:
            continue # skip blank lines silently
        if ' ' in line:
            line = line.split(' ')[0]  # handle 'git log' input
        matched = get_matched_defconfig(line)
        if not matched:
            print >> sys.stderr, "warning: %s:%d: no defconfig matched '%s'" % \
                                                 (defconfigs_file, i + 1, line)

        defconfigs += matched

    # use set() to drop multiple matching
    return [ defconfig[len('configs') + 1:]  for defconfig in set(defconfigs) ]

def get_all_defconfigs():
    """Get all the defconfig files under the configs/ directory."""
    defconfigs = []
    for (dirpath, dirnames, filenames) in os.walk('configs'):
        dirpath = dirpath[len('configs') + 1:]
        for filename in fnmatch.filter(filenames, '*_defconfig'):
            defconfigs.append(os.path.join(dirpath, filename))

    return defconfigs

def color_text(color_enabled, color, string):
    """Return colored string."""
    if color_enabled:
        # LF should not be surrounded by the escape sequence.
        # Otherwise, additional whitespace or line-feed might be printed.
        return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else ''
                           for s in string.split('\n') ])
    else:
        return string

def show_diff(a, b, file_path, color_enabled):
    """Show unidified diff.

    Arguments:
      a: A list of lines (before)
      b: A list of lines (after)
      file_path: Path to the file
      color_enabled: Display the diff in color
    """

    diff = difflib.unified_diff(a, b,
                                fromfile=os.path.join('a', file_path),
                                tofile=os.path.join('b', file_path))

    for line in diff:
        if line[0] == '-' and line[1] != '-':
            print color_text(color_enabled, COLOR_RED, line),
        elif line[0] == '+' and line[1] != '+':
            print color_text(color_enabled, COLOR_GREEN, line),
        else:
            print line,

def update_cross_compile(color_enabled):
    """Update per-arch CROSS_COMPILE via environment variables

    The default CROSS_COMPILE values are available
    in the CROSS_COMPILE list above.

    You can override them via environment variables
    CROSS_COMPILE_{ARCH}.

    For example, if you want to override toolchain prefixes
    for ARM and PowerPC, you can do as follows in your shell:

    export CROSS_COMPILE_ARM=...
    export CROSS_COMPILE_POWERPC=...

    Then, this function checks if specified compilers really exist in your
    PATH environment.
    """
    archs = []

    for arch in os.listdir('arch'):
        if os.path.exists(os.path.join('arch', arch, 'Makefile')):
            archs.append(arch)

    # arm64 is a special case
    archs.append('aarch64')

    for arch in archs:
        env = 'CROSS_COMPILE_' + arch.upper()
        cross_compile = os.environ.get(env