Skip to content
Snippets Groups Projects
Select Git revision
0 results
Search by author
  • Any Author
0 authors
  1. Jan 26, 2023
    • Masahiro Yamada's avatar
      scripts: handle BrokenPipeError for python scripts · 87c7ee67
      Masahiro Yamada authored 
      In the follow-up of commit fb3041d6 ("kbuild: fix SIGPIPE error
message for AR=gcc-ar and AR=llvm-ar"), Kees Cook pointed out that
tools should _not_ catch their own SIGPIPEs [1] [2].

Based on his feedback, LLVM was fixed [3].

However, Python's default behavior is to show noisy bracktrace when
SIGPIPE is sent. So, scripts written in Python are basically in the
same situation as the buggy llvm tools.

Example:

  $ make -s allnoconfig
  $ make -s allmodconfig
  $ scripts/diffconfig .config.old .config | head -n1
  -ALIX n
  Traceback (most recent call last):
    File "/home/masahiro/linux/scripts/diffconfig", line 132, in <module>
      main()
    File "/home/masahiro/linux/scripts/diffconfig", line 130, in main
      print_config("+", config, None, b[config])
    File "/home/masahiro/linux/scripts/diffconfig", line 64, in print_config
      print("+%s %s" % (config, new_value))
  BrokenPipeError: [Errno 32] Broken pipe

Python documentation [4] notes how to make scripts die immediately and
silently:

  """
  Piping output of your program to tools like head(1) will cause a
  SIGPIPE signal to be sent to your process when the receiver of its
  standard output closes early. This results in an exception like
  BrokenPipeError: [Errno 32] Broken pipe. To handle this case,
  wrap your entry point to catch this exception as follows:

    import os
    import sys

    def main():
        try:
            # simulate large output (your code replaces this loop)
            for x in range(10000):
                print("y")
            # flush output here to force SIGPIPE to be triggered
            # while inside this try block.
            sys.stdout.flush()
        except BrokenPipeError:
            # Python flushes standard streams on exit; redirect remaining output
            # to devnull to avoid another BrokenPipeError at shutdown
            devnull = os.open(os.devnull, os.O_WRONLY)
            os.dup2(devnull, sys.stdout.fileno())
            sys.exit(1)  # Python exits with error code 1 on EPIPE

    if __name__ == '__main__':
        main()

  Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid
  BrokenPipeError. Doing that would cause your program to exit
  unexpectedly whenever any socket connection is interrupted while
  your program is still writing to it.
  """

Currently, tools/perf/scripts/python/intel-pt-events.py seems to be the
only script that fixes the issue that way.

tools/perf/scripts/python/compaction-times.py uses another approach
signal.signal(signal.SIGPIPE, signal.SIG_DFL) but the Python
documentation clearly says "Don't do it".

I cannot fix all Python scripts since there are so many.
I fixed some in the scripts/ directory.

[1]: https://lore.kernel.org/all/202211161056.1B9611A@keescook/
[2]: https://github.com/llvm/llvm-project/issues/59037
[3]: https://github.com/llvm/llvm-project/commit/4787efa38066adb51e2c049499d25b3610c0877b
[4]: https://docs.python.org/3/library/signal.html#note-on-sigpipe



Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: default avatarNick Desaulniers <ndesaulniers@google.com>
Reviewed-by: default avatarNicolas Schier <nicolas@fjasle.eu>
      87c7ee67
  3. Sep 19, 2021
    • Ariel Marcovitch's avatar
      checkkconfigsymbols.py: Remove skipping of help lines in parse_kconfig_file · aa0f5ea1
      Ariel Marcovitch authored 
      
When parsing Kconfig files to find symbol definitions and references,
lines after a 'help' line are skipped until a new config definition
starts.

However, Kconfig statements can actually be after a help section, as
long as these have shallower indentation. These are skipped by the
parser.

This means that symbols referenced in this kind of statements are
ignored by this function and thus are not considered undefined
references in case the symbol is not defined.

Remove the 'skip' logic entirely, as it is not needed if we just use the
STMT regex to find the end of help lines.

However, this means that keywords that appear as part of the help
message (i.e. with the same indentation as the help lines) it will be
considered as a reference/definition. This can happen now as well, but
only with REGEX_KCONFIG_DEF lines. Also, the keyword must have a SYMBOL
after it, which probably means that someone referenced a config in the
help so it seems like a bonus :)

The real solution is to keep track of the indentation when a the first
help line in encountered and then handle DEF and STMT lines only if the
indentation is shallower.

Signed-off-by: default avatarAriel Marcovitch <arielmarcovitch@gmail.com>
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      aa0f5ea1
    • Ariel Marcovitch's avatar
      checkkconfigsymbols.py: Forbid passing 'HEAD' to --commit · d62d5aed
      Ariel Marcovitch authored 
      
As opposed to the --diff option, --commit can get ref names instead of
commit hashes.

When using the --commit option, the script resets the working directory
to the commit before the given ref, by adding '~' to the end of the ref.

However, the 'HEAD' ref is relative, and so when the working directory
is reset to 'HEAD~', 'HEAD' points to what was 'HEAD~'. Then when the
script resets to 'HEAD' it actually stays in the same commit. In this
case, the script won't report any cases because there is no diff between
the cases of the two refs.

Prevent the user from using HEAD refs.

A better solution might be to resolve the refs before doing the
reset, but for now just disallow such refs.

Signed-off-by: default avatarAriel Marcovitch <arielmarcovitch@gmail.com>
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      d62d5aed
  5. Sep 02, 2021
    • Ariel Marcovitch's avatar
      checkkconfigsymbols.py: Fix the '--ignore' option · 1439ebd2
      Ariel Marcovitch authored 
      
It seems like the implementation of the --ignore option is broken.

In check_symbols_helper, when going through the list of files, a file is
added to the list of source files to check if it matches the ignore
pattern. Instead, as stated in the comment below this condition, the
file should be added if it doesn't match the pattern.

This means that when providing an ignore pattern, the only files that
will be checked will be the ones we want the ignore, in addition to the
Kconfig files that don't match the pattern (the check in
parse_kconfig_files is done right)

Signed-off-by: default avatarAriel Marcovitch <arielmarcovitch@gmail.com>
Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
      1439ebd2
  7. Aug 14, 2020
  9. May 30, 2019
  11. Feb 03, 2017
  13. Jan 19, 2017
  15. Oct 28, 2016
  17. Aug 28, 2016
  19. Aug 27, 2016
  21. Aug 18, 2016
  23. May 01, 2016
  25. Dec 14, 2015
  27. Aug 04, 2015
  29. May 24, 2015
  31. Mar 24, 2015
  33. Mar 16, 2015
    • Valentin Rothberg's avatar
      checkkconfigsymbols.py: make it Git aware · b1a3f243
      Valentin Rothberg authored 
      
The script now supports to check a specified commit or a specified range
of commits (i.e., commit1..commit2).  Developers and maintainers are
encouraged to use this functionality before sending or merging patches
to avoid potential bugs and to keep the code, documentation, etc. clean.

This patch adds the following options to the script:

 -c COMMIT, --commit=COMMIT
                  Check if the specified commit (hash) introduces
                  undefined Kconfig symbols.

 -d DIFF, --diff=DIFF
                  Diff undefined symbols between two commits.  The input
                  format bases on Git log's 'commmit1..commit2'.

  --force         Reset current Git tree even when it's dirty.

Note that the first two options require to 'git reset --hard' the user's
Git tree.  This hard reset is necessary to keep the script fast, but it
can lead to the loss of uncommitted data.  Hence, the script aborts in
case it is executed in a dirty tree.  It won't abort if '--force' is
passed.

If neither -c nor -d is specified, the script defaults to check the
entire local tree (i.e., the previous behavior).

Signed-off-by: default avatarValentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
      b1a3f243
  35. Mar 09, 2015
  37. Nov 09, 2014
  39. Nov 07, 2014
    • Valentin Rothberg's avatar
      checkkconfigsymbols.sh: reimplementation in python · 24fe1f03
      Valentin Rothberg authored 
      
The scripts/checkkconfigsymbols.sh script searches Kconfig features
in the source code that are not defined in Kconfig. Such identifiers
always evaluate to false and are the source of various kinds of bugs.
However, the shell script is slow and it does not detect such broken
references in Kbuild and Kconfig files (e.g., ``depends on UNDEFINED´´).
Furthermore, it generates false positives. The script is also hard to
read and understand, and is thereby difficult to maintain.

This patch replaces the shell script with an implementation in Python,
which:
    (a) detects the same bugs, but does not report previous false positives
    (b) additionally detects broken references in Kconfig and all
        non-Kconfig files, such as Kbuild, .[cSh], .txt, .sh, defconfig, etc.
    (c) is up to 75 times faster than the shell script
    (d) only checks files under version control

The new script reduces the runtime on my machine (i7-2620M, 8GB RAM, SSD)
from 3m47s to 0m3s, and reports 938 broken references in Linux v3.17-rc1;
419 additional reports of which 16 are located in Kconfig files,
287 in defconfigs, 63 in ./Documentation, 1 in Kbuild.

Moreover, we intentionally include references in comments, which have been
ignored until now. Such comments may be leftovers of features that have
been removed or renamed in Kconfig (e.g., ``#endif /* CONFIG_MPC52xx */´´).
These references can be misleading and should be removed or replaced.

Note that the output format changed from (file list <tab> feature) to
(feature <tab> file list) as it simplifies the detection of the Kconfig
feature for long file lists.

Signed-off-by: default avatarValentin Rothberg <valentinrothberg@gmail.com>
Signed-off-by: default avatarStefan Hengelein <stefan.hengelein@fau.de>
Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
      24fe1f03