Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions doc/man/zkg.1
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,7 @@ usage: zkg refresh [\-h] [\-\-aggregate] [\-\-fail\-on\-aggregate\-problems] [\-
.INDENT 7.0
.TP
.B \-\-aggregate=False
Crawls the urls listed in package source zkg.index files and aggregates the metadata found in their zkg.meta (or legacy bro\-pkg.meta) files. The aggregated metadata is stored in the local clone of the package source that zkg uses internally for locating package metadata. For each package, the metadata is taken from the highest available git version tag or the default branch, like \(dqmain\(dq or \(dqmaster\(dq, if no version tags exist
Crawls the urls listed in package source zkg.index files and aggregates the metadata found in their zkg.meta files. The aggregated metadata is stored in the local clone of the package source that zkg uses internally for locating package metadata. For each package, the metadata is taken from the highest available git version tag or the default branch, like \(dqmain\(dq or \(dqmaster\(dq, if no version tags exist
.TP
.B \-\-fail\-on\-aggregate\-problems=False
When using \-\-aggregate, exit with error when any packages trigger metadata problems. Normally such problems only cause a warning.
Expand Down Expand Up @@ -721,7 +721,7 @@ state_dir =
# A subdirectory named \(dqpackages\(dq is always created within the
# specified path and the package manager will copy the directory
# specified by the \(dqscript_dir\(dq option of each package\(aqs zkg.meta
# (or legacy bro\-pkg.meta) file there.
# file there.
# If left blank or with \-\-user this defaults to <state_dir>/script_dir.
# In Zeek\-bundled installations, it defaults to
# <zeek_install_prefix>/share/zeek/site.
Expand All @@ -734,7 +734,7 @@ script_dir =
# A subdirectory named \(dqpackages\(dq is always created within the
# specified path and the package manager will copy the directory
# specified by the \(dqplugin_dir\(dq option of each package\(aqs zkg.meta
# (or legacy bro\-pkg.meta) file there.
# file there.
# If left blank or with \-\-user this defaults to <state_dir>/plugin_dir.
# In Zeek\-bundled installations, it defaults to
# <zeek_install_prefix>/lib/zeek/plugins.
Expand Down Expand Up @@ -766,7 +766,7 @@ default = https://github.com/zeek/package\-template
[user_vars]

# For any key in this section that is matched for value interpolation
# in a package\(aqs zkg.meta (or legacy bro\-pkg.meta) file, the corresponding
# in a package\(aqs zkg.meta file, the corresponding
# value is substituted during execution of the package\(aqs \(gabuild_command\(ga.
# This section is typically automatically populated with the
# the answers supplied during package installation prompts
Expand Down
14 changes: 4 additions & 10 deletions doc/source.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,6 @@ usernames or some unique way of identifying the organization/person that
maintains Zeek packages. However, a source is free to use a flat organization
with a single, top-level :file:`zkg.index`.

.. note::

The magic index file name of :file:`zkg.index` is available :program:`since
zkg v2.0`. For compatibility purposes, the old index file name of
:file:`bro-pkg.index` is also still supported.

After creating a git repo for the package source and adding package index files
to it, it's ready to be used by :ref:`zkg <zkg>`.

Expand All @@ -36,7 +30,7 @@ to it, it's ready to be used by :ref:`zkg <zkg>`.
Package Index Files
-------------------

Files named :file:`zkg.index` (or the legacy :file:`bro-pkg.index`) are used to
Files named :file:`zkg.index` are used to
describe the :doc:`Zeek Packages <package>` found within the package source.
They are simply a list of git URLs pointing to the git repositories of
packages. For example::
Expand Down Expand Up @@ -75,9 +69,9 @@ Aggregating Metadata
--------------------

The maintainer/operator of a package source may choose to periodically aggregate
the metadata contained in its packages' :file:`zkg.meta` (and legacy
:file:`bro-pkg.meta`) files. The :ref:`zkg refresh <refresh-command>`
is used to perform the task. For example:
the metadata contained in its packages' :file:`zkg.meta` files.
The :ref:`zkg refresh <refresh-command>` is used to perform the task. For
example:

.. code-block:: console

Expand Down
2 changes: 0 additions & 2 deletions testing/baselines/tests.install-bro-pkg-warning/stderr

This file was deleted.

4 changes: 2 additions & 2 deletions testing/baselines/tests.refresh/agg.errout
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
XXXX-XX-XX XX:XX:XX DEBUG found source clone of "one" at <...>/one
XXXX-XX-XX XX:XX:XX DEBUG refresh "one": pulling <...>/one
XXXX-XX-XX XX:XX:XX WARNING <...>/bro-pkg.meta: missing metadata file
XXXX-XX-XX XX:XX:XX WARNING skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta (or bro-pkg.meta) metadata file
XXXX-XX-XX XX:XX:XX WARNING <...>/zkg.meta: missing metadata file
XXXX-XX-XX XX:XX:XX WARNING skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta metadata file
XXXX-XX-XX XX:XX:XX DEBUG metadata refresh: 8 additions (alice/bar, alice/baz, alice/foo, alice/i-have-no-scripts, alice/new_pkg, alice/qux, bob/corge, bob/grault), 0 changes, 0 removals
XXXX-XX-XX XX:XX:XX INFO committed package source "one" metadata update
XXXX-XX-XX XX:XX:XX DEBUG fetch package one<...>/foo
4 changes: 2 additions & 2 deletions testing/baselines/tests.refresh/agg.fail.errout
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
XXXX-XX-XX XX:XX:XX DEBUG found source clone of "one" at <...>/one
XXXX-XX-XX XX:XX:XX DEBUG refresh "one": pulling <...>/one
XXXX-XX-XX XX:XX:XX WARNING <...>/bro-pkg.meta: missing metadata file
XXXX-XX-XX XX:XX:XX WARNING skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta (or bro-pkg.meta) metadata file
XXXX-XX-XX XX:XX:XX WARNING <...>/zkg.meta: missing metadata file
XXXX-XX-XX XX:XX:XX WARNING skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta metadata file
XXXX-XX-XX XX:XX:XX DEBUG metadata refresh: 8 additions (alice/bar, alice/baz, alice/foo, alice/i-have-no-scripts, alice/new_pkg, alice/qux, bob/corge, bob/grault), 0 changes, 0 removals
XXXX-XX-XX XX:XX:XX DEBUG fetch package one<...>/foo
2 changes: 1 addition & 1 deletion testing/baselines/tests.refresh/agg.fail.out
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
Refresh package source: one
No membership changes
WARNING: Metadata aggregated, but excludes the following packages due to described problems:
<...>/bad_pkg: missing zkg.meta (or bro-pkg.meta) metadata file
<...>/bad_pkg: missing zkg.meta metadata file
Refresh installed packages
No new outdated packages
2 changes: 1 addition & 1 deletion testing/baselines/tests.refresh/agg.out
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Refresh package source: one
No membership changes
WARNING: Metadata aggregated, but excludes the following packages due to described problems:
<...>/bad_pkg: missing zkg.meta (or bro-pkg.meta) metadata file
<...>/bad_pkg: missing zkg.meta metadata file
Pushed aggregated metadata
Refresh installed packages
No new outdated packages
3 changes: 0 additions & 3 deletions testing/tests/install-bro-pkg-warning

This file was deleted.

54 changes: 10 additions & 44 deletions zeekpkg/manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,6 @@
from .package import (
BUILTIN_SCHEME,
BUILTIN_SOURCE,
LEGACY_METADATA_FILENAME,
LEGACY_PLUGIN_MAGIC_FILE,
LEGACY_PLUGIN_MAGIC_FILE_DISABLED,
METADATA_FILENAME,
PLUGIN_MAGIC_FILE,
PLUGIN_MAGIC_FILE_DISABLED,
Expand Down Expand Up @@ -195,18 +192,17 @@ class Manager:

script_dir (str): the directory where the package manager will
copy each installed package's `script_dir` (as given by its
:file:`zkg.meta` or :file:`bro-pkg.meta`). Each package gets a
subdirectory within `script_dir` associated with its name.
:file:`zkg.meta`). Each package gets a subdirectory within
`script_dir` associated with its name.

plugin_dir (str): the directory where the package manager will
copy each installed package's `plugin_dir` (as given by its
:file:`zkg.meta` or :file:`bro-pkg.meta`). Each package gets a
subdirectory within `plugin_dir` associated with its name.
:file:`zkg.meta`). Each package gets a subdirectory within
`plugin_dir` associated with its name.

bin_dir (str): the directory where the package manager will link
executables into that are provided by an installed package through
`executables` (as given by its :file:`zkg.meta` or
:file:`bro-pkg.meta`)
`executables` (as given by its :file:`zkg.meta`)

source_clonedir (str): the directory where the package manager
will clone package sources. Each source gets a subdirectory
Expand Down Expand Up @@ -409,22 +405,12 @@ def _write_plugin_magic(self, ipkg: InstalledPackage) -> None:
When the package doesn't include a plugin, or when the plugin
directory already contains a correctly named magic file, this
function does nothing.

Until Zeek 6.1, the magic file was named __bro_plugin__. zkg implements
a fallback for recognizing the older name so that newer zkg versions
continue to work with older Zeek versions for some time longer.
"""
package_dir = pathlib.Path(self.plugin_dir) / ipkg.package.name

magic_paths_enabled = [
package_dir / PLUGIN_MAGIC_FILE,
package_dir / LEGACY_PLUGIN_MAGIC_FILE,
]
magic_paths_enabled = [package_dir / PLUGIN_MAGIC_FILE]

magic_paths_disabled = [
package_dir / PLUGIN_MAGIC_FILE_DISABLED,
package_dir / LEGACY_PLUGIN_MAGIC_FILE_DISABLED,
]
magic_paths_disabled = [package_dir / PLUGIN_MAGIC_FILE_DISABLED]

for path_enabled, path_disabled in zip(
magic_paths_enabled,
Expand Down Expand Up @@ -3396,7 +3382,7 @@ def ignore(_: str, files: list[str]) -> list[str]:
rval = []

for f in files:
if f in {".git", "bro-pkg.meta", "zkg.meta"}:
if f in {".git", "zkg.meta"}:
rval.append(f)

return rval
Expand Down Expand Up @@ -3451,12 +3437,7 @@ def _get_package_metadata(parser: configparser.ConfigParser) -> dict[str, str]:


def _pick_metadata_file(directory: str) -> str:
rval = os.path.join(directory, METADATA_FILENAME)

if os.path.exists(rval):
return rval

return os.path.join(directory, LEGACY_METADATA_FILENAME)
return os.path.join(directory, METADATA_FILENAME)


def _parse_package_metadata(
Expand All @@ -3466,9 +3447,7 @@ def _parse_package_metadata(
"""Return string explaining why metadata is invalid, or '' if valid."""
if not parser.read(metadata_file):
LOG.warning("%s: missing metadata file", metadata_file)
return (
f"missing {METADATA_FILENAME} (or {LEGACY_METADATA_FILENAME}) metadata file"
)
return f"missing {METADATA_FILENAME} metadata file"

if not parser.has_section("package"):
LOG.warning("%s: metadata missing [package]", metadata_file)
Expand Down Expand Up @@ -3521,19 +3500,6 @@ def _info_from_clone(
default_branch=default_branch,
)

if (
os.path.basename(metadata_file) == LEGACY_METADATA_FILENAME
and package.qualified_name() not in _legacy_metadata_warnings
):
LOG.warning(
"Package %s is using the legacy bro-pkg.meta metadata file. "
"While bro-pkg.meta still functions, it is recommended to "
"use zkg.meta instead for future-proofing. Please report this "
"to the package maintainers.",
package.qualified_name(),
)
_legacy_metadata_warnings.add(package.qualified_name())

metadata = _get_package_metadata(metadata_parser)

return PackageInfo(
Expand Down
19 changes: 7 additions & 12 deletions zeekpkg/package.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@

#: The name of files used by packages to store their metadata.
METADATA_FILENAME = "zkg.meta"
LEGACY_METADATA_FILENAME = "bro-pkg.meta"

TRACKING_METHOD_VERSION = "version"
TRACKING_METHOD_BRANCH = "branch"
Expand All @@ -26,8 +25,6 @@

PLUGIN_MAGIC_FILE = "__zeek_plugin__"
PLUGIN_MAGIC_FILE_DISABLED = "__zeek_plugin__.disabled"
LEGACY_PLUGIN_MAGIC_FILE = "__bro_plugin__"
LEGACY_PLUGIN_MAGIC_FILE_DISABLED = "__bro_plugin__.disabled"


def name_from_path(path: str) -> str:
Expand Down Expand Up @@ -309,7 +306,7 @@ class PackageInfo:
packages

metadata (dict of str -> str): the contents of the package's
:file:`zkg.meta` or :file:`bro-pkg.meta`
:file:`zkg.meta`

versions (list of str): a list of the package's availabe git version
tags
Expand All @@ -323,10 +320,9 @@ class PackageInfo:
invalid_reason (str): this attribute is set when there is a problem
with gathering package information and explains what went wrong.

metadata_file: the absolute path to the :file:`zkg.meta` or
:file:`bro-pkg.meta` for this package. Use this if you'd like to
parse the metadata yourself. May not be defined, in which case the
value is None.
metadata_file: the absolute path to the :file:`zkg.meta` for this package.
Use this if you'd like to parse the metadata yourself. May not be
defined, in which case the value is None.
"""

def __init__(
Expand Down Expand Up @@ -443,10 +439,9 @@ class Package:
or if it's located in a top-level :file:`zkg.index` file.

metadata (dict of str -> str): the contents of the package's
:file:`zkg.meta` or :file:`bro-pkg.meta` file. If the package has
not been installed then this information may come from the last
aggregation of the source's :file:`aggregate.meta` file (it may not
be accurate/up-to-date).
:file:`zkg.meta`. If the package has not been installed then this
information may come from the last aggregation of the source's
:file:`aggregate.meta` file (it may not be accurate/up-to-date).
"""

def __init__(
Expand Down
13 changes: 6 additions & 7 deletions zeekpkg/source.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
"""
A module containing the definition of a "package source": a git repository
containing a collection of :file:`zkg.index` (or legacy :file:`bro-pkg.index`)
files. These are simple INI files that can describe many Zeek packages. Each
section of the file names a Zeek package along with the git URL where it is
located and metadata tags that help classify/describe it.
containing a collection of :file:`zkg.index` files. These are simple INI files
that can describe many Zeek packages. Each section of the file names a Zeek
package along with the git URL where it is located and metadata tags that help
classify/describe it.
"""

import configparser
Expand All @@ -18,7 +18,6 @@

#: The name of package index files.
INDEX_FILENAME = "zkg.index"
LEGACY_INDEX_FILENAME = "bro-pkg.index"
#: The name of the package source file where package metadata gets aggregated.
AGGREGATE_DATA_FILE = "aggregate.meta"

Expand Down Expand Up @@ -112,8 +111,8 @@ def package_index_files(self) -> list[str]:
pass

for filename in files:
if filename in {INDEX_FILENAME, LEGACY_INDEX_FILENAME}:
rval.append(os.path.join(root, filename))
if filename == INDEX_FILENAME:
rval.append(os.path.join(root, INDEX_FILENAME))

return sorted(rval)

Expand Down
4 changes: 2 additions & 2 deletions zkg
Original file line number Diff line number Diff line change
Expand Up @@ -2841,8 +2841,8 @@ def argparser() -> argparse.ArgumentParser:
"--aggregate",
action="store_true",
help="Crawls the urls listed in package source zkg.index files and"
" aggregates the metadata found in their zkg.meta (or legacy"
" bro-pkg.meta) files. The aggregated metadata is stored in the local"
" aggregates the metadata found in their zkg.meta"
" files. The aggregated metadata is stored in the local"
" clone of the package source that zkg uses internally for locating"
" package metadata."
" For each package, the metadata is taken from the highest available"
Expand Down
6 changes: 3 additions & 3 deletions zkg.config
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ state_dir =
# A subdirectory named "packages" is always created within the
# specified path and the package manager will copy the directory
# specified by the "script_dir" option of each package's zkg.meta
# (or legacy bro-pkg.meta) file there.
# file there.
# If left blank or with --user this defaults to <state_dir>/script_dir.
# In Zeek-bundled installations, it defaults to
# <zeek_install_prefix>/share/zeek/site.
Expand All @@ -47,7 +47,7 @@ script_dir =
# A subdirectory named "packages" is always created within the
# specified path and the package manager will copy the directory
# specified by the "plugin_dir" option of each package's zkg.meta
# (or legacy bro-pkg.meta) file there.
# file there.
# If left blank or with --user this defaults to <state_dir>/plugin_dir.
# In Zeek-bundled installations, it defaults to
# <zeek_install_prefix>/lib/zeek/plugins.
Expand Down Expand Up @@ -79,7 +79,7 @@ default = https://github.com/zeek/package-template
[user_vars]

# For any key in this section that is matched for value interpolation
# in a package's zkg.meta (or legacy bro-pkg.meta) file, the corresponding
# in a package's zkg.meta file, the corresponding
# value is substituted during execution of the package's `build_command`.
# This section is typically automatically populated with the
# the answers supplied during package installation prompts
Expand Down
Loading