BUGS

(Fixed 1.35b) Fixed Gemini-identified bug in time string passed to
journalctl --since.

(Fixed 1.35) Finally fixed issue which required double dereference
on the action hash which had a latent bug for any configs that had
rules without defined actions (i.e., using the default).

(Fixed 1.34b) Fixed about 18 minor issues identified in Claude
(Opus 4.7) security and error review. Most significant were
an operator precedence bug in validate_request, a latent bug
in direct_read_process_acct_log and read_process_acct_log
(using global $logfile as parameter instead of $acct_log),
ensuring all bytes are read in priv_listener, fixing inconsistent
macro regex check (to permit whitespace in values in both places),
other improved edge case error checking, some debug or error
message typos, corrected comments.

(Fixed 1.33a) Better timespec validation and error reporting, fix
major bug that caused times directives to be out of alignment with
match/exclude/action triplets.

(Fixed 1.33) Require exclude directives in every rule to avoid
triplets (or now potential quadruplets with "times:" directive for
time constraints) getting out-of-sync.

(Fixed 1.32) Fix multiple bugs in process accounting handling for Linux,
including adding explicit ACK for file descriptor passing with privsep
and order of gzip/live file tests.

(Fixed 1.31d) Unpack OpenBSD commands with Z24 instead of A24 for
process accounting to avoid null-terminated strings.

(Fixed 1.31c) Remove deprecated "tmppath" pledge. Fix bug in mailcert.pl
sample script.

(Fixed 1.31a) Wrap log lines of more than 990 characters before sending
in email (RFC 5321).

(Fixed 1.29b) Strip carriage returns from log output which can be rejected
by sendmail.

(Fixed 1.28a) Edge case in session match handling causes failure after
handling a rotated log file.

(Fixed 1.27c) Use SHA256 hashes for process accounting records to mitigate
the edge case of either rapid log rotation/culling or long delays between
runs where the first available modified file is longer than the last size
but is not the same file.

(Fixed 1.27a) Fix bugs in Linux handling of checktime on process accounting logs.

(Warn 1.26d) Warn of bug in macOS perl affecting Privileges::Drop if present.

(Fixed 1.26a) Bugs in fd handling of privsep.

(Fixed 1.26) &first_log_line_sha256_digest needs to look at the uncompressed
line if it's a gzip (especially since before log rotation it likely wasn't
gzipped).

(Fixed 1.25b) Fix regexp in &identify_rotated_logs to match gzips again.

(Fixed 1.25b) Don't abort on non-OpenBSD systems with process accounting logs
if the older rotated logs don't exist yet (OpenBSD code already OK).

(Fixed 1.25a) Fail if can't create temp dir, rearrange unveil order to make
that less likely to happen in testing, improve macOS process accounting sample.

(Fixed 1.25) Fixed parsing bugs in macOS process accounting.

(Fixed 1.23a) Linux process accounting doesn't properly handle the flag/time
boundary when times have more digits than usual.

(Fixed 1.21c) Allow whitespace in macro definitions.

(Fixed 1.21b) Process accounting log handling uses getpwuid assuming
all users are defined in the passwd file.

(Fixed 1.21a) Insufficient unveiling causes failures in
include-macro-signedfile (but, strangely, caused no such issues for
<signedfile:xx> for values using the exact same syntax to Signify.pm.

(Fixed 1.20b) Aborts and complains in process accounting log processing
on a new system that doesn't yet have the max rotated process accounting
log in place yet.

(Fixed 1.20a) -c option doesn't properly handle the fact that the same
log file may be defined in different ways for different hosts -- it
either needs to run through the config separately for each defined
host or it needs to specifically check to make sure there are no
conflicts within a single host's context.

(Not fixed--likely config error.) Apparently it is possible to set up session-with/session-without rules
that result in a blank notification message. (Likely due to excess
capture group in match rules.)

(DONE) Could use general cleanup, more sensible config file parsing (e.g.,
make use of global/host/log context and restrict directives appropriately,
build some useful subroutines to eliminate duplication, have some errors
displayed instead of everything emailed.

Macro post-processing will improperly match IP addresses that contain
other IP addresses, and do substitution or appending inside the full
IP address. One way to solve (a tough one) is to have post-processing track
which macros are used in each match line and only do appending or substitution
for those, even including the context of the match from the match line,
which should be sufficient to prevent the false positive matches.
Easier quick fix is to special-case IP address macros, and when searching
for them, anchor each end against likely separators (beginning of line,
:, whitespace for the left hand side, end of line, :, ., whitespace for
the right hand side). (DONE, also uses right bracket for right hand side)

FEATURES:

* (Done 1.35a) Rename processes when privilege separation is used.

* (Done 1.34a) Move umask earlier to limit permissions on size file.
Add more comprehensive post-install message to OpenBSD package (suitable
for use with install.pl script on macOS and Linux).

* (Done 1.34) Added "all except <host list>" syntax for "hosts:" directive.

* (Done 1.33b) Added short-hand for *:MM-MM time ranges in define_time.
* (Done 1.33) added define_time for defining time ranges and times
directive for making time constraints on match/exclude rule
applicability.

* (Done 1.31) Implemented optional extra script execution action in
a single match/exclude/action triplet.

* (Done 1.30) Implemented script execution option.

* (Done 1.29a) Improve request_id format, completely mitigate replay
attacks. Make use of Signify optional (unneeded if there are no signed
macro files).

* (Done 1.29) Formalize and secure interprocess communications for
privilege separation.

* (Done 1.28) Precompile match/exclude regular expressions for
efficiency, simplify &match_line logic.

* (Done 1.27b) Read up to 64-record blocks at a time from process
accounting logs for greater efficiency.

* (Done 1.27) Add direct process accounting for Linux and macOS; using
lastcomm is still default for any OS other than OpenBSD, Linux, or
macOS and available by changing $USE_LASTCOMM = 1. (Direct use of
process accounting files could fail if formats change in future OS
releases; it will fail on Linux distros that use process accounting
record formats after acct_v3, but it works on Proxmox/Debian which
is my main use case.) Could make it a config file option if needed.
Direct processing is done non-privileged; on Linux that's with a
privileged open file descriptor passed for the first file, and
a temp file copy of the gunzipped files for the later files (privileged
process does the gunzipping).

* (DONE 1.26b) Add support for gzipped process accounting files for all
platforms (needed for Linux).

* (DONE 1.26a) Add privsep directive to config file (overrideable
by command line option to enable but not to disable).

* (DONE 1.26) Add privilege separation when run as root
  (use _reportnew:_reportnew user and group if present). root
  for config parse, temp file dir creation, journalctl actions,
  gunzips, and log file opens, _reportnew for all log
  processing including all parts of process accounting (no privs required).
  New requirements when priv sep is used:
  IO::FDPass (Linux: libio-fdpass-perl),
  Privileges::Drop (Linux: libprivileges-drop-perl)

(DONE 1.26) Set up a separate user, use privilege separation and pledge (pledge and
unveil DONE, not privsep and separate user). Fork child and drop privs
for each check_logfile call? Or pull out the root functionality
into a small component that opens the log file handles and passes
them to the child processes using IO::FDPass (in CPAN but not standard)?
Former seems simpler and could allow a bit of parallelization; there's
no central coordination required since each log generates a separate
action. (Child would need to return new sizes etc. so parent can write
it to the size file at the end of the loop. Could have a new option to
read and write size file that is for the child process to write it out
and the parent to read it in, no lock required.  Open it for writing
before dropping privs, in the temp dir.  Would need to drop privs in
check_logfile, but this won't work for cyclog/multilog, not sure about
where there are rotated logs, either.) If gunzip check/temp gunzip is moved
out of check_logfile--like process accounting, make it a log type--then
file handle could be passed in, and the open placed in the parent process
and the check_logfile done in a child process.

* (DONE 1.25b) Added more commands to macOS process accounting sample.

* (DONE 1.25b) Don't invoke shell when doing journalctl operations.

* (DONE 1.25b) Added minimal email address validation for master_notify and
notify action.

* (DONE 1.25) Add macOS process accounting sample to reportnew.conf.

* (DONE 1.23b) Add Linux process accounting sample to reportnew.conf
  appropriate for Proxmox/Debian.

* (DONE 1.23) Support Linux journal logs. (Support syslog by identifier
  or facility, and ".service" units.)

* (DONE 1.22) Support Linux process accounting logs (at least for
  Debian/Proxmox). (Done with lastcomm output, not native parsing of
  file, saving for a later effort.)

* (DONE 1.21) Move size file to /var/run/reportnew, never unveil config
  file dir for writing or file creation. OR just actually create the size
  file before locking things down, and do another pledge to remove
  cpath/unveil.

* (DONE 1.21) Add include-macro-file: / include-macro-signedfile:
   (1) Move macro parsing to a separate subroutine, that parses a line
       at a time and can accept or reject "<file:/signed-file:xx>" values
       per flag.
   (2) Use that subroutine to implement the new directive. Can have macro
       values that are included within those included macro files.

* (DONE) Option (1): Add ability to include a file of macro
definitions, which may be signed. Option (2): Replace
begin-host/end-host with "hosts: <all|<host list>>" which can be used
repeatedly throughout a config to define sections relevant to each
host.  Went with option (2); begin-host/end-host remain for backwards
compatibility.  This allows a multi-host config to be much shorter by
removing redundancy; my own config was reduced by more than 33%. The
implementation means that config parsing skips all lines not relevant
to the current host; if you want to do a check of config syntax, a new
-c (config check) option has been added which will spot errors.

* (DONE) Add ability to include files signed with a detached signify signature,
with signify pubkey file specified in config prior to use in global
section of config. .sig file is 152 chars, verification occurs at time
macro is loaded in config parsing. Verification code can be borrowed from
sigtree.

* (DONE) Handle process accounting logs directly on OpenBSD rather than via
  lastcomm. Only real obstacle is no way to call devname system call
  in perl without XSUB, so a lookup table is temporarily hard-coded
  and can be updated with data from:
     stat -f "%r,%N%n" /dev/tty*
  and a little editing.

* (DONE)  The cyclog/multilog logic for looking at prior log files should
  also be implemented for standard logfiles that have been rotated.
  
* Provide a way to write match output to a new log. (new action)
  (can be approximated with "alert" action)

* Would be nice to use IP files as macros, but not clear how to make
  a macro match an IP address (v4 or v6) that falls within a CIDR range
  referenced in a file via regexp. Requires special-casing the match itself,
  not just a macro. (Perhaps combine with next suggested feature, match-table?)
  Could be done as a pair of new types of post-processing macros, ip4/ip6,
  that would be implemented either globally or at the host level, in order
  to label--but that would be REALLY expensive; it doesn't work as a
  pre-processing macro but could be a special type of match or exclude?
  (match: ip4-file:filename / exclude: ip6-file:filename)? that would only
  run once per reportnew execution so wouldn't be too expensive.
  This isn't that useful for pf logs, where it would be more useful to
  use labels which are displayed in the log output.

* Add match-table and exclude-table, which give a filename of a lookup
  table -- extra argument that goes with match/exclude, or replaces it?
  Should tell what part of the log string to compare against lines of the
  table. Table contents should also be able to be regexps. (Would this
  be any more efficient? If not, no value.)

* Add action to re-log specific indicators from a log entry into
  an indicator log in something like Fortinet's log format, which
  identifies source host, source log, date/time from the original log
  entry, the specific indicators (e.g., source IP/port, dest IP/port,
  client IP, protocol, permitted or blocked, mail from/rcpt to/subject,
  etc. Such a log would be useful for auto-generation of block and monitor
  lists for pf. This might be a replacement for the below script idea?

* Create a separate parsing script for both pf and DNS query logs,
    which keeps a database of observed IP addresses/ports and IP
    addresses/domain queries, along with first seen time, last seen
    time, number of times seen.  Generate alerts for novelty,
    i.e., newly appearing, newly appearing after an absence, and
    appearing in multiple places/contexts for the first time (pf).

It would be nice to add a time component after the regexp. Maybe
   days:<day-of-week-list-or-range> hours:<hour-of-week-list-or-range>

I.e.:

log: /var/account/acct
match: /root/
exclude: /acceptable commands/
match: /lippard/ hours:0-5,22-24
exclude: none
notify: user@host

Better:  allow triples per log:
match: /xxx/ <time option>
exclude: /yyy/ <time option>
action: [notify emailaddr|alert]
(or notify: emailaddr for backwards compatibility)
match:
exclude:
action:
etc. Support at least ten per log.
DONE.

Maybe it would be better to just use action: execute and pass to a
script. (also DONE)

Time constraints: original idea:
Time options:
  [days: <day-list>] [hours: <hour-list>]
  <day-list>: [<day>|<day-range>][,<day>|<day-range>]
  <day>: [<dow>|<dom>]
  <day-range>: [<day>-<day>|weekends|weekday]
  <hour-list>: [<hour>|<hour-range>][,<hour>|<hour-range>]
  <hour>: 0-24
  <hour-range>: <hour>-<hour>

Data structure: Array of days, array of hours, where each element is
an individual day/hour or range, in numeric form.

Algorithm:  Match requires that the time stamp of log entry matches the
conjunction of one or more day and hour (or day and hour ranges).  It's
the conjunction of day/hour, disjunction within day/hour.  (What if I want
a day/hour conjunction?  Hmm. Should allow multiple days/hours pairs.)

Set up some examples...

match: /lippard use of sudo/ days:Sun-Thu hours:0-5,22-24 days:Fri-Sat hours:05
exclude: none
action: text user@host

Actually, don't like weekdays/weekends, as the above example illustrates,
unless "weekend" means Fri-Sat, not Sat-Sun.
(Above DONE in 1.33 with different implementation (separate times: directive,
define_time: directive)

Would also like to be able to do field matching, which requires parsing
out fields...

Note that lastcomm will take a username or commandname as an argument
and do that level of filtering for you.


Would also like to add ability to customize subject on notification:

default notify-subject: %hostname% %logfile%
notify-subject:

Make session matching work across log rotation boundaries.

Create all temp files in one temp dir, perhaps use to save data for matching across log rotation boundaries.
   All hash arrays are per-log, so could create a separate file for each match/exclude/action rule per log.
   Need to carry the session_match_array across, as well as the notify_arrays.

three conditions for needing temp dir:
(1) process accounting logs (known from config) [currently single file -- should put it in the temp dir though]
(2) session matches (known from config) -- now no longer thinking this needs temp dir
(3) rotated logs (known before check_logfile invocation)  &gunzip_logfile, &cleanup_gunzip_temp

Session matching as implemented:

match: session-with /[a-f0-9]{16} smtp/
exclude: session-without /([a-f0-9]+) smtp failed-command/
action: notify ...

will collect the match: session-with lines, and will collect the matching group from the exclude line, and only notify on what matches both.

Other things I might want to be able to do:
Match all lines from one match to another?  match: start-with / exclude: end-after?

Definitely want to be able to do this matching across log rotations.
Two options:
(1) Rebuild implementation of log rotation and cyclog/multilog handling to not require multiple calls to check_logfile (i.e., do it internally).
(2) Save the relevant arrays (notify_arrays, session_match_arrays) to a file, hold off on actions to the next check_logfile invocation (i.e., add an option to check_logfile
    to save the data and restore the data, and not do actions until the last call).
(3) for the current kind of session match, it would be sufficient to grep or zgrep one prior log file [keeping in mind that the current logfile might be a rotated one] for
    the collected matches, without bothering to save any information across invocations of check_logfile. This is probably simplest -- pass a flag that indicates to do this.

It might be best to build log objects that track multiple names, rotation format and type, etc.

