A problem arising from milter-regex. This fails to accept known-good regular expressions, directly taken from a working i386 system.
I believe the problem lies in the regex library, as a test program fails to compile regular expressions that contain backslashed special characters:
The salient chunk of my test program is regex_t re; if( regcomp( &re, argv[1], REG_ICASE ) ) { printf("bad re\n");
which works on "simple" things: # ./a '123' 'abc123def' re <<123>> string <<abc123def>>
matching:- <<123>>
but fails on \s and \t etc: # ./a '\s' 'abc def' re <<\s>> string <<abc def>>
bad re
although this also works # ./a '' 'abc\def' re <<\\>> string <<abc\def>>
matching:- <<\>>
(test program takes the re and a test string as its two args)
I'd not be surprised if this is another char <==> int problem, but the regex stuff is a tad more complex than spfmilter was.
Can anyone check this out please?
Didn't find your answer? Ask the community — no account required.
L
Lew Pitcher
In general, it would be helpful to know /why/ regcomp(3) disliked a given regex. Try using regerror(3) [1]. Something like this (caution: code neither syntax checked nor tested) ... regex_t re; int regcomp_rc;
if(regcomp_rc = regcomp(&re, argv[1], REG_ICASE)) { char regcomp_error[256]; /* or some other large-enough size */
regerror(regcomp_rc,argv[1],regcomp_error,sizeof regcomp_error); printf("bad re: regcomp() = %d (%s\n",regcomp_rc,regcomp_error); /* ... other error handling as required */ } could give you a better idea of why regcomp() didnt like a given regex.
re_format(7) [2] gives a list of handled backslash-escaped sequences, and '\t' isn't one of the handled sequences. Given that, regex(7) says that an atom may be "... a '' followed by any other character (matching that character taken as an ordinary character, as if the '' had not been present) ..."
So, it looks like regcomp() /should/ handle your test case here.
It looks to me like the regcomp(3) backslash-handling logic may be rejecting anything that doesn't match it's list of handled characters (although it /should/ handle your '\t' as 't', according to the docs).
[1]
formatting link
formatting link
HTH
C
Christian Weisgerber
I think your arm64 system is at a different revision of FreeBSD than your i386 one.
What are "\s" and "\t" supposed to mean? In traditional regular expressions, they have no meaning. In that case, the '' used to be ignored, i.e., they were equivalent to plain "s" and "t".
However, that was changed in this commit...
regex(3): Interpret many escaped ordinary characters as EESCAPE
formatting link
... so such sequences would now result in an error.
Subsequently, some GNU extensions have been added that give new meaning to "\s" but not to "\t".
M
Mike Scott
.....
Thanks for the responses.
Firstly, I have to confess to some history: way, way back, I modified milter-regex to use pcre rather than libc's regex routines. That's probably why my patterns still have \s strings and the like: these are valid in pcre as \s -> whitespace and \t -> tab etc.
That said, the "proper" package code from freebsd was reinstated several years ago, and both systems (i386 and arm64) are running the same packaged version 2.7.2. (It means my re's certainly haven't worked for a while, but that's a separate issue: ooops!) I have the exact same milter-regex config file on both machines.
On the arm64 box (fbsd 13.0), I get logged: parse_ruleset: /usr/local/etc/milter-regex.conf:196: regcomp: ^\s*Fwd.?\s*$: trailing backslash (\)
As has been pointed out, \s may not mean what I wanted it to: but is nevertheless valid, and that re should be accepted as equivalent to ^s*Fwd.?s*$ (man re_format is unambiguous on this)
On the i386 (running fbsd 11.4), the file compiles happily in full. Hence my supposition about errors in the regex library.
The error returned in my test code on the arm64 from regcomp() is 5 (REG_EESCAPE). On the i386 I get
% ./a '\b' '123abc4 56' re <<\b>> string <<123abc4 56>>
matching:- <>
and on the arm64: root@kirk:/usr/plumtree/config/milter-regex # ./a '\b' '123abc4 56' re <<\b>> string <<123abc4 56>>
bad re 5
Hmmm. I'm wondering about char's and int's. It's been a long, long while since I looked into the depths of Henry Spencer's original code (that on a Sun): I have a vague recollection of liberties being taken with them but IMWBW.
FWIW the test code, hacked from elsewhere, is
#include <stdio.h>
#include <regex.h>
#include <stdlib.h>
#define MAXMATCH 100
int main(int argc, char *argv[]) { regex_t re; regmatch_t matches[MAXMATCH];
printf("matching:- <<"); int p; for( p = matches[0].rm_so; p < matches[0].rm_eo; ++p ) printf("%c", argv[2][p]); printf(">>\n");
}
C
Christian Weisgerber
Again: This is an intentional change in behavior that was at some point introduced in FreeBSD's libc regex code.
Specifically this commit, which is in 13.x but not in 11.x:
formatting link
Here's full commit message:
------------------->
regex(3): Interpret many escaped ordinary characters as EESCAPE
In IEEE 1003.1-2008 [1] and earlier revisions, BRE/ERE grammar allows for any character to be escaped, but "ORD_CHAR preceded by an unescaped <backslash> character [gives undefined results]".
Historically, we've interpreted an escaped ordinary character as the ordinary character itself. This becomes problematic when some extensions give special meanings to an otherwise ordinary character (e.g. GNU's \b, \s, \w), meaning we may have two different valid interpretations of the same sequence.
To make this easier to deal with and given that the standard calls this undefined, we should throw an error (EESCAPE) if we run into this scenario to ease transition into a state where some escaped ordinaries are blessed with a special meaning -- it will either error out or have extended behavior, rather than have two entirely different versions of undefined behavior that leave the consumer of regex(3) guessing as to what behavior will be used or leaving them with false impressions.
This change bumps the symbol version of regcomp to FBSD_1.6 and provides the old escape semantics for legacy applications, just in case one has an older application that would immediately turn into a pumpkin because of an extraneous escape that's embedded or otherwise critical to its operation.
This is the final piece needed before enhancing libregex with GNU extensions and flipping the switch on bsdgrep.
[1]
formatting link
PR: 229925 (exp-run, courtesy of antoine) Differential Revision:
formatting link
<-------------------
D
druck
I'm not sure how many decades ago you are claiming for traditional reg ex, but \s and \t have been any white space and tab for a long as I can remember.
---druck
L
Lew Pitcher
Yah.... no.
In POSIX regular expressions, neither \s nor \t have any documented "special" meaning; for BREs, "The interpretation of an ordinary character preceded by a backslash ( '' ) is undefined, except for: * The characters ')', '(', '{', and '}' * The digits 1 to 9 inclusive (see BREs Matching Multiple Characters) * A character inside a bracket expression" and for EREs, "An ordinary character is any character in the supported character set, except for the ERE special characters listed in ERE Special Characters. The interpretation of an ordinary character preceded by a backslash ( '' ) is undefined." where, ERE Special Characters consists of a handful of punctuation characters, and no alphabetics [1].
A common implementation of the POSIX regular expression parser defines a regular expression atom, in part, as "..., a '' followed by one of the characters "^.[$()|*+?{\" (matching that character taken as an ordinary character), a '' followed by any other character (matching that character taken as an ordinary character, as if the '' had not been present), ..." [2]
In neither case do either \s or \t have any "special" meaning.
OTOH, Perl-compatable regular expressions recognize \s and \t as having special meanings, with \s meaning "any white space character", and \t meaning "tab (hex 09)"
It is worth noting that the OP was asking about POSIX regular expressions, as handled by the POSIX regcomp(3) interface, and /not/ pcre regular expressions.
HTH
[1]
formatting link
[2]
formatting link
A
Ahem A Rivet's Shot
They are in many places (including pcre) but not re_format(7).
A
A. Dumas
Not only pcre, also enhanced or extended. I don't have FreeBSD here but this is from the MacOS man page which is based on BSD: (conclusion below that)
----- ENHANCED FEATURES When the REG_ENHANCED flag is passed to one of the regcomp() variants, additional features are activated. Like the enhanced regex implementations in scripting languages such as perl(1) and python(1), these additional features may conflict with the IEEE Std 1003.2 (``POSIX.2'') standards in some ways. Use this with care in situations which require portability (including to past versions of the Mac OS X using the previous regex implementation).
For enhanced basic REs, `+', `?' and `|' remain regular characters, but `\+', `\?' and `\|' have the same special meaning as the unescaped characters do for extended REs, i.e., one or more matches, zero or one matches and alteration, respectively. For enhanced extended REs, back references are available. Additional enhanced features are listed below.
Within a bracket expression, most characters lose their magic. This also applies to the additional enhanced features, which don't operate inside a bracket expression.
Assertions (available for both enhanced basic and enhanced extended REs) In addition to `^' and `$' (the assertions that match the null string at the beginning and end of line, respectively), the following assertions become available:
[...]
Shortcuts (available for both enhanced basic and enhanced extended REs) The following shortcuts can be used to replace more complicated bracket expressions.
[...] \s Matches a space character. This is equivalent to `[[:space:]]'. [...]
Literal Sequences (available for both enhanced basic and enhanced extended REs) Literals are normally just ordinary characters that are matched directly. Under enhanced mode, certain character sequences are converted to specific literals.
[...] \t The ``horizontal-tab'' character (ASCII code 9). [...]
-----
So in practice it turns out that, using the built-in BSD-based grep on MacOS without any flags, it does support both \s and \t.
L
Lew Pitcher
[snip]
From the OP OP> The salient chunk of my test program is OP> regex_t re; OP> if( regcomp( &re, argv[1], REG_ICASE ) ) { OP> printf("bad re\n");
If I correctly understand the documentation you posted, to get the BSD regex "Enhanced features" that include enhanced escape parsing, the OP would have had to specify the REG_ENHANCED flag to regcomp()
I note that the OP /did not/ include this flag, nor has this flag been discussed elsethread. I also note that the OP /did not/ include the REG_EXTENDED flag, so his regcomp() will interpret the regex as a BRE.
Still, it is up to the implementation as to how it will handle the expansion of those escaped characters that the POSIX standard leaves undefined (in BRE only; they are well-defined in ERE, which the OP is not using).
M
Mike Scott
Thanks to everyone for comments.
I'll note my own error in accidentally trying to use pcre-style expressions in regex; I also take on board the changes being made: but just perhaps, the man pages should also be kept in step with the code -- and incompatible changes like that noted above perhaps merit a warning in a large flashing fluorescent font for some years.
Meanwhile I've removed the offending RE's from the milter's config file, and it runs happily. Whether it runs /correctly/ remains to be seen.
Thanks again.
Join the Discussion
Have something to add? Share your thoughts — no account required.
Didn't find your answer?
Ask the community — no account required
Report Content
You are reporting this content to the moderators. They will look at it
ASAP.