A rather hectic week on p5p,
when it was revealed that signed/unsigned comparisons and unchecked format strings to printf and sprintf could cause serious problems in poorly written applications.
s?printfIt turned out that a nasty sprintf format string could cause havoc in the webmin application suite (a set of web scripts geared towards systems administration).
Not the kind of place you want havoc to occur.
Rafael noted that this could lead to a buffer overrun in the interpreter,
by taking advantage of a signed/unsigned conversion bug in printf (which is pretty much all hand-rolled and not the printf of the underlying C standard library),
and that the next major release will apply taint checks to format strings.
Andy questioned whether it was really possible to create a buffer overrun,
and Gisle Aas responded with a tiny one-liner:
$ perl -e 'printf "%4294967295d"' Segmentation fault (core dumped)
In a subsequent thread, Andy was rather dismayed to learn that pretty-printing a variable through a %d format string makes it lose its taintedness. In later developments, Jan Dubois pointed out that Python does not have this flaw:
>>> print "%4294967295d" % 1
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: width too big
Andy subsequently decided not to post a rebuttal to the News.com article, since, to paraphrase Nathan Torkington: "everybody fucked up", and the best that can be done is to get the fix into 5.8.8, and get 5.8.8 out the door. Nicholas Clark replied that it would take a couple of weeks, which would take us right up to Christmas time. Not the kind of time you want a Perl upgrade to occur.
Philippe M. Chiasson cooked up a patch that produced the following behaviour:
$ perl -e 'printf("%04294967294d",1)'
panic: memory wrap at -e line 1.
That patch was applied by Rafael, but Gisle still managed to punch a hole through it with sprintf "%#.4294967295b". But made up for it by fixing it. Dave Mitchell supplied a patch to fix the signed/unsigned mismatch in the printf code. Hugo van der Sanden had a minor quibble with the change in behaviour, and Nicholas provided a clearer change.
Gisle thought about patching the code and documentation for Sys::Syslog, to prevent the possibility of using %n. Ronald Kimball improved the patch with a better regular expression to strip out %n.
(Summariser's note: %n, in case you weren't aware (I had to go and look it up in the documentation), takes the current number of characters emitted so far by the format string, and stores that count in the next variable appearing in the argument list; problems occur when there is no variable to take the result).
Gisle then came back later with a patch for sprintf, to prevent constant folding from taking place. Hugo appreciated the patch, and suggested a long-term plan. (Constant folding in this context meaning something like):
perl -MO=Deparse -e '$a = sprintf "%g", 2/3' $a = '0.666667';
Which stops bad things happening when %g is replaced by %99g (where 99 is a very large number). But in general, constant folding is a Good Thing, and a concensus seems to be forming around the idea that it should be possible to back out of a constant folding attempt during compilation without killing the compile, and defer the resolution until run-time.
Andy started to look at GCC's warnings of signed/unsigned comparisons, and picked a bit of low-hanging fruit in pp_pack.c. He also heard back from Jack Louis, who reported the the initial integer overflow problem. Dave Mitchell noted that one of them had already been fixed in blead. Andy forwarded another message from Jack showing how the exploit could be brought to bear on Webmin.
Joshua ben Jore pointed to a couple of threads he wrote on Perlmonks, showing the results of the code he wrote to look for uses of printf and sprintf with non-constant format parameters.
Executive summary: the problems will be fixed in 5.8.8, and a series of patches will be made available for all the 5.8 releases.
The article on News.com http://news.com.com/2100-1002_3-5975954.html Andy Lester's call for input http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00794.html sprintf and tainting http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00809.html Andy's first approximation to a PR response http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00814.html Andy declines to respond http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00002.html Philippe's patch http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00008.html Dave's patch http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00032.html Gisle's patch http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00086.html Disabling constant folding of sprintf http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00015.html Andy's patch of pp_pack.c http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00034.html Word back from the original finder of the integer overflow http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00037.html Details of a possible exploit http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00042.html The message sent to bugtraq http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00053.html Joshua's findings http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00054.html
John E. Malmberg was having difficulty tracking down why this test file was failing on VMS, and had to resort to inserting print statements to trace what was happening. Rafael Garcia-Suarez explained that it was hard to find, because in fact it is created in 00_setup.t. Ronald J Kimball thought it rather dubious that two different test files cannot be run independently of each other (this precludes, amongst other things, being able to run tests in a massively parallel manner).
Looking in the wrong place http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00764.html
my $var = undef fails to set $var when re-runErland Sommarskog posted bug #37776 showing that a declaration and assignment of a variable to undef doesn't work when the assignment is run subsequently. It turns out that it was due to an optimisation that was, well, wrong. This behaviour, according to Robin Houston, is a side-effect of change #22520. Rafael fixed it with change #26226.
Can't go there again http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00754.html
$ENV{PATH}Nick Ing-Simmons ran into a problem with Cwd's use of grep on the list of directories in $ENV{PATH}. This usually works well, but if your PATH happens to contain automounted directories that are not there, bad things happen. Indeed, Nick's Cwd was taking minutes to load. This can be construed as an abuse of grep, because only the first result is needed, but grep, by design, will always scan the entire list it is given. Nick proposed a number of ways out of the problem.
Graham Barr suggested first from List::Util. Ken Williams said that Cwd contains lots of ancient voodoo, and because it is so low on the CPAN dependency graphs, that a foreach is probably the only wise path to take. Some patches were put forward.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00018.html
I32 for arrays on 64 platformsJan Dubois noticed that the internal structures for arrays use 32 bits for index computations, thus limiting arrays on 64 bit architectures to only 2**32 elements. An array that size would consume a non-trivial amount of memory, but Jan felt that it should be fixed in blead, even if it wouldn't start being hit by applications for some time yet. Or otherwise, paraphrasing Bill Gates, that "2**32 array elements will be big enough for everyone." The concensus seems to be to use an IV instead.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00041.html
Last week in his quest to const, Andy Lester stumbled across some redundant code that he was able to chop out. In response Yitzchak Scott-Thoennes asked whether Andy was considering investigating the regparm attribute of the gcc compiler, which indicates that the parameters of the function are to be passed in registers, for a nice speed boost. But implementing this would add considerable complexity to the codebase.
The regparm attribute http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00050.html Declaring attributes in gcc http://gcc.gnu.org/onlinedocs/gcc-4.0.2/gcc/Function-Attributes.html
Alberto Simões was writing POD assuming Latin-1, but noted that it gets mangled on a system that uses UTF-8 by default, and wondered what the correct fix was. Russ Allbery replied that the correct solution was to use the =encoding directive. POD translators that are based on Pod::Simple get this for free. Other translators including pod2man and pod2txt may not.
Worse, pod2man has difficulty in dealing with non-ASCII characters because of limitations in nroff implementations. Russ is hoping to get around to adding a switch to pod2man to tell it to "assume groff", which does not how to generate UTF-8 output.
Tels noted that the POD in blead does not contain any =encoding directives, and that it probably should. Then Sadahiro Tomoyuki started talking about EBCDIC and my head exploded.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00062.html
Philippe M. Chiasson wrote to say that the Archive of Perl Changes is now running on more powerful hardware (a shade less than ten times more powerful, if you lend any credence to BogoMIPS).
The main change is that rsync://ftp.linux.activestate.com/ became rsync://public.activestate.com/. Abe Timmerman experienced a bit of transient grief with his Test::Smoke kit, but everything was sorted out in the end.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00082.html
John Peacock released version-0.50. Much of the change involves improvements to the documentation.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00076.html
Andreas König released CPAN-1.80. Lots of new goodies, including support for sudo and new commands recent and perldoc. Now runs (again) under 5.005_04.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00105.html
1512 as of Monday the 5th. All the tickets that were opened last week were commented on, which made Robert Spier happy.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00153.html
podlators 2.00 released by Russ Allbery. The underlying POD parsing is now handled by Pod::Simple, rather than Pod::Parser. Stever Peters planned to add it the core. Tels was very happy, and showed how this would let him write custom POD paragraphs.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00760.html
Ulrich Windl filed bug report #37781 show how to make the debugger crash. Richard Foley replied with a couple of message IDs showing what the probable fix would be, and otherwise how to work around it.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00777.html
Torsten Förtsch queried a strange split feature, wondering why the trailing empty elements of the split are discarded. H.Merijn Brand explained that it was operating according to spec, and showed a snippet that let Torsten achieve the desired result.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00774.html
Redundant SvUTF8_on() calls were removed from the codebase in a couple of places, thanks to careful observation from Gisle.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-11/msg00813.html
Tk compatibility was reported broken on blead by Gisle on the 23rd of November. Andreas König traced the fault back to change #26110. The fix had already been unwound in maint, and Rafael unwound it in blead. But the bug that the change tried to fix in the first place, as Nicholas reminded us, is still there.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00009.html
arenas by SV-type work continued. Jim Cromie smoked the latest blead and more or less came up with a clean bill of health. There were a couple of compiler squawks, and one test failure that Jim had difficulty in deciding whether it was because blead was in a state of flux, or whether it was because of his patch since "monkeying with arenas affects everything."
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00035.html
and a patch to unify PL_body_arenaroots[] to a single variable:
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00059.html
Sadahiro Tomoyuki improved his XS-assisted SWASHGET patch.
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2005-12/msg00111.html
This summary was written by David Landgren. Adriano and I are moving to a Monday night publishing schedule, rather than Sunday night, to give us a bit more time.
One thing I keep failing to mention in these summaries is the tireless effort that Steve Peters puts into delving into the bug queue and closing out fixed bugs and reviving the lost, the forgotten and the ignored. The number of open bugs for Perl5 has been pretty stable over the last few months (and no doubt longer, but I never paid close attention before), and this is in no small part due to Steve's diligence.
Unfortunately, as most of this activity is just one-shot messages to the list, it's nearly impossible to summarise, so casual readers of this summary have no idea of the work Steve does. So, thank-you Steve.
Information concerning bugs referenced in this summary (as #nnnnn) may be viewed at http://rt.perl.org/rt3/Ticket/Display.html?id=nnnnn
Information concerning patches to maint or blead referenced in this summary (as #nnnnn) may be viewed at http://public.activestate.com/cgi-bin/perlbrowse?patch=nnnnn
Weekly summaries are published on http://use.perl.org/ and posted on a mailing list, (subscription: perl5-summary-subscribe@perl.org). The archive is at http://dev.perl.org/perl5/list-summaries/. Corrections and comments are welcome.
If you found this summary useful or enjoyable, please consider contributing to the Perl Foundation to help support the development of Perl.