This Fortnight on perl5-porters - 12-25 December 2005

This Fortnight on perl5-porters - 12-25 December 2005

Perl 5 turns 18 and is given some lovely birthday presents, such as Perl 6's switch, smart match and say. Constants get smaller, faster and cheaper. So with all that and more, it's probably time for 5.9.3 to escape from the lab.

Patching sprintf, all the time, everywhere

The sprintf patch effort proceeded smoothly this week. Andy Dougherty phoned in to give the patching of Solaris 8 a clean bill of health. But Nicholas Clark wanted to get the other segmentation violations nailed down as well, and suggested a new set of patches. Which didn't work on Windows, and so the set was refined.

Dominic Dunlop reported perls 5.8.0 through 5.8.7 on Mac OS/X was looking good, albeit with minor unrelated warnings. Steve Hay reported success on Windows for 5.8.0, .2, .3 and .7. Nicholas published the patches on CPAN.

Gisle Aas modified the way sprintf obtains indexes and widths from the format string to check for overflows. As a result numbers larger than IV_MAX+1 (already ridiculously large on 32 bit hardware) can no longer be used directly to specify a width, but as a) such values are probably not used in practice and b) there are still indirect means of specifying Very Large Widths (such as %*d), it's not much of a loss.

More on funny file names

Last week, Craig Berry pointed out an annoyance with Pod::Simple: its test suite contains directories with names such as lib/Pod/Simple/t/test^lib. This is a hassle on VMS, because circumflex is not a legal character for a filename, and he hoped that the directories could be renamed by changing the circumflex to an underscore.

This week, John E. Malmberg pointed out other problems of filename portability. Nicholas Clark reported that the directories had been renamed to something sensible, but was now struggling with trying to get Perforce to delete the now-empty troublesomely^named^directories. Rafael came to the rescue with the correct incantation.

strict 'refs' doesn't apply inside defined

Nicholas Clark noticed that the following snippet produced no error:

  use strict;
  $a = "Not documented"; print defined @$a;
  # prints nothing, no error message
  $a = "Not documented"; print defined %$a;
  # prints nothing, no error message

... which is certainly a bug, and was worried that production code might be relying on this misbehaviour. Since applying defined to arrays or hashes is vaguely deprecated, Rafael Garcia-Suarez thought that it wasn't very important, and probably some odd side-effect of the optimiser. Robin Houston came to the heart of the problem by showing that that defined() doesn't autovivify stash entries when used via a symbolic reference:

   % perl -le '$n="z"; %$n; print(exists $::{$n} ?  "yes" : "no")'

   % perl -le '$n="z"; defined%$n; print(exists $::{$n} ? "yes" : "no")'

... and also showed where in the source this was happening.

Rafael Garcia-Suarez noticed that all the internal datatypes exhibit similar behaviour, but when he patched the necessary functions to fix it up an odd assortment of tests began to fail. One of the more icky cases was DBM_Filter, with the snippet:

  C<if (!defined %{"${class}::"})>

Rafael want to at least fix up defined($$foo), but Robin was doubtful, thinking that there's probably code Out There that relies on this behaviour. And we got back to the comment in Perl_ck_defined() that has a warning comment about not breaking Tk (This issue has been hovering in the fringes of a number of threads in the past few months).

Nicholas then posted a particularly brilliant analysis of the problem. It's well worth reading if you want to get a feel for what goes on in perl's guts

It comes down to the fact that people are not so much trying to see whether hashes (or. more precisely, stashes -- symbol tables hashes) are defined per se, but rather they are trying to determine what packages have been loaded below the current package (that is, A::B wants to know if A::B::C or A::B::D are loaded. In the current state of affairs, just thinking about A::B::C makes it spring into life. So Rafael applied change #26370 to tweak the lexer to make it all work again.

sprintf of version objects

Gisle Aas filed bug #37897 regarding how sprintf deals with version objects, as part of the fallout from the webmin/syslog debacle. John Peacock was at a bit of a loss as to where to start looking, and said that sv_vcatpvfn() was an example of overloaded operation gone wrong. After mulling over the problem for a while, John came up with a simpler approach that Gisle liked. So John went ahead an implemented it. Gisle came back with a couple of suggestions to tighten it up.

H.Merijn Brand came up with a good test case:

  # all on one line
  ($_=sprintf"%vx\n",$^V)=~s^\d\d.3\d.2^Just \
  Anoth^;s$.3..2$r P$;s;.3.;rl Hacker,;;print

At the end of the thread, John reminded us about far more than we ever wanted to know about version objects.

Pod::Man creates empty man pages

Randal L. Schwartz wrote in to say that podlators-2.0 broke an interface assumption (a parser object could not be reused) which caused a number of things (heavy lifters like pod2man, ExtUtils::MakeMaker:MM and Module::Build) to fail. Russ Allbery said that pod2man was fixed in the podlators distribution, but the main reason is Pod::Simple objects are single use once only. The new maintainer is interested in resolving the issue, but the solution is not simple.

Ken Williams was ready to patch Module::Build, but pointed out that if someone was using M::B or EU::MM and after upgrading Pod::Man noticed that their man pages stopped working, then their first reflex would be to see about seeing what was wrong with Pod::Man. It might not be obvious that the correct fix would be to upgrade M::B or EU::MM. So at some point in time Pod::Man does need to do The Right Thing.

Calling perl's guts simultaneously from many threads

Every once in a while, someone pops into p5p with a very (on the surface) simple question that turns out to be fiendishly complex. Mark Glines won this week's award for the best question.

It turns out that Mark has written a Perl XS wrapper for the Fuse project.

Fuse, as I understand it from Perl's perspective, is that you just use Fuse and you create a filesystem whose entries can be mapped to Perl routines. The Perl interface, until now, has been a single-threaded implementation (mainly because it's been around since 5.6.1). Mark recently looked at 5.8's thread support and decided it looked much better, and set about investigating how to make a multiple-threaded version of Fuse. He had a number of questions.

Nick Ing-Simmons dived in with a number of detailed responses to Mark's questions, explaining how perl's threads behave. In fact, with a bit of help from Nick and Yitzchak Scott-Thoennes, Mark was well on his way to getting it all work quite well.

The Ponie Pump(?:king|queen) chair is up for grabs

Two and a half years ago, Fotango began sponsoring the Perl 6 development effort, in the shape of Ponie, or an implementation of Perl 5 on Parrot. Nicholas worked on the project during his stay at Fotango, but he's now moved on, and Ponie is looking for a new manager. If you're interested, drop Nick and/or Jesse a line at

Eliminating null macros.

Nicholas wondered whether it would be worth reducing the use of macros by changing things like Nullsv to (SV*)0, the reason being that it's three things that people have to remember, and three things that newcomers have to learn.

Andy Lester thought it might be better to replace the Null[a-z]+ variants with NULL. Jan Dubois remembered a case where it does matter: when using vararg-style functions, H.Merijn pointed out that the explicit casts gives the compiler something bare NULLs don't: hints about alignment. Specifically, on some architectures (ok, HP) the following is safe:

  long l; *((char *)&l) = 'a';


  char a[160]; *((long *)a) = 100L;

may result in phase-of-the-moon errors. Nicholas applied a couple of experimental patches to see what it would look like.

PeekMessage() call in win32\win32.c win32_async_check fixed

Following on from last week's thread about alarm() on Win32, Jan checked in some code to address the problems. Rafael had a bit of trouble with the associated test suite, and it was all sorted out in the end

Change 26165 broke ext/threads/t/stress_re.t test on Win32

Steve Hay observed that somewhere between changes 26150 and 26185, changes to sv.c caused the above test file to fail. After doing a binary search among those change, he pinned the problem on change 26165, but couldn't see what in the patch was causing the problem.

Nicholas and Steve went over the parts of 26165 and began to apply them individually, looking for mistakes. It turned out to be a trailing comma at the end of a very heavily-casted subtraction.

After a lot of patient untangling of the initial patch, Nicholas got it all sorted out. A fascinating case of debugging by mail. All fixed up in 26378.

Reducing ithreads overhead

Brendan O'Dea told of how Debian now builds a threaded perl by default, and one of the biggest complaints is the slowdown that that induces, and worse, segfaults. A couple of ideas where discussed, but no firm conclusions were made.

Smart match is done

Robin Houston wrapped up his work on the new smart match operator, the ~~ operator from Perl 6. The patch was over 100 kilobytes. Nicholas carefully applied two parts of it, that corrected a couple of spelling errors in comments. which tickled Robin no end.

After having looked more carefully at the patch, Nicholas decomposed into six discrete components. Rafael was a bit dubious about the use feature approach to implementing smart match (and switch, and say)... all of which are provided by this patch. He thought it a pity that a pragma was required to get at recent coolness added to the language, but backwards compatibility being what it was, that it was the best way forward.

Yves Orton ironed a few kinks out the patch to get it running cleanly under threads and Windows.

Robin later followed up with another version of the patch incorporating Yves's changes and a tweaked version of smart match semantics (subject to more possible semantic changes when Larry and/or Damian touch base). Applied by Rafael as change 26400 and a tweak in 26416.

Slimming down constants

Nicholas Clark noted that, despite optimisations and inlining, constants take up a lot of memory:

  $ perl -MDevel::Size=size -le 'sub c () {3}; print size(\&c)'

... and wondered whether there was a viable way of reducing their size. One approach would be to store the scalar value directly, but that would require some collusion between the optimiser and the pp_entersub opcode. Over seven years ago, Ilya Zakharevich produced a patch to bring the cost of a subroutine declaration from 500 bytes down to 100 (change 965), one possibly non-documented side effect of which is:

  $ perl -lwe 'sub foo {}; sub bar; sub baz ($); \
    print "($_)" foreach $::{foo}, $::{bar}, $::{baz}'

Nicholas wondered whether it would be worth replacing this mechanism by another whereby something else than a typeglob could appear in symbol tables and could be used for storing constants. For modules sych as Socket and Fcntl that implement constants internally, it would become possible to generate true inlineable constants at require time, rather than at run time.

The main stumbling block is that as a constant is secretly prototyped as () behind the scenes, the trick is figure out how to distinguish between a true prototyped routine and a constant value.

After some more legwork, Nicholas discovered that pp_entersub can be fed just about any old thing, which would therefore require a non-trivial amount of cooperation from higher-level ops. And the worst of it is that as a general rule, most constant subs are never actually called in any particular program.

The new scheme that Nicholas came up with is quite clever. Now, when a reference to a constant value is stored in the symbol table, like

  $foo::{bar} = \3;

It stays that way (occupying only as much space as a reference) until it gets called upon. At that point, it springs into sub foo::bar () {3} and can be inlined as needed. What is more, any constant-building mechanism that looks like this get the optimisation for free. But wait! there's more! The first time Nicholas took the new code for a test drive, he picked up an error (a missing semi-colon in Tie::File).

Benchmarking the new technique revealed that 1% less memory is consumed when Fcntl is required and a massive 12% less when used. And all its constants are now inlined. The POSIX shows even better improvements (which will put to rest the meme about writing use POSIX () to avoid importing the universe). Bonus!

Even better still, the memory saved is that much less memory needed to be cloned when threads are created. A truly momentous change.

Later on, Nicholas figured that it should be possible to teach Devel::Peek how to dump out the value of constant subroutines, and committed a patch (26404) to do just that. Dave Mitchell liked the output.

//g loops infinitely on tainted data

Steve Peters revived an old bug (#8262) where Kirrelly "Skud" Roberts noticed that the follow snippet went into an infinite loop when tainting is active

  while($ENV{LANG} =~ m/(.)/g ) {
    print "saw '$1'\n";

Sadahiro Tomoyuki noticed that it occurs with certain, but not all, tainted values. For instance $ARGV[0] loops, but its copy $copy = shift does not. Mike Guy reasoned that bad magic was causing the trouble, no doubt some interaction between magic and taint causing pos() to be reset.

This last clue was sufficient for the light-bulb to go off over Dave Mitchell's head, and create a patch that fixes the problem.

Perl is 18 years old

Dave Mitchell noted that Perl is now old enough to drink hard liquor, vote and join the army. Randal Schwartz remarked that that's a long time to be printing "Hello, world". Joshua Juran recommended that Randal purchase a book called, um, "Learning Perl", currently in its third edition. brian d foy suggested that Joshua buy a new copy, since it's now in its fourth edition.

Plugins for Lint

Joshua ben Jore sent in a patch against blead to teach B::Lint to accept plugins. The particular scratch he wanted to itch was the ability to check for unchecked system calls and non-constant sprintf formats. Rafael wondered whether Joshua planned to backport the current B::Lint checks as plugins in the new regime. Joshua wasn't convinced of the value in doing so, to which Hugo van der Sanden pointed out that it would provide a good set of examples to show how future plugin authors could write their own. But as the plugin mechanism has a certain overhead, the cost would be too great to convert everything over.

Making the sort pragma lexically scoped

Robin Houston thought that the use sort pragma (as in use sort '_quicksort', use sort _'mergesort') should be lexically scoped. And having a couple of moments to spare after having written switch and other goodies, whipped up a patch to do just that. As a result he had to rewrite part of the test suite which was relying on the previous behaviour, but that no-one minded that.

Nicholas did raise a concern about pragmata namespaces, which Robin addressed in a followup patch. All applied by Rafael.

Robin also supplied a patch to test for %^H (the hints hash) being propagated into eval.

Is perlivp too fussy?

Gisle Aas doesn't like the way perlivp (Perl Installation Verification Procedure) moans about incorrectly installed *.ph files, which isn't really a problem under ActiveState Perl, and supplied a patch to silence these warnings.

Rafael noted that perlivp doesn't play nicely in this day and age of package-managed operating systems, and that Mandriva doesn't bother distributing it. Gisle thought that another solution would be to get rid of it altogether. Michael Cummings (who does Perl on Gentoo) voted for either removing it, or at least updating it so that it at least prints out a description of each test as it runs.

Should err be a feature?

Up until now, err has been implemented as a weak keyword, which in a nutshell means that it operates as a keyword only if a subroutine of the same name has not been declared previously. Now that Robin has introduced the feature pragma, he wondered whether it would not make more sense to recast err as a feature.

In case you haven't been following Perl 6 too closely, err is to // (defined or) as and is to &&. Rafael quite rightly pointed out that since // isn't a feature, (because it doesn't clash with anything apart from an obscure regexp construct) it seems odd to require err to be featured. The crux of the matter is the trade-off between backwards compatibility and using new, ah, features.

Also proposed was the idea of use feature ':5.9.3', use feature ':5.10' and use feature ':all' as ways of getting everything new that the interpreter has to offer, with possibly various degrees of pain inflicted on getting your code to compile.

Sébastien Asperghis-Tramoni fired up his own private gonzui search tool, and discovered that 69 distributions on CPAN currently define sub err.

Hot on the heels was the need for a command line switch that would have the same effect, in order to err, say or smart match in a one-liner. Since the best candidate (-f) has recently been used for another purpose, Robin posted the list of unused command-line switches

  b E g G H j J k K L N o O q Q r R y Y z Z

... and wondered which one would be the best for this purpose. Abigail suggested -6 only not really. More interesting was Abigail's idea to just enable the whole lot when -e is used, since in a one-liner it would be highly doubtful that err ever needs to be diambiguated, or else -e uses all features, and -E uses none.

Then it started to get really silly, with Tels suggesting to use æ (ae ligature).

In a followup post, Robin produced a patch that does the reverse of Abigail's sensible idea: that is, -E is like -e with the addition of all features.

Time to release 5.9.3?

After the discussion featuring err, Rafael thought that with the addition of all these new shiny, erm, features, it must be time to roll out a new version of bleadperl to give it some wider exposure. Some things that need to be done are: get feedback from Larry and Damian on smart match, better (read: more) tests for modules like DBI and Tk and implementing feature 'err'.

Robin promised to wrap up the documentation for the work he's done so far, on the understanding that semantics may change from here to 5.10, and Rafael said that it might be nice to try for 1/1/2006 as a release date. Robin checked in some documentation for switch and smart match a few days later.

A long discussion followed revolving around err, and whether it could be possible to promote it to a full keyword. Rafael disliked the idea, because it would introduce source-level incompatibilities,

Gisle proposed cutting the Gordian knot by just simply removing the err keyword altogether. H.Merijn said that this would remove a certain amount of orthogonality from ||/or and &&/and. Yves thought that err was a pretty dumb name, and put forward dor as an alternative. Randy W. Sims concurred, and thought that default would be better than err. But as it comes from Perl 6, it appears we don't have that luxury.

Yves's objection is that err sounds too much like error, as in Danger! Danger!, when it really means "I half expect this to not work, but that's ok. Another valid argument was that the err object in Visual Basic is the way you throw errors, so VB refugees are going to have a hard time unlearning it in Perl.

To summarise: just about everyone thinks that err sucks, but no-one can think of anything better.

The summariser thinks that since the purpose of the operator is to let you try out something, and if that doesn't work, get something else, then it should be called the Jagger-Richards operator, because you can't always get what you want, but you might find you get what you need.

Rafael wondered whether, with Robin's implementation of switch in the interpreter, whether it would be possible to fast-track the deprecation of Robin pointed out that the semantics of his switch are not quite the same as Damians source filter switch. In the end, though, since 21 CPAN modules (again thanks to a search from Sébastien's) already use Switch, it'll stay.

Adding feature bundles and featuring err

After having done six Impossible Things before breakfast, Robin finished up with a patch to implement feature bundles, as discussed above.

Should occurences of I32 be replaced with IV or STRLEN?

After having started to look at the array index issue, Jan Dubois noted that there are many other cases where signed/unsigned types are used inconsistently.

Worse, in the case of 64-bit builds, some temporary copies of things get shoved into hard-coded 32-bit variables, which admits the possibility of nasty wrap-around errors.

Rafael wondered what impact the proposed change would have on the recompilation of XS modules, and other checks that might cause slowdowns.

New Modules

Ken Williams uploaded a beta PathTools (3.14_02). This incorporates the "stop at first directory in PATH" fix that was giving Nick Ing-Simmons grief a few weeks ago. If all goes well, a 3.15 will be out shortly.

Perl5 Bug Summary

Steve Peters closed out at least two bugs that I was paying attention to, 7567 and 23907. And 23357 and 27319 as well.

Robert Spier noted that in the past seven weeks, all bugs have been replied to. 1511 open tickets as of December 19.

  The list

In Brief

Sorting within a sort does not set $a, $b was filed as bug #36430 back in June 2005. Robin (Houston, I suppose, the RT interface isn't clear on the matter) poked at it in late October. Steve Peters applied a change to fix it, only to realise that it had already been fixed by change #25953.

Jarkko Hietaniemi followed up on his valgrind findings from last month and October with a short snippet that demonstrates ampersandy leakage:

  print eval '$&';

A problem using local with threads::shared (bug 37671) inched forward a bit closer to resolution when Steve Peters realised that a more restrictive approach was needed when dealing with magic and Yitzchak agreed with the idea.

Jim Cromie refined his work on arenaroots.

Thanks to last week's summary, Andy Lester realised that he had let a message from Jim slip, and answered Jim's questions.

Vadim Konovalov noted that s{}{goto uncool;uncool:;}e produces the error Can't find label uncool at -e line 1, but that s{}{do{goto uncool if 1;uncool:;}}e (wrapping the inner statement in a do block) works. After a moment of astonished silence, Chip Salzenberg said that this was perhaps the most evil thing he'd ever seen.

Rafael discovered that the INSTALLSCRIPT doesn't follow changes to INSTALLDIR and that it's probably a bug. No-one complained about possible backwards compatibility breakage, but for the time being things are staying as they are.

Andy Lester announced the Sys::Syslog and sprintf fixes on the Perl foundation blog:

Jerry D. Hedden posted bug #37946 concerning refaddrs of threads::shared variables. Rather than staying constant, it flip-flops between two addresses. Nicholas Clark explained why it was so. Dave Mitchell scuffed his toe, looking for tuits.

See also his other bug, #37919

Yves Orton supplied a patch to fix the script embedded in patchlevel.h in order to make it work on Windows.

Andrew Savige started tracking down perl memory leaks with valgrind and noticed a leak with constant subs with prototypes and asked for some tips on how to proceed. Andrew, meet Jarkko. Jarkko, meet Andrew.

Nicholas looked at how Perl_yylex tokenises version strings and couldn't see how a certain piece of code should be reached. Robin Houston supplied a one-liner to show how it could, and Nicholas thanked him for the insight, noting that there now remained only 3000-odd lines of similarly incomprehensible code.

About this summary

This summary was written by David Landgren. Sorry about the delay, Santa ate all my tuits. No doubt Adriano will do a much better job next week. This summary took me far too long to write, and in the interest of matrimonial harmony, hasn't had the care it deserves applied, so the quotient of typos, wordos and other oddities will probably be higher than usual.

Information concerning bugs referenced in this summary (as #nnnnn) may be viewed at

Information concerning patches to maint or blead referenced in this summary (as #nnnnn) may be viewed at

If you want a bookmarklet approach to viewing bugs and change reports, there are a couple of bookmarklets that you might find useful on my page of Perl stuff:

Weekly summaries are published on and posted on a mailing list, (subscription: The archive is at Corrections and comments are welcome.

If you found this summary useful or enjoyable, please consider contributing to the Perl Foundation to help support the development of Perl.