Revision history for Ekorn-Pipeline Unreleased v0.3.2 2026-11-15 dump_to redacts the secrets in what it writes, whenever Text::Redact::Secrets is installed, so a dump sent to a log doesn't leak a password, a token or a DSN. It ships roadmap 2026-0038. Nothing changes unless Text::Redact::Secrets is installed: then dumps are redacted, and a module that's there but broken makes dump_to croak, with a new message. What a stage sees, and every message there was before, are as they were. - dump_to redacts with Text::Redact::Secrets' redact when it's installed, a v1.x, and passes a self-test, redacting a few made-up secrets in a dump: each secret it finds in what's written, such as a password, a token or the password in a DSN, is written as *REDACTED*. Only the dump is redacted: the value passed on is as it came. The module is looked for once, at the first dump. If it's there but fails to load, isn't a v1.x, or fails the test, dump_to croaks, on that dump and each after, rather than write a dump that may hold secrets, with a new message, "Text::Redact::Secrets ..., so dump_to won't write a dump that may hold secrets". local $Ekorn::Pipeline::Stages::Taps::DumpSecrets = 1 writes secrets as they are, read as each value is dumped; while it's true, the module is neither loaded nor tested. It isn't a dependency: cpanfile recommends it, so installers that take recommendations install it. The user guide and the reference show it. (roadmap 2026-0038) v0.3.1 2026-11-14 Sources, stages, gathers and sinks, each made afresh for a run and told how it ended, with words to make them and | to join them. Pipelines can be made with pipeline(), and named, and a function whose values can be code references can take its stages as a pipeline given last. pipeline() is one of the words in Ekorn::Pipeline::Factories, short functions that make Ekorn's types: stage, gather, source, generator with yield, sink_one and sink_many. The README's CPAN badge keeps up with releases, and the how-to says where a stage belongs. Stages::Flow has take_first, take_last and take_single, and take_if and drop_if, which filter. Ekorn::Pipeline::Tooling exports every stage, source, sink and word from one module, by tags. Ekorn::Pipeline::JSON reads and writes JSON and JSON Lines, as sources, stages or sinks, and Ekorn::Pipeline::Sugar names pipelines as the program compiles, each with a function of its name. A run's parts can be made afresh for it, and told how it ended: sources that close what they open, however the run ends; sinks that consume every value and give a result, committing only when the run is done; and gathers with an end as well as a flush, what v0.2.0 called the end function; Sources and Sinks have them ready-made, and Operators joins them with |. A part's set-up, or a package's role method, can return its parts by name, as a hash reference, rather than as a list in a set order. An iterator is an object, which can read what's left in one call, end its run early, and say how it ended, and which ends its run when it's let go. Pushing values through plain subs is several times faster, reading an iterator's values in one call a tenth faster, and a stop about three times cheaper. It ships nineteen roadmap items, 2026-0009, 2026-0010, 2026-0011, 2026-0013, 2026-0014, 2026-0016, 2026-0017, 2026-0018, 2026-0021, 2026-0024, 2026-0025, 2026-0026, 2026-0027, 2026-0028, 2026-0029, 2026-0030, 2026-0031, 2026-0032 and 2026-0034. It changes what some code sees, as the entries marked "Changed" say: a gather, an iterator, every canned stage, and the stages a pipeline gives back are objects, called as code references, not code references themselves; a gather's message names the flush and the end; a stage's error names the named pipeline it's in; the messages for what isn't a stage or a source say what one is; and a pipeline named alone, pipeline('clean'), is the pipeline itself, so its stages are numbered where it's given. Otherwise, existing code runs as before. - Ekorn::Pipeline::Sugar names pipelines as the program compiles: use Ekorn::Pipeline::Sugar pipeline => {name => [@stages]}, or an array of pairs, names each in the package that uses it, and gives that package a function of each name, which runs values through the pipeline, and, given a pipeline last, through that too. A definition can name any other in the same use, in either order. Mistakes, such as a name taken already, or a circle of names, croak as the use compiles. import_into does the same for a package given, for other modules' sugar. Sub::Util, which comes with Perl, is a new dependency. (roadmap 2026-0010) - Ekorn::Pipeline::JSON makes from_json and to_json, for one JSON document, and from_jsonl and to_jsonl, for JSON Lines: each a source or a sink given a file or a filehandle, and a stage given neither. from_json gives an array's elements, or the document itself; from_jsonl gives each line's document, read a block at a time, as values are pulled. to_json writes every value as one array, and to_jsonl a value a line. As stages, each passes on what a file would hold, so to_json and from_json undo each other, as to_jsonl and from_jsonl do. The encoder is JSON::MaybeXS's, configured by a json option, or replaced by an object of the caller's. Tooling exports all four, by a :json tag. JSON::MaybeXS is a new dependency. The user guide reads and writes JSON in chapters 5 and 6, and walks through how a module like it is put together in chapter 10. (roadmap 2026-0034) - Ekorn::Pipeline::Tooling exports what callers use to do work, from one module, so there's nothing to remember about which module makes which: tags :core (Stop and gather), :flow, :batch and :taps, :stages for the three, :sources, :sinks, :words (the factories) and :all. Each tag holds every name its module exports, as the same sub, and nothing is exported by default. It leaves out is_stage, is_sink and croakf, which are for extending the package, and the operators, which stay a pragma. The user guide's examples, the README's and the reference's At a glance import from it. (roadmap 2026-0031) - The reference has a section, Making each kind of part, for developers: a table of what each piece of a part, its set-up, step, flush, end and context, is called with, returns, and whether it may stop the run, and a worked example of each kind of part, made with a word and as a class, from a stage with state to an object that's both a source and a sink, committing only a run that went through. Every example in the README, the user guide, the reference and the modules' POD writes a pipeline of two or more stages a stage to a line, and paragraphs are filled evenly again. - The role methods, by which any object can be a source, a stage or a sink, are named in capitals, EKORN_SOURCE, EKORN_STAGE, EKORN_SINK_ONE and EKORN_SINK_MANY, as Perl names subs that its runtime calls, not their caller, such as DESTROY, and as CPAN names hooks for another package, such as TO_JSON. No release had them in lower case (roadmap 2026-0027). - A set-up, a gather's, a source's or a sink's, and a role method, EKORN_SOURCE, EKORN_STAGE, EKORN_SINK_ONE or EKORN_SINK_MANY, can return its parts by name, as a hash reference with the keys step, flush, end and context, giving only those it has, in place of a list in a set order with undef for one left out before one given: gather { {step => $step, end => $end} }. Other keys are ignored, and a hash without what its part needs croaks, as a list does. The list still works. Ekorn's own sources, sinks and stages, and an iterator's EKORN_SOURCE, return theirs by name (roadmap 2026-0026). - Ekorn::Pipeline::Sources has ready-made sources, needing only Perl, none exported by default: from_array(@values); from_fh($fh), a filehandle's records, as $/ divides them, chomped, left open; from_file($path, $layers), a file's records, opened through the I/O layers given, if any, for each run, and closed however it ends; and counter($start, $step), without end. With $/ undefined, from_fh and from_file slurp, an empty file giving none. (roadmap 2026-0016) - Ekorn::Pipeline::Sinks has ready-made sinks, needing only Perl, none exported by default: keep_all, every item; keep($n), the first $n, stopping there; keep_first, stopping at it; keep_single, warning at a second; keep_last; count_items; and to_fh($fh) and to_file($path, $layers), writing each item followed by $/, as from_fh and from_file take it off, and giving how many. to_file opens its file for each run, through the I/O layers given, if any, and closes it however the run ends. (roadmap 2026-0017) - Ekorn::Pipeline::Operators, used in a scope, puts |, |= and @{} in force there for Ekorn's types: $source | $stage | $sink gives the sink's result, $stage | $stage a chain of stages, $source | $stage a source, and @{ $source } every value, read through. A chain starts with a source or a stage made with a word, never a plain code reference, whose | is Perl's bitwise OR. Out of scope, they croak "Expected use Ekorn::Pipeline::Operators in scope, for %s". A package's own source class chains by overloading | and @{} with Operators' chain and drain. (roadmap 2026-0018) - Changed: a stage that isn't one croaks "Expected stage #%d to be a stage: a code reference, a pipeline, or an object offering a stage's role method", where it said "Expected stage #%d to be a code reference or a pipeline", and, given to pipeline or named, "... a pipeline or its name, or an object offering a stage's role method"; both left out the parts made for a run. A pipeline whose overridden stages method returns something that isn't a stage has a message of its own, "Expected stage #%d, a pipeline, to give only stages from its stages method", in place of one saying it should be a pipeline. Stage->new, given one of Ekorn's types, croaks "Expected a code reference, not one of Ekorn's types". - Changed: iterator, given something other than a source, croaks "Expected a source: a code reference, or an object offering EKORN_SOURCE", where it said "Expected a code reference", which was accurate only until sources could be objects. - Changed: a stage that dies within a named pipeline is named with it: after its place, the innermost named pipeline and the stage's place there, "Pipeline stage #1.2 (main::clean#2) failed at ...", and an end, "Pipeline end of stage #1.2 (main::clean#2) failed at ...". A stage of no named pipeline is named as before. A pipeline's name method gives its first name, qualified, or undef. (roadmap 2026-0013) - Ekorn::Pipeline::Stages::Flow exports two more stages on request: take_if { ... }, passing on each value the block is true of, and drop_if { ... }, each it's false of, testing every value, as grep does, where take_while and drop_while stop testing at the first value that fails. A stage ending in a test passes on true or false; take_if passes on the value. (roadmap 2026-0030) - Faster: a pipeline whose stages are all plain subs, called with values, runs them in a plain loop, with none of the machinery for flushes, ends and sinks: a call of a few values is about twelve times faster, and a call of many through one stage three and a half times. Every other run is a little faster too: a push hands its values over at once, and less is done for each value. Nothing else changes. (roadmap 2026-0028) - Faster: pulled runs, as DBIx-Squirrel measured them. An iterator's all collects every value in one pass of the run, rather than a call for each, about a tenth faster through one plain stage, and each pull copies less. Nothing else changes. (roadmap 2026-0032) - Faster: a stop, thrown with die rather than Carp's croak, which added nothing to it but cost, about three times cheaper, as DBIx-Squirrel measured it. A catcher gets the same object as before. - Changed: pipeline_in, and so the word pipeline, given one name and nothing else, returns the pipeline named itself, not a new pipeline holding it, if it's of the invocant's class or a subclass. So a stage of a named pipeline given second is "Pipeline stage #2.2 (main::clean#2)", not "#2.1.2". named on a class still makes a new pipeline, even of one name. (roadmap 2026-0029) - The message for a gather's block that returns the wrong things now ends "and optionally a flush and an end", where it ended "and optionally an end function": what v0.2.0 called the end function is now the flush, and an end, told how the run ended, may follow it. Code matching the old wording must change. (roadmap 2026-0014) - A run's parts can be made for it. Ekorn::Pipeline::Source makes a source from a set-up, called at the start of each run, which returns its step and, optionally, its end and a context; iterator takes one, or anything offering EKORN_SOURCE. A source given to new as a stage croaks. (roadmap 2026-0014) - Ekorn::Pipeline::Sink makes sinks, with new_sink_one, giving one value, and new_sink_many, giving a list: a set-up returns the step, the flush, giving the result, and optionally an end. A pipeline ending in a sink gives its result: in scalar context, a sink of one's value, or a sink of many's list as an array reference. is_sink, exported on request, says "one", "many" or false. (roadmap 2026-0014) - Each end, a source's, a gather's or a sink's, is called once, when the run has ended, however it ended, told "done", "stopped" or "died", the source's first. A run that dies calls every end, told "died", before throwing its error; an end that dies is reported as "Pipeline end of %s failed at %s line %d:", or, in a run already dying, warns "The end of %s died while the run was already dying: %s". A gather's block may return an end, third, after its flush. (roadmap 2026-0014) - new takes, as a stage, any object offering EKORN_STAGE, EKORN_SINK_ONE or EKORN_SINK_MANY, called at the start of each run, so a package can deliver parts for Ekorn without using its classes; one offering more than one croaks. is_stage is true for them. run_context gives a stage the context its run's source gave. (roadmap 2026-0014) - Changed: iterator returns an Ekorn::Pipeline::Iterator, an object called as a code reference, as before, with all, first, single, finish and ended. Let go before its values run out, it ends its run, so a source closes what it opened. An iterator can be a source for another pipeline, once. (roadmap 2026-0014) - Changed: a gather is an Ekorn::Pipeline::Gather object, called as a code reference, not a blessed code reference, and stages returns a pipeline's stages as objects: a code reference given to new comes back as an Ekorn::Pipeline::Stage of it. Code testing them with is_coderef or ref sees the change. (roadmap 2026-0014) - New messages, for sources, sinks, parts and ends, are listed in the POD's DIAGNOSTICS and the reference's Diagnostics. (roadmap 2026-0014) - The how-to, Ekorn::Pipeline::Manual::HowTo, is rewritten as a user guide. It tours the package from the bottom up, a layer at a time: stages, pipelines pushed and pulled, stopping, parts made for each run, sources, sinks, composition, the operators, functions that take stages, and parts for other packages; then errors and debugging, and the questions people ask first, such as how fast it is, how it compares with map, grep, options and callbacks, and where it falls short. Each of its examples runs, and prints what it says. - The README is rewritten for this release. It shows the shape of a run, from a source through stages to a sink, with sections on parts made for each run, sources, sinks, joining them with |, named pipelines, and parts for other packages, and lists the modules that come with it. - The manuals' tables show a | written in them as |, not \|. - The reference has a section, Runs and how they end, and another, Delivering parts for Ekorn; the how-to has recipes for sources and sinks that close and commit however a run ends, and for reading an iterator's values in one call. (roadmap 2026-0014) - pipeline(), exported on request by Ekorn::Pipeline::Factories, makes a new pipeline from stages, as new does, but a stage can also be the name of a pipeline: pipeline('clean', take(5)). It's Ekorn::Pipeline->pipeline_in(scalar caller, @stages), the one method that reads names, making its invocant's class. (roadmap 2026-0011, 2026-0021) - Ekorn::Pipeline::Factories has the words that make Ekorn's types, each one line over a class method, none exported by default: pipeline, stage, gather (Gather's own), source, generator and yield, sink_one and sink_many. generator gives a source of what its block passes to yield, as a Python generator does, collecting it all in one step; yield outside one croaks "Expected yield to be called inside a generator". (roadmap 2026-0021) - Ekorn::Pipeline->named(clean => @stages) defines a named pipeline, and $p->named('clean') names one you have. A name is kept in the caller's package unless it's qualified, whatever class defined it, is resolved when a pipeline is made, and can't be defined twice, or removed. Only pipeline() and named() read names: new doesn't, and is_stage stays false for a string. (roadmap 2026-0011) - Five new messages, for pipeline() and named(): "Expected stage #%d to be a code reference, a pipeline or the name of one", "No pipeline named %s", "A pipeline named %s is already defined", "Expected a name, made of words joined by ::" and "Expected only a name, naming a pipeline you have". (roadmap 2026-0011) - The reference, the how-to and the README show a pipeline given last as the way to take stages after values that can be code references, and how to name a pipeline you use often. (roadmap 2026-0011) - The reference has a section on the words, and its examples, and the how-to's, make sources and sinks with them. (roadmap 2026-0021) - The README's CPAN badge comes from shields.io, which keeps up with a release within minutes, in place of badge.fury.io's, which GitHub showed out of date for hours. (roadmap 2026-0009) - The how-to has a recipe, "Decide where a stage belongs": under Ekorn::Pipeline::Stages, or with the module whose values it needs to know about. Its cpanfile example asks for v0.2.0. - Ekorn::Pipeline::Stages::Flow exports three more stages on request: take_first, which is take(1); take_last, passing on the last value once there are no more; and take_single, passing on the one value, and, at a second, warning "Expected only one item, but there were more" and stopping. Each, with take($n), is the stage of a keep sink, passing on what the sink would give. (roadmap 2026-0024) - Changed: every canned stage, in Stages::Flow, Batch and Taps, is an object called as a code reference: a Gather, or, for take_while, peek and dump_to, an Ekorn::Pipeline::Stage. Code testing one with ref or is_coderef sees the change. (roadmap 2026-0024) v0.2.0 2026-11-09 Messages say a stage can be a code reference or a pipeline. It ships one roadmap item, 2026-0008. Two messages change, so it's v0.2.0: code that matches the old wording, as a string or an anchored pattern, needs updating. Nothing else changes in behaviour. DBIx::Squirrel signed this release off before it went out. - "Expected stage #%d to be a code reference" is now "Expected stage #%d to be a code reference or a pipeline". (roadmap 2026-0008) - "Expected stage #%s, made with gather, to return one or two code references" is now "Expected stage #%s, made with gather, to return a code reference or a pipeline, and optionally an end function". (roadmap 2026-0008) - The DIAGNOSTICS, the reference, the how-to and the README say the same, and no longer explain the old wording. (roadmap 2026-0008) v0.1.3 2026-11-09 A pipeline can be used as a stage, and the docs are plainer. Pipelines can be built from other pipelines without ->stages, and a gather block can return one. The documentation is rewritten in plainer English. It ships three roadmap items, 2026-0004, 2026-0006 and 2026-0007. Existing code runs as before, and every message a pipeline gives today reads as before. The one exception is a subclass of Ekorn::Pipeline used with an import list, which now croaks, as below. DBIx::Squirrel signed this release off before it went out. - Ekorn::Pipeline->new accepts a pipeline as a stage, and splices its stages in where it stands. They run in the new pipeline's run, so a take inside limits the whole run. The stages come from the pipeline's stages method, so a subclass can override it. (roadmap 2026-0004) - In error messages, a stage inside a pipeline used as a stage is named by its position in each pipeline: "Pipeline stage #2.3 failed", or "#1.2.3" three levels down. DIAGNOSTICS shows "#%s" for these messages. Other stages are named as before. (roadmap 2026-0004) - is_stage, exported by Ekorn::Pipeline on request, is true for anything new accepts as a stage: a code reference or a pipeline. Use it in a function that takes its stages from the end of its arguments, so it accepts pipelines too. Ask for v0.1.3 on the use line, as v0.1.2 lets "use Ekorn::Pipeline qw(is_stage)" compile. (roadmap 2026-0004) - Ekorn::Pipeline now has an import method, from Exporter, for is_stage. A subclass inherits it, so "use My::Subclass qw(name)" croaks unless My::Subclass exports that name; before, the list was ignored. (roadmap 2026-0004) - A gather block can return a pipeline as its stage. Its stages are made for each run in its place, gather stages inside it afresh, and the block's end function, if any, is called after theirs. A stage inside it is named as "#2.3" in messages. The message for a bad gather block is unchanged. (roadmap 2026-0007) - The README, the POD, the reference and the how-to are rewritten in shorter sentences, saying the same. Two out-of-date claims in the README are fixed: Ekorn::Pipeline is on CPAN, and DBIx::Squirrel uses it. (roadmap 2026-0006) - The README has MetaCPAN and Kwalitee badges. v0.1.2 2026-11-08 The reference and the how-to on MetaCPAN, as POD manuals. It adds documentation only: the code is as in v0.1.1, and it ships no roadmap items. - The reference and the how-to ship as POD, Ekorn::Pipeline::Manual::Reference and Ekorn::Pipeline::Manual::HowTo, made from their Markdown, so MetaCPAN and perldoc have them with the modules. v0.1.1 2026-11-08 The pipeline from DBIx-Squirrel v2.0.2, on its own, so that any function can take stages from its caller. It ships three roadmap items, settled after v0.1.0's trial: a pipeline's stages can be spliced into another, with the stages method (2026-0001); an iterator ends once a stage dies (2026-0002); and a pipeline's count in scalar context is promised (2026-0003). - Requires Perl 5.36 or later. - Ekorn::Pipeline runs values through a list of stages, each a code reference given one value, in $_ and as its argument, and returning nothing (a filter), one value (a map) or several (a flatMap). A pipeline is called with values, or pulls them from a source with an iterator, calling it only as results are wanted. - Once a stage dies, an iterator has ended, as a pushed run has: every call after returns an empty list, and neither the source nor any end function is called again. In v0.1.0's trial it carried on, without the value that died. (roadmap 2026-0002) - Called in scalar context, a pipeline returns how many results came out, as an array would; the POD now promises it. (roadmap 2026-0003) - A pipeline's stages method returns its stages, so they can be spliced into another pipeline, or given to any function that takes stages, running as if listed there. (roadmap 2026-0001) - SECURITY.md says how to report a vulnerability, privately, and CONTRIBUTING.md how to report a bug or ask for a feature; the distribution takes issues, not pull requests. - Ekorn::Pipeline::Stop ends a pipeline early, passing on any values it carries, after which the source isn't called again. A stop not made by throw, and so not an array, carries nothing. - Ekorn::Pipeline::Gather makes a stage afresh for each run, with an optional end function, called once the values run out. - Ekorn::Pipeline::Pipes gives a class pipe and tap methods, which run the object through stages. - Canned stages: take, drop, take_while and drop_while in Ekorn::Pipeline::Stages::Flow; batch and collect in Ekorn::Pipeline::Stages::Batch; peek, count_into and dump_to in Ekorn::Pipeline::Stages::Taps. - A stage that dies is named by its place, at the caller's line, and so are mistakes in making a pipeline or a canned stage. - Modules of stages can live under any namespace: general stages under Ekorn::Pipeline::Stages, and those for one kind of value or source with the module that knows about it, built from Gather, Stop and croakf, which reports at the caller's line. - Versions: a v0.1.x only fixes or adds. A change to how stages behave, or a public name dropped or renamed, waits for v0.2.0. - A how-to, docs/HOWTO.md, with recipes for callers, function authors and stage authors, and a reference, docs/REFERENCE.md, describing every module and how a pipeline behaves, with notes for stage authors. Both are on GitHub, linked from the README.