================================================== Changes from 2015-12-05 00:00:00 +0000 to present. ================================================== ------------------------------------------ version 0.006 at 2017-12-04 00:18:54 +0000 ------------------------------------------ Change: b1231e5a3f970d07f07ec4e00cc4746634293c09 Author: Brad Lhotsky Date : 2017-12-04 01:18:50 +0000 Release 0.006 with the indexers fixed and the iptables context. Change: a121f4f64caa7b60bbbe9da6b422fb7e33991799 Author: Brad Lhotsky Date : 2017-12-04 01:14:28 +0000 Fix the eris-es-indexer.pl The config was pointing to the wrong depth in the hash. Ensure when the config is passed from the commandline, eris::schemas are instantiated correctly. Fix the mapping for the geo_point field in the geoip mapping. Add the eris::log::context::iptables to parse iptables logs into the indexes. ------------------------------------------ version 0.005 at 2017-12-03 23:10:58 +0000 ------------------------------------------ Change: a4aa7aa00b3bb87e3a11f1e40fc1bd2d1d949688 Author: Brad Lhotsky Date : 2017-12-04 00:10:58 +0000 Release version 0.005 Update documentaiton to demonstrate enabling the debug dictionary in the syslog schema. Change: 537771e0cbd49dea95f9f2d8358cfa7275d5a260 Author: Brad Lhotsky Date : 2017-11-24 11:15:46 +0000 Correct the shebang line for install Change: b2090fc15bdee9533c4732afbbc73c74114051cf Author: Brad Lhotsky Date : 2017-11-24 09:34:31 +0000 Use auto-detection of MinimumPerl I incorrectly set the minimum Perl version. Removing this hard-coded config detects the correct minimum Perl version. ------------------------------------------ version 0.004 at 2017-11-16 17:43:11 +0000 ------------------------------------------ Change: b7737fd29826f94cfab9f91d02bfb142ff7a9dac Author: Brad Lhotsky Date : 2017-11-16 18:43:11 +0000 Remove App::ElasticSearch::Utilites from the prereqs and fix some bugs with the indexer. Change: ed5eb795a6a7beb21e6ce36c66b527e5b727fb40 Author: Brad Lhotsky Date : 2017-11-16 09:56:57 +0000 Remove the eris::dictionary global singleton It makes more sense to allow the schema to define it's own dictionary. Dictionaries can now be configured per-schema allowing them to be as configurable as necessary. Allow hash flattening of the documents and enable that option in eris-context.pl. Change: eea981c6695f603eccde4865581b84a689a877c5 Author: Brad Lhotsky Date : 2017-11-05 10:33:28 +0000 Regenerated README Change: 6c9167d2ed6e6f2b6b655b9a2137482791c418b2 Author: Brad Lhotsky Date : 2017-11-04 16:20:31 +0000 Catch documentation up to the current state of affairs. Change: 0d15fa7fcf4b0fad7d8f83904ce5c1edcc47dc9a Author: Brad Lhotsky Date : 2017-11-04 15:53:56 +0000 Removing the type library since I'm not using it anymore Change: 773f3a247bb14dde25e4f393e5ac6dfbf7c132e6 Author: Brad Lhotsky Date : 2017-11-04 15:53:22 +0000 Added all POD required for author tests to pass Add version tags in the modules where they were missing Add abstracts everywhere they were missing Ensure all the final POD elements were closed Change: 55293219e715ac9668c09c25afe80fa901ae917c Author: Brad Lhotsky Date : 2017-11-04 13:04:27 +0000 Fix Perl::Critic and POD syntax errors. Change: 129c0539a53e7f28f44e506c412c231a053fb76c Author: Brad Lhotsky Date : 2017-10-29 08:09:15 +0000 Fix parse and pod errors. Change: ddfe01b50fc4e2893924feb3d036ec7559059b9c Author: Brad Lhotsky Date : 2017-10-29 07:46:18 +0000 Started documenting the overall project goals and design Change: 128d6c3fc0ec36504055fcbfaa379012e0e018d4 Author: Brad Lhotsky Date : 2017-10-28 11:54:01 +0000 Documentation added to more classes. Change: 570dfb821e79bbffd202520ead08731837e28daf Author: Brad Lhotsky Date : 2017-10-28 10:47:53 +0000 Documentation added to roles * Technical debt being collected * Migrate the '_build_name' method all the way back to eris::role::plugin. Push down smarter logic for automatically determining the name of a plugin. Require a namespace parameter for eris::role::plugin that's automatically passed from the eris::role::pluggable consumer from it's required parameter. This makes naming the consumers easier and smarter. Change: 21799c506625b11dcd27696297fda40de866d865 Author: Brad Lhotsky Date : 2017-10-27 17:36:48 +0000 Allow schemas to choose not to be final. This means a single log entry can be interpretted by more than one schema. This might be useful for storing events in a large short term index, but particular events in a longer term index. Change: fdf10a5c1ad228c6555e57d858e97089baecc45c Author: Brad Lhotsky Date : 2017-10-26 23:04:28 +0000 Working out the kinks in cleaning up the separation of eris::log and the underlying elasticsearch schemas. Change: e24a07509590e27cdfa8e228bdd3a48b2e0f284a Author: Brad Lhotsky Date : 2017-10-26 19:14:51 +0000 Separation of the schema and parsing done! Change: 27201c0df79a737549e701596e710d7797521c1e Author: Brad Lhotsky Date : 2017-10-20 21:14:58 +0000 Reworking the system * Contextualizing and Storage separated so you can apply different storage rules to the same message * Working out how to mimic the ES mappings Change: 06deb277d97b92779b1539ec7a851242eedbdd73 Author: Brad Lhotsky Date : 2017-08-15 20:19:02 +0000 Store the raw message * Add the raw context and update the elasticsearch mappings to store the raw data without indexing it. * Anchor dhcpd parser * Fix protocol extraction in pfsense::filterlog Change: bc4da89bcb71189294d75cf95cfedc1ea0ec2eb8 Author: Brad Lhotsky Date : 2017-07-14 23:32:17 +0000 Add static context to add k/v pairs to every log event. Add a special "double star" matcher to match every event. Add a "SuppressWarnings" variable to the contexts. If a context wishes to be silently ignored, it can set "our $SuppressWarnings". Use both of these features in the static context. The advantage is one less subroutine dispatch if the static context isn't configured. Change: 867feb6940ef4d11b275bcd6e56acf63296d3558 Author: Brad Lhotsky Date : 2017-07-14 18:22:46 +0000 Overhaul of the reference implementations. * eris-eris-client.pl - Added options to control the flow of information. Added graphite output for statistics reporting. Currently only "dispatched" statistics work. Reads the config file for a client section, which is then passed to the POE::Component::Client::eris constructor, e.g.: client: Subscribe: [ "sshd", "sudo", "kernel" ] Match: [ "error", "failed", "failure" ] * eris-es-indexer.pl - Added support for ES versioning via the --es-version option. Defaults to '5'. Supports the following options via the config file: es_addr: a host in the cluster to index to es_default_type: Type to index message es_default_index: 'syslog' Index to write the message to TODO: eris-es-indexer.pl should autodetect the version of the cluster and apply the appropriate mapping. es_addr should accept an array Change: 385513cd4c1e713dd2565d693832960110f4ada2 Author: Brad Lhotsky Date : 2017-07-06 07:38:26 +0000 More cleanup, record pid and program sub in a CEE Compatible way. Change: 93fa0a2b5926c7d1c697f90cdf0149e26c7dff9c Author: Brad Lhotsky Date : 2017-07-05 07:37:51 +0000 Fix up tags and streamline context calls. Change: abb30c813e3589349dc66f8d89058fb08ab50471 Author: Brad Lhotsky Date : 2017-07-05 06:54:45 +0000 Fix protocol handling Normalize protocols to lowercase. Use 'proto_app' instead of 'proto' per CEE. Add 'service' of 'firewall' to the logs. Change: fe5849d08fd174325074fb84bfd0db4791da3ba6 Author: Brad Lhotsky Date : 2017-07-05 06:30:33 +0000 Added parsing for pfSense's CSV filterlog. Changes to the eris-context.pl tool to accomodate mixed case names. pfSense::filterlog can parse out IPv(4|6) and TCP/UDP meta-data Change: 9d845334e6c321fad188936fa1ba8228e8b8da49 Author: Brad Lhotsky Date : 2017-07-05 05:28:58 +0000 Added dhcpd parser. Fixed up older contexts to take advantage of better logic. Added help option to the eris-context script ------------------------------------------ version 0.003 at 2017-03-01 06:24:50 +0000 ------------------------------------------ Change: acee6050a6cae7fee3db0b8f962404302c6a8c1a Author: Brad Lhotsky Date : 2017-03-01 07:24:50 +0000 Add ability to swap index basenames and types via contexts. This patch allows a context to set a different index base name or type. This will be sent along to the cluster in the bulk api end point. Change: 63c8d264be2e86678ecbecf1262cfa53c7afde6f Author: Brad Lhotsky Date : 2017-02-02 20:54:29 +0000 Update location of GeoIP2 mmdb, rename 'desc' to 'name' for clarity. Change: 2579cad2f0a63c7a6ba4578b6844769085d822ef Author: Brad Lhotsky Date : 2017-02-02 09:26:43 +0000 eris::log->add_tags() created to allow contexts to tag messages, total_time added to allow easy searching. Change: 1307cf25da94f1b47ce7afac0f548a78f0a9de7d Author: Brad Lhotsky Date : 2017-02-02 08:40:51 +0000 Added suricata/snort log decoding to the library. Change: e18c8d66f30ec0cf1bbbdfe708ffa87a58757fb6 Author: Brad Lhotsky Date : 2016-12-16 07:47:54 +0000 Bump release in the docs. ------------------------------------------ version 0.002 at 2016-12-16 06:45:24 +0000 ------------------------------------------ Change: 503862bb34803e756ed43f25d4b887a7fcc9d039 Author: Brad Lhotsky Date : 2016-12-16 07:45:24 +0000 Fix race condition in the scheduling that was preventing bulk writes from occurring. Change: 05590cf74a82b9d5ae36af97e7a676c9cea7c60c Author: Brad Lhotsky Date : 2016-12-16 07:01:14 +0000 Postfix Context: Make sure the K/V pairs extract before trying to use them in the log context Change: d4fb6699edf3ad40b36411413bcf4ec6547a8ee3 Author: Brad Lhotsky Date : 2016-11-29 01:36:59 +0000 Update ignore list. Change: 47ab113720483941f3e2c617e00617a1fe98a8d8 Author: Brad Lhotsky Date : 2016-11-28 20:39:17 +0000 Cron support on FreeBSD, Invalid user tracking on sshd. Change: be5ead9c3f26270d43d3313370678a6e1dc9532b Author: Brad Lhotsky Date : 2016-11-26 03:19:14 +0000 CROND and postfix parsers added, dictionary fixes. Change: 936fd1c3ae01e64843c5ff1b615c8fcfea171a9f Author: Brad Lhotsky Date : 2016-11-23 21:38:10 +0000 Conversion to Moo and Type::Tiny I'm not utilizing the full Moosey stack, so I converted to Moo. If I need to use Moose for something later, the Type::Tiny conversion will make that trivial. Restructured the contexts, decoders, and dictionaries to use the eris::role::plugin to remove duplicate code. --------------------------------------------- version 0.001_01 at 2016-11-20 04:14:44 +0000 --------------------------------------------- Change: 4101a001d306c9780e0f9884c126472fc0c5268a Author: Brad Lhotsky Date : 2016-11-20 05:14:44 +0000 Ignore the dzil distribution builds. Change: 30f6b501ec302af320b1b65744421ed465c98807 Author: Brad Lhotsky Date : 2016-11-20 05:11:19 +0000 Added rudimentary README to the dist. Change: 9169cc3302c39d40b86aadaf46cd443225283e58 Author: Brad Lhotsky Date : 2016-11-20 05:10:35 +0000 Append newlines to the STDERR log messages. Change: 7f4526adf6caea02a9535d97a094fd065bf387b1 Author: Brad Lhotsky Date : 2016-11-20 05:00:21 +0000 Automatically manage index templates This patch adds a simple template checking in to the worker startup. Change: a07d8b9fdfe20e4af22803a25f48315a0f051b99 Author: Brad Lhotsky Date : 2016-11-20 03:38:55 +0000 Prerelease alpha The eris-eris-client.pl can work with a POE::Component::Server::eris server to connect and stream the logs to an ElasticSearch cluster. Stats are bubbled up from the workers to the server, and documents land in the index! Change: b4f4bc012cf0d2cb71376fce5aacf60a1a0fd5f0 Author: Brad Lhotsky Date : 2016-11-19 19:09:58 +0000 Output basic stats for testing purposes. Change: b4ceb83d54897b2ca949a2b13f1106ddf05958fc Author: Brad Lhotsky Date : 2016-11-19 19:06:06 +0000 Allow worker pools to be a configurable size. Change: a7f8a9663379711fa7b2813335e700ac982c6760 Author: Brad Lhotsky Date : 2016-11-19 18:59:56 +0000 Added eris-eris-client.pl Uses POE::Component::Client::eris to connect to a local syslog server as it's source. Change: ef7b6799ce619d9f490c0fc329144e30b17c9b14 Author: Brad Lhotsky Date : 2016-11-19 18:24:43 +0000 Prerelease Candidate 1 dzil configs updated to reflect my standard dzil layout. Renaming the scripts in the bin directory to have clearer, more appropriate names and avoid collisions. Restructure the timing data so it can be easily indexed in ElasticSearch. Convert the config file to an optional parameter so the utilities run cleanly from the the dist. Change: b0554bc33417834348579abedecb7ea202a63722 Author: Brad Lhotsky Date : 2016-11-19 17:29:36 +0000 Remove static license file. Change: 725576bc10475a79cf33017a197c0a56c60b5fcb Author: Brad Lhotsky Date : 2016-09-21 22:28:38 +0000 Delete postfix.pm This was just a placeholder, removing to remove confusion. Change: ce5b8590f3e645374af835df96a86b437d9898f7 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Catch errors and remove the input streams from the system. Bubble stats up to the main process for handling. Change: ecf1e1deb89f853e8cb92d355b14975da6b785e4 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Getting close to a releasable package. Change: 3190609932c28ae1a828a9519000b5e2443adafa Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 more stuff Change: bd85f8eff29ff4969875268e738816e6f3fa7c91 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Allow context/complete objects to be duplicated as in the case of the GeoIP context. Change: acb23a094a573a1d22b8697a4b6d571e135dbc07 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 GeoIP Context Established Change: 815920085af300f40e6406620e759587eb5604e3 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Pass plugin configs down to the context objects Change: b6d6c0831413c34a51b856db09923586980fd767 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Add timing information to contextualize script. Change: 0cf3b9ba37b8e59faaedae37a73b2aa7939cd4ce Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Added sudo and yum log parsers. Change: 58c41cc831df9eea401fdf2b35473f054670e08b Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Added sudoers parsing and dictionary entries for src/dst usernames. Change: 4788e0dacf903066deb68f018e3fe90d5a0c1eb2 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 pause. Change: 70f28672cc14b2fcdafb2e57ca344e0ba85a66d8 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Added an empty postfix context and fixed output from the library functions. Change: c08994a3cffeda013ede875892358418c0952ee7 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Adding timing element to log object. Change: 25c4a3f9d29775cba801c82d71ecc4dc1521efaa Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Added basic context establishment Added dictionary lookups for fields Fixes to the decoders to behave Change: 338bea1dad1d28ef2e7d66ec3d242beff1e9d2f7 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Simplify the contexts by providing a SubType to coerce things into the eris::log::context->target() code ref. It will return true if the context matches the field / target pair, or false otherwise. Change: 89279c0b767432a17b1b0dd5a9c53f43eb62d41b Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Typos, syntax, and logic fix ups. Included a demo script to use to test the whole eris:* name space. Decoders and object creation are working. Change: dac379507a39fc7e6f4e530913da04a6a0916f82 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 Ignore the main config file. Change: c307f13774d6829af2e51ce221f07e28db49d7b6 Author: Brad Lhotsky Date : 2016-09-21 22:20:22 +0000 initial revision Change: 8a3d2b7f64ba059fcb3cd2cd12f12bad620c61b7 Author: Brad Lhotsky Date : 2016-09-21 22:19:43 +0000 Initial commit ================ End of releases. ================