Log::ParallelCluster computing framework | |
Download |
Log::Parallel Ranking & Summary
Advertisement
- License:
- Perl Artistic License
- Price:
- FREE
- Publisher Name:
- David Muir Sharnoff
- Publisher web site:
- http://search.cpan.org/~muir/
Log::Parallel Tags
Log::Parallel Description
Cluster computing framework Log::Parallel is the main driver module at the heart of a cluster computing framework used for batch log processing. It sets things up, figures out what jobs can run and in what order, and queues them up to run.Everything it does is driven from the configuration data, probably parsed by Config::YAMLMacros and validated by Log::Parallel::ConfigCheck.Only one program, process_logs, is expected to use this module. As such, documentation for the API for this module in particular is not particularly relevant, but this is as good a place as any to document the overall system.process_logs is the driver script for processing data logs through a series of jobs specified in a configuration file.SYNOPSIS % bin/process_logs -c config_file use Log::Parallel; use Log::Parallel::ConfigCheck qw(validate_config); use Proc::JobQueue::DependencyQueue; opitons(); run($opts); validate_config($config); add_recnums($config); $files_by_recnum{$_->{recnum}} = get_files_by_srec($_, $config->{hostsinfo}) for @{$config->{sources}}; my $dependency_graph = make_dependency_graph(make_task_list($opts, $config, %files_by_recnum)) my $job_queue = new Proc::JobQueue::DependencyQueue(dependency_graph => $dependency_graph, hosts => [], hold_all => 1); setup_slave_hosts($config, $job_queue); Requirements: · Perl
Log::Parallel Related Software