MogileFS Distributed (meta) file system. Spray files across cheap disks on your network. Pay less for storage. No proprietary on-disk file formats.
Download

MogileFS Ranking & Summary

MogileFS Tags

automate replicate file system file replication meta file system automate replication

MogileFS Description

Distributed (meta) file system. Spray files across cheap disks on your network. Pay less for storage. No proprietary on-disk file formats. MogileFS is an open source distributed filesystem. MogileFS is not:· POSIX Compliant -- you don't run regular Unix applications or databases against MogileFS. It's meant for archiving write-once files and doing only sequential reads. (though you can modify a file by way of overwriting it with a new version)· Completely portable ... yet -- there still is some Linux-ism in the code, at least in the HTTP transport cod, but the plan is to scrap that and make it portable Here are some key features of "MogileFS": · Application level -- no special kernel modules required. · No single point of failure -- all three components of a MogileFS setup (storage nodes, trackers, and the tracker's database(s)) can be run on multiple machines, so there's no single point of failure. (you can run trackers on the same machines as storage nodes, too, so you don't need 4 machines...) A minimum of 2 machines is recommended. · Automatic file replication -- files, based on their "class", are automatically replicated between enough different storage nodes as to satisfy the minimum replica count as requested by their class. For instance, for a photo hosting site you can make original JPEGs have a minimum replica count of 3, but thumbnails and scaled versions only have a replica count of 1 or 2. If you lose the only copy of a thumbnail, the application can just rebuild it. In this way, MogileFS (without RAID) can save money on disks that would otherwise be storing multiple copies of data unnecessarily. · "Better than RAID" -- in a non-SAN RAID setup, the disks are redundant, but the host isn't. If you lose the entire machine, the files are inaccessible. MogileFS replicates the files between devices which are on different hosts, so files are always available. · Flat Namespace -- Files are identified by named keys in a flat, global namespace. You can create as many namespaces as you'd like, so multiple applications with potentially conflicting keys can run on the same MogileFS installation. · Shared-Nothing -- MogileFS doesn't depend on a pricey SAN with shared disks. Every machine maintains its own local disks. · No RAID required -- Local disks on MogileFS storage nodes can be in a RAID, or not. It's cheaper not to, as RAID doesn't buy you any safety that MogileFS doesn't already provide. · Local filesystem agnostic -- Local disks on MogileFS storage nodes can be formatted with your filesystem of choice (ext3, XFS, etc..). MogileFS does its own internal directory hashing so it doesn't hit filesystem limits such as "max files per directory" or "max directories per directory". Use what you're comfortable with. What's New in This Release: · fsck will now fix broken 'devcount' cols. · schema version bump to 12, and new delete worker code to use the new queue system. delete should be faster and more scalable. · add server setting queue_rate_for_fsck to allow rate limiting how fast the new distributed fsck creates jobs. the original default of 10k could utterly defeat wimpy databases. · Support 0 byte files. · Fix a bug where rebalance would accidentally delete copies of a fid. · Create new internal queue management system, with new job 'JobMaster'. This greatly reduces database load and increases throughput for workers using the new system. · Add 'no_unreachable_tracking' config. Disables more of the old replication code. Use this if you have no 1.x trackers. · Add hook for alternate replication sources. Obscure, but can be nice. · Add replicate_order_final_choices hook, so multidc replication may pull from the local network if one copy already exists. · Fix subtle bugs in process management. · Allow overriding of default classes.

MogileFS Related Software