¶ PFR System Overview
¶ Introduction
In many environments, restore of part of an object is desired especially when the overall file size is large and the restore is for only a small part of a file/object. The BlackPearl itself understands byte offsets for partial restores and can supply a part of a file/object as a get when referenced as specific bytes from beginning of file (start and stop). That specific portion is retrieved and available back to the client.
In environments where the file/object in question is a media file, the client normally works in time or frame offset from beginning of file and not byte offsets. Some clients, such as AVID PAM systems, have that conversion built into their software such that when a user requests a specific time segment to be retrieved, the system converts that internally to a byte offset to make the actual get/restore request to BlackPearl. For clients that do not have this feature, Spectra provides an optional Partial File Restore (PFR) system.
An installation guide can be found here.
¶ Basic Theory of Operation
A BlackPearl system with PFR consists of four main parts:
- Storage – this can be a tape library, object storage disk, cloud, and/or another BlackPearl for redundancy in the same or alternate site.
- The BlackPearl itself – A hardware appliance from Spectra which consists of a fast cache, a policy engine, and data movers to archive and restore content to the storage.
- BlackPearl RioBroker – A software product from Spectra that acts as an interface to the client software (MAM, workflow engine), has a job queue, and powerful data movers to mount external primary storage and move files into and out of BlackPearl. RioBroker is normally run on its own physical server (Windows) or a VM instance provided by the customer.
- Spectra PFR software – A software product from Spectra, using the Marquis PFR engine, which is normally run on its own physical or virtual server. For smaller operations, it can also be run on the same physical server as RioBroker. The PFR software is a separate charge and can be scaled for performance.
The Spectra PFR engine is a separate software package from RioBroker. The PFR software acts as a plugin for RioBroker and performs two specific functions:
- Indexing - Upon initial archive, when commanded by the client, a media file on a primary storage location is first indexed by the PFR plugin. Essentially the file is read in place and a small index file created that is able to correlate time/frame offset information to byte offset from beginning of file. That file is then stored in a predetermined location so that upon the client later requesting a partial file restore, the system can find and use that index to determine the byte offset. After indexing, the RioBroker then initiates a data move (copy) operation from the files original location on primary storage into the BlackPearl cache where the BlackPearl policy engine engages and makes all requested final copies to locations and media specified in the BlackPearl policy. Note that currently, RioBroker creates a. unique identifier for a specific file, so index files are specific to that index operation, that specific file, and that RioBroker installation.
- PFR restore – Upon a user initiating a partial file restore operation using the client, a request is made to RioBroker using time/frame offsets, a series of events is created:
- RioBroker passes the object name and time offsets to the PFR plugin and receives the appropriate byte offsets.
- RioBroker initiates a get/restore operation to BlackPearl for the byte offset portion of the file and moves that segment to a predetermined disk landing area.
- The PFR plugin then rewraps the segment into a usable media file. Modern containers and codecs require specific header and footer information in order for the segment to become a valid media file that can be used in an editor or player.
- Upon completion of the rewrap, the file is automatically moved by RioBroker to the final location requested by the user
¶ Clustering Options for Redundancy and/or Speed
- Storage – BlackPearl is able to make as many copies, in as many locations and desired for redundancy.
- BlackPearl can replicate all data under policy control to another BlackPearl in another location
- BlackPearl can be set up in a dual-head configuration with automatic failover to its peer upon any lack of connectivity. This option is typically referred to as “HotPair”
- RioBroker can be set up as a VM with automatic failover under control of the hypervisor controller
- RioBroker can be set up as a cluster with multiple unique software instances (minimum 3 in odd numbers) which share a common sharded database and automatically farm out jobs to cluster members both for speed and redundancy.
- The PFR plugin is also clusterable within specific guidelines:
- Indexing is a relatively fast operation, and as such, RioBroker makes indexing requests to a specific PFR software server and is not expandable beyond one server today. (Note that Spectra is actively monitoring architectures in use by customers and will consider adding a clusterable indexer if the need arises)
- Rewrap/restore is a more resource intensive operation and, while most installations are fully satisfied by a single PFR software instance, the software includes a clustering capability if the customer determines that a large number of concurrent PFR restores are needed in a short period of time. Generally, 3-4 rewrap operations can happen simultaneously and then a queue is established for more requests. If a customer needs a lot of rewraps to happen in a short period of time, a cluster expansion is a good option. This is typically the case when deadlines are short in news or similar environments. If a PFR software cluster is purchased (see “multiple servers” quoting option below), the operation is seamless and invisible to both the user and client software. Rewrap requests are processed by RioBroker, passed to the master PFR engine server, and the PFR software itself farms out rewrap operations to PFR cluster members.