MME Architecture

Overview

The QNX Aviage multimedia suite consists of several packages, including the multimedia core package, codec packages that provide WMA9, MP3, and AAC decoding and encoding, and software packages that support iPod and PlaysForSure media players.

The major component of the multimedia core package is the MultiMedia Engine (MME). The MME provides the main interfaces for configuring and controlling your multimedia applications. Designed to run on the QNX Neutrino OS, which can be installed on a wide variety of hardware platforms, the MME provides consumer-product developers a component-based solution that reduces the work required to develop and deliver multimedia products to their end customers.

The MME is designed to simplify and speed development of end-user applications that require device and filesystem access, content synchronization, playback control, and audio and graphics delivery. It handles multiple clients, sessions and streams, and abstracts hardware and protocol dependencies from other functional areas. It provides integration with a wide variety of media sources, including those requiring Digital Rights Management (DRM), and provides a high-level API for media transport control, device control and browsing, and media library management; and it automatically detects media devices and integrates their contents into a general database view. The applications the MME can be used to develop include:

The MME lets Human-Machine Interface (HMI) developers apply their talents to designing the best possible user experience instead of focussing on managing the media. When you build a client application that uses the MME, you can focus on:

You need to know about the configurations for your system's storage devices, but you can leave a long list of responsibilities to the MME:

MME functional areas

The MME is designed to bring together media management and playback control, providing a single, consistent interface for client applications. Internally, it has the broad functional areas described below.

Mediastore access

Mediastores are any source for multimedia data, including hard drives, DVDs, CDs and media devices such as iPods or MP3 players. The MME's mediastore access capabilities include:

Mediastore content management

The MME's mediastore content-management capabilities include:

Media playback and recording

The MME's media playback and recording capabilities include:

The MME interface

The MME API includes a primary interface and a secondary interface. The primary interface (mme) offers the media management functionality required of a multimedia middleware platform, while the secondary interface (qdb) offers the required database functionality. Together the primary and secondary interfaces offer multimedia applications a consistent API that provides:

Component-based architecture

The MME is comprised of several independent components. Each MME component executes independently as a Neutrino resource manager process. A resource manager is a user-level server process that accepts messages from other programs and, optionally, communicates with hardware.

The MME's component-based architecture delivers:

The MME resource managers

The MME resource managers can be placed into these groups:

The resource managers that provide the interfaces to multimedia client applications are:

Both the mme and the qdb resource managers support device-specific functionality within a POSIX framework. Together they make up the interface to HMI client applications, providing them with an API to control, browse, copy or rip, and play media, as well as the ability to monitor and manage multimedia processing. The mme controls the low-level resource managers that directly access and process media data.


MME resource managers


MME architecture showing mme and qdb resource managers.

Multimedia client applications don't normally interface with the MME's lower-level resource managers. You may nonetheless find it useful to know about these resource managers and understand what they do, especially if you are tuning your system configuration. The MME's low-level resource managers include:

Other low-level resource managers that are not specific to the MME, but which the MME uses, include:


MME components


High-level view of the MME components.

The MME database

The MME database is a repository of all information regarding media devices, media files and media streams known to the current MME instance. In addition to being a compendium of the metadata of all media files seen by the MME, this database stores MME system configuration and runtime states, such as track sessions and synchronization progress marks. Key data that the MME stores in its database includes:

A typical MME configuration might host the MME's database out of a RAM filesystem to enable fast performance on systems without persistent storage. In this type of implementation, the database could be made effectively persistent by configuring it to synchronize itself with a copy in flash memory.

Media file information and metadata

When the MME detects a mediastore, it synchronizes the mediastore content with its database. During synchronization, it generates a file ID with the file information and metadata for each media file described in its database. Client applications can reference this file ID to control media playback and other operations through the MME. Client applications can also pass SQL queries through MME API functions to select tracks for a particular album or artist to place in a track session.

To optimize performance when a mediastore is reinserted, the MME maintains synchronized media file information and metadata in its database even after a source mediastore is removed. However, to maintain database size within acceptable limits, when the MME adds new media entries to its database it may “prune” the database by removing stale entries to make room for the new information.

Runtime state and synchronization progress markers

The MME stores in its database the runtime state of track sessions, and the progress state of mediastore synchronizations.

The abstraction layer: mme

The mme resource manager is an abstraction layer and database manager in the MME. The functions it provides can be grouped into the following activities:

The mme resource manager has no device-specific APIs. It delegates state-sensitive device handling to device drivers, such as the drivers in the io-fs family.

Mediastore detection and data synchronization

Mediastore detection is the process by which the MME detects the presence of new mediastores. Synchronization is the process by which the MME examines mediastores and updates its database with information about the media files on the stores and with the metadata for this media. Through the mme resource manager, the MME performs mediastore synchronization in the following situations:

Synchronizer selection

The mme resource manager uses several types of synchronizers: mediastore, metadata and playlist synchronizers. These synchronizers rate themselves based on their ability to handle different types of mediastores, files and playlists so that when the mme resource manager receives a request to synchronize a mediastore, it can select the best synchronizer for each task.

Synchronization processes

After it has chosen the most appropriate synchronizers for the task, the mme resource manager begins its three-step synchronization operation — media identification, metadata retrieval and playlist retrieval — to populate the MME library with the media relevant information and metadata.

Synchronization progresses as a recursive tree walk of the filesystem directory structure for the mediastore. The mme resource manager performs this recursive tree walk of the filesystem automatically, regardless of why the synchronization was started, except in the case of synchronizations initiated by the client application with a specific request to not perform this recursive tree-walk.


Synchronization sequence


Illustration of the synchronization recursive tree walk

Filesystem change handling

The mme resource manager doesn't detect filesystem changes, such as media file creations and deletions, performed within filesystems. A client application that is aware of the changes must, therefore, instruct the MME to re-synchronize the modified mediastore. This re-synchronization can be done in a very broad or a very specific manner, depending on the needs of the application.

Media ripping

The MME currently supports the following modes for media ripping operations:

These ripping modes define the MME mediacopier's overall behavior and its interaction with playback track sessions. They can be configured by the client application developer.


Note: The term “ripping” refers to both the copying and encoding of media files into a new format, and to the simple copy of media files without changing their formats.

Media ripping set up

The mme resource manager uses a copy queue table which lists the files to be ripped or copied, along with the details for each operation: source, destination, format, bitrate, etc. The client application can call an MME API function to insert entries into this table, specifying destination file parameters such as encoding format, name template, and destination folder paths, then call another API function to start the media copy or ripping operation.

Metadata for copied and ripped files

The media ripping operation uses the MME's metadata synchronizers to obtain metadata for the ripped media files (Gracenote, cdtext, etc.) and to automatically:

Background (muted) media ripping

Background ripping operations are performed independently of playback operations. However, when accessing media sources (such as CD drives) that don't support multiple readers well, the mme resource manager gives precedence to playback operations. If a user tries to play media from a mediastore being used for background ripping, the mediacopier:


Background ripping


MME background ripping operation.

When playback on the mediastore ends and it becomes available for ripping operations again, the client application can instruct the mme resource manager to resume the operation. The mme resource manager will resume the ripping process, starting with the first entry in the copy queue table. In most cases, this entry will be the entry that was dropped when playback started. However, because the MME doesn't prevent the client application from inserting new entries into the copy queue when ripping is blocked by playback, it can't guarantee that a resumed ripping operation will begin with the file that was dropped when the operation was blocked.

The mediacopier blocks when it has processed all entries in the copy queue table.

Priority background ripping

The mme resource manager's priority background ripping mode grants the mediacopier priority access to the mediastore over playback operations. If a playback process needs to access tracks from a mediastore being accessed by the ripping operation, it uses the ripped tracks, not the source tracks on the source mediastore.

Priority background copying and ripping assumes that the system has sufficient computing resources to stay ahead of any requested playback and ensure continuous play. Priority background copy and ripping proceeds as follows:

As with background media ripping, the mediacopier blocks when it has processed all entries in the copy queue table.


Priority background ripping


MME priority background ripping operation.

Media browsing

The MME offers users a directed synchronization API function to browse media files inside a device as a hierarchical tree. MME client applications can direct synchronization to start from a device's root directory (“/” or from any other directory whose path is known) and use the metadata from the synchronized media to provide the client application with the information the end-user needs in order to continue browsing. For example, the client application can begin browsing an iPod device at a known path location such as /Music/Genres, then use the data it retrieves to compose folder and file lists that the end-user can use to continue browsing.

Media playback

The MME uses the io-media resource manager to handle media playback and rendering, and to implement graph classes for media stream rendering and transport control. The client application can send messages requesting play, pause, stop, change volume, etc. to the MME, which transmits these to io-media. When it receives a play command, io-media issues commands to the correct graph instance to start playback. During playback, io-media sends media state notifications (position, speed, direction, etc.) to the MME, which then forwards these back to the user application.

Drive and changer support

The MME's extensible architecture supports both internal and external changers. With appropriate components, the MME can access internal changers through a devb driver, and external changers through a bus.

The MME supports the creation of track sessions that span multiple disks. It monitors disk availability in the disk changer and issues slot change requests as needed. It looks for disk insertions, ejections and slot changes, and updates its database to track what media is present in the changer, and what media is active in the changer.

Filesystem management: io-fs

The MME supports media files on multiple filesystems, and automatically responds to the presence of new devices and mediastores. Supported filesystems include:

To interact with media device filesystems, the MME works through the io-media resource manager, which interacts differently with different filesystems. The io-media resource manager interacts:

The io-fs resource managers are a filesystem framework with media device specific extensions that support media devices such as iPods and PFS devices. In the MME, the io-fs resource managers provide:

The current set of iofs modules includes:

Device discovery

The io-fs resource managers offer automatic device discovery. When they detect a new device on the MME system, they retrieve information from the device about its capabilities and make this information available to the MME. This information may include:

iofs-tmpfs module

The iofs-tmpfs module is integrated as part of io-fs-media. It is a driver used by the MME to avoid the performance costs of running on slow storage devices, such as flash drives. It provides the MME with a POSIX filesystem interface that allows RAM to be used as the storage medium. To have the MME's implementation of the QDB database engine run only in RAM, simply configure the QDB database to use the filesystem mount path configured in tmpfs.

iofs-pfs module

The MME uses the iofs-pfs module to connect to, and browse and play media on PlaysForSure (PFS) devices. PFS is a Microsoft media standard for devices that use the Media Transport Protocol (MTP) and implement Digital Rights Management (DRM).

PFS connectivity

The io-fs implementation of PFS connectivity comprises three layers:

PFS module layer

The io-fs PFS module layer is responsible for presenting a filesystem view of the device to the io-fs resource manager. When io-fs initializes the PFS module, it sets up a structure filled with function pointers that it can call into. In this way, the PFS module can “translate” POSIX commands into MTP requests, and MTP requests into POSIX commands.

MTP layer

The MTP layer is Microsoft-supplied software that handles MTP messages. QNX has modified the MTP library to run in the QNX Neutrino environment.

Devices that support MTP provide a view of media content consisting of objects with properties. These objects and their properties are accessed via a command-and-response protocol with an optional data transfer phase.

PTP layer

The PTP layer handles Picture Transfer Protocol (PTP) messages. PTP is an implementation of the Still Image class of USB service, a protocol used to access multimedia content on digital cameras and PFS devices. The io-fs PTP layer communicates directly with io-usb, the Neutrino USB driver.

The figure below shows the relationship between the MME system, the io-fs PFS module, the MTP and PTP layers, and io-usb.


MME and media-fs pfs module


The MME and the iofs-pfs module.

Digital Rights Management (DRM)

The MME's iofs-pfs module uses Indirect License Acquisition (ILA) to obtain the key to decrypt DRM protected media content — to play a song, for example. The iofs-pfs module obtains this key in a license response message from a PlaysForSure device when it registers itself with that device. Registration with PFS devices must be renewed periodically. Registration with a PFS device is initiated when a user first attempts to access DRM protected content on the device. When the MME's iofs-pfs module first encounters DRM protected content on a PFS device, it:

DRM content decryption uses an AES block cipher with a 128-bit key. The AES key is unique to each playback session: it is different every time a song is played.

When the iofs-pfs module registers itself with the PFS device, it sends the device a certificate. The certificate contains the 1024-bit RSA public key that the device will use to encrypt the seed used to determine the AES key.

When the user selects DRM-protected content:

iofs-ipod module

The iofs-ipod module provides the MME with a filesystem view of a connected iPod device.

Using the iPod directory structure

The iofs-ipod module creates a directory structure from a connected iPod by querying the device's internal database. Each item on an iPod's menu is a database query. For example, selecting Albums queries the database for albums. Each further item on an iPod's menu is a subquery of the query represented by the parent menu item. Using this organizing principle, the iPod module creates for the MME a filesystem directory structure that resembles an iPod menu structure.

This filesystem directory lets the HMI perform operations on the iPod via the MME. For example, performing cd Music; ls through the MME produces the same result as a user selecting the “Music” option on the iPod. Both actions yield the same item list.

Supported iPod functionality

The MME can assume responsibility for sending control commands to the iPod to initiate playback, to stop or pause play, etc., and for routing the iPod's analog audio output directly to an amplifier. However, since iPod devices do not export their digital content, playback of content on an iPod device remains on the iPod device itself.

Media transport, control and rendering: io-media

The io-media resource manager provides low-level transport, control and rendering of media data. HMI access to io-media and its functionality is provided through the MME API and its mme abstraction layer.

The io-media resource manager uses graphs to process media tracks. Different graph classes perform different tasks, such as playing or ripping media tracks. The MME specifies the task, the media input source, the output device, the class of graph it requires, and a name for the graph instance it will use. The io-media resource manager uses this information to create a graph instance, which processes the media track.


io-media block diagram


The io-media resource manager in the MME.

In the figure above, control lines are shown as dashed lines, and data lines are shown as solid lines.

During media processing, the MME can send commands to io-media — for example, to pause playback or stop recording a stream, and io-media communicates progress and events, such as read errors, back to the MME, which reports these back to its HMI client application.

The MME can request multiple tasks from io-media; for each task io-media creates a new graph instance to handle the processing. At any point in time, each client application connection to the MME may have one instance of a playing graph, plus other graph instances performing background activities, such as ripping or capturing a live media stream to a buffer.

Different io-media variants bring together as a single executable selections of different modules, with each module implementing a selection of graphs.

trackplayer io-media graph class

trackplayer is a general purpose class of io-media graph dedicated to playing standard CDs (LPCM), MP3 files and WMA encoded files. It can interface to a standard CD or DVD drive, and includes multiple components that perform tasks, such as decoding, parsing and streaming.

The trackplayer is mounted on the QNX filesystem as /dev/io-media/trackplayer, and instances of the class appear as files in that directory. Depending on the performance of the platform, the MME can use several instances of trackplayer to drive separate audio streams to different output zones. For instance, in an automobile, passengers in the front seats and in the back seats could listen to different audio streams, with each stream being handled by a different instance of trackplayer.

The dvdtrackplayer and dvdnavigator io-media graph classes

The dvdtrackplayer graph class plays DVD audio tracks and DVD video chapters. The dvdnavigator graph class allows the client application to send button commands, select from disk menus, and otherwise interact with DVD player hardware. It requires CSS/CPPM decryption, and it interfaces exclusively to either a DVD drive or a DVD changer.

MMF: the multimedia framework

Many io-media graphs are built from filters. The multimedia framework (MMF) is a collection of filters for implementing specific media stream processing functions, such as the parsing or decryption of data. To process a multimedia stream, io-media uses only the filters it needs to get the multimedia data, process it, and send it to a target device.

The MMF can be used to create any of:

In addition, because the MMF uses the QNX Addon Interfaces library to define its standard interfaces, new components such as codecs written by QNX or by third parties are easily incorporated into the MMF as they become available. In many cases, these additions can be made without recompiling the MMF or the applications that use it.

MMF filters

MMF filters:

Filters are classified by the specific tasks they perform. These classifications are described in the table below.

Classification Description Examples
Reader Reads data from a source. Audio card readers, data stream readers and video capture card readers.
Parser Parses a stream of data into component parts. MPEG bit stream parsers, WAV format parsers and AVI format parsers.
Decoder Decodes compressed or encoded data. MPEG audio and video decoders, and DIVX decoders.
Encoder Encodes raw data into a specific format. MPEG audio and video encoders.
Writer Sends data to a destination. Audio card writers, video output writers and output file writers.