Skip to main content

Start an Ingest

Ingest is a process that can create multiple media entities and relations by uploading a single JSON file into the Media Service. This guide walks you through the process of preparing ingest files and metadata to perform successful ingest operations.

This guide assumes the following prerequisites:

  • You have access to your Mosaic environment with a deployed Media Service from the Mosaic Media Template.
  • The Video Encoding Profiles are set up, and at least one video is located in the Azure Blob Storage source location.
  • The Image Acquisition Profile is set up, and there is at least one image in the Azure Blob Storage source location.
  • The Localization Service is enabled in both Admin Portal and using IS_LOCALIZATION_ENABLED environment variable in the Media Service.

Ingest Process Overview

This chapter gives an overview of the Media Service ingest. It describes how to create a single movie with only its title property filled.

  1. Start by creating a JSON file with the following contents (or download an example file):

    {
    "name": "Minimal Movie Ingest",
    "items": [
    {
    "type": "MOVIE",
    "external_id": "avatar_external_id",
    "data": { "title": "Avatar" }
    }
    ]
    }
  2. From the Management System Homepage, navigate to the Ingest workflow. In the Ingest Explorer Station, click the New button on the right.

    Ingest Explorer station

    overview_explorer

  3. Click the folder icon next to the Document field. Choose your JSON file and click Open. Start the ingest process by clicking the Proceed button on the right.

    New ingest process

    overview_upload

  4. This opens the Ingest Process details station that shows the progress for each entity that is ingested.

    Ingest progress

    overview_status

  5. When a single ingested item is green, it can be checked in the Movies Explorer and in the movies details page of the imported "Avatar" movie.

    Movies Explorer

    overview_movie

This is the most basic version of processing an ingest file. However, the process is still the same when using other entity types like TV shows, seasons, or episodes and all possible sets of metadata. Only the uploaded JSON ingest file is different. The next sections describe some more complex ingest examples.

Ingest of a Movie with More Metadata

In the previous example, the ingested movie was almost empty. This example expands on the previous one by adding all possible movie properties.

  1. Create a JSON file with the following contents (or download an example file):

    {
    "name": "Movie with Metadata",
    "items": [
    {
    "type": "MOVIE",
    "external_id": "avatar_external_id",
    "data": {
    "title": "Avatar",
    "original_title": "James Cameron's Avatar",
    "description": "Avatar is a 2009 American epic science fiction film...",
    "synopsis": "In 2154, humans have depleted Earth's natural resources...",
    "released": "2009-12-10",
    "studio": "20th Century Fox",
    "tags": ["3D", "SciFi", "Highlight"],
    "cast": ["Sam Worthington", "Zoe Saldana", "Sigourney Weaver"],
    "production_countries": [
    "United States of America",
    "Estonia",
    "Germany",
    "COL",
    "ESP"
    ],
    "genres": ["Sci-Fi", "Drama"],
    "licenses": [
    {
    "start": "2020-08-01T00:00:00.000+00:00",
    "end": "2020-08-30T23:59:59.999+00:00",
    "countries": ["AW", "AT", "FI"]
    }
    ]
    }
    }
    ]
    }
  2. The ingest process is the same as in the previous example. The resulting movie looks like this:

    movie_metadata

    1. Some explanations:

      • The ingest process decides whether to add or update a media entity based on its external_id value. The external ID is a unique identifier, defined by the external system that defines the JSON ingest file. In this case, the first example created the Avatar movie with the external ID "avatar_external_id". This second example updated the existing Avatar movie as the external_id in the system matched the one in the second ingest file. If the external_id values were different, a new movie would have been created.
      • The Media Service defines all movie properties as optional, except for title.
      • original_title, description, synopsis, and studio are all free-text properties.
      • released is a date property and must follow the following format: YYYY-MM-DD
      • tags, cast, and production_countries are array properties in the ingest document that expect free-text values.
      • genres is an array property that references genres that are already defined as "Movies Genres" in the settings area. Performing an ingest and referencing a non-existent genre results in a failure of the metadata update for one specific media entity.
      • licenses is an array of objects:

Ingest of a Movie with Video Relations

So far, the two examples ingested only movie details that have references within the Media Service (movies to genres, tags, ...). However, it is also possible to orchestrate the ingest across service boundaries. A movie entity and other entities can have a video relation. Videos are encoded by the Encoding Service and managed by the Video Service. To be processed, video files must be available in the blob storage. The video encoder process gets the access credentials from the Video Encoding Acquisition profile.

This example assumes that the acquisition profile is set up correctly and the blob storage container is defined as source. To make sure of this, you can go to the Video Encoding Acquisition profile details page and check the Container Name value.

video_acquisition_profile

Providing a Source Video File

When working with videos, each video should be put under its own folder with all its files (e.g. the video stream, audio tracks, subtitle, and caption files). This example provides a sample video with all related files that you can put into your blob storage. Download the zip file, unpack it (e.g. with 7-zip), and upload the resulting folder into your blob storage.

The provisioning of the videos is likely done by your content service provider. In this example, we use the link:https://azure.microsoft.com/en-us/features/storage-explorer/[Azure Storage Explorer] to manually upload a video into the source container.

Microsoft Azure Storage Explorer

azure_storage_explorer

  1. Under Emulator & Attached, right-click Storage Accounts.

  2. Select Connect to Azure Storage.

  3. Choose a Storage account or service option.

  4. Choose a Connection string (Key or SAS) option.

  5. Paste your Azure storage connection string value into the Connection String field.

  6. Click connect and a connection should appear in the Storage Explorer UI.

    Blob Storage Container "source"

    azure_storage_container

  7. Upload the extracted video folder into the source blob container as `"test_video"``. It should look like this:

    Source container root level

    sample-file-folder

    Contents of the "test_video" folder

    sample-file-folder-contents

Start an Ingest Process with the Main Video

Now that the video file is available in the blob storage, it can be referenced in a JSON file.

  1. Create a JSON file with the following contents (or download an example file):

    {
    "name": "Movie with Video",
    "items": [
    {
    "type": "MOVIE",
    "external_id": "avatar_external_id",
    "data": {
    "title": "Avatar",
    "main_video": {
    "source": "test_video",
    "profile": "DEFAULT"
    }
    }
    }
    ]
    }
    • source is a file path to a folder in blob storage that contains the files for a video. In this case, the folder is located in the root of blob storage, so it’s just a folder name.
    • profile defines, which of the Video Processing profiles should be used. This value is optional. If it isn’t specified, the default processing profile is used.
  2. After this file is ingested, the existing "Avatar" movie has a video relation to the "test_video".

    Ingest status of the metadata update and video import

    video_ingest_status

It’s important to note, that the video status does not reflect the actual video encoding status. It denotes that the video encoding has successfully started, a database entry for the video has been created, and the video is assigned to the Avatar movie.

Start an Ingest Process with Trailer Videos

Each movie can have only one main video but multiple trailers. The process of preparing the ingest file with video trailer data is very similar to the main video example. For this particular example, the movie should get two trailers, and the trailers should have a dedicated sub-folder named trailers, located in blob storage container root. You can use the same example video files also for the trailers.

  1. Upload the example video files into the folder trailers/test_video_1 and trailers/test_video_2, respectively:

    Trailer video folders

    azure_storage_container_trailers

  2. Each folder should have the same example video files in them:

    Example video files in the trailer folders

    azure_storage_container_trailer_files

  3. The container root should now include two folders: test_video and trailers.

    Root "source" folders

    azure_storage_container_with_trailers

  4. Create a JSON file with the following contents (or download an example file):

    {
    "name": "Movie with Trailers",
    "items": [
    {
    "type": "MOVIE",
    "external_id": "avatar_external_id",
    "data": {
    "title": "Avatar",
    "trailers": [
    {
    "source": "trailers/test_video_1",
    "profile": "DEFAULT"
    },
    {
    "source": "trailers/test_video_2",
    "profile": "DEFAULT"
    }
    ]
    }
    }
    ]
    }
    note

    This time we used a relative path to a trailer folder instead of just the folder name. The same logic would have applied to the main video, if the main video folder was located in a subfolder.

After this file is ingested, the movie has relations to the trailer videos.

Ingest of a Movie with Image Relations

The preparations for an ingest that references images are very similar to video ingest. Each image is just a single file. It does not consist of potentially multiple files like videos do. Therefore, all images can be copied to the root blob storage container directly if you want to.

To ingest a movie with image relations:

  1. You need to have access to the blob storage container that is defined in the Image Acquisition Profile. Use the Microsoft Azure Storage Explorer to connect to the image source blob storage, the same way it was done for videos.

  2. Upload any two images as avatar_1.jpg and avatar_2.jpg.

    Images source root folder

    azure_storage_container_with_images

  3. Create a JSON file with the following contents (or download an example file):

    {
    "name": "Movie with images",
    "items": [
    {
    "type": "MOVIE",
    "external_id": "avatar_external_id",
    "data": {
    "title": "Avatar",
    "images": [
    {
    "path": "avatar_1.jpg",
    "type": "COVER"
    },
    {
    "path": "avatar_2.jpg",
    "type": "TEASER"
    }
    ]
    }
    }
    ]
    }
    • path is a file path to an image file in blob storage. In this case, the images are located in the root of the blob storage container, so it’s just a file name.
    • type is a pre-defined value for the image type that the movie supports. The Media Template supports the COVER and TEASER image types.

After this file is ingested, the Avatar movie shall have relations to the two images.

Ingest of a Movie with Localizations

This example shows how to ingest a movie with localizations.

  1. Make sure that you have et-EE and de-DE locales defined in the Localization Service settings.

  2. Create a JSON file with the following contents (or download an example file):

    {
    "name": "Movie with Localizations",
    "items": [
    {
    "type": "MOVIE",
    "external_id": "avatar_external_id",
    "data": {
    "title": "Avatar",
    "description": "Avatar is a 2009 American epic science fiction film...",
    "synopsis": "In 2154, humans have depleted Earth's natural resources...",
    "localizations": [
    {
    "language_tag": "de-DE",
    "title": "Avatar – Aufbruch nach Pandora",
    "description": "Avatar ist ein US-amerikanischer Science-Fiction-Epos aus dem Jahr 2009...",
    "synopsis": "Im Jahr 2154 haben die Menschen die natürlichen Ressourcen der Erde erschöpft..."
    },
    {
    "language_tag": "et-EE",
    "title": "Avatar",
    "description": "Avatar on 2009. aasta Ameerika eepiline ulmefilm...",
    "synopsis": "Aastal 2154 on inimesed ammendanud Maa loodusvarad..."
    }
    ]
    }
    }
    ]
    }

After this file is ingested, you can observe the localization workflows and see the updated localization values.

Combined Movie Example

The chapters above described how to ingest different parts of movie relations and metadata. A combined example would look like the following (download JSON):

{
"name": "Full Movie",
"items": [
{
"type": "MOVIE",
"external_id": "avatar_external_id",
"data": {
"title": "Avatar",
"original_title": "James Cameron's Avatar",
"description": "Avatar is a 2009 American epic science fiction film...",
"synopsis": "In 2154, humans have depleted Earth's natural resources...",
"released": "2009-12-10",
"studio": "20th Century Fox",
"tags": ["3D", "SciFi", "Highlight"],
"cast": ["Sam Worthington", "Zoe Saldana", "Sigourney Weaver"],
"production_countries": [
"United States of America",
"Estonia",
"Germany",
"COL",
"ESP"
],
"genres": ["Sci-Fi", "Drama"],
"licenses": [
{
"start": "2020-08-01T00:00:00.000+00:00",
"end": "2020-08-30T23:59:59.999+00:00",
"countries": ["AW", "AT", "FI"]
}
],
"main_video": {
"source": "test_video",
"profile": "DEFAULT"
},
"trailers": [
{
"source": "trailers/test_video_1",
"profile": "DEFAULT"
},
{
"source": "trailers/test_video_2",
"profile": "DEFAULT"
}
],
"images": [
{
"path": "avatar_1.jpg",
"type": "COVER"
},
{
"path": "avatar_2.jpg",
"type": "TEASER"
}
],
"localizations": [
{
"language_tag": "de-DE",
"title": "Avatar – Aufbruch nach Pandora",
"description": "Avatar ist ein US-amerikanischer Science-Fiction-Epos aus dem Jahr 2009...",
"synopsis": "Im Jahr 2154 haben die Menschen die natürlichen Ressourcen der Erde erschöpft..."
},
{
"language_tag": "et-EE",
"title": "Avatar",
"description": "Avatar on 2009. aasta Ameerika eepiline ulmefilm...",
"synopsis": "Aastal 2154 on inimesed ammendanud Maa loodusvarad..."
}
]
}
}
]
}

Ingest Example for All Media Types

Apart from Movies, the Media Service also supports TV Shows, Seasons, and Episodes. Those can also be ingested in the same ingest JSON file. There are some differences between these types. Overall, the process is the same. Here is an example for all types (JSON file, image files, and video files):

{
"name": "Full Ingest Example",
"items": [
{
"type": "MOVIE",
"external_id": "avatar",
"data": {
"title": "Avatar",
"original_title": "James Cameron's Avatar",
"description": "Avatar is a 2009 American epic science fiction film...",
"synopsis": "In 2154, humans have depleted Earth's natural resources...",
"released": "2009-12-10",
"studio": "20th Century Fox",
"tags": ["3D", "SciFi", "Highlight"],
"cast": ["Sam Worthington", "Zoe Saldana", "Sigourney Weaver"],
"production_countries": [
"United States of America",
"Estonia",
"Germany",
"COL",
"ESP"
],
"genres": ["Sci-Fi", "Drama"],
"licenses": [
{
"start": "2020-08-01T00:00:00.000+00:00",
"end": "2020-08-30T23:59:59.999+00:00",
"countries": ["AW", "AT", "FI"]
}
],
"main_video": {
"source": "example_videos/videos/avatar",
"profile": "DEFAULT"
},
"trailers": [
{
"source": "example_videos/trailers/avatar_1",
"profile": "DEFAULT"
},
{
"source": "example_videos/trailers/avatar_2",
"profile": "DEFAULT"
}
],
"images": [
{
"path": "example_images/covers/avatar_1.jpg",
"type": "COVER"
},
{
"path": "example_images/teasers/avatar_1.jpg",
"type": "TEASER"
}
],
"localizations": [
{
"language_tag": "de-DE",
"title": "Avatar – Aufbruch nach Pandora",
"description": "Avatar ist ein US-amerikanischer Science-Fiction-Epos aus dem Jahr 2009...",
"synopsis": "Im Jahr 2154 haben die Menschen die natürlichen Ressourcen der Erde erschöpft..."
},
{
"language_tag": "et-EE",
"title": "Avatar",
"description": "Avatar on 2009. aasta Ameerika eepiline ulmefilm...",
"synopsis": "Aastal 2154 on inimesed ammendanud Maa loodusvarad..."
}
]
}
},
{
"type": "EPISODE",
"external_id": "mandalorian_s2_e1",
"data": {
"index": 1,
"parent_external_id": "mandalorian_s2",
"title": "The Marshal",
"original_title": "Chapter 9: The Marshal",
"description": "After the stories of Jango and Boba Fett, another warrior emerges in the Star Wars universe...",
"synopsis": "The Mandalorian is drawn to the Outer Rim in search of others of his kind.",
"released": "2020-10-30",
"studio": "Lucasfilm",
"tags": ["star wars", "mandalorian", "bounty hunter"],
"genres": ["Action", "Sci-Fi"],
"cast": ["Pedro Pascal", "Carl Weathers", "Gina Carano"],
"production_countries": ["USA"],
"licenses": [
{
"start": "2020-08-01T00:00:00.000+00:00",
"end": "2020-08-30T23:59:59.999+00:00",
"countries": ["AW", "AT", "FI"]
}
],
"main_video": {
"source": "example_videos/videos/mandalorian_s2_e1",
"profile": "DEFAULT"
},
"trailers": [
{
"source": "example_videos/trailers/mandalorian_s2_e1_t1",
"profile": "DEFAULT"
},
{
"source": "example_videos/trailers/mandalorian_s2_e1_t2",
"profile": "DEFAULT"
}
],
"images": [
{
"path": "example_images/covers/mandalorian_s2_e1_t1.jpg",
"type": "COVER"
},
{
"path": "example_images/teasers/mandalorian_s2_e1_t1.jpg",
"type": "TEASER"
}
],
"localizations": [
{
"language_tag": "de-DE",
"title": "The Mandalorian",
"description": "Nach den Geschichten von Jango und Boba Fett taucht ein weiterer Krieger im Star Wars-Universum auf...",
"synopsis": "Der Mandalorianer zieht es auf der Suche nach Artgenossen in den Outer Rim."
},
{
"language_tag": "et-EE",
"title": "The Mandalorian",
"description": "Pärast Jango ja Boba Fetti lugusid kerkib Tähesõdade universumis esile veel üks sõdalane...",
"synopsis": "Mandaloriat tõmbab Välisääre poole, et otsida teisi omasuguseid."
}
]
}
},
{
"type": "SEASON",
"external_id": "mandalorian_s2",
"data": {
"index": 2,
"parent_external_id": "mandalorian",
"description": "After the stories of Jango and Boba Fett, another warrior emerges in the Star Wars universe...",
"synopsis": "A Mandalorian bounty hunter tracks a target for a well-paying, mysterious client.",
"released": "2020-10-30",
"studio": "Lucasfilm",
"tags": ["star wars", "mandalorian", "bounty hunter"],
"genres": ["Action", "Sci-Fi"],
"cast": ["Pedro Pascal", "Carl Weathers", "Gina Carano"],
"production_countries": ["USA"],
"licenses": [
{
"start": "2020-08-01T00:00:00.000+00:00",
"end": "2020-08-30T23:59:59.999+00:00",
"countries": ["AW", "AT", "FI"]
}
],
"trailers": [
{
"source": "example_videos/trailers/mandalorian_s2_t1",
"profile": "DEFAULT"
},
{
"source": "example_videos/trailers/mandalorian_s2_t2",
"profile": "DEFAULT"
}
],
"images": [
{
"path": "example_images/covers/mandalorian_s2.jpg",
"type": "COVER"
},
{
"path": "example_images/teasers/mandalorian_s2.jpg",
"type": "TEASER"
}
],
"localizations": [
{
"language_tag": "de-DE",
"description": "Nach den Geschichten von Jango und Boba Fett taucht ein weiterer Krieger im Star Wars-Universum auf...",
"synopsis": "Ein mandalorianischer Kopfgeldjäger verfolgt ein Ziel für einen gut zahlenden, mysteriösen Kunden."
},
{
"language_tag": "et-EE",
"description": "Pärast Jango ja Boba Fetti lugusid kerkib Tähesõdade universumis esile veel üks sõdalane...",
"synopsis": "Mandaloriast pärit pearahakütt jälitab hästi maksva ja salapärase kliendi sihtmärki."
}
]
}
},
{
"type": "TVSHOW",
"external_id": "mandalorian",
"data": {
"title": "Mandalorian",
"original_title": "The Mandalorian",
"description": "After the stories of Jango and Boba Fett, another warrior emerges in the Star Wars universe...",
"synopsis": "The travels of a lone bounty hunter in the outer reaches of the galaxy, far from the authority of the New Republic.",
"released": "2019-11-12",
"studio": "Lucasfilm",
"tags": ["star wars", "mandalorian", "bounty hunter"],
"genres": ["Action", "Sci-Fi"],
"cast": ["Pedro Pascal", "Carl Weathers", "Gina Carano"],
"production_countries": ["USA"],
"licenses": [
{
"start": "2020-08-01T00:00:00.000+00:00",
"end": "2020-08-30T23:59:59.999+00:00",
"countries": ["AW", "AT", "FI"]
}
],
"trailers": [
{
"source": "example_videos/trailers/mandalorian_t1",
"profile": "DEFAULT"
},
{
"source": "example_videos/trailers/mandalorian_t2",
"profile": "DEFAULT"
}
],
"images": [
{
"path": "example_images/covers/mandalorian_1.jpg",
"type": "COVER"
},
{
"path": "example_images/teasers/mandalorian_1.jpg",
"type": "TEASER"
}
],
"localizations": [
{
"language_tag": "de-DE",
"title": "The Mandalorian",
"description": "Nach den Geschichten von Jango und Boba Fett taucht ein weiterer Krieger im Star Wars-Universum auf...",
"synopsis": "Die Reisen eines einsamen Kopfgeldjägers in den äußeren Bereichen der Galaxis, weit entfernt von der Autorität der Neuen Republik."
},
{
"language_tag": "et-EE",
"title": "The Mandalorian",
"description": "Pärast Jango ja Boba Fetti lugusid kerkib Tähesõdade universumis esile veel üks sõdalane...",
"synopsis": "Üksildase pearahaküti reisid galaktika äärealadel, kaugel Uue Vabariigi autoriteedist."
}
]
}
}
]
}

All four entity types have similar metadata, but there are some differences:

  • A season does not have a title but a required numeric index property.
  • An episode has both a title and a numeric index property which are both required.
  • All entity types can have trailers, but only Movie and Episode types can have a main_video.
  • A season has a parent_external_id which should have a value of a TV Show external_id. This is used to link a season to a TV show.
  • An episode has a parent_external_id which should have a value of a Season external_id. This is used to link an Episode to a Season.

Checking JSON Validity

When a JSON file is uploaded, it is validated against a JSON schema that is defined in the Media Service. This schema can be used to check if your JSON ingest file is valid without uploading it to the Media Service. You can use an online tool like https://www.jsonschemavalidator.net/ to get the validation result.

There are also custom validation rules that are not reflected by the JSON schema. There is, for example, a check for duplicate external_ids, since those should be unique per item type. There is no way to check for those custom rules without actually uploading the JSON file.

The schema file provided in this chapter is a copy of the schema that is provided (and potentially customized) in the Media Service. Please use your custom/latest version of this file to use it for validations.