Start an Ingest
Ingest is a process that can create multiple media entities and relations by uploading a single JSON file into the Media Service. This guide walks you through the process of preparing ingest files and metadata to perform successful ingest operations.
This guide assumes the following prerequisites:
- You have access to your Mosaic environment with a deployed Media Service from the Mosaic Media Template.
- The Video Encoding Profiles are set up, and at least one video is located in the Azure Blob Storage source location.
- The Image Acquisition Profile is set up, and there is at least one image in the Azure Blob Storage source location.
- The Localization Service is enabled in both Admin Portal and using
IS_LOCALIZATION_ENABLED
environment variable in the Media Service.
Ingest Process Overview
This chapter gives an overview of the Media Service ingest. It describes how to create a single movie with only its title property filled.
-
Start by creating a JSON file with the following contents (or download an example file):
{
"name": "Minimal Movie Ingest",
"items": [
{
"type": "MOVIE",
"external_id": "avatar_external_id",
"data": { "title": "Avatar" }
}
]
} -
From the Management System Homepage, navigate to the Ingest workflow. In the Ingest Explorer Station, click the New button on the right.
Ingest Explorer station
-
Click the folder icon next to the Document field. Choose your JSON file and click Open. Start the ingest process by clicking the Proceed button on the right.
New ingest process
-
This opens the Ingest Process details station that shows the progress for each entity that is ingested.
Ingest progress
-
When a single ingested item is green, it can be checked in the Movies Explorer and in the movies details page of the imported "Avatar" movie.
Movies Explorer
This is the most basic version of processing an ingest file. However, the process is still the same when using other entity types like TV shows, seasons, or episodes and all possible sets of metadata. Only the uploaded JSON ingest file is different. The next sections describe some more complex ingest examples.
Ingest of a Movie with More Metadata
In the previous example, the ingested movie was almost empty. This example expands on the previous one by adding all possible movie properties.
-
Create a JSON file with the following contents (or download an example file):
{
"name": "Movie with Metadata",
"items": [
{
"type": "MOVIE",
"external_id": "avatar_external_id",
"data": {
"title": "Avatar",
"original_title": "James Cameron's Avatar",
"description": "Avatar is a 2009 American epic science fiction film...",
"synopsis": "In 2154, humans have depleted Earth's natural resources...",
"released": "2009-12-10",
"studio": "20th Century Fox",
"tags": ["3D", "SciFi", "Highlight"],
"cast": ["Sam Worthington", "Zoe Saldana", "Sigourney Weaver"],
"production_countries": [
"United States of America",
"Estonia",
"Germany",
"COL",
"ESP"
],
"genres": ["Sci-Fi", "Drama"],
"licenses": [
{
"start": "2020-08-01T00:00:00.000+00:00",
"end": "2020-08-30T23:59:59.999+00:00",
"countries": ["AW", "AT", "FI"]
}
]
}
}
]
} -
The ingest process is the same as in the previous example. The resulting movie looks like this:
-
Some explanations:
- The ingest process decides whether to add or update a media entity based on
its
external_id
value. The external ID is a unique identifier, defined by the external system that defines the JSON ingest file. In this case, the first example created the Avatar movie with the external ID"avatar_external_id"
. This second example updated the existing Avatar movie as theexternal_id
in the system matched the one in the second ingest file. If theexternal_id
values were different, a new movie would have been created. - The Media Service defines all movie properties as optional, except for
title
. original_title
,description
,synopsis
, andstudio
are all free-text properties.released
is a date property and must follow the following format: YYYY-MM-DDtags
,cast
, andproduction_countries
are array properties in the ingest document that expect free-text values.genres
is an array property that references genres that are already defined as "Movies Genres" in the settings area. Performing an ingest and referencing a non-existent genre results in a failure of the metadata update for one specific media entity.licenses
is an array of objects:start
andend
values should be defined in the date-time link:https://json-schema.org/understanding-json-schema/reference/string.html#dates-and-times[ISO 8601] format.countries
is an array of link:https://www.iban.com/country-codes[ISO 3166 Alpha-2] country codes.- Each license object must have at least one property.
- The ingest process decides whether to add or update a media entity based on
its
-
Ingest of a Movie with Video Relations
So far, the two examples ingested only movie details that have references within the Media Service (movies to genres, tags, ...). However, it is also possible to orchestrate the ingest across service boundaries. A movie entity and other entities can have a video relation. Videos are encoded by the Encoding Service and managed by the Video Service. To be processed, video files must be available in the blob storage. The video encoder process gets the access credentials from the Video Encoding Acquisition profile.
This example assumes that the acquisition profile is set up correctly and the
blob storage container is defined as source
. To make sure of this, you can go
to the Video Encoding Acquisition profile details page and check the Container Name
value.
Providing a Source Video File
When working with videos, each video should be put under its own folder with all its files (e.g. the video stream, audio tracks, subtitle, and caption files). This example provides a sample video with all related files that you can put into your blob storage. Download the zip file, unpack it (e.g. with 7-zip), and upload the resulting folder into your blob storage.
The provisioning of the videos is likely done by your content service provider. In this example, we use the link:https://azure.microsoft.com/en-us/features/storage-explorer/[Azure Storage Explorer] to manually upload a video into the source container.
Microsoft Azure Storage Explorer
-
Under
Emulator & Attached
, right-clickStorage Accounts
. -
Select
Connect to Azure Storage
. -
Choose a
Storage account or service
option. -
Choose a
Connection string (Key or SAS)
option. -
Paste your Azure storage connection string value into the
Connection String
field. -
Click connect and a connection should appear in the Storage Explorer UI.
Blob Storage Container "source"
-
Upload the extracted video folder into the
source
blob container as `"test_video"``. It should look like this:Source container root level
Contents of the "test_video" folder
Start an Ingest Process with the Main Video
Now that the video file is available in the blob storage, it can be referenced in a JSON file.
-
Create a JSON file with the following contents (or download an example file):
{
"name": "Movie with Video",
"items": [
{
"type": "MOVIE",
"external_id": "avatar_external_id",
"data": {
"title": "Avatar",
"main_video": {
"source": "test_video",
"profile": "DEFAULT"
}
}
}
]
}source
is a file path to a folder in blob storage that contains the files for a video. In this case, the folder is located in the root of blob storage, so it’s just a folder name.profile
defines, which of the Video Processing profiles should be used. This value is optional. If it isn’t specified, the default processing profile is used.
-
After this file is ingested, the existing "Avatar" movie has a video relation to the
"test_video"
.Ingest status of the metadata update and video import
It’s important to note, that the video status does not reflect the actual video encoding status. It denotes that the video encoding has successfully started, a database entry for the video has been created, and the video is assigned to the Avatar movie.
Start an Ingest Process with Trailer Videos
Each movie can have only one main video but multiple trailers. The process of
preparing the ingest file with video trailer data is very similar to the main
video example. For this particular example, the movie should get two trailers,
and the trailers should have a dedicated sub-folder named trailers
, located in
blob storage container root. You can use the same example video files also for the
trailers.
-
Upload the example video files into the folder
trailers/test_video_1
andtrailers/test_video_2
, respectively:Trailer video folders
-
Each folder should have the same example video files in them:
Example video files in the trailer folders
-
The container root should now include two folders:
test_video
andtrailers
.Root "source" folders
-
Create a JSON file with the following contents (or download an example file):
{
"name": "Movie with Trailers",
"items": [
{
"type": "MOVIE",
"external_id": "avatar_external_id",
"data": {
"title": "Avatar",
"trailers": [
{
"source": "trailers/test_video_1",
"profile": "DEFAULT"
},
{
"source": "trailers/test_video_2",
"profile": "DEFAULT"
}
]
}
}
]
}noteThis time we used a relative path to a trailer folder instead of just the folder name. The same logic would have applied to the main video, if the main video folder was located in a subfolder.
After this file is ingested, the movie has relations to the trailer videos.
Ingest of a Movie with Image Relations
The preparations for an ingest that references images are very similar to video ingest. Each image is just a single file. It does not consist of potentially multiple files like videos do. Therefore, all images can be copied to the root blob storage container directly if you want to.
To ingest a movie with image relations:
-
You need to have access to the blob storage container that is defined in the Image Acquisition Profile. Use the Microsoft Azure Storage Explorer to connect to the image source blob storage, the same way it was done for videos.
-
Upload any two images as
avatar_1.jpg
andavatar_2.jpg
.Images source root folder
-
Create a JSON file with the following contents (or download an example file):
{
"name": "Movie with images",
"items": [
{
"type": "MOVIE",
"external_id": "avatar_external_id",
"data": {
"title": "Avatar",
"images": [
{
"path": "avatar_1.jpg",
"type": "COVER"
},
{
"path": "avatar_2.jpg",
"type": "TEASER"
}
]
}
}
]
}path
is a file path to an image file in blob storage. In this case, the images are located in the root of the blob storage container, so it’s just a file name.type
is a pre-defined value for the image type that the movie supports. The Media Template supports theCOVER
andTEASER
image types.
After this file is ingested, the Avatar movie shall have relations to the two images.
Ingest of a Movie with Localizations
This example shows how to ingest a movie with localizations.
-
Make sure that you have
et-EE
andde-DE
locales defined in the Localization Service settings. -
Create a JSON file with the following contents (or download an example file):
{
"name": "Movie with Localizations",
"items": [
{
"type": "MOVIE",
"external_id": "avatar_external_id",
"data": {
"title": "Avatar",
"description": "Avatar is a 2009 American epic science fiction film...",
"synopsis": "In 2154, humans have depleted Earth's natural resources...",
"localizations": [
{
"language_tag": "de-DE",
"title": "Avatar – Aufbruch nach Pandora",
"description": "Avatar ist ein US-amerikanischer Science-Fiction-Epos aus dem Jahr 2009...",
"synopsis": "Im Jahr 2154 haben die Menschen die natürlichen Ressourcen der Erde erschöpft..."
},
{
"language_tag": "et-EE",
"title": "Avatar",
"description": "Avatar on 2009. aasta Ameerika eepiline ulmefilm...",
"synopsis": "Aastal 2154 on inimesed ammendanud Maa loodusvarad..."
}
]
}
}
]
}
After this file is ingested, you can observe the localization workflows and see the updated localization values.
Combined Movie Example
The chapters above described how to ingest different parts of movie relations and metadata. A combined example would look like the following (download JSON):
{
"name": "Full Movie",
"items": [
{
"type": "MOVIE",
"external_id": "avatar_external_id",
"data": {
"title": "Avatar",
"original_title": "James Cameron's Avatar",
"description": "Avatar is a 2009 American epic science fiction film...",
"synopsis": "In 2154, humans have depleted Earth's natural resources...",
"released": "2009-12-10",
"studio": "20th Century Fox",
"tags": ["3D", "SciFi", "Highlight"],
"cast": ["Sam Worthington", "Zoe Saldana", "Sigourney Weaver"],
"production_countries": [
"United States of America",
"Estonia",
"Germany",
"COL",
"ESP"
],
"genres": ["Sci-Fi", "Drama"],
"licenses": [
{
"start": "2020-08-01T00:00:00.000+00:00",
"end": "2020-08-30T23:59:59.999+00:00",
"countries": ["AW", "AT", "FI"]
}
],
"main_video": {
"source": "test_video",
"profile": "DEFAULT"
},
"trailers": [
{
"source": "trailers/test_video_1",
"profile": "DEFAULT"
},
{
"source": "trailers/test_video_2",
"profile": "DEFAULT"
}
],
"images": [
{
"path": "avatar_1.jpg",
"type": "COVER"
},
{
"path": "avatar_2.jpg",
"type": "TEASER"
}
],
"localizations": [
{
"language_tag": "de-DE",
"title": "Avatar – Aufbruch nach Pandora",
"description": "Avatar ist ein US-amerikanischer Science-Fiction-Epos aus dem Jahr 2009...",
"synopsis": "Im Jahr 2154 haben die Menschen die natürlichen Ressourcen der Erde erschöpft..."
},
{
"language_tag": "et-EE",
"title": "Avatar",
"description": "Avatar on 2009. aasta Ameerika eepiline ulmefilm...",
"synopsis": "Aastal 2154 on inimesed ammendanud Maa loodusvarad..."
}
]
}
}
]
}
Ingest Example for All Media Types
Apart from Movies, the Media Service also supports TV Shows, Seasons, and Episodes. Those can also be ingested in the same ingest JSON file. There are some differences between these types. Overall, the process is the same. Here is an example for all types (JSON file, image files, and video files):
{
"name": "Full Ingest Example",
"items": [
{
"type": "MOVIE",
"external_id": "avatar",
"data": {
"title": "Avatar",
"original_title": "James Cameron's Avatar",
"description": "Avatar is a 2009 American epic science fiction film...",
"synopsis": "In 2154, humans have depleted Earth's natural resources...",
"released": "2009-12-10",
"studio": "20th Century Fox",
"tags": ["3D", "SciFi", "Highlight"],
"cast": ["Sam Worthington", "Zoe Saldana", "Sigourney Weaver"],
"production_countries": [
"United States of America",
"Estonia",
"Germany",
"COL",
"ESP"
],
"genres": ["Sci-Fi", "Drama"],
"licenses": [
{
"start": "2020-08-01T00:00:00.000+00:00",
"end": "2020-08-30T23:59:59.999+00:00",
"countries": ["AW", "AT", "FI"]
}
],
"main_video": {
"source": "example_videos/videos/avatar",
"profile": "DEFAULT"
},
"trailers": [
{
"source": "example_videos/trailers/avatar_1",
"profile": "DEFAULT"
},
{
"source": "example_videos/trailers/avatar_2",
"profile": "DEFAULT"
}
],
"images": [
{
"path": "example_images/covers/avatar_1.jpg",
"type": "COVER"
},
{
"path": "example_images/teasers/avatar_1.jpg",
"type": "TEASER"
}
],
"localizations": [
{
"language_tag": "de-DE",
"title": "Avatar – Aufbruch nach Pandora",
"description": "Avatar ist ein US-amerikanischer Science-Fiction-Epos aus dem Jahr 2009...",
"synopsis": "Im Jahr 2154 haben die Menschen die natürlichen Ressourcen der Erde erschöpft..."
},
{
"language_tag": "et-EE",
"title": "Avatar",
"description": "Avatar on 2009. aasta Ameerika eepiline ulmefilm...",
"synopsis": "Aastal 2154 on inimesed ammendanud Maa loodusvarad..."
}
]
}
},
{
"type": "EPISODE",
"external_id": "mandalorian_s2_e1",
"data": {
"index": 1,
"parent_external_id": "mandalorian_s2",
"title": "The Marshal",
"original_title": "Chapter 9: The Marshal",
"description": "After the stories of Jango and Boba Fett, another warrior emerges in the Star Wars universe...",
"synopsis": "The Mandalorian is drawn to the Outer Rim in search of others of his kind.",
"released": "2020-10-30",
"studio": "Lucasfilm",
"tags": ["star wars", "mandalorian", "bounty hunter"],
"genres": ["Action", "Sci-Fi"],
"cast": ["Pedro Pascal", "Carl Weathers", "Gina Carano"],
"production_countries": ["USA"],
"licenses": [
{
"start": "2020-08-01T00:00:00.000+00:00",
"end": "2020-08-30T23:59:59.999+00:00",
"countries": ["AW", "AT", "FI"]
}
],
"main_video": {
"source": "example_videos/videos/mandalorian_s2_e1",
"profile": "DEFAULT"
},
"trailers": [
{
"source": "example_videos/trailers/mandalorian_s2_e1_t1",
"profile": "DEFAULT"
},
{
"source": "example_videos/trailers/mandalorian_s2_e1_t2",
"profile": "DEFAULT"
}
],
"images": [
{
"path": "example_images/covers/mandalorian_s2_e1_t1.jpg",
"type": "COVER"
},
{
"path": "example_images/teasers/mandalorian_s2_e1_t1.jpg",
"type": "TEASER"
}
],
"localizations": [
{
"language_tag": "de-DE",
"title": "The Mandalorian",
"description": "Nach den Geschichten von Jango und Boba Fett taucht ein weiterer Krieger im Star Wars-Universum auf...",
"synopsis": "Der Mandalorianer zieht es auf der Suche nach Artgenossen in den Outer Rim."
},
{
"language_tag": "et-EE",
"title": "The Mandalorian",
"description": "Pärast Jango ja Boba Fetti lugusid kerkib Tähesõdade universumis esile veel üks sõdalane...",
"synopsis": "Mandaloriat tõmbab Välisääre poole, et otsida teisi omasuguseid."
}
]
}
},
{
"type": "SEASON",
"external_id": "mandalorian_s2",
"data": {
"index": 2,
"parent_external_id": "mandalorian",
"description": "After the stories of Jango and Boba Fett, another warrior emerges in the Star Wars universe...",
"synopsis": "A Mandalorian bounty hunter tracks a target for a well-paying, mysterious client.",
"released": "2020-10-30",
"studio": "Lucasfilm",
"tags": ["star wars", "mandalorian", "bounty hunter"],
"genres": ["Action", "Sci-Fi"],
"cast": ["Pedro Pascal", "Carl Weathers", "Gina Carano"],
"production_countries": ["USA"],
"licenses": [
{
"start": "2020-08-01T00:00:00.000+00:00",
"end": "2020-08-30T23:59:59.999+00:00",
"countries": ["AW", "AT", "FI"]
}
],
"trailers": [
{
"source": "example_videos/trailers/mandalorian_s2_t1",
"profile": "DEFAULT"
},
{
"source": "example_videos/trailers/mandalorian_s2_t2",
"profile": "DEFAULT"
}
],
"images": [
{
"path": "example_images/covers/mandalorian_s2.jpg",
"type": "COVER"
},
{
"path": "example_images/teasers/mandalorian_s2.jpg",
"type": "TEASER"
}
],
"localizations": [
{
"language_tag": "de-DE",
"description": "Nach den Geschichten von Jango und Boba Fett taucht ein weiterer Krieger im Star Wars-Universum auf...",
"synopsis": "Ein mandalorianischer Kopfgeldjäger verfolgt ein Ziel für einen gut zahlenden, mysteriösen Kunden."
},
{
"language_tag": "et-EE",
"description": "Pärast Jango ja Boba Fetti lugusid kerkib Tähesõdade universumis esile veel üks sõdalane...",
"synopsis": "Mandaloriast pärit pearahakütt jälitab hästi maksva ja salapärase kliendi sihtmärki."
}
]
}
},
{
"type": "TVSHOW",
"external_id": "mandalorian",
"data": {
"title": "Mandalorian",
"original_title": "The Mandalorian",
"description": "After the stories of Jango and Boba Fett, another warrior emerges in the Star Wars universe...",
"synopsis": "The travels of a lone bounty hunter in the outer reaches of the galaxy, far from the authority of the New Republic.",
"released": "2019-11-12",
"studio": "Lucasfilm",
"tags": ["star wars", "mandalorian", "bounty hunter"],
"genres": ["Action", "Sci-Fi"],
"cast": ["Pedro Pascal", "Carl Weathers", "Gina Carano"],
"production_countries": ["USA"],
"licenses": [
{
"start": "2020-08-01T00:00:00.000+00:00",
"end": "2020-08-30T23:59:59.999+00:00",
"countries": ["AW", "AT", "FI"]
}
],
"trailers": [
{
"source": "example_videos/trailers/mandalorian_t1",
"profile": "DEFAULT"
},
{
"source": "example_videos/trailers/mandalorian_t2",
"profile": "DEFAULT"
}
],
"images": [
{
"path": "example_images/covers/mandalorian_1.jpg",
"type": "COVER"
},
{
"path": "example_images/teasers/mandalorian_1.jpg",
"type": "TEASER"
}
],
"localizations": [
{
"language_tag": "de-DE",
"title": "The Mandalorian",
"description": "Nach den Geschichten von Jango und Boba Fett taucht ein weiterer Krieger im Star Wars-Universum auf...",
"synopsis": "Die Reisen eines einsamen Kopfgeldjägers in den äußeren Bereichen der Galaxis, weit entfernt von der Autorität der Neuen Republik."
},
{
"language_tag": "et-EE",
"title": "The Mandalorian",
"description": "Pärast Jango ja Boba Fetti lugusid kerkib Tähesõdade universumis esile veel üks sõdalane...",
"synopsis": "Üksildase pearahaküti reisid galaktika äärealadel, kaugel Uue Vabariigi autoriteedist."
}
]
}
}
]
}
All four entity types have similar metadata, but there are some differences:
- A
season
does not have atitle
but a required numericindex
property. - An
episode
has both atitle
and a numericindex
property which are both required. - All entity types can have
trailers
, but only Movie and Episode types can have amain_video
. - A
season
has aparent_external_id
which should have a value of a TV Showexternal_id
. This is used to link a season to a TV show. - An
episode
has aparent_external_id
which should have a value of a Seasonexternal_id
. This is used to link an Episode to a Season.
Checking JSON Validity
When a JSON file is uploaded, it is validated against a JSON schema that is defined in the Media Service. This schema can be used to check if your JSON ingest file is valid without uploading it to the Media Service. You can use an online tool like https://www.jsonschemavalidator.net/ to get the validation result.
There are also custom validation rules that are not reflected by the JSON
schema. There is, for example, a check for duplicate external_ids
, since those
should be unique per item type. There is no way to check for those custom rules
without actually uploading the JSON file.
The schema file provided in this chapter is a copy of the schema that is provided (and potentially customized) in the Media Service. Please use your custom/latest version of this file to use it for validations.