This fixer ran extremely slow for large DBs due to mikro-orm having to
handle many thousands of entities, which it is not primed to do.
Instead, we can generate the updates using existing DB data with direct
updates. However, we will still have to implement a background updater
if we need to pull information from the media source (i.e. it is missing
from the DB)
This commit introduces a new HLS stream mode, modeled off of the great
"HLS segmenter" in ErsatzTV. It also introduces the concept of "channel
stream modes" which are customizable in the UI.
Overall this commit performs a ton of refactoring and cleanup around the
streaming pipeline, consolidating a lot of logic and moving things
around to make all parts of it more flexible, understandable, and
extensible in the future.
This commit features a major rewrite and refactoring of the streaming
pipeline and class hierachy in Tunarr. It introduces new default
streaming modes in an attempt to stabilize transitions between program
streams, reuse underlying resources (e.g. ffmpeg processes), reduce the
complexity of interaction between various streaming class components,
and increase flexibility for future development.
Some notable changes:
* The default MPEG-TS stream is now backed by HLS. IPTV and Web
streaming sessions are shared
* Transcode readahead implementation. This should create smoother
streams. For FFMPEG 7 (and future versions) we use the native
"readrate_initial_burst" option. For earlier versions we use an
artificial readahead. **NOTE**: FFMPEG <7 artificial readahead might
have unintended side effects (e.g. loudness normalization issues) and
as such FFMPEG 7+ is recommended
* Separation between building an ffmpeg process, spawning the process,
and the notion of a 'transcode session'. These are all separate
classes now with narrow concerns
* Other misc refactoring to cleanup code and remove leftover superfluous
logic from DTV
This commit includes a huge amount of changes, including support for
adding Jellyfin servers as media sources and streaming content from
them.
These are breaking changes and touch almost every corner of the code,
but also pave the way for a lot more flexibility on the backend for
addinng different sources.
The commit also includes performance improvements to the inline modal,
lots of code cleanup, and a few bug fixes I found along the way.
Fixes#24
Includes:
* Batching inserts to external IDs
* Only saving critical external IDs (plex rating key) synchronously on
lineup update. Non-critical external ID inserts are pushed to
background tasks
* Program grouping / hierarchy updating has been pushed to a background
task
* Optimized verison of content program converter function that is
synchronous. This is much more performant for 1000s of items,
specifically when no async actions (selecting extra data) is
necessary. We saw a ~89% reduction in time spent here (>1s to ~30ms)
* A few places were selecting all channel programs when they only needed
to select the channel
* Only select program IDs at the outset of lineup update, instead of all
program details. We only need the IDs for relation diffing
Still some things we haven't solved:
1. There is non-trivial overhead in the mikro-orm ORM framework to
mapping 1000s of selected entities. In a 2k program channel case,
when loading all necessary entities (relations), we can see ~6k
entities loaded by the framework. The select from the DB only takes
about 800ms, but the entity mapping step can take >3s in this case.
One solution here is to use a simpler library for these super large
selects (kysely?)
2. It's probably overkill to have both Zodios on the client and then Zod
checking incoming types on the server. On huge request, this can add
~100ms or more (server side) to requests as Zod validates incoming
requests against the schema
3. We should think about replacing Zod on the server side with JSON
Schema. There are converters out there. We have a lot invested in
Zod, so a converter would be a good first step.
4. There's clearly some I/O contention happening in certain situations
... background tasks that query DB or Plex, getting responses to flow, logging, etc. I think most of it is DB-related.
5. Unclear if there is any actually _fast_ way to insert the amount of
data we are currently generating for large channels.
Latest migration incorrectly created the conditional indexes, which
potentially allowed duplicates into the table. There was also a bug
before the latest migration that allowed dupes, which would make the
migration fail when reinserting the data into the new table. Easiest
solution here, since this data 1. still largely exists for programs in
the program table, and 2. is largely unused, is to completely drop the
table, recreate it correctly, and let the BackfillProgramExternalIds run
and fix everything up.
Some verification:
No external_key mismatches after migration
> sqlite> select eid.*, p.title from program_external_id eid join
program p on eid.program_uuid = p.uuid where p.external_key !=
eid.external_key and eid.source_type = 'plex';
There are only 3 programs that we were not able to find rating keys for
however, this means that the programs will be "broken" anyway and will
have to be readded in the UI, since their rating_keys are dangling in
Plex.
> sqlite> select count(p.uuid) from program p left join
program_external_id eid on p.uuid = eid.program_uuid where eid.uuid is
null;
count(p.uuid)
-------------
3
Fixes#537
* This is a mess
* Fixes
* IDK about this
* Checkpoint
* Checkpoint
* Save external_ids for programs using Plex provided "Guid" field
This is a rather large change to a do a relative simple thing: save IDs
for external sources, such as IMDB/TMDB/TVDB, using IDs provided from
Plex.
It does a few things:
* Creates a delineation in the program_external_id table between
"single" and "multi" external IDs. Single IDs are global, while multi
IDs are scoped to some external source (e.g. Plex server)
* Starts the migration for Programs to be "source agnostic". Eventually,
Programs should be a logical entity that has >=1 "source". An ideal
end state would be for a Tunarr instance to have multiple "sources"
where Programs are deduped across them, with the ability to pick the
source for each program per channel, have source fallbacks, and search
for desired settings (e.g. audio language, subtitles) _across_ sources
for a given program. Additionally, saving non-streaming-source external
IDs opens up another avenue of metadata collection for Tunarr
* Implements uniqueness in the program_external_id table using partial
indexes. This made for a pretty messy change and I'm not super happy
with it.
* Migrates some queries away from the source/external_key fields on
Program to joins against the program_external_id table
* Implements some backfill mechanisms for these IDs
* Attempt 1: Include package.json in the final binary
* Bundle native deps alongside; bundle migrations in main bundle file; zip everything up
* Upload the zip to the edge release
Motiviation:
Adding this table is a bit of a future-proofing mechanism. We want to
decouple the notion of a 'program' from its external references.
Currently, the only external reference Tunarr knows of is 'plex'.
Eventually, we will add different sources for programming. In theory, a
program could exist in multiple sources. And not only in multiple media
sources, but on other sites that contain more metadata, such as IMDB and
TMDB. The new DB table generalizes Program to have a 1:M relationship
with external references. Additionally, this allows us to maintain
stable and _unstable_ IDs for various sources (ex. plex rating key)
without losing metadata sourcing for a Program.
Changes:
* Migration to add `program_external_id` table
* Backfill fixer task to fill in details from existing programs in the
DB
* Changes to program DB save path to save to new table and to start
saving Plex GUID
* Self-heal when Plex rating key changes. This code is triggered when
starting a stream for an item. If we have a Plex GUID available, we
will attempt to heal the item in the DB and start the stream
* Use the new table in the stream item calculator
TODO:
* Start saving Plex GUID for program_groupings, which have a separate
external key table
* Start saving other external IDs for programs, such as IMDB ID and TMDB
ID
Initially we were trying to mimic the dizqueTV data model. Programs in
DTV saved both Plex's 'ratingKey' and 'key' values. These are
effectively the same thing, from what I can tell, with the 'key' just
being a 'ratingKey' prefixed by '/library/metadata/'.
The issue in this bug was that on the frontend, we were creating
'uniqueId' using the 'ratingKey' and on the backend, we were creating it
using the 'key'. We would end up with LineupPrograms without a proper DB
Program reference, which would break various things on the save path.
This change does a few things:
1. Standardizes on Plex ratingKey as the 'externalKey' on a Program
2. Deprecates the plexRatingKey field on the Program, since eventually
these will not be Plex-specific anyway and it just causes more
confusion
3. Updates references to the aforementioned fields
Next steps:
1. Use branded types for additional string type safety - we have a lot
of srtings flying around!
* This is a nightmare
* Checkpointing.... getting closer
* First cut - packaging the server in a docker container
* Remove busted bundles
* Minify build
* Some common commands for building - we're going to look into proper monorepo solutions soon
* Remove dependency on serve-static
* Add web serving, full-stack docker target, and Nvidia container support
* Remove test action graph for now
