Improve Meilisearch Index upgrade workflow (reindex_studio)

@regisb

Currently the reindex_studio command is required before you can use any of Studio's search features. In addition, because the index format has changed between Teak and Ulmo, it will be necessary for operators to rebuild the search index during the upgrade to Ulmo.

The functionality to create/update the Meilisearch index schema (for Studio, but also for courseware search, forum, etc.) should be moved to database migrations - see @regisb 's suggestions in the comments below.
The reindex_studio command currently requires --experimental. Now that it's been used in two full releases (Sumac & Teak), the experimental flag should be removed.
The reindex_studio command can take a long time to run (hours on large instances), so Tutor should not necessarily run a full synchronous index rebuild during its upgrade script. See this bug report for example.
- the simplest approach is just to run reindex_studio --initial during the upgrade, (i.e. create a blank index with the new format - search results will be empty but the search code will work without error), and tell the user to run reindex_studio manually to fill in the content (which will be fast if they have little content, or could take ~ a day on very large instances).
- a more sophisticated approach could be to check if the instance has < 50 courses + libraries and in that case run the full reindex_studio to completion, but if it has more, then just run reindex_studio --initial and tell the user to complete it on their own time.
- an even better approach would be to just create the blank index (reindex_studio --initial) during init/upgrade and then kick off an async celery task to populate it incrementally in the background.
The reindex_studio command has two modes: "normal" (create a new index, populate it, then cut over to it once it's fully populated. If interrupted, must start over again.) or "incremental" (start or continue populating the current index. If interrupted, can be resumed anytime). The "normal" mode should be removed and the "incremental" mode should be the only way of populating the index. This does mean that there could be a period of time during upgrades when Studio search results are incomplete, but it's worth it for simplicity. Also the incremental mode indexes newest content first, so should quickly fill in the content that users most care about.
In coordination with Tutor, we should upgrade to the latest version of Meilisearch - especially if features like "Reduce indexing memory usage", "update documents with function", and contains filters are stable because those will be valuable features for us. (Not to mention High Availability, if/when it's implemented.)
- Note that upgrades to Meilisearch 1.13+ are much easier than prior versions (prior versions required backup/restore, new versions just require an upgrade command like Mongo).

In the future we should also think about how to consolidate our search index management so that one command manages all the search indexes, rather than studio and LMS being separate. (Not to mention teams, forum, etc.). But this is blocked until we know if we're going to be supporting TypeSense or any other alternative to Meilisearch or not.

@ormsbee

@ormsbee @kdmccormick @jmakowski1123 This is a (hopefully small and easy) backend story related to Studio search that I think is important to get done before Ulmo as well.

Have you considered moving the reindex_studio to a database migration script? I mean: we should keep the manage.py management command for manually running that step when necessary, but if the reindex command should be run at precise moments in time, then we would be better off making that command part of the ./manage.py cms migrate workflow.

My reasoning is as follows: as a Tutor maintainer, if I need to run reindex_studio --initial at some moments, but not other, then I need to keep track of whether that command was ever run before. Something like the following:

Was the index ever created? If not, then run reindex_studio --initial. If yes...
... does the index have the latest structure? If not, then run reindex_studio --initial.

As a Tutor maintainer I would be much better off if the meilisearch migrations were integrated with other regular migrations. What do you think?

I definitely considered that and in many ways I would prefer that as well, but the problem is that (depending on how we implement it), it creates the potential for "migrations" that either (1) take a very long time to run, or (2) erase your entire search index and require you to manually "re-index" stuff later. And neither of these are expected outcomes of running the migrate command.

If there is some way that our migration could kick off an async celery task to re-populate the index after it is modified to the new format, then that could work. But again I don't think people expect migrate to launch celery tasks, so that also seems unexpected.

Open to ideas or things that I may have missed.

Oh right, there is also the challenge that changing the Meilisearch index configuration depends on Meilisearch being enabled or not. And in general I think migrations that depend on the state of the instance's config at the time the migration was run can be problematic. (e.g. imagine if someone has the Meilisearch setting off, runs migrations, then turns the setting on - now their index doesn't exist but all migrations are "already applied")

So while I like the simplicity of setting the index config during a migration, I would personally be more comfortable doing that only if we made Meilisearch a dependency of the overall platform and knew that everyone would be using it; if we're going to be supporting "Meilisearch or Elasticsearch" or "Meilisearch or TypeSense" or "no search features" as options, then I think a migration is less appropriate.

I partly agree with your reasons not to include meilisearch migrations in the migrate command. I suggest the following:

Perform index structure migrations in model migrations: these migrations should only concern the index structure, and not the index content, such that migrations don't take a very long time to run. (is this possible at all?)
Move these migrations to a "meilisearch" Django application, which is added to INSTALLED_APPS only if meilisearch is enabled. Thus, migrations are run only if the feature is enabled.
Allow the reindex_studio command to index course content asynchronously (by triggering celery tasks), and only for courses that really need it. (i.e: for courses that don't have any content, if that makes sense?)

I'm not sure if my suggestions make sense, as I didn't look at the detailed implementation of reindex_studio.

@regisb

@regisb Good ideas. I think we can find something like that that will work.

Perform index structure migrations in model migrations: these migrations should only concern the index structure, and not the index content, such that migrations don't take a very long time to run. (is this possible at all?)

It's sort of possible. We can definitely create and update the (empty) index structure with migrations, but modifying the configuration of an index that's already full of data can be very slow (according to the docs). So in the case where the index already exists and is populated with data and we need to change its configuration, it's best to wipe it out and create a new empty index with the right config. (Otherwise the migration could take a long time, causing the same problem we're trying to solve here.) That said, I have never actually tested how long this takes, especially compared to re-indexing all the content.

Move these migrations to a "meilisearch" Django application, which is added to INSTALLED_APPS only if meilisearch is enabled. Thus, migrations are run only if the feature is enabled.

Oh, that should have been obvious to me. That makes a lot of sense.

Allow the reindex_studio command to index course content asynchronously (by triggering celery tasks), and only for courses that really need it. (i.e: for courses that don't have any content, if that makes sense?)

I think we still have to do all courses, in principle. But we do start with the most recent courses first, to reduce the downtime. People are much more likely to care if a newly created course is temporarily missing from the search index compared to one that's ten years old.

This is on the release planning list for Ulmo. Is that possible / happening or should I take it off?

I think it's very important and would really like to see it happen, but nobody has been able to commit to doing it yet.

This likely won't make the Ulmo cut, but consider for backport.

@bradenmacdonald

@bradenmacdonald @regisb

I’ve tried to incorporate the suggestions discussed above. This is an initial attempt and still needs some refactoring, but I’d like your thoughts on the overall architecture first.

In this setup, any Meilisearch schema change would require a new Django migration to update index structure.(This can be improved)

Does this approach align with what you had in mind, or would you prefer anything else?

@jawad-khan

@jawad-khan Thanks! I think that approach could work for Ulmo as it's mostly a simplification of the existing code. I commented with some suggestions to actually include index setup as part of the command.

We need to try the "index update" option on some large datasets and see if it actually takes a long time for Meilisearch to change an index in-place or not. (@samuelallan72 can you help with that? since we have a perfect dataset ready to go). Then we can decide if future index changes will be implemented as django migrations as discussed above, or if we should stick to something like I suggested on that PR.

@bradenmacdonald

@bradenmacdonald absolutely, this sounds perfect for the deployment we have already. 👍

@kdmccormick

@kdmccormick @ormsbee We're working on implementing this, and @farhaanbukhsh will post an updated plan / prototype shortly. Do we just need to support Meilisearch, or is Typesense or Elasticsearch also in scope?

Since this is Studio-only, it's okay to stick with just Meilisearch for now.

Thank you.

@bradenmacdonald

@bradenmacdonald Thanks. cc @samuelallan72 @ormsbee

This is the summary of the plan that we have:

When MEILISEARCH_ENABLED = True, platform setup should ensure that the Studio Meilisearch index is ready to use from the start: the index must exist, be correctly configured, and be populated. The implementation should split this into two parts.

Part 1

We handle index creation and configuration synchronously during setup, since these are relatively cheap operations. To support that, we introduce a dedicated meilisearch_migration app whose only responsibility is to own Django migrations related to Meilisearch.

Extend init_index very similar this change in @samuelallan72's test:

it should

create the index if it does not exist
inspect the current index settings, and
reconfigure the index when there is a mismatch in attributes such as distinct, filterable, searchable, or ranking rules.

It should also detect primary key mismatches; unlike other settings changes, a primary key change should force the index to be dropped and recreated. A separate reset_studio_index path can exist for explicit full resets, but the migration flow should use the non-reset configuration path.

Migration failures should surface clearly rather than being silently suppressed so operators know when setup did not complete correctly.

Part 2

The second phase is index population, which is the expensive part and should happen asynchronously. The reindex_studio flow should default to incremental population so the --incremental flag is no longer required, and it should clean up the incremental index table after a successful run.

Once that is in place, the Tutor initialization flow can be updated so it triggers population via the Celery-backed reindex_studio task instead of only initializing the index. That gives us asynchronous execution, task logging, and better operational visibility during setup.

Progress can be observed through Celery logs, but the Meilisearch task list should be treated as the primary source of truth for verifying whether indexing completed successfully.

As follow-up work, it would be useful to document how operators can inspect task status, and a Meilisearch dashboard could be considered later.

Task List

create the meilisearch_migration app
make configure_studio_index able to validate existing index settings
add support for primary key drift detection and full index recreation when needed
wire configure_studio_index into a Django RunPython migration
make incremental mode the default behavior for reindex_studio
clean up the incremental table after successful population
update Tutor init flow to trigger asynchronous population through Celery
test creation, reconfiguration, reset, migration flow, and async population end to end

create the index if it does not exist

There's not just one index though. There are six:

comments and comment_threads (I think those are out of scope, and the forums service already manages them)
course info index - browse for courses, powers http://local.openedx.io:8000/courses, part of edx-search
course team index - ?????
course content index - LMS courseware content, part of edx-search
studio content index - CMS courseware & library content, not edx-search

We really need to plan a refactor of edx-search and provide a mechanism for registering these use cases in some way. I think for now it's OK to have a meilisearch_migration that knows about all of these things, but please think about the long-term approach. I think ideally, edx-search would provide some mechanism for apps to register as having an index, or at least some helpers for creating "index migrations" in each app.

reconfigure the index when there is a mismatch in attributes such as distinct, filterable, searchable, or ranking rules.

So what should a developer do when they change one of these things? Will they need to create a new migration? If so, why do we need code to inspect the index settings; we could just include a list of changes in each migration.

Part 2

Sounds good to me.

Progress can be observed through Celery logs, but the Meilisearch task list should be treated as the primary source of truth for verifying whether indexing completed successfully.

Viewing the IncrementalIndexCompleted could be a better way to recommend since it would work with all search engines, e.g. Typesense, in the future?

@bradenmacdonald

There's not just one index though. There are six:

Thank you @bradenmacdonald for pointing that out, we tried to limit the scope of work to just openedx-platform and not do much of edx-search refactor, so that when in future we need to do the refactor, this work can act as a foundation stone and learning on the path we should take.

course team index - ?????

This made me curious, so I dug into it and found out that it is used under "Teams" tab in learning MFE

Implementation PRs:

https://github.com/openedx/openedx-platform/pull/9472/changes

And this is not a part of edx-search.

Hence, this can be a good case study for us here to migrate it to this workflow. But the work here is mainly concentrated on studio content index and if we have the time left we can do that same for course teams.

I think for now it's OK to have a meilisearch_migration that knows about all of these things, but please think about the long-term approach. I think ideally, edx-search would provide some mechanism for apps to register as having an index, or at least some helpers for creating "index migrations" in each app.

Ideally, I think all the changes related to searches and indexes should be a part of edx-search. This forms a clear segregation that anything to deal with searches and indexes should be done through edx-search but since we already have to many wires here, this effort can help us to untangle the wires and then move those modules into edx-search.

If you see we are redoing/duplicating few things for example creating index is already done in edx-search/meilisearch, and we do it in edx-platform as well.

One good pattern I see here is if we can unify the use of all tools elastic_search/typesense/meilisearch, and create an abstraction over them, where we expose only a selective api from edx-search for openedx-platform to communicate with these databases. This is a draft idea I have about refactoring edx-search and openedx-platform. This needs a lot of polishing and deeper thought.

So what should a developer do when they change one of these things? Will they need to create a new migration? If so, why do we need code to inspect the index settings; we could just include a list of changes in each migration.

From what I can think, Django migration is just a tool to run scripts so the developer doesn't have to introduce a new migration at all. The script that is being triggered through migration uses the same constants that are used for querying, so in case any of these attributes change, this script automatically runs a migration/updation of the index by checking the current state of attributes and that is why the checks are required.

I have added better clarification in #36868 (comment)

Viewing the IncrementalIndexCompleted could be a better way to recommend since it would work with all search engines, e.g. Typesense, in the future?

👍🏾 makes sense.

@farhaanbukhsh

@farhaanbukhsh

From what I can think, Django migration is just a tool to run scripts so the developer doesn't have to introduce a new migration at all. The script that is being triggered through migration uses the same constants that are used for querying, so in case any of these attributes change, this script automatically runs a migration/updation of the index by checking the current state of attributes and that is why the checks are required.

A Django migration will only be run once though. So we would need a new migration whenever something changes, otherwise the changes will not be applied when deployed.

A Django migration will only be run once though. So we would need a new migration whenever something changes, otherwise the changes will not be applied when deployed.

Sorry for the misunderstanding. I may have made statements that conveyed different meanings let me try to clarify this. :)

From what I can think, Django migration is just a tool to run scripts so the developer doesn't have to introduce a new migration at all. The script that is being triggered through migration uses the same constants that are used for querying, so in case any of these attributes change, this script automatically runs a migration/updation of the index by checking the current state of attributes and that is why the checks are required.

I was trying to use Django migration to run a single script each time, which takes care of any change in the attributes, ranking rules and primary key. This script has to be triggered every time any changes are made in the index. Hence, in theory, we might be creating a new migration with the same content.

And while trying to answer you, I was thinking if we could just run one migration without marking it complete, and that made me land up on post_migrate signal, which I think is better suited for our requirement than Django migrations. The post_migrate runs at the end of the migrate command, even if no migration is applied. I think our use case is much suited to this approach rather than going with Django migration, where we can catch the post_migrate signal and trigger the script to see if MEILISEARCH_ENABLE is true and carry on the checks as mentioned above.

This also has a benefit where later we can move the signal handler and the script to do the checks and evolve the index to edx-search.

@farhaanbukhsh

@farhaanbukhsh Yes, we need to do a complete refactor of edx-search at some point, so please add any ideas you have to that issue. But for now, we just need a better way to work with the system we have, to help make index management more automatic.

The post_migrate runs at the end of the migrate command, even if no migration is applied. I think our use case is much suited to this approach rather than going with Django migration, where we can catch the post_migrate signal and trigger the script to see if MEILISEARCH_ENABLE is true and carry on the checks as mentioned above.

Nice. If all the index changes can be auto-detected, then that sounds like a much better approach, yep.

But if we ever have to manually write index migrations like "rename this index field", or "create this non-null field with default value X", then I don't think it would work? Unless we have some other place to make the script able to handle those cases.

@bradenmacdonald

@bradenmacdonald this is still something I am testing but it's up for review, the phase 1 of the plan is completed you can have a look #38384 and review it.

@bradenmacdonald

@bradenmacdonald Thank you for the amazing reviews and pushing this work. Now I think we have overhangio/tutor#1374 pending to get this in Tutor 😄

perf: wrap reindex_course loop in modulestore().bulk_operations (#39134)

Navigation Menu

Repository navigation

Improve Meilisearch Index upgrade workflow (reindex_studio) #36868

10 remaining items

Part 1

Part 2

Task List

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Improve Meilisearch Index upgrade workflow (reindex_studio) #36868

Description

Activity

bradenmacdonald commented on Jun 6, 2025

regisb commented on Jun 9, 2025

bradenmacdonald commented on Jun 9, 2025

bradenmacdonald commented on Jun 9, 2025

regisb commented on Jun 12, 2025

bradenmacdonald commented on Jun 12, 2025

sarina commented on Sep 4, 2025

bradenmacdonald commented on Sep 4, 2025

ormsbee commented on Oct 17, 2025

jawad-khan commented on Nov 12, 2025

bradenmacdonald commented on Nov 18, 2025

samuelallan72 commented on Nov 19, 2025

10 remaining items

bradenmacdonald commented on Apr 1, 2026

ormsbee commented on Apr 1, 2026

farhaanbukhsh commented on Apr 9, 2026

Part 1

Part 2

Task List

bradenmacdonald commented on Apr 10, 2026

farhaanbukhsh commented on Apr 12, 2026

samuelallan72 commented on Apr 13, 2026

farhaanbukhsh commented on Apr 13, 2026

bradenmacdonald commented on Apr 13, 2026

farhaanbukhsh commented on Apr 20, 2026

farhaanbukhsh commented on Apr 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions