[SYNPY-1893] Add search index tutorial - #1447
Conversation
|
|
||
|
|
||
| # --8<-- [start:search_configuration] | ||
| def create_search_configuration() -> str: |
There was a problem hiding this comment.
Is this function used anywhere in the tutorial?
There was a problem hiding this comment.
It isn't executed by design, because of the nuance described in the markdown file
|
|
||
|
|
||
| # --8<-- [start:apply_search_configuration] | ||
| def create_index_with_configuration(search_configuration_id: str) -> SearchIndex: |
There was a problem hiding this comment.
Is this function used anywhere in the tutorial?
There was a problem hiding this comment.
I'm still thinking whether this "sage only" section should live here or if it should actually live within the "explanations" folder as an "advanced topic" since this isn't "open to the world"
There was a problem hiding this comment.
I would elect to have both. Where the tutorial touches on the concept, but then links to a more in depth explanation that talks about it more
There was a problem hiding this comment.
I'll create a ticket for the "advanced topic" for later, but it would be good for samia and I to coordinate on what that looks like
| "type": "custom", | ||
| "tokenizer": "standard", | ||
| "filter": [ | ||
| "lowercase", |
There was a problem hiding this comment.
If it isn't mentioned in the tutorial a not about casing should be included. In all of the synonyms that I created for our portal I lowercase the synonyms inside the SynonymSet. The reason for this is that when this default_search filter pipeline is used it will lowercase the user's search terms before applying synonyms. If the synonyms themselves are not lowercase then it would not match to a synonym and expand it.
| ``` | ||
| </details> | ||
|
|
||
| ### Cursor paging with `search_after` |
There was a problem hiding this comment.
search_after also needs to be used when paginating beyond 10k rows. That was what I recall from the opensearch docs that the offset working right.
There was a problem hiding this comment.
Do you think that we should just have the search_after? or is having both pagination and search after critical?
There was a problem hiding this comment.
Using just search_after would be better if the framing is "Enumerate every result", but if the framing is I want the 504th result, then offset would be used.
Both can be kept because they are genuinely different use cases
| ## Advanced: Tune matching with synonyms and analyzers | ||
|
|
||
| !!! warning "Restricted and permanent" | ||
| Creating and updating the following resources is restricted to Sage Bionetworks employees, |
There was a problem hiding this comment.
Everything including SearchIndex creation is gated behind the Sage Bio employee restriction
There was a problem hiding this comment.
I see, so there's the setup, and then there's the usage. Perhaps I'll shuffle things around so that the setup is first, then the usage of the indexes is second?
There was a problem hiding this comment.
Sounds fine to me. The querying part is free to use by everyone whereas the setup is sage only.
BryanFauble
left a comment
BryanFauble
left a comment
There was a problem hiding this comment.
There are a few suggested changes to improve this, but I'll approve now because it's shaping up well.
Feel free to ignore the build status on this PR since it isn't affecting anything tested.
| * a [Table][synapseclient.models.Table] | ||
| * an [EntityView][synapseclient.models.EntityView] | ||
| * a [DatasetCollection][synapseclient.models.DatasetCollection] | ||
| * a [MaterializedView][synapseclient.models.MaterializedView] |
There was a problem hiding this comment.
@BryanFauble Are VirtualTable or Datasets supported?
| GET, so ask for the largest `size` you can use rather than walking a big index in small | ||
| pages. | ||
|
|
||
| ## Advanced: Tune matching with synonyms and analyzers |
There was a problem hiding this comment.
@BryanFauble what does the index.query functions look like in this tutorial if the following is all set up already?
would we still have to fine-tune the query parameters like we do in sections 3,4,5,6,7?
| </details> | ||
|
|
||
| Hits come back ranked by relevance, and each one carries its score on | ||
| [`hit.score`][synapseclient.models.SearchHit] along with the `row_id` and `row_version` |
There was a problem hiding this comment.
Not returning row_id and row_version, but look at hit score
| </details> | ||
|
|
||
| **Note**: Highlighting, like relevance scoring, depends on the column being indexed as | ||
| analyzed text. Step 9 covers how to control that with a |
| * `must` clauses have to match and **do** contribute to the relevance score | ||
| * `filter` and `must_not` clauses have to match (or not match) but **do not** affect | ||
| the score — use these for hard constraints like a numeric cutoff | ||
| * `should` clauses boost the rows that match them without excluding the rows that don't |
There was a problem hiding this comment.
More explanation of how this is actually working would be great.
Problem:
No search index tutorial
Solution:
Added tutorial
Testing:
https://synapsepythonclient--1447.org.readthedocs.build/en/1447/