Maestro Dev Documentation

CONTRIBUTING.md

@@ -0,0 +1,103 @@
+# Contributing to Overture
+
+We appreciate your interest in contributing to the Overture project. We are the Genome Informatics Software Engineering team from Ontario Institute for Cancer Research. At OICR we develop new software, databases and other necessary components to store, organize and compute over the large and complex datasets being generated by our cancer research programs. Embodying OICR's values of collaboration and community, we are firm believers in open-source and open-science. As such we strongly believe in the collective power of expertise and shared resources.
+
+## Code of Conduct
+
+By participating in this project, you are expected to abide by our [Code of Conduct](https://docs.overture.bio/community/code-of-conduct). Please take the time to read it carefully before contributing.
+
+## Get Involved
+
+**Getting Started:** Our primary platform for community support, feature requests, and general discussions is GitHub Discussions. This allows us to keep all conversations in one place and make them easily searchable for future reference.
+
+
+**Community Support:** Our primary platform for community support, feature requests, and general discussions is [GitHub Discussions](https://github.com/overture-stack/docs/discussions). This allows us to keep all conversations in one place and make them easily searchable for future reference.
+
+- **Getting Help:** If you need assistance with Overture, please create a [new discussion in our support category](https://github.com/overture-stack/docs/discussions/categories/support).
+  - Before creating a new discussion, please search existing discussions to see if your question has already been answered.
+- **Feature Requests & Proposals:** We love hearing your ideas for improving Overture! Before making a feature request, please check our [**feature roadmap**](https://github.com/orgs/overture-stack/projects/11/views/1) to see if your idea is already planned.
+  - If your idea isn't on the roadmap, feel free to [**submit a feature request**](https://github.com/overture-stack/docs/discussions/categories/ideas) by creating a new discussion in our Ideas category 
+- **Report a Potential Bug:** We use GitHub Issues primarily for tracking confirmed bugs and ticketing development tasks. If you come across a potential bug or issue, please first post it to our [**GitHub support discussion forum**](https://github.com/overture-stack/docs/discussions/categories/support).
+  - This allows us to confirm the issue and gather more information if needed. If we determine that further development is required, we will create and tag you into a GitHub Issue from your discussion post.
+
+## Our Development Process
+
+We use GitHub issues and pull requests for communication related to code changes. 
+
+### Branch Organization
+
+We use the following standard branches:
+
+- `main` is for stable production code
+- `develop` is the integration branch for new features
+- `feature/<name>` for feature branches
+- `release/v<version>` for release branches
+- `hotfix/<name>` for hotfix branches
+
+## Pull Requests
+
+### Submitting a Pull Request
+
+We welcome and encourage pull requests from the community. To submit a pull request, please follow these steps:
+
+1. **Fork the Repository**: Fork the Overture repository on GitHub.
+2. **Clone Your Fork**: Clone your forked repository to your local machine.
+3. **Create a New Branch**: Create a new branch for your changes. Use lowercase and hyphens (e.g., `feature/user-authentication`). Include ticket/issue numbers when applicable (e.g., `feature/PROJ-123-user-authentication`).
+4. **Make Your Changes**: Implement your changes and commit them to your branch. Write clear, concise commit messages in present tense (e.g., "Add feature" not "Added feature"). Reference issue numbers in commits when applicable.
+5. **Push Your Changes**: Push your changes to your forked repository.
+6. **Submit a Pull Request**: Open a pull request against the main repository.
+
+### Best Practices
+
+1. **Keep PRs as small as possible:** Focus on one feature or bug fix per pull request. Break large changes into smaller, more manageable pieces making it easier for reviewers to understand and approve your changes.
+
+2. **Use descriptive titles:** Start with a verb (e.g., "Add", "Fix", "Update", "Refactor"), briefly summarize the main purpose of the PR and include the issue number if applicable (e.g., "Fix user authentication bug (#123)").
+
+3. **Describe how you tested it:** Explain the testing process you followed and mention any new automated tests you've added.
+
+4. **Provide a clear description:** Explain the purpose of your changes and list the main modifications you've made. Mention any potential side effects or areas that might need extra attention.
+
+5. **Link related issues:** Reference any related issues or pull requests. Use GitHub keywords to automatically link issues (e.g., "Closes #123", "Fixes #456").
+6. **Keep the PR's branch up-to-date:** Regularly rebase your branch on the latest main branch and resolve any merge conflicts promptly.
+
+7. **Respond to feedback:** Be open to suggestions and willing to make changes. Address all comments from reviewers. If you disagree with a suggestion, explain your reasoning politely.
+
+8. **Include documentation updates:** If your changes affect user-facing features, update or create and issue detailing the relevant changes need to the documentation. Where appropriate include inline comments for complex code sections.
+
+10. **Be patient:** Reviewers will likely be unable to respond immediately. However, feel free to follow up politely if you haven't received feedback after a reasonable time.
+
+### Using Draft Pull Requests
+
+Draft Pull Requests are an excellent way to document work in progress and facilitate early feedback. Use them to:
+
+- Organize your thoughts and process
+- Share early work and ideas with the team
+- Get feedback on implementation approaches before finalizing code
+- Track progress on long-running features
+
+Guidelines for Draft Pull Requests:
+
+1. **Creation**:
+   - Open a pull request and select "Create draft pull request"
+   - Clearly mark the title with [WIP] or [DRAFT] prefix
+2. **Description**:
+   - Outline the current state of the work
+   - List planned tasks or improvements
+   - Highlight areas where feedback is specifically needed
+3. **Updates**:
+   - Regularly update the description or provide comments following commits with progress notes
+- Use task lists (using `- [ ]` in Markdown) to track completion of sub-tasks
+4. **Collaboration**:
+   - Encourage early feedback and discussion
+   - Use the pull request comments for design discussions
+5. **Finalization**:
+   - Complete all planned work and address feedback
+   - Update tests and documentation
+   - Click "Ready for review" to move out of draft state
+
+### Merging a Pull Request
+
+- Ensure all CI checks pass
+- Obtain the required number of approvals
+- Use the project's specified merge strategy (Typically squash and merge)
+- Delete the source branch after merging if no longer needed

README.md

@@ -1,116 +1,56 @@
-<h1 align="center">Maestro</h1>
-<p align="center">Organize geographically distributed data stored in Song and Score, with a single, configurable index.</p>
-<p align="center">
-    <a href="https://github.com/overture-stack/maestro">
-        <img alt="Under Development" 
-            title="Under Development" 
-            src="http://www.overture.bio/img/progress-horizontal-RC.svg" width="320" />
-    </a>
-</p>
-
-[![Documentation Status](https://readthedocs.org/projects/maestro-overture/badge/?version=latest)](https://maestro-overture.readthedocs.io/en/latest/?badge=latest)
-[![Slack](http://slack.overture.bio/badge.svg)](http://slack.overture.bio)
-
-## Documentation:
-Documentation is hosted on :
-- github pages: https://overture-stack.github.io/maestro/
-- read the docs: https://maestro-overture.readthedocs.io/en/latest/ (using mkdocs https://docs.readthedocs.io/en/stable/intro/getting-started-with-mkdocs.html)
-
-## Introduction
-Meastro was created to enable genomic researchers to enhance their Overture SONGs by building indexes, elastic search
-by default, that makes searching Analyses and Studies much more powerful and easier.
-
-## TLDR; 
-Skip down to the How to section it has the steps to get started.
-
-### Features:
-- Supports indexing from multiple metadata repositories (SONG).
-- Multiple indexing requests: analysis, study, full repository.
-- Event driven indexing.
-    - Integration with SONG to index published analysis and delete suppressed / unpublished analyses
-- Ability to Exclude analysis based on different Ids: Study, Analysis, Donor, Sample Or file.
-- Slack web hook integration
-
-### Design Goals:
-- Reactive
-    - Event driven
-    - Elastic
-    - Resiliency & Fault tolerance
-- Failure audit
-    - Dead letters queue for faulty messages to be retried later and reviewed.
-    - Human readable Error log
-- Runtime configurability
-- Extendable: separate domain from infrastructure & configuration
-
-## Technologies & libraries:
-- Java 11 - OpenJDK 11
-- Maven 3 (YOU NEED MAVEN 3.5+, if you don't want to use the wrapper or your IDE points to older version)
-- lombok
-- Spring boot 2
-    - Spring webflux & project reactor
-    - Spring retry
-    - Spring Cloud
-        - Streams (for Kafak integration)
-        - config client
-    - Spring boot admin + client
-- Elasticsearch 7+
-- Apache Kafka
-- resilience4j:
-    - retry module
-- vavr 
-- Testing libraries:
-    - Junit 5
-    - Mockito 2
-    - testcontainers
-    - Spring cloud contract wiremock
-
-## Structure
-The project is following the ports/adapters architecture, where the domain is completely isolated from external infrastructure
-and frameworks.
-- Two maven modules:
-    - maestro domain
-      - the core features and framework independent logic that is portable and contains the main indexing, rules, notifications
-      logic as specified by the business features. Has packages like:
-          - entities : contains POJOs and entities
-          - api: the logic that fulfills the business features
-          - ports: contains the interfaces needed by the api to communicate with anything outside the indexing context.
-
-    - maestro app:
-       - The main runnable (spring boot app)
-       - Contains the infrastructure and adapters (ports implementations) that is needed to connect the domain
-         with the outside world like elastic search, song web clients, configuration files etc.
-         It also has the Spring framework configurations here to keep technologies outside of the domain.
-
-# Dependencies:
-To Successfully run Maestro (as is) you need the following services to be deployed and configure it to use them:
-- [Elasticsearch](https://www.elastic.co/products/elasticsearch)
-- [Apache Kafka](https://kafka.apache.org/)
-- [SONG](https://github.com/overture-stack/SONG)
-
-you can check the sample docker compose files under `./run/docker-compose` for containerized versions of elastic & kafka.
-for SONG please check the SONG github repo [here](https://github.com/overture-stack/SONG/tree/develop/dev) 
-on how to run it with docker. Or you can run it as jar.
-
-## How to:
-- Swagger API access: 
-    - localhost:11235/api-docs
-
-Note: if you don't/can't use the Makefile, look inside it for the shell commands and replicate them.
-- Compile: `make` 
-- Test: `make test`
-- Package: `make package`
-- Run:
-    - Source:
-        - Development:
-            1. `make docker-start-dev` starts the infrastructure containers
-                - kafka
-                - elastic search
-                - other helper tools if you want like kafka rest proxy
-            2. `make run` OR start maestro (Maestro.java) from the IDE/cmd as a java application
-    - Docker:
-        - Repository: https://hub.docker.com/r/overture/maestro
-        - `make docker-start` starts maestro from a docker image along with all needed infrastructure
-    - helm:
-        - Repository: https://overture-stack.github.io/charts-server/
-        - see : https://github.com/overture-stack/helm-charts for instructions and the maestro folder for examples
-
+# Maestro
+
+Maestro enables researchers to enhance their Overture SONG deployments by building powerful search indexes for Analyses and Studies. It organizes geographically distributed data stored in Song and Score into a single, configurable index.
+
+</br>
+
+> 
+> <div>
+> <img align="left" src="ov-logo.png" height="60"/>
+> </div>
+> 
+> *Maestro is part of [Overture](https://www.overture.bio/), a collection of open-source software microservices used to create platforms for researchers to organize and share genomics data.*
+> 
+> 
+
+## Documentation
+
+Technical resources for those working with or contributing to the project are available from our official documentation site, the following content can also be read and updated within the `/docs` folder of this repository.
+
+- **[Maestro Overview](https://docs.overture.bio/docs/core-software/Maestro/overview)** 
+- [**Setting up the Development Enviornment**](https://docs.overture.bio/docs/core-software/Maestro/setup)
+- [**Common Usage Docs**](https://docs.overture.bio/docs/core-software/Maestro/setup)
+
+## Development Environment
+
+- [Java 11 (OpenJDK)](https://openjdk.java.net/projects/jdk/11/)
+- [Maven 3.5+](https://maven.apache.org/) (or use provided wrapper)
+- [VS Code](https://code.visualstudio.com/) or preferred Java IDE
+- [Docker](https://www.docker.com/) Container platform
+
+## Support & Contributions
+
+- For support, feature requests, and bug reports, please see our [Support Guide](https://docs.overture.bio/community/support).
+- For detailed information on how to contribute to this project, please see our [Contributing Guide](https://docs.overture.bio/docs/contribution).
+
+## Related Software 
+
+The Overture Platform includes the following Overture Components:
+
+</br>
+
+|Software|Description|
+|---|---|
+|[Score](https://github.com/overture-stack/score/)| Transfer data to and from any cloud-based storage system |
+|[Song](https://github.com/overture-stack/song/)| Catalog and manage metadata associated to file data spread across cloud storage systems |
+|[Maestro](https://github.com/overture-stack/maestro/)| Organizing your distributed data into a centralized Elasticsearch index |
+|[Arranger](https://github.com/overture-stack/arranger/)| A search API with reusable search UI components |
+|[Stage](https://github.com/overture-stack/stage)| A React-based web portal scaffolding |
+|[Lyric](https://github.com/overture-stack/lyric)| A model-agnostic, tabular data submission system |
+|[Lectern](https://github.com/overture-stack/lectern)| Schema Manager, designed to validate, store, and manage collections of data dictionaries.  |
+
+If you'd like to get started using our platform [check out our quickstart guides](https://docs.overture.bio/guides/getting-started)
+
+## Funding Acknowledgement
+
+Overture is supported by grant #U24CA253529 from the National Cancer Institute at the US National Institutes of Health, and additional funding from Genome Canada, the Canada Foundation for Innovation, the Canadian Institutes of Health Research, Canarie, and the Ontario Institute for Cancer Research.