diff --git a/README.md b/README.md index ff0dee76a..642758d1a 100644 --- a/README.md +++ b/README.md @@ -11,12 +11,20 @@ description: >- Welcome to OpenAdapt's developer API! Here you'll find all the documentation you need to get up and running with our software. -## Want to jump right in? +## Want to jump to user guide? -Get started on installing OpenAdapt by following our [quick-start.md](getting-started/welcome-to-openadapt/quick-start.md "mention") +Get started on installing OpenAdapt by following our [user-guide.md](getting-started/welcome-to-openadapt/user-guide.md "mention") -{% content-ref url="getting-started/welcome-to-openadapt/quick-start.md" %} -[quick-start.md](getting-started/welcome-to-openadapt/quick-start.md) +{% content-ref url="getting-started/welcome-to-openadapt/user-guide.md" %} +[user-guide.md](getting-started/welcome-to-openadapt/user-guide.md) +{% endcontent-ref %} + +## Want to jump to developer guide? + +Get started with testing/development with OpenAdapt by following our [developer-guide.md](getting-started/welcome-to-openadapt/developer-guide.md "mention") + +{% content-ref url="getting-started/welcome-to-openadapt/developer-guide.md" %} +[developer-guide.md](getting-started/welcome-to-openadapt/developer-guide.md) {% endcontent-ref %} ## Already Familiar? diff --git a/SUMMARY.md b/SUMMARY.md index 45897a954..653a7b92f 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -3,7 +3,11 @@ ## Getting Started * [🤖 Welcome to OpenAdapt!](README.md) - * [Quick Start](getting-started/welcome-to-openadapt/quick-start.md) + * [User Guide](getting-started/welcome-to-openadapt/user-guide.md) + * [Developer Guide](getting-started/welcome-to-openadapt/developer-guide.md) + +## Legacy Guide +* [User Guide](legacy/getting-started/quick-start.md) ## Reference diff --git a/getting-started/welcome-to-openadapt/assets/app-tray.png b/getting-started/welcome-to-openadapt/assets/app-tray.png new file mode 100644 index 000000000..e3cca5dee Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/app-tray.png differ diff --git a/getting-started/welcome-to-openadapt/assets/client.png b/getting-started/welcome-to-openadapt/assets/client.png new file mode 100644 index 000000000..716963493 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/client.png differ diff --git a/getting-started/welcome-to-openadapt/assets/dashboard-settings.png b/getting-started/welcome-to-openadapt/assets/dashboard-settings.png new file mode 100644 index 000000000..ac3056da2 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/dashboard-settings.png differ diff --git a/getting-started/welcome-to-openadapt/assets/dashboard.png b/getting-started/welcome-to-openadapt/assets/dashboard.png new file mode 100644 index 000000000..82ba4fca8 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/dashboard.png differ diff --git a/getting-started/welcome-to-openadapt/assets/developer-run-local.png b/getting-started/welcome-to-openadapt/assets/developer-run-local.png new file mode 100644 index 000000000..e7b899998 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/developer-run-local.png differ diff --git a/getting-started/welcome-to-openadapt/assets/download-page.png b/getting-started/welcome-to-openadapt/assets/download-page.png new file mode 100644 index 000000000..f36d84665 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/download-page.png differ diff --git a/getting-started/welcome-to-openadapt/assets/downloaded-zip.png b/getting-started/welcome-to-openadapt/assets/downloaded-zip.png new file mode 100644 index 000000000..29f8c1367 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/downloaded-zip.png differ diff --git a/getting-started/welcome-to-openadapt/assets/pip-install.png b/getting-started/welcome-to-openadapt/assets/pip-install.png new file mode 100644 index 000000000..62a7c6d14 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/pip-install.png differ diff --git a/getting-started/welcome-to-openadapt/assets/poetry-shell.png b/getting-started/welcome-to-openadapt/assets/poetry-shell.png new file mode 100644 index 000000000..6b78ed102 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/poetry-shell.png differ diff --git a/getting-started/welcome-to-openadapt/assets/post-recording-start.png b/getting-started/welcome-to-openadapt/assets/post-recording-start.png new file mode 100644 index 000000000..6ad03ee97 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/post-recording-start.png differ diff --git a/getting-started/welcome-to-openadapt/assets/record-prompt.png b/getting-started/welcome-to-openadapt/assets/record-prompt.png new file mode 100644 index 000000000..aaffa3196 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/record-prompt.png differ diff --git a/getting-started/welcome-to-openadapt/assets/recording-saved.png b/getting-started/welcome-to-openadapt/assets/recording-saved.png new file mode 100644 index 000000000..ed1202d6f Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/recording-saved.png differ diff --git a/getting-started/welcome-to-openadapt/assets/replay-list.png b/getting-started/welcome-to-openadapt/assets/replay-list.png new file mode 100644 index 000000000..614f0a01f Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/replay-list.png differ diff --git a/getting-started/welcome-to-openadapt/assets/replay-strategy-prompt.png b/getting-started/welcome-to-openadapt/assets/replay-strategy-prompt.png new file mode 100644 index 000000000..caeb4eae0 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/replay-strategy-prompt.png differ diff --git a/getting-started/welcome-to-openadapt/assets/scrubbing-processing.png b/getting-started/welcome-to-openadapt/assets/scrubbing-processing.png new file mode 100644 index 000000000..e91222628 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/scrubbing-processing.png differ diff --git a/getting-started/welcome-to-openadapt/assets/start-recording-commad.png b/getting-started/welcome-to-openadapt/assets/start-recording-commad.png new file mode 100644 index 000000000..a0242f96a Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/start-recording-commad.png differ diff --git a/getting-started/welcome-to-openadapt/assets/stop-recording.png b/getting-started/welcome-to-openadapt/assets/stop-recording.png new file mode 100644 index 000000000..7d836e8d9 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/stop-recording.png differ diff --git a/getting-started/welcome-to-openadapt/assets/visualize-1.png b/getting-started/welcome-to-openadapt/assets/visualize-1.png new file mode 100644 index 000000000..b4b653563 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/visualize-1.png differ diff --git a/getting-started/welcome-to-openadapt/assets/visualize-2-1.png b/getting-started/welcome-to-openadapt/assets/visualize-2-1.png new file mode 100644 index 000000000..7186f66a7 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/visualize-2-1.png differ diff --git a/getting-started/welcome-to-openadapt/assets/visualize-2.png b/getting-started/welcome-to-openadapt/assets/visualize-2.png new file mode 100644 index 000000000..296768479 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/visualize-2.png differ diff --git a/getting-started/welcome-to-openadapt/assets/visualize-3.png b/getting-started/welcome-to-openadapt/assets/visualize-3.png new file mode 100644 index 000000000..2ebe65310 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/visualize-3.png differ diff --git a/getting-started/welcome-to-openadapt/assets/visualize-list.png b/getting-started/welcome-to-openadapt/assets/visualize-list.png new file mode 100644 index 000000000..73978b11b Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/visualize-list.png differ diff --git a/getting-started/welcome-to-openadapt/assets/visualize-userapp-1.png b/getting-started/welcome-to-openadapt/assets/visualize-userapp-1.png new file mode 100644 index 000000000..04843a9a2 Binary files /dev/null and b/getting-started/welcome-to-openadapt/assets/visualize-userapp-1.png differ diff --git a/getting-started/welcome-to-openadapt/developer-guide.md b/getting-started/welcome-to-openadapt/developer-guide.md new file mode 100644 index 000000000..5f0c4aec1 --- /dev/null +++ b/getting-started/welcome-to-openadapt/developer-guide.md @@ -0,0 +1,226 @@ +--- +description: Working with OpenAdapt +--- + +{% hint style="info" %} +**Experiencing difficulty? Get support on our** [**Discord**](https://discord.gg/QKPuDqhDHF)**.** +{% endhint %} + +## Installation + +OpenAdapt **requires Python3.10**, you can download it [here](https://www.python.org/downloads/) or try our installer, which will download Python for you. + +{% tabs %} +{% tab title="Poetry (Recommended)" %} + +First, clone & navigate to the repository: + +``` +git clone https://github.com/OpenAdaptAI/OpenAdapt.git +cd OpenAdapt +``` + +If [poetry](https://python-poetry.org) is not installed, you can use pip to install it: + +``` +pip3 install poetry +``` + +Lastly, run these lines to setup the environment: + +``` +poetry install +poetry shell +poetry run postinstall +cd openadapt && alembic upgrade head && cd .. +``` + +{% endtab %} + +{% tab title="Git" %} + +Prerequisite: + +- Python 3.10 +- Git +- Tesseract (for OCR) +- nvm (node version manager) + +```batch +git clone https://github.com/OpenAdaptAI/OpenAdapt.git +cd OpenAdapt +python3.10 -m venv .venv +source .venv/bin/activate +pip install wheel +pip install -r requirements.txt +pip install -e . +alembic upgrade head +``` + +{% endtab %} + +{% tab title="Scripted Installation" %} +[Download](https://openadapt.ai/#start) + +## Windows: + +- Press Windows Key, type "powershell", and press Enter +- Copy and paste the following command into the terminal, and press Enter (If Prompted for `User Account Control`, click 'Yes'): +- `Start-Process powershell -Verb RunAs -ArgumentList '-NoExit', '-ExecutionPolicy', 'Bypass', '-Command', "iwr -UseBasicParsing -Uri 'https://raw.githubusercontent.com/OpenAdaptAI/OpenAdapt/main/install/install_openadapt.ps1' | Invoke-Expression"` + +## MacOS: + +- Download and install Git and Python 3.10 +- Press Command+Space, type "terminal", and press Enter +- Copy and paste the following command into the terminal, and press Enter: + `/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/OpenAdaptAI/OpenAdapt/HEAD/install/install_openadapt.sh)"` + +{% endtab %} +{% endtabs %} + +## Permissions(IMPORTANT) + +See how to set up system permissions on macOS [here](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/permissions_in_macOS.md). + +{% hint style="info" %} +**Good to know:** After installing OpenAdapt, run\ +`pytest` to verify that the installation was successful. +{% endhint %} + +# Usage + +## Shell + +Run this in every new terminal window once (while inside the `OpenAdapt` root directory) before running any `openadapt` commands below. This is required to activate the pythom environment: + +`poetry shell` + +You should see the something like this: + +``` +% poetry shell +Using python3.10 (3.10.13) +... +(openadapt-py3.10) % +``` + +Notice the environment prefix `(openadapt-py3.10)`. + + + +## Run app locally for testing + +`python -m openadapt.entrypoint` + +CLI would look something like this after the command + + + +After it runs successfully, you will see an app tray at top right of screen + + + +## Make your first recording + +{% hint style="info" %} + +Create a new recording by running the following command: + +To make your first recording, try the following command: + +```sh +python -m openadapt.record "testing out openadapt" +``` + + + +Wait until all three event writers have started: + +```bash +| INFO | __mp_main__:write_events:230 - event_type='screen' starting +| INFO | __mp_main__:write_events:230 - event_type='action' starting +| INFO | __mp_main__:write_events:230 - event_type='window' starting +``` + + + +To stop recording, focus the terminal and send **CTRL+C (SIGINT)**. The recording is finished saving when you see something like this: + +```bash +| INFO | __main__:record:637 - joining... +| INFO | __mp_main__:write_events:221 - event_type='window' done +| INFO | __mp_main__:write_events:221 - event_type='action' done +| INFO | __main__:read_screen_events:356 - done +| INFO | __main__:process_events:126 - done +| INFO | __mp_main__:write_events:221 - event_type='screen' done +| INFO | __main__:record:652 - saved recording_timestamp=1686243306.393652 +| INFO | __mp_main__:performance_stats_writer:433 - performance stats writer done +``` + + + +Current limitations: + +- Recording should be short (i.e. under a minute), as they are somewhat memory intensive, and there is currently an [open issue](https://github.com/OpenAdaptAI/OpenAdapt/issues/5) describing a possible memory leak. +- The only touchpad and trackpad gestures currently supported are pointing the cursor and left or right clicking, as described in this [open issue](https://github.com/OpenAdaptAI/OpenAdapt/issues/145) + +## Visualize + +- Quickly visualize the latest recording you created by running the following command: + +`python -m openadapt.visualize` + +This will generate an HTML file and open a tab in your browser that looks something like this: + + + +- For a more powerful dashboard, run: + +`python -m openadapt.app.dashboard.run` + + + + + + +- For a desktop app-based visualization, run: + +`python -m openadapt.app.visualize` + +Read more on [openadapt.visualize](../../reference/api-reference/openadapt-module/visualize.py.md). + +This will open a scrollable window that looks something like this: + + + +## Playback + +You can play back the recording using the following command: + +`python -m openadapt.replay NaiveReplayStrategy` + +Other replay strategies include: + +- [StatefulReplayStrategy](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/stateful.py): Early proof-of-concept which uses the OpenAI GPT-4 API with prompts constructed via OS-level window data. +- [(\*)VisualReplayStrategy](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/visual.py): Uses [Fast Segment Anything Model (FastSAM)](https://github.com/CASIA-IVA-Lab/FastSAM) to segment active window. +- [(\*)VanillaReplayStrategy](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/vanilla.py): Assumes the model is capable of directly reasoning on states and actions accurately. With future frontier models, we hope that this script will suddenly work a lot better. + +{% hint style="info" %} +**Good to know:** The (\*) prefix indicates strategies which accept an "instructions"\ +parameter that is used to modify the recording. +e.g.: `python -m openadapt.replay VanillaReplayStrategy --instructions "calculate 9-8"` +{% endhint %} + +See [strategy list](https://github.com/OpenAdaptAI/OpenAdapt/tree/main/openadapt/strategies) for a complete list. More ReplayStrategies coming soon! + +# Features + +## System Tray Icon and Client GUI App(work-in-progress) + +`python3 -m openadapt.start` + + + +## Detailed performance monitoring via [pympler](https://pympler.readthedocs.io/en/latest/) and [tracemalloc](https://docs.python.org/3/library/tracemalloc.html): + + diff --git a/getting-started/welcome-to-openadapt/user-guide.md b/getting-started/welcome-to-openadapt/user-guide.md new file mode 100644 index 000000000..2817adc7b --- /dev/null +++ b/getting-started/welcome-to-openadapt/user-guide.md @@ -0,0 +1,128 @@ +--- +description: Installing OpenAdapt +--- + +{% hint style="info" %} +**Experiencing difficulty? Get support on our** [**Discord**](https://discord.gg/QKPuDqhDHF)**.** +{% endhint %} + +OpenAdapt **requires Python3.10**, you can download it [here](https://www.python.org/downloads/) or try our installer, which will download Python for you. + + +## Download OpenAdapt + +We have builds available for Windows and Mac + +{% hint style="info" %} +**IMPORTANT:** If you are on MAC, you would need to provide some [permissions](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/permissions_in_macOS.md) +{% endhint %} + +- Download the OpenAdapt application from [here](https://openadapt.ai/#start) + +![image](./assets/download-page.png) + +- Extract the zip + +![image](./assets/downloaded-zip.png) + +- Click on the extracted application + +- It creates an app-tray on top right. This app tray can be used to control the application. + +![image](./assets/app-tray.png) + + +## Start Recording + +Now after the application is installed, you can start recording following the steps explained. + +- Click the record button in the app-tray + +- It opens up a prompt to name your recording + +![image](./assets/record-prompt.png) + +- Give it a name and click ok. Now the recording starts. + + +## Stop Recording + +To end the recording you can simply + +- Click the stop recording button in the app-tray and the recording is saved. + +![image](./assets/stop-recording.png) + + +## Replay Recording + +To replay the saved recordings + +- Click on replay in the app-tray. It opens list of all the recordings. + +![image](./assets/replay-list.png) + +- Click on the recording you want to replay. This will open up a prompt to choose the **replay strategy** & **replay instructions** + +![image](./assets/replay-strategy-prompt.png) + +{% hint style="info" %} +**Good to know:** Read more about strategies [here](./developer-guide.md/#playback) +{% endhint %} + +- Here is a quick [demo](https://openadapt.ai/#demo-video) of how to use replay strategy & instructions. + + +## Visualize Recording + +You can visualize data about your recording events too. + +- Click on visualize in the app-tray + +- Choose the recording you want to visualize + +![image](./assets/visualize-list.png) + +- It opens an HTML page with extensive details about the events in your recordings + +![image](./assets/visualize-userapp-1.png) + + +## Delete Recording + +If you want to delete any recording from OpenAdapt, you can do it easily. + +- Click on delete button in app-tray + +- A list of available recordings appear + +- Choose the recording to delete + + +## Dashboard + +You can now control OpenAdapt from the GUI recently introduced. You can easily see all the **recordings**, **get recording data**, **change setting such as API keys**, choose to **scrubb a recording**, **replays** etc + +- Click on dashboard in the app-tray + +- A GUI opens on your browser similar to this + +![image](./assets/dashboard.png) + +- On settings page you can change API keys, turn on scrub etc + +![image](./assets/dashboard-settings.png) + + +## Scrubbing + +You can now scrub a recording of your choice and it gets processed. + +- In the dashboard settings, turn on scrubbing. + +- Navigate to scrubbing and choose a recording to scrub. Choose the provider + +- Now wait for the recording to be processed + +![image](./assets/scrubbing-processing.png) + diff --git a/getting-started/welcome-to-openadapt/quick-start.md b/legacy/getting-started/quick-start.md similarity index 100% rename from getting-started/welcome-to-openadapt/quick-start.md rename to legacy/getting-started/quick-start.md