Swirl

1.5k 132

04 May, 2024

Python

What is Swirl

Swirl is an open source software that simultaneously searches multiple content sources and returns AI ranked results. Prompt your choice of Generative AI using the top N results to get answers incorporating your own data.

Swirl Features

Swirl can connect to:

Databases (SQL, NoSQL, Google BigQuery)
Public data services (Google Programmable Search Engines, ArXiv.org, etc.)
Enterprise sources (Microsoft 365, Jira, Miro, etc.)

And generate insights with AI and LLMs like ChatGPT. Start discovering and generating the answers you need based on your data.

Swirl is as simple as ABC: (a) Download YML, (b) Start in Docker, (c) Search with Swirl. From there, add credentials to preloaded SearchProviders to get results from more sources.

✦	Feature
📌	Microsoft 365 integration and OAUTH2 support
🔍	SearchProvider configurations for all included Connectors. They can be organized with the active, default and tags properties.
✏️	Adaptation of the query for each provider such as rewriting `NOT term` to `-term` , removing NOTted terms from providers that don’t support NOT, and passing down the AND, + and OR operators.
⏳	Synchronous or asynchronous search federation via APIs
🛎️	Optional subscribe feature to continuously monitor any search for new results
🛠️	Pipelining of Processor stages for real-time adaptation and transformation of queries, responses and results
🗄️	Results stored in SQLite3 or PostgreSQL for post-processing, consumption and/or analytics
➡️	Built-in Query Transformation support, including re-writing and replacement
📖	Matching on word stems and handling of stopwords via NLTK
🚫	Duplicate detection on field or by configurable Cosine Similarity threshold
🔄	Re-ranking of unified results using Cosine Vector Similarity based on spaCy’s large language model and NLTK
🎚️	Result mixers order results by relevancy, date or round-robin (stack) format, with optional filtering of just new items in subscribe mode
📄	Page through all results requested, re-run, re-score and update searches using URLs provided with each result set
📁	Sample data sets for use with SQLite3 and PostgreSQL
✒️	Optional spell correction using TextBlob
⌛	Optional search/result expiration service to limit storage use
🔌	Easily extensible Connector and Mixer objects

🔎 How Swirl Works

Swirl adapts and distributes user queries to anything with a search API - search engines, databases, noSQL engines, cloud/SaaS services, data siloes, etc. and uses Large Language Models to re-rank the unified results without extracting or indexing anything.

Swirl Diagram

🔌 List of Connectors

Connectors_2

Get Started

🚀 Try Swirl with ChatGPT

Swirl with ChatGPT as a configured AI Model

Swirl with ChatGPT as a configured AI Model.

🔥 Try Swirl Now In Docker

Prerequisites

To run Swirl in Docker, you must have the latest Docker app for MacOS, Linux, or Windows installed and running locally.
Windows users must also install and configure either the WSL 2 or the Hyper-V backend, as outlined in the System Requirements for installing Docker Desktop on Windows.

Start Swirl in Docker

Warning

Make sure the Docker app is running before proceeding!

Download the YML file: https://raw.githubusercontent.com/swirlai/swirl-search/main/docker-compose.yaml

curl https://raw.githubusercontent.com/swirlai/swirl-search/main/docker-compose.yaml -o docker-compose.yaml

Optional: To enable Swirl’s Real-Time Retrieval Augmented Generation (RAG) in Docker, run the following commands from the Console using a valid OpenAI API key:

export MSAL_CB_PORT=8000

export MSAL_HOST=localhost

export OPENAI_API_KEY=‘<your-OpenAI-API-key>’

:key: Check out OpenAI’s YouTube video if you don’t have an OpenAI API Key.

In MacOS or Linux, run the following command from the Console:

docker-compose pull && docker-compose up

In Windows, run the following command from PowerShell:

docker compose up

After a few minutes the following or similar should appear:

swirl_docker_1

Open this URL with a browser: http://localhost:8000 (or http://localhost:8000/galaxy)
If the search page appears, click Log Out at the top, right. The Swirl login page will appear.
Enter the username admin and password password, then click Login.
Enter a search in the search box and press the Search button. Ranked results appear in just a few seconds:

galaxy_ui_2

To view the raw JSON, open http://localhost:8000/swirl/search/

The most recent Search object will be displayed at the top. Click on the result_url link to view the full JSON Response.

Notes 📝

Warning

The Docker version of Swirl does not retain any data or configuration when shut down!

:key: Swirl includes five (5) Google Programmable Search Engines (PSEs) to get you up and running right away. The credentials for these are shared with the Swirl Community.

:key: Using Swirl with Microsoft 365 requires installation and approval by an authorized company Administrator. For more information, please review the M365 Guide or contact us.

Next Steps 👇

Check out the details of our latest release!
Head over to the Quick Start Guide and install Swirl locally!