Pathway
Pathway is a Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG. It provides an easy-to-use Python API with a scalable Rust engine for incremental computation, handling both batch and streaming data. It is designed for developers building data pipelines that require low-latency processing and easy deployment.
🎬 KEGG Pathway Analysis Tutorial for Beginners | Learn, Analyze & Interpret with Real Examples | · Biology Lectures
✨ Key features
- Wide range of connectors including Kafka, GDrive, PostgreSQL, SharePoint, and Airbyte.
- Supports stateless and stateful transformations like joins, windowing, and sorting.
- Persistence to save computation state for restart after updates or crashes.
- Consistency management for late and out-of-order data with at-least-once semantics.
- Scalable Rust engine enabling multithreading, multiprocessing, and distributed computations.
- LLM helpers for building RAG pipelines with wrappers, parsers, and vector index.
🎯 Use cases
- Real-time ETL pipelines from Kafka or other sources.
- Event-driven pipelines with alerting and monitoring.
- Real-time analytics with linear regression on streaming data.
- Building live RAG applications with private documents and LLMs.
- Unstructured data to SQL conversion on-the-fly.
📦 Installation
🧰 Requirements: Python 3.10 or above, available on MacOS and Linux; other systems require a virtual machine.
pip install -U pathway
🚀 Usage
import pathway as pw
class InputSchema(pw.Schema):
value: int
input_table = pw.io.csv.read("./input/", schema=InputSchema)
filtered_table = input_table.filter(input_table.value >= 0)
result_table = filtered_table.reduce(
sum_value = pw.reducers.sum(filtered_table.value)
)
pw.io.jsonlines.write(result_table, "output.jsonl")
pw.run()
⚠️ Good to know
Pathway is available on MacOS and Linux only; other systems need a virtual machine. The free version provides at-least-once consistency, while exactly-once is enterprise-only.
❓ FAQ
What Python version is required?
Pathway requires Python 3.10 or above.
Can I use Pathway on Windows?
Pathway is available on MacOS and Linux; Windows users should run it on a virtual machine.
Does Pathway support stateful operations?
Yes, Pathway supports stateful transformations such as joins, windowing, and sorting.
How can I deploy Pathway in production?
Pathway can be deployed using Docker and Kubernetes, and there is an enterprise version for distributed scaling.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.