Skip to main content

· 25 min read
tison

What is Apache OpenDAL?

Apache OpenDAL is a data access layer provided in the form of a software library. It allows users to access data stored on various storage services simply and efficiently through a unified API. You can consider it as an improved implementation of an S3 SDK, or use the unified OpenDAL API to simplify the work of configuring and accessing different data storage services (such as S3, HDFS, GCS, Aliyun OSS, etc.).

OpenDAL is provided as a library, so there is no need to deploy additional services when using OpenDAL. The core of OpenDAL is written in Rust. During the project's incubation and growth, the community has also developed bindings for languages such as Java, Python, Node.js, and C, to support the convenient integration of OpenDAL capabilities into programs written in these languages.

The following diagram lists the users of Apache OpenDAL's multi-language implementations:

Apache OpenDAL Users

You can use the unified APIs of OpenDAL as follows:

async fn do_business() -> Result<()> {
let mut builder = services::S3::default();
builder.bucket("test");

let op = Operator::new(builder)?
.layer(LoggingLayer::default())
.finish();

// Write Data
op.write("hello.txt", "Hello, World!").await?;
// Read Data
let bytes = op.read("hello.txt").await?;
// Fetch Metadata
let meta = op.stat("hello.txt").await?;
// Delete Data
op.delete("hello.txt").await?;

Ok(())
}

As you can see, the API for data reading and writing is carefully designed. Users wanting to access data stored on different services only need to modify the configuration of the Operator, without having to change any code for the actual read and write operations.

· 13 min read
tison

Apache Kvrocks is a distributed key value NoSQL database that uses RocksDB as storage engine and is compatible with Redis protocol. It aims to address the high memory cost and limited capacity issues of Redis. You can use Kvrocks as a drop-in replacement as a persistent variant of Redis.

Kvrocks was initially developed internally by Meitu Inc. and was open-sourced and developed independently in 2019. In April 2022, Kvrocks entered the ASF Incubator with Chen Liang as the Champion.

During the one-year incubation period, the number of contributors and users more than doubled, and four different release managers successfully completed the release of four official versions that complied with the Apache release standards. In April 2023, Apache Kvrocks initiated the graduation process and discussions, ultimately completing the process and officially graduating in June 2023.

In early 2022, I was nominated as an official member of the ASF by Willem Ning Jiang. Then I became qualified as a mentor to participate in mentoring projects. At that time, I happened to come across the incubation proposal for Kvrocks and after reading it, I believed it was a promising project, so I volunteered to become one of the mentors.

Apache Kvrocks is the first incubator project (podling) that I mentored to graduate successfully. During the incubation period, I put my open-source philosophy into practice, helping the project maintainers understand the principles of The Apache Way, and also participating in the development and growth of the project. It can be said that Kvrocks has helped me validate many ideas about producing open-source software and community development.

This article will start from the incubation process and my involvement, introducing the community status, achievements, and future directions of Apache Kvrocks.