[DISCUSS] Aegis MCP Governance Gateway Incubator Proposal

--000000000000a1ea53065831c9f7
Content-Type: text/plain; charset="UTF-8"

Hello everyone,
I would like to propose a project called Aegis MCP Governance Gateway for
the Apache Incubator, and start a discussion. It is a working reference
implementation, and I am bringing it here mainly because it needs a
community rather than a single author, which I state plainly in the
proposal.
*In short: *when an AI agent needs to operate data systems like Flink,
Kafka, Spark, or Iceberg, it usually drives each engine control plane
directly or connects to a pile of separate MCP servers, each with its own
security posture, no shared audit, and no common policy. An MCP server that
can reach a control plane is a privileged client, so shipping one safely
means shipping an access-control product. Aegis puts one governed control
plane in front of all of those engines, so every agent action runs through
a single fail-closed chain of identity, scope, policy, approval for
destructive actions, egress and request-forgery defense, rate limiting,
output redaction, and a tamper-evident audit trail. Engines plug in through
a small service provider interface and contribute no policy.
Current state, stated honestly. There is a working Java 21 build: a gateway
core, four in-depth adapters (Flink, Kafka, Spark, Iceberg), thinner
adapters for other Apache systems, and a runnable distribution. It builds
cleanly with 309 tests passing across all modules. A published
design-conformance matrix maps every design claim to Done or Roadmap.
Several capabilities ship as roadmap items and are not sold as done:
externalized high-availability state, CIMD and SPIFFE identity, a full
Cedar runtime, vault-backed credentials, and an OpenTelemetry SDK bridge.
Parts of the code were developed with AI-assisted tools under my direction
and review, which I disclose up front for IP clearance.
*Biggest risk:  *The project is currently a single contributor. Growing a
diverse, multi-organization community is the main reason I am bringing it
here. I am currently in active conversations with engineers in the
ecosystem and expect to add a few of them as initial committers in the next
few days, prior to a formal vote
What I am looking for now: a Champion (an ASF Member) and mentors (three or
more IPMC members), ideally including people from the Flink, Kafka, Spark,
or Iceberg communities. The full proposal follows, and the reference
implementation is at https://github.com/vaquarkhan/aegis-mcp-gateway. I
welcome feedback on scope, name (a name search is still to be run), risks,
and anything I have missed.
Thank you,
 Viquar Khan
https://www.linkedin.com/in/vaquar-khan-b695577/

Apache Incubator Proposal: Aegis MCP Governance Gateway
This is the proposal for entry into the Apache Incubator, written for
discussion on the Incubator general list. Aegis is a working codename; a
name search will be run and the ASF trademark process followed if the
project is accepted. The document follows the ASF Incubator proposal
template and states the maturity of the code honestly, including what is
implemented and what is deferred to the roadmap.
Field
Value
Proposal name
Aegis MCP Governance Gateway (codename)
Proposed short name
aegis (subject to name search)
Proposal date
to be set at submission
Sponsor
Apache Incubator PMC
Champion
to be identified (ASF Member)
Mentors
to be recruited (three or more IPMC members)
Initial code
Java 21 multi-module reference implementation, 36 Maven modules, 309 tests
passing
Reference repository
https://github.com/vaquarkhan/aegis-mcp-gateway (private for now, access on
request)
License
Apache License 2.0
Discussion list
general at incubator.apache.org
Abstract
Aegis MCP Governance Gateway is a vendor-neutral, secure-by-default Model
Context Protocol (MCP) gateway that puts one governed, auditable control
plane in front of many Apache data and streaming engines, so an AI agent
never talks to Flink, Kafka, Spark, or Iceberg directly. Every agent action
passes one fail-closed governance chain: secure-by-default tool exposure,
per-caller authentication and scope, a policy decision point, approval for
destructive operations, egress and server-side request forgery defense,
rate limiting, circuit breaking, prompt-injection screening, output
bounding and redaction, tool-catalog integrity, and a tamper-evident
hash-chained audit trail. Engines plug in through a small service provider
interface and contribute only tool definitions and backend calls.
Proposal
We propose to develop, under Apache governance, a Layer 7 policy MCP
gateway plus a family of engine adapters. A proxy forwards bytes and a
router dispatches by capability, but neither is a trust boundary. A gateway
is: it knows the tool, the principal, the budget, and the policy for every
call, and it can deny, redact, gate, and record. The gateway core is
stateless at the protocol edge so it can scale behind an ordinary load
balancer once shared governance state is externalized, and it is designed
to align with OAuth 2.1, SPIFFE, SLSA, OpenTelemetry, and the NIST AI Risk
Management Framework and ISO/IEC 42001.
The initial contribution is a working Java reference implementation: a
reusable gateway core, four in-depth engine adapters (Flink, Kafka, Spark,
Iceberg), and a further set of thinner adapters for the wider Apache data
platform. During incubation the priorities are community building, IP
clearance, and promoting the roadmap items to full implementations.
Background
The Model Context Protocol has become a common way for AI agents to reach
tools and data. As organizations connect agents to distributed Apache data
estates, an agent that needs Flink, Kafka, Spark, and Iceberg must either
drive each engine control plane directly or connect to a sprawl of
independent, mostly read-only MCP servers, each with its own security
posture, no shared audit, and no unified policy.
An MCP server that can reach a control plane (a JobManager REST API, a
Kafka AdminClient, a SQL gateway, an Iceberg catalog) is a privileged
client. Shipping one safely is effectively shipping an access-control
product: authentication, authorization, approval for destructive actions,
redaction of secrets and personal data, rate and cost limits, and forensic
audit. Many existing options lack these controls and carry
single-maintainer sustainability risk. A single governed gateway, developed
in the open under neutral governance, gives the ecosystem a shared standard
for how agents operate Apache systems safely.
Rationale
A single governed gateway resolves the problems of the decentralized
approach:

   - Security. One place enforces identity, authorization, approval,
   redaction, egress, and audit, instead of each server reimplementing or
   omitting them.
   - Maintainability. Adding an engine is a small adapter, not a new
   security model. There is one chain to review and harden, so the security
   review surface stays flat as engines are added.
   - Governance. One control plane for policy, budget, and provenance, with
   a stable deny taxonomy so operators can alert on codes such as
   POLICY_DENIED or APPROVAL_REQUIRED across every engine with one rule.

Doing this under Apache governance keeps it neutral across vendors and
engines rather than leaving agent access to fragmented single-vendor
servers.
Initial goals

   1. Establish the project and practice the Apache Way: transparency,
   consensus, meritocracy, and community over code.
   2. Complete IP clearance for the existing code and land it under
   Apache-2.0 with correct LICENSE and NOTICE files.
   3. Publish and stabilize the adapter service provider interface so
   per-engine adapters are small, reviewable contributions.
   4. Promote the roadmap items to full implementations: externalized
   high-availability state, CIMD and SPIFFE identity, a full Cedar runtime,
   vault-backed credential exchange, and an OpenTelemetry SDK bridge.
   5. Produce the first Apache source releases and grow a diverse,
   multi-organization contributor base.

Current status
Aegis is a new codebase with a working multi-module reference
implementation. It is not yet a community. The design, threat model,
low-level design, and standards research are complete and public. The code
builds green and is tested. A published design-conformance matrix maps
every design claim to Done or Roadmap, so the maturity line is explicit.
Verified build and test state

   - 36 Maven modules: a gateway core, adapters for 33 Apache engines (four
   in-depth today, Flink, Kafka, Spark, and Iceberg, and 29 thinner
   integrations covering Cassandra, HBase, Hive, Pulsar, Airflow, NiFi, Druid,
   Pinot, Doris, Impala, Calcite, Hudi, Paimon, Kudu, Ignite, BookKeeper,
   Ozone, Hadoop, ZooKeeper, Solr, Superset, Arrow Flight, Beam, Storm, Flume,
   ActiveMQ, CouchDB, Ranger, and Atlas), and a shaded runnable distribution.
   These are engine adapters the gateway supports, which is distinct from
   external adopters.
   - Builds with JDK 21 and runs on JDK 17 or newer with Maven 3.9 or newer.
   - 309 unit and integration tests across all modules, all passing, zero
   failures and zero errors on a clean build.

Implemented in 0.1.0

   - The governance chain with ordered steps and first-denial-wins
   semantics.
   - Identity: shared bearer and a hashed multi-caller token file with
   per-caller scope, and an OAuth 2.1 resource-server mode with JWKS
   validation. Constant-time secret comparison and fail-closed startup
   validation.
   - Approvals: HMAC-SHA256 signed, TTL-bounded, single-use,
   tool-and-scope-bound tokens with replay rejection.
   - Policy: a builtin decision point, a live fail-closed OPA HTTP decision
   point, and a Cedar-style deny-file or HTTP delegate.
   - Per-caller outbound credential pass-through so a downstream engine can
   see a per-caller Authorization value when one was bound at admission.
   - Egress and server-side request forgery defense: allow list plus
   unconditional deny of cloud metadata and non-routable ranges, including
   numeric IP encodings and resolved addresses, with redirects disabled on
   outbound clients.
   - Rate limiting, circuit breaking, per-tool timeout, read-only SQL
   guard, identifier validation, output bounding, and data-loss-prevention
   redaction.
   - Tool-catalog integrity with durable pins and a fail-closed switch, a
   tamper-evident hash-chained audit, Prometheus metrics, and GenAI-shaped
   span log lines.
   - Two transports: stdio and authenticated Streamable HTTP with TLS, plus
   health, readiness, and metrics endpoints. A CycloneDX software bill of
   materials is produced at build time.

Deferred to the roadmap (honestly labeled, not sold as implemented)

   - CIMD and SPIFFE authentication modes (they refuse to start today).
   - A full Cedar policy runtime, and vault-backed or token-exchange
   credential resolution.
   - Externalized high-availability state (shared nonce, rate, and audit
   stores). Until then, run a single replica; multi-replica replay protection
   and rate limits require the shared store.
   - The MCP Tasks extension for long-running jobs, an OpenTelemetry SDK
   bridge, a full Merkle and signature-based reconciliation proof, and true
   semantic routing and caching.

These are tracked in the roadmap and the design-conformance matrix that
ship with the code.
Meritocracy, community, core developers, alignment
The project will operate meritocratically from day one, with technical
decisions on the dev list and committership earned through contribution.
The community is currently centered on the initial author, Viquar Khan, who
has around 23 years in software, has worked in the open since 2013, and has
contributed to Apache Spark, Apache Iceberg, Apache Kafka, and other
open-source projects. The author also maintains MCP-related open-source
packages that have together been downloaded on the order of half a million
times, so there is an existing user base to draw contributors from. Growing
a committer base across the Flink, Kafka, Spark, and Iceberg communities
and across employers is the primary goal of incubation. Aegis consumes
public engine interfaces only and does not fork or modify any engine core.
Known risks
Orphaned products
Low to moderate. Agentic access to data systems is a growing need and the
governance gap is real, so demand is unlikely to vanish. The initial author
has maintained open-source projects since 2013 rather than abandoning them,
and the MCP-related packages already have a real user base, which lowers
the abandonment risk. Mitigation: build a multi-organization community,
keep the core small and standards aligned, and keep adapters independent so
the project survives the departure of any single contributor.
Inexperience with open source
Low. This is not the situation here. The initial author has about 23 years
in software, has worked in open source since 2013, and has contributed to
Apache Spark, Apache Iceberg, Apache Kafka, and other projects, so the
norms of public development, code review, and releases are familiar.
Mentors will still help with the ASF-specific mechanics of voting, board
reports, and IP clearance, which are process details rather than a gap in
open-source experience.
Homogenous developers
 This is the real risk and we state it plainly: the podling has one
committer today. However, we are in active discussions with potential
co-committers and expect to formally add them to this proposal in the
coming days. What further reduces this risk is that  the author is already
embedded in the exact communities this project draws from (Spark, Iceberg,
Kafka) and has an existing user base for the MCP packages, so there is a
concrete pool to recruit from rather than a cold start. Mitigation: recruit
committers from more than one organization before graduation, shape the
adapter contract so per-engine contributions are natural, and use
good-first-issue and adapter-sized work to onboard newcomers. Growing the
committer base is the main reason we are here.
Reliance on salaried developers
Low current reliance; the work to date is largely independent. The project
will attract contributors with varied motivations and employers so no
single company can stall it.
Relationships with other Apache products
Aegis consumes the public interfaces of Flink, Kafka, Spark, Iceberg, and
other Apache data systems and does not fork or modify them. We will
coordinate on the relevant dev lists and will not duplicate or compete with
an engine that ships its own MCP surface; where an engine has one, Aegis
composes with it. Aegis is not proposed for any engine core.
Excessive fascination with the Apache brand
The project seeks Apache governance for its community model and neutrality,
not for the brand. Until and unless accepted, it uses a neutral name and
does not use the Apache marks or the feather. The breadth of engine
adapters reflects a common governance surface, not an attempt to claim the
ecosystem; only four adapters are in-depth today and the rest are
explicitly thin.
Documentation
Public documentation exists and will be maintained in the repository: an
end-to-end design with architecture decision records, a low-level design
with architecture, sequence, and state diagrams, a threat and governance
model, a design-conformance matrix, standards research, and operations and
adapter guides.
Initial source
The initial source is the Java multi-module gateway described above,
currently in a private repository at
https://github.com/vaquarkhan/aegis-mcp-gateway under the initial author.
Access is available on request to anyone reviewing this proposal, and the
author will make it public before the vote if the community prefers to read
the code first. It includes the gateway core, the four in-depth adapters,
the thinner adapters, the shaded distribution, runnable examples, and the
documentation set. On acceptance it will be donated under a Software Grant
and made public.
Source and intellectual property submission plan
The initial author will submit the code under a Software Grant Agreement
and complete IP clearance early in incubation, with ICLAs from every
committer. Every source file will carry the standard Apache-2.0 header with
correct LICENSE and NOTICE files; any premature ASF-ownership wording will
be corrected to the standard header at donation time.
Transparency on tooling: parts of the codebase were developed with
AI-assisted coding tools under the author's direction and review. The
author asserts authorship and the right to contribute the work, and will
confirm provenance and authorship during IP clearance so the grant is
clean. We raise this proactively so the IPMC can review it rather than
discover it.
External dependencies
Runtime dependencies are Apache-2.0 or compatible, including the MCP Java
SDK, a servlet container for the HTTP transport, SLF4J, Logback, and Nimbus
for JOSE and JWT, with JUnit for tests. Per-engine adapters add that engine
to the official client, for example the Kafka clients library. A full
dependency review and a CycloneDX software bill of materials are produced.
No GPL or ASF category X dependency is introduced into the core.
Cryptography
Aegis uses standard, library-based cryptography: HMAC-SHA256 for approval
tokens, TLS for the HTTP transport, and JSON Web Token validation for
OAuth. Signature-based integrity is an optional gate. The project will
comply with ASF export notification requirements.
Required resources
Mailing lists

   - [email protected]
   - [email protected]
   - [email protected]

Source control

   - A Git repository under the ASF (gitbox with a GitHub mirror).

Issue tracking

   - ASF Jira or GitHub issues per current ASF practice.

Other

   - Website and wiki space, and continuous integration through ASF
   infrastructure.

Initial committers

   - Viquar Khan *(Note: We are in active conversations with potential
   contributors and expect to add a few additional initial committers here in
   the next few days before moving to a formal vote.)*

   - To be recruited before and during incubation: committers from the
   Flink, Kafka, Spark, and Iceberg communities and from additional
   organizations. Adapter-sized work is designed to make these contributions
   natural.

Affiliations
To be completed as committers are confirmed. Diversity of affiliation is an
explicit goal; the current single-affiliation status is the main risk being
addressed.
Sponsors
Champion
To be identified. An ASF Member Champion will help finalize this proposal,
recruit mentors, and shepherd the discussion and vote on the Incubator
general list.
Nominated mentors
Three or more IPMC members to be recruited, ideally including participants
from the Flink, Kafka, and Iceberg communities.
Sponsoring entity
The Apache Incubator PMC.
Roadmap during incubation
Phase
Focus
Bootstrap
IP clearance, name search, infra, initial committers, first dev-list
decisions
0.2
Externalized HA state, vault-backed credentials, OAuth multi-issuer, OTel
SDK bridge, supply-chain CI (enforcer, CVE scan, signing)
0.3
CIMD and SPIFFE identity, full Cedar runtime, MCP Tasks, signature-based
reconciliation proof, semantic routing and caching
Graduation readiness
Diverse community, cadence of Apache releases, healthy governance
References

   - Apache Incubator proposal guide:
   https://incubator.apache.org/guides/proposal.html
   - Apache Incubator Cookbook: https://incubator.apache.org/cookbook/
   - The Apache Way: https://www.apache.org/theapacheway/
   - Model Context Protocol: https://modelcontextprotocol.io/

--000000000000a1ea53065831c9f7--