When addressing technology challenges within the payment industry, two crucial aspects of programming come into play: security and responsiveness. Security ensures the protection of sensitive financial data, while responsiveness pertains to the speed and efficiency of the application. In the process of developing a state-of-the-art payment application, I found myself at a juncture where extensive research into various frameworks was necessary to meet my design criteria. I ultimately opted for the Java programming language as the foundation for my application, prompting me to explore Java-based I/O frameworks that prioritize security and speed. After thorough research, I concluded that Apache Netty best suited my needs. However, implementing Netty presented its own set of challenges. Understanding these challenges and my solution to overcome them may be helpful to other developers working in payment systems.
Apache Netty is a powerful, I/O-optimized, non-opinionated, asynchronous I/O framework built in Java, designed for rapid development of maintainable high performance protocol servers and clients (?https://netty.io/). This tool provides the ability to work with many protocols and offers valuable features beyond just I/O, such as ByteBuf, which I used to overcome one of the challenges.
While the architects and developers of Netty have designed it for ease of use, certain obstacles arise when working with applications that require optimization. In my case, the first hurdle I encountered was in designing the system to handle both synchronous and asynchronous requests.
Handling both synchronous and asynchronous tasks may seem straightforward, but achieving this goal while minimizing the overhead of creating a bootstrap is where the problem began. Duplicating code may seem like an easy way out, but I was seeking a solution that I would not regret in the future. After exploring a multitude of options, I found that Java’s CompletableFuture provided a better solution for handling this problem, while still maintaining simplicity.
It is critical to understand the basics of Netty before attempting to design a solution. Every Netty I/O process, be it a Client or Server, has the following components:
- EventLoopGroups: For a server, it is advised to have two EventLoopGroups; one to handle incoming socket connections and the other to handle I/O operations. However, for a Client, only one EventLoopGroup is needed to handle socket read and write operations.
- Bootstrap: Each I/O process requires a bootstrap, which essentially informs Netty how to handle a connection and data.
- Channel Initializer : Channel Initializer configures the incoming and outgoing handlers, essentially forming a duplex data pipeline. One of these handlers can be configured to include SSL for secure communication, alongside codec and incoming message handlers.
EventLoopGroups can be reused, and Channel Initializer, with the exception of activating SSL as needed, can be achieved easily. However, Bootstrap comes with its own set of issues.
For me, the problem with the Bootstrap was the host and port part, connecting to the remote host. After isolating that with relatively little difficulty, I used an empty bootstrap with no host and tried to connect to the remote host each time the application made calls. While the below code looks easy, there are other challenges with this approach.
bootstrap.connect(uri.getHost(), port);
The primary requirement was to correlate the response to the request correctly, and this reveals the problem. If I have to map the response, Channel Initializer needs to be created every time with a new handler so it can be mapped, which negates the purpose of reusing the bootstrap. The code to achieve the reusable bootstrap looks as follows:
public class NettySSLHttpClient {
private final EventLoopGroup group = new NioEventLoopGroup();
private final Bootstrap bootstrap;
private final SslContext sslContext;
public String httpResponse;
public NettySSLHttpClient() throws Exception {
bootstrap = new Bootstrap();
sslContext = SslContextBuilder
.forClient()
.trustManager(InsecureTrustManagerFactory.INSTANCE)
.build();
bootstrap
.group(group)
.channel(io.netty.channel.socket.nio.NioSocketChannel.class)
.handler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) {
ChannelPipeline p = ch.pipeline();
p.addLast(sslContext.newHandler(ch.alloc()));
p.addLast(new HttpClientCodec());
p.addLast(new HttpObjectAggregator(1024 * 1024)); // Max content length
p.addLast(new SimpleChannelInboundHandler<FullHttpResponse>() {
@Override
protected void channelRead0(ChannelHandlerContext ctx, FullHttpResponse msg)
{
httpResponse = msg.content().toString(CharsetUtil.UTF_8);
ctx.close();
}
});
} });
}
public String process(String url, String data)
throws URISyntaxException, InterruptedException {
URI uri = new URI(url);
String host = uri.getHost();
int port = uri.getPort() == -1 ? 443 : uri.getPort();
String path = uri.getPath();
ChannelFuture future = bootstrap.connect(host, port).sync();
DefaultFullHttpRequest request =
new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, path);
request.headers().set(HttpHeaderNames.HOST, host);
future.channel().writeAndFlush(request).sync();
future.channel().closeFuture().sync();
return httpResponse;
}
public void shutdown() {
group.shutdownGracefully();
}
}
While the above code works well for both synchronous and asynchronous operations, with some adjustments, upon closer observation, a new problem arises: because every request needs to wait for one object to complete its cycle, rendering the use of Netty loses its usefulness. Although Netty works fine for both asynchronous and synchronous modes, it processes messages sequentially. Additionally, we are now extracting content and passing it on. If a FullHTTPResponse is passed, it would not work due to the nature of ByteBuf. If we need more data such as headers or response code, we need to build a new object and return, which is not very effective or efficient. The following code works well with one request at time, but for concurrent requests it falls apart:
public static void main(String[] args) throws Exception {
NettySSLHttpClient client = new NettySSLHttpClient();
System.out.println(client.process("https://api.namefake.com"));
System.out.println(client.process("https://api.namefake.com"));
}
In a payment application, there is a high possibility of initiating two transactions at the same time, down to the nanosecond. If the same program were to be used in this case, the result would be disastrous. Additionally, because the code is clumsy and not easily extendable, maintenance would be quite challenging.
To address this problem, I designed the following solution, which worked quite well in production. First, it is important to build a reusable configuration:
public class Netty {
private Bootstrap bootstrap;
private EventLoopGroup childgroup;
private ChannelInitializer<Channel> pipeline;
private Map<ChannelId, Future<FullHttpResponse>> channelMap;
public Netty() {
bootstrap = new Bootstrap();
childgroup = new NioEventLoopGroup();
pipeline = new HttpChannelPipeline();
channelMap = new HashMap<>();
buildBootStrap(bootstrap);
}
public void buildBootStrap(Bootstrap bootStrap) {
bootstrap.group(childgroup)
.channel(NioSocketChannel.class)
.option(ChannelOption.SO_KEEPALIVE, false)
.option(ChannelOption.TCP_NODELAY, true)
.option(ChannelOption.SO_LINGER, 0)
.handler(pipeline);
}
}
The next step is the HttpChannelPipeline, which is instantiated only once:
public class HttpChannelPipeline extends HttpChannelPipeline<Channel> {
@autowired
private MessageHandler handler;
@Override
protected void initChannel(Channel ch) throws SSLException {
SslContext sslContext = SslContextBuilder
.forClient()
.trustManager(InsecureTrustManagerFactory.INSTANCE)
.build();
ChannelPipeline p = ch.pipeline();
p.addLast(sslContext.newHandler(ch.alloc()));
p.addLast(new HttpClientCodec());
p.addLast(new HttpObjectAggregator(1024 * 1024));
// add incoming message handler
p.addLast(handler)
}
}
The reusable configuration is now complete. The final step requires correlating the response with the result. In the following code, the handler will be injected with a Map resource, so every time there is a change in the Map object, the updates are reflected in both the handler and a ChannelListener, which is used to listen for connection success events and write requests to the host/server.
public class HttpFutureListener implements ChannelFutureListener {
@Autowired
private Map<ChannelId, Future<FullHttpResponse>> channelMap;
@Override
public void operationComplete(ChannelFuture future) throws Exception {
Channel channel = future.channel();
Future<FullHttpResponse> httpFuture = channelMap.get(channel.id());
channel.attr(TcpConstants.RESPONSE_FUTURE).set(httpFuture);
if (future.isSuccess()) {
channel.writeAndFlush(httpFuture.getRequestObject());
}
}
}
Next, we must integrate the MessageHandler, a shareable component and a stateless singleton object that can be shared with all the channels.
public class MessageHandler extends SimpleChannelInboundHandler<FullHttpResponse> {
@Override
protected void channelRead0(ChannelHandlerContext ctx, FullHttpResponse msg) throws Exception {
Future<FullHttpResponse> responseFuture = ctx.channel()
.attr(TcpConstants.RESPONSE_FUTURE)
.get();
responseFuture.complete(msg.copy);
} }
The final part of the solution is all about processing the HTTP request, as follows:
public Future<FullHttpResponse> processAsync(FullHttpRequest request, TimeUnit unit, int timeout) {
URI uri = getUri(request.uri());
int port = uri.getPort();
final Future<FullHttpResponse> responseFuture = new CompletableFuture<>(request);
CompletableFuture.supplyAsync(() -> {
ChannelFuture future = getbootstrap().connect(uri.getHost(), port);
channelMap.put(future.channel().id(), responseFuture);
// successChannelListner is a HttpChannelListner's singleton instance
future.addListener(this.sucessChannelListner)
.awaitUninterruptibly(timeout, unit);
return true;
});
return responseFuture;
}
}
Usage of this component is very easy; it can be at the caller’s discretion to make it synchronous or asynchronous.
Netty httpClient = new Netty();
httpClient.processAsync(request).get();
httpClient.processAsync(request);
Either way, in a synchronous call, the caller is blocked, but the actual HTTP client component remains asynchronous and non-blocking. There may be more efficient ways to achieve this, but I managed to achieve a single-digit millisecond response in benchmarks (provided there is a good network).
This solves the first problem. The second problem involves Java’s SSLEngine.
While I had hoped that the steps above would resolve my issues, running Java profiling revealed secure data in the heap dump. This prompted the question: is Java SSL really secure?
In my expert opinion, this is a gray area. For non-payment applications, I believe it is secure enough. But for payment-related applications subject to PCI (News - Alert)-DSS and/or PCI-SSF audits, it is not very secure. Java doesn’t support a zero-copy mechanism out of the box. After spending many hours profiling and studying heap dumps, I am assured that Java’s SSL Engine makes copies of data, and they are not explicitly swept with random data—they are simply released for the garbage collector to pick up and release. However, in Java, garbage collection is not guaranteed, and that’s where the problem lies.
There may be good reasons why JVM/Java engineers implemented it in that way, perhaps for caching or other performance-related decisions. Unfortunately, that is not acceptable in the payment industry or for PCI certification.
Java copies data coming from remote servers, whether it’s from sockets, HTTP, or any other source. These copies can be clearly seen in data structures such as Char[], Byte[], ByteBuffer, and especially in PlainText object types. Below is a snippet of Java code that copies data and performs a clear call, which only resets data pointers, but does not scramble or clear the actual data:
// from Java library in the stack trace for SSL read and write
// bb is of type ByteBuffer
// this.bb = ByteBuffer.allocate(INITIAL_BYTE_BUFFER_CAPACITY);
private void writeBytes() throws IOException {
bb.flip();
int lim = bb.limit();
int pos = bb.position();
assert (pos <= lim);
int rem = (pos <= lim ? lim - pos : 0);
if (rem > 0) {
if (ch != null) {
int wc = ch.write(bb);
assert wc == rem : rem;
} else {
out.write(bb.array(), bb.arrayOffset() + pos, rem);
}
}
bb.clear();
}
This snippet demonstrates how Java copies data from the original source and performs a clear operation, but it only resets the data pointers and does not securely erase or clear the actual data. The following is code and details from the Java specification (https://docs.oracle.com/javase/8/docs/api/java/nio/Buffer.html#clear--):
ByteBuffer java.nio.ByteBuffer.clear()
Clear this, Buffer. The position is set to zero, the limit is set to the capacity, and the mark is discarded.
Invoke this method before using a sequence of channel-read or put operations to fill this buffer. For example:
{@snippet lang=java : buf.clear(); // Prepare buffer for reading in.read(buf); // Read data }
This method does not actually erase the data in the buffer, but it is named as if it did, because it will most often be used in situations in which that might as well be the case.
// Fill the destination buffers.
if ((dsts != null) && (dstsLength > 0)) {
ByteBuffer fragment = plainText.fragment;
int remains = fragment.remaining();
// Should have enough room in the destination buffers.
int limit = dstsOffset + dstsLength;
for (int i = dstsOffset;
((i < limit) && (remains > 0)); i++) {
int amount = Math.min(dsts[i].remaining(), remains);
fragment.limit(fragment.position() + amount);
dsts[i].put(fragment);
remains -= amount;
if (!dsts[i].hasRemaining()) {
dstsOffset++;
}
}
if (remains > 0) {
throw context.fatal(Alert.INTERNAL_ERROR,
"no sufficient room in the destination buffers");
}
}
}
finalPlaintext = plainText;
}
return finalPlaintext;
}
}
The snippet above from Java demonstrates copying the data into dsts[i], then copying the reference of the object to another PlainText object, which keeps the data in the heap for the garbage collector to clean up, leaving behind potentially sensitive data. This process does not securely erase or clear the actual data, posing a security risk.
The issue of sensitive data lingering in memory, even for a short period, poses significant security concerns. The solution lies in leveraging Netty’s zero-copy feature and utilizing ByteBuf data type effectively. The difference is that traditional data copying involves transferring data from one buffer to another, resulting in memory duplication and potential security risks. However, with Netty’s zero-copy feature and the use of ByteBuf, data can be accessed and manipulated directly, without the need for copying, thus minimizing memory usage and enhancing security. This approach not only improves performance, but also mitigates the risk of exposing sensitive data in memory.
Upon closer examination, Netty does not explicitly clear out memory either (implemented to optimize performance). Instead, it resets pointers and reuses the same ByteBuf object. However, an interesting aspect is that Netty allows users to make changes and clean up the data. This flexibility empowers developers to manage memory effectively and ensure the security of sensitive data within their applications. Following is the example use case code for how to leverage Netty’s flexibility:
public class HeapBuf extends UnpooledHeapByteBuf {
private static final int DEFAULT_INITIAL_CAPACITY = 256;
private static final int DEFAULT_MAX_CAPACITY = Integer.MAX_VALUE;
public HeapBuf() {
super(UnpooledByteBufAllocator.DEFAULT, DEFAULT_INITIAL_CAPACITY, DEFAULT_MAX_CAPACITY);
}
protected void freeArray(byte[] array) {
super.freeArray(array);
Arrays.fill(array, (byte) '*');
}
}
The snippet above demonstrates how to extend ByteBuf and can be implemented where secure data is present. If not, let Netty take care of the data. This cleanup method does come with a performance cost, but I consider it a necessary trade-off to ensure that secure data doesn’t fall into the wrong hands.
In the realm of securing network communications, a critical aspect is the SSL/TLS engine, itself. Netty, being a versatile framework, seamlessly integrates with OpenSSL, offering robust encryption capabilities right out of the box. Through meticulous configuration, I’ve observed significant improvements in both performance and security as compared to the traditional Java SSLEngine.
Additionally, to further bolster our cryptographic defenses, I’ve opted to leverage Bouncy Castle as a security provider for Java. Bouncy Castle’s extensive cryptographic algorithms and features surpass those provided by standard Java libraries, enhancing our ability to secure sensitive data and communications.
By harnessing the power of OpenSSL integration and Bouncy Castle’s advanced cryptographic capabilities, I have fortified our network communications against potential vulnerabilities and attacks, ensuring a high level of security while maintaining optimal performance in our applications.
The snippet below provides an example of how to set up Bouncy Castle as the cipher provider and configure Netty to utilize OpenSSL. It’s crucial to pay close attention here, as improper configuration may lead Netty to fall back to Java SSL as a safety measure, which could potentially introduce unintended security vulnerabilities.
System.setProperty("io.netty.handler.ssl.openssl.library.path", "/path/to/openssl/libraries");
Security.addProvider(new BouncyCastleProvider());
// Create an OpenSSL-based SSL context
SslContext sslContext = SslContextBuilder.forServer()
.sslProvider(io.netty.handler.ssl.SslProvider.OPENSSL)
.build();
As you consider Java’s SSL handling and memory management nuances in Java, it is important to remain vigilant about security considerations, especially in sensitive applications like payment processing. By leveraging Netty’s features and implementing careful memory management practices, we can strive to maintain the integrity and security of our applications.
Akhileshwar Padala manages software engineering at Allied Electronics, Inc., an omnichannel company serving the retail petroleum industry since 1978. As head of engineering, Akhil designs and architects innovative fuel software solutions, integrating advanced technologies including Artificial Intelligence (AI), Computer Vision, Machine Learning (ML), Cloud Computing, and Internet of Things (IoT), to build robust software and hardware products for demanding fuel retailing environments. Among his many accomplishments, he developed a state-of-the-art Fuel Payment Application capable of processing payments in real-time within 10 milliseconds. Akhil is a recognized technology leader in the fuel retailing industry and a frequent speaker at industry conferences. He received his B.S. degree in Computer Science from Jawaharlal Nehru Technological University in Hyderabad, India, and earned a Master’s degree in Computer Science from Governors State University in University Park, Illinois (US).
