Protobuf – Improving the Communication Process of Marketplace Mediated Networks

Fyber’s Marketplace is a technology platform, enabling mobile app and game developers to sell their ad space inventory to advertisers, by adding Fyber’s Marketplace SDK to their apps which are installed on billions of mobile devices worldwide. 

 

In traditional ad mediation setups, Fyber’s SDK communicates directly with Fyber’s Exchange- a backend system that performs the programmatic auction, based on the following stages:

  1. Collection of data by the SDK 
  2. Completion of the auction stage 
  3. Returning HTML elements and JS codes wrapped in Fyber’s custom template.
    These are returned from the backend to the SDK, eventually rendering it and collecting data on the entire flow.

 

The Problem

In-app bidding has evolved in recent years as part of the ad-tech industry, allowing App developers running this setup to rely on a 3rd party SDK to mediate between Fyber’s SDK to the backend.

 

Two predefined string signals provide an indirect communication method, meaning the two can no longer communicate directly.

  • (The Token) passed by Fyber’s SDK to the 3rd party SDK
  • (The ADM) gave feedback from Fyber’s backend

 

These industry-standard signals are an integral part of all in-app bidding implementations. To support them, however, Fyber must adhere to a set of limitations:

  • Maximal signal sizes
  • Creation time latency
  • Obfuscation of business-sensitive data flowing through the 3rd party SDK.

 

These limitations obviate a need to find a desirable solution for using the right technology that supports infrastructure for future development.

 

The Solution

Set out below are the stages we went through to find the best solution to the problem.

The First Stage – Defining  the Structure

 

We first defined the structured data that all systems reading and writing said signals must be familiar with. We went over every data point currently supported both in the SDK and the backend. We optimized them by either deprecating obsolete material, enumerating constant data or shifting logic from the SDK to the backend. But this wasn’t enough – our tokens were still too big. We had to serialize our signals.

 

The Second Stage – Choosing a Serialization Technology

 

There are many options to choose from – Protocol Buffers, Thrift, Ion, Avro, JSON, XML and many more. After some benchmarking, we saw the best performance with Protobuf and decided to go with it.

 

Protobuf is an open-source data format created and popularized by Google. It is entirely cross-platform and allows developers to describe their data once and then generate matching code in many popular programming languages including Java, C++, Scala, Objective-C, Python, and C#. It also serves as the basis for Google’s custom Remote Processing Call protocol – gRPC.

 

Apart from the excellent performance, the ability to quickly and automatically generate multi-language code is essential to Fyber. Our Exchange backend system is written in a combination of Java and Scala; we have both an Android (Java) and iOS (Objective-C) SDK as well as a testing system (extensively used in this project) written in Kotlin.

 

The Third Stage – a Common Repository

 

Fyber’s software development philosophy dictates the SDK and backend, although engineered by teams from the same product line, be released separately from each other so as not to impede the development of either product. However, an SDK cannot be rolled back once published, therefore, users decide whether to upgrade their applications (and thus Fyber’s SDK) to newer versions. The backend must remain fully backward compatible with all SDK versions until they are no longer supported. The next step was building the infrastructure to support these abilities while working with Protobuf.

 

We created a common repository for the Protobuf files and a CI/CD process. Running backward compatibility and breaking changes checks, lint checks generate code, artifacts and sometimes even create pull requests for upgrading the new artifacts in each relevant project.

 

Furthermore, all examples are from the Scala process responsible for performing these validations and generating backend JAVA code. Similar methods exist for generating classes in other languages, to be used in the different systems involved. 

 

We use an SBT plugin called sbt-protobuf, transforming .proto files into classes. To enable that plugin, we added the following to the plugins.sbt file:

 

 

And the following to the build.sbt file:

 

Backward Compatibility Validation

 

To ensure the backend is always backward compatible with all previous SDK versions, we have to make sure the latest .proto file is backward compatible with all .proto files ever created. This isn’t easy to achieve, so instead, we decide on the following.

  1. Check compatibility with the latest .proto file, and ensure no one can update a .proto file without running backward compatibility rules.
  2. Define all fields as optional, and never delete any area from .proto files. This is already a rare case that generally only happens when a significant SDK version is no longer supported, so we made this logical decision.

 

By doing both, we made ourselves transitively backward compatible. Now, we had to build an infrastructure for running these backward compatibility rules. There are numerous rules to enforce – making sure nothing was removed; field types hadn’t changed, all fields are defined as optional, etc. Unfortunately, there’s no official solution to completely solve this issue.

 

We were about to write such a tool ourselves when we discovered Buf. Buf is a powerful tool for Protobuf based API development that provides, among other things, a linter and breaking changes detection capabilities. We incorporated both into our CI/CD process. 

 

The linter and the breaking changes detection tools split their rules into categories allowing you to choose the proper authorities that match your code and organizational rquirements. They are configured through a .yaml file where you can set which type to use, which rules to exclude and which files should be ignored – either entirely or from certain rule checks.

 

This is our buf.yaml file:

 

 

The CI/CD Process

 

At Fyber, we use Jenkins for running our CI pipeline. The following steps, defined in the Jenkinsfile, are executed every time a change is pushed to our protobuf repository:

 

  • Check whether the file has changed or not

 

  • Run lint validations to enforce a good API design and structure

 

  • Extract the most recent version tag from Git and run breaking changes checks against it

 

  • If no breaking changes were detected
    • Increment the version and push a new tag to Git
    • Compile the .proto file using the sbt-protobuf plugin. This can also be performed using Buf’s generate command.
    • Publish a new version of the code to our Artifactory

 

Once the CI pipeline is completed successfully, pull requests are opened to the different projects by making calls to Fyber’s Git repository API, and each engineering team is then responsible for its own CD pipeline.

Conclusion

In this post, we explored the background and motivation for using Protocol Buffers at Fyber, as well as the reasons we decided to choose Protobuf as our serialization technology due to its excellent performance and the ability to quickly and automatically generate multi-language code.

Read these next