A/B Test

1. Functional Overview

An A/B test is a comparative experiment that divides users into control and test groups in proportion, applying different configurations to each group. By comparing results using distinct advertising strategies across test groups, you can determine which strategy yields better performance.

Taku A/B test supports the following features:

• Multi-group support: Up to 10 test groups can be configured. 
• Dual-scenario testing: Supports placement A/B test and segment A/B test. 
• Multiple ad space aggregation: Placement A/B tests allow selection of multiple Taku placement simultaneously; segment A/B tests permit selection of only one segment per test. Taku also supports viewing A/B test data at the application level.
• One-click optimization: After completing an A/B test, users can instantly select the optimal test group.
• Historical data tracking: Users can review historical A/B test group data and configuration settings.

2. Operation Guide

2.1 A/B Test usage process

2.1.1 Create A/B Test

You can create tests through either Mediation or A/B Test.  

[Mediation] → [A/B Test] → [Add A/B Test]

[Advanced]→[A/B Test] → [+ A/B Test]

2.1.2 Add Detailed AB Test Configuration

1. Test Types

AB test includes Placement A/B Test and Segment A/B Test. Click the question mark【?】to the right of the subheading to view dimension differences. 

If you create using Segment, you can set up multiple A/B groups (up to 10 groups) and select multiple Taku placement for A/B test within the same application (segment A/B test supports only single segment creation).

Note: Only one type of A/B test can be created under the same placement.

2. Traffic distribution

It is set proportionally. To achieve automatic equal distribution, turn on the switch to the right of the title.

3. Configuration Group

After the A/B Test is created, the original WaterFall configuration of the mediation will be automatically synchronized to the group with the smallest ID in the A/B Test (for example, if three groups A\B\C are configured, it will be automatically copied to group A). You can use the Copy WaterFall function to synchronize the original WaterFall configuration of the mediation of other groups (B/C).

4. Effective Date

You can choose Now, Manual, or Assign.

After completing all configurations, click Submit to create successfully.

If you select Manual, the test will enter the "Waiting" status; you can click Start Test on the Mediation or AB Test page to launch it officially.

2.1.3 Grouping Settings: WaterFall

After completing the A/B test, you can configure traffic groups and ad sources for each A/B test group in Mediation. Use the Create Segment/Add AD Source - Copy from Another Segment to quickly complete WaterFall configuration.

2.1.4 Edit Test

After filtering the corresponding A/B test on the Mediation page, select More Actions to edit the A/B test configuration.

Or [Advanced] - [A/B Test] - [Edit] to adjust the configuration

2.1.5 End Testing  

When concluding A/B test, you can select All Placements to choose the optimal grouping strategy with one click, or select Choose one or more placement(s) to choose the optimal grouping strategy for each spot individually.

2.2 A/B Test Data Viewing Method  

2.2.1 A/B Test Section Homepage View  

The AB testing page consists of two lists: the AB Testing list (showing ongoing or pending tests) and the Historical AB Tests list (showing completed tests). The historical list directly displays test results.

The list item can be expanded to view key data comparisons during the experiment (e.g. Est. Revenue, eCPM changes).

Click Placement or Segment in the list to view details.

For an ongoing AB test, go to the corresponding placement on the Mediation page; for historical AB tests, go to the Historical Traffic Data page.

2.2.2 Ongoing A/B Test

• View on Mediation Page

You can filter date ranges and test names on the Mediation page to view detailed data for each group; click Data Detail to navigate to the Details of A/B Test Data page.

• View the A/B test page.

Click View Data to navigate to the Details of A/B Test Data page.

For multi-placement testing, you can view each placement individually or access total data for all placements. The dashboard displays summary data by testing phase by default, but you can switch to using estimated application data. A dropdown menu allows comparison of specific metrics and supports viewing daily data trends.

A/B Test supports data metrics are detailed in Part 3 of this document.

2.2.3 Historical AB Test Configuration and Data

• View on the Mediation Page

Or go to [Report] - [Full Report] - [Dimension], and select [AB Test] to view

In historical Traffic data, you can filter all historical A/B test data by the A/B test group name.

Note: After distinguishing A/B groups, please pay attention to metrics such as "estimated data" when observing data. For example: estimated eCPM, estimated revenue, and estimated ARPU.

The metrics with "API" in the A/B group are data of the entire code bit dimension, that is, the actual revenue of this code bit. In fact, no split will occur.

For example: CSJ code position "xxxx" is configured in an A/B Test with a 60/40 distribution ratio, and the actual revenue API is 100 RMB. The estimated revenue in group A is 60 RMB, and in group B it is 40 RMB. However, the revenue API in both groups will be consistent with the "actual revenue pulled back from the mediation network".

That is, whether watching in group A or group B, the revenue API of CSJ code position "xxxx" will be equal to 100.

• View the AB test page.

To view the historical A/B test list, click View Data to navigate to the A/B Test Data Details page.

2.2.4 View A/B Test Data in the Full Report

After the A/B Test is completed, Taku SDK strategy cache problem. Unselected grouping strategies will generate legacy data within a certain period of time. If you find that the placement data queried in the full report is inconsistent with the data impressioned in the mediation, you can select AB test in the data dimension of the full report to further determine whether the data difference is caused by the old AB test strategy.

3. A/B test supports data metrics

4. FAQ

(1) How to allocate devices for the A/B Test function?

If the AB strategy remains unchanged, the device ID will be fixedly allocated to one of the groups. If the strategy changes, such as modifying the volume cut permission, the device ID will be reallocated.

(2) The cut-in ratio set in the A/B Test does not match the cut-in ratio of the actual placement, but the DAU cut-in ratio of the App is close to the cut-in ratio set in the A/B Test?

Under normal circumstances, it is normal for the gap between the cut-in ratio set in the AB experiment and the cut-in ratio of the placements in the experiment to be 5%.

When the gap exceeds 5%, you can check whether there are any test groups in all the placements participating in the experiment that have no ad source configured. When the test group is not configured with any available ad source, Taku adopts the "Automatically allocate impression ads" strategy by default, that is, the traffic of this test group will be cut to other test groups with available ad sources, thereby affecting the actual cut-in ratio of the placement.

(3) When a single placement participates in an A/B Test, the A/B Test results of the placement are inconsistent with the A/B Test results of the App dimension?

Under normal circumstances, the A/B Test results of a single placement will be consistent with the A/B Test results of the App dimension. When there is a discrepancy, it may be that the samples participating in the A/B Test are unevenly cut.

It is recommended that you close the existing A/B Test and create a new A/B Test. In the new A/B Test, you can consider using the AABB test (two AA groups as the control group and two BB groups as the test group). When the test results of the two AA groups are close, it means that the A/B Test samples are evenly cut. This A/B Test can consider the comparison effect of the AB groups.

(4) When multiple placements participate in the experiment, the summary data of each placement participating in the experiment is inconsistent with the experimental results of the App dimension?

When this happens, it means that the test results of your current placements participating in the A/B Test and the overall App dimension show the "Simpson's Paradox". In this case, you can select the optimal strategy test grouping strategy based on the A/B Test results of each placement.

At the same time, it is recommended that you create different A/B Tests for the above placements participating in the A/B Test and re-verify the existing test conclusions.