IAP Submission Fixture

• programming
• nodejs
• javascript
• unity
• portfolio

Description

This is a special tool written with NodeJS to submit IAP Items to apple’s new App Store Connect API.

Business Purpose

When BBB4 was originally programmed, Apple used an XML file for submitting In App Purchase items. Once we’d finished the core app, Apple changed their submission policy to use their new and poorly documented App Store Connect API (ASCA from now on), so I was forced to write a new tool to submit our IAP items to Apple.

Structure

Rather than complicating the existing system (and introducing new bugs), I decided to build an external tool that would consume the old data and translate it to the new API. I assumed that the required data would be basically the same, which in retrospect may have been fallacious, but it turned out that I didn’t need any more information to submit IAP items via the new API than I had already generated in the original XML technique.

The fixture consumes an apple ITMSP file. This “file” on apple is actually a directory with an XML file in it with a ton of metadata, as well as a a single review screenshot for each IAP Item. Because the Bridal bouquet Builder app has hundreds of items, it’s necessary to upload them all with an automated process to avoid manual errors and save time for the client.

The core module is Upload.JS. This handles each step of the process synchronously. Each step is broken out into its own operation file. Each operation is an asynchronous function (because the API calls are asynchronous to begin with), but is wrapped in an await to guarantee each step completes before the next begins. A number of helper functions are stored in a utilities directory.

Process

The program is a single NodeJS project. It has a single entry point and is called with npm run upload🧱¹. On startup, it looks for a config file that contains the following:

• the base ITMSP directory
• the API key for the App Store Connect API
• the key id
• the issuer id of the key
• the app id

The config isn’t committed to source control, to protect our API keys.

Once the config is parsed, we create a rest client using the node-rest-client package, create a new token manager, and pass all three down to the Upload function.

The Upload function then performs the following steps.

1. GetIapItems() retrieves a list of all IAP items in the app from ASCA.
2. Reads the ITMSP XML file for the proposed IAP Items. Since the XML file is generated by the BBB4 build process, it’s the source of truth.
3. Find new IAP items by comparing the ITMSP XML and the ASCA data. Any IAP items in the ITMSP but not on ASCA are new, and are added to ASCA. Any IAP items on ASCA but not in the ITMSP are deprecated and skipped by other processes
4. GetIapItems() again to ensure we’ve got the new item ids from ASCA.
5. Screenshots() asks ASCA for the existence and checksum of the screenshot associated with each unsubmitted IAP item.
   • We skip submitted items because once an item is submitted, it doesn’t need a new review screenshot to be submitted.
   • Screenshots() compares the checksums of unsubmitted items. It deletes stale screenshots, then creates new screenshot requests for each new item.
   • It then performs uploads according to the instructions sent back in the response for the “create-new” request.
   • Once the upload is done, it commits the screenshot to close the upload process.
6. Localization() simply compares the ASCA description and name for each item and sends changes for those that need to be changed.
7. Pricing() is a nightmarish hellscape.
   • Each IAP item on ASCA has N pricing schedules.
   • Each pricing schedule has something like 16,625 different price tier options
   • The actual customer price is not provided in any of the IAP data.
   • The ID or URL to the specific price tier is never given and is not associated with any part of the IAP Item in any way.
   • Here’s what I had to do for every single IAP item, of ~300
     • Get a single IAP Item’s id (easy)
     • Filter the price schedules by the territory of USA (the territories aren’t obviously listed on the response for all schedules, so I had no idea this was needed)
     • Receive about 200 price points
     • loop through the 200 price points to find a single price point with the metadata startDate: null² which is the correct price for the IAP Item and also completely fucking insane³
     • simply get the pricePoint.attributes.priceTier. There, see? very easy.
   • Did I mention that almost none of this is documented, because Apple only documents the first-order API calls for ASCA? Yeah. So every time you receive a response and that response has a call URL in it, that call is undocumented.
8. GetIapItems() again to ensure we’ve got the latest data on all our IAP items from ASCA
9. SubmitUnsumbitted() sorts all our items into three buckets:
   • Approved: we no longer care about these. They’re done.
   • Waiting for Review: we don’t care about these in this fixture, because they’ll be submitted with the app the next time the app itself is submitted for review
   • Ready for submission: We create a submission request for each of these and send it to ASCA.
     • because we valdidate the response of each of these, the program will throw a fit if one of the submissions doesn’t complete, and we know we’ve got a problem.

Challenges

Unnecessary Complication

For reference, the old ITMSP looked like this.

<package version = "software5.7" xmlns = "http://apple.com/itunes/importer" >
  <provider> XXXXXXXXXX </provider>
  <team_id> XXXXXXXXXX </team_id>
  <software>
    <vendor_id >bbb4</vendor_id>
    <software_metadata>
      <in_app_purchases>
        <in_app_purchase>
          <product_id>astilbe___blush_pink</product_id>
          <reference_name>astilbe___blush_pink</reference_name>
          <type>non-consumable</type>
          <products>
            <product>
              <cleared_for_sale>true</cleared_for_sale>
              <wholesale_price_tier>1</wholesale_price_tier>
            </product>
          </products>
          <locales>
            <locale name="en-US">
              <title>AstiBlusPink</title>
              <description>Astilbe - Blush Pink</description>
            </locale>
          </locales>
          <review_screenshot>
            <file_name>astilbe___blush_pink.png</file_name>
            <size>158693</size>
            <checksum>3324AEE777E810FE791AD88F79FCEDD2</checksum>
          </review_screenshot>
        </in_app_purchase>
...

Pretty simple, right?

Well here’s the same data for Google.

Product ID,Published State,Purchase Type,Auto Translate,Locale; Title; Description,Auto Fill Prices,Price,Pricing Template ID
astilbe___blush_pink,published,managed_by_android,false,en_US; Astilbe - Blush Pink; Astilbe - Blush Pink,true,990000,

That’s right. I spent several weeks reverse constructing the original ITMSP XML file from a dummy app I built. The new ASCA version has taken months to figure out, including several weeks of waiting for Apple to get back to me with exactly null response.

And the entire google IAP catalog system I wrote in about a day.

A

DAY.

It’s a MF-ing CSV file.

Saying that this is overcomplicated is like say that the observable universe is “pretty big, I guess”.

Documentation

It really doesn’t help that this API’s documentation consists of little more than the name of each TOP LEVEL⁴ api call, what the names of the fields it takes are, and the numbers of the error codes. No explanation whatsoever about each item. Dozens of times I looked at the documentation and it would just ask for “id.”

What ID? The ID of the IAP Item I’m trying to change? The ID of the ID of the element of the IAP item I’m trying to change? The ID of a new create request for the element of the IAP item I’m trying to change? The ID of the Price Schedule? Or the Price Point? What’s the difference?

I’m mildly surprised this didn’t get me to quit programming altogether.

No Batch Options for Pricing

The pricing loop takes about 13-23 minutes now. That’s because I have to check every single price individually. There’s no call for GET https://api.appstoreconnect.apple.com/v2/inAppPurchases/${appleIapId}/GetIapPricesPleaseAsshat or anything.

You can see in the screenshot that almost everything except the pricing gets done in a couple of seconds before pricing starts. Each call for GetIapItems() takes between 7-45 seconds. Pricing takes almost that long per item.

Will to Live

This was such a headache that I almost quit. The whole project. At the end. When the entire app was already done. Every. single. call. was a nightmare to figure out, and it was like turning over a rotten log with each maggot being a new call to make.

Points of Interest

Token Manager

So partway through pricing, I got an error that my request was not authorized. It turned out that being forced to do pricing item-by-item took so long that my token had timed out. The max token length is 20 minutes. So of course I had to overhaul the project and replace every call to the simple token object generated at the beginning with a new tokenManager object and ask for the current token each time.

• In the beginning, it’s created with CreateTokenManager. That returns a new tokenManager object that we reference later.
• Each time a token is created (including at the beginning) the manager stores the creation date, and the token.
• When a token is requested using tokenManager.GetToken() the manager checks the current time against the last creation time.
  • if the token is older than 19 minutes, it creates a new token and resets the creation time.
  • if the token is more recent than 19 minutes, it simply returns the current token.

PromiseRestCall

When making an http request in node, normally, you make a call, provide a callback, and let the callback do the rest.

Since this whole thing needs to be done in discrete steps, I needed things to be much more serial than parallel. So I built a function called PromiseRestCall that promisifies the node-rest-client calls and checks each one for the right status code in the response. If it gets the wrong response, it throws and error.

In higher-level functions, I can then await those PromiseRestCalls and boom, a synchronous system that takes forever but does it all in the right order.

It’s totally plausible that I could let some parts (at least the pricing loop) be parallel, but I don’t really have the time to add that optimization at this point. It works, and for now that’s good enough.

Screenshot Uploading

So the process goes like this:

1. get all the screenshots from ASCA
2. compare each of them to the ones from ITMSP
3. delete stale screenshots*
4. get all items with empty screenshots
5. for each new screenshot…
   1. Create a new upload request
   2. receive upload instructions*
   3. upload data*
   4. commit data

*Not documented. Lammogotem⁵

So aside from the usual problems with undocumented API calls, the real problem here is that the upload never returned anything useful. I always got an empty buffer no matter what I tried. I even tried sending incorrect and malformed data (repeatedly) to this call, and it was just like ¯\_(ツ)_/¯

Then I assumed that for some reason it the buffer would always be empty. I don’t know why, but maybe that’s just what it does. So I moved on to the commit the data step. That failed, of course, but didn’t elaborate as to why. ¯\_(ツ)_/¯

After a week or two of emailing apple and almost giving up to do the whole thing manually everytime, I finally got mad one monday morning and simply wrote the HTTP request method from scratch.

Remember node-rest-call🧱⁶? Well, it turns out that something about how it formats PUT requests was utterly incompatible with Apple’s App Store Connect API. So rewriting node-rest-call instantly fixed the problem.

Critiques

Apple.

What I learned

Do not Apple.