Model testing

Steps before releasing a new model in production

Step 1 : test the application

If you are creating a model for a new country, check what is current status of this items in the application.

  • Do we have the building polygons for the whole country : check on the map view in different places and in the database with the count

  • Do we have default building information (build years etc)

  • Do we have the addresses

  • Do we have the demographic/economic data

  • Do we have the building prototypes and other data related to the technical side of the buildings

 

Step 2 : evaluate the performance of the model

The target is to be sure that the model performance is on a sufficient level in larger test samples, to do that check the different metrics in ML hub :

  • Features importance

  • Accuracy

  • Bias

  • Summary statistics of the variables : mean, …

  • Average price by sqm by municipality / postcode

 

Step 3 : verify the model deployment

a) Model API

The target is to be sure that the API works and that it performs in expected way, you need to do that as soon as the model is ready.

2 things to test :

  • use the same data test as previously in the model performance and check if all the results are the same

  • use a new test file, a list of curl in fact, same at each release (with all locations, locations without data, all variables and values, all building types, different month for the same unit, location outside the coverage) and analyse the results : do the results make sense ?

For the second point, the purpose is to have in ML hub a table like below, where the curl are tested automatically and give the results.

 

Model API : value

Curl

 

Model API : value

Curl

Apartment Helsinki 30 m2

190000 €

{}

Apartment Helsinki 50 m2

210000 €

{}

Row house Helsinki

ERROR

{}

b) Import / Application API

The target is to test that the integration between the model and the application works. Also check that the application produces the same results as the model endpoint, use the same test cases as in 3a.

 

Model API : value

Application API : value

Test

 

Model API : value

Application API : value

Test

Apartment Helsinki 30 m2

190000 €

190000 €

OK

Apartment Helsinki 50 m2

210000 €

230000 €

PROBLEM

Row house Helsinki

ERROR

ERROR

PROBLEM

 

Step 4 : check by manual testing

The final step is to check that “everything” works, you can do that as soon as the back end integration is ready. This step is primordial when it’s a new country. You need to take some ad in the current market and check them in the application. Take a portfolio with enough units, more than 20 units and with :

  • Multiple locations

  • All variables

  • All building types

  • Valuation date

  • Locations without data

  • Locations outside the coverage

 

 

Now you can release the model